Investigating lateral movements with Amazon Detective investigation and Security Lake integration
According to the MITRE ATT&CK framework, lateral movement consists of techniques that threat actors use to enter and control remote systems on a network. In Amazon Web Services (AWS) environments, threat actors equipped with illegitimately obtained credentials could potentially use APIs to interact with infrastructures and services directly, and they might even be able to use APIs to evade defenses and gain direct access to Amazon Elastic Compute Cloud (Amazon EC2) instances. To help customers secure their AWS environments, AWS offers several security services, such as Amazon GuardDuty, a threat detection service that monitors for malicious activity and anomalous behavior, and Amazon Detective, an investigation service that helps you investigate, and respond to, security events in your AWS environment.
<p>After the service is turned on, Amazon Detective automatically collects logs from your AWS environment to help you analyze and investigate security events in-depth. At <a href="https://aws.amazon.com/blogs/aws/top-announcements-of-aws-reinvent-2023/" target="_blank" rel="noopener">re:Invent 2023</a>, Detective released <a href="https://docs.aws.amazon.com/detective/latest/userguide/investigations-about.html" target="_blank" rel="noopener">Detective Investigations</a>, a one-click investigation feature that automatically investigates <a href="https://aws.amazon.com/iam/" target="_blank" rel="noopener">AWS Identity and Access Management (IAM)</a> users and roles for indicators of compromise (IoC), and <a href="https://docs.aws.amazon.com/detective/latest/userguide/securitylake-integration.html" target="_blank" rel="noopener">Security Lake integration</a>, which enables customers to retrieve log data from <a href="https://aws.amazon.com/security-lake/" target="_blank" rel="noopener">Amazon Security Lake</a> to use as original evidence for deeper analysis with access to more detailed parameters.</p>
<p>In this post, you will learn about the use cases behind these features, how to run an investigation using the Detective Investigation feature, and how to interpret the contents of investigation reports. In addition, you will also learn how to use the Security Lake integration to retrieve raw logs to get more details of the impacted resources.</p>
<h2>Triage a suspicious activity</h2>
<p>As a security analyst, one of the common workflows in your daily job is to respond to suspicious activities raised by security event detection systems. The process might start when you get a ticket about a GuardDuty finding in your daily operations queue, alerting you that suspicious or malicious activity has been detected in your environment. To view more details of the finding, one of the options is to use the GuardDuty console.</p>
<p>In the GuardDuty console, you will find more details about the finding, such as the account and AWS resources that are in scope, the activity that caused the finding, the IP address that caused the finding and information about its possible geographic location, and times of the first and last occurrences of the event. To triage the finding, you might need more information to help you determine if it is a false positive.</p>
<p>Every GuardDuty finding has a link labeled <strong>Investigate with Detective</strong> in the details pane. This link allows you to pivot to the Detective console based on aspects of the finding you are investigating to their respective entity profiles. The finding <strong>Recon:IAMUser/MaliciousIPCaller.Custom</strong> that’s shown in Figure 1 results from an API call made by an IP address that’s on the custom threat list, and GuardDuty observed it made API calls that were commonly used in reconnaissance activity, which commonly occurs prior to attempts at compromise. To investigate this finding, because it involves an IAM role, you can select the <strong>Role session</strong> link and it will take you to the role session’s profile in the Detective console.</p>
<div id="attachment_34169" class="wp-caption aligncenter">
<img aria-describedby="caption-attachment-34169" src="https://infracom.com.sg/wp-content/uploads/2024/05/img1-5.png" alt="Figure 1: Example finding in the GuardDuty console, with Investigate with Detective pop-up window" width="780" class="size-full wp-image-34169">
<p id="caption-attachment-34169" class="wp-caption-text">Figure 1: Example finding in the GuardDuty console, with <strong>Investigate with Detective</strong> pop-up window</p>
</div>
<p>Within the AWS Role session profile page, you will find security findings from GuardDuty and <a href="https://aws.amazon.com/security-hub/" target="_blank" rel="noopener">AWS Security Hub</a> that are associated with the AWS role session, API calls the AWS role session made, and most importantly, new behaviors. Behaviors that deviate from expectations can be used as indicators of compromises to give you more information to determine if the AWS resource might be compromised. Detective highlights new behaviors first observed during the <a href="https://docs.aws.amazon.com/detective/latest/userguide/scope-time-managing.html" target="_blank" rel="noopener">scope time</a> of the events related to the finding that weren’t observed during the Detective baseline time window of 45 days.</p>
<p>If you switch to the <strong>New behavior</strong> tab within the AWS role session profile, you will find the <strong>Newly observed geolocations</strong> panel (Figure 2). This panel highlights geolocations of IP addresses where API calls were made from that weren’t observed in the baseline profile. Detective determines the location of requests using <a href="https://support.maxmind.com/hc/en-us/sections/4407519834267-IP-Geolocation" target="_blank" rel="noopener">MaxMind GeoIP</a> databases based on the IP address that was used to issue requests.</p>
<div id="attachment_34170" class="wp-caption aligncenter">
<img aria-describedby="caption-attachment-34170" src="https://infracom.com.sg/wp-content/uploads/2024/05/img2-5.png" alt="Figure 2: Detective’s Newly observed geolocations panel" width="780" class="size-full wp-image-34170">
<p id="caption-attachment-34170" class="wp-caption-text">Figure 2: Detective’s <strong>Newly observed geolocations</strong> panel</p>
</div>
<p>If you choose <strong>Details</strong> on the right side of each row, the row will expand and provide details of the API calls made from the same locations from different AWS resources, and you can drill down and get to the API calls made by the AWS resource from a specific geolocation (Figure 3). When analyzing these newly observed geolocations, a question you might consider is why this specific AWS role session made API calls from <strong>Bellevue, US</strong>. You’re pretty sure that your company doesn’t have a satellite office there, nor do your coworkers who have access to this role work from there. You also reviewed the <a href="https://aws.amazon.com/cloudtrail/" target="_blank" rel="noopener">AWS CloudTrail</a> management events of this AWS role session, and you found some unusual API calls for services such as IAM.</p>
<div id="attachment_34171" class="wp-caption aligncenter">
<img aria-describedby="caption-attachment-34171" src="https://infracom.com.sg/wp-content/uploads/2024/05/img3-5.png" alt="Figure 3: Detective’s Newly observed geolocations panel expanded on details" width="780" class="size-full wp-image-34171">
<p id="caption-attachment-34171" class="wp-caption-text">Figure 3: Detective’s <strong>Newly observed geolocations</strong> panel expanded on details</p>
</div>
<p>You decide that you need to investigate further, because this role session’s anomalous behavior from a new geolocation is sufficiently unexpected, and it made unusual API calls that you would like to know the purpose of. You want to gather anomalous behaviors and high-risk API methods that can be used by threat actors to make impacts. Because you’re investigating an AWS role session rather than investigating a single role session, you decide you want to know what happened in other role sessions associated with the AWS role in case threat actors spread their activities across multiple sessions. To help you examine multiple role sessions automatically with additional analytics and threat intelligence, Detective introduced the Detective Investigation feature at re:Invent 2023.</p>
<h2>Run an IAM investigation</h2>
<p>Amazon Detective Investigation uses machine learning (ML) models and AWS threat intelligence to automatically analyze resources in your AWS environment to identify potential security events. It identifies tactics, techniques, and procedures (TTPs) used in a potential security event. The MITRE ATT&CK framework is used to classify the TTPs. You can use this feature to help you speed up the investigation and identify indicators of compromise and other TTPs quickly.</p>
<p>To continue with your investigation, you should investigate the role and its usage history as a whole to cover all involved role sessions at once. This addresses the potential case where threat actors assumed the same role under different session names. In the AWS role session profile page that’s shown in Figure 4, you can quickly identify and pivot to the corresponding AWS role profile page under the <strong>Assumed role</strong> field.</p>
<div id="attachment_34172" class="wp-caption aligncenter">
<img aria-describedby="caption-attachment-34172" src="https://infracom.com.sg/wp-content/uploads/2024/05/img4-5.png" alt="Figure 4: Detective’s AWS role session profile page" width="780" class="size-full wp-image-34172">
<p id="caption-attachment-34172" class="wp-caption-text">Figure 4: Detective’s AWS role session profile page</p>
</div>
<p>After you pivot to the AWS role profile page (Figure 5), you can run the automated investigations by choosing <strong>Run investigation</strong>.</p>
<div id="attachment_34173" class="wp-caption aligncenter">
<img aria-describedby="caption-attachment-34173" src="https://infracom.com.sg/wp-content/uploads/2024/05/img5.jpg" alt="Figure 5: Role profile page, from which an investigation can be run" width="780" class="size-full wp-image-34173">
<p id="caption-attachment-34173" class="wp-caption-text">Figure 5: Role profile page, from which an investigation can be run</p>
</div>
<p>The first thing to do in a new investigation is to choose the time scope you want to run the investigation for. Then, choose <strong>Confirm</strong> (Figure 6).</p>
<div id="attachment_34174" class="wp-caption aligncenter">
<img aria-describedby="caption-attachment-34174" src="https://infracom.com.sg/wp-content/uploads/2024/05/img6.jpg" alt="Figure 6: Setting investigation scope time" width="780" class="size-full wp-image-34174">
<p id="caption-attachment-34174" class="wp-caption-text">Figure 6: Setting investigation scope time</p>
</div>
<p>Next, you will be directed to the <strong>Investigations</strong> page (Figure 7), where you will be able to see the status of your investigation. Once the investigation is done, you can choose the hyperlinked investigation ID to access the investigation report.</p>
<div id="attachment_34175" class="wp-caption aligncenter">
<img aria-describedby="caption-attachment-34175" src="https://infracom.com.sg/wp-content/uploads/2024/05/img7-1.png" alt="Figure 7: Investigations page, with new report" width="780" class="size-full wp-image-34175">
<p id="caption-attachment-34175" class="wp-caption-text">Figure 7: Investigations page, with new report</p>
</div>
<p>Another way to run an investigation is to choose <strong>Investigations</strong> on the left menu panel in the Detective console, and then choose <strong>Run investigation</strong>. You will then be taken to the page where you will specify the AWS role Amazon Resource Number (ARN) you’re investigating, and the scope time (Figure 8). Then you can choose <strong>Run investigation</strong> to commence an investigation.</p>
<div id="attachment_34176" class="wp-caption aligncenter">
<img aria-describedby="caption-attachment-34176" src="https://infracom.com.sg/wp-content/uploads/2024/05/img8-1.png" alt="Figure 8: Configuring a new investigation from scratch rather than from an existing finding" width="780" class="size-full wp-image-34176">
<p id="caption-attachment-34176" class="wp-caption-text">Figure 8: Configuring a new investigation from scratch rather than from an existing finding</p>
</div>
<p>Detective also offers <a href="https://docs.aws.amazon.com/detective/latest/APIReference/API_StartInvestigation.html" target="_blank" rel="noopener">StartInvestigation</a> and <a href="https://docs.aws.amazon.com/detective/latest/APIReference/API_GetInvestigation.html" target="_blank" rel="noopener">GetInvestigation</a> APIs for running Detective Investigations and retrieving investigation reports programmatically.</p>
<h2>Interpret the investigation report</h2>
<p>The investigation report (Figure 9) includes information on anomalous behaviors, potential TTP mappings of observed CloudTrail events, and indicators of compromises of the resource (in this example, an IAM principal) that was investigated.</p>
<p>At the top of the report, you will find a severity level computed based on the observed behaviors during the scope window, as well as a summary statement to give you a quick understanding of what was found. In Figure 9, the AWS role that was investigated engaged in the following unusual behaviors:</p>
<ul>
<li><strong>Seven tactics</strong> showing that the API calls made by this AWS role were mapped to seven tactics of the MITRE ATT&CK framework.</li>
<li><strong>Eleven cases of impossible travel</strong> representing API calls made from two geolocations that are too far apart for the same user to have physically travelled between them to make the calls from both, within the time span involved.</li>
<li><strong>Zero flagged IP</strong> addresses. Detective would flag IP addresses that are considered suspicious according to its threat intelligence sources.</li>
<li><strong>Two new Autonomous System Organizations (ASOs)</strong> which are entities with assigned Autonomous System Numbers (ASNs) as used in Border Gateway Protocol (BGP) routing.</li>
<li><strong>Nine new user agents</strong> were used to make API calls that weren’t observed in the 45 days prior to the events being investigated.</li>
</ul>
<p>These indicators of compromise represent unusual behaviors that have either not been observed before in the AWS account involved or that are intrinsically considered high risk. The following summary panel includes the report that shows a detailed breakdown of the investigation results.</p>
<p>Unusual activities are important factors that you should look for during investigations, and sudden behavior change can be a sign of compromise. When you’re investigating an AWS role that can be assumed by different users from different AWS Regions, you are likely to need to examine activity at the granularity of the specific AWS role session that made the APIs calls. Within the report, you can do this by choosing the hyperlinked role name in the summary panel, and it will take you to the AWS role profile page.</p>
<div id="attachment_34177" class="wp-caption aligncenter">
<img aria-describedby="caption-attachment-34177" src="https://infracom.com.sg/wp-content/uploads/2024/05/img9.png" alt="Figure 9: Investigation report summary page" width="780" class="size-full wp-image-34177">
<p id="caption-attachment-34177" class="wp-caption-text">Figure 9: Investigation report summary page</p>
</div>
<p>Further down on the investigation report is the <strong>TTP Mapping from CloudTrail Management Events</strong> panel. Detective Investigations maps CloudTrail events to the MITRE ATT&CK framework to help you understand how an API can be used by threat actors. For each mapped API, you can see the tactics, techniques, and procedures it can be used for. In Figure 10, at the top there is a summary of TTPs with different severity levels. At the bottom is a breakdown of potential TTP mappings of observed CloudTrail management events during the investigation scope time.</p>
<p>When you select one of the cards, a side panel appears on the right to give you more details about the APIs. It includes information such as the IP address that made the API call, the details of the TTP the API call was mapped to, and if the API call succeeded or failed. This information can help you understand how these APIs can potentially be used by threat actors to modify your environment, and whether or not the API call succeeded tells you if it might have affected the security of your AWS resources. In the example that’s shown in Figure 10, the IAM role successfully made API calls that are mapped to <a href="https://attack.mitre.org/tactics/TA0008/" target="_blank" rel="noopener">Lateral Movement</a> in the ATT&CK framework.</p>
<div id="attachment_34178" class="wp-caption aligncenter">
<img aria-describedby="caption-attachment-34178" src="https://infracom.com.sg/wp-content/uploads/2024/05/img10.jpg" alt="Figure 10: Investigation report page with event ATT CK mapping" width="780" class="size-full wp-image-34178">
<p id="caption-attachment-34178" class="wp-caption-text">Figure 10: Investigation report page with event ATT CK mapping</p>
</div>
<p>The report also includes additional indicators of compromise (Figure 11). You can find these if you select the <strong>Indicators</strong> tab next to <strong>Overview</strong>. Within this tab, you can find the indicators identified during the scope time, and if you select one indicator, details for that indicator will appear on the right. In the example in Figure 11, the IAM role made API calls with a user agent that wasn’t used by this IAM role or other IAM principals in this account, and indicators like this one show sudden behavior change of your IAM principal. You should review them and identify the ones that aren’t expected. To learn more about indicators of compromise in Detective Investigation, see the <a href="https://docs.aws.amazon.com/detective/latest/userguide/detective-investigations.html#investigations-report-understand" target="_blank" rel="noopener">Amazon Detective User Guide</a>.</p>
<div id="attachment_34179" class="wp-caption aligncenter">
<img aria-describedby="caption-attachment-34179" src="https://infracom.com.sg/wp-content/uploads/2024/05/img11.jpg" alt="Figure 11: Indicators of compromise identified during scope time" width="780" class="size-full wp-image-34179">
<p id="caption-attachment-34179" class="wp-caption-text">Figure 11: Indicators of compromise identified during scope time</p>
</div>
<p>At this point, you’ve analyzed the new and unusual behaviors the IAM role made and learned that the IAM role made API calls using new user agents and from new ASOs. In addition, you went through the API calls that were mapped to the MITRE ATT&CK framework. Among the TTPs, there were three API calls that are classified as lateral movements. These should attract attention for the following reasons: first, the purpose of these API calls is to gain access to the EC2 instance involved; and second, <span>ec2-instance-connect:SendSSHPublicKey</span> was run successfully.</p>
<p>Based on the procedure description in the report, this API would grant threat actors temporary SSH access to the target EC2 instance. To gather original evidence, examine the raw logs stored in Security Lake. Security Lake is a fully managed security data lake service that automatically centralizes security data from AWS environments, SaaS providers, on-premises sources, and other sources into a purpose-built data lake stored in your account.</p>
<h2>Retrieve raw logs</h2>
<p>You can use Security Lake integration to retrieve raw logs from your Security Lake tables within the Detective console as original evidence. If you haven’t enabled the integration yet, you can follow the <a href="https://docs.aws.amazon.com/detective/latest/userguide/securitylake-integration.html" target="_blank" rel="noopener">Integration with Amazon Security Lake</a> guide to enable it. In the context of the example investigation earlier, these logs include details of which EC2 instance was associated with the <span>ec2-instance-connect:SendSSHPublicKey</span> API call. Within the AWS role profile page investigated earlier, if you scroll down to the bottom of the page, you will find the <strong>Overall API call volume</strong> panel (Figure 12). You can search for the specific API call using the <strong>Service</strong> and <strong>API method</strong> filters. Next, choose the magnifier icon, which will initiate a Security Lake query to retrieve the raw logs of the specific CloudTrail event.</p>
<div id="attachment_34180" class="wp-caption aligncenter">
<img aria-describedby="caption-attachment-34180" src="https://infracom.com.sg/wp-content/uploads/2024/05/img12.png" alt="Figure 12: Finding the CloudTrail record for a specific API call held in Security Lake" width="780" class="size-full wp-image-34180">
<p id="caption-attachment-34180" class="wp-caption-text">Figure 12: Finding the CloudTrail record for a specific API call held in Security Lake</p>
</div>
<p>You can identify the target EC2 instance the API was issued against from the query results (Figure 13). To determine whether threat actors actually made an SSH connection to the target EC2 instance as a result of the API call, you should examine the EC2 instance’s profile page:</p>
<div id="attachment_34181" class="wp-caption aligncenter">
<img aria-describedby="caption-attachment-34181" src="https://infracom.com.sg/wp-content/uploads/2024/05/img13.png" alt="Figure 13: Reviewing a CloudTrail log record from Security Lake" width="780" class="size-full wp-image-34181">
<p id="caption-attachment-34181" class="wp-caption-text">Figure 13: Reviewing a CloudTrail log record from Security Lake</p>
</div>
<p>From the profile page of the EC2 instance in the Detective console, you can go to the <strong>Overall VPC flow volume</strong> panel and filter the <a href="https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html" target="_blank" rel="noopener">Amazon Virtual Private Cloud (Amazon VPC) flow logs</a> using the attributes related to the threat actor identified as having made the SSH API call. In Figure 14, you can see the IP address that tried to connect to 22/tcp, which is the SSH port of the target instance. It’s common for threat actors to change their IP address in an attempt to evade detection, and you can remove the IP address filter to see inbound connections to port 22/tcp of your EC2 instance.</p>
<div id="attachment_34182" class="wp-caption aligncenter">
<img aria-describedby="caption-attachment-34182" src="https://infracom.com.sg/wp-content/uploads/2024/05/img14.png" alt="Figure 14: Examining SSH connections to the target instance in the Detective profile page" width="780" class="size-full wp-image-34182">
<p id="caption-attachment-34182" class="wp-caption-text">Figure 14: Examining SSH connections to the target instance in the Detective profile page</p>
</div>
<h2>Iterate the investigation</h2>
<p>At this point, you’ve made progress with the help of Detective Investigations and Security Lake integration. You started with a GuardDuty finding, and you got to the point where you were able to identify some of the intent of the threat actors and uncover the specific EC2 instance they were targeting. Your investigation shouldn’t stop here because you’ve successfully identified the EC2 instance, which is the next target to investigate.</p>
<p>You can reuse this whole workflow by starting with the EC2 instance’s <strong>New behavior</strong> panel, run Detective Investigations on the IAM role attached to the EC2 instance and other IAM principals you think are worth taking a closer look at, then use the Security Lake integration to gather raw logs of the APIs made by the EC2 instance to identify the specific actions taken and their potential consequences.</p>
<h2>Conclusion</h2>
<p>In this post, you’ve seen how you can use the Amazon Detective Investigation feature to investigate IAM user and role activity and use the Security Lake integration to determine the specific EC2 instances a threat actor appeared to be targeting.</p>
<p>The Detective Investigation feature is automatically enabled for both existing and new customers in <a href="https://docs.aws.amazon.com/general/latest/gr/detective.html" target="_blank" rel="noopener">AWS Regions that support Detective</a> where Detective has been activated. The <a href="https://docs.aws.amazon.com/detective/latest/userguide/securitylake-integration.html" target="_blank" rel="noopener">Security Lake integration</a> feature can be enabled in your Detective console. If you don’t currently use Detective, you can <a href="https://aws.amazon.com/detective/pricing/" target="_blank" rel="noopener">start a free 30-day trial</a>. For more information on Detective Investigation and Security Lake integration, see <a href="https://docs.aws.amazon.com/detective/latest/userguide/detective-investigations.html" target="_blank" rel="noopener">Investigating IAM resources using Detective investigations</a> and <a href="https://docs.aws.amazon.com/detective/latest/userguide/securitylake-integration.html" target="_blank" rel="noopener">Security Lake integration</a>.</p>
<p> <br>If you have feedback about this post, submit comments in the<strong> Comments</strong> section below. If you have questions about this post, <a href="https://console.aws.amazon.com/support/home" target="_blank" rel="noopener noreferrer">contact AWS Support</a>.</p>
<p><strong>Want more AWS Security news? Follow us on <a title="Twitter" href="https://twitter.com/AWSsecurityinfo" target="_blank" rel="noopener noreferrer">X</a>.</strong></p>
<!-- '"` -->