One-Stop IT Service Provider - To enable complicated analytics, export historical Safety Hub findings to an S3 bucket.

Posted on November 2, 2022December 27, 2022 by admin

To enable complicated analytics, export historical Safety Hub findings to an S3 bucket.

 <a href="https://aws.amazon.com/security-hub/" target="_blank" rel="noopener noreferrer">     AWS Safety Hub     </a>      is really a cloud safety posture management service which you can use to execute security best exercise checks, aggregate alerts, and automate remediation. Protection Hub has out-of-the-package integrations with numerous      <a href="https://docs.aws.amazon.com/securityhub/latest/userguide/securityhub-internal-providers.html" target="_blank" rel="noopener noreferrer">     AWS solutions     </a>      and over 60      <a href="https://docs.aws.amazon.com/securityhub/latest/userguide/securityhub-partner-providers.html" target="_blank" rel="noopener noreferrer">     companion products     </a>     . Safety Hub centralizes results across your AWS accounts and      <a href="https://docs.aws.amazon.com/securityhub/latest/userguide/finding-aggregation.html" target="_blank" rel="noopener noreferrer">     backed AWS Regions     </a>      right into a individual      <a href="https://docs.aws.amazon.com/securityhub/latest/userguide/designate-orgs-admin-account.html" target="_blank" rel="noopener noreferrer">     delegated administrator     </a>      accounts in your      <a href="https://docs.aws.amazon.com/securityhub/latest/userguide/finding-aggregation.html" target="_blank" rel="noopener noreferrer">     aggregation Area     </a>      of choice, developing a single pane of cup to consolidate and look at individual security findings.

 <pre>          <code>        &lt;p&gt;Because you can find numerous probable integrations across accounts and Regions, your delegated administrator accounts in the aggregation Region may have thousands of Protection Hub findings. To execute complex analytics or device learning over the existing (historical) results which are maintained in Safety Hub, it is possible to export results to an &lt;a href="https://aws.amazon.com/s3/" target="_blank" rel="noopener noreferrer"&gt;Amazon Simple Storage Services (Amazon S3)&lt;/the&gt; bucket. To export new findings which have recently been created, you can apply the answer in the &lt;a href="https://github.com/aws-samples/aws-security-hub-findings-export" focus on="_blank" rel="noopener noreferrer"&gt;aws-security-hub-findings-export &lt;/the&gt;GitHub repository. Nevertheless, Security Hub has information export &lt;a href="https://docs.aws.amazon.com/securityhub/best and newest/userguide/securityhub_limits.html" focus on="_blank" rel="noopener noreferrer"&gt;API price quotas&lt;/the&gt;, which will make exporting a lot of results challenging.&lt;/p&gt;

In this website post, we provide a good example treatment for export your historical Security Hub results to an S3 bucket in your account, even though you have a lot of findings. We stroll you through the the different parts of the remedy and demonstrate how to utilize the option after deployment.
<h2>Prerequisites</h2>
To deploy the perfect solution is, complete the next prerequisites:
<ol>
<li><a href=”https://docs.aws.amazon.com/securityhub/latest/userguide/securityhub-settingup.html” focus on=”_blank” rel=”noopener noreferrer”>Enable Security Hub</the>.</li>
<li>In order to export Security Hub results for several accounts, designate a <a href=”https://docs.aws.amazon.com/securityhub/latest/userguide/designate-orgs-admin-accounts.html” focus on=”_blank” rel=”noopener noreferrer”>Protection Hub administrator accounts</the>.</li>
<li>If you need to export Security Hub results across several Regions, enable <a href=”https://docs.aws.amazon.com/securityhub/most recent/userguide/finding-aggregation-enable.html” focus on=”_blank” rel=”noopener noreferrer”>cross-Area aggregation</the>.</li>
</ol>
<h2>Solution architecture&lt and overview;/h2>
In this solution, you utilize the next AWS services and functions:
<ul>
<li>Safety Hub export orchestration
<ul>
<li><a href=”https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=2ahUKEwi7x-T_zvj4AhVnEFkFHWMzA48QFnoECBoQAQ&url=https%3A%2F%2Faws.amazon.com%2Fstep-functions%2F&usg=AOvVaw0M8Bj8XfPo60Y9sDwzakOd” target=”_blank” rel=”noopener noreferrer”>AWS Phase Functions</the> can help you orchestrate automation and long-running work, which are essential to the solution. You want the ability to operate a workflow all night due to the Protection Hub API rate limitations and amount of findings and items. </li>
<li><a href=”https://aws.amazon.com/lambda/” focus on=”_blank” rel=”noopener noreferrer”>AWS Lambda</a> functions deal with the logic for exporting and storing findings within an effective and cost-effective manner. It is possible to customize Lambda features to most use situations.</li>
</ul> </li>
<li>Storage space of exported findings
</li>
<li>Work status tracking
<ul>
<li><a href=”https://aws.amazon.com/eventbridge/” focus on=”_blank” rel=”noopener noreferrer”>Amazon EventBridge</a> tracks modifications in the standing of the Step Features workflow. The answer can operate for over 100 hours; through the use of EventBridge, you don’t need to manually check the position. </li>
<li><a href=”https://aws.amazon.com/sns/” focus on=”_blank” rel=”noopener noreferrer”>Amazon Basic Notification Assistance (Amazon SNS)</the> supplies you with notifications once the long-running careers are complete or if they might have issues.</li>
<li><a href=”https://aws.amazon.com/systems-supervisor/” target=”_blank” rel=”noopener noreferrer”>AWS Techniques Manager</the> <a href=”https://aws.amazon.com/systems-manager/features/#Parameter_Shop” target=”_blank” rel=”noopener noreferrer”>Parameter Shop</a> offers a quick solution to track overall standing by keeping a numeric count of effectively exported results that you could compare with the amount of results shown in the Safety Hub dashboard.</li>
</ul> </li>
</ul>
Number 1 displays the architecture for the answer, deployed in the Protection Hub delegated administrator accounts in the aggregation Area. The figure shows several Safety Hub member accounts to illustrate ways to export findings for a whole <a href=”https://aws.amazon.com/businesses/” focus on=”_blank” rel=”noopener noreferrer”>AWS Businesses</a> organization from the single delegated administrator accounts.
<div id=”attachment_27432″ course=”wp-caption aligncenter”>
<img aria-describedby=”caption-attachment-27432″ src=”https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3electronic253a90eee5098477c95c23d/2022/10/12/img1-1-1019×1024.png” alt=”Shape 1: High-level summary of process and assets deployed inside the Security Hub accounts” width=”760″ course=”size-large wp-picture-27432″>
Figure 1: High-level summary of process and sources deployed in the Protection Hub accounts
</div>
As shown in Shape 1, the workflow after deployment is really as follows:
<ol>
<li>The Stage Features workflow for the Safety Hub export is invoked.</li>
<li>The Action Functions workflow invokes an individual Lambda function that will the following:
<ol>
<li>Retrieves Protection Hub findings with an Dynamic position and places them in a temporary document.</li>
<li>Pushes the document as an item to Amazon S3.</li>
<li>Provides the worldwide count of exported results from the Step Features workflow to the Systems Supervisor parameter for validation and monitoring purposes.</li>
<li>Repeats actions b-c for approximately 10 minutes to obtain the most findings whilst preventing the Lambda functionality from timing out there.</li>
<li>In case a <a href=”https://docs.aws.amazon.com/cli/current/reference/securityhub/get-results.html” focus on=”_blank” rel=”noopener noreferrer”>nextToken</the> exists, pushes the <period>nextToken</period> to the result of the Step Features. <blockquote>
Take note: If the amount of products in the result is smaller sized than the amount of products came back by the API contact, then your return output carries a <period>nextToken</period>, which may be approved to a subsequent control to retrieve another set of products.
</blockquote> </li>
</ol> </li>
<li>The Phase Functions workflow undergoes a <period>Choice</period> state the following:
<ul>
<li>In case a Safety Hub <period>nextToken</period> is present, Step Features invokes the Lambda functionality again.</li>
<li>In case a Protection Hub <period>nextToken</period> isn’t present, Stage Functions finishes the workflow successfully.</li>
</ul> </li>
<li>An EventBridge guideline tracks adjustments in the standing of the Step Features workflow and sends events to an SNS subject. Clients to the SNS subject receive a notification once the status of the Action Functions workflow modifications.</li>
</ol>
<h2>Deploy the alternative</h2>
It is possible to deploy the perfect solution is through either the <a href=”https://aws.amazon.com/console/” focus on=”_blank” rel=”noopener noreferrer”>AWS Management System</the> or the <a href=”https://aws.amazon.com/cdk/” focus on=”_blank” rel=”noopener noreferrer”>AWS Cloud Advancement Package (AWS CDK)</the>.
To deploy the answer (system)
<ul>
<li>In your delegated administrator Security Hub account, release the <a href=”http://aws.amazon.com/cloudformation” focus on=”_blank” rel=”noopener noreferrer”>AWS CloudFormation</the> template by selecting the next Release Stack button. It will require about ten minutes for the CloudFormation stack to perform. <a href=”https://gaming console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/fresh?stackName=securityhubhistoricalexport&templateURL=https://awsiammedia.s3.amazonaws.com/public/sample/1391-security-hub-continuous-export/securityhubhistoricalexport.yaml” rel=”noopener noreferrer” focus on=”_blank”><img loading=”lazy” src=”https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3electronic253a90eee5098477c95c23d/2019/06/05/launch-stack-switch.png” alt=”Launch Stack” width=”190″ height=”36″ class=”aligncenter size-complete wp-picture-10149″></the>
<blockquote>
Notice: The stack will start in america East (N. Virginia) Area (<period>us-east-1</period>). If you work with cross-Area aggregation, deploy the perfect solution is in to the Region where Safety Hub results are consolidated. It is possible to <a href=”https://awsiammedia.s3.amazonaws.com/general public/sample/1391-security-hub-continuous-export/securityhubhistoricalexport.yaml” focus on=”_blank” rel=”noopener noreferrer”>download the CloudFormation template for the answer</the>, change it, and deploy it to your selected Area.
</blockquote> </li>
</ul>
To deploy the answer (AWS CDK)
<ol>
<li>Download the code from our <a href=”https://github.com/aws-samples/aws-security-hub-findings-historical-export” focus on=”_blank” rel=”noopener noreferrer”>aws-security-hub-findings-historical-export</the> GitHub repository, where one can also donate to the sample code. The CDK initializes your uploads and environment the Lambda assets to Amazon S3. After that, you deploy the perfect solution is back.</li>
<li>When you are authenticated in the protection tooling account, work the next commands in your terminal. Ensure that you replace <period></period> with the accounts quantity, and replace <period></period> with the AWS Area where you intend to deploy the answer. <program code>cdk bootstrap aws:// /
cdk deploy SechubHistoricalPullStack</program code></li>
</ol>
<h2>Solution validation&lt and walkthrough;/h2>
Given that you’ve successfully deployed the perfect solution is, you can view each facet of the automation workflow doing his thing.
Prior to the workflow is began by you, you need to sign up to the SNS topic in order that you’re notified of status changes within the Step Functions workflow. Because of this example, you shall use email notification.
A subscription to the SNS subject
<ol>
<li>Open up the <a href=”https://us-east-1.system.aws.amazon.com/sns/v3/house?region=us-east-1#/dashboard” target=”_blank” rel=”noopener noreferrer”>Amazon SNS gaming console</the>.</li>
<li>Head to Subjects and pick the Protection_Hub_Export_Position subject.</li>
<li>Choose Create membership.</li>
<li>For Protocol, select E-mail.</li>
<li>For Endpoint, enter the e-mail address where you intend to receive notifications.</li>
<li>Choose Create registration.</li>
<li>Following the subscription is established by you, head to your confirm and e-mail the subscription.</li>
</ol>
You’re subscribed to the SNS topic now, so any right period that the Step Features status changes, you shall get a notification. Let’s stroll through how to operate the export remedy.
To perform the export option
<ol>
<li>Open up the <a href=”https://us-east-1.system.aws.amazon.com/claims/home?region=us-east-1″ focus on=”_blank” rel=”noopener noreferrer”>Amazon Step Features gaming console</the>.</li>
<li>In the still left navigation pane, choose State devices.</li>
<li>Pick the new state model called sec_hub_locating_export.</li>
<li>Choose Begin execution.</li>
<li>On the Begin execution web page, for Title – optional and Insight – optional, depart the default ideals and choose Begin execution.
<div id=”attachment_27435″ course=”wp-caption aligncenter”>
<img aria-describedby=”caption-attachment-27435″ src=”https://d2908q01vomqb2.cloudfront.net/22d200f8670dbdb3electronic253a90eee5098477c95c23d/2022/10/12/img2-1-1024×446.png” alt=”Body 2: Example input ideals for execution of the Phase Functions workflow” width=”700″ class=”size-large wp-image-27435″>
Figure 2: Example input ideals for execution of the Stage Functions workflow
</div> </li>
<li>This can start the Step Features workflow and redirect one to the Graph watch. If successful, you shall note that the overall Execution position and a status end up being had by each stage of Successful.</li>
<li>For long-running jobs, you will see the CloudWatch log team linked to the Lambda function to see the logs.</li>
<li>To track the real number of Safety Hub findings which have been exported, open up the <a href=”https://us-east-1.system.aws.amazon.com/systems-manager/house?region=us-east-1″ focus on=”_blank” rel=”noopener noreferrer”>Systems Manager gaming console</the>, select Parameter Shop, and choose the &lt then;period>/sechubexport/findingcount</period> parameter. Under Worth, you shall start to see the total number of Protection Hub findings which have been exported, as shown in Body 3.
<div id=”attachment_27436″ course=”wp-caption aligncenter”>
<img aria-describedby=”caption-attachment-27436″ src=”https://infracom.com.sg/wp-content/uploads/2022/11/img3-1-1024×285-1.png” alt=”Amount 3: Systems Manager Parameter Store worth for the amount of Security Hub results exported” width=”700″ course=”size-large wp-picture-27436″>
Figure 3: Systems Manager Parameter Shop value for the amount of Security Hub results exported
</div> </li>
</ol>
According to the true amount of Security Hub findings, this process may take some right time. This is because of the &lt primarily;a href=”https://docs.aws.amazon.com/securityhub/recent/userguide/securityhub_limits.html” focus on=”_blank” rel=”noopener noreferrer”>GetFindings quota</a> of 3 requests per 2nd. Each <period>GetFindings</period> request can come back no more than 100 findings, which means this means that you may get to 300 results per second up. On average, the answer can export about 1 million findings each hour. If you possess numerous findings, you can begin the finding export procedure and await the SNS subject to notify you once the process is full.
<h2>How exactly to customize the solution</h2>
The perfect solution is offers a general framework to assist you export your historical Safety Hub findings. There are several ways that it is possible to customize this solution predicated on your needs. Listed below are some enhancements that you could consider.
<h3>Modification the Security Hub getting filter</h3>
The answer pulls all findings with &lt currently;span>RecordState: Dynamic</period>, which pulls the energetic Security Hub results in the AWS accounts. It is possible to update the Lambda functionality code, the &lt specifically;span>finding_filtration system</period> JSON worth within the <period>create_filtration system functionality, to pull findings to use case. For illustration, to obtain all active Protection Hub results from the <a href=”https://docs.aws.amazon.com/securityhub/most recent/userguide/securityhub-standards-fsbp.html” focus on=”_blank” rel=”noopener noreferrer”>AWS Foundational Safety GUIDELINES standard</the>, upgrade the Lambda function program code the following.
<div course=”hide-language”>
<pre course=”unlimited-height-code”><program code class=”lang-text”>
WorkflowState: [

                 "Value": "NEW ",
                 "Comparison": "EQUALS"
             ,
         ],
         "RecordState": [

                 "Value": "ACTIVE",
                 "Comparison": "EQUALS"
             ,
         ]
    &lt;/program code&gt;&lt;/pre&gt;

</div>
<h3>Export a lot more than 100 million Protection Hub results</h3>
The example solution can export about 100 million Security Hub findings. This amount depends upon the speed of which findings could be exported primarily, because of the following elements:

In order to export a lot more than 100 million Security Hub results, do among the following:

<blockquote>
Be aware: In the event that you implement either of the solutions, be sure that the <period>nextToken</period> also gets exceeded to the brand new Step Features execution by updating the Lambda functionality program code to parse and move the <period>nextToken</period> received within the last request.
</blockquote>
<h3>Increase the export</h3>
One method to raise the export bandwidth, and decrease the overall execution period, would be to run the export job inside parallel across the specific Security Hub member accounts instead of from the one delegated administrator account.
You could utilize <a href=”https://docs.aws.amazon.com/AWSCloudFormation/best and newest/UserGuide/what-is-cfnstacksets.html” focus on=”_blank” rel=”noopener noreferrer”>CloudFormation StackSets</the> to deploy this alternative in each Safety Hub member accounts and send the results to a centralized S3 bucket. You’ll want to modify the perfect solution is to permit an S3 bucket to become provided being an input, and all of the Lambda functionality <a href=”https://aws.amazon.com/iam/” focus on=”_blank” rel=”noopener noreferrer”>Identity and Accessibility Management (IAM)</the> roles would want cross-account usage of the S3 bucket and corresponding <a href=”https://aws.amazon.com/kms/” focus on=”_blank” rel=”noopener noreferrer”>AWS Key Administration Program (AWS KMS)</the> key. You’ll should also make improvements in each member accounts to iterate through the many Regions where the Security Hub results exist.
<h2>Following steps</h2>
The perfect solution is in this post is made to help out with the retrieval and export of most existing findings currently in Protection Hub. Once you run this treatment for export historical findings effectively, it is possible to continuously export new Safety Hub findings utilizing the sample answer in the <a href=”https://github.com/aws-samples/aws-security-hub-findings-export” focus on=”_blank” rel=”noopener noreferrer”>aws-security-hub-findings-export</the> GitHub repository.
Given that you’ve exported the Security Hub results, you can &lt create and run;a href=”https://docs.aws.amazon.com/athena/most recent/ug/data-sources-glue.html” focus on=”_blank” rel=”noopener noreferrer”>custom made complex queries&lt or even reporting;/a> contrary to the S3 bucket through the use of <a href=”https://aws.amazon.com/athena/” focus on=”_blank” rel=”noopener noreferrer”>Amazon Athena</the> and <a href=”https://aws.amazon.com/glue/” focus on=”_blank” rel=”noopener noreferrer”>AWS Glue</a>. Additionally, it is possible to run device analytics and learning capabilities through the use of services like <a href=”https://aws.amazon.com/pm/sagemaker” focus on=”_blank” rel=”noopener noreferrer”>Amazon SageMaker</the> or <a href=”https://aws.amazon.com/lookout-for-metrics/” focus on=”_blank” rel=”noopener noreferrer”>Amazon Lookout for Metrics</the>.
<h2>Bottom line</h2>
In this article, you deployed a remedy to export the prevailing Security Hub results in your account to a central S3 bucket, to enable you to apply complex device and analytics understanding how to those findings. We walked you through how exactly to use the remedy and apply it for some example use instances after you effectively exported existing results across your AWS atmosphere. Now your security group can use the info in the S3 bucket for predictive analytics and determine if you can find Security Hub results and specific assets that might have to be prioritized for review because of deviation from normal habits. Additionally, you may use this treatment for enable more technical analytics on multiple areas by <a href=”https://docs.aws.amazon.com/AmazonS3/most recent/userguide/storage-inventory-athena-query.html” focus on=”_blank” rel=”noopener noreferrer”>querying complex and big datasets along with AWS Athena.</the>
When you have feedback concerning this post, submit remarks in the Remarks area below. Should you have questions concerning this post, take up a thread on <a href=”https://repost.aws/tags/TAFZPV4oyuS6-TWxLQfz5qSQ?forumID=283″ target=”_blank” rel=”noopener noreferrer”>AWS Protection Hub re:Write-up</the>.
 Want a lot more AWS Security news? Stick to us on <a name=”Twitter” href=”https://twitter.com/AWSsecurityinfo” focus on=”_blank” rel=”noopener noreferrer”>Twitter</the>.

<!– ‘”` –>

To enable complicated analytics, export historical Safety Hub findings to an S3 bucket.

Share this:

Like this: