This post is section of a series about how exactly Amazon Web Services (AWS) might help your US federal agency meet up with the requirements of the President’s Executive Order on Improving the Nation’s Cybersecurity . Become familiar with ways to use AWS information security practices to greatly help meet the requirement to boost logging and log retention practices in your AWS environment.

Improving the security and operational readiness of applications depends on improving the observability of the applications and the infrastructure which they operate. For the customers, this means questions of how exactly to gather the proper telemetry data, how exactly to store it over its lifecycle securely, and how exactly to analyze the data to make it actionable. These questions undertake more importance as our federal customers seek to boost their collection and management of log data in every their IT environments, including their AWS environments, as mandated by the executive order.

Given the fascination with the technologies used to aid log and logging retention, we’d prefer to share our perspective. This starts having an summary of logging concepts in AWS, including log management and storage, and proceeds to how exactly to gain actionable insights from that logging data. This post shall address how exactly to improve logging and log retention practices in keeping with the Security and Operational Excellence pillars of the AWS Well-Architected Framework.

Log activity and actions inside your AWS account

AWS offers you extensive logging capabilities to supply visibility into activity and actions inside your aws account. A security best practice would be to establish a wide variety of detection mechanisms across all your AWS accounts. You start with services such as for example AWS CloudTrail, AWS Config, Amazon CloudWatch, Amazon GuardDuty, and AWS Security Hub offers a foundation upon which it is possible to base detective controls, remediation actions, and forensics data to aid incident response. Here’s more detail on what these services might help you get more security insights into your AWS workloads:

• AWS CloudTrail provides event history for several of one’s AWS account activity, including API-level actions taken through the AWS Management Console, AWS SDKs, command line tools, along with other AWS services. You should use CloudTrail to recognize who or what took which action, what resources were applied, once the event occurred, along with other details. If your agency uses AWS Organizations, it is possible to automate this technique for several of the accounts in the business.
• CloudTrail logs could be delivered from all your accounts right into a centralized account. This places all logs in a controlled tightly, central location, rendering it simpler to both protect them in addition to to store and analyze them. Much like AWS CloudTrail, it is possible to automate this technique for several of the accounts in the organization using AWS Organizations.  CloudTrail could be configured to &lt also;a href=”https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AWS-API-Usage-Metrics.html” target=”_blank” rel=”noopener noreferrer”>emit metrical data in to the CloudWatch monitoring service, giving near real-time insights in to the using various services.
• CloudTrail log file integrity validation produces and cyptographically signs a digest file which has references and hashes for each CloudTrail file that has been delivered for the reason that hour. This helps it be infeasible to change computationally, delete or forge CloudTrail log files without detection. Validated log files are invaluable in security and forensic investigations. For instance, a validated log file lets you assert that the log file itself have not changed positively, or that one user credentials performed specific API activity.
• AWS Config monitors and records your AWS resource configurations and lets you automate the evaluation of recorded configurations against desired configurations. For instance, you should use AWS Config to verify that resources are encrypted, multi-factor authentication (MFA) is enabled, and logging is fired up, and you may use AWS Config rules to recognize noncompliant resources. Additionally, it is possible to review changes in relationships and configurations between AWS resources and dive into detailed resource configuration histories, assisting you to determine when compliance status changed and the nice reason behind the change.
• Amazon GuardDuty is really a threat detection service that continuously monitors for malicious activity and unauthorized behavior to safeguard your AWS accounts and workloads. Amazon GuardDuty analyzes and processes the next data sources: VPC Flow Logs, AWS CloudTrail management event logs, CloudTrail Amazon Simple Storage Service (Amazon S3) data event logs, and DNS logs. It uses threat intelligence feeds, such as for example lists of malicious IP domains and addresses, and machine understanding how to identify potential threats inside your AWS environment.
• AWS Security Hub offers a single place that aggregates, organizes, and prioritizes your security alerts, or findings, from multiple AWS services and optional third-party products to offer a thorough view of security alerts and compliance status.

You ought to know that a lot of AWS services usually do not ask you for for enabling logging (for instance, AWS WAF) however the storage of logs shall incur ongoing costs. Consult the AWS service’s pricing page to comprehend cost impacts always. Related services such as for example Amazon Kinesis Data Firehose (used to stream data to storage services), and Amazon Simple Storage Service (Amazon S3), used to store log data, will incur charges.

Start service-specific logging as desired

Once you’ve the foundational logging services configured and enabled, next turn your focus on service-specific logging. Many AWS services produce service-specific logs offering more information. These services could be configured to record and distribute information that is essential to understand their internal state, including application, workload, user activity, dependency, and transaction telemetry. Here’s a sampling of key services with service-specific logging features:

• Amazon CloudWatch offers you data and actionable insights to monitor your applications, react to system-wide performance changes, optimize resource utilization, and obtain a unified view of operational health. CloudWatch collects monitoring and operational data by means of logs, metrics, and events, offering you a unified view of AWS resources, applications, and services that operate on on-premises and AWS servers. It is possible to gain additional operational insights from your own AWS compute instances (Amazon Elastic Compute Cloud, or EC2) in addition to on-premises servers utilizing the CloudWatch agent. Additionally, you should use CloudWatch to detect anomalous behavior in your environments, set alarms, visualize metrics and logs hand and hand, take automated actions, troubleshoot issues, and find out insights to smoothly keep your applications running.
• Amazon CloudWatch Logs is really a element of Amazon CloudWatch used to monitor, store, and access your log files from Amazon Elastic Compute Cloud (Amazon EC2) instances, AWS CloudTrail, Route 53, along with other sources. CloudWatch Logs lets you centralize the logs from all your systems, applications, and AWS services that you utilize, in a single, scalable service highly. After that you can view them easily, search them for specific error patterns or codes, filter them predicated on specific fields, or archive them for future analysis securely. CloudWatch Logs lets you see all your logs, of their source regardless, as a frequent and single flow of events ordered by time, and you may query them and sort them predicated on other dimensions, group them by specific fields, create custom computations with a robust query language, and visualize log data in dashboards.
• Traffic Mirroring lets you achieve full packet capture (in addition to filtered subsets) of network traffic from an elastic network interface of EC2 instances within your VPC. After that you can send the captured traffic to out-of-band monitoring and security appliances for content inspection, threat monitoring, and troubleshooting.
• The Elastic Load Balancing service provides access logs that capture detailed information regarding requests that are delivered to your load balancer. Each log contains information like the right time the request was received, the client’s Ip, latencies, request paths, and server responses. The precise information logged varies by load balancer type:
• Amazon S3 access logs record the S3 account and bucket which are being accessed, the API action, and requester information.
• AWS Web Application Firewall (WAF) logs record web requests which are processed by AWS WAF, and indicate if the requests matched AWS WAF rules and what actions, if any, were taken. These logs are sent to Amazon S3 through the use of Amazon Kinesis Data Firehose.
• Amazon Relational Database Service (Amazon RDS) log files can be downloaded or published to Amazon CloudWatch Logs. Log settings are specific to each database engine. Agencies use these settings to use their desired logging configurations and chose which events are logged.  Amazon Aurora and Amazon RDS for Oracle also support a real-time logging feature called “database activity streams” which gives a lot more detail, and can’t be accessed or modified by database administrators.
• Amazon Route 53 provides choices for logging for both public DNS query requests in addition to Route53 Resolver DNS queries:
  • Route 53 Resolver DNS query logs record DNS responses and queries that result from your VPC, that use an inbound Resolver endpoint, that use an outbound Resolver endpoint, or that work with a Route 53 Resolver DNS Firewall.
  • Route 53 DNS public query logs record queries to Route 53 for domains you have hosted with AWS, like the subdomain or domain that has been requested; enough time and date of the request; the DNS record type; the Route 53 edge location that taken care of immediately the DNS query; and the DNS response code.
• Amazon Elastic Compute Cloud (Amazon EC2) instances may use the unified CloudWatch agent to get logs and metrics from Linux, macOS, and Windows EC2 instances and publish them to the Amazon CloudWatch service.
• Elastic Beanstalk logs could be streamed to CloudWatch Logs. You can even utilize the AWS Management Console to request the final 100 log entries from the net and application servers, or request a lot of money of most log files that’s uploaded to Amazon S3 as a ZIP file.
• Amazon CloudFront logs record user requests for content that’s cached by CloudFront.

Store and analyze log data

Given that you’ve enabled foundational and service-specific logging in your AWS accounts, that data must be managed and persisted throughout its lifecycle. AWS offers a selection of services and answers to consolidate your log data and store it, secure usage of it, and perform analytics.

Store log data

The principal service for storing all this logging data is Amazon S3. Amazon S3 is fantastic for this role, because it’s an extremely scalable, resilient object storage service highly. AWS offers a rich group of multi-layered capabilities to secure log data that’s stored in Amazon S3, including encrypting objects (log records), preventing deletion (the S3 Object Lock feature), and using lifecycle policies to transition data to lower-cost storage over time (for instance, to S3 Glacier). Usage of data in Amazon S3 could be restricted through &lt also;a href=”http://aws.amazon.com/iam” target=”_blank” rel=”noopener noreferrer”>AWS Identity and Access Management (IAM) policies, AWS Organizations service control policies (SCPs), S3 bucket policies, Amazon S3 Access Points, and AWS PrivateLink interfaces. While S3 is specially simple to use with other AWS services given its various integrations, many customers centralize their storage and analysis of these on-premises log data also, or log data from other cloud environments, on AWS using S3 and the analytic features described below.

If your AWS accounts are organized in a multi-account architecture, you may make usage of the AWS Centralized Logging solution. This solution enables organizations to get, analyze, and display CloudWatch Logs data within a dashboard. AWS services generate log data, such as for example audit logs for access, configuration changes, and billing events. Furthermore, web servers, applications, and os’s all generate log files in a variety of formats. The &lt can be used by this solution;a href=”http://aws.amazon.com/elasticsearch-service” target=”_blank” rel=”noopener noreferrer”>Amazon Elasticsearch Service (Amazon ES) and Kibana to deploy a centralized logging solution that delivers a unified view of all log events. In conjunction with other AWS-managed services, this solution offers you a turnkey environment to begin with logging and analyzing your AWS applications and environment.

You may make use of services such as for example &lt also;a href=”https://aws.amazon.com/kinesis/data-firehose/” target=”_blank” rel=”noopener noreferrer”>Amazon Kinesis Data Firehose, used to move log information to S3, Amazon ES, or any third-party service that’s given an HTTP endpoint, such as for example Datadog, New Relic, or Splunk.

Finally, you should use Amazon EventBridge to route and integrate event data between AWS services also to third-party solutions such as for example software as something (SaaS) providers or help desk ticketing systems. EventBridge is really a serverless event bus service which allows one to connect your applications with data from the selection of sources. EventBridge delivers a blast of real-time data from your applications, SaaS applications, and AWS services, and routes that data to targets such as for example &lt then;a href=”http://aws.amazon.com/lambda” target=”_blank” rel=”noopener noreferrer”>AWS Lambda.

Analyze log data and react to incidents

Because the final part of managing your log data, you should use AWS services such as for example Amazon Detective, Amazon ES, CloudWatch Logs Insights, and Amazon Athena to investigate your log gain and data operational insights.

• Amazon Detective allows you to analyze, investigate, and identify the primary cause of security findings or suspicious activities quickly. Detective collects log data from your own AWS resources automatically. It uses machine learning then, statistical analysis, and graph theory to assist you visualize and conduct faster and much more efficient security investigations.
• Incident Manager is really a component of AWS Systems Manger which lets you automatically take action whenever a critical issue is detected by an Amazon CloudWatch alarm or Amazon Eventbridge event. Incident Manager executes pre-configured response plans to activate responders via phone and SMS calls, enable chat notifications and commands using AWS Chatbot, and execute AWS Systems Manager Automation runbooks. The Incident Manager console integrates with AWS Systems Manager OpsCenter to assist you track incidents and post-incident action items from the central place that also synchronizes with popular third-party incident management tools such as for example Jira Service Desk and ServiceNow.
• Amazon Elasticsearch Service (Amazon ES) is really a managed service that collects fully, indexes, and unifies metrics and logs across your environment to offer unprecedented visibility into your applications and infrastructure. With Amazon ES, the scalability is got by you, flexibility, and security you will need for probably the most demanding log analytics workloads. It is possible to configure a CloudWatch Logs log group to stream data it receives to your Amazon ES cluster in near real-time by way of a CloudWatch Logs subscription.
• CloudWatch Logs Insights lets you interactively search and analyze your log data in CloudWatch Logs.
• Amazon Athena can be an interactive query service which you can use to investigate data in Amazon S3 through the use of standard SQL. Athena is serverless, so there is absolutely no infrastructure to manage, and you also just pay for the queries that you run.

Conclusion

As called out in the executive order, information from systems and network logs is invaluable for both investigation and remediation services. AWS offers a broad group of services to get an unprecedented quantity of data at suprisingly low cost, optionally store it for extended periods of time in tiered storage, and analyze that telemetry information from your own cloud-based workloads. These insights shall assist you to enhance your organization’s security posture and operational readiness and, as a result, enhance your organization’s capability to deliver on its mission.

Next steps

For more information about how AWS might help certain requirements are met by you of the executive order, start to see the other post in this series:

When you have feedback concerning this post, submit comments in the Comments section below.

Want more AWS Security how-to content, news, and show announcements? Follow us on Twitter.