Amazon Elasticsearch Service (Amazon ES) is really a fully managed assistance which you can use to deploy, secure, and run Elasticsearch cost-effectively from scale. The ongoing program provides assistance for open-resource Elasticsearch APIs, managed Kibana, and integration with Logstash along with other AWS providers. Amazon ES offers a deep security design that spans several layers of conversation and supports fine-grained accessibility handle at the cluster, index, document, and field degree, on a per-user schedule. The service’s safety plugin integrates with federated identity providers for Kibana login.

A standard use case for Amazon ES is log analytics. Clients configure their apps to store log information to the Elasticsearch cluster, where in fact the data could be queried for insights in to the use and functionality of the applications as time passes. In many cases, customers reviewing those insights ought never to have access to everything from the log information. The log information for a web program, for example, might are the supply IP addresses of incoming requests. Privacy guidelines in lots of nations require that those information be masked, or in part wholly. This write-up explains how to create field masking inside your Amazon ES domain.

Field masking can be an option to field-level security that enables you to anonymize the info in a field instead of remove it altogether. When making a role, put in a set of fields to mask. Field masking affects whether you can view the contents of a field once you search. You should use field masking to either perform random hash or pattern-based substitution of sensitive information from users, who shouldn’t get access to that information.

By using field masking, Amazon ES creates a hash of the specific field values before returning the serp’s. It is possible to apply field masking on a per-role basis, supporting different degrees of visibility with regards to the identity of an individual making the query. Currently, field masking is designed for string-based fields. A search result with a masked field (clientIP) appears like this:

  "_index": "web_logs",
  "_type": "_doc",
  "_id": "1",
  "_score": 1,
  "_source": 
    "agent": "Mozilla/5.0 (X11; Linux x86_64; rv:6.0a1) Gecko/20110421 Firefox/6.0a1",
    "bytes": 0,
    "clientIP": "7e4df8d4df7086ee9c05efe1e21cce8ff017a711ee9addf1155608ca45d38219",
    "host": "www.example.com",
    "extension": "txt",
    "geo": 
      "src": "EG",
      "dest": "CN",
      "coordinates": 
        "lat": 35.98531194,
        "lon": -85.80931806
      
    ,
    "machine": 
      "ram": 17179869184,
      "os": "win 7"
    
  

To follow along in this article, make certain an Amazon is had by you ES domain with Elasticsearch version 6.7 or more, sample data loaded (this example uses the net logs data given by Kibana), and usage of Kibana by way of a role with administrator privileges for the domain.

Configure field masking

Field masking is managed by defining specific access controls within the Kibana visualization system. Have to develop a new Kibana role you’ll, define the fine-grained access-control privileges for that role, specify which fields to mask, and apply that role to specific users.

You should use either the Kibana console or direct-to-API calls to create field masking. Inside our first example, we’ll utilize the Kibana console.

To configure field masking in the Kibana console

1. Log directly into Kibana, pick the Security pane, and choose Roles, as shown in Figure 1.

   

   Figure 1: Choose security roles

2. Choose the plus sign (+) to make a new role, as shown in Figure 2.

   

   Figure 2: Create role

3. Choose the Index Permissions tab, and choose Add index permissions, as shown in Figure 3.

   

   Figure 3: Set index permissions

4. Add index patterns and appropriate permissions for data access. Start to see the Amazon ES documentation for information on configuring fine-grained access control.
5. Once you’ve set Index Patterns, Permissions: Action Groups, Document Level Security Query, and Include or exclude fields, you should use the Anonymize fields entry to mask the clientIP, as shown in Figure 4.

   

   Figure 4: Anonymize field

6. Choose Save Role Definition.
7. Next, you need to generate a number of users and apply the role to the brand new users. Get back to the Security page and choose Internal User Database, as shown in Figure 5.

   

   Figure 5: Select Internal User Database

8. Choose the plus sign (+) to make a new user, as shown in Figure 6.

   

   Figure 6: Create user

9. Add a password and username, and under Open Distro Security Roles, choose the role es-mask-role, as shown in Figure 7.

   

   Figure 7: Choose the username, password, and roles

10. Choose Submit.

If you prefer, it is possible to perform exactly the same task utilizing the Amazon ES REST API using Kibana dev tools.

Utilize the following API to make a role as described in below snippet and shown in Figure 8.

PUT _opendistro/_security/api/roles/es-mask-role

  "cluster_permissions": [],
  "index_permissions": [
    
      "index_patterns": [
        "web_logs"
      ],
      "dls": "",
      "fls": [],
      "masked_fields": [
        "clientIP"
      ],
      "allowed_actions": [
        "data_access"
      ]
    
  ]

Sample response:

  "status": "CREATED",
  "message": "'es-mask-role' created."

Utilize the following API to make a user with the role as described in below snippet and shown in Figure 9.

PUT _opendistro/_security/api/internalusers/es-mask-user

  "password": "xxxxxxxxxxx",
  "opendistro_security_roles": [
    "es-mask-role"
  ]

Sample response:

  "status": "CREATED",
  "message": "'es-mask-user' created."

Verify field masking

It is possible to verify field masking by owning a simple search query using Kibana dev tools (GET web_logs/_search) and retrieving the info first utilizing the kibana_user (without field masking), and utilizing the es-mask-user (with field masking) you merely created.

Query responses run by the kibana_user (all access) have the initial values in every fields, as shown in Figure 10.

Figure 11, following, shows a good example of what you will see if you logged in because the es-mask-user. In this full case, the clientIP field is hidden because of the es-mask-role you created.

Use pattern-based field masking

Than developing a hash rather, you can use a number of regular replacement and expressions strings to mask a field. The syntax is :://::.

You should use either the Kibana console or direct-to-API calls to create pattern-based field masking. In the next example, clientIP is masked so that the final three elements of the Ip are masked by xxx utilizing the pattern is clientIP::/[0-9]1,3.[0-9]1,3.[0-9]1,3$/::xxx.xxx.xxx>. You see only the initial part of the Ip, as shown in Figure 12.

Run the search query to verify that the final three elements of clientIP are masked by custom characters and only the initial part is proven to the requester, as shown in Figure 13.

Conclusion

Field level security ought to be the primary approach for ensuring data access security – however if there are particular business requirements that can’t be met with this particular approach, field masking may provide a viable alternative then. Through the use of field masking, it is possible to selectively allow or stop your users from seeing personal information such as for example personally identifying information (PII) or personal healthcare information (PHI). To find out more about fine-grained access control, start to see the Amazon Elasticsearch Service Developer Guide.

When you have feedback concerning this post, submit comments in the Comments section below. When you have questions concerning this post, take up a new thread on the Amazon Elasticsearch Service forum or contact AWS Support.

Want more AWS Security how-to content, news, and show announcements? Follow us on Twitter.