Modern web application stacks are filled with secrets. From SSH keys to API credentials, building and deploying a piece of software to AWS involves handling a lot of secret values that each need to be properly protected. One of the first steps in protecting a secret is identifying secrets in your environment that lack encryption or access controls.. Finding unencrypted secrets in AWS can be a daunting task, but there are several tools and approaches that can help you identify and remediate them.
One of the most popular tools for finding hidden information in your AWS environment is Amazon Macie. This user-friendly service focuses on data security and privacy, using machine learning to automatically find and categorize sensitive information in your AWS space. Macie can help you find unprotected secrets by looking through your storage areas like S3 buckets, EBS volumes, and RDS databases. It can also spot sensitive information in less structured places, such as logs and application files. Macie is not built, however, to find secrets in places like the AWS System Manager Parameter Store (SSM) which is a managed key/value solution that is intended for storing easily variable data.
At Jahnel Group, we were tasked by a client with identifying unencrypted secrets within SSM and remediating them. Typically, we would look to use open-source tools like Prowler and Scout Suite which can perform security audits of your AWS environment to identify secrets in services that Macie lacks coverage over. Prowler is a command-line tool that automates security checks on your AWS environment. It can help you identify security risks, including unencrypted secrets, in your AWS account. Scout Suite is another open-source multi-cloud security auditing tool that scans your AWS account for security risks, including unencrypted secrets.
These tools, however, both lack the same capability as Macie to identify secrets stored in unencrypted SSM parameters or remediate them. Usually, people turn to AWS Secrets Manager to keep their secrets safe within AWS. But sometimes, SSM might still be used in older systems or for cost savings reasons, as it doesn’t have a monthly fee for standard parameters, while Secrets Manager does have a small monthly cost per secret and a tiny fee for every 10,000 times a secret is accessed. However, Secrets Manager offers better security features (like automatic key updates), making it a better choice for new apps. For those new to AWS, there are the AWS services we are going to be focusing on for the remainder of this article:
– Amazon Key Management Service (KMS) is a managed service offered by AWS that makes it easy for you to create and manage cryptographic keys and control their use across a wide range of AWS services and applications. KMS is designed to help you protect sensitive data by providing an additional layer of security through encryption. You can create your own (customer managed) KMS keys or use service specific KMS keys provided by AWS.
– Amazon SSM (Systems Manager) is a suite of managed services that are used to configure cloud applications. In this article, we’ll be focusing on AWS SSM parameters, a managed key/value store offered by AWS to make it easier to update settings without modifying code. An AWS SSM (Systems Manager) parameter is like a container for storing sensitive information or settings that your applications or scripts need to work properly. Instead of putting these details directly into your code, you save them as parameters in AWS SSM. By doing this, you can keep your sensitive data more secure and easily manage it.
– Amazon IAM, or Identity and Access Management, is a service in AWS that helps you control who can access and use your AWS resources. With IAM, you can create users, groups, and roles, and set up rules called “policies” to define what each user, group, or role is allowed to do. This way, you can make sure that the right people have access to the right resources, keeping your AWS environment more secure and organized.
–AWS CloudTrail is a service provided by AWS that allows users to record and monitor activity in their AWS accounts. It captures detailed information about API calls made by users, as well as activity related to management events, and stores this information in log files. These log files can be used for security analysis, compliance auditing, and troubleshooting. CloudTrail also supports integration with other AWS services, such as S3, CloudWatch, and Lambda, allowing users to analyze and automate their AWS activity data.
Since our client has thousands of SSM parameters across dozens of AWS accounts, we chose to design and implement a custom solution to identify and remediate SSM parameters that lacked encryption in a semiautomated manner with the ability to rollback the change in the event of a regression due to legacy code. Our approach consisted of a series of 3 stages. First, we used a custom Go script to concurrently scan the AWS accounts belonging to our client and identify SSM parameters containing high entropy strings or with keywords that would indicate the presence of a secret. High entropy strings are sequences of characters that appear to be random and have a high degree of unpredictability. These strings are commonly used to generate secure passwords, cryptographic keys, and other secrets that need to be kept confidential. The randomness of high entropy strings makes them difficult to guess, making them ideal for use as secure secrets.
Secondly, we built a custom Go script to pull access logs and other activity from Cloudtrail to identify which users and services were interacting with these SSM parameters so we could determine which secrets were leftovers from deprecated legacy services and could be deleted and which required remediation. Amazon CloudTrail is a service in AWS records and logs all API activity in your AWS account. It provides a complete audit trail of all events and changes made within your AWS account, including who made the changes, when the changes were made, and what specific changes were made. Cloudtrail enabled us to see which services were using the SSM parameters with plaintext secrets so we could identify dependencies on these parameters and ensure that they could consume encrypted SSM parameters with tightened access controls.
Next, we went into the codebases for the services that had noncompliant SSM parameters and worked with the product teams to determine if any changes were required to the codebases to support encrypting the SSM parameter after remediation. After confirming the service had the correct IAM policies and SDK configurations to support encrypted SSM parameters, we used a final SSM script to delete the unencrypted SSM parameter, cache the configuration in an encrypted state for rollback and create a duplicate encrypted SSM parameter. For popular AWS SDKs like boto3 for Python, decryption can be enabled in advance of this change, enabling the code changes required to support encrypted SSM parameters to be deployed in advance of the remediation efforts.
Encryption of SSM parameters does not provide the same level of fine grained access control that Secrets Manager or other tools like Vault can provide, but it is an important best practice for ensuring Secrets are securely managed in the Cloud. In order to limit access to the secrets, you could implement an IAM policy that restricts access to the KMS key that was used to encrypt you secrets containing SSM parameters using an IAM policy similar to the example below:
|
AWSTemplateFormatVersion: '2010-09-09' |
Although a more complex IAM policy will likely be required in your AWS account in order to facilitate access to the SSM secret from your services and/or CI/CD pipelines, this template illustrates how restricting access to a subset of allowlisted entities can be used to protect your secrets within SSM.
We strongly recommend utilizing IAM policies to limit access to encrypted SSM keys with sensitive information to only those users and services that need access to them. Additionally, Cloudtrail should be configured to provide audit logging of access to these parameters for compliance and auditing purposes. AWS has a lot of great resources for configuring and querying Cloudtrail to provide an audit trail of activity within your cloud environment. Security in cloud environments like AWS is complex and multifaceted and there is no single “silver bullet” for keeping secrets safe. By following best practices and leveraging the robust security features of AWS, you can rest assured that your secrets are locked up tight in the cloud, safe from prying eyes and unauthorized access.
Author Bio
Zach Phillips-Gary is a Senior Site Reliability Engineer at Jahnel Group, Inc., a custom software development firm based in Schenectady, NY. Jahnel Group is a consulting firm that specializes in helping companies leverage technology to improve their business operations. We provide end-to-end strategic consulting and deployment services, helping companies optimize their operations and reduce costs through the use of technology. Jahnel Group is an Advanced Tier AWS Services Partner, with expertise in AWS Lambdas, Amazon API Gateway, Amazon DynamoDB, and other AWS services.
