Keeping track of your application's secrets can be overwhelming. Password managers like AWS Secrets Manager are great for securely storing your passwords, certificates, and tokens, but they can’t import secrets to applications.
To beat the overwhelm of managing secrets, you can use External Secrets Operator to import them into your application.
Why Not Use AWS Secrets Manager?
Password managers provide a single source of truth for credentials like admin accounts created during provisioning and other non-human IDs necessary to the application.
While AWS Secrets Manager allows us to store text files, certificates, and JSON objects, it’s missing a method to sync with Kubernetes secrets. Enter the External Secrets Operator.
How External Secrets Operator Works
External Secrets Operator (ESO) allows you to pull credentials stored in AWS Secrets Manager and sync them with a Kubernetes secret. This ensures that the Secret Manager remains the single source of truth for credentials.
An external secret object contains information about the object in a vault, such as AWS Secrets Manager, and also contains information about how to use that secret to create a Kubernetes secret. Inside the cluster, there is a pod that is checking the vault regularly (such as every minute) and will ensure that the Kubernetes secret is up to date.
Additionally, one secret can map to many places—even in different clusters or namespaces. For example, you only need a single account to pull images from Docker Hub, but each namespace will need those credentials. By using the External Secrets Operator, you can easily update credentials in the AWS Secrets manager and have them automatically synced to a Kubernetes Cluster as a Kubernetes secret.
Note: There is another project called Kubernetes External Secrets which is extremely similar to ESO. Kubernetes External Secrets has been deprecated in favor of ESO.
How to Implement External Secrets Operator
Now that you have the background, let’s get started. Follow these steps to install and configure ESO.
1. Install ESO via Helm
Run the following commands to install ESO via Helm.
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets \
-n external-secrets \
2. Create a user and configure permissions
Next, set up a user and make sure that they have the proper permissions to pull from the secret store. Once the user and permissions are configured, copy the IAM Policy below and save it as policy.json.
After the IAM Policy is saved, create the policy.
aws iam create-policy --policy-name secrets-reader --policy-document file://policy.json
Get the policy ARN from the output. Then create the group and attach the policy to the new group.
aws iam create-group --group-name secret-readers
aws iam attach-group-policy --policy-arn "arn:aws:iam::1234567890:policy/secrets-reader" --group-name secret-readers
Now create a user and add them to the group.
aws iam create-user --user-name external-secrets
aws iam add-user-to-group --group-name secret-readers --user-name external-secrets
3. Create a secret with the user information
This'll be the last secret you create in Kubernetes. Create an access key, and use that key to create the secret. This will be necessary in every namespace.
aws iam create-access-key --user-name external-secrets > creds.json
AK=$(cat creds.json | jq -r .AccessKey.AccessKeyId)
SK=$(cat creds.json | jq -r .AccessKey.SecretAccessKey)
kubectl create secret generic aws-secret --from-literal=access-key=$AK --from-literal=secret=$SK -n default
How Does ESO Work?
When we install ESO, four Custom Resource Definitions are created. The first two are namespace specific—SecretStore and ExternalSecret.
The SecretStore contains information to connect to the vault, in this case, AWS. The ExternalSecret contains the information about what data to retrieve, how to format that data, and what to call the secret.
The next two Custom Resource Definitions are ClusterSecretStore and ClusterExternalSecret. The ClusterSecretStore is similar to SecretStore, but it can be used by ExternalStore in all namespaces. This can be used if all of your namespaces share a single vault.
Like ClusterSecretStore, ClusterExternalSecret is similar to ExternalSecret but it has additional fields for which namespace(s) to add the secret. It also supports wildcards. For example, you may want to add credentials to pull from a private docker image repository to all namespaces.
Create Your First External Secret
To create an external secret, start by putting something in the safe.
aws secretsmanager create-secret \
--name sm-secret \
Then create a SecretStore. The SecretStore will contain information about the vault that stores the secrets. Create the SecretStore by copying the following into a file called ss.yaml.
Next, apply the YAML to your cluster.
kubectl apply -f ss.yaml -n default
Finally, we create the YAML to create the ExternalSecret. This YAML contains the information about where the secret is stored in our vault in addition to where it should be stored in Kubernetes. Name this hello.yaml.
Now apply the YAML.
kubectl apply -f hello.yaml -n default
Kubernetes will apply the External secret to Kubernetes. It will recognize that there is a new object and will retrieve the secret from AWS and sync it with a Kubernetes secret that needs to be created. This takes about 10 seconds, after which you can check to see if it was created.
kubectl get secret hello
If you inspect the secret you will find a field called value with Hello-World encrypted.
While the process above is good for passing a single value, there are likely scenarios where you'll want to see multiple values. See these examples for more ways to implement ESO.
Example 1: Docker Credentials
This assumes that you have a secret called docker-config in your safe containing a JSON docker token.
- secretKey: mysecret
Example 2: Retrieve Username and Password.
For this example, the username and password would be stored as a JSON key-value pair.
- secretKey: userename
- secretKey: password
For more examples, see the project page at https://external-secrets.io/.
Optimize Your Secrets Management
External Secrets Operator provides an automated method to take secrets from vault to application.
Use ESO to convert secrets from a vault such as AWS Secrets Manager to a Kubernetes secret, which can then be added to an application.
This provides a single source of truth for the secret, allowing you to make one change that will be propagated throughout the Kubernetes infrastructure.
Have questions? Drop into Bitovi's Community Slack, and talk to us in the
Need More Help?
Bitovi's team of DevOps experts can help you with anything from secrets management to CI/CD. Book a free consultation and tell us about your current project.