Optimizing Secrets Management with External Secrets Operator

Keeping track of your application's secrets can be overwhelming. Password managers like AWS Secrets Manager are great for securely storing your passwords, certificates, and tokens, but they can’t import secrets to applications.

To beat the overwhelm of managing secrets, you can use External Secrets Operator to import them into your application.

Why Not Use AWS Secrets Manager?

Password managers provide a single source of truth for credentials like admin accounts created during provisioning and other non-human IDs necessary to the application.

While AWS Secrets Manager allows us to store text files, certificates, and JSON objects, it’s missing a method to sync with Kubernetes secrets. Enter the External Secrets Operator.

How External Secrets Operator Works

External Secrets Operator (ESO) allows you to pull credentials stored in AWS Secrets Manager and sync them with a Kubernetes secret. This ensures that the Secret Manager remains the single source of truth for credentials.

An external secret object contains information about the object in a vault, such as AWS Secrets Manager, and also contains information about how to use that secret to create a Kubernetes secret. Inside the cluster, there is a pod that is checking the vault regularly (such as every minute) and will ensure that the Kubernetes secret is up to date.

Additionally, one secret can map to many places—even in different clusters or namespaces. For example, you only need a single account to pull images from Docker Hub, but each namespace will need those credentials. By using the External Secrets Operator, you can easily update credentials in the AWS Secrets manager and have them automatically synced to a Kubernetes Cluster as a Kubernetes secret.

Note: There is another project called Kubernetes External Secrets which is extremely similar to ESO. Kubernetes External Secrets has been deprecated in favor of ESO.

How to Implement External Secrets Operator

Now that you have the background, let’s get started. Follow these steps to install and configure ESO.

1. Install ESO via Helm

Run the following commands to install ESO via Helm.

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets \
   external-secrets/external-secrets \
    -n external-secrets \
    --create-namespace

2. Create a user and configure permissions

Next, set up a user and make sure that they have the proper permissions to pull from the secret store. Once the user and permissions are configured, copy the IAM Policy below and save it as policy.json.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetResourcePolicy",
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret",
                "secretsmanager:ListSecretVersionIds"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

After the IAM Policy is saved, create the policy.

aws iam create-policy --policy-name secrets-reader --policy-document file://policy.json

Get the policy ARN from the output. Then create the group and attach the policy to the new group.

aws iam create-group --group-name secret-readers
aws iam attach-group-policy --policy-arn "arn:aws:iam::1234567890:policy/secrets-reader" --group-name secret-readers

Now create a user and add them to the group.

aws iam create-user --user-name external-secrets
aws iam add-user-to-group --group-name secret-readers --user-name external-secrets

3. Create a secret with the user information

This'll be the last secret you create in Kubernetes. Create an access key, and use that key to create the secret. This will be necessary in every namespace.

aws iam create-access-key --user-name external-secrets > creds.json
AK=$(cat creds.json | jq -r .AccessKey.AccessKeyId)
SK=$(cat creds.json | jq -r .AccessKey.SecretAccessKey)
kubectl create secret generic aws-secret --from-literal=access-key=$AK --from-literal=secret=$SK -n default

How Does ESO Work?

When we install ESO, four Custom Resource Definitions are created. The first two are namespace specific—SecretStore and ExternalSecret.

The SecretStore contains information to connect to the vault, in this case, AWS. The ExternalSecret contains the information about what data to retrieve, how to format that data, and what to call the secret.

The next two Custom Resource Definitions are ClusterSecretStore and ClusterExternalSecret. The ClusterSecretStore is similar to SecretStore, but it can be used by ExternalStore in all namespaces. This can be used if all of your namespaces share a single vault.

Like ClusterSecretStore, ClusterExternalSecret is similar to ExternalSecret but it has additional fields for which namespace(s) to add the secret. It also supports wildcards. For example, you may want to add credentials to pull from a private docker image repository to all namespaces.

Create Your First External Secret

To create an external secret, start by putting something in the safe.

aws secretsmanager create-secret \
     --name sm-secret \
     --secret-string Hello-World

Then create a SecretStore. The SecretStore will contain information about the vault that stores the secrets. Create the SecretStore by copying the following into a file called ss.yaml.

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: secret-store
spec:
  provider:
    aws:  
      service: SecretsManager 
      region: us-east-1   
      auth:
        secretRef:
          accessKeyIDSecretRef: 
            name: aws-secret 
            key: access-key  
          secretAccessKeySecretRef:
            name: aws-secret
            key: secret

Next, apply the YAML to your cluster.

kubectl apply -f ss.yaml -n default

Finally, we create the YAML to create the ExternalSecret. This YAML contains the information about where the secret is stored in our vault in addition to where it should be stored in Kubernetes. Name this hello.yaml.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: sm-secret
spec:
  refreshInterval: 1m
  secretStoreRef:
    name: secret-store 
    kind: SecretStore
  target:
    name: hello
  data:
secretKey: value 
    remoteRef:
      key: sm-secret

Now apply the YAML.

kubectl apply -f hello.yaml -n default

Kubernetes will apply the External secret to Kubernetes. It will recognize that there is a new object and will retrieve the secret from AWS and sync it with a Kubernetes secret that needs to be created. This takes about 10 seconds, after which you can check to see if it was created.

kubectl get secret hello

Example of optimizing secrets management with external secrets operator

If you inspect the secret you will find a field called value with Hello-World encrypted.

Examples

While the process above is good for passing a single value, there are likely scenarios where you'll want to see multiple values. See these examples for more ways to implement ESO.

Example 1: Docker Credentials

This assumes that you have a secret called docker-config in your safe containing a JSON docker token.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: docker-config
spec:
  refreshInterval: 1h
  secretStoreRef:
name: secret-store
kind: SecretStore
  target:
name: docker-config-secret
template:
  type: kubernetes.io/dockerconfigjson
  data:
    .dockerconfigjson: ""
  data:
  - secretKey: mysecret
remoteRef:
  key: docker-config

Example 2: Retrieve Username and Password.

For this example, the username and password would be stored as a JSON key-value pair.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: demo-user
spec:
  refreshInterval: 1m
  secretStoreRef:
name: secret-store
kind: SecretStore
  target:
name: demo-creds
  data:
  - secretKey: userename
remoteRef:
  key: demo-user-pass
  property: username
  - secretKey: password
remoteRef:
  key: demo-user-pass
  property: password

More Examples

For more examples, see the project page at https://external-secrets.io/.

Optimize Your Secrets Management

External Secrets Operator provides an automated method to take secrets from vault to application.

Use ESO to convert secrets from a vault such as AWS Secrets Manager to a Kubernetes secret, which can then be added to an application.

This provides a single source of truth for the secret, allowing you to make one change that will be propagated throughout the Kubernetes infrastructure.

Have questions? Drop into Bitovi's Community Slack, and talk to us in the #devops channel!

Need More Help?

Bitovi's team of DevOps experts can help you with anything from secrets management to CI/CD. Book a free consultation and tell us about your current project.