Kubernetes Secrets and the External Secrets Operator

Create and consume secrets in Kubernetes on IronSled — kubectl create secret from literals/files/YAML, Opaque data vs stringData, mounting secrets as environment variables or files, image-pull secrets, rotating secrets and restarting pods, and using the External Secrets Operator (ESO) to sync secrets from AWS Secrets Manager into your namespace.

Secrets hold the sensitive values your application needs at runtime — database passwords, API keys, tokens, and certificates. On IronSled you have two ways to manage them: create Kubernetes Secret objects directly, or let the External Secrets Operator (ESO) sync them into your namespace from AWS Secrets Manager. This page covers both, plus how to consume a secret from a pod and how to rotate one.

Kubernetes Secrets are base64-encoded, not encrypted, in the object definition. Never commit a raw Secret manifest to Git. For anything sensitive in a shared or production environment, prefer the External Secrets Operator so the source of truth stays in AWS Secrets Manager.

Understanding Kubernetes Secrets

A Kubernetes Secret is an object that stores sensitive data as key-value pairs. Secret values are base64-encoded (not encrypted by default) and can be:

Exposed to a container as environment variables
Mounted into a container as files
Used by the kubelet to pull images from a private registry

Secrets are namespaced — a Secret lives in one namespace and is only usable by pods in that same namespace.

Creating Secrets

From literal values

kubectl create secret generic my-secret \
  --from-literal=username=admin \
  --from-literal=password='S3cr3tP@ss!' \
  -n my-namespace

# Verify
kubectl get secret my-secret -n my-namespace

From files

echo -n 'admin' > ./username.txt
echo -n 'S3cr3tP@ss!' > ./password.txt

kubectl create secret generic my-secret \
  --from-file=username=./username.txt \
  --from-file=password=./password.txt \
  -n my-namespace

rm ./username.txt ./password.txt

From a YAML manifest (base64-encoded)

Values in the data field must be base64-encoded:

echo -n 'admin' | base64          # YWRtaW4=
echo -n 'S3cr3tP@ss!' | base64    # UzNjcjN0UEBzcyE=

apiVersion: v1
kind: Secret
metadata:
  name: my-secret
  namespace: my-namespace
type: Opaque
data:
  username: YWRtaW4=
  password: UzNjcjN0UEBzcyE=

Using stringData (no base64)

Use stringData to provide plain-text values that Kubernetes encodes for you:

apiVersion: v1
kind: Secret
metadata:
  name: my-secret
  namespace: my-namespace
type: Opaque
stringData:
  username: admin
  password: S3cr3tP@ss!

Secret Types

Docker registry (image pull)

kubectl create secret docker-registry regcred \
  --docker-server=registry.example.com \
  --docker-username=user \
  --docker-password=password \
  --docker-email=user@example.com \
  -n my-namespace

TLS certificate

kubectl create secret tls my-tls-secret \
  --cert=path/to/cert.pem \
  --key=path/to/key.pem \
  -n my-namespace

Using Secrets in Pods

As environment variables

Map individual keys with secretKeyRef:

spec:
  containers:
  - name: my-container
    image: my-image
    env:
    - name: DB_USERNAME
      valueFrom:
        secretKeyRef:
          name: my-secret
          key: username
    - name: DB_PASSWORD
      valueFrom:
        secretKeyRef:
          name: my-secret
          key: password

Or import every key at once with envFrom:

spec:
  containers:
  - name: my-container
    image: my-image
    envFrom:
    - secretRef:
        name: my-secret

As a mounted volume

spec:
  containers:
  - name: my-container
    image: my-image
    volumeMounts:
    - name: secret-volume
      mountPath: /etc/secrets
      readOnly: true
  volumes:
  - name: secret-volume
    secret:
      secretName: my-secret

For image pull

spec:
  containers:
  - name: my-container
    image: registry.example.com/my-image:tag
  imagePullSecrets:
  - name: regcred

Viewing and Managing Secrets

# List secrets
kubectl get secrets -n my-namespace

# Describe (shows metadata, not values)
kubectl describe secret my-secret -n my-namespace

# View the full object (values base64-encoded)
kubectl get secret my-secret -n my-namespace -o yaml

# Decode a specific key
kubectl get secret my-secret -n my-namespace \
  -o jsonpath='{.data.password}' | base64 -d

# Delete a secret
kubectl delete secret my-secret -n my-namespace

Rotating Secrets

Update a secret in place and roll the workload so pods pick up the new value:

# Update the secret
kubectl create secret generic my-secret \
  --from-literal=password='NewP@ssword!' \
  --dry-run=client -o yaml | kubectl apply -f -

# Restart pods to pick up the change
kubectl rollout restart deployment my-app -n my-namespace

Secrets consumed as environment variables do not update automatically — a pod restart is required. Secrets mounted as volumes update automatically after a short delay (unless mounted with subPath, which disables auto-updates).

External Secrets Operator (ESO)

IronSled clusters run the External Secrets Operator (ESO), which syncs secrets from AWS Secrets Manager into native Kubernetes Secret objects in your namespace. This keeps the source of truth in a dedicated, audited secret store while your application still reads an ordinary Kubernetes Secret. During onboarding, an AWS role scoped to your project is created so you can store secrets under a fixed naming convention:

<environment>/ironsled/applications/<project>/<secret>
# Example: dev/ironsled/applications/my-app/db-connection

Create an ExternalSecret

Add an ExternalSecret to your Helm chart or manifest. It references the cluster's shared ClusterSecretStore and writes a Kubernetes Secret named by target.name. Use dataFrom.extract to pull every key from a JSON secret (typical for RDS credentials):

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: my-app
  namespace: my-namespace
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-cluster-secret-store
    kind: ClusterSecretStore
  target:
    name: my-app
    creationPolicy: Owner
  dataFrom:
    - extract:
        key: dev/ironsled/applications/my-app/db-connection

To map specific keys instead of extracting everything, use data with a remoteRef per key:

spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-cluster-secret-store
    kind: ClusterSecretStore
  target:
    name: my-app
    creationPolicy: Owner
  data:
    - secretKey: db-host
      remoteRef:
        key: dev/ironsled/applications/my-app/db-connection
        property: host
    - secretKey: db-password
      remoteRef:
        key: dev/ironsled/applications/my-app/db-connection
        property: password
    - secretKey: db-user
      remoteRef:
        key: dev/ironsled/applications/my-app/db-connection
        property: username

Consume the synced Secret

ESO creates a normal Kubernetes Secret (named by target.name), so your Deployment references it the same way as any other Secret:

spec:
  containers:
  - name: my-app
    image: registry.example.com/my-app:latest
    env:
    - name: DATABASE_PASSWORD
      valueFrom:
        secretKeyRef:
          name: my-app
          key: password

data vs. dataFrom

data maps individual remote keys (or properties within a JSON secret) to specific keys in the generated Kubernetes Secret. Each entry sets a secretKey and a remoteRef (key, optional property, optional version). Use it when you want explicit control over each key.
dataFrom pulls multiple values at once — extract copies all key-value pairs from a JSON secret into the Kubernetes Secret; find discovers secrets by name pattern or path. Use it for bulk sync without listing each key.

Best Practices

Never commit secrets to Git. Add secret manifests, *.key, and *.pem files to .gitignore, and prefer ESO so plain-text values never live in the repo.
Restrict access with RBAC — grant get on only the specific Secrets a workload needs, rather than blanket secret access in the namespace.
Rotate regularly, and remember env-var secrets require a kubectl rollout restart to take effect.
Isolate by namespace — keep each application's secrets in its own namespace.

Troubleshooting

Secret not found

kubectl get secret my-secret -n my-namespace
kubectl get secrets -A | grep my-secret

Pod cannot access a secret

kubectl get pod my-pod -n my-namespace -o yaml | grep -A5 secretKeyRef
kubectl auth can-i get secrets -n my-namespace \
  --as=system:serviceaccount:my-namespace:default

Secret not updating in a pod — env-var secrets do not auto-update; restart the workload with kubectl rollout restart deployment my-app -n my-namespace. Volume-mounted secrets update automatically unless mounted with subPath.

ExternalSecret not syncing — check its status and events for the reason (missing remote key, store permissions, or a malformed remoteRef):

kubectl describe externalsecret my-app -n my-namespace

Kubernetes Secrets and the External Secrets Operator

On this page