Secrets

Secrets store and manage sensitive data like passwords, OAuth tokens, SSH keys, and TLS certificates. Similar to ConfigMaps, Secrets can be injected into pods as environment variables or mounted as files, but they’re designed specifically for sensitive information with additional security considerations.

What Are Secrets?

A Secret is a Kubernetes object that stores sensitive data in an encoded format. While not encrypted by default, Secrets provide a way to separate sensitive information from pod definitions and container images, reducing the risk of accidental exposure.

graph TB A[Secret] --> B[Sensitive Data] B --> C[Passwords] B --> D[API Keys] B --> E[TLS Certificates] B --> F[Tokens] C --> G[Injected into Pods] D --> G E --> G F --> G style A fill:#e1f5ff style B fill:#ffe1e1 style G fill:#fff4e1

Why Use Secrets?

Secrets provide:

✅ Separation of sensitive data - Keep secrets out of images and pod definitions
✅ Access control - Kubernetes RBAC controls who can access Secrets
✅ Audit trail - Track Secret access through Kubernetes audit logs
✅ Base64 encoding - Basic obfuscation (not encryption by default)
✅ Multiple consumption methods - Environment variables, volume mounts, image pull secrets
✅ Encryption at rest - Can be enabled for etcd encryption

Secret vs ConfigMap

Secrets are similar to ConfigMaps but designed for sensitive data:

graph TB subgraph secret[Secret] A[Secret] --> B[Sensitive Data] B --> C[Base64 Encoded] B --> D[Passwords, Keys, Certificates] B --> E[RBAC Protected] end subgraph configmap[ConfigMap] F[ConfigMap] --> G[Non-Sensitive Data] G --> H[Plain Text] G --> I[Configuration, Settings] G --> J[Less Restricted] end style A fill:#ffe1e1 style B fill:#fff4e1 style F fill:#e1f5ff style G fill:#e8f5e9

Use Secrets for:

Passwords, tokens, API keys
TLS certificates and keys
SSH keys
Docker registry credentials
Any sensitive data

Use ConfigMaps for:

Application configuration
Non-sensitive settings
Environment variables (non-sensitive)

Secret Types

Kubernetes provides several built-in Secret types:

Opaque - Arbitrary user-defined data (default)
kubernetes.io/dockerconfigjson - Docker registry credentials
kubernetes.io/tls - TLS certificate and key
kubernetes.io/service-account-token - Service account token
kubernetes.io/basic-auth - Basic authentication
kubernetes.io/ssh-auth - SSH authentication

Creating Secrets

1. From Literal Values

kubectl create secret generic my-secret \
  --from-literal=username=admin \
  --from-literal=password=secret123

2. From Files

# From a single file
kubectl create secret generic my-secret --from-file=password.txt

# From multiple files
kubectl create secret generic my-secret \
  --from-file=username=./username.txt \
  --from-file=password=./password.txt

# From a directory
kubectl create secret generic my-secret --from-file=./secrets/

3. From YAML

apiVersion: v1
kind: Secret
metadata:
  name: my-secret
type: Opaque
data:
  username: YWRtaW4=  # base64 encoded "admin"
  password: c2VjcmV0MTIz  # base64 encoded "secret123"

Note: Values in YAML must be base64 encoded. Encode with:

echo -n "admin" | base64
# Output: YWRtaW4=

4. TLS Secret

# From certificate and key files
kubectl create secret tls my-tls-secret \
  --cert=tls.crt \
  --key=tls.key

Or from YAML:

apiVersion: v1
kind: Secret
metadata:
  name: my-tls-secret
type: kubernetes.io/tls
data:
  tls.crt: <base64-encoded-cert>
  tls.key: <base64-encoded-key>

5. Docker Registry Secret

kubectl create secret docker-registry my-registry-secret \
  --docker-server=docker.io \
  --docker-username=myuser \
  --docker-password=mypassword \
  --docker-email=[email protected]

Using Secrets in Pods

Secrets can be used in the same ways as ConfigMaps:

1. As Environment Variables

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
  - name: app
    image: my-app:latest
    env:
    - name: DB_PASSWORD
      valueFrom:
        secretKeyRef:
          name: my-secret
          key: password
    # Or inject all keys
    envFrom:
    - secretRef:
        name: my-secret

2. As Volume Mounts

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
  - name: app
    image: my-app:latest
    volumeMounts:
    - name: secret-volume
      mountPath: /etc/secret
      readOnly: true
  volumes:
  - name: secret-volume
    secret:
      secretName: my-secret

3. For Image Pull Secrets

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  imagePullSecrets:
  - name: my-registry-secret
  containers:
  - name: app
    image: private-registry.io/my-app:latest

Or at the ServiceAccount level (recommended):

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-serviceaccount
imagePullSecrets:
- name: my-registry-secret

Secret Volume Mounts

When mounted as a volume, Secret keys become filenames, and decoded values become file contents:

graph LR A[Secret] --> B[username: admin<br/>base64: YWRtaW4=] A --> C[password: secret123<br/>base64: c2VjcmV0MTIz] B --> D[/etc/secret/username<br/>decoded: admin] C --> E[/etc/secret/password<br/>decoded: secret123] style A fill:#ffe1e1 style D fill:#fff4e1 style E fill:#fff4e1

Mounting Specific Keys

volumes:
- name: secret-volume
  secret:
    secretName: my-secret
    items:
    - key: username
      path: db/username
    - key: password
      path: db/password
    defaultMode: 0400  # Read-only permissions

Security Considerations

Base64 Encoding is Not Encryption

Secrets are base64 encoded, not encrypted. Base64 is encoding, not encryption—anyone with access can decode:

# Viewing a Secret decodes it automatically
kubectl get secret my-secret -o jsonpath='{.data.password}' | base64 -d

Enable Encryption at Rest

For production, enable etcd encryption at rest:

# Encryption configuration
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
  - secrets
  providers:
  - aescbc:
      keys:
      - name: key1
        secret: <base64-encoded-key>
  - identity: {}  # Fallback

RBAC for Secret Access

Use RBAC to control who can access Secrets:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: secret-reader
rules:
- apiGroups: [""]
  resources: ["secrets"]
  resourceNames: ["my-secret"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-secrets
subjects:
- kind: User
  name: developer
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: secret-reader
  apiGroup: rbac.authorization.k8s.io

Secret Rotation

Rotate Secrets regularly:

# Update Secret
kubectl create secret generic my-secret \
  --from-literal=password=newpassword \
  --dry-run=client -o yaml | kubectl apply -f -

# Restart pods to pick up new Secret (if using env vars)
kubectl rollout restart deployment my-deployment

Secret Lifecycle

graph TD A[Secret Created] --> B[Secret Stored in etcd] B --> C[Referenced by Pod] C --> D[Secret Available in Pod] E[Secret Updated] --> F{Usage Type} F -->|Volume Mount| G[Auto-Updated in Pod] F -->|Environment Var| H[Requires Pod Restart] I[Secret Deleted] --> J{Referenced by Pods?} J -->|Yes| K[Pods May Fail] J -->|No| L[Secret Removed] style A fill:#e1f5ff style D fill:#e8f5e9 style G fill:#fff4e1 style K fill:#ffe1e1

Common Use Cases

1. Database Credentials

apiVersion: v1
kind: Secret
metadata:
  name: db-secret
type: Opaque
stringData:  # Automatically base64 encoded
  username: admin
  password: secretpassword
  host: database.example.com
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    spec:
      containers:
      - name: app
        image: my-app:latest
        env:
        - name: DB_USER
          valueFrom:
            secretKeyRef:
              name: db-secret
              key: username
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-secret
              key: password
        - name: DB_HOST
          valueFrom:
            secretKeyRef:
              name: db-secret
              key: host

2. TLS Certificates

apiVersion: v1
kind: Secret
metadata:
  name: tls-secret
type: kubernetes.io/tls
data:
  tls.crt: <base64-encoded-certificate>
  tls.key: <base64-encoded-key>
---
apiVersion: v1
kind: Pod
metadata:
  name: nginx-tls
spec:
  containers:
  - name: nginx
    image: nginx:latest
    volumeMounts:
    - name: tls
      mountPath: /etc/nginx/tls
  volumes:
  - name: tls
    secret:
      secretName: tls-secret

3. Docker Registry Authentication

apiVersion: v1
kind: Secret
metadata:
  name: registry-secret
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: <base64-encoded-docker-config>
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    spec:
      imagePullSecrets:
      - name: registry-secret
      containers:
      - name: app
        image: private-registry.io/my-app:latest

4. API Keys

apiVersion: v1
kind: Secret
metadata:
  name: api-keys
type: Opaque
stringData:
  stripe-key: sk_live_...
  sendgrid-key: SG....
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    spec:
      containers:
      - name: app
        image: my-app:latest
        envFrom:
        - secretRef:
            name: api-keys

Best Practices

Enable encryption at rest - Encrypt Secrets in etcd for production
Use RBAC - Restrict who can read and modify Secrets
Use stringData for convenience - Kubernetes encodes it automatically

stringData:  # Automatically base64 encoded
  password: mypassword

Don’t commit Secrets to Git - Use external secret management tools
Rotate Secrets regularly - Change passwords and keys periodically
Use volume mounts for files - Better for certificates and keys
Set appropriate file permissions - Use defaultMode for mounted Secrets

volumes:
- name: secret-volume
  secret:
    secretName: my-secret
    defaultMode: 0400  # Read-only

Use namespaces - Isolate Secrets with namespaces
Limit Secret size - Secrets have a 1 MiB size limit
Use external secret stores - Consider tools like External Secrets Operator, Vault, etc.
Monitor Secret access - Enable audit logging
Use immutable Secrets - For Secrets that never change (Kubernetes 1.19+)

apiVersion: v1
kind: Secret
metadata:
  name: my-secret
immutable: true
data:
  # ...

Common Operations

Create Secret

# From literal
kubectl create secret generic my-secret --from-literal=key=value

# From file
kubectl create secret generic my-secret --from-file=password.txt

# From YAML
kubectl create -f secret.yaml

View Secret

# List Secrets (data is hidden)
kubectl get secrets

# View Secret metadata
kubectl get secret my-secret -o yaml

# Decode Secret value
kubectl get secret my-secret -o jsonpath='{.data.password}' | base64 -d

Update Secret

# Edit Secret
kubectl edit secret my-secret

# Update from file
kubectl create secret generic my-secret \
  --from-literal=password=newpassword \
  --dry-run=client -o yaml | kubectl apply -f -

# Restart pods (if using env vars)
kubectl rollout restart deployment my-deployment

Delete Secret

kubectl delete secret my-secret

Troubleshooting

Secret Not Found

# Verify Secret exists
kubectl get secret my-secret

# Check namespace
kubectl get secret my-secret -n <namespace>

# Check pod references
kubectl describe pod my-pod | grep -A 10 "Environment\|Mounts"

Access Denied

# Check RBAC permissions
kubectl auth can-i get secrets/my-secret

# Check Role/RoleBinding
kubectl get role,rolebinding | grep secret

Secret Not Updating

# For env vars, restart pods
kubectl rollout restart deployment my-deployment

# For volume mounts, check sync period
kubectl describe pod my-pod | grep "Events"