Secrets Management in Kubernetes: Vault vs External Secrets

Introduction

By late 2019, secrets management had become a critical security concern for Kubernetes deployments. While Kubernetes provided a native Secrets API, teams needed more sophisticated solutions for secret rotation, audit logging, and integration with external systems. HashiCorp Vault and the emerging External Secrets Operator offered different approaches to managing sensitive information in Kubernetes.

This mattered because secrets sprawl and static credentials were common security vulnerabilities. Teams needed solutions that could rotate secrets automatically, provide audit trails, and integrate with existing identity systems. Vault and External Secrets Operator addressed these needs with different architectural approaches and trade-offs.

Historical note: HashiCorp Vault had been available since 2015, but 2019 saw increased adoption for Kubernetes secrets management. External Secrets Operator was in early development, providing a Kubernetes-native approach to external secret management.

Kubernetes Secrets: Built-in Limitations

Native Secrets API

Kubernetes Secrets provide basic secret storage:

apiVersion: v1
kind: Secret
metadata:
  name: my-secret
type: Opaque
data:
  username: YWRtaW4=
  password: cGFzc3dvcmQ=

Limitations

• No Encryption at Rest: Secrets stored in etcd are base64-encoded, not encrypted (unless etcd encryption is enabled).
• No Rotation: Manual secret rotation requires updating Secrets and restarting pods.
• No Audit Trail: Limited visibility into secret access and usage.
• No External Integration: Difficult to integrate with external secret management systems.

HashiCorp Vault Integration

Vault Architecture

• Centralized Secret Store: Vault provides a centralized secret management system.
• Dynamic Secrets: Generate short-lived credentials on-demand.
• Secret Engines: Support for various secret backends (AWS, Azure, databases, etc.).
• Access Control: Fine-grained access policies for secret access.

Vault Agent Integration

# Vault Agent Sidecar
apiVersion: v1
kind: Pod
metadata:
  name: app-with-vault
spec:
  containers:
  - name: app
    image: my-app:latest
    env:
    - name: DB_PASSWORD
      valueFrom:
        secretKeyRef:
          name: vault-secret
          key: password
  - name: vault-agent
    image: vault:latest
    args:
    - agent
    - -config=/vault/config/agent.hcl
    volumeMounts:
    - name: vault-secrets
      mountPath: /vault/secrets
  volumes:
  - name: vault-secrets
    emptyDir: {}

Vault Kubernetes Auth

# ServiceAccount for Vault authentication
apiVersion: v1
kind: ServiceAccount
metadata:
  name: app-service-account
---
# Vault policy
path "secret/data/app/*" {
  capabilities = ["read"]
}

External Secrets Operator

Operator Architecture

• Kubernetes-Native: Uses CRDs to manage external secrets.
• Multiple Backends: Supports Vault, AWS Secrets Manager, Azure Key Vault, GCP Secret Manager, etc.
• Automatic Sync: Automatically syncs secrets from external systems to Kubernetes Secrets.
• Secret Rotation: Can trigger pod restarts when secrets are updated.

External Secrets Configuration

# SecretStore (Vault backend)
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      server: "https://vault.example.com:8200"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "app-role"
          serviceAccountRef:
            name: "external-secrets-sa"
---
# ExternalSecret
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: app-secrets
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: app-secrets
    creationPolicy: Owner
  data:
  - secretKey: db-password
    remoteRef:
      key: secret/data/app
      property: password

Comparison: Vault vs External Secrets Operator

Comparison: Kubernetes Secrets vs External Management

Best Practices

Secret Rotation

• Automated Rotation: Use Vault dynamic secrets or scheduled rotation.
• Graceful Rollout: Coordinate secret rotation with pod restarts.
• Backward Compatibility: Support multiple secret versions during rotation.

Secret Access

• Least Privilege: Grant minimal access to secrets.
• Service Accounts: Use dedicated service accounts for secret access.
• Audit Logging: Enable audit logging for secret access.

Secret Storage

• Encryption: Ensure secrets are encrypted at rest.
• Backup: Backup secret stores regularly.
• Disaster Recovery: Plan for secret store failures.

Practical Considerations

Vault Deployment

Vault requires infrastructure:

• High Availability: Vault cluster for production.
• Storage Backend: etcd, Consul, or cloud storage.
• Network Access: Pods need network access to Vault.
• Authentication: Configure Kubernetes auth method.

External Secrets Operator

External Secrets Operator is simpler:

• Operator Deployment: Deploy operator once.
• CRD Management: Manage secrets via CRDs.
• Backend Configuration: Configure SecretStore for each backend.
• Automatic Sync: Operator handles secret synchronization.

Getting Started

Vault Integration

# Install Vault
helm repo add hashicorp https://helm.releases.hashicorp.com
helm install vault hashicorp/vault

# Configure Kubernetes auth
vault auth enable kubernetes
vault write auth/kubernetes/config \
  token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
  kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

External Secrets Operator

# Install External Secrets Operator
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets

# Create SecretStore
kubectl apply -f secretstore.yaml

# Create ExternalSecret
kubectl apply -f externalsecret.yaml

Caveats & Lessons Learned

• Secret Sprawl: Too many secrets make management difficult; consolidate where possible.
• Rotation Coordination: Secret rotation requires pod restarts; plan for downtime.
• Backend Dependencies: External secret management depends on external systems; plan for failures.
• Access Control: Proper access control is critical; use RBAC and Vault policies.

Common Failure Modes

• “Secret sync failures”: External Secrets Operator can’t sync from backend; check connectivity and credentials.
• “Vault unavailable”: Pods can’t access Vault; ensure network connectivity and authentication.
• “Secret rotation breaks apps”: Apps fail after secret rotation; coordinate rotation with deployments.

Conclusion

Secrets management in Kubernetes had evolved significantly by late 2019. While Kubernetes Secrets provided basic functionality, HashiCorp Vault and External Secrets Operator offered enterprise-grade solutions for secret rotation, audit logging, and external system integration. Teams could now choose between Vault’s centralized approach and External Secrets Operator’s Kubernetes-native pattern based on their infrastructure and operational preferences.

For organizations with enterprise security requirements, external secret management became essential for production deployments. It demonstrated that Kubernetes security didn’t have to rely on base64-encoded secrets—it could leverage sophisticated secret management systems with rotation, audit trails, and integration capabilities.

The patterns and practices established with Vault and External Secrets Operator would influence the development of advanced secret management tools (K8s Pro Sentinel, Sealed Secrets) and set the foundation for zero-trust secret management in Kubernetes. Secrets management had evolved from a basic storage mechanism to a comprehensive security capability.