StatefulSets

StatefulSets are designed for stateful applications that need stable network identity, stable persistent storage, and ordered deployment and scaling. Unlike Deployments, where pods are interchangeable, StatefulSet pods have unique identities that persist across rescheduling.

What Are StatefulSets?

A StatefulSet manages pods that have stable, unique network identities and stable storage. Each pod in a StatefulSet gets a persistent identifier that is maintained even if the pod is rescheduled to a different node. This makes StatefulSets ideal for databases, message queues, and other distributed systems that require stable identity and persistent data.

Why Use StatefulSets?

StatefulSets provide features essential for stateful applications:

✅ Stable pod identity - Each pod has a persistent, unique name
✅ Stable network identity - Predictable hostnames and DNS entries
✅ Stable storage - Each pod gets its own persistent volume
✅ Ordered deployment - Pods created and terminated in order
✅ Ordered scaling - Scale up and down in a controlled sequence
✅ Graceful shutdown - Pods terminated in reverse order

StatefulSet vs Deployment

The key difference is how pods are identified and managed:

Use Deployments when:

• Pods are stateless and interchangeable
• Any pod can handle any request
• Storage can be shared or ephemeral

Use StatefulSets when:

• Pods have unique identity and roles
• Each pod needs its own persistent storage
• Pods must start/stop in order
• Stable network identity is required

Stable Pod Identity

Each pod in a StatefulSet gets a stable name based on the StatefulSet name and an ordinal index:

<statefulset-name>-<ordinal>

Example: A StatefulSet named mysql with 3 replicas creates:

• mysql-0
• mysql-1
• mysql-2

These names persist even if pods are deleted and recreated.

Stable Network Identity

StatefulSets create stable network identities using headless Services. Each pod gets a stable DNS name:

<pod-name>.<service-name>.<namespace>.svc.cluster.local

For a pod mysql-0 in a StatefulSet with service mysql:

mysql-0.mysql.default.svc.cluster.local

This allows applications to discover and connect to specific pod instances reliably.

Stable Storage

Each pod in a StatefulSet gets its own PersistentVolumeClaim (PVC). The PVC is created automatically with a stable name:

<volume-claim-template-name>-<statefulset-name>-<ordinal>

Example: For a volumeClaimTemplate named data in a StatefulSet mysql:

• data-mysql-0
• data-mysql-1
• data-mysql-2

When a pod is deleted and recreated, it reattaches to the same PVC, preserving data.

Ordered Deployment and Scaling

StatefulSets deploy and scale pods in a strict order:

Ordered Deployment

Pods are created sequentially, starting from index 0:

1. mysql-0 starts and becomes Ready
2. mysql-1 starts and becomes Ready
3. mysql-2 starts and becomes Ready

Ordered Scaling

Scaling Up:

• Pods are created in order (0, 1, 2, …)

Scaling Down:

• Pods are terminated in reverse order (2, 1, 0)
• Each pod must be fully terminated before the next one starts terminating

Basic StatefulSet Example

Here’s a StatefulSet running MySQL with persistent storage:

apiVersion: v1
kind: Service
metadata:
  name: mysql
  labels:
    app: mysql
spec:
  ports:
  - port: 3306
    name: mysql
  clusterIP: None  # Headless service
  selector:
    app: mysql
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
spec:
  serviceName: mysql  # Must match Service name
  replicas: 3
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - name: mysql
        image: mysql:8.0
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-secret
              key: password
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: data
          mountPath: /var/lib/mysql
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: fast-ssd
      resources:
        requests:
          storage: 10Gi

Key fields:

• serviceName - Name of the headless Service (required)
• volumeClaimTemplates - Templates for creating PVCs for each pod
• podManagementPolicy - Ordered (default) or Parallel

StatefulSet Lifecycle

Update Strategies

StatefulSets support two update strategies:

RollingUpdate (Default)

Updates pods one at a time in reverse ordinal order:

spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 0  # All pods updated (can use for canary)

You can use partition for canary updates:

• partition: 2 - Only update pods with index >= 2
• partition: 1 - Update pods with index >= 1
• partition: 0 - Update all pods

OnDelete

Pods are updated only when manually deleted:

spec:
  updateStrategy:
    type: OnDelete

Use Cases

StatefulSets are ideal for:

✅ Databases - MySQL, PostgreSQL, MongoDB clusters
✅ Message queues - RabbitMQ, Kafka
✅ Distributed systems - etcd, Consul, Elasticsearch
✅ Applications requiring stable identity - Any app where pods have unique roles
✅ Applications needing persistent storage - Each pod needs its own data directory

Example: MySQL Replication

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
spec:
  serviceName: mysql
  replicas: 3
  template:
    spec:
      containers:
      - name: mysql
        image: mysql:8.0
        args:
        - --server-id=$(echo $HOSTNAME | sed 's/mysql-//')
        - --log-bin
        - --gtid-mode=ON
        - --enforce-gtid-consistency
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-secret
              key: password
        volumeMounts:
        - name: data
          mountPath: /var/lib/mysql
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 50Gi

Best Practices

1. Always create a headless Service - Required for stable network identity

2. Use volumeClaimTemplates - Ensures each pod gets persistent storage

3. Set resource requests and limits - Important for scheduling

4. Use appropriate access modes - ReadWriteOnce for databases, ReadWriteMany if shared storage needed

5. Plan for ordered operations - Be aware that scaling takes time due to ordering

6. Use health probes - Ensure pods are ready before continuing deployment

livenessProbe:
  exec:
    command:
    - /bin/sh
    - -c
    - mysqladmin ping -h localhost
  initialDelaySeconds: 30
  periodSeconds: 10

readinessProbe:
  exec:
    command:
    - /bin/sh
    - -c
    - mysqladmin ping -h localhost
  initialDelaySeconds: 5
  periodSeconds: 5

7. Handle graceful shutdown - Use preStop hooks for clean termination

lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "mysqladmin shutdown"]

8. Monitor pod identities - Ensure applications use stable DNS names

9. Test backup and restore - Verify data persistence across pod rescheduling

10. Use podManagementPolicy: Parallel for independent pods - Only if pods don’t need ordered startup

Common Operations

View StatefulSet Status

# List StatefulSets
kubectl get statefulsets
kubectl get sts

# Detailed information
kubectl describe statefulset mysql

# View pods (note the stable names)
kubectl get pods -l app=mysql

Scaling

# Scale up
kubectl scale statefulset mysql --replicas=5

# Scale down (pods terminated in reverse order)
kubectl scale statefulset mysql --replicas=2

Updating

# Update container image
kubectl set image statefulset mysql mysql=mysql:8.0.33

# Check rollout status
kubectl rollout status statefulset mysql

# Rollback
kubectl rollout undo statefulset mysql

Accessing Pods

# Access a specific pod by name
kubectl exec -it mysql-0 -- mysql -u root -p

# Connect using stable DNS
kubectl run -it --rm client --image=mysql:8.0 --restart=Never -- mysql -h mysql-0.mysql

Troubleshooting

Pods Not Starting

# Check StatefulSet events
kubectl describe sts mysql

# Check pod events
kubectl describe pod mysql-0

# Check PVCs
kubectl get pvc -l app=mysql

Storage Issues

# Verify PVCs are bound
kubectl get pvc

# Check storage class
kubectl get storageclass

# Verify pod can mount volume
kubectl describe pod mysql-0

Network Issues

# Verify headless service exists
kubectl get svc mysql

# Test DNS resolution
kubectl run -it --rm test --image=busybox --restart=Never -- nslookup mysql-0.mysql

See Also

• Deployments - For stateless applications
• Persistent Volumes - Understanding persistent storage
• Services - Headless Services for StatefulSets
• Storage Classes - Dynamic storage provisioning