EKS Storage

EKS supports multiple storage options for persistent data: EBS volumes for block storage, EFS for shared file storage, and FSx for Lustre for high-performance computing workloads. Understanding when to use each storage type and how to configure them is essential for running stateful applications on EKS.

Storage Overview

Kubernetes abstracts storage through PersistentVolumes (PVs) and PersistentVolumeClaims (PVCs). On EKS, these map to AWS storage services:

graph TB APP[Application Pod] --> PVC[PersistentVolumeClaim] PVC --> PV[PersistentVolume] PV --> STORAGE[AWS Storage] STORAGE --> EBS[EBS Volume Block Storage] STORAGE --> EFS[EFS Filesystem Shared File Storage] STORAGE --> FSX[FSx for Lustre High-Performance] style APP fill:#e1f5ff style PVC fill:#fff4e1 style PV fill:#e8f5e9 style EBS fill:#f3e5f5

Storage Classes:

Define storage types and parameters
Enable dynamic provisioning
Specify volume binding modes
Configure default storage class

PersistentVolumeClaims:

Request storage from a storage class
Specify size and access modes
Automatically provision volumes

PersistentVolumes:

Actual storage resources
Created by storage provisioner
Bound to PVCs

EBS CSI Driver

The EBS Container Storage Interface (CSI) driver provides block storage using Amazon EBS volumes. EBS volumes are network-attached block storage devices that persist independently of EC2 instance lifecycles.

Architecture

graph TB POD[Pod] --> PVC[PVC: 20Gi] PVC --> SC[StorageClass: gp3] SC --> CSI[EBS CSI Driver] CSI --> EBS[EBS Volume gp3, 20Gi] EBS --> NODE[Worker Node] POD -->|Mount| NODE style POD fill:#e1f5ff style EBS fill:#fff4e1 style CSI fill:#e8f5e9

Installation

The EBS CSI driver is available as an EKS add-on:

# Install EBS CSI driver add-on
aws eks create-addon \
  --cluster-name my-cluster \
  --addon-name aws-ebs-csi-driver \
  --addon-version latest

# Or using eksctl
eksctl create addon \
  --name aws-ebs-csi-driver \
  --cluster my-cluster \
  --version latest

IAM Role for EBS CSI Driver

The driver needs IAM permissions to create and manage EBS volumes:

# Create IAM policy
cat > ebs-csi-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:CreateSnapshot",
        "ec2:AttachVolume",
        "ec2:DetachVolume",
        "ec2:ModifyVolume",
        "ec2:DescribeAvailabilityZones",
        "ec2:DescribeInstances",
        "ec2:DescribeSnapshots",
        "ec2:DescribeTags",
        "ec2:DescribeVolumes",
        "ec2:DescribeVolumesModifications"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "ec2:CreateTags"
      ],
      "Resource": [
        "arn:aws:ec2:*:*:volume/*",
        "arn:aws:ec2:*:*:snapshot/*"
      ],
      "Condition": {
        "StringEquals": {
          "ec2:CreateAction": [
            "CreateVolume",
            "CreateSnapshot"
          ]
        }
      }
    },
    {
      "Effect": "Allow",
      "Action": [
        "ec2:DeleteTags"
      ],
      "Resource": [
        "arn:aws:ec2:*:*:volume/*",
        "arn:aws:ec2:*:*:snapshot/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "ec2:CreateVolume"
      ],
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "ec2:VolumeType": [
            "gp3",
            "gp2"
          ]
        }
      }
    },
    {
      "Effect": "Allow",
      "Action": [
        "ec2:CreateVolume"
      ],
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "ec2:VolumeType": [
            "io1",
            "io2"
          ]
        },
        "StringLike": {
          "ec2:Encrypted": "true"
        }
      }
    },
    {
      "Effect": "Allow",
      "Action": [
        "ec2:DeleteVolume"
      ],
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "ec2:VolumeSource": "volume"
        }
      }
    },
    {
      "Effect": "Allow",
      "Action": [
        "ec2:DeleteSnapshot"
      ],
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "ec2:SnapshotSource": "snapshot"
        }
      }
    }
  ]
}
EOF

# Create IAM role with IRSA
eksctl create iamserviceaccount \
  --name ebs-csi-controller-sa \
  --namespace kube-system \
  --cluster my-cluster \
  --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
  --approve \
  --role-only \
  --role-name AmazonEKS_EBS_CSI_DriverRole

Storage Classes

Define storage classes for different EBS volume types:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  encrypted: "true"
  kmsKeyId: arn:aws:kms:us-west-2:123456789012:key/12345678-1234-1234-1234-123456789012
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
reclaimPolicy: Delete

EBS Volume Types:

Type	Use Case	IOPS	Throughput	Cost
gp3	General purpose (default)	3,000-16,000	125-1,000 MB/s	Low
gp2	General purpose (legacy)	3-16,000	125-250 MB/s	Medium
io1	High IOPS	100-64,000	125-1,000 MB/s	High
io2	High IOPS (latest)	100-64,000	125-1,000 MB/s	High
st1	Throughput optimized	500	500 MB/s	Low
sc1	Cold storage	250	250 MB/s	Very Low

Storage Class Parameters:

type - EBS volume type (gp3, gp2, io1, io2, st1, sc1)
encrypted - Enable encryption (true/false)
kmsKeyId - KMS key for encryption
fsType - Filesystem type (ext4, xfs)
volumeBindingMode - WaitForFirstConsumer or Immediate

Creating Persistent Volumes

Dynamic Provisioning (Recommended):

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: gp3
  resources:
    requests:
      storage: 100Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - name: mysql
        image: mysql:8.0
        volumeMounts:
        - name: data
          mountPath: /var/lib/mysql
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: secretpassword
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: mysql-pvc

Access Modes:

ReadWriteOnce - Single node read/write (EBS default)
ReadOnlyMany - Multiple nodes read-only (not supported by EBS)
ReadWriteMany - Multiple nodes read/write (not supported by EBS, use EFS)

Volume Expansion

EBS volumes can be expanded:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3-expandable
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
allowVolumeExpansion: true  # Enable expansion

Expand a PVC:

# Edit PVC to increase size
kubectl patch pvc mysql-pvc -p '{"spec":{"resources":{"requests":{"storage":"200Gi"}}}}'

# Verify expansion
kubectl get pvc mysql-pvc

Volume Snapshots

Create snapshots for backups:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: ebs-snapshot-class
driver: ebs.csi.aws.com
deletionPolicy: Retain
---
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: mysql-snapshot
spec:
  volumeSnapshotClassName: ebs-snapshot-class
  source:
    persistentVolumeClaimName: mysql-pvc

Restore from snapshot:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pvc-restored
spec:
  dataSource:
    name: mysql-snapshot
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  accessModes:
    - ReadWriteOnce
  storageClassName: gp3
  resources:
    requests:
      storage: 100Gi

EFS CSI Driver

Amazon Elastic File System (EFS) provides shared file storage that can be accessed by multiple pods simultaneously. EFS is network-attached storage that supports NFS protocol.

Architecture

graph TB POD1[Pod 1] --> PVC1[PVC: efs] POD2[Pod 2] --> PVC2[PVC: efs] POD3[Pod 3] --> PVC3[PVC: efs] PVC1 --> EFS[EFS Filesystem] PVC2 --> EFS PVC3 --> EFS EFS --> MOUNT1[Mount Target 1 AZ-1] EFS --> MOUNT2[Mount Target 2 AZ-2] EFS --> MOUNT3[Mount Target 3 AZ-3] style POD1 fill:#e1f5ff style POD2 fill:#e1f5ff style POD3 fill:#e1f5ff style EFS fill:#fff4e1

Installation

# Install EFS CSI driver
kubectl apply -k "github.com/kubernetes-sigs/aws-efs-csi-driver/deploy/kubernetes/overlays/stable/?ref=release-1.7"

# Create IAM role for EFS CSI driver
eksctl create iamserviceaccount \
  --name efs-csi-controller-sa \
  --namespace kube-system \
  --cluster my-cluster \
  --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEFSCSIDriverPolicy \
  --approve

Creating EFS Filesystem

# Create EFS filesystem
aws efs create-file-system \
  --creation-token eks-efs \
  --performance-mode generalPurpose \
  --throughput-mode provisioned \
  --provisioned-throughput-in-mibps 100 \
  --encrypted \
  --region us-west-2

# Create mount targets in each subnet
aws efs create-mount-target \
  --file-system-id fs-12345678 \
  --subnet-id subnet-12345678 \
  --security-groups sg-12345678

Storage Class for EFS

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: efs
provisioner: efs.csi.aws.com
parameters:
  provisioningMode: efs-ap
  fileSystemId: fs-12345678
  directoryPerms: "0755"
  gidRangeStart: "1000"
  gidRangeEnd: "2000"
  basePath: "/dynamic_provisioning"

Using EFS

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: shared-storage
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: efs
  resources:
    requests:
      storage: 1Gi  # EFS doesn't enforce size limits
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: app
        image: nginx:latest
        volumeMounts:
        - name: shared
          mountPath: /shared
      volumes:
      - name: shared
        persistentVolumeClaim:
          claimName: shared-storage

EFS Use Cases:

Shared configuration files
Content management systems
Shared application data
Log aggregation
Multi-pod read/write access

FSx for Lustre

Amazon FSx for Lustre provides high-performance file storage optimized for compute-intensive workloads like machine learning, high-performance computing, and media processing.

Architecture

graph TB POD1[Pod 1] --> PVC1[PVC: fsx] POD2[Pod 2] --> PVC2[PVC: fsx] PVC1 --> FSX[FSx for Lustre] PVC2 --> FSX FSX --> S3[S3 Bucket Data Repository] style POD1 fill:#e1f5ff style POD2 fill:#e1f5ff style FSX fill:#fff4e1 style S3 fill:#e8f5e9

Installation

# Install FSx CSI driver
kubectl apply -k "github.com/kubernetes-sigs/aws-fsx-csi-driver/deploy/kubernetes/overlays/stable/?ref=release-1.7"

# Create IAM role
eksctl create iamserviceaccount \
  --name fsx-csi-controller-sa \
  --namespace kube-system \
  --cluster my-cluster \
  --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonFSxCSIDriverPolicy \
  --approve

Creating FSx Filesystem

# Create FSx for Lustre filesystem
aws fsx create-file-system \
  --file-system-type LUSTRE \
  --storage-capacity 1200 \
  --subnet-ids subnet-12345678 \
  --security-group-ids sg-12345678 \
  --lustre-configuration DeploymentType=SCRATCH_2

Storage Class

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fsx-lustre
provisioner: fsx.csi.aws.com
parameters:
  subnetId: subnet-12345678
  securityGroupIds: sg-12345678
  deploymentType: SCRATCH_2
  perUnitStorageThroughput: "200"

FSx Use Cases:

Machine learning training
High-performance computing
Media processing
Scientific computing
Data analytics workloads

Multi-AZ Storage Considerations

EBS Volumes

EBS volumes are zonal resources:

graph TB subgraph az1[Availability Zone 1] N1[Node 1] --> EBS1[EBS Volume 1] P1[Pod 1] --> EBS1 end subgraph az2[Availability Zone 2] N2[Node 2] --> EBS2[EBS Volume 2] P2[Pod 2] --> EBS2 end style EBS1 fill:#e1f5ff style EBS2 fill:#fff4e1

Considerations:

EBS volumes are tied to a specific availability zone
Pods must be scheduled in the same AZ as the volume
Use WaitForFirstConsumer binding mode
Replicate data across AZs for high availability

EFS Filesystems

EFS is regional and accessible from all AZs:

graph TB EFS[EFS Filesystem Regional] --> MT1[Mount Target AZ-1] EFS --> MT2[Mount Target AZ-2] EFS --> MT3[Mount Target AZ-3] subgraph az1[AZ-1] N1[Node 1] --> MT1 P1[Pod 1] --> N1 end subgraph az2[AZ-2] N2[Node 2] --> MT2 P2[Pod 2] --> N2 end style EFS fill:#e1f5ff style P1 fill:#fff4e1 style P2 fill:#fff4e1

Benefits:

Accessible from all availability zones
Automatic replication across AZs
No AZ affinity required
Perfect for shared storage

Best Practices

Use gp3 for Most Workloads - Best price/performance for general use
Enable Encryption - Always encrypt EBS volumes and EFS filesystems
Use WaitForFirstConsumer - For EBS to ensure proper AZ placement
Plan for Multi-AZ - Use EFS for shared data, replicate EBS for critical data
Monitor Storage Usage - Set up CloudWatch alarms for volume usage
Use Snapshots - Regular EBS snapshots for backups
Right-Size Volumes - Start small, expand as needed (gp3 supports expansion)
Use EFS for Shared Data - When multiple pods need access
Consider FSx for HPC - For compute-intensive workloads
Tag Resources - Tag volumes for cost allocation

Common Issues

Volume Not Attaching

Problem: Pod stuck in Pending, volume not attaching

Solutions:

Check EBS CSI driver logs
Verify IAM permissions
Check security group rules
Verify volume is in same AZ as node

EFS Mount Timeout

Problem: EFS mount fails or times out

Solutions:

Verify mount targets in all subnets
Check security group rules (port 2049)
Verify network connectivity
Check EFS CSI driver logs

Volume Expansion Fails

Problem: Volume expansion doesn’t work

Solutions:

Verify storage class allows expansion
Check EBS volume type supports expansion
Ensure pod is running (expansion requires mounted volume)
Check EBS CSI driver version

EKS Storage

Storage Overview

EBS CSI Driver

Architecture

Installation

IAM Role for EBS CSI Driver

Storage Classes

Creating Persistent Volumes

Volume Expansion

Volume Snapshots

EFS CSI Driver

Architecture

Installation

Creating EFS Filesystem

Storage Class for EFS

Using EFS

FSx for Lustre

Architecture

Installation

Creating FSx Filesystem

Storage Class

Multi-AZ Storage Considerations

EBS Volumes

EFS Filesystems

Best Practices

Common Issues

Volume Not Attaching

EFS Mount Timeout

Volume Expansion Fails

See Also