AWS VPC CNI 1.5: IP Address Optimization for EKS

Introduction

On July 15, 2019, AWS released VPC CNI Plugin version 1.5, focusing on IP address optimization and better resource utilization. The release improved the WARM_IP_TARGET parameter behavior to automatically return unassigned IP addresses back to VPC subnets, addressing one of the most common operational challenges with VPC CNI: IP address exhaustion.

IP address management in VPC CNI is a balancing act: pre-allocate too many IPs and you waste subnet capacity; pre-allocate too few and pods wait for IP allocation. Version 1.5 made this balance more automatic and efficient.

Key Improvements

1. WARM_IP_TARGET Enhancement: Improved behavior to return unassigned IPs back to subnets automatically.
2. IP Address Recycling: Better management of IP address lifecycle, reducing waste.
3. Default Configuration: Made VPC CNI 1.5 the default for new EKS clusters.
4. Operational Improvements: Better logging and metrics for IP address management.

WARM_IP_TARGET Improvements

The WARM_IP_TARGET parameter controls how many IP addresses to keep warm (pre-allocated) on each node:

apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-vpc-cni
  namespace: kube-system
data:
  WARM_IP_TARGET: "2"

What Changed in 1.5:

• Automatic Return: Unused IPs are automatically returned to the subnet when not needed.
• Better Thresholds: More intelligent decision-making about when to keep vs. return IPs.
• Reduced Waste: Less IP address waste during pod churn and scaling events.

IP Address Lifecycle

1. Pre-allocation: VPC CNI pre-allocates IPs based on WARM_IP_TARGET.
2. Pod Assignment: IPs are assigned to pods when they start.
3. Pod Termination: When pods terminate, IPs become available.
4. Automatic Return: Unused IPs are returned to the subnet if not needed for new pods.

Getting Started

VPC CNI 1.5 became the default for new EKS clusters. To upgrade existing clusters:

kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/release-1.5/config/master/aws-k8s-cni.yaml

Monitor IP address usage:

kubectl logs -n kube-system -l app=aws-node | grep -i "ip.*target"

Configuration Tuning

For High Pod Churn

WARM_IP_TARGET: "1"

Lower values reduce IP waste but may increase pod startup time during bursts.

For Stable Workloads

WARM_IP_TARGET: "5"

Higher values improve pod startup time but consume more IP addresses.

For Large Clusters

WARM_IP_TARGET: "2"
MINIMUM_IP_TARGET: "2"

Balance between startup performance and IP utilization.

Use Cases

• Dynamic Workloads: Clusters with frequent pod creation and termination benefit from automatic IP return.
• IP-Constrained Environments: Environments with limited subnet IP addresses need efficient utilization.
• Cost Optimization: Reducing IP waste helps optimize VPC subnet usage.
• Large Clusters: Better IP management supports larger cluster sizes.

Operational Benefits

• Reduced IP Waste: Automatic return of unused IPs prevents accumulation.
• Better Scalability: More efficient IP usage supports larger clusters.
• Simplified Operations: Less manual intervention needed for IP management.
• Cost Savings: Better IP utilization reduces need for larger subnets.

Comparison: 1.1 vs 1.5

Common Patterns

• Gradual Scaling: Use lower WARM_IP_TARGET for gradual scaling patterns.
• Burst Workloads: Use higher WARM_IP_TARGET for workloads with sudden scaling.
• Mixed Workloads: Balance WARM_IP_TARGET based on dominant workload pattern.
• Monitoring: Track IP utilization metrics to tune WARM_IP_TARGET over time.

Limitations

• Subnet Size: Still constrained by VPC subnet IP address limits.
• ENI Limits: EC2 instance ENI limits still apply.
• Regional Constraints: Works only in AWS regions with EKS support.
• Tuning Required: Optimal WARM_IP_TARGET varies by workload pattern.

Looking Ahead

VPC CNI 1.5 set the foundation for:

• MINIMUM_IP_TARGET: Future parameter to ensure minimum IP availability.
• Better Metrics: Enhanced IP utilization metrics and monitoring.
• Advanced Features: Pod security groups and automatic subnet discovery.
• Performance Improvements: Faster IP allocation and pod startup.

Summary

AWS VPC CNI 1.5 addressed one of the most common operational challenges: IP address waste. By automatically returning unused IPs to subnets and improving the WARM_IP_TARGET behavior, it made VPC CNI more efficient and easier to operate at scale.