AWS VPC CNI 1.6: Pod Start Time Optimization

Introduction

On February 10, 2020, AWS released VPC CNI Plugin version 1.6, focusing on pod startup performance and improved IP address management. The release introduced the MINIMUM_IP_TARGET parameter to ensure IP addresses are available when pods need them, reducing pod start latency—a critical metric for autoscaling and rapid deployments.

Pod startup time matters when you’re autoscaling, doing rapid deployments, or running serverless-style workloads. Waiting for IP allocation can turn a 2-second pod start into a 10-second start, which compounds across hundreds of pods. VPC CNI 1.6 addressed this by ensuring IPs are ready before pods need them.

Key Features

1. MINIMUM_IP_TARGET: New parameter ensures minimum IP addresses are always available.
2. Reduced Pod Start Time: Faster pod startup by pre-allocating IPs based on minimum target.
3. Peered VPC Support: Improved support for VPC peering scenarios.
4. Better IP Management: Enhanced IP address allocation and recycling logic.

MINIMUM_IP_TARGET Parameter

The MINIMUM_IP_TARGET parameter ensures a minimum number of IP addresses are always available on each node:

apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-vpc-cni
  namespace: kube-system
data:
  WARM_IP_TARGET: "2"
  MINIMUM_IP_TARGET: "2"

How It Works:

• Guarantees Availability: Ensures IPs are available even during IP return operations.
• Reduces Latency: Pods don’t wait for IP allocation when MINIMUM_IP_TARGET is met.
• Works with WARM_IP_TARGET: Complements WARM_IP_TARGET for optimal IP management.

Pod Start Time Improvement

Before 1.6:

• Pods might wait for IP allocation if WARM_IP_TARGET IPs were returned to subnet.
• IP allocation could add 1-3 seconds to pod startup time.

After 1.6:

• MINIMUM_IP_TARGET ensures IPs are always available.
• Pod startup time reduced, especially during scaling events.

Getting Started

VPC CNI 1.6 became the default for new EKS clusters. To upgrade:

kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/release-1.6/config/master/aws-k8s-cni.yaml

Configure minimum IP target:

apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-vpc-cni
  namespace: kube-system
data:
  WARM_IP_TARGET: "2"
  MINIMUM_IP_TARGET: "2"

Configuration Examples

For Autoscaling Workloads

WARM_IP_TARGET: "3"
MINIMUM_IP_TARGET: "2"

Ensures IPs are available when autoscaler creates new pods.

For Stable Workloads

WARM_IP_TARGET: "1"
MINIMUM_IP_TARGET: "1"

Lower values for workloads with predictable scaling patterns.

For High-Churn Workloads

WARM_IP_TARGET: "5"
MINIMUM_IP_TARGET: "3"

Higher values for workloads with frequent pod creation/termination.

Peered VPC Support

VPC CNI 1.6 improved support for VPC peering scenarios:

• Route Propagation: Better handling of routes in peered VPCs.
• IP Allocation: Improved IP allocation across peered VPC boundaries.
• Connectivity: Enhanced pod-to-pod connectivity across VPC peering.

Use Cases

• Autoscaling: Faster pod startup improves autoscaling responsiveness.
• Rapid Deployments: Reduced pod start time speeds up deployment cycles.
• Serverless Workloads: Better performance for serverless-style workloads with frequent scaling.
• High-Availability: Faster pod recovery improves high-availability scenarios.

Performance Impact

• Pod Start Time: Reduced by 1-3 seconds in typical scenarios.
• Scaling Events: Faster scaling during sudden traffic increases.
• IP Availability: More predictable IP availability during peak usage.

Comparison: 1.5 vs 1.6

Operational Considerations

• IP Address Planning: MINIMUM_IP_TARGET consumes more IPs; plan subnet sizes accordingly.
• Tuning: Balance WARM_IP_TARGET and MINIMUM_IP_TARGET based on workload patterns.
• Monitoring: Track pod start times and IP utilization to optimize configuration.
• Cost: Higher IP consumption may require larger subnets.

Common Patterns

• Autoscaling: Use MINIMUM_IP_TARGET to ensure IPs for autoscaler-created pods.
• Burst Workloads: Higher MINIMUM_IP_TARGET for workloads with sudden scaling.
• Cost Optimization: Lower MINIMUM_IP_TARGET for cost-sensitive environments.
• Performance Optimization: Higher MINIMUM_IP_TARGET for latency-sensitive workloads.

Limitations

• IP Consumption: MINIMUM_IP_TARGET increases IP address consumption.
• Subnet Limits: Still constrained by VPC subnet IP address limits.
• ENI Limits: EC2 instance ENI limits still apply.
• Tuning Required: Optimal values vary by workload and cluster size.

Looking Ahead

VPC CNI 1.6 set the foundation for:

• Pod Security Groups: Enhanced security group support for pods.
• Automatic Subnet Discovery: Dynamic subnet discovery and IP allocation.
• Performance Improvements: Continued optimization of IP allocation and pod startup.
• Advanced Features: Additional AWS networking feature integrations.

Summary

AWS VPC CNI 1.6 addressed pod startup performance by introducing MINIMUM_IP_TARGET to guarantee IP availability. This improvement reduced pod start latency, making VPC CNI more suitable for autoscaling, rapid deployments, and latency-sensitive workloads.