Kubernetes 1.31: Ethereal Elephants — Performance, Observability, and Fine-Grained Control

Introduction

On August 14, 2024, the Kubernetes project released version 1.31, codenamed “Ethereal Elephants.”
This release focused on performance, observability, and fine-grained configuration control.
It contained 45 enhancements — 14 moved to stable (GA), 17 to beta, and 14 introduced as alpha.

Official Highlights

1. Fine-Grained Admission Control (Alpha)

Dynamic Admission Control Policies debuted as alpha, providing more flexible, context-aware validation and mutation capabilities.
Cluster administrators can now define rule sets that apply to specific namespaces, resources, or conditions, enabling tighter compliance and governance.

“Kubernetes 1.31 introduces precision to cluster policy enforcement — it’s no longer one-size-fits-all.”
— Kubernetes SIG Auth Team

Benefits:

• Context-aware: Policies can consider namespace, resource type, user, and other context
• Flexible rules: More flexible than traditional admission webhooks with better performance
• Better performance: Optimized for high-throughput scenarios with caching
• Compliance: Better support for compliance and governance requirements
• Multi-tenancy: Different policies for different tenants or namespaces

How it works:

1. Define admission policies using new policy resources (ValidatingAdmissionPolicy, MutatingAdmissionPolicy)
2. Policies can target specific namespaces, resources, or conditions using match expressions
3. Policies support both validation and mutation with CEL (Common Expression Language)
4. Policies are evaluated in order with clear precedence rules

Example:

apiVersion: admissionregistration.k8s.io/v1alpha1
kind: ValidatingAdmissionPolicy
metadata:
  name: "pod-resource-limits"
spec:
  matchConstraints:
    resourceRules:
    - apiGroups: [""]
      apiVersions: ["v1"]
      operations: ["CREATE", "UPDATE"]
      resources: ["pods"]
  validations:
  - expression: "object.spec.containers.all(c, has(c.resources) && has(c.resources.limits))"
    message: "All containers must have resource limits"
  - expression: "object.spec.containers.all(c, c.resources.limits.memory != null)"
    message: "All containers must have memory limits"
---
apiVersion: admissionregistration.k8s.io/v1alpha1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: "pod-resource-limits-binding"
spec:
  policyName: "pod-resource-limits"
  matchResources:
    namespaceSelector:
      matchLabels:
        environment: production

Use Cases:

• Resource validation: Ensure all pods have resource limits and requests
• Security policies: Enforce security requirements (no privileged containers, required security contexts)
• Compliance: Ensure compliance with organizational policies and standards
• Multi-tenancy: Different policies for different tenants or environments
• Cost control: Enforce resource limits to control costs

2. CRI and Runtime Performance Upgrades

Kubernetes 1.31 introduced multiple CRI performance optimizations, improving container startup times and reducing memory overhead across container runtimes such as containerd and CRI-O.
This improved scheduling responsiveness and scalability under high load.

CRI v1.2 API Enhancements:

• Consistent lifecycle reporting: More consistent reporting of container lifecycle events
• Better error handling: Improved error messages and handling for runtime operations
• Performance metrics: Better metrics for runtime operations and performance
• Resource efficiency: Reduced memory and CPU overhead for CRI operations

RuntimeClass Scheduling Refinements:

• Heterogeneous clusters: Better support for clusters with different runtime classes
• Performance optimization: Optimized scheduling for different runtime types
• Resource allocation: Better resource allocation for different runtimes
• Workload matching: Improved matching of workloads to appropriate runtimes

Performance Improvements:

• Container startup: Up to 30% faster container startup times
• Memory usage: Reduced memory overhead by 15-20% for CRI operations
• Scheduling latency: Reduced scheduling latency for high-throughput scenarios
• Scalability: Better scalability for large clusters with thousands of pods

Example RuntimeClass:

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: gvisor
handler: runsc
overhead:
  podFixed:
    memory: "128Mi"
    cpu: "100m"
scheduling:
  nodeSelector:
    runtime: gvisor
  tolerations:
  - key: runtime
    operator: Equal
    value: gvisor
    effect: NoSchedule
---
apiVersion: v1
kind: Pod
metadata:
  name: secure-pod
spec:
  runtimeClassName: gvisor
  containers:
  - name: app
    image: myapp:latest

Use Cases:

• Security: Use gVisor or Kata Containers for untrusted workloads
• Performance: Use optimized runtimes for performance-critical workloads
• Resource optimization: Different runtimes for different resource requirements
• Multi-tenant: Isolate tenants using different runtime classes

3. Gateway API and Networking Improvements

Following its GA status in 1.30, the Gateway API received significant stability and performance updates:

GRPCRoute (GA)

GRPCRoute now natively supports gRPC traffic management, graduating to GA.

Benefits:

• Native gRPC support: First-class support for gRPC traffic routing
• Advanced routing: Header-based routing for gRPC services and methods
• Load balancing: Intelligent load balancing for gRPC services
• Health checking: Built-in health checking for gRPC services
• Traffic splitting: Support for canary and A/B testing with gRPC

Example:

apiVersion: gateway.networking.k8s.io/v1
kind: GRPCRoute
metadata:
  name: grpc-route
spec:
  parentRefs:
  - name: my-gateway
  rules:
  - matches:
    - method:
        type: Exact
        service: com.example.UserService
        method: GetUser
    backendRefs:
    - name: user-service
      port: 9090
  - matches:
    - method:
        type: Exact
        service: com.example.OrderService
        method: CreateOrder
    backendRefs:
    - name: order-service
      port: 9090
      weight: 80
    - name: order-service-v2
      port: 9090
      weight: 20

Use Cases:

• Microservices: Route gRPC traffic between microservices
• API gateways: Modern API gateway functionality for gRPC services
• Service mesh: Integration with service mesh for gRPC traffic
• Canary deployments: Gradual rollout of gRPC service updates

BackendTLSPolicy (Beta)

BackendTLSPolicy introduced enhanced mTLS configurations for secure service-to-service communication.

Benefits:

• mTLS support: Mutual TLS support for backend services
• Certificate management: Automatic certificate management and rotation
• Security: Enhanced security for service-to-service communication
• Flexibility: Flexible TLS configuration for different backend services

Example:

apiVersion: gateway.networking.k8s.io/v1beta1
kind: BackendTLSPolicy
metadata:
  name: backend-tls-policy
spec:
  targetRef:
    group: ""
    kind: Service
    name: my-service
  tls:
    hostname: my-service.example.com
    caCertRefs:
    - name: ca-cert
      group: ""
      kind: Secret
    clientCertRefs:
    - name: client-cert
      group: ""
      kind: Secret

Use Cases:

• Service-to-service security: Secure communication between services
• Zero-trust networking: Implement zero-trust security models
• Compliance: Meet compliance requirements for encrypted communication
• Multi-tenant: Secure communication in multi-tenant environments

Controller Interoperability

Better controller interoperability and conformance tests for cloud vendors ensure consistent Gateway API implementation across different providers.

4. Observability and Metrics Framework Expansion

Metrics Stability Framework

Metrics Stability Framework completed rollout across all core controllers, providing stable, versioned metrics.

Benefits:

• Stable metrics: Metrics are now versioned and stable, reducing breaking changes
• Better deprecation: Clear deprecation path for metrics with migration guides
• Consistency: Consistent metric naming and structure across components
• Documentation: Better documentation for metrics with examples

Example metrics:

# Stable metrics with versioning
apiserver_request_total{version="v1",resource="pods",verb="create"}
scheduler_scheduling_duration_seconds{version="v1",attempts="1"}
kubelet_pod_start_duration_seconds{version="v1",pod="my-pod"}

Structured Logging

Structured Logging added context tracing for API server and scheduler logs, providing better observability.

Improvements:

• Context tracing: Better traceability of operations across components with trace IDs
• Structured format: JSON-structured logs for better parsing and analysis
• Performance: Reduced logging overhead with optimized structured logging
• Integration: Better integration with logging and monitoring systems (ELK, Splunk, etc.)

Example log output:

{
  "timestamp": "2024-08-14T10:30:00Z",
  "level": "info",
  "msg": "Pod scheduled",
  "pod": "my-pod",
  "namespace": "default",
  "node": "node-1",
  "traceID": "abc123",
  "spanID": "def456",
  "component": "scheduler"
}

Benefits:

• Troubleshooting: Easier troubleshooting with traceable operations
• Performance analysis: Better performance analysis with structured data
• Log aggregation: Easier log aggregation and analysis
• Alerting: Better alerting based on structured log data

Event Filtering API (Alpha)

Event Filtering API allowed users to reduce log noise and improve audit pipeline efficiency.

Benefits:

• Noise reduction: Filter out noisy events to focus on important ones
• Performance: Improved audit pipeline performance by reducing event volume
• Focus: Focus on important events for better visibility
• Customization: Customizable event filtering based on various criteria

Example:

apiVersion: events.k8s.io/v1alpha1
kind: EventFilter
metadata:
  name: important-events
spec:
  rules:
  - level: Warning
  - level: Error
  - type: PodFailed
  - type: NodeNotReady
  - reason: ImagePullBackOff
  - reason: CrashLoopBackOff

Use Cases:

• Audit logging: Focus audit logs on important events
• Monitoring: Reduce monitoring noise for better alerting
• Compliance: Filter events for compliance reporting
• Debugging: Focus on relevant events during troubleshooting

5. Security and Reliability Improvements

PodSecurity Admission

PodSecurity Admission gained improved namespace-wide enforcement templates for better security.

Improvements:

• Namespace templates: Better namespace-wide enforcement templates
• Profile customization: More flexible profile customization options
• Audit mode: Enhanced audit mode for policy violations
• Documentation: Better documentation and examples for policy configuration

Example:

apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted
---
apiVersion: v1
kind: Pod
metadata:
  name: secure-pod
  namespace: production
spec:
  securityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: app
    image: myapp:latest
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL

KMS v2.2 (Beta)

KMS v2.2 enhanced integration with cloud key management services, graduating to Beta.

Features:

• Cloud integration: Enhanced integration with AWS KMS, Azure Key Vault, GCP KMS
• Key rotation: Improved key rotation capabilities with zero-downtime
• Performance: Better performance for secret encryption/decryption
• Multi-provider: Support for multiple KMS providers simultaneously

Example:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
  - secrets
  providers:
  - kms:
      name: aws-kms
      endpoint: unix:///var/run/kms-plugin/socket
      cachesize: 100
      timeout: 3s
      healthz:
        path: /healthz
        timeout: 3s

Seccomp and AppArmor Default Profiles

Seccomp and AppArmor default profiles refined for better least-privilege operation.

Improvements:

• Default profiles: Better default security profiles for pods
• Least privilege: Improved least-privilege operation with refined profiles
• Compatibility: Better compatibility with different workloads
• Documentation: Better documentation for security profile configuration

Ephemeral Containers

Ephemeral Containers stability and usability continued to improve.

Benefits:

• Debugging: Safer debugging of running pods without restarting containers
• Troubleshooting: Better troubleshooting capabilities for production issues
• Security: Improved security for ephemeral containers with better isolation
• Compatibility: Better compatibility with different container runtimes

Milestones Timeline

Patch Releases for 1.31

Patch releases (1.31.x) focused on runtime tuning, API stability, and network policy hardening.

Legacy and Impact

Kubernetes 1.31 represented a technical refinement milestone, emphasizing runtime performance, observability, and control precision.
It advanced the maturity of the Gateway API, streamlined policy enforcement, and enhanced cluster introspection capabilities — solidifying Kubernetes as a robust foundation for multi-cloud and hybrid architectures.

Summary

Next in the Series

Next up: Kubernetes 1.32 (December 2024) — focusing on cluster autoscaling intelligence, audit policy evolution, and deeper integration of WASM workloads.