Kubernetes 1.29: Mandala — Extending Performance and Security

Introduction

On December 13, 2023, the Kubernetes project announced version 1.29, codenamed “Mandala.”
This release built on Kubernetes’ foundation of stability and scalability, introducing key improvements in security, node management, and job scheduling.
It included 49 enhancements — 18 graduated to stable (GA), 18 to beta, and 13 newly introduced as alpha.

Official Highlights

1. Job Lifecycle Management Enhancements

Kubernetes 1.29 introduced major updates to Job and CronJob behavior, improving backoff policies, error handling, and retry logic for more reliable and predictable batch processing across large clusters.

Benefits:

• Better error handling: Improved retry logic with configurable backoff policies
• Reliability: More predictable job execution and completion
• Control: Enhanced suspend/resume capabilities for long-running workloads
• Indexed jobs: Better support for parallel batch processing with index-based tasks

Indexed Jobs Improvements: Indexed Jobs enable parallel execution of batch workloads with unique indices per task, perfect for processing large datasets:

apiVersion: batch/v1
kind: Job
metadata:
  name: indexed-job
spec:
  completions: 5
  parallelism: 3
  completionMode: Indexed
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: worker
        image: batch-worker:latest
        command:
        - /bin/sh
        - -c
        - process-item.sh $JOB_COMPLETION_INDEX
        env:
        - name: JOB_COMPLETION_INDEX
          valueFrom:
            fieldRef:
              fieldPath: metadata.labels['batch.kubernetes.io/job-completion-index']

Suspend/Resume APIs: Control job execution with new suspend/resume capabilities:

# Suspend a running job
kubectl patch job my-job -p '{"spec":{"suspend":true}}'

# Resume a suspended job
kubectl patch job my-job -p '{"spec":{"suspend":false}}'

Backoff Policy Improvements:

apiVersion: batch/v1
kind: Job
metadata:
  name: job-with-backoff
spec:
  backoffLimit: 5
  backoffLimitPerIndex: 3
  maxFailedIndexes: 2
  template:
    spec:
      restartPolicy: OnFailure
      containers:
      - name: task
        image: task-runner:latest

Verification:

# Check job status
kubectl get jobs

# View job events
kubectl describe job indexed-job

# Check job completion
kubectl get pods -l job-name=indexed-job

“Kubernetes 1.29 delivers deeper control and smarter automation for complex workloads.”
— Kubernetes 1.29 Release Team

2. Node & Scheduling Improvements

NodeLogQuery API (Alpha) - Kubernetes 1.29 introduces native support for retrieving node-level logs via the API server, eliminating the need for SSH access to nodes for log inspection.

Benefits:

• Centralized access: Retrieve logs through Kubernetes API without node access
• Security: No SSH required for log access, better security posture
• Observability: Better integration with monitoring and logging tools
• Auditability: API-based access enables better audit trails

Requirements:

• Feature gate NodeLogQuery must be enabled
• Kubelet must be configured with log access enabled
• Proper RBAC permissions for log access

Example:

# Query node logs via API
kubectl get --raw "/api/v1/nodes/node-name/proxy/logs/?query=kubelet" | jq .

# Query specific container logs
kubectl get --raw "/api/v1/nodes/node-name/proxy/logs/?query=kubelet&container=kubelet" | jq .

RBAC Configuration:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: node-log-reader
rules:
- apiGroups: [""]
  resources: ["nodes/proxy/logs"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: node-log-reader-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: node-log-reader
subjects:
- kind: User
  name: admin
  apiGroup: rbac.authorization.k8s.io

Verification:

# Enable feature gate (requires cluster configuration)
# Add to kubelet: --feature-gates=NodeLogQuery=true

# Test log query
kubectl get --raw "/api/v1/nodes/$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')/proxy/logs/" | head -20

Additional Improvements:

• Node Lifecycle Controller was improved for better handling of preemptions and taints
• PodTopologySpread and Priority & Fairness scheduling received key performance enhancements for large-scale clusters

3. Security and Policy Hardening

Security took center stage in Kubernetes 1.29 with significant improvements to encryption, admission control, and authentication:

KMS v2 API as Default: The KMS v2 API becomes the default encryption method for secrets at rest, providing enhanced security and performance:

# Encryption configuration (KMS v2 default)
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
  - secrets
  providers:
  - kms:
      apiVersion: v2  # v2 is now default
      name: kms-provider
      endpoint: unix:///tmp/kms-provider.sock

PodSecurity Admission Enhancements: PodSecurity Admission receives new enforcement levels and improved audit logging:

• Enforcement modes: Enforce, Audit, Warn (per namespace)
• Improved audit logging: Better tracking of policy violations
• Namespace labeling: Simplified namespace configuration

# Namespace with PodSecurity
apiVersion: v1
kind: Namespace
metadata:
  name: secure-namespace
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

Ephemeral Containers Stability: Ephemeral containers receive stability improvements for safer debugging operations:

# Create ephemeral container for debugging
kubectl debug pod/my-pod -it --image=busybox --target=my-container

# Debug without modifying pod spec
kubectl debug node/node-name -it --image=busybox

Authn/Authz Improvements: Authentication and authorization receive improvements:

• Simplified token management
• Better role binding updates
• Enhanced service account token handling

Verification:

# Check encryption configuration
kubectl get encryptionconfig

# Verify PodSecurity labels
kubectl get namespaces --show-labels | grep pod-security

# Test ephemeral container
kubectl debug pod/test-pod -it --image=busybox

4. Networking & Storage Updates

IPv4/IPv6 Dual-Stack Production Readiness: Kubernetes 1.29 achieves full production readiness for dual-stack networking across major CNI plugins, enabling seamless IPv4 and IPv6 support.

Benefits:

• Modern networking: Native support for IPv6 alongside IPv4
• IPv6 migration: Smooth transition path to IPv6-only deployments
• Hybrid environments: Support for both IPv4 and IPv6 in the same cluster
• CNI compatibility: Broad support across major CNI plugins (Calico, Cilium, Flannel, etc.)

Configuration Example:

apiVersion: v1
kind: Service
metadata:
  name: dual-stack-service
spec:
  type: ClusterIP
  ipFamilyPolicy: RequireDualStack
  ipFamilies:
  - IPv4
  - IPv6
  selector:
    app: my-app
  ports:
  - port: 80

CSI Migration Completion: All legacy in-tree storage drivers have completed migration to CSI:

• Consistent storage interface across all providers
• Better extensibility and maintainability
• Improved performance and reliability
• Future-proof storage architecture

VolumePopulator API (Beta): The VolumePopulator API improves dynamic storage population for better data management:

apiVersion: populators.storage.k8s.io/v1beta1
kind: VolumePopulator
metadata:
  name: data-populator
spec:
  sourceKind:
    group: example.com
    kind: DataSource
  target:
    path: /data

NetworkPolicy Validation: NetworkPolicy validation has been tightened for better conformance and security:

• Improved validation rules
• Better error messages
• Enhanced security checks
• Consistent behavior across implementations

Verification:

# Check dual-stack service
kubectl get svc dual-stack-service -o yaml

# Verify CSI drivers
kubectl get csidrivers

# Test NetworkPolicy
kubectl get networkpolicies -A

5. Observability and Extensibility

• Structured Logging continued to progress toward full adoption.
• Metrics Stability Framework now covers all core components.
• kubeadm added enhanced preflight checks and improved version skew handling.
• kubectl debug gained new options for node and container inspection.

Milestones Timeline

Patch Releases for 1.29

Patch releases (1.29.x) focused on API polish, performance tuning, and security patching.

Legacy and Impact

Kubernetes 1.29 continued the project’s focus on secure and scalable orchestration.
With improvements in job scheduling, node observability, and encryption, this release positioned Kubernetes as an even more reliable foundation for enterprise-scale workloads in hybrid and multi-cloud environments.

Getting Started

Upgrade Path

Prerequisites:

• Kubernetes 1.28+ cluster
• Backup etcd and cluster state
• Review deprecation notices

Upgrade Steps:

# For kubeadm clusters
kubeadm upgrade plan
kubeadm upgrade apply v1.29.0

# Verify upgrade
kubectl get nodes
kubectl version

# Check deprecated APIs
kubectl get --raw /api/v1 | grep -i deprecated

Feature Gates:

# Enable alpha features (if needed)
--feature-gates=NodeLogQuery=true,VolumePopulator=true

# Check current feature gates
kubectl get --raw /metrics | grep feature_gate

Migration Guide:

• Review 1.29 CHANGELOG
• Test in non-production environment first
• Update custom controllers and operators
• Review deprecated API usage
• Verify KMS v2 configuration for secret encryption

Compatibility:

• Supported upgrade path: 1.28.x → 1.29.x
• kubectl version: 1.29+ recommended
• Minimum node versions: 1.27+ for 1.29 control plane

Summary

Next in the Series

Next up: Kubernetes 1.30 (April 2024) — the release that refined Sidecar Containers, improved Gateway API, and expanded WASM runtime integration.