Upgrades & Version Skew

Keeping Kubernetes clusters up-to-date is essential for security, new features, and bug fixes. However, upgrading a Kubernetes cluster requires careful planning because different components can run different versions—a concept called “version skew.” Understanding version skew policies and upgrade strategies ensures upgrades happen safely without breaking running applications.

Think of upgrading Kubernetes like renovating a building while people are still working inside. You need to upgrade components in the right order, ensure compatibility between old and new versions, and minimize disruption to ongoing operations.

Why Upgrade?

Regular upgrades provide:

Security Patches - Fix vulnerabilities and security issues
New Features - Access to latest capabilities and improvements
Bug Fixes - Resolve issues and improve stability
Performance Improvements - Better resource utilization and faster operations
Compatibility - Stay compatible with tools, libraries, and managed services

Version Skew Policy

Kubernetes allows components to run different versions simultaneously, but only within specific limits. This version skew policy enables rolling upgrades where you upgrade components one at a time rather than shutting everything down.

The general rule is:

kube-apiserver is the reference version—all components must be compatible with it
kubelet can be up to 2 minor versions older than kube-apiserver
kubectl can be up to 1 minor version older or newer than kube-apiserver
kube-controller-manager, kube-scheduler should match kube-apiserver (no skew)
kube-proxy can be up to 2 minor versions older than kube-apiserver

graph TB A[kube-apiserver v1.28.0] --> B[Reference Version] B --> C[kubelet v1.26.0 to v1.28.0 ±2 minor versions] B --> D[kubectl v1.27.0 to v1.29.0 ±1 minor version] B --> E[controller-manager v1.28.0 Must match] B --> F[scheduler v1.28.0 Must match] B --> G[kube-proxy v1.26.0 to v1.28.0 ±2 minor versions] style A fill:#e1f5ff style B fill:#fff4e1 style C fill:#e8f5e9 style D fill:#e8f5e9 style E fill:#f3e5f5 style F fill:#f3e5f5 style G fill:#e8f5e9

Upgrade Strategies

In-Place Upgrades

Upgrade the existing cluster by updating components in place. This is the most common approach:

Control Plane First:

Upgrade kube-apiserver, controller-manager, and scheduler (usually done together)
Upgrade etcd if needed (often compatible across multiple versions)
Upgrade worker nodes one at a time or in small batches

Worker Nodes:

Drain the node (safely evict pods)
Upgrade kubelet and kube-proxy
Upgrade container runtime if needed
Uncordon the node (mark it schedulable again)

graph TD A[Start Upgrade] --> B[Upgrade Control Plane] B --> C[Upgrade etcd if needed] C --> D[Upgrade Worker Node 1] D --> E[Drain Node] E --> F[Upgrade kubelet] F --> G[Upgrade kube-proxy] G --> H[Upgrade Container Runtime] H --> I[Uncordon Node] I --> J{More Nodes?} J -->|Yes| D J -->|No| K[Upgrade Complete] style A fill:#e1f5ff style B fill:#fff4e1 style D fill:#f3e5f5 style K fill:#e8f5e9

Migration Upgrades

For major version upgrades or when in-place upgrades aren’t possible, create a new cluster and migrate workloads:

Create new cluster with target Kubernetes version
Migrate workloads and data
Switch traffic to new cluster
Decommission old cluster

This approach requires more planning and coordination but provides a clean upgrade path.

Upgrade Process with kubeadm

When using kubeadm, upgrades follow a structured process:

Control Plane Upgrade

Upgrade kubeadm on the control plane node
Upgrade control plane with kubeadm upgrade apply
Manually upgrade kubelet and restart it
Verify cluster is healthy

Worker Node Upgrade

Upgrade kubeadm on the worker node
Drain the node to evict pods safely
Upgrade kubelet configuration with kubeadm upgrade node
Upgrade kubelet and restart it
Uncordon the node to make it schedulable again
Verify the node is Ready

Version Compatibility

Before upgrading, verify compatibility:

API Version Compatibility - Check if deprecated APIs are removed in the target version
Feature Gates - Understand which features are enabled/disabled in the new version
Add-on Compatibility - Ensure cluster add-ons (CNI, CSI, monitoring) support the target version
Application Compatibility - Verify applications work with the new Kubernetes version

Best Practices

Read Release Notes - Understand breaking changes and new features
Test in Non-Production - Upgrade a test cluster first
Backup etcd - Always backup cluster state before upgrading
Upgrade in Minor Steps - Don’t skip minor versions (upgrade 1.26 → 1.27 → 1.28, not 1.26 → 1.28)
Monitor During Upgrade - Watch cluster health and application metrics
Have a Rollback Plan - Know how to revert if something goes wrong
Coordinate with Teams - Communicate upgrade schedules and potential impacts
Upgrade During Low Traffic - Schedule upgrades during maintenance windows when possible

Common Upgrade Scenarios

Minor Version Upgrade (1.27 → 1.28)

Typically straightforward with minimal breaking changes. Follow standard in-place upgrade process.

Patch Version Upgrade (1.28.0 → 1.28.5)

Usually safe and recommended for security patches. Can often be done with minimal planning.

Major Version Upgrade (v1 → v2)

Would require significant planning, testing, and potentially migration. Kubernetes hasn’t released v2 yet, but this would be a major undertaking when it happens.

Deprecated API Removal

When Kubernetes removes deprecated APIs, you must update all resources using those APIs before upgrading. Tools like pluto can scan for deprecated API usage.

Upgrade Tools

kubeadm upgrade - Built-in upgrade commands for kubeadm clusters
kubectl convert - Convert resources between API versions
API deprecation scanners - Tools to find deprecated API usage
Managed service upgrade UIs - Cloud providers offer upgrade interfaces for managed clusters

Topics

Upgrades & Version Skew - This page covers upgrade strategies and version management