kubeadm Air-Gapped Bootstrapping: Official Guide

Introduction

In October 2023, the Kubernetes project published an official guide for bootstrapping air-gapped clusters with kubeadm, providing step-by-step instructions for deploying Kubernetes in environments without direct internet access. This guide addressed a critical gap: while air-gapped bootstrapping had been possible for years, the official documentation made it accessible to teams with strict security requirements or disconnected network architectures.

This mattered because air-gapped deployments represented some of the most challenging Kubernetes use cases: government systems, financial services, industrial control systems, and secure research environments. The official guide provided a standardized, tested approach that teams could follow with confidence, reducing the risk of deployment failures and security vulnerabilities.

Historical note: While kubeadm had supported offline deployments since 2018, the October 2023 guide represented the first comprehensive, officially maintained documentation for air-gapped bootstrapping, making it a reference implementation for the community.

Official Air-Gapped Workflow

Phase 1: Preparation (Internet-Connected Environment)

Download Kubernetes Images

# Download all required container images
kubeadm config images pull

# List downloaded images
kubeadm config images list

# Save images to tar archive
docker save $(kubeadm config images list) -o k8s-images.tar

Download Kubernetes Binaries

# Download kubeadm, kubelet, kubectl
K8S_VERSION=v1.28.0
ARCH=amd64

curl -LO "https://dl.k8s.io/release/${K8S_VERSION}/bin/linux/${ARCH}/kubeadm"
curl -LO "https://dl.k8s.io/release/${K8S_VERSION}/bin/linux/${ARCH}/kubelet"
curl -LO "https://dl.k8s.io/release/${K8S_VERSION}/bin/linux/${ARCH}/kubectl"

# Download checksums
curl -LO "https://dl.k8s.io/release/${K8S_VERSION}/bin/linux/${ARCH}/kubeadm.sha256"
curl -LO "https://dl.k8s.io/release/${K8S_VERSION}/bin/linux/${ARCH}/kubelet.sha256"
curl -LO "https://dl.k8s.io/release/${K8S_VERSION}/bin/linux/${ARCH}/kubectl.sha256"

# Verify checksums
sha256sum -c kubeadm.sha256
sha256sum -c kubelet.sha256
sha256sum -c kubectl.sha256

Download CNI Plugins

# Download CNI plugins
CNI_VERSION=v1.3.0
curl -LO "https://github.com/containernetworking/plugins/releases/download/${CNI_VERSION}/cni-plugins-linux-${ARCH}-${CNI_VERSION}.tgz"

Create Offline Bundle

# Create directory structure
mkdir -p k8s-offline-bundle/{images,binaries,cni,scripts}

# Move files
mv k8s-images.tar k8s-offline-bundle/images/
mv kubeadm kubelet kubectl k8s-offline-bundle/binaries/
mv cni-plugins-linux-${ARCH}-${CNI_VERSION}.tgz k8s-offline-bundle/cni/

# Create transfer script
cat > k8s-offline-bundle/transfer.sh << 'EOF'
#!/bin/bash
# Script to transfer bundle to air-gapped environment
# Usage: ./transfer.sh <air-gapped-host>
scp -r k8s-offline-bundle/ $1:/tmp/
EOF
chmod +x k8s-offline-bundle/transfer.sh

Phase 2: Transfer to Air-Gapped Environment

# Transfer bundle using secure method
# Option 1: USB drive
cp -r k8s-offline-bundle/ /media/usb-drive/

# Option 2: Secure file transfer (if isolated network link exists)
./k8s-offline-bundle/transfer.sh air-gapped-host

# Option 3: Optical media
tar -czf k8s-offline-bundle.tar.gz k8s-offline-bundle/
# Burn to DVD/Blu-ray

Phase 3: Installation (Air-Gapped Environment)

Set Up Private Container Registry

# Option 1: Harbor (recommended for production)
# Install Harbor on air-gapped host
# See Harbor documentation for offline installation

# Option 2: Simple Docker registry
docker run -d -p 5000:5000 --name registry \
  -v /var/lib/registry:/var/lib/registry \
  registry:2

Load Images into Private Registry

# Load images from tar archive
docker load -i /tmp/k8s-offline-bundle/images/k8s-images.tar

# Tag and push to private registry
PRIVATE_REGISTRY=private-registry.local:5000
for image in $(docker images --format "{{.Repository}}:{{.Tag}}" | grep k8s.gcr.io); do
  docker tag $image ${PRIVATE_REGISTRY}/${image#k8s.gcr.io/}
  docker push ${PRIVATE_REGISTRY}/${image#k8s.gcr.io/}
done

Install Kubernetes Binaries

# Install binaries
sudo cp /tmp/k8s-offline-bundle/binaries/kubeadm /usr/local/bin/
sudo cp /tmp/k8s-offline-bundle/binaries/kubelet /usr/local/bin/
sudo cp /tmp/k8s-offline-bundle/binaries/kubectl /usr/local/bin/

# Make executable
sudo chmod +x /usr/local/bin/kubeadm
sudo chmod +x /usr/local/bin/kubelet
sudo chmod +x /usr/local/bin/kubectl

Install CNI Plugins

# Extract CNI plugins
sudo mkdir -p /opt/cni/bin
sudo tar -xzf /tmp/k8s-offline-bundle/cni/cni-plugins-linux-amd64-v1.3.0.tgz -C /opt/cni/bin

Initialize Cluster with Private Registry

# Create kubeadm configuration
cat > kubeadm-config.yaml << EOF
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
kubernetesVersion: v1.28.0
imageRepository: private-registry.local:5000
networking:
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/12
controlPlaneEndpoint: "192.168.1.100:6443"
---
apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: "192.168.1.100"
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///var/run/containerd/containerd.sock
EOF

# Initialize cluster
sudo kubeadm init --config=kubeadm-config.yaml

Security Considerations

Image Verification

# Verify image signatures (if available)
cosign verify --key cosign.pub k8s.gcr.io/kube-apiserver:v1.28.0

# Verify binary checksums
sha256sum -c kubeadm.sha256
sha256sum -c kubelet.sha256
sha256sum -c kubectl.sha256

Certificate Management

# Generate certificates offline
kubeadm init phase certs all --config kubeadm-config.yaml

# Backup certificates
sudo tar -czf k8s-certs.tar.gz /etc/kubernetes/pki

# Store securely (encrypted storage, secure backup location)

Network Security

Private Registry: Use TLS for private registry connections.
Network Isolation: Ensure air-gapped network is properly isolated.
Access Control: Restrict access to air-gapped environment.
Audit Logging: Enable audit logging for compliance.

CNI Plugin Installation

Calico (Offline)

# Download Calico manifests (on internet-connected machine)
curl https://raw.githubusercontent.com/projectcalico/calico/v3.26.0/manifests/calico.yaml -o calico.yaml

# Modify image references
sed -i 's|docker.io/calico|private-registry.local:5000/calico|g' calico.yaml

# Transfer to air-gapped environment
# Apply on air-gapped cluster
kubectl apply -f calico.yaml

Flannel (Offline)

# Download Flannel manifest
curl https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml -o flannel.yaml

# Modify image references
sed -i 's|docker.io/flannel|private-registry.local:5000/flannel|g' flannel.yaml

# Transfer and apply
kubectl apply -f flannel.yaml

Add-On Installation

CoreDNS (Offline)

# Download CoreDNS manifest
kubectl get configmap coredns -n kube-system -o yaml > coredns-configmap.yaml

# Modify image references
sed -i 's|k8s.gcr.io/coredns|private-registry.local:5000/coredns|g' coredns-configmap.yaml

# Apply
kubectl apply -f coredns-configmap.yaml

High Availability Setup

External etcd (Air-Gapped)

# Set up external etcd cluster (3 nodes)
# Download etcd binaries
ETCD_VERSION=v3.5.9
curl -LO "https://github.com/etcd-io/etcd/releases/download/${ETCD_VERSION}/etcd-${ETCD_VERSION}-linux-amd64.tar.gz"

# Transfer to air-gapped environment
# Extract and install on etcd nodes

# Configure kubeadm for external etcd
cat > kubeadm-config-ha.yaml << EOF
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
kubernetesVersion: v1.28.0
imageRepository: private-registry.local:5000
etcd:
  external:
    endpoints:
    - https://etcd-1:2379
    - https://etcd-2:2379
    - https://etcd-3:2379
    caFile: /etc/kubernetes/pki/etcd/ca.crt
    certFile: /etc/kubernetes/pki/etcd/etcd.crt
    keyFile: /etc/kubernetes/pki/etcd/etcd.key
EOF

Update Strategies

Downloading Updates

# On internet-connected machine
# Download new Kubernetes version
K8S_VERSION=v1.29.0
kubeadm config images pull --kubernetes-version ${K8S_VERSION}

# Save images
docker save $(kubeadm config images list --kubernetes-version ${K8S_VERSION}) -o k8s-images-v1.29.0.tar

# Download binaries
curl -LO "https://dl.k8s.io/release/${K8S_VERSION}/bin/linux/amd64/kubeadm"
curl -LO "https://dl.k8s.io/release/${K8S_VERSION}/bin/linux/amd64/kubelet"
curl -LO "https://dl.k8s.io/release/${K8S_VERSION}/bin/linux/amd64/kubectl"

Applying Updates

# On air-gapped cluster
# Load new images
docker load -i k8s-images-v1.29.0.tar

# Tag and push to private registry
# (same process as initial installation)

# Upgrade cluster
sudo kubeadm upgrade plan
sudo kubeadm upgrade apply v1.29.0

Practical Considerations

Planning Checklist

Before deploying air-gapped clusters:

Inventory Requirements: List all required images, binaries, and dependencies.
Registry Setup: Configure and populate private container registry.
Network Planning: Plan internal DNS and network connectivity.
Image Verification: Verify and sign all container images.
Documentation: Document all manual steps and configurations.
Testing: Test the entire process in a non-production environment first.

Transfer Methods

USB Drives: Physical transfer for initial setup (encrypt drives).
Internal Network: If air-gapped network has internal connectivity.
Secure File Transfer: SFTP/SCP over isolated network links.
Optical Media: DVDs or Blu-ray for large image sets.

Troubleshooting

Common issues and solutions:

“Image pull errors”: Verify private registry is accessible and images are properly tagged.
“DNS resolution failures”: Configure internal DNS for private registry hostname.
“Certificate errors”: Ensure TLS certificates are properly configured for private registry.

Conclusion

The official kubeadm air-gapped bootstrapping guide published in October 2023 provided a standardized, tested approach for deploying Kubernetes in disconnected environments. It addressed a critical need for organizations with strict security requirements, making air-gapped deployments more accessible and reliable.

The guide demonstrated that air-gapped Kubernetes deployments were not just possible but practical with proper planning and tooling. It provided teams with a reference implementation they could follow with confidence, reducing deployment risks and security vulnerabilities.

For organizations requiring air-gapped deployments, the official guide represented a significant milestone: Kubernetes was now officially supported in the most challenging deployment scenarios. This opened up container orchestration to use cases that previously seemed impossible, from government systems to industrial control systems to secure research environments.

Table of Contents