Solutions
Audit & Optimization Platform Design Automation Team Structure
Documentation Blog
Get Started
Kubernetes Operations 2020

Bootstrapping in Air-Gapped Environments

K8s Guru
6 min read
#air-gapped #offline #bootstrap #security #operations
Bootstrapping in Air-Gapped Environments

Table of Contents

Introduction

By late 2020, bootstrapping Kubernetes clusters in air-gapped environments had become a critical requirement for organizations with strict security policies, regulatory compliance needs, or disconnected network architectures. While tools like kubeadm, k0s, and k3s had improved offline capabilities, successfully deploying Kubernetes without internet access required careful planning and specific tooling.

This mattered because air-gapped deployments represented some of the most challenging Kubernetes use cases: government systems, financial services, industrial control systems, and secure research environments. These deployments couldn’t rely on cloud services, public container registries, or online package repositories—everything had to be self-contained.

Historical note: Air-gapped bootstrapping had been possible with kubeadm since 2018, but 2020 saw significant improvements in tooling (k0s airgap bundles, kubeadm offline mode) and documentation, making it more accessible to teams.

Air-Gapped Bootstrapping Challenges

Image Distribution

  • Container Images: All container images (Kubernetes components, CNI plugins, add-ons) must be pre-downloaded and transferred.
  • Binary Distribution: Kubernetes binaries (kubeadm, kubelet, kubectl) must be available offline.
  • Dependency Management: Operating system packages, container runtimes, and tools must be pre-installed.

Network Isolation

  • No Internet Access: Clusters cannot reach public container registries or package repositories.
  • Private Registries: Must set up and populate private container registries.
  • Internal DNS: DNS resolution must work within the air-gapped network.

Security Considerations

  • Image Verification: Container images must be verified and signed before transfer.
  • Binary Integrity: Kubernetes binaries must be verified against checksums.
  • Certificate Management: TLS certificates must be generated and distributed securely.

kubeadm Offline Mode

Pre-Downloading Images

# On internet-connected machine
# Download all required images
kubeadm config images pull

# Save images to tar file
docker save $(kubeadm config images list) -o k8s-images.tar

# Transfer to air-gapped environment
scp k8s-images.tar air-gapped-host:/tmp/

Loading Images

# On air-gapped host
# Load images into local registry
docker load -i /tmp/k8s-images.tar

# Tag and push to private registry
for image in $(docker images --format "{{.Repository}}:{{.Tag}}" | grep k8s.gcr.io); do
  docker tag $image private-registry.local:5000/$image
  docker push private-registry.local:5000/$image
done

kubeadm Init with Private Registry

# Configure kubeadm to use private registry
kubeadm init \
  --image-repository private-registry.local:5000 \
  --pod-network-cidr=10.244.0.0/16 \
  --apiserver-advertise-address=192.168.1.10

k0s Air-Gap Bundles

k0s provides built-in air-gap support:

# On internet-connected machine
# Create air-gap bundle
k0s airgap bundle

# Transfer bundle
scp k0s-airgap-bundle.tar air-gapped-host:/tmp/

# On air-gapped host
# Install using air-gap bundle
sudo k0s install controller --airgap-file /tmp/k0s-airgap-bundle.tar
sudo k0s start

The air-gap bundle includes:

  • k0s binary
  • All container images
  • Dependencies

k3s Air-Gapped Installation

k3s supports offline installation:

# On internet-connected machine
# Download k3s binary and images
curl -sfL https://get.k3s.io -o install.sh
INSTALL_K3S_SKIP_DOWNLOAD=true ./install.sh

# Export images
k3s ctr images export k3s-images.tar $(k3s ctr images list -q)

# Transfer to air-gapped environment
scp k3s-images.tar install.sh air-gapped-host:/tmp/

# On air-gapped host
# Install k3s offline
INSTALL_K3S_SKIP_DOWNLOAD=true INSTALL_K3S_EXEC="--system-default-registry private-registry.local:5000" ./install.sh

# Load images into private registry
docker load -i /tmp/k3s-images.tar
# Tag and push to private registry

Private Container Registry Setup

Harbor Registry

Harbor is a popular choice for air-gapped environments:

# Install Harbor
helm repo add harbor https://helm.goharbor.io
helm install harbor harbor/harbor \
  --set expose.type=nodePort \
  --set expose.tls.enabled=false \
  --set persistence.enabled=true

# Configure kubeadm to use Harbor
kubeadm init \
  --image-repository harbor.local:30002 \
  --pod-network-cidr=10.244.0.0/16

Docker Registry

Simple Docker registry for basic use cases:

# Run private registry
docker run -d -p 5000:5000 --name registry registry:2

# Configure kubeadm
kubeadm init \
  --image-repository localhost:5000 \
  --pod-network-cidr=10.244.0.0/16

Image Mirroring Strategies

Manual Mirroring

# Mirror images manually
for image in $(kubeadm config images list); do
  docker pull $image
  docker tag $image private-registry.local:5000/$image
  docker push private-registry.local:5000/$image
done

Automated Mirroring

Use tools like skopeo for automated mirroring:

# Install skopeo
apt-get install -y skopeo

# Mirror images
skopeo copy docker://k8s.gcr.io/kube-apiserver:v1.20.0 \
  docker://private-registry.local:5000/kube-apiserver:v1.20.0

CNI Plugin Installation

CNI plugins must also be available offline:

# Download CNI plugins
wget https://github.com/containernetworking/plugins/releases/download/v0.9.1/cni-plugins-linux-amd64-v0.9.1.tgz

# Transfer to air-gapped host
scp cni-plugins-linux-amd64-v0.9.1.tgz air-gapped-host:/tmp/

# On air-gapped host
# Extract to CNI bin directory
mkdir -p /opt/cni/bin
tar -xzf /tmp/cni-plugins-linux-amd64-v0.9.1.tgz -C /opt/cni/bin

Add-On Installation

Add-ons (CoreDNS, metrics-server, etc.) require offline installation:

# Download add-on manifests
kubectl apply -f https://raw.githubusercontent.com/coredns/deployment/master/kubernetes/coredns.yaml.sed

# For air-gapped, download and modify manifests
wget https://raw.githubusercontent.com/coredns/deployment/master/kubernetes/coredns.yaml.sed
# Edit to use private registry
sed -i 's|k8s.gcr.io|private-registry.local:5000|g' coredns.yaml.sed
kubectl apply -f coredns.yaml.sed

Security Considerations

Image Verification

# Verify image signatures
cosign verify --key cosign.pub k8s.gcr.io/kube-apiserver:v1.20.0

# Verify binary checksums
wget https://dl.k8s.io/v1.20.0/bin/linux/amd64/kubeadm.sha256
sha256sum -c kubeadm.sha256

Certificate Management

# Generate certificates offline
kubeadm init phase certs all --config kubeadm-config.yaml

# Backup certificates
tar -czf k8s-certs.tar.gz /etc/kubernetes/pki

Practical Considerations

Planning Checklist

Before deploying air-gapped clusters:

  1. Inventory Requirements: List all required images, binaries, and dependencies.
  2. Registry Setup: Configure and populate private container registry.
  3. Network Planning: Plan internal DNS and network connectivity.
  4. Image Verification: Verify and sign all container images.
  5. Documentation: Document all manual steps and configurations.

Transfer Methods

Options for transferring files to air-gapped environments:

  • USB Drives: Physical transfer for initial setup.
  • Internal Network: If air-gapped network has internal connectivity.
  • Secure File Transfer: SFTP/SCP over isolated network links.
  • Optical Media: DVDs or Blu-ray for large image sets.

Update Strategies

Updating air-gapped clusters requires:

  1. Download Updates: On internet-connected machine, download new images and binaries.
  2. Verify Updates: Verify signatures and checksums.
  3. Transfer Updates: Move updates to air-gapped environment.
  4. Apply Updates: Update cluster using offline methods.

Caveats & Lessons Learned

  • Image Size: Container images can be large; plan for significant storage and transfer time.
  • Dependency Chains: Some images depend on others; ensure all dependencies are mirrored.
  • Version Compatibility: Ensure all components (kubeadm, kubelet, images) are version-compatible.
  • Registry Availability: Private registry must be highly available; single points of failure cause cluster issues.

Common Failure Modes

  • “Image pull errors”: Private registry not accessible or images not properly mirrored.
  • “DNS resolution failures”: Internal DNS not configured correctly for private registry.
  • “Certificate errors”: TLS certificates not properly configured for private registry.

Conclusion

Bootstrapping Kubernetes in air-gapped environments in 2020 required careful planning, but tools like kubeadm, k0s, and k3s had improved offline capabilities. The key was preparation: downloading all images, setting up private registries, and planning for updates.

For organizations with strict security requirements or disconnected networks, air-gapped Kubernetes deployments were not just possible but increasingly practical. The tooling and documentation improvements in 2020 made it easier for teams to deploy Kubernetes in the most challenging environments, opening up container orchestration to use cases that previously seemed impossible.

As Kubernetes continued to mature, air-gapped deployments would become even more streamlined, with better tooling, documentation, and support for offline operations. The ability to run Kubernetes completely disconnected from the internet would remain a critical capability for many organizations.