kubespray Evolution: Ansible-Based Production Clusters

Introduction

By late 2021, kubespray had evolved into a mature, production-ready tool for deploying Kubernetes clusters using Ansible. Unlike kubeadm’s opinionated approach or managed services’ cloud-specific focus, kubespray provided maximum flexibility for teams deploying on bare metal, heterogeneous cloud environments, or complex on-premises infrastructure.

This mattered because kubespray filled a critical gap: teams needing deep customization and control over their Kubernetes deployments. While kubeadm worked well for standard setups and managed services handled cloud deployments, kubespray enabled teams to customize networking, storage, add-ons, and cluster configurations to match their specific requirements.

Historical note: kubespray (originally kargo) had been evolving since 2016, but by 2021 it had matured into a stable, well-maintained project with broad community support and production deployments across industries.

kubespray Highlights

Ansible-Based Architecture

• Playbook-Driven: Uses Ansible playbooks for cluster provisioning and management.
• Inventory-Based: Defines clusters using Ansible inventory files.
• Idempotent: Ansible’s idempotent nature ensures safe re-runs and updates.
• Extensible: Easy to customize playbooks and roles for specific requirements.

Infrastructure Flexibility

• Multi-Cloud: Supports AWS, Azure, GCP, OpenStack, and more.
• Bare Metal: Excellent support for on-premises and bare metal deployments.
• Heterogeneous: Can mix different machine types and configurations in a single cluster.
• Network Options: Supports multiple CNI plugins (Calico, Flannel, Cilium, Weave, etc.).

Production Features

• High Availability: Full HA support with multiple control plane nodes.
• Upgrade Workflows: Ansible playbooks for in-place upgrades.
• Add-On Management: Integrated support for common add-ons (DNS, monitoring, ingress).
• Security Hardening: Options for CIS Kubernetes Benchmark compliance.

kubespray vs kubeadm vs Cluster API

Architecture Patterns

Inventory Structure

kubespray uses Ansible inventory files to define clusters:

# inventory/my-cluster/hosts.yaml
all:
  hosts:
    node1:
      ansible_host: 192.168.1.10
      ip: 192.168.1.10
    node2:
      ansible_host: 192.168.1.11
      ip: 192.168.1.11
    node3:
      ansible_host: 192.168.1.12
      ip: 192.168.1.12
  children:
    kube_control_plane:
      hosts:
        node1:
        node2:
        node3:
    kube_node:
      hosts:
        node1:
        node2:
        node3:
    etcd:
      hosts:
        node1:
        node2:
        node3:

Cluster Configuration

# inventory/my-cluster/group_vars/all/all.yml
kube_version: v1.22.0
container_manager: containerd
kube_network_plugin: calico
calico_ipip_mode: "CrossSubnet"
calico_vxlan_mode: "Never"

Getting Started with kubespray

1. Clone kubespray

git clone https://github.com/kubernetes-sigs/kubespray.git
cd kubespray

2. Create Inventory

# Copy sample inventory
cp -rfp inventory/sample inventory/my-cluster

# Edit inventory
vi inventory/my-cluster/hosts.yaml

3. Configure Cluster

# Edit cluster configuration
vi inventory/my-cluster/group_vars/all/all.yml
vi inventory/my-cluster/group_vars/k8s_cluster/k8s-cluster.yml

4. Deploy Cluster

# Install dependencies
pip3 install -r requirements.txt

# Deploy cluster
ansible-playbook -i inventory/my-cluster/hosts.yaml cluster.yml -b -v

Advanced Configurations

Multi-Master HA Setup

# inventory/my-cluster/hosts.yaml
all:
  children:
    kube_control_plane:
      hosts:
        master1:
        master2:
        master3:
    kube_node:
      hosts:
        worker1:
        worker2:
        worker3:
    etcd:
      hosts:
        etcd1:
        etcd2:
        etcd3:

Custom CNI Configuration

# inventory/my-cluster/group_vars/all/all.yml
kube_network_plugin: cilium
cilium_enable_hubble: true
cilium_hubble_relay_enabled: true
cilium_hubble_ui_enabled: true

Bare Metal Deployment

# inventory/my-cluster/group_vars/all/all.yml
kube_network_plugin: calico
calico_ipip_mode: "Never"
calico_vxlan_mode: "Always"
loadbalancer_apiserver_type: "none"
loadbalancer_apiserver_address: "192.168.1.100"

Upgrade Workflows

kubespray provides Ansible playbooks for upgrades:

# Upgrade cluster
ansible-playbook -i inventory/my-cluster/hosts.yaml upgrade-cluster.yml -b -v \
  -e kube_version=v1.23.0

The upgrade playbook handles:

• Control plane component updates
• Node updates
• Add-on compatibility
• Rollback procedures

Add-On Management

kubespray includes support for common add-ons:

# inventory/my-cluster/group_vars/k8s_cluster/addons.yml
dashboard_enabled: true
helm_enabled: true
local_path_provisioner_enabled: true
metrics_server_enabled: true

Practical Considerations

Ansible Requirements

kubespray requires:

• Ansible: Version 2.9+ with Python 3
• Python Dependencies: Various Python packages for cloud providers
• SSH Access: Ansible needs SSH access to all nodes
• Privileged Access: Ansible needs sudo/root access for cluster setup

Inventory Management

• Version Control: Store inventory files in Git for version control.
• Environment Separation: Use separate inventories for dev, staging, production.
• Dynamic Inventories: Use Ansible dynamic inventories for cloud providers.

Network Planning

kubespray requires careful network planning:

• Pod CIDR: Plan pod network CIDR to avoid conflicts.
• Service CIDR: Plan service network CIDR.
• Node Networking: Ensure nodes can communicate with each other.

Comparison: kubespray vs kubeadm

Choose kubespray When:

• Deep Customization: Need extensive customization of cluster components.
• Bare Metal: Deploying on bare metal or complex on-premises infrastructure.
• Ansible Expertise: Team has Ansible expertise and existing Ansible workflows.
• Heterogeneous Infrastructure: Mixing different machine types and configurations.

Choose kubeadm When:

• Standard Setup: Standard Kubernetes setup is sufficient.
• Simplicity: Want simpler, more opinionated tooling.
• Learning: Learning Kubernetes internals with minimal abstraction.
• CI/CD Integration: Need lightweight tooling for CI/CD pipelines.

Caveats & Lessons Learned

• Ansible Learning Curve: kubespray requires Ansible knowledge; team must understand Ansible concepts.
• Inventory Management: Inventory files can become complex; use version control and documentation.
• Upgrade Complexity: Upgrades require careful planning and testing; not as automated as managed services.
• Resource Requirements: Ansible control node needs resources to run playbooks.

Common Failure Modes

• “Ansible connection failures”: SSH access issues prevent cluster deployment; verify SSH connectivity.
• “Inventory errors”: Incorrect inventory configuration causes deployment failures; validate inventory files.
• “Network conflicts”: CIDR conflicts cause networking issues; plan network ranges carefully.

Conclusion

kubespray’s evolution by late 2021 had established it as the go-to tool for complex, customized Kubernetes deployments. Its Ansible-based architecture, infrastructure flexibility, and production features made it ideal for teams needing deep control over their cluster configurations.

While kubeadm remained popular for standard setups and managed services handled cloud deployments, kubespray filled a critical niche: bare metal, heterogeneous infrastructure, and highly customized deployments. Its maturity and community support made it a reliable choice for production deployments across industries.

For teams with Ansible expertise or complex infrastructure requirements, kubespray provided the flexibility and control needed to deploy Kubernetes exactly as required. It demonstrated that Kubernetes deployment tools didn’t have to be opinionated—they could be flexible, extensible, and adaptable to any environment.