Node & Kubelet

Worker nodes are where your applications actually run in Kubernetes. Understanding nodes and the kubelet (the primary agent on each node) is essential because they’re responsible for executing the decisions made by the control plane and reporting back the status of your workloads.

Think of a node like a worker in a factory. The control plane (management) decides what needs to be done, and the node (worker) actually does it. The kubelet is like the foreman on the factory floor—it receives instructions, coordinates the work, and reports back on progress.

What is a Node?

A node is a worker machine in Kubernetes. It can be a physical machine or a virtual machine. Each node runs:

• kubelet - Agent that communicates with the control plane
• kube-proxy - Network proxy for service networking
• Container runtime - Software that runs containers (containerd, CRI-O, Docker)

Nodes are where your pods (and thus your containers) actually run. The control plane schedules pods onto nodes, and nodes execute them.

Node Types

Kubernetes distinguishes between:

• Control plane nodes - Run control plane components (API server, scheduler, etc.)
• Worker nodes - Run your application workloads

In many setups, especially for learning or small deployments, the same machines can run both control plane and worker components. For production, they’re typically separated.

Node Components

Kubelet

The kubelet is the primary node agent. It’s responsible for:

• Registering the node with the API server
• Watching for pod assignments
• Starting and stopping containers
• Reporting node and pod status
• Running health checks (liveness and readiness probes)

kube-proxy

kube-proxy maintains network rules on nodes. It’s responsible for:

• Implementing the Service abstraction
• Load balancing traffic to pods
• Maintaining iptables or IPVS rules
• Enabling service discovery

Container Runtime

The container runtime is the software responsible for running containers. Kubernetes supports any runtime that implements the Container Runtime Interface (CRI), such as:

• containerd - Industry-standard container runtime
• CRI-O - Lightweight, OCI-compliant runtime
• Docker - Uses containerd under the hood (deprecated in favor of containerd)

Kubelet Deep Dive

The kubelet is the most important component on a worker node. It’s the bridge between the control plane and the containers running on the node.

Kubelet Responsibilities

Node Registration

When a node starts, the kubelet registers itself with the API server:

• Reports node information (CPU, memory, OS, kernel version)
• Updates node status periodically
• Handles node heartbeat (node lease)

Pod Management

The kubelet watches for pods assigned to its node:

• Receives pod specifications from the API server
• Creates pod sandbox (network namespace)
• Starts containers via container runtime
• Monitors container health
• Restarts containers if they fail
• Cleans up when pods are deleted

Status Reporting

The kubelet continuously reports status:

• Pod status (running, pending, failed)
• Container status (running, waiting, terminated)
• Resource usage (CPU, memory)
• Node conditions (ready, memory pressure, disk pressure)

Health Checks

The kubelet runs probes defined in pod specifications:

• Liveness probe - Determines if container is alive (restarts if fails)
• Readiness probe - Determines if container is ready to serve traffic
• Startup probe - Determines if container has started (for slow-starting apps)

How Kubelet Works

1. Watch for assignments - Kubelet watches API server for pods assigned to its node
2. Create pod sandbox - Sets up network namespace for the pod
3. Start containers - Instructs container runtime to start each container
4. Monitor health - Runs probes and monitors container status
5. Report status - Continuously updates API server with pod status
6. Handle failures - Restarts containers or reports failures as needed

Kubelet Configuration

The kubelet can be configured via:

• Command-line flags - Traditional configuration method
• Config file - YAML configuration file (preferred)
• Dynamic Kubelet Config - Runtime configuration updates (deprecated)

Key configuration options:

• Node IP - IP address the kubelet uses
• Pod CIDR - IP range for pods on this node
• Container runtime endpoint - CRI socket path
• Image pull secrets - Credentials for private registries
• Resource limits - CPU and memory limits for pods

Container Runtime Interface (CRI)

The Container Runtime Interface (CRI) is a plugin interface that enables kubelet to use different container runtimes without recompiling. This abstraction allows Kubernetes to support multiple runtimes.

CRI Operations

The kubelet communicates with the container runtime through CRI:

• Image operations - Pull, list, remove images
• Container operations - Create, start, stop, remove containers
• Pod operations - Create, start, stop pod sandboxes
• Status - Get container and pod status

Supported Runtimes

containerd - Most common production runtime:

• Industry standard
• Part of CNCF
• Used by Docker, Kubernetes, and others
• Lightweight and performant

CRI-O - Lightweight alternative:

• Designed specifically for Kubernetes
• OCI-compliant
• Minimal overhead
• Good for resource-constrained environments

Node Registration and Communication

Node Registration

When a node starts:

1. Kubelet starts and reads configuration
2. Kubelet creates Node object in API server (or updates existing)
3. Kubelet reports node information:
   • Capacity (CPU, memory, storage)
   • Allocatable resources
   • Conditions (ready, memory pressure, etc.)
   • Labels and annotations

Node Heartbeat

The kubelet periodically updates the node status:

• Node status - Updates every 10 seconds (configurable)
• Node lease - Lightweight heartbeat every 40 seconds
• Pod status - Updates as pod state changes

If the kubelet stops reporting, the control plane marks the node as NotReady and reschedules pods.

Node Conditions

Nodes report various conditions:

• Ready - Node is healthy and ready to accept pods
• MemoryPressure - Node has memory pressure
• DiskPressure - Node has disk pressure
• PIDPressure - Node has too many processes
• NetworkUnavailable - Node network is not configured correctly

Node Lifecycle

Node States

• Ready - Node is healthy and can accept pods
• NotReady - Node is not healthy (kubelet not reporting, issues detected)
• Unknown - Node status can’t be determined (communication issues)

Node Failure Handling

When a node fails or becomes unreachable:

1. Control plane detects - Node stops sending heartbeats
2. Node marked NotReady - After timeout period
3. Pods marked for deletion - Pods on failed node marked as terminating
4. Pods rescheduled - Control plane creates new pods on healthy nodes
5. Node removed - After grace period, node object may be deleted

kube-proxy

kube-proxy is a network proxy that runs on each node and maintains network rules that enable Service abstraction.

kube-proxy Modes

iptables mode (default) - Uses iptables rules:

• Fast and efficient
• No userspace proxying
• Kernel-level forwarding

IPVS mode - Uses IPVS (IP Virtual Server):

• Better performance for large clusters
• More load balancing algorithms
• Still kernel-level

userspace mode (deprecated) - Userspace proxying:

• Slower but more flexible
• Rarely used in production

How kube-proxy Works

When a Service is created:

1. API server stores Service - Service definition stored
2. kube-proxy watches - Detects new/updated Service
3. Endpoints discovered - Finds pods matching Service selector
4. Rules created - Creates iptables/IPVS rules
5. Traffic routed - Traffic to Service IP routed to pod IPs

Node Resources

Resource Capacity

Each node reports its resource capacity:

• CPU - Number of CPU cores
• Memory - Total memory
• Storage - Available storage
• Pods - Maximum number of pods (typically 110)

Resource Allocation

The scheduler uses node resources to make scheduling decisions:

• Requests - Guaranteed resources for pods
• Limits - Maximum resources pods can use
• Allocatable - Resources available for pods (capacity minus system reserves)

Resource Pressure

Nodes can experience resource pressure:

• Memory pressure - Low available memory
• Disk pressure - Low available disk space
• PID pressure - Too many processes

When pressure is detected, the node may evict pods to free resources.

Node Management

Adding Nodes

To add a node to a cluster:

1. Install kubelet and kube-proxy - Install Kubernetes components
2. Configure kubelet - Point to API server, set credentials
3. Join cluster - Use kubeadm join or manual configuration
4. Node appears - Node object created in API server
5. Pods scheduled - Scheduler can now assign pods to new node

Removing Nodes

To remove a node:

1. Drain node - Evict all pods from node
2. Cordon node - Mark node as unschedulable
3. Delete node - Remove node object from API server

Node Maintenance

For node maintenance:

1. Cordon node - Prevent new pods from being scheduled
2. Drain node - Evict existing pods gracefully
3. Perform maintenance - Update, reboot, etc.
4. Uncordon node - Allow pods to be scheduled again

Key Takeaways

• Nodes are worker machines that run your applications
• Kubelet is the primary agent on each node, managing pods and reporting status
• Container runtime (containerd/CRI-O) actually runs containers
• kube-proxy handles Service networking and load balancing
• Nodes register with the API server and send periodic heartbeats
• Node failures trigger pod rescheduling to healthy nodes
• CRI allows Kubernetes to support different container runtimes

See Also

• Kubernetes Architecture - Overall architecture
• Control Plane Components - Control plane details
• etcd Basics - State storage
• Pods 101 - What runs on nodes
• Services - How kube-proxy enables Services