Solutions
Audit & Optimization Platform Design Automation Team Structure
Documentation Blog
Get Started

kubectl debug

kubectl debug is a powerful debugging feature that allows you to create ephemeral containers in running pods or copy pods for debugging. It’s especially useful for debugging distroless containers or minimal images that don’t have debugging tools installed.

What is kubectl debug?

kubectl debug enables debugging by:

  • Ephemeral containers - Add temporary containers to running pods
  • Pod copying - Create a copy of a pod with debugging tools
  • Node debugging - Debug nodes with special debugging containers
  • Troubleshooting - Investigate issues without modifying original containers
graph TB A[kubectl debug] --> B[Running Pod] A --> C[Copy Pod] A --> D[Node] B --> E[Ephemeral Container] E --> F[Debug Tools] C --> G[New Pod Copy] G --> H[Original Containers] G --> I[Debug Container] D --> J[Node Debug Container] J --> K[Node Access] style A fill:#e1f5ff style E fill:#e8f5e9 style G fill:#fff4e1 style J fill:#f3e5f5

Ephemeral Containers

Ephemeral containers are temporary containers that run alongside existing containers in a pod. They’re useful for debugging without restarting the pod or modifying the original containers.

When to Use Ephemeral Containers

  • Distroless images - Containers without shells or debugging tools
  • Minimal images - Containers with limited tools
  • Running pods - Debug without restarting
  • Investigation - Quick troubleshooting

Prerequisites

Ephemeral containers require:

  • Kubernetes 1.23+ (beta in 1.23, stable in 1.25)
  • Ephemeral containers feature gate enabled (if < 1.25)
  • Container runtime support

Basic Usage

# Debug running pod
kubectl debug <pod-name> -it --image=busybox

# Debug pod in namespace
kubectl debug <pod-name> -n <namespace> -it --image=busybox

# Debug with specific command
kubectl debug <pod-name> -it --image=busybox -- /bin/sh

Example: Debugging a Distroless Container

Original pod:

apiVersion: v1
kind: Pod
metadata:
  name: my-app
spec:
  containers:
  - name: app
    image: gcr.io/distroless/base:latest
    # No shell available

Debug with ephemeral container:

# Add ephemeral container
kubectl debug my-app -it --image=busybox --target=app

# Now you have shell access
# Inspect files, processes, network

Accessing Original Container

Use --target to share process namespace:

# Share process namespace with app container
kubectl debug <pod-name> -it --image=busybox --target=app

# Now you can see processes from app container
ps aux

# Access app container's filesystem
ls /proc/1/root/

Common Debugging Images

# Busybox (lightweight, includes common tools)
kubectl debug <pod-name> -it --image=busybox

# netshoot (network debugging tools)
kubectl debug <pod-name> -it --image=nicolaka/netshoot

# Ubuntu (full OS with all tools)
kubectl debug <pod-name> -it --image=ubuntu

# Alpine (lightweight Linux)
kubectl debug <pod-name> -it --image=alpine

Pod Copy Mode

Copy mode creates a new pod based on an existing pod, allowing you to modify it for debugging.

When to Use Copy Mode

  • Modify configuration - Change container command or args
  • Add debugging tools - Add containers with debugging tools
  • Safe experimentation - Test changes without affecting original
  • Preserve original - Original pod remains unchanged

Basic Usage

# Copy pod
kubectl debug <pod-name> -it --copy-to=debug-pod --image=busybox

# Copy with modifications
kubectl debug <pod-name> -it --copy-to=debug-pod --image=busybox -- sh

Example: Debugging Crash Loop

Original pod:

apiVersion: v1
kind: Pod
metadata:
  name: crashing-app
spec:
  containers:
  - name: app
    image: my-app:latest
    command: ["/app/start.sh"]
    # Pod keeps crashing

Debug by copying:

# Copy pod and change command
kubectl debug crashing-app -it --copy-to=debug-pod \
  --image=my-app:latest \
  --container=app \
  -- sh

# Now debug-pod runs with shell instead of start.sh
# Investigate what's wrong with start.sh

Copying with Shared Volumes

# Copy pod preserving volumes
kubectl debug <pod-name> -it --copy-to=debug-pod \
  --share-processes \
  --image=busybox

Node Debugging

Debug nodes by creating a debugging container that has access to the node’s filesystem and processes.

When to Use Node Debugging

  • Node-level issues - Problems affecting the entire node
  • System investigation - Check system files, logs, processes
  • Kubelet debugging - Investigate kubelet issues
  • Container runtime - Debug container runtime problems

Basic Usage

# Debug node
kubectl debug node/<node-name> -it --image=busybox

# Debug node with host network
kubectl debug node/<node-name> -it --image=busybox --host-network

Example: Debugging Node Issues

# Access node
kubectl debug node/worker-node-1 -it --image=busybox

# Inside debug container:
# Check kubelet logs
cat /host/var/log/kubelet.log

# Check system logs
cat /host/var/log/messages

# Check node processes
chroot /host ps aux

# Check disk usage
chroot /host df -h

# Check network
chroot /host ip addr

Accessing Host Filesystem

The host filesystem is mounted at /host:

# Access host root
cd /host

# Check kubelet logs
cat /host/var/log/kubelet.log

# Check container logs
ls /host/var/log/pods/

# Check system files
chroot /host /bin/sh

Practical Examples

Debugging Application Issues

# Step 1: Add ephemeral container
kubectl debug my-app -it --image=busybox --target=app

# Step 2: Inspect processes
ps aux

# Step 3: Check network connections
netstat -an

# Step 4: Inspect files
ls -la /proc/1/root/app/

# Step 5: Check environment
env

Network Debugging

# Use netshoot image
kubectl debug my-app -it --image=nicolaka/netshoot

# Now you have network tools:
# - curl, wget (HTTP)
# - dig, nslookup (DNS)
# - ping, traceroute (connectivity)
# - tcpdump (packet capture)

# Test connectivity
curl http://other-service:8080

# Check DNS
dig other-service.default.svc.cluster.local

# Capture packets
tcpdump -i any -n

File System Inspection

# Debug pod
kubectl debug my-app -it --image=busybox --target=app

# Access container filesystem
cd /proc/1/root

# Inspect files
ls -la /proc/1/root/app/
cat /proc/1/root/app/config.json

# Check mounted volumes
mount | grep /proc/1/root

Process Investigation

# Debug with process namespace sharing
kubectl debug my-app -it --image=busybox --target=app

# List processes
ps aux

# Check process details
cat /proc/1/status

# Check open files
lsof -p 1

# Check network connections
netstat -anp

Comparison with kubectl exec

kubectl exec

# Requires shell in container
kubectl exec -it <pod-name> -- /bin/sh

# Limitations:
# - Container must have shell
# - Can't add tools
# - Same environment as container

Use when:

  • Container has shell
  • Tools available in container
  • Standard debugging

kubectl debug

# Works with any container
kubectl debug <pod-name> -it --image=busybox

# Advantages:
# - Works with distroless
# - Can add any tools
# - Ephemeral (doesn't affect original)

Use when:

  • Distroless or minimal images
  • Need additional tools
  • Want to preserve original container

Limitations

Ephemeral Containers

  • Cannot be added to pods with restartPolicy=Never
  • Cannot modify existing containers
  • Cannot access all container resources
  • Ephemeral (removed when pod terminates)

Pod Copy Mode

  • Creates new pod (uses resources)
  • Original pod must be running or completed
  • Some fields cannot be copied

Node Debugging

  • Requires privileged access
  • May impact node performance
  • Security considerations

Best Practices

1. Use Appropriate Images

Choose debugging images based on needs:

  • busybox: Lightweight, common tools
  • netshoot: Network debugging
  • ubuntu: Full OS with all tools

2. Clean Up Debug Pods

Remove debug pods after debugging:

kubectl delete pod debug-pod

3. Use Target Container

When debugging specific container:

kubectl debug <pod-name> --target=<container-name> -it --image=busybox

4. Share Process Namespace

For process inspection:

kubectl debug <pod-name> --target=app --share-processes -it --image=busybox

5. Document Debugging

Document debugging steps and findings for future reference.

6. Security Considerations

  • Ephemeral containers have same security context as pod
  • Be careful with privileged containers
  • Don’t expose sensitive data in debug sessions

Troubleshooting

Ephemeral Containers Not Working

# Check Kubernetes version
kubectl version

# Verify feature gate (if < 1.25)
# Check API server configuration

# Check container runtime support
# Ephemeral containers require CRI support

Permission Denied

# Check pod security context
kubectl get pod <pod-name> -o jsonpath='{.spec.securityContext}'

# Verify debug container has required permissions
# May need to adjust security context

Debug Container Not Starting

# Check debug container status
kubectl describe pod <pod-name>

# Check events
kubectl get events --field-selector involvedObject.name=<pod-name>

# Verify image is available
kubectl debug <pod-name> --image=busybox --dry-run=client -o yaml

See Also