Debugging Toolkit

Debugging Kubernetes applications requires a variety of tools and techniques. This guide covers the essential debugging tools available in Kubernetes, from kubectl commands to ephemeral containers, helping you diagnose and resolve issues efficiently.

Overview

Kubernetes provides several built-in tools for debugging:

kubectl logs - View container logs
kubectl exec - Execute commands in running containers
kubectl debug - Create ephemeral containers for debugging
kubectl describe - Get detailed resource information
kubectl get events - View cluster events
kubectl port-forward - Forward ports for local access

graph TB A[Debugging Tools] --> B[kubectl logs] A --> C[kubectl exec] A --> D[kubectl debug] A --> E[kubectl describe] A --> F[kubectl get events] A --> G[kubectl port-forward] B --> B1[View Logs] B --> B2[Follow Logs] B --> B3[Previous Instance] C --> C1[Execute Commands] C --> C2[Interactive Shell] C --> C3[Run Diagnostics] D --> D1[Ephemeral Containers] D --> D2[Node Debugging] D --> D3[Copy Pod] E --> E1[Resource Details] E --> E2[Events] E --> E3[Configuration] F --> F1[Cluster Events] F --> F2[Resource Events] F --> F3[Watch Events] G --> G1[Pod Port Forward] G --> G2[Service Port Forward] G --> G3[Debug Services] style A fill:#e1f5ff style D fill:#e8f5e9 style E fill:#fff4e1

When to Use Each Tool

kubectl logs

Use for:

Viewing application output
Debugging application errors
Monitoring application behavior
Checking startup logs

# Basic usage
kubectl logs <pod-name>

# Follow logs
kubectl logs -f <pod-name>

# Previous instance (crash loops)
kubectl logs <pod-name> --previous

kubectl exec

Use for:

Running commands in running containers
Interactive debugging sessions
Checking file system
Testing connectivity
Inspecting environment

# Interactive shell
kubectl exec -it <pod-name> -- /bin/sh

# Run command
kubectl exec <pod-name> -- env

# Specific container
kubectl exec -it <pod-name> -c <container-name> -- /bin/sh

kubectl debug

Use for:

Debugging containers without exec (distroless images)
Adding debugging tools to minimal containers
Node-level debugging
Inspecting containers with missing shells

# Debug pod
kubectl debug <pod-name> -it --image=busybox

# Copy pod with debug container
kubectl debug <pod-name> -it --image=busybox --copy-to=debug-pod

kubectl describe

Use for:

Understanding resource state
Viewing events
Checking configuration
Diagnosing why resources are in current state

# Describe pod
kubectl describe pod <pod-name>

# Describe service
kubectl describe service <service-name>

# Describe node
kubectl describe node <node-name>

kubectl get events

Use for:

Understanding what happened
Chronological event timeline
Identifying issues
Monitoring resource changes

# All events
kubectl get events -A

# Watch events
kubectl get events -w

# Filter by resource
kubectl get events --field-selector involvedObject.name=<pod-name>

kubectl port-forward

Use for:

Accessing services locally
Testing services
Debugging connectivity
Local development

# Forward pod port
kubectl port-forward <pod-name> 8080:80

# Forward service port
kubectl port-forward svc/<service-name> 8080:80

Debugging Workflow

A systematic approach to debugging:

graph TD A[Issue Detected] --> B[Check Resource Status] B --> C{Status Known?} C -->|No| D[Describe Resource] C -->|Yes| E[Check Events] D --> E E --> F{Events Show Issue?} F -->|Yes| G[Check Logs] F -->|No| H[Check Configuration] G --> I{Logs Reveal Issue?} I -->|Yes| J[Execute Into Pod] I -->|No| K[Use Debug Container] J --> L[Inspect Internals] K --> L L --> M{Issue Found?} M -->|Yes| N[Apply Fix] M -->|No| O[Port Forward] O --> P[Test Locally] P --> Q{Issue Reproduced?} Q -->|Yes| N Q -->|No| R[Check Dependencies] style A fill:#e1f5ff style G fill:#fff4e1 style L fill:#e8f5e9 style N fill:#e8f5e9

Common Debugging Scenarios

Pod Not Starting

# Step 1: Check pod status
kubectl get pod <pod-name>

# Step 2: Describe pod for events
kubectl describe pod <pod-name>

# Step 3: Check logs (if container started)
kubectl logs <pod-name>

# Step 4: Check previous instance
kubectl logs <pod-name> --previous

CrashLoopBackOff

# Step 1: Check logs from previous instance
kubectl logs <pod-name> --previous

# Step 2: Describe pod for events
kubectl describe pod <pod-name>

# Step 3: Execute into pod (if possible)
kubectl exec -it <pod-name> -- /bin/sh

# Step 4: Use debug container if no shell
kubectl debug <pod-name> -it --image=busybox

Service Not Accessible

# Step 1: Check service endpoints
kubectl get endpoints <service-name>

# Step 2: Check service configuration
kubectl describe service <service-name>

# Step 3: Port forward to test
kubectl port-forward svc/<service-name> 8080:80

# Step 4: Test from pod
kubectl exec -it <pod-name> -- curl http://<service-name>

ImagePullBackOff

# Step 1: Check image pull secrets
kubectl get pod <pod-name> -o jsonpath='{.spec.imagePullSecrets}'

# Step 2: Describe pod for events
kubectl describe pod <pod-name>

# Step 3: Check image name and tag
kubectl get pod <pod-name> -o jsonpath='{.spec.containers[*].image}'

# Step 4: Test image pull manually (if on node)
docker pull <image-name>

Network Issues

# Step 1: Check pod network
kubectl exec -it <pod-name> -- ip addr

# Step 2: Test DNS resolution
kubectl exec -it <pod-name> -- nslookup <service-name>

# Step 3: Test connectivity
kubectl exec -it <pod-name> -- ping <target>

# Step 4: Check network policies
kubectl get networkpolicies -A

Debugging Decision Tree

graph TD A[Debugging Need] --> B{What to Debug?} B -->|Application| C[Application Debugging] B -->|Configuration| D[Configuration Debugging] B -->|Network| E[Network Debugging] B -->|Resource| F[Resource Debugging] C --> C1[kubectl logs] C1 --> C2[kubectl exec] C2 --> C3[kubectl debug] D --> D1[kubectl describe] D1 --> D2[kubectl get -o yaml] D2 --> D3[Check ConfigMaps/Secrets] E --> E1[kubectl exec ping] E1 --> E2[kubectl exec curl] E2 --> E3[kubectl port-forward] F --> F1[kubectl describe] F1 --> F2[kubectl get events] F2 --> F3[Check Resources] style A fill:#e1f5ff style C3 fill:#e8f5e9 style D3 fill:#fff4e1 style E3 fill:#f3e5f5

Tips and Best Practices

1. Start with Events

Events provide chronological context:

kubectl describe <resource> | grep -A 10 Events

2. Check Previous Instance

For crash loops, always check previous instance:

kubectl logs <pod-name> --previous

3. Use Appropriate Tools

Use kubectl exec for containers with shells
Use kubectl debug for distroless/minimal images
Use kubectl logs for initial investigation

4. Debug in Order

Check status
Check events
Check logs
Execute into container
Use debug container if needed

5. Keep Debug Containers

Consider keeping debug containers in your toolchain:

busybox
netshoot (network debugging)
curl (HTTP debugging)
dig (DNS debugging)

6. Use Namespace Context

Always specify namespace when needed:

kubectl logs <pod-name> -n <namespace>

7. Combine Commands

Chain commands for efficient debugging:

kubectl get pods | grep Error | awk '{print $1}' | xargs -I {} kubectl logs {}

Topics

kubectl debug - Using ephemeral containers for debugging
Events - Understanding and using Kubernetes events

Debugging Toolkit

Overview

When to Use Each Tool

kubectl logs

kubectl exec

kubectl debug

kubectl describe

kubectl get events

kubectl port-forward

Debugging Workflow

Common Debugging Scenarios

Pod Not Starting

CrashLoopBackOff

Service Not Accessible

ImagePullBackOff

Network Issues

Debugging Decision Tree

Tips and Best Practices

1. Start with Events

2. Check Previous Instance

3. Use Appropriate Tools

4. Debug in Order

5. Keep Debug Containers

6. Use Namespace Context

7. Combine Commands

Topics

See Also