Container & Pod logs

Understanding how to view and work with container logs is essential for debugging applications in Kubernetes. This guide covers kubectl logs usage, multi-container pods, log aggregation, and practical examples for common scenarios.

How Container Logs Work

Every container in Kubernetes writes logs to stdout and stderr. The container runtime captures these streams, and kubelet makes them available through the Kubernetes API and node filesystem.

graph TB A[Container Process] --> B[stdout] A --> C[stderr] B --> D[Container Runtime] C --> D D --> E[Kubelet] E --> F[Kubernetes API] E --> G[Node Filesystem] F --> H[kubectl logs] G --> I[/var/log/pods/] style A fill:#e1f5ff style D fill:#e8f5e9 style E fill:#fff4e1 style H fill:#f3e5f5

kubectl logs Command

Basic Usage

# View logs from a pod
kubectl logs <pod-name>

# View logs from pod in namespace
kubectl logs <pod-name> -n <namespace>

# View logs with timestamps
kubectl logs <pod-name> --timestamps

Example:

kubectl logs my-app-7d8f9b5c4-x2k9j

Output:

2024-01-15T10:30:45.123456789Z Starting application...
2024-01-15T10:30:46.234567890Z Listening on port 8080
2024-01-15T10:30:47.345678901Z Health check endpoint ready

Following Logs

Stream logs in real-time (like tail -f):

# Follow logs
kubectl logs -f <pod-name>

# Follow logs with timestamps
kubectl logs -f <pod-name> --timestamps

# Follow logs from multiple pods
kubectl logs -f -l app=my-app

Viewing Previous Instance

For crash loops, view logs from the previous container instance:

# View logs from previous instance
kubectl logs <pod-name> --previous

# Previous instance with timestamps
kubectl logs <pod-name> --previous --timestamps

This is crucial for debugging crash loops, as the current instance may crash before logs can be viewed.

Limiting Output

# View last N lines
kubectl logs <pod-name> --tail=100

# View logs since specific time
kubectl logs <pod-name> --since=1h

# View logs since specific time
kubectl logs <pod-name> --since-time='2024-01-15T10:00:00Z'

# Limit to last 10 minutes
kubectl logs <pod-name> --since=10m

Output Formatting

# Include timestamps
kubectl logs <pod-name> --timestamps

# Prefix with pod name (useful for multiple pods)
kubectl logs -l app=my-app --prefix=true

# Custom date format (requires log format support)
kubectl logs <pod-name> --timestamps

Multi-Container Pods

When a pod has multiple containers, you need to specify which container’s logs to view.

Listing Containers

# List containers in pod
kubectl get pod <pod-name> -o jsonpath='{.spec.containers[*].name}'

# Get container names
kubectl get pod <pod-name> -o json | jq '.spec.containers[].name'

Viewing Specific Container Logs

# View logs from specific container
kubectl logs <pod-name> -c <container-name>

# Follow logs from specific container
kubectl logs -f <pod-name> -c <container-name>

# View logs from all containers
kubectl logs <pod-name> --all-containers=true

Example pod with multiple containers:

apiVersion: v1
kind: Pod
metadata:
  name: multi-container-app
spec:
  containers:
  - name: app
    image: my-app:latest
  - name: sidecar
    image: fluent-bit:latest
  - name: log-processor
    image: log-processor:latest

Viewing logs:

# Application container
kubectl logs multi-container-app -c app

# Sidecar container
kubectl logs multi-container-app -c sidecar

# All containers
kubectl logs multi-container-app --all-containers=true

Log Aggregation

Multiple Pods with Labels

Aggregate logs from multiple pods using label selectors:

# View logs from all pods with label
kubectl logs -l app=my-app

# Follow logs from all matching pods
kubectl logs -f -l app=my-app

# With prefix to identify pod
kubectl logs -f -l app=my-app --prefix=true

# Multiple label selectors
kubectl logs -l app=my-app,environment=production

Example with prefix:

kubectl logs -f -l app=my-app --prefix=true

Output:

[my-app-7d8f9b5c4-x2k9j] 2024-01-15T10:30:45Z Request received
[my-app-7d8f9b5c4-z7m3n] 2024-01-15T10:30:45Z Request received
[my-app-7d8f9b5c4-x2k9j] 2024-01-15T10:30:46Z Processing request

Aggregating by Namespace

# All pods in namespace
kubectl logs -l app=my-app -n production

# All pods in all namespaces (use with caution)
kubectl logs -l app=my-app --all-namespaces

Combining with Other Commands

# Get pod names and view their logs
kubectl get pods -l app=my-app -o name | xargs -I {} kubectl logs {}

# View logs from running pods only
kubectl get pods -l app=my-app --field-selector=status.phase=Running -o name | xargs -I {} kubectl logs {}

Log Format and Structure

Default Log Format

Kubernetes logs include:

Timestamp - When log was written
Stream - stdout or stderr
Log level (F for output)
Content - Actual log message

Format:

<timestamp> <stream> F <log-content>

Example:

2024-01-15T10:30:45.123456789Z stdout F INFO: Application started
2024-01-15T10:30:46.234567890Z stderr F ERROR: Connection failed

Structured Logging

For better parsing, use structured formats (JSON):

{"timestamp":"2024-01-15T10:30:45Z","level":"INFO","message":"Request processed","method":"GET","path":"/api/users","status":200,"duration_ms":45,"request_id":"abc123"}

Benefits:

Easy parsing by log aggregation tools
Better filtering and searching
Support for nested structures
Consistent format across services

Parsing Structured Logs

# Parse JSON logs with jq
kubectl logs <pod-name> | jq '.'

# Filter by level
kubectl logs <pod-name> | jq 'select(.level == "ERROR")'

# Extract specific fields
kubectl logs <pod-name> | jq '.message, .duration_ms'

Practical Examples

Debugging Application Startup

# View logs with timestamps
kubectl logs <pod-name> --timestamps

# View from beginning
kubectl logs <pod-name> --tail=0

# Follow logs during startup
kubectl logs -f <pod-name> --timestamps

Finding Errors

# Search for errors
kubectl logs <pod-name> | grep -i error

# Case-insensitive search
kubectl logs <pod-name> | grep -i "error\|exception\|failed"

# Count errors
kubectl logs <pod-name> | grep -i error | wc -l

# View errors with context
kubectl logs <pod-name> | grep -i error -A 5 -B 5

Monitoring Request Patterns

# Follow logs to monitor requests
kubectl logs -f <pod-name> | grep "GET\|POST\|PUT\|DELETE"

# Count requests by method (requires structured logs)
kubectl logs <pod-name> | jq 'select(.method) | .method' | sort | uniq -c

# Monitor slow requests
kubectl logs <pod-name> | jq 'select(.duration_ms > 1000)'

Debugging Crash Loops

# Step 1: View logs from previous instance
kubectl logs <pod-name> --previous

# Step 2: View logs from current attempt
kubectl logs <pod-name>

# Step 3: Follow logs to see crash in real-time
kubectl logs -f <pod-name>

# Step 4: Get pod details for events
kubectl describe pod <pod-name>

Time-Based Analysis

# Logs from last hour
kubectl logs <pod-name> --since=1h

# Logs from last 30 minutes
kubectl logs <pod-name> --since=30m

# Logs since specific time
kubectl logs <pod-name> --since-time='2024-01-15T10:00:00Z'

# Combine with filtering
kubectl logs <pod-name> --since=1h | grep -i error

Aggregating from Deployment

# View logs from all replicas
kubectl logs -l app=my-app --prefix=true

# Follow logs from all replicas
kubectl logs -f -l app=my-app --prefix=true

# View logs from specific replica
kubectl logs <pod-name> -l app=my-app

Troubleshooting

Missing Logs

Problem: kubectl logs returns nothing or “No logs available”

Causes:

Container hasn’t started yet
Container crashed immediately
Application not writing to stdout/stderr
Logs written to files instead of streams

Solutions:

# Check pod status
kubectl get pod <pod-name>

# Check previous instance
kubectl logs <pod-name> --previous

# Check if container is running
kubectl exec <pod-name> -- ps aux

# Verify application writes to stdout
kubectl exec <pod-name> -- echo "test" | kubectl logs <pod-name>

Truncated Logs

Problem: Logs are cut off or incomplete

Causes:

Log rotation during high-volume logging
Buffer overflow
Application not flushing buffers

Solutions:

# View logs from previous instances
kubectl logs <pod-name> --previous

# Check all log files (requires node access)
# Logs stored at: /var/log/pods/<namespace>_<pod-name>_*/

High Log Volume

Problem: Too many logs causing performance issues

Solutions:

Use appropriate log levels
Implement log sampling
Use structured logging efficiently
Configure log rotation appropriately

No Timestamps

Problem: Logs missing timestamp information

Solutions:

# View with timestamps
kubectl logs <pod-name> --timestamps

# Or configure application to include timestamps in log format

Best Practices

1. Always Check Previous Instance

For crash loops, always check the previous instance:

kubectl logs <pod-name> --previous

2. Use Timestamps

Include timestamps for better debugging:

kubectl logs <pod-name> --timestamps

3. Follow Logs During Testing

Watch logs in real-time when testing:

kubectl logs -f <pod-name>

4. Use Labels for Aggregation

Label your pods consistently:

labels:
  app: my-app
  version: v1.0
  environment: production

5. Structured Logging

Use structured formats (JSON) for better parsing:

{"timestamp":"2024-01-15T10:30:45Z","level":"INFO","message":"..."}

6. Include Context

Add contextual information to logs:

Request IDs
User IDs
Operation names
Durations

7. Appropriate Log Levels

Use log levels appropriately:

DEBUG: Detailed debugging info
INFO: General information
WARN: Warning messages
ERROR: Error messages
FATAL: Critical errors

Container & Pod logs

How Container Logs Work

kubectl logs Command

Basic Usage

Following Logs

Viewing Previous Instance

Limiting Output

Output Formatting

Multi-Container Pods

Listing Containers

Viewing Specific Container Logs

Log Aggregation

Multiple Pods with Labels

Aggregating by Namespace

Combining with Other Commands

Log Format and Structure

Default Log Format

Structured Logging

Parsing Structured Logs

Practical Examples

Debugging Application Startup

Finding Errors

Monitoring Request Patterns

Debugging Crash Loops

Time-Based Analysis

Aggregating from Deployment

Troubleshooting

Missing Logs

Truncated Logs

High Log Volume

No Timestamps

Best Practices

1. Always Check Previous Instance

2. Use Timestamps

3. Follow Logs During Testing

4. Use Labels for Aggregation

5. Structured Logging

6. Include Context

7. Appropriate Log Levels

See Also