Logs & Output Streams

Understanding how Kubernetes captures and handles logs from containers is fundamental to effective debugging and monitoring. This guide explains stdout/stderr redirection, log collection, rotation, and best practices for containerized applications.

Understanding Container Logs

In Kubernetes, container logs are captured from standard output (stdout) and standard error (stderr) streams. Unlike traditional applications that write to log files, containerized applications should write all output to these streams.

stdout and stderr

stdout (Standard Output)

Normal program output goes to stdout:

• Application logs (INFO, DEBUG levels)
• Application output
• Successful operations

stderr (Standard Error)

Error output goes to stderr:

• Error messages
• Warnings
• Stack traces
• Fatal errors

Why Use stdout/stderr?

Kubernetes automatically captures and aggregates logs from these streams:

• Automatic collection - No need to configure log file locations
• Pod-level aggregation - All containers in a pod can be accessed together
• API access - Access logs via kubectl logs
• Standard practice - Follows 12-factor app principles

How Kubernetes Captures Logs

Container Runtime Capture

The container runtime (containerd, CRI-O, Docker) captures stdout/stderr:

Log Storage Locations

Logs are stored on the node’s filesystem:

Pod logs:

/var/log/pods/<namespace>_<pod-name>_<pod-uid>/
  <container-name>/
    <instance-number>.log

Container logs (symlinks):

/var/log/containers/<pod-name>_<namespace>_<container-name>-<container-id>.log

These paths are symlinks managed by kubelet for easier access.

Viewing Logs

kubectl logs

The primary way to view logs:

# View logs from pod
kubectl logs <pod-name>

# Follow logs in real-time
kubectl logs -f <pod-name>

# View logs from previous container instance
kubectl logs <pod-name> --previous

# View last N lines
kubectl logs <pod-name> --tail=100

# View logs with timestamps
kubectl logs <pod-name> --timestamps

# View logs since specific time
kubectl logs <pod-name> --since=1h

# View logs from specific time range
kubectl logs <pod-name> --since-time='2024-01-01T00:00:00Z'

Multi-Container Pods

# List containers in pod
kubectl get pod <pod-name> -o jsonpath='{.spec.containers[*].name}'

# View logs from specific container
kubectl logs <pod-name> -c <container-name>

# View logs from all containers
kubectl logs <pod-name> --all-containers=true

Aggregating Logs from Multiple Pods

# View logs from all pods with label
kubectl logs -l app=my-app

# View logs from all pods in namespace
kubectl logs -l app=my-app -n production

# Aggregate and follow
kubectl logs -f -l app=my-app --prefix=true

Streaming Logs

# Follow logs (like tail -f)
kubectl logs -f <pod-name>

# Follow logs from multiple pods
kubectl logs -f -l app=my-app

# Follow with prefix showing pod name
kubectl logs -f -l app=my-app --prefix=true

Log Format

Default Log Format

By default, Kubernetes logs include:

• Timestamp - When the log entry was written
• Stream - stdout or stderr
• Content - The actual log message

Example:

2024-01-15T10:30:45.123456789Z stdout F This is a log message
2024-01-15T10:30:45.234567890Z stderr F This is an error message

Structured Logging

For better parsing and analysis, use structured formats (JSON):

{"timestamp":"2024-01-15T10:30:45Z","level":"INFO","message":"Request processed","method":"GET","path":"/api/users","status":200,"duration_ms":45}

Benefits:

• Easy parsing by log aggregation tools
• Better filtering and searching
• Support for nested structures
• Consistent format across services

Log Rotation

Kubernetes automatically rotates logs to prevent disk space issues:

Rotation Policy

• Size limit: 10MB per log file (default)
• Number of files: Maximum 5 log files per container (default)
• Rotation: When size limit is reached
• Cleanup: Oldest files are deleted automatically

Configuration

Log rotation is controlled by kubelet configuration:

# kubelet config (partial)
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
containerLogMaxSize: 10Mi
containerLogMaxFiles: 5

Log Retention

• Logs are kept on the node until rotated out
• No built-in long-term retention
• For retention, use centralized logging solutions

Log Aggregation Patterns

Sidecar Pattern

A sidecar container collects logs from the application container:

apiVersion: v1
kind: Pod
metadata:
  name: app-with-log-sidecar
spec:
  containers:
  - name: app
    image: my-app:latest
    volumeMounts:
    - name: logs
      mountPath: /var/log/app
  - name: log-collector
    image: fluent-bit:latest
    volumeMounts:
    - name: logs
      mountPath: /var/log/app
      readOnly: true
  volumes:
  - name: logs
    emptyDir: {}

DaemonSet Pattern

A DaemonSet runs on every node to collect logs:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
spec:
  selector:
    matchLabels:
      name: fluent-bit
  template:
    metadata:
      labels:
        name: fluent-bit
    spec:
      containers:
      - name: fluent-bit
        image: fluent/fluent-bit:latest
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers

Common Log Output Issues

Missing Logs

Problem: Logs not appearing in kubectl logs

Causes:

• Application not writing to stdout/stderr
• Logs written to files instead of streams
• Container crashed before logs were flushed

Solution:

# Check if container is running
kubectl get pod <pod-name>

# Check previous instance logs
kubectl logs <pod-name> --previous

# Verify application writes to stdout
kubectl exec <pod-name> -- env | grep LOG

Truncated Logs

Problem: Logs are cut off or incomplete

Causes:

• Log rotation during high-volume logging
• Buffer overflow
• Application not flushing buffers

Solution:

# View logs from previous instances
kubectl logs <pod-name> --previous

# Check all rotated log files on node
# (requires node access)
ls -la /var/log/pods/<namespace>_<pod-name>_*/

High Log Volume

Problem: Too many logs causing performance issues

Causes:

• Verbose logging (DEBUG level)
• Logging in tight loops
• Including unnecessary data

Solution:

• Use appropriate log levels
• Implement log sampling
• Use structured logging efficiently
• Configure log rotation appropriately

No Timestamps

Problem: Logs missing timestamp information

Causes:

• Application not including timestamps
• Using –timestamps flag not set

Solution:

# View with timestamps
kubectl logs <pod-name> --timestamps

# Or configure application to include timestamps

Best Practices

1. Write to stdout/stderr

Always write logs to stdout/stderr, not files:

# Good
import sys
print("Log message", file=sys.stdout)

# Bad
with open("/var/log/app.log", "a") as f:
    f.write("Log message\n")

2. Use Structured Logging

Use JSON or other structured formats:

import json
import sys

log_entry = {
    "timestamp": "2024-01-15T10:30:45Z",
    "level": "INFO",
    "message": "Request processed",
    "request_id": "abc123"
}
print(json.dumps(log_entry), file=sys.stdout)

3. Set Appropriate Log Levels

Use log levels appropriately:

• DEBUG: Detailed information for debugging
• INFO: General informational messages
• WARN: Warning messages
• ERROR: Error messages
• FATAL: Critical errors

4. Include Context

Add contextual information to logs:

{
  "timestamp": "2024-01-15T10:30:45Z",
  "level": "INFO",
  "message": "Request processed",
  "user_id": "123",
  "request_id": "abc123",
  "duration_ms": 45,
  "status_code": 200
}

5. Avoid Sensitive Data

Never log:

• Passwords
• API keys
• Tokens
• Personal information (PII)
• Credit card numbers

6. Flush Logs Regularly

Ensure logs are flushed, especially for crash scenarios:

import sys
sys.stdout.flush()
sys.stderr.flush()

7. Use Log Solutions

For production, use log solutions:

• ELK Stack
• Loki
• Datadog
• CloudWatch (AWS)
• Azure Monitor (Azure)

Accessing Logs Directly on Node

If you have node access, you can view logs directly:

# View pod logs
cat /var/log/pods/<namespace>_<pod-name>_<pod-uid>/<container-name>/<instance>.log

# Follow logs
tail -f /var/log/pods/<namespace>_<pod-name>_<pod-uid>/<container-name>/<instance>.log

# View container logs (symlink)
tail -f /var/log/containers/<pod-name>_<namespace>_<container-name>-<container-id>.log

Log Levels and Filtering

Filtering by Log Level

If using structured logging with levels:

# View only ERROR logs (requires jq)
kubectl logs <pod-name> | jq 'select(.level == "ERROR")'

# View logs from last hour
kubectl logs <pod-name> --since=1h

Searching Logs

# Search for specific text
kubectl logs <pod-name> | grep "ERROR"

# Case-insensitive search
kubectl logs <pod-name> | grep -i "error"

# Count occurrences
kubectl logs <pod-name> | grep -c "ERROR"

See Also

• Container & Pod logs - Detailed guide on viewing container logs
• Node & Sidecar logging - Advanced logging patterns
• Log Solutions - Solutions for log aggregation