Logging

Logging is one of the three pillars of observability in Kubernetes, alongside metrics and traces. Understanding how logging works in Kubernetes is essential for debugging applications, monitoring system health, and maintaining production workloads.

The Three Pillars of Observability

Observability in Kubernetes consists of three complementary signals:

graph TB A[Observability] --> B[Logs] A --> C[Metrics] A --> D[Traces] B --> B1[What Happened?] B --> B2[Event Records] B --> B3[Text-Based] C --> C1[How Much?] C --> C2[Numerical Data] C --> C3[Time-Series] D --> D1[Why?] D --> D2[Request Flow] D --> D3[Distributed Context] style A fill:#e1f5ff style B fill:#fff4e1 style C fill:#e8f5e9 style D fill:#f3e5f5

Logs answer “what happened?” by providing event records in text format. They’re essential for debugging and understanding application behavior.

Metrics answer “how much?” by providing numerical data over time. They’re ideal for dashboards and alerting.

Traces answer “why?” by showing request flows across distributed systems. They help identify bottlenecks and dependencies.

Kubernetes Logging Architecture

Kubernetes doesn’t provide a built-in centralized logging solution. Instead, it captures container logs at the node level, and you need additional components to collect, store, and analyze them.

graph TB A[Application Container] --> B[stdout/stderr] B --> C[Container Runtime] C --> D[Kubelet] D --> E[Node Log Files] E --> F[Log Collection Agent] F --> G[Centralized Storage] G --> H[Log Analysis Tool] I[System Components] --> J[kubelet.log] I --> K[api-server.log] I --> L[etcd.log] J --> F K --> F L --> F style A fill:#e1f5ff style B fill:#fff4e1 style D fill:#e8f5e9 style F fill:#f3e5f5 style G fill:#ffe1e1 style H fill:#fff4e1

Log Lifecycle

Understanding how logs flow through the system helps you troubleshoot and design logging solutions:

1. Application Logging

Applications write logs to stdout or stderr. Kubernetes captures everything written to these streams.

# Example: Application writing to stdout
apiVersion: v1
kind: Pod
metadata:
  name: app-logging
spec:
  containers:
  - name: app
    image: nginx:latest
    # Logs go to stdout/stderr automatically

2. Container Runtime Capture

The container runtime (containerd, CRI-O, Docker) captures stdout/stderr from containers and stores them in a log format.

3. Kubelet Collection

Kubelet reads logs from the container runtime and makes them available via the Kubernetes API or node filesystem.

4. Node Storage

By default, logs are stored on the node’s filesystem at /var/log/pods/ and /var/log/containers/. These logs are rotated automatically.

5. Centralized Collection

A log collection agent (running as a DaemonSet) reads logs from nodes and forwards them to centralized storage (Elasticsearch, Loki, CloudWatch, etc.).

6. Analysis and Visualization

Log analysis tools (Kibana, Grafana, CloudWatch Insights) allow you to search, filter, and visualize logs.

Key Concepts

Log Rotation

Kubernetes automatically rotates logs to prevent disk space issues:

Container logs are rotated when they reach 10MB
Maximum of 5 log files per container
Old log files are automatically deleted

Log Retention

By default, Kubernetes keeps logs on nodes until they’re rotated out. For long-term retention, you need centralized logging.

Multi-Container Pods

Each container in a pod has separate log streams. You need to specify which container’s logs to view:

# View logs from specific container
kubectl logs <pod-name> -c <container-name>

Logging Patterns

There are several patterns for collecting logs in Kubernetes:

Sidecar Pattern

A sidecar container in the same pod collects and forwards logs:

apiVersion: v1
kind: Pod
metadata:
  name: app-with-sidecar
spec:
  containers:
  - name: app
    image: my-app:latest
  - name: log-collector
    image: fluent-bit:latest
    volumeMounts:
    - name: logs
      mountPath: /var/log/app

DaemonSet Pattern

A DaemonSet runs a log collector on every node:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
spec:
  selector:
    matchLabels:
      name: fluent-bit
  template:
    metadata:
      labels:
        name: fluent-bit
    spec:
      containers:
      - name: fluent-bit
        image: fluent/fluent-bit:latest
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers

Viewing Logs

The most common way to view logs is using kubectl logs:

# View logs from a pod
kubectl logs <pod-name>

# Follow logs in real-time
kubectl logs -f <pod-name>

# View logs from previous container instance
kubectl logs <pod-name> --previous

# View logs from specific container in multi-container pod
kubectl logs <pod-name> -c <container-name>

# View logs with timestamps
kubectl logs <pod-name> --timestamps

# View last N lines
kubectl logs <pod-name> --tail=100

# View logs from all pods matching label
kubectl logs -l app=my-app

Best Practices

Write to stdout/stderr - Don’t write logs to files in containers. Use stdout/stderr instead.
Use structured logging - Use JSON or other structured formats for easier parsing and analysis.
Include context - Add timestamps, log levels, and contextual information.
Avoid sensitive data - Never log passwords, tokens, or other sensitive information.
Set appropriate log levels - Use DEBUG, INFO, WARN, ERROR appropriately.
Centralize logs - Use a centralized logging solution for production clusters.
Monitor log volume - Too many logs can impact performance and storage costs.
Use log aggregation - Aggregate logs from multiple pods/services for easier analysis.

Common Use Cases

Debugging application issues - Find errors and understand application behavior
Security auditing - Track access and detect suspicious activity
Performance analysis - Identify slow requests and bottlenecks
Compliance - Meet regulatory requirements for log retention
Troubleshooting deployments - Understand why pods fail to start

Topics

Container & Pod logs - How to view and work with container logs
Node & Sidecar logging - Advanced logging patterns
Log Solutions - Solutions for log aggregation and analysis