Graylog

Graylog is an open-source log management platform designed for centralized log collection, processing, and analysis. It provides powerful search capabilities, alerting, and dashboards, making it an excellent choice for Kubernetes log management.

What is Graylog?

Graylog is a log aggregation and analysis tool that:

Collects logs - From multiple sources via various inputs
Processes logs - Using extractors, pipelines, and stream rules
Stores logs - In Elasticsearch or OpenSearch
Analyzes logs - Through search, dashboards, and alerts
Alerts - Based on log patterns and conditions

graph TB A[Log Sources] --> B[Graylog Inputs] B --> C[Extractors/Pipelines] C --> D[Streams] D --> E[Elasticsearch/OpenSearch] E --> F[Graylog UI] G[Kubernetes Pods] --> B H[Node Logs] --> B I[Application Logs] --> B C --> C1[Parse Fields] C --> C2[Enrich Data] C --> C3[Transform] D --> D1[Route Logs] D --> D2[Filter] F --> F1[Search] F --> F2[Dashboards] F --> F3[Alerts] style A fill:#e1f5ff style B fill:#e8f5e9 style C fill:#fff4e1 style E fill:#f3e5f5 style F fill:#ffe1e1

Architecture

Graylog consists of:

Graylog Server - Processing and management
MongoDB - Configuration and metadata storage
Elasticsearch/OpenSearch - Log data storage
Graylog Web UI - User interface

graph TB A[Graylog Server] --> B[MongoDB] A --> C[Elasticsearch] A --> D[Web UI] E[Inputs] --> A F[GELF] --> E G[Beats] --> E H[Syslog] --> E A --> I[Extractors] A --> J[Pipelines] A --> K[Streams] K --> C style A fill:#e1f5ff style B fill:#e8f5e9 style C fill:#fff4e1 style D fill:#f3e5f5

Installation

Using Helm

# Add Graylog Helm repository
helm repo add graylog https://charts.graylog.org
helm repo update

# Install Graylog with dependencies
helm install graylog graylog/graylog \
  --namespace logging \
  --create-namespace \
  --set mongodb.mongodbUsername=graylog \
  --set mongodb.mongodbPassword=graylog \
  --set elasticsearch.clusterName=graylog

Manual Deployment

MongoDB

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mongodb
  namespace: logging
spec:
  serviceName: mongodb
  replicas: 1
  selector:
    matchLabels:
      app: mongodb
  template:
    metadata:
      labels:
        app: mongodb
    spec:
      containers:
      - name: mongodb
        image: mongo:6.0
        ports:
        - containerPort: 27017
        volumeMounts:
        - name: data
          mountPath: /data/db
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: mongodb-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: mongodb
  namespace: logging
spec:
  selector:
    app: mongodb
  ports:
  - port: 27017
    targetPort: 27017

Elasticsearch

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch
  namespace: logging
spec:
  serviceName: elasticsearch
  replicas: 3
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
      - name: elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
        env:
        - name: discovery.type
          value: single-node
        - name: ES_JAVA_OPTS
          value: "-Xms1g -Xmx1g"
        - name: xpack.security.enabled
          value: "false"
        ports:
        - containerPort: 9200
        volumeMounts:
        - name: data
          mountPath: /usr/share/elasticsearch/data
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: elasticsearch-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: elasticsearch
  namespace: logging
spec:
  selector:
    app: elasticsearch
  ports:
  - port: 9200
    targetPort: 9200

Graylog Server

apiVersion: apps/v1
kind: Deployment
metadata:
  name: graylog
  namespace: logging
spec:
  replicas: 1
  selector:
    matchLabels:
      app: graylog
  template:
    metadata:
      labels:
        app: graylog
    spec:
      containers:
      - name: graylog
        image: graylog/graylog:5.2
        env:
        - name: GRAYLOG_PASSWORD_SECRET
          valueFrom:
            secretKeyRef:
              name: graylog-secret
              key: password-secret
        - name: GRAYLOG_ROOT_PASSWORD_SHA2
          valueFrom:
            secretKeyRef:
              name: graylog-secret
              key: root-password-sha2
        - name: GRAYLOG_HTTP_EXTERNAL_URI
          value: http://graylog.logging.svc.cluster.local:9000/
        - name: GRAYLOG_ELASTICSEARCH_HOSTS
          value: http://elasticsearch:9200
        - name: GRAYLOG_MONGODB_URI
          value: mongodb://mongodb:27017/graylog
        ports:
        - containerPort: 9000
          name: http
        - containerPort: 12201
          name: gelf-udp
        - containerPort: 12201
          name: gelf-tcp
          protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: graylog
  namespace: logging
spec:
  selector:
    app: graylog
  ports:
  - port: 9000
    targetPort: 9000
    name: http
  - port: 12201
    targetPort: 12201
    name: gelf-udp
    protocol: UDP
  - port: 12201
    targetPort: 12201
    name: gelf-tcp
    protocol: TCP
  type: ClusterIP

Input Configuration

GELF UDP Input

GELF (Graylog Extended Log Format) is commonly used:

Go to System > Inputs in Graylog UI
Click Launch new input
Select GELF UDP
Configure:
- Title: Kubernetes GELF UDP
- Port: 12201
- Bind address: 0.0.0.0
Click Launch

Beats Input

For Filebeat/Beats integration:

Select Beats input type
Configure port: 5044
Launch input

Sending Logs to Graylog

Using Fluent Bit

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logging
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush         1
        Log_Level     info
    
    [INPUT]
        Name              tail
        Path              /var/log/pods/**/*.log
        Parser            cri
        Tag               kubernetes.*
    
    [FILTER]
        Name                kubernetes
        Match               kubernetes.*
        Kube_URL            https://kubernetes.default.svc:443
        Merge_Log           On
    
    [OUTPUT]
        Name  gelf
        Match *
        Host  graylog.logging.svc.cluster.local
        Port  12201
        Mode  udp
        Gelf_Short_Message_Key  message
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: logging
spec:
  selector:
    matchLabels:
      name: fluent-bit
  template:
    metadata:
      labels:
        name: fluent-bit
    spec:
      containers:
      - name: fluent-bit
        image: fluent/fluent-bit:latest
        volumeMounts:
        - name: config
          mountPath: /fluent-bit/etc
        - name: varlog
          mountPath: /var/log
      volumes:
      - name: config
        configMap:
          name: fluent-bit-config
      - name: varlog
        hostPath:
          path: /var/log

Using Filebeat

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: logging
data:
  filebeat.yml: |
    filebeat.inputs:
    - type: container
      paths:
        - /var/log/containers/*.log
    
    output.logstash:
      hosts: ["graylog.logging.svc.cluster.local:5044"]
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: logging
spec:
  selector:
    matchLabels:
      name: filebeat
  template:
    metadata:
      labels:
        name: filebeat
    spec:
      containers:
      - name: filebeat
        image: docker.elastic.co/beats/filebeat:8.11.0
        volumeMounts:
        - name: config
          mountPath: /usr/share/filebeat/filebeat.yml
          subPath: filebeat.yml
        - name: varlog
          mountPath: /var/log
          readOnly: true
      volumes:
      - name: config
        configMap:
          name: filebeat-config
      - name: varlog
        hostPath:
          path: /var/log

Extractors and Pipelines

Extractors

Extractors parse fields from log messages:

Go to Inputs
Click on input name
Go to Extractors tab
Click Load extractors or Add extractor

Example: JSON Extractor

Type: JSON
Field: message
Target field: Leave empty (extract to root)

Example: Grok Pattern

Type: Grok pattern
Pattern: %{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:message}

Pipelines

Pipelines provide more advanced processing:

Go to System > Pipelines
Create new pipeline
Add rules with conditions and actions

Example Pipeline Rule:

rule "Extract Kubernetes Fields"
when
  has_field("kubernetes")
then
  set_field("pod_name", $message.kubernetes.pod_name);
  set_field("namespace", $message.kubernetes.namespace);
  set_field("container_name", $message.kubernetes.container_name);
end

Streams

Streams route and filter logs:

Creating a Stream

Go to Streams
Click Create stream
Configure:
- Title: Production Errors
- Description: Error logs from production namespace
Add stream rules:
- Field: kubernetes.namespace
- Type: equals
- Value: production
- AND
- Field: level
- Type: equals
- Value: ERROR
Start stream

Stream Rules

Common stream rule types:

equals - Exact match
contains - Substring match
regex - Regular expression
greater than/less than - Numeric comparison
field presence - Check if field exists

Dashboards

Creating Dashboards

Go to Dashboards
Click Create dashboard
Add widgets:
- Message count - Number of messages
- Quick values - Field value counts
- Field statistics - Numeric field stats
- Search result count - Count based on search
- World map - Geographic visualization

Example Dashboard Widgets

Error Rate:

Widget: Message count
Query: level:ERROR
Time range: Last 1 hour

Top Error Messages:

Widget: Quick values
Field: message
Query: level:ERROR
Limit: 10

Alerting

Creating Alerts

Go to Alerts
Create alert condition
Configure:
- Stream - Which stream to monitor
- Condition - When to trigger (e.g., message count > 100)
- Time range - Evaluation window
- Notifications - Email, HTTP callback, etc.

Alert Conditions

Message count - Trigger when count exceeds threshold
Field value - Trigger based on field value
Field content - Trigger when message matches pattern

Notifications

Email - Send email alerts
HTTP callback - POST to webhook
Slack - Send to Slack channel

Best Practices

1. Index Management

Configure index sets:

Default retention: 30 days
Index rotation: Daily or by size
Index templates for consistent mapping

2. Resource Allocation

Graylog Server: Minimum 2GB RAM
Elasticsearch: Scale based on log volume
MongoDB: 1GB RAM sufficient for metadata

3. Input Optimization

Use appropriate input type for source
GELF for structured logs
Syslog for system logs
Beats for file-based collection

4. Stream Organization

Create streams by environment (dev, staging, prod)
Separate streams by log level
Use streams for routing and filtering

5. Performance

Limit extractor processing
Use pipelines efficiently
Optimize searches with proper indexes
Monitor Graylog performance

6. Security

Enable authentication
Use RBAC for user permissions
Encrypt connections (TLS)
Secure MongoDB access

Accessing Graylog

Port Forwarding

kubectl port-forward -n logging svc/graylog 9000:9000

Access at: http://localhost:9000

Default username: admin
Default password: (set via secret)

Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: graylog
  namespace: logging
spec:
  rules:
  - host: graylog.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: graylog
            port:
              number: 9000

Troubleshooting

Check Graylog Logs

# View Graylog server logs
kubectl logs -n logging -l app=graylog

# Check input status
# Go to System > Inputs in UI

Verify Elasticsearch Connection

# Test Elasticsearch connectivity
kubectl exec -n logging graylog-0 -- \
  curl -s http://elasticsearch:9200/_cluster/health

Check MongoDB Connection

# Test MongoDB connectivity
kubectl exec -n logging graylog-0 -- \
  mongo --eval "db.adminCommand('ping')" mongodb://mongodb:27017/graylog

No Logs Appearing

Check input is running
Verify logs are being sent to correct port
Check network connectivity
Verify extractors/pipelines are working
Check stream rules aren’t filtering out logs