Grafana

Grafana is an open-source analytics and interactive visualization platform that allows you to create dashboards for metrics, logs, and traces. It’s the most popular tool for visualizing Prometheus metrics, Loki logs, and distributed traces, making it a central component of the Kubernetes observability stack.

What is Grafana?

Grafana provides:

Unified visualization - Single platform for metrics, logs, and traces
Rich dashboards - Create beautiful, interactive dashboards
Multiple data sources - Connect to Prometheus, Loki, Elasticsearch, and many others
Alerting - Built-in alerting system with notifications
Exploration - Interactive querying and data exploration
Collaboration - Share dashboards and work as a team

graph TB A[Grafana] --> B[Data Sources] B --> C[Prometheus] B --> D[Loki] B --> E[Elasticsearch] B --> F[Tempo] B --> G[Other Sources] A --> H[Dashboards] H --> I[Panels] I --> J[Graphs] I --> K[Tables] I --> L[Gauges] A --> M[Alerting] M --> N[Notifications] style A fill:#e1f5ff style B fill:#e8f5e9 style H fill:#fff4e1 style M fill:#f3e5f5

Grafana in the Observability Stack

Grafana sits at the top of your observability stack, querying data from various sources:

graph TB A[Grafana] --> B[Query Layer] B --> C[Prometheus] B --> D[Loki] B --> E[Tempo] C --> F[Metrics] D --> G[Logs] E --> H[Traces] F --> I[Time-Series Data] G --> J[Log Aggregation] H --> K[Distributed Traces] A --> L[Dashboards] A --> M[Alerts] style A fill:#e1f5ff style C fill:#e8f5e9 style D fill:#fff4e1 style E fill:#f3e5f5 style L fill:#ffe1e1

Key Concepts

Data Sources

Data sources are connections to where your data lives:

Prometheus - Metrics and time-series data
Loki - Log aggregation
Tempo - Distributed tracing
Elasticsearch - Logs and search
InfluxDB - Time-series database
Many others - 50+ supported data sources

Dashboards

Dashboards are collections of panels that visualize your data:

Organized views - Group related metrics together
Customizable - Full control over layout and styling
Interactive - Zoom, pan, refresh, filter
Shareable - Export/import dashboards as JSON

Panels

Panels are individual visualizations within a dashboard:

Graph - Time-series line charts
Table - Tabular data display
Stat - Single value or gauge
Gauge - Circular or linear gauges
Heatmap - Color-coded matrix
Many more - 20+ panel types

Variables

Variables make dashboards dynamic and reusable:

Template variables - Filter dashboards by namespace, pod, service, etc.
Query variables - Populate from data source queries
Custom variables - Manual lists or text inputs

Installation

Using Helm (Recommended)

# Add Grafana Helm repository
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

# Install Grafana
helm install grafana grafana/grafana \
  --namespace monitoring \
  --create-namespace \
  --set persistence.enabled=true \
  --set persistence.size=10Gi

Using kube-prometheus-stack

If you installed Prometheus using kube-prometheus-stack, Grafana is included:

helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace

Manual Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      containers:
      - name: grafana
        image: grafana/grafana:latest
        ports:
        - containerPort: 3000
        env:
        - name: GF_SECURITY_ADMIN_PASSWORD
          valueFrom:
            secretKeyRef:
              name: grafana-secret
              key: admin-password
        volumeMounts:
        - name: storage
          mountPath: /var/lib/grafana
      volumes:
      - name: storage
        persistentVolumeClaim:
          claimName: grafana-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: grafana
  namespace: monitoring
spec:
  selector:
    app: grafana
  ports:
  - port: 3000
    targetPort: 3000
  type: ClusterIP

Accessing Grafana

Port Forwarding

kubectl port-forward -n monitoring svc/grafana 3000:3000

Access: http://localhost:3000

Default username: admin
Get password: kubectl get secret -n monitoring grafana -o jsonpath="{.data.admin-password}" | base64 -d

Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: grafana
  namespace: monitoring
spec:
  rules:
  - host: grafana.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: grafana
            port:
              number: 3000

Basic Workflow

1. Add Data Source

Go to Configuration > Data Sources
Click Add data source
Select your data source type (e.g., Prometheus)
Configure connection settings
Click Save & Test

2. Create Dashboard

Go to Dashboards
Click New > Dashboard
Click Add visualization
Select data source and query
Choose visualization type
Customize panel settings
Click Apply and Save dashboard

3. Add Panels

Click Add panel in dashboard
Or use + Add visualization
Configure query and visualization
Set panel title and description

Common Use Cases

Kubernetes Cluster Monitoring

Monitor cluster health, resource usage, and pod status:

Node CPU/memory usage
Pod resource consumption
Namespace resource quotas
Cluster capacity planning

Application Performance

Track application metrics and performance:

Request rates and latencies
Error rates
Database query performance
Cache hit rates

Infrastructure Monitoring

Monitor infrastructure components:

API server performance
etcd health
Network throughput
Storage usage

Log Analysis

View and analyze logs alongside metrics:

Correlate errors with metrics
Search log messages
Filter by labels/tags
Time-based log exploration

Integration with Prometheus

Grafana works seamlessly with Prometheus:

# Example: Prometheus data source configuration
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus.monitoring.svc.cluster.local:9090
    isDefault: true

Example PromQL Query in Grafana

# CPU usage percentage
100 - (avg(irate(container_cpu_usage_seconds_total[5m])) * 100)

# Memory usage
container_memory_usage_bytes / container_spec_memory_limit_bytes * 100

# Request rate
rate(http_requests_total[5m])

Best Practices

Organize dashboards - Use folders and tags to organize dashboards
Use variables - Make dashboards reusable with template variables
Set refresh intervals - Configure appropriate auto-refresh rates
Dashboard annotations - Add annotations for deployments and incidents
Alert rules - Use Grafana alerts for critical metrics
Dashboard provisioning - Store dashboards as code (ConfigMaps/Helm)
Performance - Optimize queries to avoid slow dashboards
Access control - Configure RBAC and authentication properly

Dashboard Provisioning

Store dashboards as YAML/JSON for version control:

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-dashboards
  namespace: monitoring
data:
  kubernetes.json: |
    {
      "dashboard": {
        "title": "Kubernetes Cluster",
        "panels": [...]
      }
    }

Reference in Grafana ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana
  namespace: monitoring
data:
  grafana.ini: |
    [dashboards]
    default_home_dashboard_path = /var/lib/grafana/dashboards/kubernetes.json

Topics

Dashboards - Creating and managing dashboards
Data Sources - Configuring data sources