Prometheus 2.0: New Storage Engine and HA-Friendly Scraping

Introduction

At PromCon 2017 on November 8, the Prometheus team announced Prometheus 2.0, a major overhaul of the beloved monitoring system. The headline change: a brand-new time series database designed for higher ingestion rates, faster queries and more reliable storage—perfect for busy Kubernetes clusters.

Key Upgrades

• TSDB Rewrite: Blocks-based storage with 2-hour chunks, memory-efficient postings lists, and background compaction drastically reduces disk I/O.
• Write-Ahead Log (WAL): Durable WAL makes scrapes crash-safe, enabling faster restarts and fewer data gaps.
• Staleness Handling: Prometheus now marks stale series automatically, eliminating “stuck” metrics after pods roll or jobs terminate.
• Remote Write/Read Enhancements: gRPC-like batching and backpressure controls improve integrations with Cortex, Thanos and InfluxDB.
• Alertmanager v0.15+ Compatibility: Improved label handling and inhibit rules make incident routing more expressive.

Migration Guidance

1. Backup existing Prometheus 1.x data; 2.0 cannot read old TSDB files.
2. Deploy 2.0 in parallel and dual-scrape targets before cutting over.
3. Update recording/alerting rules to account for staleness markers (or on() patterns).
4. Monitor WAL size and retention; default retention is 15d with --storage.tsdb.retention=15d.

Sample Kubernetes manifest snippet:

spec:
  containers:
  - name: prometheus
    image: prom/prometheus:v2.0.0
    args:
    - "--config.file=/etc/prometheus/prometheus.yaml"
    - "--storage.tsdb.path=/prometheus"
    - "--storage.tsdb.retention=30d"
    - "--web.enable-lifecycle"

Operational Notes (TSDB Reality Checks)

• The new TSDB is faster, but it’s also more disk-sensitive: give Prometheus a real persistent volume (IOPS matter) if you want predictable compactions and query latency.
• Keep an eye on WAL growth during outages; if scrape targets flap or remote write backs up, the WAL can grow faster than expected.
• Treat retention as a capacity planning input: longer retention increases compaction work and storage needs (and makes node reschedules more painful).

Performance Wins

• Benchmarks show 3–10x query speedups for range queries, especially on high-cardinality label sets.
• WAL amortizes disk writes, allowing Prometheus to survive node evictions or kubelet restarts with minimal data loss.
• Memory footprint becomes more predictable; target metadata now stored in dedicated indexes.

Kubernetes Integrations

• kube-prometheus manifests updated to 2.0 with new recording rules, Grafana dashboards and Alertmanager pipelines.
• Operators (CoreOS Prometheus Operator) added automated upgrades, persistent volume migrations and Thanos remote read configuration.
• Federation patterns improved; upstream clusters can scrape leaf Prometheus servers that expose 2.0 metrics without compatibility issues.

Summary