Metrics & Monitoring

Problem Statement & Requirements

Design a large-scale metrics collection, storage, alerting, and visualization platform that ingests millions of time-series data points per second from infrastructure and applications, stores them efficiently, and powers real-time dashboards and alerts — similar to Datadog, Prometheus + Grafana, New Relic, or Splunk.

Functional Requirements

• Metrics ingestion — collect counters, gauges, histograms, and summaries from agents, SDKs, and push APIs
• Time-series storage — store metric data points (timestamp, value, tags) with configurable retention
• Querying — flexible query language for aggregation (sum, avg, p99, rate), grouping (by host, service, region), and time-range selection
• Dashboards — build and share real-time dashboards with charts, tables, heatmaps, and topN lists
• Alerting — define threshold, anomaly, and composite alerts; route notifications (PagerDuty, Slack, email)
• Tagging / Dimensions — every metric has tags (key:value) for flexible slicing (e.g., http.requests{service=api,status=200,region=us-east})
• Downsampling / Rollups — automatically aggregate older data to coarser granularity (1s → 1m → 1h) to save storage
• Anomaly detection — ML-based detection of unexpected patterns (spike, drop, trend shift)
• Service-level objectives (SLOs) — define and track SLIs/SLOs with error-budget burn-rate alerts

Non-Functional Requirements

• High write throughput — ingest 10M+ data points per second
• Low query latency — dashboard queries return in < 500 ms even over weeks of data
• High availability — 99.99% for ingestion (if monitoring goes down, you're flying blind)
• Durability — metrics data must not be lost once acknowledged
• Horizontal scalability — scale ingestion and storage independently
• Retention — configurable: raw data 15 days, 1-min rollups 3 months, 1-hour rollups 2 years
• Multi-tenancy — support thousands of customers with isolation and per-tenant rate limits

Out of Scope

• Log aggregation (ELK/Splunk detailed design)
• Distributed tracing (Jaeger/Zipkin)
• APM (Application Performance Monitoring) code-level profiling
• Synthetic monitoring (external probing)

Scale Estimations

Traffic

Storage

Bandwidth

Hardware Estimate

This first layer defines the system boundary. The next chapters add application logic and data dependencies. After examining the request paths and tradeoffs, we will add operational safeguards and bring the complete architecture together.

API Design

Submit Metrics (Push API)

POST /api/v1/series
Content-Type: application/json
DD-API-KEY: <api_key>

Request:
{
  "series": [
    {
      "metric": "http.requests",
      "type": "count",
      "interval": 10,
      "points": [[1712150400, 1523], [1712150410, 1487]],
      "tags": ["service:api", "status:200", "region:us-east"],
      "host": "web-server-42"
    },
    {
      "metric": "system.cpu.user",
      "type": "gauge",
      "points": [[1712150400, 72.5]],
      "tags": ["host:web-server-42", "env:production"],
      "host": "web-server-42"
    }
  ]
}

Response (202 Accepted):
{
  "status": "ok",
  "points_accepted": 3
}

Query Metrics

POST /api/v1/query
Authorization: Bearer <token>

Request:
{
  "query": "avg:http.latency{service=api,region=us-east} by {status_code}",
  "from": 1712146800,    // epoch seconds
  "to": 1712150400,      // last 1 hour
  "rollup": {
    "method": "avg",
    "interval": 60        // 1-minute buckets
  }
}

Response (200 OK):
{
  "series": [
    {
      "metric": "http.latency",
      "tags": {"service": "api", "region": "us-east", "status_code": "200"},
      "pointlist": [
        [1712146800, 23.4],
        [1712146860, 25.1],
        [1712146920, 22.8],
        ...
      ],
      "unit": "milliseconds",
      "query_index": 0
    },
    {
      "metric": "http.latency",
      "tags": {"service": "api", "region": "us-east", "status_code": "500"},
      "pointlist": [[1712146800, 187.3], ...],
      "unit": "milliseconds",
      "query_index": 0
    }
  ],
  "query_time_ms": 87
}

Create Alert

POST /api/v1/monitors
Authorization: Bearer <token>

Request:
{
  "name": "High API Error Rate",
  "type": "metric_alert",
  "query": "avg(last_5m):sum:http.errors{service=api} / sum:http.requests{service=api} > 0.05",
  "message": "@pagerduty-oncall API error rate is above 5%! Check {{host}}.",
  "tags": ["team:platform", "severity:critical"],
  "options": {
    "thresholds": {
      "critical": 0.05,
      "warning": 0.02
    },
    "notify_no_data": true,
    "no_data_timeframe": 10,
    "evaluation_delay": 60,
    "renotify_interval": 300
  }
}

Response (201 Created):
{
  "monitor_id": 12345,
  "name": "High API Error Rate",
  "status": "OK",
  "created_at": "2026-04-03T10:00:00Z"
}

Create Dashboard

POST /api/v1/dashboards
Authorization: Bearer <token>

Request:
{
  "title": "API Health Dashboard",
  "widgets": [
    {
      "type": "timeseries",
      "title": "Request Rate",
      "queries": [
        {"query": "sum:http.requests{service=api} by {status_code}.as_rate()", "display_type": "bars"}
      ],
      "time": {"live_span": "1h"}
    },
    {
      "type": "query_value",
      "title": "p99 Latency",
      "queries": [
        {"query": "p99:http.latency{service=api}", "aggregator": "last"}
      ],
      "conditional_formats": [
        {"comparator": ">", "value": 500, "palette": "red"},
        {"comparator": ">", "value": 200, "palette": "yellow"}
      ]
    },
    {
      "type": "toplist",
      "title": "Top Error Endpoints",
      "queries": [
        {"query": "sum:http.errors{service=api} by {endpoint}.as_count()"}
      ],
      "limit": 10
    }
  ]
}

Data Model

Time-Series Data Point

On-Disk Storage Layout (Time-Partitioned Blocks)

Gorilla Compression (Core Innovation)

Tag Inverted Index

Core Design Decisions

Decision 1: Push vs Pull Collection Model

Decision 2: Time-Series Storage Engine

Decision 3: Alert Evaluation Architecture

Detailed Flow Diagrams

Metrics Ingestion Flow

Dashboard Query Flow

Alert Evaluation + Notification Flow

Downsampling & Retention

Multi-Tenancy

Handling Edge Cases

High-Cardinality Tags

Problem: Metric http.requests tagged with request_id (unique per request)
  → Billions of unique series → unbounded storage growth
  → Queries become impossibly slow (scan billions of series)

Detection:
  Monitor: unique_series_count per (tenant, metric_name)
  If growth rate > 10K new series/hour → alert

Prevention:
  1. Client-side: agent rejects tags with cardinality > threshold
  2. Server-side: ingestion worker checks new series rate
     → If > 100K unique series for one metric → drop new series
     → Return warning header to agent
  3. Documentation: educate users about low-cardinality tags only
     ✅ service, host, region, status_code, endpoint
     ❌ request_id, user_id, session_id, trace_id

Late-Arriving Data Points

Problem: Agent buffers during network outage → sends 30 min of old data at once

Impact:
  • Data points arrive for already-flushed blocks
  • Alert evaluations may need recalculation

Solution:
  • Accept late data up to a configurable window (e.g., 1 hour)
  • Write to a "late-arrival" WAL segment
  • Merge into existing blocks during next compaction
  • For alerts: re-evaluate sliding windows when late data arrives
  • Reject data older than 1 hour (return 400 with "too old" error)

Clock Skew Across Agents

Problem: Different hosts have slightly different clocks
  → Data points at "same time" have different timestamps
  → Aggregation across hosts produces incorrect results

Solution:
  1. NTP synchronization on all hosts (< 100ms drift acceptable)
  2. Ingestion gateway records server-side receive timestamp
  3. If |agent_timestamp - server_timestamp| > 5 minutes:
     → Override with server timestamp
     → Log warning to agent
  4. Queries tolerate minor drift via configurable alignment window

Alert Flapping

Problem: Metric oscillates around threshold → alert fires/recovers every minute
  → Oncall engineer gets 30 notifications in 30 minutes

Solution: Hysteresis + evaluation window

  Alert: http.error_rate > 5% for 5 minutes
  Recovery: http.error_rate < 3% for 5 minutes   ← different threshold!

  The gap (3% to 5%) is the hysteresis band.
  Must go BELOW 3% to recover, not just below 5%.

  Additional measures:
  • Renotify interval: minimum 5 minutes between notifications
  • Auto-mute if alert fires > 10 times in 1 hour → likely flapping
  • Dashboard flag: "This alert is flapping — consider adjusting threshold"

Tradeoffs & Design Decisions Summary

Reliability & Fault Tolerance

Single Points of Failure & Mitigations

Graceful Degradation

Tier 1 (single TSDB node down):
  → Queries return partial results (other nodes still respond)
  → Warning banner: "Some data may be incomplete"
  → Writes to that shard go to replica

Tier 2 (Kafka down):
  → Agents buffer locally (1 hour of data in memory/disk)
  → Alert evaluations continue on existing recent data
  → When Kafka recovers: agents flush buffered data (late arrival)

Tier 3 (Tag index down):
  → Cannot resolve tag queries (which series match "service=api"?)
  → Fallback: if query specifies exact series_id → still works
  → Dashboard shows "Tag resolution degraded"
  → Rebuild index from TSDB metadata (~minutes)

Tier 4 (Alert engine down):
  → NO ALERTS FIRING — this is the worst failure mode
  → Must be highest priority to recover
  → Mitigation: dead-man's switch alert
    → External service expects heartbeat every 5 min
    → If no heartbeat → alert via independent channel
    → "Alert engine is down" — monitors the monitors

High-Level Architecture

Component Breakdown

Full System Architecture (Production-Grade)

SLO Monitoring (Bonus Deep Dive)

Interview Tips

1. Start with the write path — 10M points/sec — "Time-series is a write-dominated workload." Explain: agents batch + compress → push to ingestion gateway → Kafka buffer → ingestion workers → TSDB with WAL. Kafka decouples collection from storage and absorbs bursts.

2. Gorilla compression is the key insight — Don't just say "compress data." Explain: delta-of-delta for timestamps (1 bit for regular intervals), XOR for values (encode only changed bits). 1.37 bytes/point average = 11.7× compression. This is why a monitoring system storing 864B points/day needs only ~26 TB.

3. Time-partitioned blocks — 2-hour immutable blocks. Active block in memory (WAL for durability). Old blocks compressed, indexed, read-optimized on SSD. Deletion = drop a block file (instant, no GC). This is the Prometheus TSDB model.

4. Tag inverted index with Roaring Bitmaps — Query service=api AND region=us-east resolves to: bitmap A ∩ bitmap B = matching series IDs. Roaring Bitmaps are compact (~200 GB for 1B series) and support fast set operations. This is how tag-based queries stay fast.

5. Rollup/downsampling is critical for long-term queries — Raw 10s data retained 15 days, 1-min rollups 3 months, 1-hour rollups 2 years. Query engine auto-selects the right tier based on time range. Transparent to the user. This is how a dashboard over "last 6 months" returns in < 500 ms.

6. Alert evaluation is streaming, not polling — 5M rules evaluated every 60s = 83K evals/sec. If each queries TSDB = overload. Instead: as data points arrive, route to matching alert rules (inverted index: metric+tags → rule_ids). Most data matches zero rules. Only evaluate relevant rules.

7. Multi-tenancy is about isolation — Per-tenant rate limits, series quotas, query timeouts. Cardinality explosion protection: if a tenant's metric has > 100K unique tag combinations, reject new series. One bad tenant must not affect others.

8. High-cardinality tags are the #1 operational issue — request_id, user_id, trace_id as tags = billions of unique series = storage explosion + slow queries. Detect + prevent at ingestion. This is the most common real-world problem in monitoring systems.

9. Meta-monitoring — who monitors the monitors? — Dead-man's switch: external service expects heartbeat every 5 min. If missing → alert via independent channel (SMS, not the monitoring system itself). If your monitoring is down, you're flying blind.

10. SLO burn-rate alerting shows depth — Don't just describe threshold alerts. Explain: SLI as a time-series, error budget calculation, burn rate = actual/budgeted error rate, multi-window alerting (fast burn pages, slow burn creates ticket). This is the Google SRE Book approach and demonstrates production maturity.