{"type":"excalidraw","version":2,"source":"https://AwesomeSystemDesign.dev","elements":[{"angle":0,"strokeColor":"#6577aa","backgroundColor":"#eef1fc","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-1"],"frameId":null,"roundness":null,"boundElements":[{"type":"text","id":"shape-1"}],"updated":0,"link":null,"locked":false,"id":"shape-0","type":"rectangle","seed":41,"x":36,"y":32,"width":390,"height":738,"customData":{"revealStep":1}},{"angle":0,"strokeColor":"#394557","backgroundColor":"transparent","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-1"],"frameId":null,"roundness":null,"boundElements":null,"updated":0,"link":null,"locked":false,"id":"shape-1","type":"text","seed":114,"x":56,"y":50,"width":350,"height":702,"fontSize":18,"fontFamily":1,"text":"INGESTION PLANE\n\n• Collection Agents\nPer-host agent (DaemonSet in k8s):\n• System metrics: CPU, memory, disk\nI/O, network\n• Container metrics: cgroups stats\nper container\n• Integration metrics: MySQL, Redis,\nKafka, Nginx, etc.\n• Custom app metrics: StatsD,\nDogStatsD, OpenTelemetry\nClient-side aggregation:\n• Counters: aggregate 10s of\nincrements → one data point\n• Histograms: compute p50/p95/p99\nlocally, send summaries\n→ Reduces network by 100-1000× vs\nsending every event\n\n• Ingestion Gateway\n• Authenticate (API key →\ntenant_id)\n• Validate metric format\n• Per-tenant rate limiting (burst:\n100K pts/s, sustained: 50K)\n• Produce to Kafka with tenant\npartition key\n• Return 202 Accepted (async\nprocessing)","originalText":"INGESTION PLANE\n\n• Collection Agents\nPer-host agent (DaemonSet in k8s):\n• System metrics: CPU, memory, disk\nI/O, network\n• Container metrics: cgroups stats\nper container\n• Integration metrics: MySQL, Redis,\nKafka, Nginx, etc.\n• Custom app metrics: StatsD,\nDogStatsD, OpenTelemetry\nClient-side aggregation:\n• Counters: aggregate 10s of\nincrements → one data point\n• Histograms: compute p50/p95/p99\nlocally, send summaries\n→ Reduces network by 100-1000× vs\nsending every event\n\n• Ingestion Gateway\n• Authenticate (API key →\ntenant_id)\n• Validate metric format\n• Per-tenant rate limiting (burst:\n100K pts/s, sustained: 50K)\n• Produce to Kafka with tenant\npartition key\n• Return 202 Accepted (async\nprocessing)","textAlign":"center","verticalAlign":"middle","containerId":"shape-0","lineHeight":1.3,"autoResize":false,"customData":{"revealStep":1}},{"angle":0,"strokeColor":"#628572","backgroundColor":"#edf5ee","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-2"],"frameId":null,"roundness":null,"boundElements":[{"type":"text","id":"shape-3"}],"updated":0,"link":null,"locked":false,"id":"shape-2","type":"rectangle","seed":187,"x":466,"y":32.00000000000006,"width":390,"height":995.4,"customData":{"revealStep":2}},{"angle":0,"strokeColor":"#394557","backgroundColor":"transparent","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-2"],"frameId":null,"roundness":null,"boundElements":null,"updated":0,"link":null,"locked":false,"id":"shape-3","type":"text","seed":260,"x":486,"y":50.00000000000006,"width":350,"height":959.4,"fontSize":18,"fontFamily":1,"text":"STORAGE PLANE\n\n• Time-Series Database (TSDB\nCore)\nData model: series_id =\nhash(metric_name + sorted_tags)\ne.g.,\nhash(\"http.requests{service=api,stat\nus=200}\")\nStorage layout: Partitioned by\nTIME (e.g., 2-hour blocks) Within\neach block: series grouped together\nGorilla compression for timestamps +\nvalues\nWrite path: append to current\nactive block (in-memory WAL) Read\npath: identify relevant time blocks →\nscan series\n\n• Tag Index (Inverted Index)\ntag_key=tag_value → set of\nseries_ids e.g., service=api →\n{series_1, series_2, ...series_50K}\nSupports:\n• Exact match: service=api\n• Wildcard: host=web-*\n• Negation: status!=500\n• Set intersection for multi-tag\nqueries\nStorage: Roaring Bitmaps for\ncompact set operations\n\n• Rollup / Downsampling Engine\nRaw (10s intervals) → retained 15\ndays 1-min rollups (avg, min, max,\nsum, count) → 3 months 1-hour rollups\n→ 2 years 1-day rollups → indefinite\nBackground job: reads raw blocks\npast retention window → Computes\naggregates → writes to rollup tier →\ndeletes raw","originalText":"STORAGE PLANE\n\n• Time-Series Database (TSDB\nCore)\nData model: series_id =\nhash(metric_name + sorted_tags)\ne.g.,\nhash(\"http.requests{service=api,stat\nus=200}\")\nStorage layout: Partitioned by\nTIME (e.g., 2-hour blocks) Within\neach block: series grouped together\nGorilla compression for timestamps +\nvalues\nWrite path: append to current\nactive block (in-memory WAL) Read\npath: identify relevant time blocks →\nscan series\n\n• Tag Index (Inverted Index)\ntag_key=tag_value → set of\nseries_ids e.g., service=api →\n{series_1, series_2, ...series_50K}\nSupports:\n• Exact match: service=api\n• Wildcard: host=web-*\n• Negation: status!=500\n• Set intersection for multi-tag\nqueries\nStorage: Roaring Bitmaps for\ncompact set operations\n\n• Rollup / Downsampling Engine\nRaw (10s intervals) → retained 15\ndays 1-min rollups (avg, min, max,\nsum, count) → 3 months 1-hour rollups\n→ 2 years 1-day rollups → indefinite\nBackground job: reads raw blocks\npast retention window → Computes\naggregates → writes to rollup tier →\ndeletes raw","textAlign":"center","verticalAlign":"middle","containerId":"shape-2","lineHeight":1.3,"autoResize":false,"customData":{"revealStep":2}},{"angle":0,"strokeColor":"#a68a53","backgroundColor":"#fff7e6","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-3"],"frameId":null,"roundness":null,"boundElements":[{"type":"text","id":"shape-5"}],"updated":0,"link":null,"locked":false,"id":"shape-4","type":"rectangle","seed":333,"x":36,"y":1067.4,"width":390,"height":901.8000000000001,"customData":{"revealStep":3}},{"angle":0,"strokeColor":"#394557","backgroundColor":"transparent","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-3"],"frameId":null,"roundness":null,"boundElements":null,"updated":0,"link":null,"locked":false,"id":"shape-5","type":"text","seed":406,"x":56,"y":1085.4,"width":350,"height":865.8000000000001,"fontSize":18,"fontFamily":1,"text":"QUERY & ALERTING PLANE\n\n• Query Engine\nQuery:\navg:http.latency{service=api,region=u\ns-east} .rollup(avg, 60) .last(\"1h\")\nExecution:\n1. Parse query\n2. Resolve tags → series_ids\n(inverted index)\n3. Select storage tier (raw? 1-min\nrollup? 1-hour?)\nbased on time range requested\n4. Scatter: fan out to storage\nnodes holding those series\n5. Gather: aggregate partial results\n6. Return time-series result set\n\n• Alert Engine\n5M alert rules, evaluated every\n30-60 seconds\nTypes:\n• Threshold: http.error_rate > 5%\nfor 5 min\n• Anomaly: CPU deviates 3σ from\nseasonal baseline\n• Change: error rate increased >\n50% vs last week\n• Composite: alert A AND alert B\nboth firing\n• SLO: error-budget burn rate > 10×\nin 5 min\nPipeline: Rule → Query TSDB →\nEvaluate condition → State\ntransition OK → WARN → ALERT →\nRECOVERED On transition: notify\nvia configured channels","originalText":"QUERY & ALERTING PLANE\n\n• Query Engine\nQuery:\navg:http.latency{service=api,region=u\ns-east} .rollup(avg, 60) .last(\"1h\")\nExecution:\n1. Parse query\n2. Resolve tags → series_ids\n(inverted index)\n3. Select storage tier (raw? 1-min\nrollup? 1-hour?)\nbased on time range requested\n4. Scatter: fan out to storage\nnodes holding those series\n5. Gather: aggregate partial results\n6. Return time-series result set\n\n• Alert Engine\n5M alert rules, evaluated every\n30-60 seconds\nTypes:\n• Threshold: http.error_rate > 5%\nfor 5 min\n• Anomaly: CPU deviates 3σ from\nseasonal baseline\n• Change: error rate increased >\n50% vs last week\n• Composite: alert A AND alert B\nboth firing\n• SLO: error-budget burn rate > 10×\nin 5 min\nPipeline: Rule → Query TSDB →\nEvaluate condition → State\ntransition OK → WARN → ALERT →\nRECOVERED On transition: notify\nvia configured channels","textAlign":"center","verticalAlign":"middle","containerId":"shape-4","lineHeight":1.3,"autoResize":false,"customData":{"revealStep":3}}],"appState":{"viewBackgroundColor":"#fbfbf8","gridSize":null},"files":{}}