{"type":"excalidraw","version":2,"source":"https://AwesomeSystemDesign.dev","elements":[{"angle":0,"strokeColor":"#6577aa","backgroundColor":"#eef1fc","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-1"],"frameId":null,"roundness":null,"boundElements":[{"type":"text","id":"shape-1"}],"updated":0,"link":null,"locked":false,"id":"shape-0","type":"rectangle","seed":41,"x":16,"y":16,"width":288,"height":1351.6000000000001,"customData":{"revealStep":1}},{"angle":0,"strokeColor":"#394557","backgroundColor":"transparent","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-1"],"frameId":null,"roundness":null,"boundElements":null,"updated":0,"link":null,"locked":false,"id":"shape-1","type":"text","seed":114,"x":36,"y":34,"width":248,"height":1315.6000000000001,"fontSize":22,"fontFamily":1,"text":"INGESTION PLANE\n\n• Collection Agents\nPer-host agent\n(DaemonSet in k8s):\n• System metrics:\nCPU, memory, disk\nI/O, network\n• Container metrics:\ncgroups stats per\ncontainer\n• Integration metrics:\nMySQL, Redis, Kafka,\nNginx, etc.\n• Custom app\nmetrics: StatsD,\nDogStatsD,\nOpenTelemetry\nClient-side\naggregation:\n• Counters:\naggregate 10s of\nincrements → one\ndata point\n• Histograms: compute\np50/p95/p99 locally,\nsend summaries\n→ Reduces network by\n100-1000× vs sending\nevery event\n\n• Ingestion Gateway\n• Authenticate (API\nkey → tenant_id)\n• Validate metric\nformat\n• Per-tenant rate\nlimiting (burst: 100K\npts/s, sustained:\n50K)\n• Produce to Kafka\nwith tenant partition\nkey\n• Return 202\nAccepted (async\nprocessing)","originalText":"INGESTION PLANE\n\n• Collection Agents\nPer-host agent\n(DaemonSet in k8s):\n• System metrics:\nCPU, memory, disk\nI/O, network\n• Container metrics:\ncgroups stats per\ncontainer\n• Integration metrics:\nMySQL, Redis, Kafka,\nNginx, etc.\n• Custom app\nmetrics: StatsD,\nDogStatsD,\nOpenTelemetry\nClient-side\naggregation:\n• Counters:\naggregate 10s of\nincrements → one\ndata point\n• Histograms: compute\np50/p95/p99 locally,\nsend summaries\n→ Reduces network by\n100-1000× vs sending\nevery event\n\n• Ingestion Gateway\n• Authenticate (API\nkey → tenant_id)\n• Validate metric\nformat\n• Per-tenant rate\nlimiting (burst: 100K\npts/s, sustained:\n50K)\n• Produce to Kafka\nwith tenant partition\nkey\n• Return 202\nAccepted (async\nprocessing)","textAlign":"left","verticalAlign":"middle","containerId":"shape-0","lineHeight":1.3,"autoResize":false,"customData":{"revealStep":1}},{"angle":0,"strokeColor":"#6577aa","backgroundColor":"#eef1fc","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-2"],"frameId":null,"roundness":null,"boundElements":[{"type":"text","id":"shape-3"}],"updated":0,"link":null,"locked":false,"id":"shape-2","type":"rectangle","seed":187,"x":16,"y":1411.6000000000001,"width":288,"height":1837.8,"customData":{"revealStep":2}},{"angle":0,"strokeColor":"#394557","backgroundColor":"transparent","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-2"],"frameId":null,"roundness":null,"boundElements":null,"updated":0,"link":null,"locked":false,"id":"shape-3","type":"text","seed":260,"x":36,"y":1429.6000000000001,"width":248,"height":1801.8,"fontSize":22,"fontFamily":1,"text":"STORAGE PLANE\n\n• Time-Series\nDatabase (TSDB\nCore)\nData model:\nseries_id =\nhash(metric_name +\nsorted_tags) e.g.,\nhash(\"http.requests{\nservice=api,status=2\n00}\")\nStorage layout:\nPartitioned by TIME\n(e.g., 2-hour blocks)\nWithin each block:\nseries grouped\ntogether Gorilla\ncompression for\ntimestamps + values\nWrite path: append\nto current active\nblock (in-memory WAL)\nRead path: identify\nrelevant time blocks\n→ scan series\n\n• Tag Index\n(Inverted Index)\ntag_key=tag_value →\nset of series_ids\ne.g., service=api →\n{series_1, series_2,\n...series_50K}\nSupports:\n• Exact match:\nservice=api\n• Wildcard:\nhost=web-*\n• Negation:\nstatus!=500\n• Set intersection\nfor multi-tag queries\nStorage: Roaring\nBitmaps for compact\nset operations\n\n• Rollup /\nDownsampling Engine\nRaw (10s intervals) →\nretained 15 days\n1-min rollups (avg, min,\nmax, sum, count) → 3\nmonths 1-hour rollups\n→ 2 years 1-day\nrollups → indefinite\nBackground job:\nreads raw blocks\npast retention\nwindow → Computes\naggregates → writes\nto rollup tier →\ndeletes raw","originalText":"STORAGE PLANE\n\n• Time-Series\nDatabase (TSDB\nCore)\nData model:\nseries_id =\nhash(metric_name +\nsorted_tags) e.g.,\nhash(\"http.requests{\nservice=api,status=2\n00}\")\nStorage layout:\nPartitioned by TIME\n(e.g., 2-hour blocks)\nWithin each block:\nseries grouped\ntogether Gorilla\ncompression for\ntimestamps + values\nWrite path: append\nto current active\nblock (in-memory WAL)\nRead path: identify\nrelevant time blocks\n→ scan series\n\n• Tag Index\n(Inverted Index)\ntag_key=tag_value →\nset of series_ids\ne.g., service=api →\n{series_1, series_2,\n...series_50K}\nSupports:\n• Exact match:\nservice=api\n• Wildcard:\nhost=web-*\n• Negation:\nstatus!=500\n• Set intersection\nfor multi-tag queries\nStorage: Roaring\nBitmaps for compact\nset operations\n\n• Rollup /\nDownsampling Engine\nRaw (10s intervals) →\nretained 15 days\n1-min rollups (avg, min,\nmax, sum, count) → 3\nmonths 1-hour rollups\n→ 2 years 1-day\nrollups → indefinite\nBackground job:\nreads raw blocks\npast retention\nwindow → Computes\naggregates → writes\nto rollup tier →\ndeletes raw","textAlign":"left","verticalAlign":"middle","containerId":"shape-2","lineHeight":1.3,"autoResize":false,"customData":{"revealStep":2}},{"angle":0,"strokeColor":"#6577aa","backgroundColor":"#eef1fc","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-3"],"frameId":null,"roundness":null,"boundElements":[{"type":"text","id":"shape-5"}],"updated":0,"link":null,"locked":false,"id":"shape-4","type":"rectangle","seed":333,"x":16,"y":3293.4,"width":288,"height":1609,"customData":{"revealStep":3}},{"angle":0,"strokeColor":"#394557","backgroundColor":"transparent","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-3"],"frameId":null,"roundness":null,"boundElements":null,"updated":0,"link":null,"locked":false,"id":"shape-5","type":"text","seed":406,"x":36,"y":3311.4,"width":248,"height":1573,"fontSize":22,"fontFamily":1,"text":"QUERY & ALERTING\nPLANE\n\n• Query Engine\nQuery:\navg:http.latency{serv\nice=api,region=us-eas\nt} .rollup(avg, 60)\n.last(\"1h\")\nExecution:\n1. Parse query\n2. Resolve tags →\nseries_ids (inverted\nindex)\n3. Select storage\ntier (raw? 1-min\nrollup? 1-hour?)\nbased on time range\nrequested\n4. Scatter: fan out\nto storage nodes\nholding those series\n5. Gather: aggregate\npartial results\n6. Return time-series\nresult set\n\n• Alert Engine\n5M alert rules,\nevaluated every\n30-60 seconds\nTypes:\n• Threshold:\nhttp.error_rate >\n5% for 5 min\n• Anomaly: CPU\ndeviates 3σ from\nseasonal baseline\n• Change: error rate\nincreased > 50% vs\nlast week\n• Composite: alert A\nAND alert B both\nfiring\n• SLO: error-budget\nburn rate > 10× in 5\nmin\nPipeline: Rule → Query\nTSDB → Evaluate\ncondition → State\ntransition OK → WARN\n→ ALERT →\nRECOVERED On\ntransition: notify via\nconfigured channels","originalText":"QUERY & ALERTING\nPLANE\n\n• Query Engine\nQuery:\navg:http.latency{serv\nice=api,region=us-eas\nt} .rollup(avg, 60)\n.last(\"1h\")\nExecution:\n1. Parse query\n2. Resolve tags →\nseries_ids (inverted\nindex)\n3. Select storage\ntier (raw? 1-min\nrollup? 1-hour?)\nbased on time range\nrequested\n4. Scatter: fan out\nto storage nodes\nholding those series\n5. Gather: aggregate\npartial results\n6. Return time-series\nresult set\n\n• Alert Engine\n5M alert rules,\nevaluated every\n30-60 seconds\nTypes:\n• Threshold:\nhttp.error_rate >\n5% for 5 min\n• Anomaly: CPU\ndeviates 3σ from\nseasonal baseline\n• Change: error rate\nincreased > 50% vs\nlast week\n• Composite: alert A\nAND alert B both\nfiring\n• SLO: error-budget\nburn rate > 10× in 5\nmin\nPipeline: Rule → Query\nTSDB → Evaluate\ncondition → State\ntransition OK → WARN\n→ ALERT →\nRECOVERED On\ntransition: notify via\nconfigured channels","textAlign":"left","verticalAlign":"middle","containerId":"shape-4","lineHeight":1.3,"autoResize":false,"customData":{"revealStep":3}}],"appState":{"viewBackgroundColor":"#fbfbf8","gridSize":null},"files":{}}