{"type":"excalidraw","version":2,"source":"https://AwesomeSystemDesign.dev","elements":[{"angle":0,"strokeColor":"#6577aa","backgroundColor":"#eef1fc","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-1"],"frameId":null,"roundness":null,"boundElements":[{"type":"text","id":"shape-1"}],"updated":0,"link":null,"locked":false,"id":"shape-0","type":"rectangle","seed":41,"x":16,"y":16,"width":288,"height":322,"customData":{"revealStep":1}},{"angle":0,"strokeColor":"#394557","backgroundColor":"transparent","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-1"],"frameId":null,"roundness":null,"boundElements":null,"updated":0,"link":null,"locked":false,"id":"shape-1","type":"text","seed":114,"x":36,"y":34,"width":248,"height":286,"fontSize":22,"fontFamily":1,"text":"For each new post:\n- Extract hashtags:\n#WorldCup, #AI\n- Extract named\nentities: \"Elon Musk\",\n\"Tesla\"\n- Extract URLs\n(trending links)\n- Detect language\nand region","originalText":"For each new post:\n- Extract hashtags:\n#WorldCup, #AI\n- Extract named\nentities: \"Elon Musk\",\n\"Tesla\"\n- Extract URLs\n(trending links)\n- Detect language\nand region","textAlign":"left","verticalAlign":"middle","containerId":"shape-0","lineHeight":1.3,"autoResize":false,"customData":{"revealStep":1}},{"angle":0,"strokeColor":"#6577aa","backgroundColor":"#eef1fc","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-2"],"frameId":null,"roundness":null,"boundElements":[{"type":"text","id":"shape-3"}],"updated":0,"link":null,"locked":false,"id":"shape-2","type":"rectangle","seed":187,"x":16,"y":382,"width":288,"height":436.40000000000003,"customData":{"revealStep":2}},{"angle":0,"strokeColor":"#394557","backgroundColor":"transparent","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-2"],"frameId":null,"roundness":null,"boundElements":null,"updated":0,"link":null,"locked":false,"id":"shape-3","type":"text","seed":260,"x":36,"y":400,"width":248,"height":400.40000000000003,"fontSize":22,"fontFamily":1,"text":"For each entity,\nmaintain:\n- count_last_5min\n- count_last_1hour\n-\ncount_last_24hours\n(baseline)\n\nImplementation: Redis\nHyperLogLog or\nCount-Min Sketch\n(approximate\ncounting,\nmemory-efficient)","originalText":"For each entity,\nmaintain:\n- count_last_5min\n- count_last_1hour\n-\ncount_last_24hours\n(baseline)\n\nImplementation: Redis\nHyperLogLog or\nCount-Min Sketch\n(approximate\ncounting,\nmemory-efficient)","textAlign":"left","verticalAlign":"middle","containerId":"shape-2","lineHeight":1.3,"autoResize":false,"customData":{"revealStep":2}},{"angle":0,"strokeColor":"#a68a53","backgroundColor":"#fff7e6","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-3"],"frameId":null,"roundness":null,"boundElements":[{"type":"text","id":"shape-5"}],"updated":0,"link":null,"locked":false,"id":"shape-4","type":"rectangle","seed":333,"x":16,"y":862.4000000000001,"width":288,"height":2266.8,"customData":{"revealStep":3}},{"angle":0,"strokeColor":"#394557","backgroundColor":"transparent","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-3"],"frameId":null,"roundness":null,"boundElements":null,"updated":0,"link":null,"locked":false,"id":"shape-5","type":"text","seed":406,"x":36,"y":880.4000000000001,"width":248,"height":2230.8,"fontSize":22,"fontFamily":1,"text":"Trending Topics\nPipeline\n\nGoal: Identify topics\nthat are spiking in\ndiscussion\nRIGHT NOW\ncompared to their\nbaseline.\n\nPipeline:\n\nAll Posts --> Kafka\n--> Storm/Flink -->\nTrending Cache\n(stream proc) (Redis)\n\nStep 1: Extract\nentities from posts\n(real-time)\n\nStep 2: Count in\nsliding windows\n\nStep 3: Compute\ntrend score\n\ntrend_score =\ncount_5min /\navg_count_5min_bas\neline\n\n\"World Cup\" normally\ngets 100 mentions\nper 5 min.\nRight now it's getting\n10,000 per 5 min.\ntrend_score =\n10,000 / 100 = 100x\n(very trending)\n\n\"Elon Musk\" normally\ngets 500 per 5 min.\nRight now it's getting\n1,000 per 5 min.\ntrend_score = 1,000\n/ 500 = 2x (mildly\ntrending)\n\nKey insight: Trending\nmeans RELATIVE\nspike, not\nabsolute volume.\n\"Weather\" always has\nhigh volume\nbut isn't trending\nunless there's a\nspike.\n\nStep 4: Rank and\nserve\n\n- Sort entities by\ntrend_score\n- Filter by\nregion/language\n- Deduplicate (merge\n\"World Cup\" and\n\"#WorldCup\")\n- Content policy\nfilter (remove\noffensive topics)\n- Cache top 30\ntrending per region in\nRedis\n- TTL: 5 minutes\n(refreshed by stream\nprocessor)","originalText":"Trending Topics\nPipeline\n\nGoal: Identify topics\nthat are spiking in\ndiscussion\nRIGHT NOW\ncompared to their\nbaseline.\n\nPipeline:\n\nAll Posts --> Kafka\n--> Storm/Flink -->\nTrending Cache\n(stream proc) (Redis)\n\nStep 1: Extract\nentities from posts\n(real-time)\n\nStep 2: Count in\nsliding windows\n\nStep 3: Compute\ntrend score\n\ntrend_score =\ncount_5min /\navg_count_5min_bas\neline\n\n\"World Cup\" normally\ngets 100 mentions\nper 5 min.\nRight now it's getting\n10,000 per 5 min.\ntrend_score =\n10,000 / 100 = 100x\n(very trending)\n\n\"Elon Musk\" normally\ngets 500 per 5 min.\nRight now it's getting\n1,000 per 5 min.\ntrend_score = 1,000\n/ 500 = 2x (mildly\ntrending)\n\nKey insight: Trending\nmeans RELATIVE\nspike, not\nabsolute volume.\n\"Weather\" always has\nhigh volume\nbut isn't trending\nunless there's a\nspike.\n\nStep 4: Rank and\nserve\n\n- Sort entities by\ntrend_score\n- Filter by\nregion/language\n- Deduplicate (merge\n\"World Cup\" and\n\"#WorldCup\")\n- Content policy\nfilter (remove\noffensive topics)\n- Cache top 30\ntrending per region in\nRedis\n- TTL: 5 minutes\n(refreshed by stream\nprocessor)","textAlign":"left","verticalAlign":"middle","containerId":"shape-4","lineHeight":1.3,"autoResize":false,"customData":{"revealStep":3}}],"appState":{"viewBackgroundColor":"#fbfbf8","gridSize":null},"files":{}}