{"type":"excalidraw","version":2,"source":"https://AwesomeSystemDesign.dev","elements":[{"angle":0,"strokeColor":"#6577aa","backgroundColor":"#eef1fc","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-1"],"frameId":null,"roundness":null,"boundElements":[{"type":"text","id":"shape-1"}],"updated":0,"link":null,"locked":false,"id":"shape-0","type":"rectangle","seed":41,"x":466,"y":32,"width":390,"height":153,"customData":{"revealStep":1}},{"angle":0,"strokeColor":"#394557","backgroundColor":"transparent","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-1"],"frameId":null,"roundness":null,"boundElements":null,"updated":0,"link":null,"locked":false,"id":"shape-1","type":"text","seed":114,"x":486,"y":50,"width":350,"height":117,"fontSize":18,"fontFamily":1,"text":"✅ Managed, infinite scale\n✅ Cheap for cold storage\n❌ Higher latency than HDFS\n❌ Per-request cost adds up at\nscale","originalText":"✅ Managed, infinite scale\n✅ Cheap for cold storage\n❌ Higher latency than HDFS\n❌ Per-request cost adds up at\nscale","textAlign":"center","verticalAlign":"middle","containerId":"shape-0","lineHeight":1.3,"autoResize":false,"customData":{"revealStep":1}},{"angle":0,"strokeColor":"#628572","backgroundColor":"#edf5ee","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-2"],"frameId":null,"roundness":null,"boundElements":[{"type":"text","id":"shape-3"}],"updated":0,"link":null,"locked":false,"id":"shape-2","type":"rectangle","seed":187,"x":36,"y":32,"width":390,"height":293.40000000000003,"customData":{"revealStep":2}},{"angle":0,"strokeColor":"#394557","backgroundColor":"transparent","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-2"],"frameId":null,"roundness":null,"boundElements":null,"updated":0,"link":null,"locked":false,"id":"shape-3","type":"text","seed":260,"x":56,"y":50,"width":350,"height":257.40000000000003,"fontSize":18,"fontFamily":1,"text":"✅ Designed for large sequential\nwrites\n✅ Built-in replication (3x)\n✅ Integrates with MapReduce for\nindexing\n❌ Not great for random reads\n❌ Small file problem (many small\npages)\n\nMitigation: Bundle pages into large\nWARC files (Web ARChive format)","originalText":"✅ Designed for large sequential\nwrites\n✅ Built-in replication (3x)\n✅ Integrates with MapReduce for\nindexing\n❌ Not great for random reads\n❌ Small file problem (many small\npages)\n\nMitigation: Bundle pages into large\nWARC files (Web ARChive format)","textAlign":"center","verticalAlign":"middle","containerId":"shape-2","lineHeight":1.3,"autoResize":false,"customData":{"revealStep":2}},{"angle":0,"strokeColor":"#a68a53","backgroundColor":"#fff7e6","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-3"],"frameId":null,"roundness":null,"boundElements":[{"type":"text","id":"shape-5"}],"updated":0,"link":null,"locked":false,"id":"shape-4","type":"rectangle","seed":333,"x":36,"y":365.40000000000003,"width":420,"height":632.7,"customData":{"revealStep":3}},{"angle":0,"strokeColor":"#394557","backgroundColor":"transparent","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-3"],"frameId":null,"roundness":null,"boundElements":null,"updated":0,"link":null,"locked":false,"id":"shape-5","type":"text","seed":406,"x":56,"y":383.40000000000003,"width":380,"height":596.7,"fontSize":17,"fontFamily":1,"text":"Content Storage Layer\n\nOption 1: Distributed File System (HDFS)\n\nOption 2: Object Storage (S3)\n\nOur Design: Hybrid\n\nRecent crawl data → HDFS (hot storage)\n- Fast access for indexing pipeline\n- Retained for 30 days\n\nArchive data → S3 (cold storage)\n- Compressed WARC files\n- Retained indefinitely\n- Used for historical analysis\n\nStorage format per page:\n\nurl: \"https://...\",\nfetched_at: \"2026-04-03T...\",\nhttp_status: 200,\nheaders: { ... },\ncontent_type: \"text/html\",\nbody: \"<html>...\",\nbody_hash: \"sha256:abc...\",\ncontent_length: 45230","originalText":"Content Storage Layer\n\nOption 1: Distributed File System (HDFS)\n\nOption 2: Object Storage (S3)\n\nOur Design: Hybrid\n\nRecent crawl data → HDFS (hot storage)\n- Fast access for indexing pipeline\n- Retained for 30 days\n\nArchive data → S3 (cold storage)\n- Compressed WARC files\n- Retained indefinitely\n- Used for historical analysis\n\nStorage format per page:\n\nurl: \"https://...\",\nfetched_at: \"2026-04-03T...\",\nhttp_status: 200,\nheaders: { ... },\ncontent_type: \"text/html\",\nbody: \"<html>...\",\nbody_hash: \"sha256:abc...\",\ncontent_length: 45230","textAlign":"left","verticalAlign":"middle","containerId":"shape-4","lineHeight":1.3,"autoResize":false,"customData":{"revealStep":3}}],"appState":{"viewBackgroundColor":"#fbfbf8","gridSize":null},"files":{}}