{"type":"excalidraw","version":2,"source":"https://AwesomeSystemDesign.dev","elements":[{"angle":0,"strokeColor":"#6577aa","backgroundColor":"#eef1fc","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-1"],"frameId":null,"roundness":null,"boundElements":[{"type":"text","id":"shape-1"}],"updated":0,"link":null,"locked":false,"id":"shape-0","type":"rectangle","seed":41,"x":16,"y":16,"width":288,"height":264.8,"customData":{"revealStep":1}},{"angle":0,"strokeColor":"#394557","backgroundColor":"transparent","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-1"],"frameId":null,"roundness":null,"boundElements":null,"updated":0,"link":null,"locked":false,"id":"shape-1","type":"text","seed":114,"x":36,"y":34,"width":248,"height":228.8,"fontSize":22,"fontFamily":1,"text":"✅ Managed, infinite\nscale\n✅ Cheap for cold\nstorage\n❌ Higher latency than\nHDFS\n❌ Per-request cost\nadds up at scale","originalText":"✅ Managed, infinite\nscale\n✅ Cheap for cold\nstorage\n❌ Higher latency than\nHDFS\n❌ Per-request cost\nadds up at scale","textAlign":"left","verticalAlign":"middle","containerId":"shape-0","lineHeight":1.3,"autoResize":false,"customData":{"revealStep":1}},{"angle":0,"strokeColor":"#6577aa","backgroundColor":"#eef1fc","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-2"],"frameId":null,"roundness":null,"boundElements":[{"type":"text","id":"shape-3"}],"updated":0,"link":null,"locked":false,"id":"shape-2","type":"rectangle","seed":187,"x":16,"y":324.8,"width":288,"height":493.6,"customData":{"revealStep":2}},{"angle":0,"strokeColor":"#394557","backgroundColor":"transparent","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-2"],"frameId":null,"roundness":null,"boundElements":null,"updated":0,"link":null,"locked":false,"id":"shape-3","type":"text","seed":260,"x":36,"y":342.8,"width":248,"height":457.6,"fontSize":22,"fontFamily":1,"text":"✅ Designed for large\nsequential writes\n✅ Built-in replication\n(3x)\n✅ Integrates with\nMapReduce for\nindexing\n❌ Not great for\nrandom reads\n❌ Small file problem\n(many small pages)\n\nMitigation: Bundle\npages into large\nWARC files (Web\nARChive format)","originalText":"✅ Designed for large\nsequential writes\n✅ Built-in replication\n(3x)\n✅ Integrates with\nMapReduce for\nindexing\n❌ Not great for\nrandom reads\n❌ Small file problem\n(many small pages)\n\nMitigation: Bundle\npages into large\nWARC files (Web\nARChive format)","textAlign":"left","verticalAlign":"middle","containerId":"shape-2","lineHeight":1.3,"autoResize":false,"customData":{"revealStep":2}},{"angle":0,"strokeColor":"#a68a53","backgroundColor":"#fff7e6","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-3"],"frameId":null,"roundness":null,"boundElements":[{"type":"text","id":"shape-5"}],"updated":0,"link":null,"locked":false,"id":"shape-4","type":"rectangle","seed":333,"x":16,"y":862.4000000000001,"width":288,"height":1237.2,"customData":{"revealStep":3}},{"angle":0,"strokeColor":"#394557","backgroundColor":"transparent","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-3"],"frameId":null,"roundness":null,"boundElements":null,"updated":0,"link":null,"locked":false,"id":"shape-5","type":"text","seed":406,"x":36,"y":880.4000000000001,"width":248,"height":1201.2,"fontSize":22,"fontFamily":1,"text":"Content Storage\nLayer\n\nOption 1: Distributed\nFile System (HDFS)\n\nOption 2: Object\nStorage (S3)\n\nOur Design: Hybrid\n\nRecent crawl data →\nHDFS (hot storage)\n- Fast access for\nindexing pipeline\n- Retained for 30\ndays\n\nArchive data → S3\n(cold storage)\n- Compressed WARC\nfiles\n- Retained\nindefinitely\n- Used for historical\nanalysis\n\nStorage format per\npage:\n\nurl: \"https://...\",\nfetched_at:\n\"2026-04-03T...\",\nhttp_status: 200,\nheaders: { ... },\ncontent_type:\n\"text/html\",\nbody: \"<html>...\",\nbody_hash:\n\"sha256:abc...\",\ncontent_length:\n45230","originalText":"Content Storage\nLayer\n\nOption 1: Distributed\nFile System (HDFS)\n\nOption 2: Object\nStorage (S3)\n\nOur Design: Hybrid\n\nRecent crawl data →\nHDFS (hot storage)\n- Fast access for\nindexing pipeline\n- Retained for 30\ndays\n\nArchive data → S3\n(cold storage)\n- Compressed WARC\nfiles\n- Retained\nindefinitely\n- Used for historical\nanalysis\n\nStorage format per\npage:\n\nurl: \"https://...\",\nfetched_at:\n\"2026-04-03T...\",\nhttp_status: 200,\nheaders: { ... },\ncontent_type:\n\"text/html\",\nbody: \"<html>...\",\nbody_hash:\n\"sha256:abc...\",\ncontent_length:\n45230","textAlign":"left","verticalAlign":"middle","containerId":"shape-4","lineHeight":1.3,"autoResize":false,"customData":{"revealStep":3}}],"appState":{"viewBackgroundColor":"#fbfbf8","gridSize":null},"files":{}}