{"type":"excalidraw","version":2,"source":"https://AwesomeSystemDesign.dev","elements":[{"angle":0,"strokeColor":"#6577aa","backgroundColor":"#eef1fc","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-1"],"frameId":null,"roundness":null,"boundElements":[{"type":"text","id":"shape-1"}],"updated":0,"link":null,"locked":false,"id":"shape-0","type":"rectangle","seed":41,"x":36,"y":32,"width":358.23532,"height":153,"customData":{"revealStep":1}},{"angle":0,"strokeColor":"#394557","backgroundColor":"transparent","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-1"],"frameId":null,"roundness":null,"boundElements":null,"updated":0,"link":null,"locked":false,"id":"shape-1","type":"text","seed":114,"x":56,"y":50,"width":318.23532,"height":117,"fontSize":18,"fontFamily":1,"text":"page_hash = SHA-256(html_body)\nCheck against content hash set\n✅ Simple, fast\n❌ Fails if even 1 byte differs\n(timestamps, ads, session tokens)","originalText":"page_hash = SHA-256(html_body)\nCheck against content hash set\n✅ Simple, fast\n❌ Fails if even 1 byte differs\n(timestamps, ads, session tokens)","textAlign":"center","verticalAlign":"middle","containerId":"shape-0","lineHeight":1.3,"autoResize":false,"customData":{"revealStep":1}},{"angle":0,"strokeColor":"#628572","backgroundColor":"#edf5ee","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-2"],"frameId":null,"roundness":null,"boundElements":[{"type":"text","id":"shape-3"}],"updated":0,"link":null,"locked":false,"id":"shape-2","type":"rectangle","seed":187,"x":434.23532,"y":32,"width":390,"height":410.40000000000003,"customData":{"revealStep":2}},{"angle":0,"strokeColor":"#394557","backgroundColor":"transparent","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-2"],"frameId":null,"roundness":null,"boundElements":null,"updated":0,"link":null,"locked":false,"id":"shape-3","type":"text","seed":260,"x":454.23532,"y":50,"width":350,"height":374.40000000000003,"fontSize":18,"fontFamily":1,"text":"SimHash (Charikar, 2002):\n1. Extract features\n(shingles/n-grams)\n2. Hash each feature to 64-bit\nvalue\n3. Sum weighted bit vectors\n4. Final hash: positive sums → 1,\nnegative → 0\n\nSimilarity: Hamming distance of\nhashes\n< 3 bits different → near-duplicate\n\n✅ Detects near-duplicates\n✅ Compact (64-bit per document)\n❌ More complex to implement","originalText":"SimHash (Charikar, 2002):\n1. Extract features\n(shingles/n-grams)\n2. Hash each feature to 64-bit\nvalue\n3. Sum weighted bit vectors\n4. Final hash: positive sums → 1,\nnegative → 0\n\nSimilarity: Hamming distance of\nhashes\n< 3 bits different → near-duplicate\n\n✅ Detects near-duplicates\n✅ Compact (64-bit per document)\n❌ More complex to implement","textAlign":"center","verticalAlign":"middle","containerId":"shape-2","lineHeight":1.3,"autoResize":false,"customData":{"revealStep":2}},{"angle":0,"strokeColor":"#a68a53","backgroundColor":"#fff7e6","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-3"],"frameId":null,"roundness":null,"boundElements":[{"type":"text","id":"shape-5"}],"updated":0,"link":null,"locked":false,"id":"shape-4","type":"rectangle","seed":333,"x":36,"y":482.40000000000003,"width":567.87788,"height":323.3,"customData":{"revealStep":3}},{"angle":0,"strokeColor":"#394557","backgroundColor":"transparent","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-3"],"frameId":null,"roundness":null,"boundElements":null,"updated":0,"link":null,"locked":false,"id":"shape-5","type":"text","seed":406,"x":56,"y":500.40000000000003,"width":527.87788,"height":287.3,"fontSize":17,"fontFamily":1,"text":"Content Fingerprinting\n\nProblem: Different URLs can serve identical content\nexample.com/page ← same content → example.com/page/\nexample.com/?ref=a ← same content → example.com/?ref=b\n\nSolution 1: Exact Match — SHA-256 of page body\n\nSolution 2: Near-Duplicate — SimHash / MinHash\n\nRecommendation: SHA-256 for exact dedup (cheap, catches\nmost duplicates) + SimHash for near-dedup (run as\nsecondary batch process)","originalText":"Content Fingerprinting\n\nProblem: Different URLs can serve identical content\nexample.com/page ← same content → example.com/page/\nexample.com/?ref=a ← same content → example.com/?ref=b\n\nSolution 1: Exact Match — SHA-256 of page body\n\nSolution 2: Near-Duplicate — SimHash / MinHash\n\nRecommendation: SHA-256 for exact dedup (cheap, catches\nmost duplicates) + SimHash for near-dedup (run as\nsecondary batch process)","textAlign":"left","verticalAlign":"middle","containerId":"shape-4","lineHeight":1.3,"autoResize":false,"customData":{"revealStep":3}}],"appState":{"viewBackgroundColor":"#fbfbf8","gridSize":null},"files":{}}