{"type":"excalidraw","version":2,"source":"https://AwesomeSystemDesign.dev","elements":[{"angle":0,"strokeColor":"#628572","backgroundColor":"#edf5ee","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-1"],"frameId":null,"roundness":null,"boundElements":[{"type":"text","id":"shape-1"}],"updated":0,"link":null,"locked":false,"id":"shape-0","type":"rectangle","seed":41,"x":30,"y":30,"width":420,"height":68,"customData":{"revealStep":1}},{"angle":0,"strokeColor":"#394557","backgroundColor":"transparent","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-1"],"frameId":null,"roundness":null,"boundElements":null,"updated":0,"link":null,"locked":false,"id":"shape-1","type":"text","seed":114,"x":50,"y":52.3,"width":380,"height":23.400000000000002,"fontSize":18,"fontFamily":1,"text":"RAG Evaluation Framework","originalText":"RAG Evaluation Framework","textAlign":"left","verticalAlign":"middle","containerId":"shape-0","lineHeight":1.3,"autoResize":false,"customData":{"revealStep":1}},{"angle":0,"strokeColor":"#a68a53","backgroundColor":"#fff7e6","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-2"],"frameId":null,"roundness":null,"boundElements":[{"type":"text","id":"shape-3"}],"updated":0,"link":null,"locked":false,"id":"shape-2","type":"rectangle","seed":187,"x":30,"y":123,"width":665.3088400000001,"height":153,"customData":{"revealStep":2}},{"angle":0,"strokeColor":"#394557","backgroundColor":"transparent","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-2"],"frameId":null,"roundness":null,"boundElements":null,"updated":0,"link":null,"locked":false,"id":"shape-3","type":"text","seed":260,"x":50,"y":141,"width":625.3088400000001,"height":117,"fontSize":18,"fontFamily":1,"text":"Dimension 1: Retrieval Quality\n• Recall@K: % of relevant chunks in top-K results\n• MRR (Mean Reciprocal Rank): how high is the first relevant?\n• NDCG: quality of ranking considering graded relevance\nMeasured via: human-labeled test set of (query, relevant_chunks)","originalText":"Dimension 1: Retrieval Quality\n• Recall@K: % of relevant chunks in top-K results\n• MRR (Mean Reciprocal Rank): how high is the first relevant?\n• NDCG: quality of ranking considering graded relevance\nMeasured via: human-labeled test set of (query, relevant_chunks)","textAlign":"left","verticalAlign":"middle","containerId":"shape-2","lineHeight":1.3,"autoResize":false,"customData":{"revealStep":2}},{"angle":0,"strokeColor":"#9180ac","backgroundColor":"#f4effb","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-3"],"frameId":null,"roundness":null,"boundElements":[{"type":"text","id":"shape-5"}],"updated":0,"link":null,"locked":false,"id":"shape-4","type":"rectangle","seed":333,"x":30,"y":301,"width":589.7902000000001,"height":176.4,"customData":{"revealStep":3}},{"angle":0,"strokeColor":"#394557","backgroundColor":"transparent","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-3"],"frameId":null,"roundness":null,"boundElements":null,"updated":0,"link":null,"locked":false,"id":"shape-5","type":"text","seed":406,"x":50,"y":319,"width":549.7902000000001,"height":140.4,"fontSize":18,"fontFamily":1,"text":"Dimension 2: Generation Quality\n• Faithfulness: are claims supported by retrieved context?\n→ LLM-as-judge: \"Is this claim supported by the context?\"\n• Relevance: does the answer address the question?\n• Completeness: did the answer cover all aspects?\n• Hallucination rate: % of claims with no source support","originalText":"Dimension 2: Generation Quality\n• Faithfulness: are claims supported by retrieved context?\n→ LLM-as-judge: \"Is this claim supported by the context?\"\n• Relevance: does the answer address the question?\n• Completeness: did the answer cover all aspects?\n• Hallucination rate: % of claims with no source support","textAlign":"left","verticalAlign":"middle","containerId":"shape-4","lineHeight":1.3,"autoResize":false,"customData":{"revealStep":3}},{"angle":0,"strokeColor":"#6577aa","backgroundColor":"#eef1fc","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-4"],"frameId":null,"roundness":null,"boundElements":[{"type":"text","id":"shape-7"}],"updated":0,"link":null,"locked":false,"id":"shape-6","type":"rectangle","seed":479,"x":30,"y":502.4,"width":644.56888,"height":129.60000000000002,"customData":{"revealStep":4}},{"angle":0,"strokeColor":"#394557","backgroundColor":"transparent","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-4"],"frameId":null,"roundness":null,"boundElements":null,"updated":0,"link":null,"locked":false,"id":"shape-7","type":"text","seed":552,"x":50,"y":520.4,"width":604.56888,"height":93.60000000000001,"fontSize":18,"fontFamily":1,"text":"Dimension 3: End-to-End\n• User satisfaction: thumbs up/down ratio\n• Answer accepted rate: did user take action based on answer?\n• Follow-up rate: high follow-ups = incomplete first answer","originalText":"Dimension 3: End-to-End\n• User satisfaction: thumbs up/down ratio\n• Answer accepted rate: did user take action based on answer?\n• Follow-up rate: high follow-ups = incomplete first answer","textAlign":"left","verticalAlign":"middle","containerId":"shape-6","lineHeight":1.3,"autoResize":false,"customData":{"revealStep":4}},{"angle":0,"strokeColor":"#628572","backgroundColor":"#edf5ee","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-5"],"frameId":null,"roundness":null,"boundElements":[{"type":"text","id":"shape-9"}],"updated":0,"link":null,"locked":false,"id":"shape-8","type":"rectangle","seed":625,"x":30,"y":657,"width":633.1972000000001,"height":153,"customData":{"revealStep":5}},{"angle":0,"strokeColor":"#394557","backgroundColor":"transparent","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-5"],"frameId":null,"roundness":null,"boundElements":null,"updated":0,"link":null,"locked":false,"id":"shape-9","type":"text","seed":698,"x":50,"y":675,"width":593.1972000000001,"height":117,"fontSize":18,"fontFamily":1,"text":"Automated evaluation pipeline (nightly):\n1. Run test suite of 500+ (query, expected_answer, sources)\n2. Compare with previous day's results\n3. Alert if any dimension regresses > 5%\n4. Track per-change: \"after adding BM25, recall improved 12%\"","originalText":"Automated evaluation pipeline (nightly):\n1. Run test suite of 500+ (query, expected_answer, sources)\n2. Compare with previous day's results\n3. Alert if any dimension regresses > 5%\n4. Track per-change: \"after adding BM25, recall improved 12%\"","textAlign":"left","verticalAlign":"middle","containerId":"shape-8","lineHeight":1.3,"autoResize":false,"customData":{"revealStep":5}},{"angle":0,"strokeColor":"#a68a53","backgroundColor":"#fff7e6","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-6"],"frameId":null,"roundness":null,"boundElements":[{"type":"text","id":"shape-11"}],"updated":0,"link":null,"locked":false,"id":"shape-10","type":"rectangle","seed":771,"x":30,"y":835,"width":558.28912,"height":176.4,"customData":{"revealStep":6}},{"angle":0,"strokeColor":"#394557","backgroundColor":"transparent","fillStyle":"solid","strokeWidth":1.5,"strokeStyle":"solid","roughness":0.7,"opacity":100,"version":1,"versionNonce":1,"isDeleted":false,"groupIds":["step-6"],"frameId":null,"roundness":null,"boundElements":null,"updated":0,"link":null,"locked":false,"id":"shape-11","type":"text","seed":844,"x":50,"y":853,"width":518.28912,"height":140.4,"fontSize":18,"fontFamily":1,"text":"Continuous improvement loop:\nUser feedback → identify failure patterns →\n→ Chunking too coarse? → reduce chunk size\n→ Wrong documents retrieved? → improve embedding model\n→ Right docs but wrong answer? → improve prompt\n→ Missing documents entirely? → check ingestion pipeline","originalText":"Continuous improvement loop:\nUser feedback → identify failure patterns →\n→ Chunking too coarse? → reduce chunk size\n→ Wrong documents retrieved? → improve embedding model\n→ Right docs but wrong answer? → improve prompt\n→ Missing documents entirely? → check ingestion pipeline","textAlign":"left","verticalAlign":"middle","containerId":"shape-10","lineHeight":1.3,"autoResize":false,"customData":{"revealStep":6}}],"appState":{"viewBackgroundColor":"#fbfbf8","gridSize":null},"files":{}}