RAG Pipeline

Problem Statement & Requirements

Design a production Retrieval-Augmented Generation system that ingests enterprise knowledge bases (documents, wikis, databases), indexes them for semantic search, retrieves relevant context at query time, and augments LLM prompts to generate accurate, grounded answers with source citations — similar to systems powering Perplexity AI, ChatGPT with search, enterprise Q&A bots (Glean, Guru), and custom knowledge assistants.

Functional Requirements

• Document ingestion — ingest documents from diverse sources: PDFs, web pages, Confluence, Notion, Slack, Google Drive, databases, APIs
• Chunking & embedding — split documents into semantic chunks, generate vector embeddings for each chunk
• Vector search (retrieval) — given a user query, find the most relevant document chunks via semantic similarity
• Hybrid search — combine vector (semantic) search with keyword (BM25) search for better recall
• LLM generation with context — send retrieved chunks as context to an LLM to generate a grounded answer
• Source citations — every claim in the answer links back to the source document and chunk
• Conversational memory — multi-turn conversations with context carried across turns
• Access control — users only see results from documents they have permission to access
• Incremental sync — documents update automatically when source changes (near real-time)
• Evaluation & feedback — track answer quality; thumbs up/down; use feedback for improvement

Non-Functional Requirements

• Answer quality — relevant, accurate, no hallucination; grounded in retrieved sources
• Low latency — retrieval + generation end-to-end < 3 seconds for simple queries
• Scalability — support 100M+ document chunks, 10K+ concurrent queries
• Freshness — new/updated documents searchable within minutes of change
• Multi-tenancy — thousands of organizations with isolated knowledge bases
• Cost efficiency — minimize embedding compute and LLM token usage

Out of Scope

• LLM training / fine-tuning
• General web search (Perplexity-scale crawling)
• Image/video understanding in documents
• Full AI agent orchestration framework

Scale Estimations

Traffic

Storage

Compute

Bandwidth

This first layer defines the system boundary. The next chapters add application logic and data dependencies. After examining the request paths and tradeoffs, we will add operational safeguards and bring the complete architecture together.

API Design

Query (Chat with Knowledge Base)

POST /api/v1/query
Authorization: Bearer <token>

Request:
{
  "query": "How does our refund policy work for international orders?",
  "conversation_id": "conv-abc123",       // for multi-turn
  "knowledge_base_id": "kb-acme-corp",
  "filters": {
    "source_type": ["confluence", "notion"],
    "date_range": {"after": "2025-01-01"},
    "tags": ["policy", "customer-facing"]
  },
  "top_k": 5,                             // number of chunks to retrieve
  "stream": true,
  "include_sources": true
}

Response (SSE stream):
data: {"type": "retrieval_complete", "sources": [
  {"chunk_id": "chk-001", "source": "Refund Policy FAQ", "url": "https://confluence.acme.com/...", "relevance": 0.94},
  {"chunk_id": "chk-002", "source": "Terms of Service v3.2", "url": "...", "relevance": 0.89},
  {"chunk_id": "chk-003", "source": "International Shipping KB", "url": "...", "relevance": 0.85}
]}

data: {"type": "content_delta", "text": "Our refund policy for international orders "}
data: {"type": "content_delta", "text": "allows returns within 30 days of delivery "}
data: {"type": "content_delta", "text": "[1]. "}
data: {"type": "content_delta", "text": "For international shipments, the return shipping "}
data: {"type": "content_delta", "text": "cost is borne by the customer unless the item was "}
data: {"type": "content_delta", "text": "defective [2]. "}
...
data: {"type": "done", "usage": {"retrieval_chunks": 5, "prompt_tokens": 1842, "completion_tokens": 287}}

Ingest Documents

POST /api/v1/knowledge-bases/{kb_id}/documents
Authorization: Bearer <token>

Request:
{
  "source": {
    "type": "confluence",
    "space_key": "ENG",
    "page_ids": ["12345", "67890"],       // specific pages, or null for full space
    "sync_mode": "incremental"            // full | incremental
  },
  "processing": {
    "chunking_strategy": "semantic",       // semantic | fixed | recursive
    "chunk_size_tokens": 512,
    "chunk_overlap_tokens": 50,
    "embedding_model": "text-embedding-3-large"
  },
  "access_control": {
    "inherit_source_permissions": true     // mirror Confluence page permissions
  }
}

Response (202 Accepted):
{
  "ingestion_job_id": "job-xyz789",
  "status": "processing",
  "documents_queued": 2,
  "estimated_chunks": 150
}

Get Ingestion Status

GET /api/v1/ingestion-jobs/{job_id}

Response (200 OK):
{
  "job_id": "job-xyz789",
  "status": "completed",
  "documents_processed": 2,
  "chunks_created": 147,
  "chunks_updated": 12,
  "chunks_deleted": 3,
  "errors": [],
  "completed_at": "2026-04-03T10:05:00Z"
}

Feedback

POST /api/v1/feedback
{
  "query_id": "q-abc123",
  "rating": "thumbs_up",               // thumbs_up | thumbs_down
  "feedback_text": "Answer was accurate but missed the EU exception",
  "correct_source": "chk-007"          // optional: which chunk should have been retrieved
}

Data Model

Document

Chunk

Conversation (Multi-Turn)

Core Design Decisions

Decision 1: Chunking Strategy — How to Split Documents

The most impactful decision for retrieval quality. Bad chunking = bad retrieval = bad answers.

Decision 2: Hybrid Retrieval (Vector + Keyword)

Decision 3: Reranking — The Quality Amplifier

Detailed Flow Diagrams

Query Flow (End-to-End)

Document Ingestion Flow

Reuse document embeddings when the content and embedding model version have not changed. Query or answer caching needs more care: permissions, document versions, and retrieval settings can change what evidence is valid. Bound retrieval candidates before reranking, and measure retrieval latency separately from generation latency to find the actual bottleneck.

Vector Database Deep Dive

HNSW Index (Hierarchical Navigable Small World)

Scaling Vector Search to 5B Vectors

Advanced Retrieval Techniques

Query Expansion / Rewriting

Parent-Child Chunk Retrieval

Handling Edge Cases

Hallucination Prevention

The #1 risk in RAG: LLM generates plausible but wrong information.

Multi-layer defense:

  Layer 1: Grounding Prompt
    System: "Answer ONLY based on the provided context.
    If the context doesn't contain the answer, say
    'I don't have enough information to answer this.'
    NEVER make up facts not in the context."

  Layer 2: Citation Enforcement
    System: "Every factual claim must include [N] citing
    the source chunk number. If you cannot cite a source,
    do not include the claim."

  Layer 3: Post-Generation Verification
    After LLM generates answer:
    → Extract all factual claims
    → For each claim: verify it exists in the retrieved chunks
    → Flag unsupported claims → remove or mark as "unverified"
    Latency cost: ~200-500 ms (can be async)

  Layer 4: Confidence Scoring
    If retrieval scores are all low (top chunk < 0.5 relevance):
    → Don't attempt to answer
    → "I couldn't find relevant information. Try rephrasing
       your question or checking [suggested sources]."

  Layer 5: User Feedback Loop
    Thumbs down → flag for review
    → Identify: was it a retrieval failure or generation failure?
    → Feed back into chunking/embedding improvement

Stale / Outdated Information

Problem: Policy changed yesterday but old version still in index

Solution: Near-real-time incremental sync

  1. Source connectors poll for changes every 5 minutes
     (or webhook-based for supported sources)
  2. Changed documents: re-chunk + re-embed ONLY changed chunks
     (content_hash comparison → skip unchanged chunks)
  3. Deleted documents: remove all chunks from vector + BM25 index
  4. Freshness metadata:
     → Each chunk carries source_last_modified timestamp
     → Retrieval can boost recent chunks: score × recency_boost
     → LLM prompt: "Note: [Source 2] was last updated 2 hours ago"

  For critical freshness:
  → Pre-retrieval filter: date_range.after = "7 days ago"
  → Or: always include most-recent version of each source document

Access Control at Scale

Problem: User queries KB but should only see documents they can access

  Enterprise scenario:
  Alice (engineering) can see: eng-docs, all-company, public
  Bob (sales) can see: sales-docs, all-company, public
  
  Implementation:

  Option A: Post-Retrieval Filtering (simple but wasteful)
    Retrieve top-100 from index (ignoring ACL)
    Filter out chunks user can't access
    Return remaining top-5
    Problem: if top 95 are restricted, return only 5 of low quality

  Option B: Pre-Retrieval Filtering (recommended)
    Each chunk has access_groups: ["engineering", "all-company"]
    At query time: filter = user's groups
    Vector search: WHERE access_groups INTERSECT user_groups
    → Only search within permitted chunks
    → Correct results, no wasted retrieval

  Option C: Separate Indexes per Access Level
    Index per permission group
    Query: search ONLY indexes user has access to
    + Fastest (no filter overhead)
    - More indexes to maintain
    - Shared documents duplicated across indexes

  Recommendation: Pre-retrieval filtering (Option B) via metadata filter
  Most vector DBs support filtered search natively (Qdrant, Pinecone, Weaviate)

Multi-Turn Conversation Context

Problem: Turn 2 query "What about for EU?" is meaningless without Turn 1

Solution: Query Contextualization via LLM

  Conversation:
  Turn 1: "How does our refund policy work?"
  Turn 2: "What about for EU customers?"

  Before retrieval, rewrite Turn 2 using conversation context:

  Prompt to small LLM (fast, cheap):
  "Given this conversation, rewrite the latest query to be standalone:
   User: How does our refund policy work?
   Assistant: [previous answer...]
   User: What about for EU customers?
   Rewritten: What is the refund policy specifically for EU customers?"

  Now search with: "What is the refund policy specifically for EU customers?"
  → Much better retrieval than searching "What about for EU?"

  Cost: ~50 tokens input, ~20 tokens output = negligible
  Latency: ~100 ms (use small model like Haiku)

Tradeoffs & Design Decisions Summary

Reliability & Fault Tolerance

Single Points of Failure & Mitigations

Graceful Degradation

Tier 1 (Reranker down):
  → Skip reranking step; use fusion scores directly
  → Slightly lower quality but functional
  → Latency actually decreases by ~100 ms

Tier 2 (BM25 index down):
  → Pure vector search only (no hybrid)
  → Exact term matches may be missed
  → Retrieval still works semantically

Tier 3 (Embedding service degraded):
  → Cache frequent query embeddings (Redis, TTL=1 hour)
  → Queue new document ingestion (process when recovered)
  → Existing indexed content still searchable

Tier 4 (Primary LLM down):
  → Failover to secondary LLM provider
  → Or: return retrieved chunks directly WITHOUT generation
    → "Here are relevant sources:" + chunk summaries
  → User gets information, just not synthesized

Tier 5 (Vector DB down):
  → Fall back to BM25 only (keyword search)
  → Significantly degraded but still useful
  → Rebuild vector index from stored chunks when recovered

High-Level Architecture

Ingestion Pipeline

Full System Architecture (Production-Grade)

Evaluation & Continuous Improvement

Interview Tips

1. Start with the retrieval-generation separation — "RAG solves the knowledge freshness problem. LLMs have a training cutoff; RAG gives them access to current, private data without retraining. The architecture has two pipelines: ingestion (document → chunks → embeddings → vector index) and query (embed query → retrieve chunks → rerank → generate with context)."

2. Chunking strategy is the #1 quality lever — "Chunk quality determines retrieval quality, which determines answer quality. Semantic chunking with heading context preserves document structure. A chunk about 'EU refund policy' prepended with 'Refund Policy > International > EU-Specific Rules' retrieves far better than a context-free 512-token block."

3. Hybrid search (vector + BM25) is essential — Don't say "just use embeddings." Explain: vector search handles semantic similarity ("return policy" ↔ "refund rules"), BM25 handles exact terms ("ERR-4231"). RRF fusion combines them. +10-30% recall improvement. This is the single biggest retrieval improvement.

4. Two-stage retrieval: recall then precision — Stage 1: fast approximate search (HNSW) returns top-50 candidates. Stage 2: cross-encoder reranker scores each (query, chunk) pair with full token-level attention. Top-5 after reranking are dramatically better than top-5 from retrieval alone.

5. Hallucination prevention is multi-layered — Grounding prompt ("only use provided context"), citation enforcement (every claim must cite [N]), post-generation verification (check each claim against sources), confidence threshold (low retrieval scores → "I don't know"). No single layer is sufficient.

6. Access control must be pre-retrieval, not post — "If I retrieve top-100 and filter to authorized-only, I might return poor-quality results. Pre-retrieval filtering (metadata filter in vector DB) ensures only authorized chunks are even considered. Most vector DBs support filtered HNSW natively."

7. Incremental ingestion saves 90%+ compute — "Hash each chunk's content. On document update, only re-embed chunks where hash changed. A policy doc with 20 chunks where 2 paragraphs changed = re-embed 2 chunks, not 20. At 100M chunks/day, this matters enormously."

8. Multi-turn requires query rewriting — "'What about for EU?' is meaningless without Turn 1. Use a small fast LLM to rewrite: 'What is the refund policy specifically for EU customers?' Then search with the rewritten query. ~100 ms cost, massive quality improvement."

9. Parent-child chunking solves the precision-context tradeoff — "Small chunks (256 tokens) for precise retrieval. But return the parent chunk (1024 tokens) to the LLM for richer context. Best of both worlds: precise recall + sufficient context for generation."

10. End with evaluation metrics — "RAG quality is measurable: retrieval recall@5, MRR, NDCG for retrieval; faithfulness, relevance, hallucination rate for generation; thumbs up/down ratio end-to-end. Nightly automated eval on a test suite of 500 queries catches regressions. Without evaluation, you're flying blind."