Back to Hub
Certification Study Notes

Retrieval-Augmented Generation (RAG)

Applied AI Architect study guide covering retrieval modes, hybrid retrieval, Reciprocal Rank Fusion (RRF), cross-encoder reranking, untrusted context & refusal gates, Agentic RAG & memory, GraphRAG, Google Cloud RAG, federated vs. ingested architectures, ACL security, semantic caching, and evaluation.

Author: Michaël Bettan
Track: Applied AI Architect / Foundation
Test Yourself: RAG Practice Quiz →
00

Overview & Core Mechanics

Definition

Retrieval-Augmented Generation (RAG) pairs an LLM's parametric knowledge with an external retrieval system. At query time it finds the relevant passages, injects them into the prompt and asks the model to answer only from that context, with citations. The model is grounded on private, current data without fine-tuning, and every claim can be traced to a source.

Why It Exists (Standalone LLM Limits)

  • Private data: The model doesn't know your wikis, tickets, contracts, or databases.
  • Freshness: Weights are frozen at the training cutoff.
  • Citations & hallucination: The model can't cite its weights, and confabulates when facts are missing.
  • Scale & cost: Stuffing everything into the prompt is expensive, slow, and loses facts "in the middle".

Three Core Phases

  • 1. Retrieval: Finds the relevant chunks in an external store (the retriever: search and recall only).
  • 2. Augmentation: Injects those chunks and the question into a prompt template (prompt hydration).
  • 3. Generation: The LLM (the generator) reads, reasons, and writes a grounded, cited answer.

Mental Model

Fine-tuning is a semester of study that teaches a student how to think and speak like a specialist. RAG hands that student an open-book exam with exactly the right pages bookmarked.

01

When to Use RAG (vs. Fine-Tuning, Long Context & Live Tools)

Approach Best For Update Speed Cost per Query Traceability
Fine-tuning Tone, format, behavior, output schemas Hours to weeks (retrain) Low inference, high training None: facts can't be traced to weights
Long-context stuffing Whole-document reasoning on a small corpus Instant High: grows with prompt size Coarse
Context caching Reusing a large, fixed prefix Minutes Lower (cached input discount) Coarse
RAG Facts from large, changing, permissioned reference corpora Seconds to minutes (index update) Low: ~1.5k–3k relevant tokens Exact (doc, chunk, page)
Live tool call (API / SQL) Live transactional state (order status, stock levels, account balances, booking availability) Real-time (system of record) Low (1 API/DB call) Exact (record ID + timestamp)

Why Long Context Alone Fails at Scale

Cost grows linearly with input tokens, time to first token grows with prompt length, recall degrades well before the advertised window is full ("lost in the middle"), and you cannot apply per-user ACLs or exact citations to one giant prompt. The production answer is often both: RAG narrows millions of documents to a few full sections (tens of thousands of tokens), then a long-context model reasons across them without fragmenting the evidence into tiny chunks.

Choose Long Context When…

  • Corpus is small (under ~50K–200K tokens)
  • Low query volume
  • The task needs the whole document at once
  • Ephemeral per-session uploads
  • Simplicity matters more than cost
VS

Choose RAG When…

  • Corpus is large or unbounded
  • High volume where unit cost and latency matter
  • Questions target specific facts
  • Shared corpus that changes continuously
  • Citations, audit, and ACLs are mandatory

Retrieval Modes & Context-Window Utilization

Information retrieval offers two complementary scoring families; production RAG usually runs both and fuses the rankings (see Retrieval):

Retrieval Mode How It Scores Relevance Strong On Weak On
Dense vector (semantic) Embeds query and chunks into one vector space; ranks by cosine or dot-product distance Paraphrase, synonyms, cross-lingual intent Exact identifiers, rare tokens, negation
Sparse keyword (BM25) Term overlap weighted by term frequency, inverse document frequency, and length normalization Product codes, error IDs, names, legal citations Paraphrased or vocabulary-mismatched questions
Hybrid Runs both in parallel, then fuses ranks with Reciprocal Rank Fusion (RRF) Mixed natural-language plus identifier queries Extra latency and two indexes to maintain

Context Rot & Window Limits

More context is not automatically better: accuracy and recall degrade as token count grows (context rot), so curate what enters the prompt instead of filling the space available. On Claude 4.5 and newer models, a request whose input plus max_tokens exceeds the window is accepted and stops with stop_reason: model_context_window_exceeded if generation reaches the limit; earlier models return a validation error unless the model-context-window-exceeded-2025-08-26 beta header is sent.

Long Context vs. RAG Cost Model

Compare three cost drivers before choosing:

  • 1. Token spend: Per-query input tokens × monthly volume, after prefix caching on static content.
  • 2. Pipeline operations: The running cost of parsing, embedding, indexing, and evaluating a retrieval stack.
  • 3. Growth headroom: The migration effort if the corpus later outgrows the context window.

Rule of Thumb

RAG supplies reference facts, fine-tuning shapes behavior, long context does deep reasoning over a pre-filtered set, and tool calls fetch live transactional state.

Common Trap: Treating Embedding Distance as Veracity

Nearest-neighbor scores rank how close a passage's meaning is to the query; they carry no information about whether the passage is current. An indexed copy of a mutable record (an order, a price, a balance) is a snapshot frozen at ingestion time, so it can never be authoritative. Keep reference knowledge in RAG and query live state through API/SQL tools against the system of record.

02

Architecture: Offline Indexing & Online Answering

Offline Ingestion & Indexing (Once per Change)
Sources
PDF, Wiki, Tickets
→
Parse
Layout, Tables, OCR
→
Chunk & Enrich
doc_id, date, acl
→
Embed
RETRIEVAL_DOCUMENT
→
Hybrid Index
Vectors + BM25
Online Retrieval & Generation (Every Query)
User Query
+ Auth Scope
→
Rewrite & Embed
RETRIEVAL_QUERY
→
Hybrid Search
Pre-filtered Top-K
→
RRF + Rerank
Cross-Encoder Top-N
→
LLM Generator
Answer + Citations
Phase Steps Key Rule
Offline: ingest and index Parse → chunk → attach metadata (doc_id, page, date, acl) → embed → store vectors, text, and metadata Expensive work happens once, so online queries stay fast
Online: retrieve and generate (Rewrite) → embed the query → hybrid search for top-K → rerank to top-N → build the prompt → generate with citations Embed queries with the same model and version as the corpus

RAG Paradigms (Gao et al. Survey)

Understanding the progression from Naïve to Modular RAG clarifies where pre-retrieval, post-retrieval, and agentic orchestration sit:

Paradigm Shape Adds Weakness
Naïve index → retrieve → generate Basic chunking + vector search Noisy context, low precision and recall
Advanced Linear, with pre- and post-retrieval stages Query rewriting, hybrid search, reranking, compression Fixed one-shot path
Modular Composable, non-linear modules Routing, iteration, memory, agentic loops More complexity, latency, and evaluation cost

Common Trap

Agentic RAG isn't a fourth official paradigm. It's an implementation of modular RAG in which an agent orchestrates the modules.

03

Indexing: Parsing, Chunking & Enrichment

Parsing Comes First

Bad parsing corrupts everything downstream, and silently: if a table is dropped or a scan is misread, the LLM won't complain—it will invent plausible facts.

Chunking Strategy How It Works Use When
Recursive character / fixed-size (baseline) Split on paragraphs, then lines, then words Default starting point for uniform text lacking structural markers
Overlap Neighboring chunks share ~10–20% Facts that span boundaries
Semantic Split where embedding similarity between sentences drops (topic shift) Long, continuous narrative prose where fixed windows slice ideas in half
Structure-aware Split Markdown by headings, source code by AST/function/class, and legal docs by clause; keep tables whole Codebases, docs with markup, manuals, tables
Parent-child (hierarchical) Embed small children (128–256 tokens), return the larger parent (1–2K tokens) Precise matching and full context (clause-numbered contracts, policy docs)
Proposition An LLM rewrites text into atomic facts High-precision factual lookup

Small Chunks (100–256 Tokens)

  • Precision: High (focused semantic vector)
  • Context: Low (pronouns lose their referents)
  • Tokens per answer: Low — more sources fit
  • Main risk: Severed facts
VS

Large Chunks (1–2K Tokens)

  • Precision: Lower (one vector averages several topics)
  • Context: High (self-contained passages)
  • Tokens per answer: High — fewer sources fit
  • Main risk: Noise, lost in the middle

Hard Constraint: Embedding Token Limit

A chunk must fit the embedding model's input limit (2,048 tokens for gemini-embedding-001 and text-embedding-005). Because autoTruncate=true by default, longer chunks are silently truncated, and their tail is never searchable.

Index-Time Enrichment What It Adds Problem It Fixes
Metadata (source, date, acl, region, is_latest) Structured fields on every chunk Filtering, recency, access control
Contextual headers (contextual retrieval) Document title, section path, and a short summary prepended before embedding and BM25 Ambiguous chunks ("Revenue grew 12%": of whom, when?)
doc2query LLM-generated questions each chunk answers Question vs. passage wording gap
SPLADE Learned expansion terms in a sparse vector BM25's blindness to synonyms

Contextual Header (Before vs. After)

  • Raw chunk: "Revenue grew 12% over the previous quarter." (Cannot match "ACME EMEA Q3 revenue" by meaning or keyword.)
  • Contextualized chunk: "[ACME Corp 2025 annual report › EMEA results › Q3] ACME's EMEA revenue grew 12% over Q2 2025." (Matches both dense and BM25!)
  • Cost optimization: Cache the full source document in the prompt prefix and run offline backfills through asynchronous batch APIs (~50% discount).

Enrichment Mechanics

  • Match on the expansion, answer from the original: Generated text can be wrong, so always send the original chunk to the LLM.
  • Parent-child mechanics: Store parent_id on each child, deduplicate by parent, and ACL-check the parent too.
  • HyDE vs. Document Expansion: HyDE expands the query at runtime (latency on every request). Document expansion expands the corpus once at index time (cost at ingestion only).
04

Embeddings & Vector Search

The Semantic Recall Role

An embedding is a dense vector (typically 768–3,072 dimensions) in which texts with similar meaning sit close together, regardless of wording. Don't just say vector search "stores embeddings"—its role is Stage 1 semantic recall, retrieving conceptually matching passages even when the user's wording shares zero keywords with the document wording (bridging the vocabulary gap where BM25 returns zero hits). The embedding model sets the ceiling on semantic recall: downstream prompt engineering can't rescue a poor vector space.

Concept What to Know
Same model both sides Query and corpus vectors must come from the same model and version. Otherwise scores still compute, but rankings are near-random.
Task types (Vertex AI) RETRIEVAL_DOCUMENT (+ title) for chunks, RETRIEVAL_QUERY for queries. Unset defaults to RETRIEVAL_QUERY, so the corpus gets query-style vectors.
Current models gemini-embedding-001 (3,072 dims) and text-embedding-005 (768 dims); output_dimensionality truncates (Matryoshka), then renormalize.
Similarity vs. distance Cosine similarity (higher = closer; equals dot product on unit-normalized vectors) vs. semantic distance (lower = closer, e.g., 1 - cos or L2). Watch cutoff direction: similarity >= τ vs. vector_distance_threshold <= d_max.
Domain fit Benchmark on your labeled queries; leaderboards are only a shortlist. Add BM25 or an adaptor for jargon.
ANN Index How It Works Strength Trade-Off
Exact k-NN Compare with every vector 100% recall Linear cost; unusable past ~10⁵ vectors
HNSW Multi-layer proximity graph Best recall vs. latency High RAM
IVF k-means clusters; search the nearest n_probe Fast partitioning Recall drops if n_probe is too small
PQ Compress sub-vectors to codebook IDs Large memory savings Lossy ranking
ScaNN Google's anisotropic quantization Highest QPS; powers Vertex AI Vector Search and AlloyDB AI Tuning required

Knobs, Quantization & Scale-Out

  • Tuning knobs: HNSW ef_search and IVF n_probe. Higher means more recall and more latency.
  • Quantization ladder: float32 (1×) → float16 (½) → int8 (¼, default at 100M+ vectors) → PQ (~10–30× smaller) → binary (1/32). Pair heavy compression with oversample + full-precision rescoring.
  • Shards vs. replicas: Shards scale corpus size; replicas scale QPS and availability. Isolate mid-size tenants via namespaces/partitions and large/regulated tenants on dedicated shards.

Vector Store Landscape

Core ANN libraries (ScaNN, Meta FAISS) power three deployment tiers:

  • 1. Local embedded stores (Chroma) for fast prototyping.
  • 2. Dedicated vector databases (Vertex AI Vector Search, Qdrant, Milvus, Pinecone, Weaviate) for high-scale ANN and payload filtering.
  • 3. Vector-enabled operational stores (PostgreSQL pgvector, AlloyDB AI, BigQuery, Redis, Elasticsearch) keeping vectors and relational rows in one ACID system without dual-write sync lag.

Common Trap: ANN Recall vs. Retrieval Recall

"ANN recall" (agreement with exact k-NN, fixed with ef_search / n_probe) isn't "retrieval recall" (agreement with human relevance labels, fixed with chunking, embeddings, and hybrid search). Diagnose which one is failing.

05

Retrieval: Hybrid Search, RRF & Reranking

Hybrid Search Decision Rule

Match the index to query diversity. Hybrid retrieval justifies its added compute and rank-fusion step whenever traffic blends exact identifier or clause lookups (sparse BM25) with broad conceptual questions (dense vector). Frame the architecture as a three-way balance across quality, latency, and ongoing maintenance (the operational burden of keeping indexes synchronized as the corpus scales). Both searches run in parallel over the same metadata-pre-filtered corpus, and their ranked lists are merged via RRF.

Dimension Dense (Vector) Sparse (BM25)
Matches Semantic recall: meaning, synonyms, paraphrases (when user wording ≠ document wording) Lexical recall: exact terms, IDs, SKUs, error codes, clause numbers, names
Misses Rare exact strings (ERR-409, §4.2(b)) Synonyms ("car" vs. "automobile")
Score Scale Cosine in [-1, 1] Unbounded [0, ∞)
Reciprocal Rank Fusion (RRF) — Rank-Only Score Merging
RRF(d) = ∑ [ 1 / (k + r_m(d)) ]   (typically k = 60)
Raw cosine and BM25 scores can't be added (different scales). With k = 60: Chunk A (#1 in BM25, #50 in dense) scores 1/61 + 1/110 ≈ 0.0255. Chunk B (#3 in BM25, #2 in dense) scores 1/63 + 1/62 ≈ 0.0320. Chunk B wins because RRF rewards consensus across retrievers.

Common Trap: Hybrid Search vs. RAG-Fusion

Both use RRF, but fuse different lists. Hybrid search fuses retrievers (BM25 + dense) for one query. RAG-Fusion fuses results from several LLM-generated query variants.

Aspect Bi-Encoder (Vector Search — Stage 1) Cross-Encoder (Reranker — Stage 2)
Input Query and document encoded separately Query + document encoded together (full cross-attention)
Speed Fast: document vectors precomputed, ANN lookup Slow: one transformer pass per (query, chunk) pair
Role & Funnel Size Stage 1 Recall over millions (top K ≈ 50–100 per retriever) Stage 2 Precision: re-score RRF-fused pool (~100–150 → final top N ≈ 3–10)
Examples SBERT, DPR, Vertex AI embeddings Vertex AI Ranking API, Cohere Rerank, BGE-Reranker

Dynamic N, Pre-Generation Refusal Gate & Precision Wording

Stop adding chunks when the cross-encoder relevance score falls below a calibrated threshold τ, and abstain immediately if even the top-1 chunk falls below τ. In interviews/exams, never say "re-rank with list"—say "fuse parallel BM25 and dense lists via RRF, then re-score the fused candidates with a pairwise cross-encoder reranker."

Query Transformation (Pre-Retrieval)

  • Query rewriting: Uses chat history to make follow-ups self-contained ("How about Emily?" → full question).
  • Routing: Sends query to the right backend (domain index, SQL, graph, web, or live API tool).
  • HyDE: LLM drafts a hypothetical answer; embed the draft and search with it.
  • Multi-query: 3–5 rephrasings of the same intent, fused with RRF.
  • Decomposition: Splits a multi-part or multi-hop question into sub-queries.
  • Self-query: LLM extracts a metadata filter + semantic query.

Metadata Filtering (Pre-Filter vs. Post-Filter)

  • Pre-filter (filtered ANN): Applied during the BM25 and ANN index scan. Returns K valid results; mandatory for ACLs (acl_groups) and excluding stale data (is_latest = true). Vertex AI: restricts / numeric_restricts.
  • Post-filter: Global top-K, then drop non-matching results. Can return zero results (recall collapse); never acceptable as primary filter for ACLs or stale data.
  • Partition / namespace: Separate index per tenant, language, or region (hard isolation).
06

Generation: Untrusted Context, Refusal Behavior & Groundedness

Untrusted Input Isolation & Fail-Closed Screening

  • Structural delimiters: Wrap each top-N chunk in <document> tags with its Source ID and effective_date. Instruct the LLM to treat <document> blocks strictly as untrusted passive data, never executable instructions.
  • Post-retrieval screening gate (fail-closed): Input guardrails only inspect the user's prompt prior to search, missing indirect prompt injections inside fetched passages or tool outputs. Run a classifier on retrieved chunks before prompt assembly and fail closed (halt rather than pass unscreened text) if the classifier errors or times out.

Context Ordering & Prompt-Cache Prefix

  • Lost in the middle: Attention is strongest at the start and end of the prompt, so put the highest-ranked chunks there.
  • Prefix cache ordering: Because prefix caches evaluate tools → system → messages in order, keep static tool definitions, system instructions, and pinned docs in the cached prefix, and place dynamically retrieved RAG chunks + the user query at the uncached tail (never inject per-query chunks or timestamps into system).
Defense Layer Technique
Retrieval Pre-filtered hybrid search + cross-encoder reranking, measured with Recall@K
Pre-generation refusal gate If top cross-encoder score < τ, short-circuit and return a deterministic refusal ("No relevant documents found")
Context & security isolation Deduplicate; pre-filter stale versions (is_latest = true); pass effective_date to surface conflicts; wrap chunks in <document> tags as untrusted data
Post-retrieval screening gate Run a classifier on fetched passages and tool payloads prior to prompt assembly; fail closed on classifier error; cap chunks and tokens
Prompt & output Closed-world rule + explicit refusal instruction; low temperature; structured output with required citations checked in code
Groundedness verification Claim-level entailment check: NLI model, LLM judge Faithfulness, or Vertex AI Check Grounding API (support_score 0–1)
Post-generation refusal gate Below groundedness threshold → block answer and fall back to safe refusal (showing closest sources) or human review
Citation Level Mechanism What It Guarantees
Level 1 (Weakest) Free-text [n] brackets Nothing: numbers can be invented
Level 2 Cite chunk IDs from the supplied set The source exists (IDs checked in code)
Level 3 Exact quote spans The quoted text exists (substring check)
Level 4 (Strongest) Native grounding metadata Claim-to-source mapping from the API: Gemini groundingMetadata (chunks + supports), Claude Citations API (returns source spans without billing output tokens for quoted text)

Common Trap: Citations ≠ Entailment

"The answer has citations, so it's grounded." A citation proves the source exists, not that it supports the sentence. Always check entailment (sampled, or on every response for high-stakes flows). Refusal is a first-class safety feature enforced both before generation (reranker threshold τ) and after generation (groundedness check).

07

Agentic RAG & Three-Layer Agent Memory

Agentic RAG: Retrieval Inside the Reasoning Loop

A fixed pipeline fails silently when retrieval returns the wrong chunks. Agentic RAG places retrieval inside the reasoning loop so the agent decides when to search, what to query, whether the returned passages are trustworthy, and when to retry or switch tools.

Agentic Pattern Mechanism Fixes
Corrective RAG (CRAG) A grader labels retrieval Correct / Ambiguous / Incorrect; Incorrect triggers query rewrite + web search Answering when the local index lacks the facts
Self-RAG The generator emits reflection tokens: [Retrieve], [IsRel], [IsSup], [IsUse] Needless retrieval and unsupported claims
Adaptive RAG A complexity router picks no retrieval, single-step, or iterative multi-hop Over-paying on simple queries, under-retrieving on hard ones

Implementation Patterns

  • 1. Explicit state-machine graphs: e.g., LangGraph StateGraph carrying typed question / documents / generation state across deterministic retrieve → grade/rewrite → generate nodes for predictable control flow.
  • 2. Autonomous tool-calling loops: e.g., LlamaIndex ReActAgent or Google ADK Agent wrapping a hybrid + cross-encoder reranked retriever as a callable QueryEngineTool.

Guardrails & Trajectory Evaluation

  • Tool description quality: The model decides whether to call retrieval by reading the tool's description. If retrieval never fires, check the trace first and fix the tool description.
  • Hard iteration cap: Prevent runaway search loops and task drift; evaluate the trajectory, not just the final answer (Vertex AI trajectory_* metrics; ADK tool_trajectory_avg_score).
Memory Layer What It Holds Scope Typical Backing Store
Short-term Current turn history and active working context Single session In-memory message list
Episodic User preferences, past sessions, prior trajectories ("what was tried") Cross-session Key-value / session store; auto-extracted via Vertex AI Memory Bank, Mem0 (auto fact extraction), or Zep (session-aware auto-summarization + entity extraction)
Semantic Permanent domain knowledge searched by meaning Cross-session / global Vector store (Chroma locally; Vertex AI Vector Search, Qdrant, Pinecone, Weaviate, or pgvector in prod); keep separate collections for user preferences vs. domain facts

Persistent External Scratchpad (Read → Refine → Repeat) & Salience Decay

08

GraphRAG, Structured & Multimodal Data

GraphRAG (Property Graphs & Community Summaries)

GraphRAG stores entities (nodes) and relationships (edges) next to—or instead of—vectors. Where vector similarity only floats topically related paragraphs to the top, a property graph (e.g., Neo4j queried via Text-to-Cypher in LlamaIndex PropertyGraphIndex or LangChain GraphQA) traverses explicit relational chains (Person -[:WORKS_ON]-> Project <-[:FUNDS]- Org) in a single deterministic hop, and builds corpus-wide summaries ("main themes across all Q3 filings") via community summaries. For purely relational queries (org charts, supply chains, legal discovery), traverse the graph directly rather than relying on vector similarity.

Dimension Vector RAG GraphRAG Hybrid (Vector + Graph)
Strength Local semantic match Multi-hop entity chains (org charts, supply chains, legal/financial links), global summaries Explicit relations + grounded passage quotes
Cost Low to moderate High (entity/relation extraction, rigid schema & graph upkeep, higher latency) Highest (two stores to sync)

Structured Data (Text-to-SQL)

Embeddings are bad at SUM, AVG, sorting, and joins. Route quantitative questions to Text-to-SQL: retrieve the relevant schema → generate read-only SQL → execute → put the exact result in the prompt. BigQuery and AlloyDB AI can also combine SQL filters and vector search in one query. For choosing between parameterized query tools and NL2SQL, and the guardrails for model-written SQL, see Knowledge engineering: structured data tools.

Multimodal RAG

Multimodal embeddings put text and images in one shared vector space, so a text query can retrieve architecture diagrams, charts, or schematics, which a multimodal generator (such as Gemini) then reads together with the text.

09

RAG on Google Cloud

This guide stays vendor-neutral. Google Cloud's managed retrieval services (Agent Search, RAG Engine, Vector Search 1.0 and Agent Retrieval, Grounding with Google Search, the Ranking and Check grounding APIs, and in-database vectors) are covered in depth in the Agent Platform guide: Grounding & Retrieval. Quick decision map:

Google Cloud Option Choose When
Agent Search (formerly Vertex AI Search) Turnkey managed RAG: connectors, parsing, chunking, hybrid retrieval and native ACLs with the least engineering
RAG Engine Code-first managed pipeline when you need control over parsing, chunking, embeddings and the vector DB
Vector Search 1.0 / Agent Retrieval Custom embeddings and ANN at large scale inside your own pipeline (you build parsing, storage and ACL restricts)
BigQuery · AlloyDB / Cloud SQL (pgvector) In-database vectors: BigQuery for analytics; AlloyDB/Cloud SQL for operational data
Grounding with Google Search Current public web facts (ADK google_search tool)

Naming Update

Older material refers to PaLM 2, Matching Engine and Vertex AI Search. Those are now Gemini, Vector Search and Agent Search on Gemini Enterprise Agent Platform.

10

Federated Retrieval vs. Ingested RAG

Dimension Ingested RAG (Copy into Your Index) Federated Retrieval (Query Live Sources)
Freshness Behind by the sync interval; needs change capture Live (as fresh as each source's own index)
Access Control ACLs replicated onto chunks; must stay in sync Source enforces native permissions per user token
Revocations Must be propagated; the lag is an exposure window Immediate
Data Copies Another copy of sensitive data to secure No copy, but query text goes to each third party
Quality & Ranking Best: your parsing, hybrid search, and reranker over one index Limited by each source's search; scores aren't comparable across sources
Latency & Availability Predictable; independent of sources Fan-out: tail latency = slowest source; outages drop sources
Operations Pipelines, re-embedding, migrations Many APIs, OAuth tokens, quotas, schema changes

Rule of Thumb & Hybrid Patterns

Asked about quality and latency? Ingest. Asked about not copying data or live permissions/state? Federate. Both? Propose the hybrid: ingest stable, high-value corpora with ACL pre-filtering + live top-N permission re-check, and federate volatile or permission-complex systems. (Vertex AI Search connectors support both.)

11

Security, Access Control & Fairness

Vector Similarity Is Permission-Blind

Decide authorization before the prompt is built: once a restricted chunk reaches the model, no system prompt instruction reliably stops it from leaking through paraphrase.

ACL Strategy How It Works Strength Weakness
Early binding ACL groups copied onto chunk metadata; pre-filter in search Fast; always K authorized results Only as current as the last sync
Late binding Check each candidate against the source at query time Always current Slower; can return zero results
Hybrid (Recommended) Pre-filter + live re-check of the final top-N only Fast and current More moving parts

ACL Hygiene & Revocations

  • Store group IDs, not user lists; resolve user → groups at query time (short cache TTL). Apply deny-overrides-allow; isolate tenants first.
  • Derived artifacts inherit the most restrictive source ACL: parent chunks, summaries, graph communities, caches, and memories.
  • Revocation fast-lane: Revoked access hides in chunk metadata, group caches, answer caches, chat history, and agent memory. Use a metadata-only ACL fast-lane (no re-embedding), webhook cache purges, and live top-N re-checks.
  • Cache keys must include permission scope, or the cache becomes an ACL bypass.

Fairness & Replayable Decision Logs

  • End-to-end fairness: Audit all four stages where demographic skew enters: (1) the indexed corpus (uneven coverage), (2) system-prompt framing, (3) few-shot demonstrations, and (4) downstream business routing. Slice evaluation metrics by demographic subgroup.
  • Replayable decision logs: Record the request payload, exact retrieved chunks (IDs, revision timestamps, text), generated response, and downstream branch taken, indexed by a correlation ID for deterministic replay (with field minimization, TTL caps, and RBAC).
Threat / Defense Layer Architectural Control (Cheap & Fast First)
1. Deterministic filters (~1 ms) At ingestion and query time, strip hidden text, HTML comments, and zero-width characters; apply regex for known attack patterns.
2. Small classifier (Model Armor) Screen the user prompt and every retrieved chunk/tool payload before prompt assembly. Fail closed: halt if it errors or times out.
3. Isolated generation Wrap chunks in <document> delimiters as untrusted data; cap chunks and tokens per request.
4. Output checks PII/DLP filtering, citation validation, and grounding check before the answer is returned.
Confused deputy / excessive agency Least privilege at the tool registry: read-only tools and credentials for RAG agents; no send/write tools. A system-prompt instruction is not a control.
Index poisoning (fake documents injected) Ingestion approval, provenance verification, write-audit logs.
Cascading injection in multi-agent flows Validate payloads at every agent handoff.
12

Freshness, Index Updates & Migration

Update Mode Typical Lag Use For
Full rebuild Hours Small corpora, embedding-model changes
Incremental batch Minutes to hours Most documents (policies, wikis)
Streaming upsert/delete Seconds Fast-changing reference content (new tickets, KB articles), revocations (Vertex AI STREAM_UPDATE, required by RAG Engine)

Preventing Orphan & Ghost Chunks

  • Deterministic chunk IDs (doc_id + chunk index), and delete all of a document's old chunks before inserting the new version (preventing orphan chunks).
  • CDC tombstones: Propagate source deletes to evict ghost vectors and purge caches.
  • Content hashing: Re-embed only changed chunks; keep is_latest in metadata and pre-filter on it; run periodic reconciliation jobs.

Blue/Green Index Migration (7 Steps)

  • 1–2. Version & backfill from raw documents (vectors cannot be converted between models).
  • 3–5. Dual-write changes, evaluate offline on golden set (including ACLs), and run shadow/canary traffic.
  • 6. Atomic alias swap: Switch the index alias and the query embedding model at the exact same moment.
  • 7. Rollback readiness: Keep the old index warm and purge the semantic cache.
13

Semantic Caching & Latency Budget

Safe Semantic Answer Caching

Embed the incoming query and, if it's close enough to a past query, return the stored answer (~15 ms instead of ~1.2 s).

  • Entity masking before embedding: "AAPL revenue in Q3 2025" → [COMPANY] revenue in [PERIOD], keeping real values as exact-match keys so Q3 is never served for Q4.
  • Intent check: Light cross-encoder confirms the candidate hit means the same thing.
  • Thresholds by risk: Strict (≥ 0.90) for legal/financial, looser (≈ 0.75) for general FAQ.
  • ACL scope in the cache key, or the cache serves one user's answer to another.

Trap: Relying on LRU for Freshness

LRU only decides which entry to drop when the cache is full; a popular, outdated answer stays hot indefinitely. Pair LRU (a size policy) with a TTL or version-hash keys (a freshness policy). Provider prefix (KV) prompt caches are keyed on exact prefix bytes, so edited chunks simply miss: they bound cost, not correctness.

Eviction / Invalidation Policy Mechanism Controls Use For
TTL (time to live) Entry expires a fixed time after it was written Freshness (bounded staleness) Content with a known update cadence
LRU (least recently used) Evicts the coldest entry when capacity is reached Memory size only, not freshness Bounding cache cost under heavy traffic
Version-hash keys Key includes the index version and content hashes of cited chunks; any change forces a miss Instant invalidation on re-index or embedding-model migration Corpora that are re-chunked or re-embedded
Event-driven purge Document-update webhooks or CDC events delete entries citing the changed chunk Near-real-time freshness Policies, pricing, legal content
Online Pipeline Step (Cache Miss) Typical Latency Main Optimization Lever
Auth + group resolution 20–50 ms Cache group membership (short TTL)
Cache lookup 10–30 ms A hit skips everything below
Query rewrite 0 or 300–500 ms Gate it: only for follow-ups or ambiguous queries
Query embedding 30–80 ms Same-region endpoint
Parallel BM25 ‖ ANN 60–150 ms ef_search / n_probe, shard by tenant
Cross-encoder rerank 100–250 ms Fewer candidates (150 → 75), GPU batching
LLM time to first token 500–1,200 ms Fewer chunks (smaller N), prompt-cached prefix, smaller model; stream
14

Evaluation, Grading Hierarchy & Rollback Gates

Metric Type Metric Focus & What It Diagnoses
Reference-required (Offline Golden Set) Recall@K, Precision@K, MRR, nDCG@K Retrieval: Did we find the right chunks in the top K (headline metric), and are they ranked well?
Reference-required (Offline Golden Set) Context Recall, Answer Correctness End-to-End (Ragas): Alignment with ground-truth answers in the golden set
Reference-free (Online) Faithfulness (Groundedness) Generation: Are answer claims entailed by retrieved context? (NLI, LLM judge, or Vertex Check Grounding API support_score)
Reference-free (Online) Context Relevance, Answer Relevance Generation: Does context support the query, and does the answer address the question without drift?
Reference-free (Online) Refusal Accuracy (Abstention) Safety: Share of unanswerable/unsupported queries where the system cleanly refuses instead of guessing
Faithfulness (Groundedness) & The RAG Triad
Faithfulness = (Claims Supported by Context) / (Total Claims in Answer)
RAG Triad (TruLens): Context Relevance (query ↔ context), Groundedness (context ↔ answer), Answer Relevance (query ↔ answer). Trap: The RAG Triad has no recall metric—if retrieval returns one narrow chunk and the model summarizes it faithfully, all three Triad scores can be 1.0 while most of the answer is missing. Always pair with Recall@K on a golden dataset (200–500 SME-labeled questions including unanswerable, permission-restricted, and multi-turn cases; tune K on the Recall@K curve knee).

Evaluation Grading Hierarchy

Default to the fastest deterministic check and escalate only when subjective judgment is needed:

  • 1. Deterministic code assertions first: JSON schema, valid citation IDs, regex/exact matches, length bounds—sub-millisecond, free, and immune to evaluator drift.
  • 2. LLM-as-a-judge for open-ended quality: Explicit rubric, chain-of-thought prior to scoring, discrete categorical labels, structured JSON output, reference-fact grounding, and human alignment checks. Always score with a separate model family or tier from the generator to prevent self-preference bias.
  • 3. Domain-expert human grading: Reserved for high-risk edge cases, novel failure modes, and curating the gold calibration set (including multi-turn, unanswerable, and ACL-restricted cases).

Per-Stratum Rollback Gates & Hybrid Routing

Lock in pass/fail migration thresholds ahead of any model or index cutover, broken down by document category rather than relying on a blended mean.

  • Example scenario: If a cheaper model hits 0.86 overall vs. 0.94 for the flagship model, but drops to 0.71–0.74 on two complex document classes against a 0.85 per-class floor, block a full cutover.
  • Architectural fix: Route those two complex document classes to the flagship tier and send the remaining high-scoring volume to the cheaper tier.
15

Failure Modes & Exam Traps

Symptom / Scenario Root Cause → Correct Architectural Fix Tempting Wrong Answer
Answer misses half a table or condition Severed context → structure-aware or parent-child chunking, overlap Bigger LLM
"What about him?" retrieves nothing useful Conversational query → query rewriting with chat history Raising top-K
Exact codes or SKUs (ERR-742) not found Dense-only search → hybrid search (BM25 + dense) with RRF Larger embedding model
High Recall@100, poor top-3 precision Bi-encoder imprecision → cross-encoder reranking Bigger chunks
Fact buried in a long prompt is ignored Lost in the middle → fewer, better chunks at start and end Longer context window
Long sessions degrade, then stop with model_context_window_exceeded Context rot and window exhaustion → curate context, compaction, check stop_reason Bigger window
Confident answer beyond the sources Gap filled from memory → context-only rule + abstain + grounding check Fine-tuning on the documents
Multi-hop / relationship question fails Flat chunk search → query decomposition, agentic loop, or GraphRAG Raising top-K
Agent loses its plan or recalls stale preferences Cluttered context, no decay → external scratchpad.md with Read → Refine → Repeat + memory decay Bigger window
Outdated or deleted content still returned Ghost or orphan chunks → delete-before-insert, CDC tombstones, reconciliation job Post-filtering results
Wrong live order status, stock, or balance Mutable state in a vector index → live API/SQL tool on the system of record Re-embedding with a larger model
Quality drifts with no code change Corpus drift or an unpinned model alias → pin model versions, rerun golden evals, reindex Bigger model
Rankings suddenly near-random Query/index embedded with different models → same model and version; atomic blue/green migration Retuning ef_search
User sees a document they shouldn't Post-filter, stale ACLs, unscoped cache → pre-filter + live top-N re-check + ACL-scoped cache keys "Don't reveal restricted data" prompt
Instruction hidden in a retrieved doc hijacks the model Only user input screened → fail-closed chunk screening, delimiters, read-only tools Input guardrail only
High RAG Triad scores, answers incomplete Triad has no recall → measure Recall@K on a golden set Tuning the generator
Cached answer for the wrong entity or date Unmasked semantic cache → entity masking + cross-encoder intent check + webhook invalidation Exact-string cache
Popular cached answer stays outdated after a policy update LRU-only cache (size policy, no freshness) → TTL + version-hash keys + event-driven purge Larger cache capacity
16

Multimedia & Deep Dive Resources

RAG Production Blueprint Video

A comprehensive walkthrough of the RAG production architecture, retrieval pipelines, and enterprise trade-offs.

Audio Deep Dive & Study Downloads

Listen to the RAG audio deep dive or view the study guide, slide deck, and architecture infographic.

Architecting Production RAG Systems (Podcast)
Open Study Notes, Slide Deck & Infographic in Hub Viewer Launch Interactive RAG Practice Quiz →

Self-Assessment Questions (Click to Reveal)

Q1. Why should live order status, inventory counts, or account balances never be served from a RAG vector index?

A vector index stores snapshots captured at ingestion time, and similarity scores say nothing about whether a snapshot is still current. Mutable transactional state belongs in live API/SQL tool calls against the authoritative database, which return the value as of the request.

Q2. Why can't raw BM25 and dense cosine similarity scores be added together directly, and how does Reciprocal Rank Fusion (RRF) solve this?

Cosine similarity is bounded in [-1, 1] while BM25 is unbounded [0, ∞), so raw score addition lets one retriever dominate. RRF merges candidate lists using only rank positions: RRF(d) = ∑ 1 / (k + r_m(d)) (with k = 60), rewarding consensus across retrievers before a cross-encoder reranker re-scores the top pool.

Q3. Why is post-filtering unacceptable for enforcing document ACLs or excluding superseded revisions (is_latest = true)?

Post-filtering retrieves a global top-K first and then drops unauthorized or stale chunks, which can leave zero valid results (recall collapse) and leak document existence via timing. Always apply metadata pre-filtering inside the ANN and BM25 index scan, paired with a live ACL re-check on the final top-N chunks.

Q4. Input guardrails inspect the user's prompt before search. Why does that leave an enterprise RAG or agent workflow vulnerable, and what is the fix?

Adversarial instructions hidden inside retrieved documents or tool responses bypass input guardrails (indirect prompt injection). Place a fail-closed post-retrieval screening classifier over fetched chunks and tool payloads before prompt hydration, wrap chunks in <document> tags as untrusted passive data, and remove write-capable tools from the generation step.

Q5. A semantic answer cache uses LRU eviction only. After a pricing policy changes, users keep receiving the old answer. Why, and what fixes it?

LRU is a size policy: it evicts the least recently used entry only when capacity runs out, so a frequently requested answer never ages out. Add a freshness policy: a TTL matched to the content update cadence, version-hash keys that include the content hashes of cited chunks, and an event-driven purge triggered by document-update webhooks.

Q6. When evaluating a cheaper generator model for migration, overall Faithfulness is 0.86 (vs. 0.94 flagship), but two complex legal document classes score 0.72 against a 0.85 per-stratum floor. What should the architect do?

Block a full cutover based on the per-stratum rollback gate, and implement hybrid model routing: route the two complex legal document classes to the flagship model while sending the remaining high-scoring categories to the cheaper model.