Vendor-neutral guide to the data and interface layer under every grounded AI agent: the 3-phase, 7-step framework (layout-aware ingestion, normalization, structure-aware chunking, metadata & provenance, hybrid indexes, knowledge graphs & ontologies, typed retrieval tools), the progressive scaling ladder, a symptom-to-fix diagnostic sheet, and how to keep the knowledge layer multimodal, permission-aware, fresh, and measurable.
Test Yourself: Knowledge Engineering Practice Quiz →Knowledge Engineering for AI agents is the discipline of transforming raw, unstructured enterprise data into structured, traceable, and mistake-proof retrieval systems and tools (Agent-Computer Interfaces) so an LLM can reason over verified ground truth.
Core thesis: an agent is only as reliable as the ground truth returned by its tools. Even the most sophisticated reasoning loop (ReAct, Evaluator-Optimizer, Orchestrator-Workers) will confidently hallucinate if its knowledge layer drops tables, bisects clauses, ignores document versions, or misses multi-hop entity relationships.
pypdf) strip table grids, scramble multi-column layouts, and return empty strings on scans.A → B → C).search_all(query: str) tool forces the model to guess query syntax and filters.Agentic Design Patterns build the nervous system (how the agent thinks, loops, and asks for human approval). Knowledge Engineering builds the library and catalog: how raw files become clean books, indexed by topic, keyword, and citation network, with a librarian desk of specialized lookup tools.
The RAG guide covers the retrieve-then-generate pipeline end to end (embeddings, retrieval, reranking, generation, evaluation). This guide focuses on the data and interface layer underneath it: turning raw sources into clean, traceable, permission-aware knowledge and exposing it through safe agent tools.
| Dimension | Naive RAG (Prototype) | Knowledge Engineering (Production) | Memory Hook |
|---|---|---|---|
| Parsing | Raw text stream (drops tables & layout) | Layout-aware vision/OCR (Docling, Document AI; preserves tables) | "Read the page layout, not just the raw bytes" |
| Chunking | Fixed size (e.g., every 500 tokens) | Structure-aware (split by section, clause, or intact table) | "Cut at chapter seams, never mid-sentence" |
| Context | Raw text only | Rich metadata & provenance (doc_id, version, date, owner) | "Every quote carries its birth certificate" |
| Search | Vector similarity only | Hybrid (Vector + BM25) + metadata pre-filters + Knowledge Graph | "Meaning + exact IDs + relationship web" |
| Agent Access | Passive prompt stuffing (1 generic search) | Active, strongly-typed Retrieval Tools (ACI / MCP) | "Specialized scalpel tools, not one blunt shovel" |
Upgrading the foundation model (e.g., from a fast tier to a frontier tier) cannot fix an upstream knowledge engineering failure. If a parser drops a table column or a 500-token split cuts out a "provided, however, that..." exception clause, a smarter model only hallucinates more persuasively.
Instead of memorizing 7 isolated items, group them into 3 logical phases: Prepare → Index → Expose.
What: extract content from raw enterprise formats (PDFs, DOCX, HTML, scans, emails, spreadsheets) with layout-aware and OCR-capable parsers instead of raw character-stream readers.
Why: standard text extractors (pypdf) fail silently on scans, multi-column layouts, and complex tables: garbage in, confident hallucination out.
Production pattern (tiered extraction & epistemic gating):
pypdf/pdfplumber on clean, single-column digital PDFs.Docling (local layout model + TableFormer + OCR) or a managed cloud parser (e.g., Google Document AI).< 0.70) before indexing or calling an LLM.What: convert raw extracted text into a clean, canonical hierarchy: Markdown with explicit # / ## / ### headings, serialized Markdown/HTML tables, structured JSON records, and normalized entities (ISO-8601 YYYY-MM-DD dates, standardized currency/units).
Why: LLMs and downstream chunkers rely on structural markers to understand scope: which sub-clause belongs to which article, which cell belongs to which column header.
Key output: a typed document object (such as DoclingDocument) preserving reading order, section hierarchy, bounding boxes, and table grids.
What: split along logical and semantic boundaries (sections, subsections, complete tables, atomic clauses), not arbitrary fixed token counts.
Why: fixed-token chunking cuts rules from their exceptions ("The vendor is liable for all damages..." in chunk 1; "...except indirect or consequential damages" in chunk 2) and separates table rows from headers.
HybridChunker): starts from the document hierarchy, splits oversized chunks and merges undersized siblings to respect the embedding tokenizer's limit.[Doc: MSA_2026 | Section 8.2: Limitation of Liability].What: a mandatory, structured metadata envelope on every chunk:
source_uri, doc_id, section_path, page_number, bounding_box.created_date, effective_date, expiration_date, version, status ("active" vs. "superseded"), owner, access_acl / tenant_id.Solves two problems:
(Source: Section 4.2, Page 14)) that humans or auditor nodes can verify deterministically.Section 12.4(b)), acronyms, and proper nouns.Why both: dense vectors blur exact alphanumeric IDs; keyword search misses paraphrases. Run both in parallel, fuse with Reciprocal Rank Fusion, then pass the top-K (≈50) through a cross-encoder reranker (e.g., Cohere Rerank, BGE-Reranker, or a managed cloud ranking API) to keep the top-N (≈5).
What: domain knowledge as explicit nodes (entities) and directed edges (relationships) governed by a formal schema (ontology).
Why: vector and keyword indexes find passages with similar text, but fail on multi-hop relational reasoning, dependency tracing, or aggregation across isolated documents.
Contract, Party, Clause), predicates (governed_by, supersedes), and constraints (RDFS, OWL, or typed schemas).[Service: Checkout-API] ──(depends_on)──► [DB: Payments-Cluster-02]
│
(hosted_in)
▼
[Policy: PCI-DSS-v4] ◄──(governed_by)─── [Region: eu-central-1]
What: wrap hybrid indexes, metadata filters, and knowledge graphs into narrow, strongly-typed, mistake-proof functions (often exposed via MCP) that the agent invokes intentionally.
Why: one vague search_all(query: str) tool is an anti-pattern: the model must guess how to phrase queries, cannot enforce deterministic metadata constraints, and floods its own context window with noisy chunks.
Literal enums, ISO-date validators, and bounded top_k so the model cannot request 500 chunks or invent category names.chunk_text, doc_id, section_path, page_number, effective_date, so generator and auditor agents can verify citations deterministically.def get_policy_by_id(policy_id: str) -> PolicyRecord: ... # exact key / BM25 lookup def search_active_clauses( category: Literal["indemnity", "termination", "ip"], jurisdiction: str, effective_after: date, top_k: int = Field(default=5, le=10), ) -> list[ChunkWithProvenance]: ... # hybrid search with mandatory pre-filters (status="active", user ACL) def trace_system_dependencies(service_name: str, max_hops: int = 2) -> SubgraphPath: ... # deterministic graph traversal
Questions such as "total refunds by region last quarter" need exact filters, joins and aggregates, which embeddings cannot compute. Expose tables through a query tool instead of indexing rows as text, and choose the most constrained tool shape that still answers the questions users actually ask.
| Tool shape | Who writes the SQL | Use when | Main risk | Core controls |
|---|---|---|---|---|
Parameterized query toolget_refunds_by_region(region, quarter) | An engineer, ahead of time; the model only fills typed arguments | Questions are known and recurring; customer-facing paths; anything that writes | Cannot answer questions outside the tool catalog | Enum and typed arguments, bound parameters, never string concatenation |
| NL2SQL over a curated layer (approved views plus metric definitions) | The model, against a small set of documented views | Analysts ask varied questions over a well-defined domain | A wrong join or metric definition that still returns a plausible number | Approved views only, a metric glossary in context, verified example queries |
| Open NL2SQL over the schema ( execute_sql) | The model, against any table it can discover | Exploration by trusted internal users in a sandbox | Expensive scans, data exposure, injected text steering the query | Everything below, plus human review before results are shared |
SELECT from approved views. Permissions enforce safety; a prompt instruction does not.SELECT, dry-run it to estimate bytes scanned, and enforce a cost ceiling and a row LIMIT.WHERE clause.bigquery-sql and postgres-sql tools run a predefined statement with bound parameters, while bigquery-execute-sql accepts model-written SQL and offers a dry run. BigQuery conversational analytics data agents sit in the middle, grounded by authored context and verified queries.Apply progressive complexity (Rule 0 from Agentic Design Patterns): add indexing and graph layers only when corpus scale or query topology proves the simpler level falls short.
1 to a few documents per run → Level 1. Hundreds to millions of documents → go to Q2.
Fact, clause, or procedure lookup → Level 2. Entity dependencies, hierarchies, cross-document rules → Level 3.
| Level | Data & Task Profile | Steps Needed | What You Skip & Why |
|---|---|---|---|
| Level 1: In-Context Processing | Analyzing 1 to a few documents at a time (e.g., one uploaded contract or invoice that fits in context) | Steps 1, 2, 4 (Ingest → Structure → Provenance) | Skip 3, 5, 6, 7. Do not chunk and embed a 10-page document just to retrieve 5 chunks from it; parse it cleanly, attach page/section provenance, and pass the structured Markdown directly. |
| Level 2: Production Agentic RAG | Searching hundreds to millions of documents for facts, clauses, policies, or procedures | Steps 1, 2, 3, 4, 5, 7 (+ Chunking, Hybrid Index, Retrieval Tools) | Skip 6 (Knowledge Graph) when self-contained sections found by semantic + keyword search and metadata filters answer the question. |
| Level 3: Graph-Augmented RAG (GraphRAG) | Domains where relationships, dependencies, hierarchies, or cross-document rules drive the answer (supply-chain impact, microservice root cause, regulatory cross-references) | All Steps 1–7 (+ Knowledge Graph / Ontology + graph tools) | Nothing skipped: structure-aware hybrid search plus explicit graph traversal, exposed as specialized tools. |
A vector database and knowledge graph for a single 5-page contract add cost and new failure modes (missed chunks) with no benefit. Conversely, stuffing millions of documents into a prompt is impossible; the scale and the query shape choose the level.
Use this table in architecture reviews and debugging sessions to pinpoint why an agent is failing.
| Agent Failure Symptom | Broken Step | Root Cause | Engineering Fix |
|---|---|---|---|
| Misses numbers inside tables or hallucinates on scanned PDFs | 01 Ingestion / 02 Structuring | Basic stream parser dropped table structure or returned empty text on scans | Tiered layout-aware/OCR parsing + pre-LLM quality gate |
| Retrieves a general rule but misses the exception right below it | 03 Structure-Aware Chunking | Fixed 500-token splitter severed the rule from its sub-clause or table header | Split on headers/clauses (HybridChunker), keep tables intact, or use parent-child chunks |
| Cites an outdated 2022 policy or leaks another customer's record | 04 Metadata & Provenance | Chunks lack lifecycle/ACL tags, or retrieval runs without pre-filtering | Attach version, status="active", effective_date, tenant_id; enforce mandatory pre-filters |
| Finds general topics but fails on exact SKUs, clause IDs, or error codes | 05 Embeddings & Hybrid Index | Pure dense embeddings smooth over exact alphanumeric strings | Hybrid search (dense + BM25) fused via RRF + cross-encoder reranking |
| Fails on multi-hop questions ("If Vendor X defaults, which SLA in Contract Y is triggered via Subsidiary Z?") | 06 Knowledge Graph & Ontology | Flat chunks share no lexical or vector overlap across the intermediate hops | Model entities and relationships in an ontology-governed graph; expose traversal tools |
| Writes overly broad, unparseable, or hallucinated search queries | 07 Retrieval Tools (ACI) | One generic search(query) tool gives the LLM too much freedom and zero guardrails | Narrow, typed MCP tools with Literal enums, date filters, and clear docstrings (Poka-Yoke) |
The 7 steps build the knowledge layer once. A production agent also needs it to handle non-text sources, respect permissions, stay current, stay separate from agent memory, and be measured. For pipeline-level depth see the RAG guide: indexing, security, freshness, and evaluation.
| Source | Prepare it as | Provenance to keep |
|---|---|---|
| Audio (calls, meetings, voicemail) | Speech-to-text transcript with speaker labels (diarization) and timestamps; chunk by speaker turn or topic | start_time, end_time, speaker, recording_id |
| Video (training, demos, recorded incidents) | Transcript + scene/shot segments + keyframe descriptions, or one multimodal embedding per segment | Timestamp range, keyframe URI |
| Images, diagrams, scanned forms | OCR for text; a vision model's caption for figures; or a multimodal embedding model that maps images and text into the same vector space | page_number, bounding_box, image_uri |
Turn every media file into time- or region-addressable units so a citation points to "minute 12:40" or "Figure 3, page 7", not just "the video".
Trap: one shared index + a superuser credential + a prompt asking the model to "respect permissions" = a data leak waiting to happen.
status="superseded" so two versions are never both "active".embedding_model and ingested_at in metadata.| Dimension | Knowledge base | Agent memory |
|---|---|---|
| Source | Curated enterprise documents and systems of record | The agent's own interactions (sessions) |
| Scope | Shared across users, filtered by ACL | Per user or per session |
| Authority | Authoritative, versioned ground truth | User-specific preferences and facts; can be wrong or stale |
| Write path | Ingestion pipeline (Steps 1–6) with owners and review | Agent extracts and consolidates facts during or after a session |
| Lifecycle | Versions, effective dates, supersession | TTL, consolidation, user deletion requests |
Do not write unverified conversation facts into the shared knowledge base, and do not store company policy in a user's memory. See the 4-tier memory model in Agentic Design Patterns → Memory.
doc_id/chunk IDs and a reference answer.| Comparison | Key Distinction |
|---|---|
| Naive RAG vs. Knowledge Engineering | Naive RAG = raw stream → fixed 500-token chunks → vector-only → prompt stuffing. Knowledge Engineering = layout parsing → structure chunking → provenance → hybrid + graph → typed ACI tools. |
| Prepare vs. Index vs. Expose | Prepare (Steps 1–4) makes data clean & traceable; Index (Steps 5–6) makes it findable by meaning, keyword & relationship; Expose (Step 7) turns indexes into mistake-proof agent tools. |
| Fixed-token vs. structure-aware chunking | Fixed-token cuts every N tokens (severing rules from exceptions and tables from headers); structure-aware splits on headings, clauses, and intact tables. |
| Metadata pre-filtering vs. post-generation citation | Both rely on Step 4: pre-filtering enforces ACLs, recency, and active status before search; provenance enables deterministic citation checks after generation. |
| Dense vs. BM25 vs. Hybrid | Dense = meaning & synonyms; BM25 = exact SKUs, error codes & clause numbers; Hybrid = both, fused via RRF + cross-encoder reranking. |
| Hybrid RAG (Step 5) vs. GraphRAG (Step 6) | Hybrid RAG retrieves relevant text passages; GraphRAG traverses explicit entity-relationship edges for multi-hop reasoning and dependency analysis. |
| Ontology vs. Knowledge Graph | Ontology = the formal schema (node types, edge predicates, constraints); Knowledge Graph = the populated instances that conform to it. |
Generic search(query) vs. specialized tools | Generic search forces the LLM to guess syntax; specialized ACI/MCP tools constrain arguments with typed schemas (Literal, dates, IDs). |
| Knowledge base vs. agent memory | Knowledge base = curated, shared (ACL-filtered), versioned ground truth; memory = per-user/session facts extracted from interactions, with TTL and consolidation. |
| Retrieval vs. generation metrics | Retrieval = recall@k, precision@k, MRR/nDCG against golden chunk IDs; generation = groundedness, citation accuracy, answer correctness. Measure both separately. |
PHASE 1: PREPARE (Clean & Traceable) 01. Ingestion Layout + OCR parsing (tables, scans, PDFs) 02. Structuring Raw text → Markdown sections / JSON / tables 03. Chunking Split by structure (headers/tables), not every N tokens 04. Provenance Tag doc_id, date, version, section, ACL PHASE 2: INDEX (Findable) 05. Hybrid Index Dense vectors (meaning) + BM25 (exact IDs) + RRF + rerank 06. Knowledge Graph Entity → Relation → Entity (multi-hop rules) PHASE 3: EXPOSE (Actionable ACI) 07. Retrieval Tools Domain-specific, typed MCP tools (Poka-Yoke) SCALING LADDER Level 1 (In-context, 1–few docs): Steps 1, 2, 4 Level 2 (Production agentic RAG): Steps 1, 2, 3, 4, 5, 7 Level 3 (GraphRAG, multi-hop): All steps 1–7
Docling, Google Document AI, LlamaParse) to preserve reading order and tables.doc_id, section_path, page_number, version, effective_date, access_acl) on every chunk for pre-search filtering and post-generation citation checks.Listen to the 55-minute Knowledge Engineering audio deep dive, open the 21-slide deck and sketchnote infographic in the Hub Viewer, or drill scenario-based questions on layout-aware parsing, structure-aware chunking, metadata pre-filters, hybrid search, knowledge graphs, typed retrieval tools, and ACL enforcement.
Q1. What are the 3 phases and 7 steps of Knowledge Engineering for AI agents?
Q2. Why is arbitrary 500-token chunking dangerous for agents?
Q3. Why do production systems use hybrid search (vector + BM25) instead of vectors alone?
Q4. When is a Knowledge Graph (Step 6) justified over standard hybrid RAG (Step 5)?
Q5. What two problems does Metadata & Provenance (Step 4) solve?
Q6. Why expose several specific retrieval tools instead of one generic search(query) tool?
Q7. Which steps do you need to analyze a single 5-page contract that fits in the context window?
Q8. You switch to a newer embedding model. Can you embed only new documents with it?
embedding_model in metadata.