# Machine Learning Certification Hub — Full Text > Independent, open-source study guides, practice quizzes, infographics, decks, videos and podcasts on vendor-neutral AI foundations and on Google Cloud and Anthropic Claude agentic AI products. Independent study notes. Not affiliated with or endorsed by Google or Anthropic. Product and certification names are trademarks of their owners. Quiz questions are original practice material, not exam questions. This file concatenates the full Markdown source of every study guide listed in https://mbettan.github.io/machine-learning-cert/llms.txt (generated from https://mbettan.github.io/machine-learning-cert/catalog.json). Each guide starts with a header block. ## Contents - ML 101 — https://mbettan.github.io/machine-learning-cert/ML101.html - Generative AI 101 — https://mbettan.github.io/machine-learning-cert/GenAI101.html - Prompt Engineering — https://mbettan.github.io/machine-learning-cert/PromptEngineering.html - Agentic Design Patterns — https://mbettan.github.io/machine-learning-cert/ADP.html - Context Engineering & Reliability — https://mbettan.github.io/machine-learning-cert/ContextEngineering.html - Agent Development Kit (ADK) — https://mbettan.github.io/machine-learning-cert/ADK.html - Model Context Protocol (MCP) — https://mbettan.github.io/machine-learning-cert/MCP.html - Agent2Agent (A2A) — https://mbettan.github.io/machine-learning-cert/A2A.html - Retrieval-Augmented Generation (RAG) — https://mbettan.github.io/machine-learning-cert/RAG.html - Knowledge Engineering — https://mbettan.github.io/machine-learning-cert/KnowledgeEngineering.html - AgentOps: Evals, Observability & Cost — https://mbettan.github.io/machine-learning-cert/AgentOps.html - Responsible AI: Guardrails, Safety & Governance — https://mbettan.github.io/machine-learning-cert/ResponsibleAI.html - Vibe Coding — https://mbettan.github.io/machine-learning-cert/vibecoding.html - Spec-Driven Development (IDD) — https://mbettan.github.io/machine-learning-cert/idd.html - Gemini & Google Models — https://mbettan.github.io/machine-learning-cert/Google.html - Gemini Enterprise & Low-Code Agents — https://mbettan.github.io/machine-learning-cert/GeminiEnterprise.html - Antigravity & Agents CLI — https://mbettan.github.io/machine-learning-cert/Antigravity.html - Agent Platform: Build, Retrieve & Run — https://mbettan.github.io/machine-learning-cert/AgentPlatform.html - Agent Platform: Identity, Security & Governance — https://mbettan.github.io/machine-learning-cert/AgentPlatformSecurity.html - Anthropic & Claude Models — https://mbettan.github.io/machine-learning-cert/Anthropic.html - Claude Platform & API — https://mbettan.github.io/machine-learning-cert/ClaudeAPI.html - Claude Code — https://mbettan.github.io/machine-learning-cert/ClaudeCode.html - Claude Agent SDK — https://mbettan.github.io/machine-learning-cert/ClaudeAgentSDK.html ================================================================================ GUIDE: ML 101 URL: https://mbettan.github.io/machine-learning-cert/ML101.html Practice quiz: https://mbettan.github.io/machine-learning-cert/quiz.html?topic=ml101 Section: Foundation · AI Fundamentals Exam relevance: Google Professional Agentic Architect (supporting), Claude Certified Architect – Foundations (supporting) ================================================================================ # Machine Learning 101: Study Guide > **Author:** Michaël Bettan Definition: Process of combining inputs to produce useful predictions on never-before-seen data. --- ## Definitions & Scope - **Artificial Intelligence (AI):** Computer think/act like humans. - **Machine Learning (ML):** Scalably solve problems using data examples (not rules). - **Deep Learning (DL):** Subtype that works even when the data consists of unstructured data, like images, speech, video, natural language text and so on. Approach based on Neural Networks. > **Core Distinction:** The basic difference between Machine Learning and other techniques in AI is that in Machine Learning machines learn. ## Standard Workflow & Lifecycle #### Problem Definition What business problem are you solving for? What is the outcome you want to achieve? #### Data Extraction Determine the data you need for training and testing your model based on the use case. #### Data Preparation Your data needs to be properly formatted before and after data import (iteratively). #### Model Training Set parameters and build your model. #### Model Troubleshooting Troubleshoot the performance. #### Model Evaluation Review model metrics and performance. #### Model Tuning Adjust parameters not learned from data. #### Model Testing Try your model on test data. #### Model Serving Make your model available to use. #### Model Monitoring Keep your model accurate. ## ML Problem Types ### Supervised Learning Learn from examples, apply label to data. - **Binary classification model:** predicts a binary outcome (one of two classes) → yes or no questions. Example: Credit card transactions (fraud or not). - **Multi-class classification model:** predicts one class from three or more discrete classes (one of a set). Example: segment customers into different personas. - **Regression model:** predicts a continuous value. Example: forecast customer spending over the next month. - **Matrix Factorization:** (matrix decomposition) used in recommender systems. Example: Netflix. ### Unsupervised & Reinforcement **Unsupervised Learning:** detect unknown patterns w/o labeled examples. - **Clustering model:** identifying similarities in groups. - **Anomaly Detection:** identifying abnormalities in data. **Reinforcement Learning:** - Learn from the environment via exploration and exploitation. Use positive/negative reinforcement to complete a task. Example: chess, maze, etc. ## Data Extraction & Preparation ### Data Extraction - **Select relevant features:** A feature is an input attribute used for model training. Features are how your model identifies patterns to make predictions, so they need to be relevant to your problem. - **Include enough data:** The more training examples you have, the better your outcome. Amount of example data required scales with the complexity of the problem you’re solving for. - **Capture variation:** Your dataset should capture the diversity of your problem space. The more diverse examples a model sees during training, the more readily it can generalize to new or less common examples. > **Data Preparation Pitfalls:** **Prevent data leakage:** when you use input features during training that "leak" information about the target that you are trying to predict which is unavailable when the model is actually served. This can be detected when a feature that is highly correlated with the target column is included as one of the input features. Strong model performance during testing, but not when deployed in production. > **Prevent prediction-serving skew:** avoiding a mismatch between the data or processing used during model training and what is available at prediction (inference) time. > **Clean up data:** clean up missing, incomplete, and inconsistent data to improve data quality before using it for training purposes. ### Model Training Fundamentals - **Feature engineering:** transforming input features to be more useful for the models. e.g. mapping categories to buckets, normalizing between -1 and 1, removing null. - **Model selection and iterations.** ## Model Troubleshooting ### Underfitting Poor performance on training and test dataset. - Increase the complexity of model - Increase the training time ### Overfitting Good perf on training, poor perf on test dataset. - Specific to the trained data, does not generalize. - **Reasons:** not enough training data, too many features (redundant), features not useful (noise). - **Combat via:** Regularization which limits information captured. - **Combat via:** Early Stopping (halting when validation loss increases). - **Combat via:** L1 / L2 Regularization. - **Combat via:** Dropout (randomly dropping nodes). - **Combat via:** Max-Norm Regularization. - **Combat via:** Data Augmentation (artificially boosting dataset diversity). ### Bias Missing relationship between features and labels → training data is not generalized. **Ideal state:** Low bias, low variance. ### Variance Sensitivity in small fluctuations in the training dataset → difference in a set of predictions. **Ideal state:** Low bias, low variance. ## Model Evaluation **Model evaluation metrics** are based on how the model performed against a slice of your dataset (the test dataset). ### Classification Metrics Score thresholds to adjust the confusion matrix (true positive, true negative, false positive and false negative). - **Accuracy:** % of total true positive & true negative / total population. - **Precision:** % of total positive class correctly classified by TP / (TP + FP). - **Recall:** % of total actual positive labels identified by TP / (TP + FN). - **F1 score:** 2 * (Precision * Recall) / (Precision + Recall). - **AUC PR, AUC ROC, Log loss:** Common classification performance metrics. ### Regression Metrics **Mean squared error:** - MAE (Mean Absolute Error) - RMSE (Root Mean Squared Error) - RMSLE (Root Mean Squared Logarithmic Error) ### Gradient Descent & Backpropagation - **Gradient Descent:** technique to minimize the loss in a neural network by calculating gradient (slope) → optimal weight. - **Batch Gradient Descent:** → slow large dataset. - **Stochastic Gradient Descent:** → random process. - **Mini-Batch Gradient Descent:** → mixed mode. - **Backpropagation:** algorithm mapping from the input-output pair to calculate the gradient descent → efficient way. > **Model Tuning & Testing:** **Hyperparameter tuning:** Adjusting the parameters of our models not from the data but the model (e.g., # layers, learning rate, etc.). > **Model Testing:** Evaluating your model metrics is primarily how you can determine whether your model is ready to deploy, but you can also test it with new data. Try uploading new data to see if the model's predictions match your expectations. Based on the evaluation metrics or testing with new data, you may need to continue improving your model's performance. > *Consideration: Never test with training data.* ## Model Serving & Monitoring ### Batch prediction Is useful for making many prediction requests at once. Batch prediction is asynchronous, meaning that the model will wait until it processes all of the prediction requests before returning the predicted values. ### Online prediction Is synchronous (real-time), meaning that it will quickly return a prediction, but only accepts one prediction request per API call. Online prediction is useful if your model is part of an application and parts of your system are dependent on a quick prediction turnaround. ### Model Monitoring - **Metrics to monitor:** traffic patterns, error rates, latency, resource utilization with Cloud Monitoring alerting. - **Data Drift:** the input data changes over time → refresh models. (Not to be confused with *training-serving skew*, a mismatch between training and serving data.) - Watch for changes in dependencies upstream in the pipeline. - Assess model prediction quality. - Test for unfairness (bias unintentionally). ## User Personas and User Stories - **Product Managers:** Insights and objectives. - **Data Analyst:** Query and analyze. - **Data Engineer:** Get clean and useful data. - **Data Scientist:** Models that work. - **ML Developer:** Intelligent applications. - **ML Engineer:** Models in production. > **ML Ops & Operationalize:** “DevOps” / automated operations for machine learning. ## Neural Networks ### Architecture & Components - **Neural Networks:** = input layer + hidden layer to output. Model composed of layers, consisted of neurons. - **Neuron:** node → combine input values to one output value. - **Epoch:** single pass through training dataset. - **Weight:** multiplication of input values. - **Bias:** value of output given a weight of 0. - **Hidden layers:** set of neurons operating from same input set. ### Network Types - **Feedforward neural network (FFNN):** information moves strictly forward from input to output. - **Recurrent neural network (RNN):** optimized for sequential data, where previous runs feed into the next. - **Deep Neural networks:** = generalization, many hidden layers. - **Wide Neural networks:** = memorization, many features. - **Deep and Wide networks:** generalization + memorization. Good fit for recommendation engines. ## Glossary ### Core Concepts - **Inference:** = scoring = predictions: applied trained model to unlabelled examples. - **Label:** The variable we are predicting (target). - **Features:** Input data used by the ML model. - **Feature Store:** Rich feature repository to serve, share and re-use ML features. - **Training-Serving Skew:** Mismatch between input features at training time and serving time. ### Data Splits - **Training Set:** Labeled examples used to optimize the model. - **Validation Set:** Disjoint subset used to adjust hyperparameters and prevent overfitting. - **Test Set:** Subset used to provide final results on new data (contains labels, model never learns from them). ### 3 Types of "Bias" - **Bias (Math):** Intercept or offset from an origin (the b in y = wx + b). - **Prediction Bias:** Difference between average of predictions and average of labels in the dataset. - **Bias (Ethics/Fairness):** Unintentional unfairness or stereotyping in algorithms or data. ### Neural Networks & Optimization - **FFNN:** Neural network without recursive connections; information moves strictly forward. - **RNN:** Network optimized for sequential data where previous runs feed into the next. - **Activation Functions:** Mathematical functions (e.g. RELU, tanh) that introduce non-linearity. - **Optimizer:** Operation that changes weights and biases to reduce loss e.g. Adagrad or Adam. - **Gradient Descent:** Technique to minimize loss by calculating gradient (slope) to find optimal weight. - **Backpropagation:** Efficient algorithm to calculate gradient descent in neural networks. - **Learning Rate:** Rate at which optimizers adjust weights and biases; risks non-convergence if too high. - **Converge:** State where loss stabilizes and the algorithm reaches an optimal answer. ### Preventing Overfitting in DNNs (Regularization) - **L1 Lasso Regularization:** Penalizes absolute weight values; drives least useful features to 0. - **L2 Ridge Regression:** Penalizes squared weights; keeps weights approximately equal in size. - **Dropouts methods:** Randomly dropping nodes during training to improve generalization error. - **Early Stopping:** Ending training when validation loss starts to increase to prevent overfitting. - **Max-Norm Regularization:** Limiting the absolute magnitude of network weights. - **Data Augmentation:** Transforming existing examples (e.g. rotating images) to artificially boost dataset diversity. ### Advanced Techniques & Data Processing - **Ensemble Learning:** Combining predictions from multiple distinct models to solve the same problem. - **Neural Architecture Search (NAS):** Automated approach to designing and selecting the best model ensemble. - **Embeddings:** Mapping discrete objects (like words) to vectors of real numbers. - **One-Hot encoding:** Mapping attribute values to a bit in a binary array. - **Normalization:** Converting numeric values to a standard range, e.g. -1 and 1. - **Tensor:** N-dimensional arrays of numbers; primary data structure in ML. ### Self-Assessment Questions Q1. What is the difference between Overfitting and Underfitting? Q2. What is "Data Leakage" in machine learning? Q3. What is the difference between Supervised and Unsupervised learning? ================================================================================ GUIDE: Generative AI 101 URL: https://mbettan.github.io/machine-learning-cert/GenAI101.html Practice quiz: https://mbettan.github.io/machine-learning-cert/quiz.html?topic=genai101 Section: Foundation · AI Fundamentals Exam relevance: Google Professional Agentic Architect (supporting), Claude Certified Architect – Foundations (supporting) ================================================================================ # Generative AI 101: Study Guide (v3.0) > **Author:** Michaël Bettan · **Version:** 3.0 (October 2026) Comprehensive foundation guide to Generative AI: Sparse MoE Transformers and native multimodality, the 9-step engineering workflow, PAA Section 3.1 model selection (LLM vs. SLM, Open-Weight vs. Proprietary, Managed SaaS/MaaS vs. Self-Hosted VPC), System 2 inference-time reasoning (`thinking_level` & Adaptive Thinking), prompting, training (SFT, RLHF, DPO, RLAIF, RLVR), L0–L4 agent capability levels, the 5-Layer Agentic Protocol Stack (`MCP`, `A2A`, `MCP UI / AG-UI / A2UI`, `Live API`, `AP2 + x402`), RAG, risks, and glossary. --- ## 01. Definitions & Scope ### What is GenAI? **Generative AI (GenAI)** encompasses machine learning models designed to generate novel content (text, images, audio, video, structured data, and executable code) by predicting patterns learned across massive training corpora. - **LLMs (Large Language Models):** The foundation of text- and code-based GenAI. Built on the **Transformer architecture** and **self-attention mechanisms** to weigh token relevance across long context windows (up to 1M–2M+ tokens). Modern frontier models predominantly use **Sparse Mixture-of-Experts (MoE)** routing—activating only a small subset of specialized expert sub-networks per token so total parameter capacity scales without multiplying per-token inference latency or FLOPs. - **Multimodal models:** Built from the ground up (*native early-fusion*) to ingest, reason across, and generate multiple modalities (text, images, audio, video, PDFs) inside a unified representation space (*examples: Google Gemini 4 Argon / 3.8 Flash, Anthropic Claude Opus 4.7 / Sonnet 4.6, OpenAI GPT-5 / GPT-4o*). - **Native Multimodal vs. Modular Perception Pipelines:** Native multimodal ingestion preserves visual layout, audio prosody, and cross-modal spatial relationships without OCR/ASR information loss. However, **modular pipelines** (such as dedicated Speech-to-Text or Document AI Layout Parser stages ahead of an LLM) remain preferable when compliance requires deterministic **PII redaction on intermediate transcripts** before data reaches the foundation model, strict bounding-box coordinate auditability, or ultra-cheap batch pre-filtering. - **Synthetic media:** The umbrella industry term for AI-generated text, images, audio, video, and tabular/training data—replacing the narrower legacy term "deepfake," which applies only to deceptive likeness synthesis. --- ## 02. Standard Workflow 1. **Define Objective:** Specify the business goal, target audience, latency/cost envelope, tone, and structured output schema. 2. **Model Selection:** Match task complexity, modality, and governance constraints to the right model tier and hosting boundary (frontier LLM vs. specialized SLM, SaaS/MaaS vs. dedicated VPC endpoint). 3. **Prompt & Context Engineering:** Craft clear system instructions, schemas, and curated context windows. 4. **Generation & Iteration:** Run baseline evaluations, inspect traces, collect human/automated feedback loops (thumbs up/down, rubric scores), and iterate. 5. **Troubleshooting:** Diagnose lazy completions, hallucinations, context truncation, or refusals by adjusting reasoning effort, decoding parameters, or context structure. 6. **Evaluation & Fact-Checking:** Cross-reference outputs with deterministic checks, autoraters, and human oversight—GenAI is probabilistic, never infallible. 7. **Advanced Customization:** Climb the customization ladder deliberately: **Prompting → RAG / Grounding → Supervised Fine-Tuning (SFT) → Preference Alignment (DPO/RLHF)**. 8. **Finalizing & Editing:** Harmonize voice, verify citations and code dependencies, and apply human editorial judgment. 9. **Deployment & Ethics:** Attach provenance/watermarks (`SynthID`, `C2PA`), enforce runtime guardrails, cite sources, and monitor production telemetry for model or data drift. --- ## 03. Ecosystem & Key Players Selecting the right model family depends on reasoning complexity, multimodal requirements, latency budget, licensing, and data-residency rules: - **Google (Gemini & Gemma):** Native multimodal Sparse MoE frontier models (**Gemini 4 Argon** for complex multi-step reasoning and agentic coding, **Gemini 3.8 Flash** as high-throughput workhorse, **Gemini 3.8 Live** for sub-second bidirectional voice/video, and **Gemini 3.5 Flash-Lite** for high-QPS routing) integrated across Google Cloud Agent Platform, Search grounding, and Workspace—paired with **Gemma 4** (`E2B`, `E4B`, `26B A4B MoE`, `31B Dense`) open-weight models and DeepMind research (`SynthID`, `AlphaEvolve`). - **Anthropic (Claude):** Safety-first frontier models (**Claude Opus 4.7**, **Claude Sonnet 4.6**, **Claude Haiku 4.5**) pioneering **Constitutional AI (RLAIF)**—aligning outputs to an explicit written constitution rather than relying solely on ad-hoc human ratings—alongside native **Adaptive Thinking**, 1M-token context engineering, mechanistic interpretability, and computer/tool use. - **Meta (Llama):** Broad open-weight model ecosystem empowering organizations to fine-tune, quantize, and deploy weights across cloud Model-as-a-Service (MaaS) catalogs, private VPCs, or air-gapped hardware. - **DeepSeek:** Demonstrated that **Sparse Mixture-of-Experts (MoE)** architectures, multi-head latent attention, and reinforcement-learned reasoning distillation can rival massive proprietary models in math and coding at a fraction of active compute. - **OpenAI (ChatGPT / GPT-5 / o-series):** Early pioneer of commercial conversational LLMs (`GPT-4`, `GPT-4o`) and inference-time reasoning scaling (`o1` through `o3`/`GPT-5`). --- ## 04. Model Selection Matrix: LLM vs. SLM, OSS vs. Proprietary & Self-Hosted vs. SaaS Architectural exams (especially **Google PAA Section 3.1** and **CCAR-F**) test model selection across **three orthogonal axes**. Never conflate model *size*, *weight licensing*, and *hosting topology*. ### Axis 1: Capacity Tier — Frontier LLM vs. Small Language Model (SLM) | Dimension | Frontier LLM (`Gemini 4 Argon`, `Claude Opus 4.7`) | Workhorse Mid-Tier (`Gemini 3.8 Flash`, `Claude Sonnet 4.6`) | Small Language Model / Lite (`Gemini 3.5 Flash-Lite`, `Gemma 4`, `Claude Haiku 4.5`, `Gemini Nano`) | | :--- | :--- | :--- | :--- | | **Best for** | Ambiguous multi-step planning, cross-document synthesis (1M+ tokens), complex debugging, novel edge cases | 80–90% of production agentic tool-calling, RAG synthesis, code generation, customer workflows | Intent routing, schema extraction, PII/safety classification, high-QPS batch tagging, on-device/edge inference | | **Latency & Cost** | Highest TTFT & per-token cost; supports deep System 2 reasoning | Balanced sub-second TTFT and strong reasoning-per-dollar | Ultra-low latency (<150 ms) and 10×–30× lower token cost; fits single GPU or mobile NPU | | **Failure Mode** | Overkill latency and token burn on deterministic or single-step tasks | May miss subtle 6+ hop constraints without explicit decomposition | Brittle on open-ended multi-hop planning or unstated domain assumptions | ### Axis 2: Weight Licensing — Proprietary Frontier vs. Open-Weight (OSS) - **Proprietary Models (`Gemini`, `Claude`, `GPT`):** Deliver peak zero-shot reasoning, native 1M+ multimodal context, built-in safety filters, and managed vendor SLAs—at the trade-off of API lock-in and vendor model-retirement lifecycles (requiring version aliases and migration eval suites). - **Open-Weight Models (`Gemma 4`, `Llama`, `Mistral`, `DeepSeek`):** Provide **weight permanence** (frozen checkpoints never deprecated by a vendor), deep **LoRA / QLoRA / full-parameter fine-tuning**, custom quantization (`FP8` / `INT4`), and portability across clouds or air-gapped environments. - **Safety Caveat:** Open weights ship without managed platform guardrails; production deployments must wrap inputs and outputs in an external runtime safety layer (such as **Model Armor** or **ShieldGemma 2**). ### Axis 3: Hosting & Consumption — Managed SaaS / MaaS vs. Dedicated Self-Hosted VPC > **Exam Trap — Open-Weight ≠ Self-Hosted GPU Ops:** Choosing an open-weight model (such as Llama or Gemma 4) does **not** automatically mean provisioning GPUs on Kubernetes. Always evaluate **Managed Model-as-a-Service (MaaS)** serverless endpoints first before paying the 24/7 idle tax of dedicated nodes. | Consumption Mode | Billing Model | Ops Overhead | Choose When | | :--- | :--- | :--- | :--- | | **Serverless SaaS / MaaS API** (First-party Gemini/Claude or Model Garden MaaS Partner/OSS) | **Pay-As-You-Go per token** (or reserved **Provisioned Throughput** in GSUs for guaranteed latency SLAs) | Zero infra management; instant autoscaling to zero | Spiky or unpredictable traffic, rapid iteration, standard enterprise privacy (zero training on customer prompts, VPC-SC, regional endpoints) | | **Dedicated Private Endpoint** (Managed Model Garden Endpoint) | **Per node-hour (GPU/TPU)** — **bills 24/7 even at 0 QPS** until undeployed | Managed container serving, manual replica/autoscaling sizing | Strict regulatory mandate prohibiting shared multi-tenant SaaS compute, hosting custom LoRA/fine-tuned open weights, or steady high-duty-cycle volume | | **Self-Hosted on GKE / Bare Metal** (`vLLM`, `JetStream`, `TGI` on GPU/TPU nodes) | **Cluster compute + storage + networking** | Highest: driver patching, KV-cache tuning, autoscaler ops | Full custom serving kernels, multi-LoRA bin-packing across dozens of adapters, or completely air-gapped / sovereign environments | ### The Multi-Model Routing Cascade & Customization Ladder 1. **Multi-Model Routing Cascade:** Do not send every turn to a frontier model. Place a fast **SLM classifier/router** (`Gemini 3.5 Flash-Lite` or `Haiku 4.5`) in front to handle simple lookups and route standard tool loops to a mid-tier model (`Gemini 3.8 Flash` / `Sonnet 4.6`), escalating to a frontier reasoning model (`Gemini 4 Argon` / `Opus 4.7`) **only** on low confidence, complex planning, or verification failures. 2. **The Customization Ladder:** - Need **fresh, mutable, or citable facts** with per-user ACLs? Use **RAG / Grounding**, **not** fine-tuning (weights freeze stale facts and cannot enforce row-level permissions). - Need **consistent domain tone, terse output formatting, or specialized syntax** (so you can shrink bloated few-shot prompts into a fast SLM)? Use **Supervised Fine-Tuning (SFT)** or **Model Distillation**. --- ## 05. System 1 vs. System 2 Reasoning: Inference-Time Compute Early LLMs operated purely in **System 1** mode—emitting the final answer token-by-token in a single forward pass with fixed compute per token. Modern frontier models add **System 2 reasoning** (*inference-time / test-time compute scaling*): spending variable internal **thinking tokens** to decompose problems, test hypotheses, catch mistakes, and plan between tool calls before emitting user-visible text. ### System 1 vs. System 2 Comparison - **System 1 (Fast / Non-Reasoning or Minimal Effort):** Immediate autoregressive decoding. Lowest Time-to-First-Token (TTFT) and lowest output-token cost. Ideal for classification, entity extraction, translation, straightforward summarization, and low-latency voice turn-taking. - **System 2 (Deliberative / Extended Reasoning):** Generates an internal chain of thought trained via reinforcement learning before responding and **interleaves reasoning between tool results** (*observe tool error → reason about root cause → choose corrective tool*). Essential for ambiguous multi-constraint planning, root-cause code debugging, theorem/math verification, and policy conflict resolution. ### 2026 Vendor Reasoning Controls | Capability | Google Gemini (`Gemini 4 Argon` & `Gemini 3.8`) | Anthropic Claude (`Claude Opus 4.7` & `Sonnet 4.6`) | | :--- | :--- | :--- | | **Primary Control Knob** | **`thinking_level`**: `MINIMAL`, `LOW`, `MEDIUM`, `HIGH` (replaces legacy manual token-count `thinking_budget` on Gemini 4) | **Adaptive Thinking** (`thinking: {type: "adaptive"}`) paired with **`output_config.effort`**: `low`, `medium`, `high`, `xhigh` (long-horizon coding/agents), `max` | | **Multi-Turn Tool State** | Emits encrypted **`thought_signatures`** alongside tool calls; the agent harness **must pass them back verbatim** on subsequent turns so multi-step reasoning state is preserved | Native **interleaved thinking** between tool calls (`budget_tokens` manual mode is deprecated on 4.6 and removed on Opus 4.7) | | **Real-Time Voice Bridge** | **`gemini-3.8-live-extended-thinking`** decouples fast conversational turn-taking from deeper background System 2 reasoning | Pair fast voice/UI acknowledgement with asynchronous tool/subagent execution | | **Sampling Interaction** | Keep default sampling settings when `thinking_level` is active to avoid destabilizing reasoning chains | Custom `temperature` / `top_p` / `top_k` (4.7+) and assistant message prefilling (4.6+) are removed; steer behavior via `effort` and `output_config.format` schemas | > **Cost & Latency Warning:** Internal thinking tokens are **billed as output tokens** and delay visible TTFT unless streamed as thought summaries. Never default every microservice call to `HIGH` / `max`—calibrate reasoning effort per route using offline evaluation sets. --- ## 06. Prompt Engineering & Frameworks ### Prompting Frameworks Reusable, structured templates designed to build consistent, high-quality prompts instead of starting from scratch: - **ReAct (Reasoning + Acting):** Alternates between reasoning steps and external tool calls (*Thought: I need live flight inventory → Action: Call Flight API → Observation: 3 seats left*). Foundational loop for AI agents. - **PACT:** Defines **P**ersona, **A**ction, **C**ontext, and **T**one. Best for marketing, copywriting, and maintaining brand consistency. - **WISER:** Defines **W**ho (Persona), **I**nstruction (Main task), **S**ubtasks (Decomposition), **E**xamples (Few-shot), and **R**eview (Self-correction). Best for complex, structured single-call workflows. ### Core Prompting Techniques - **Prompt:** A natural-language instruction, schema, and context payload sent to the model. Outputs remain probabilistic—effective engineers specify explicit constraints, edge-case rules, and verification criteria. - **Iterative Prompting:** A flexible cycle of prompting, evaluating against test cases, and refining instructions until output meets quality bars. - **Prompt Chaining:** Breaking a complex task into a fixed sequence of smaller prompts where each step's validated output feeds the next step. - **Meta-Prompts:** Prompts that inspect, critique, or generate other prompts (e.g., *"Analyze these failure traces and propose a tighter system prompt"*). - **Few-Shot Learning:** Providing 2–5 representative input-output examples inside the prompt to anchor edge-case formatting and classification boundaries. - **System Prompt:** High-priority meta-instructions defining the model's role, invariants, tool policies, and refusal boundaries. - **Chain-of-Thought (CoT):** Asking a non-reasoning model to *"think step-by-step"* in visible output tokens to improve logic and math accuracy. *(Note: On native System 2 models with `thinking_level` or Adaptive Thinking enabled, avoid rigid step-by-step CoT boilerplate—let native thinking handle internal deliberation.)* - **Subtractive method:** Best for content creators—start with a broad draft prompt and issue follow-up prompts to whittle away unwanted elements. - **Additive method:** Best for visual artists and scene builders—start with a foundational composition/background prompt and iteratively layer lighting, details, and textures. --- ## 07. How Models Are Trained ### Training Phases 1. **Pretraining:** The base model ingests trillions of tokens of diverse web, code, and multimodal data using **self-supervised next-token prediction**, acquiring broad world representations and syntax—but not reliable instruction-following or safety boundaries. 2. **Post-Training (Alignment & Reasoning):** Adapts the base checkpoint for instruction adherence, safety, and multi-step reasoning: - **SFT (Supervised Fine-Tuning):** Trains on curated, high-quality `(instruction, ideal_response)` demonstrations so the model follows user formatting and task structure. - **RLHF (Reinforcement Learning from Human Feedback):** Humans rank candidate responses to train a separate **Reward Model**, which then updates policy weights (e.g., via PPO) toward helpful, harmless outputs. - **DPO (Direct Preference Optimization):** Directly optimizes model weights on `(chosen, rejected)` preference pairs using a closed-form classification loss—bypassing the instability and memory cost of training a separate reward model. - **RLAIF (Reinforcement Learning from AI Feedback / Constitutional AI):** Uses a capable critique model guided by explicit written principles (a constitution) to generate preference labels and self-revisions at scale. - **RLVR (Reinforcement Learning with Verifiable Rewards):** Trains System 2 reasoning models against deterministic verifiers—unit-test suites, compilers, math proof checkers, and sandbox execution—rewarding multi-step chains that arrive at objectively verified solutions. 3. **Model Distillation:** Transfers domain behavior or reasoning traces from a large frontier **"Teacher"** model into a compact **"Student"** SLM, preserving task accuracy at a fraction of serving latency and cost. --- ## 08. Advanced Tactics & Troubleshooting ### Multi-Model Composition Tactics - **AI Aggregation:** Combining specialized outputs from multiple distinct GenAI models into one deliverable (e.g., Claude for technical prose, Nano Banana / Imagen for illustrations, Veo for video). - **Output Stitching:** Running multiple models in parallel on the *same* prompt, then selecting and merging the strongest segments from each response. - **AI Chaining:** Passing the output of one GenAI model or modality directly as the input prompt/asset to a downstream GenAI tool. - **Multi-Persona Prompting:** Instructing a single model call to critique a proposal from several distinct stakeholder viewpoints (e.g., *"Evaluate this launch plan as a CISO, a CFO, and an end customer"*). ### Troubleshooting Failure Modes - **The "Finger Problem":** Visual/structural anomalies (such as six-fingered hands in early image generators) caused by ambiguous statistical co-occurrence in training data rather than true physical-world comprehension. - **Model / Data Drift:** Degradation when real-world facts, APIs, or user behavior diverge from the model's static training cutoff. *Combat via: RAG/search grounding, dynamic context injection, or periodic re-tuning.* - **Overfitting (Memorization):** Training too narrowly on repeated records so the model regurgitates training text verbatim instead of generalizing. *Combat via: deduplicating training sets, regularization, adjusting sampling, and output similarity/plagiarism filters.* - **Lazy Models:** Premature truncation, placeholder comments (`// implement rest here`), or unwarranted refusals on long tasks. *Combat via: explicit completeness criteria, chunked output chaining, calibrated `effort` / `thinking_level`, and clear task framing (historically worked around in early chat models via urgency or incentive cues).* --- ## 09. Agents & Capability Levels (L0 to L4) ### What Makes an Agent An **AI agent** is a system in which a model decides, step by step, which actions to take toward a goal, observes the results, and continues until the goal is met or a stop condition fires. A chatbot answers; an agent acts. - **Model:** The reasoning engine that interprets the goal, plans, and selects the next action. - **Tools:** Functions, APIs, retrieval, and sandboxed code execution that let the model read live state and mutate external systems. - **Orchestration loop:** Harness code that runs the cycle (*gather context → decide → act → observe*), preserves `thought_signatures`/state, enforces budgets/stop conditions, and routes errors. - **Context and memory:** Curated working context placed in the window each turn, plus session state and long-term memory stored outside it (see the [Context Engineering guide](https://mbettan.github.io/machine-learning-cert/ContextEngineering.html#fundamentals)). - **Guardrails:** Deterministic permission gates, human-in-the-loop approvals, schema validation, and audit logging that keep actions within policy. ```text goal ──▶ gather context ──▶ model decides ──▶ call tool ──▶ observe result ▲ │ └───────────── repeat until done or stopped ◀──────┘ ``` - **Workflow vs. Agent:** In a **workflow**, your application code fixes the sequence of model calls and tool invocations; in an **agent**, the model chooses the sequence dynamically at run time. Workflows are cheaper, faster, and more predictable; open agent loops are reserved for tasks whose path cannot be known in advance. ### Agent Capability Levels (L0 to L4) Each step up adds a concrete runtime capability—and multiplies token cost, latency, failure modes, evaluation surface, and security exposure. | Level | Adds | Use When | Example | Go Deeper | | :--- | :--- | :--- | :--- | :--- | | **L0: model only** | Nothing beyond one model call with supplied input | The answer depends only on the prompt input and general language skill | Rewrite a station closure notice in plain language; classify a ticket | [Prompt Engineering](https://mbettan.github.io/machine-learning-cert/PromptEngineering.html) | | **L1: model with tools** | Tool calls and retrieval for live data or a single action | The task requires fresh, private, or exact data, or one external action | Look up live order status; answer from the policy library | [ADP agent primitives](https://mbettan.github.io/machine-learning-cert/ADP.html#agent-primitives), [MCP](https://mbettan.github.io/machine-learning-cert/MCP.html), [RAG](https://mbettan.github.io/machine-learning-cert/RAG.html) | | **L2: multi-step agent** | Planning across dependent steps with state and managed context | Several steps depend on earlier tool observations | Build a weekly operations report from five sources and flag anomalies | [ADP workflow patterns](https://mbettan.github.io/machine-learning-cert/ADP.html#workflow-patterns), [Context Engineering](https://mbettan.github.io/machine-learning-cert/ContextEngineering.html#long-sessions) | | **L3: multi-agent system** | Specialist agents with isolated tools, permissions, and context under a coordinator | Subtasks require distinct IAM scopes, clean context isolation, parallel branches, or an independent verifier | Read-only telemetry diagnostics agent + purchasing agent with bounded spend authority | [ADP multi-agent topologies](https://mbettan.github.io/machine-learning-cert/ADP.html#multi-agent-topologies), [A2A](https://mbettan.github.io/machine-learning-cert/A2A.html) | | **L4: self-extending system** | Creates or modifies its own tools, prompts, or subagents at run time | Tightly sandboxed research or platform environments with mandatory human review before activation | System that drafts and registers a new data connector inside an isolated sandbox | [AgentOps](https://mbettan.github.io/machine-learning-cert/AgentOps.html), [Responsible AI](https://mbettan.github.io/machine-learning-cert/ResponsibleAI.html) | > **Key Rule:** Pick the **lowest level** that solves the task. Climb one level only when an offline evaluation proves the lower level fails, and name the exact capability that justifies the climb. ### Choosing a Level - **Before L0:** If the rule is deterministic (a tax table, a date window check, an ID lookup), write deterministic code and keep the LLM out of the path. 1. **Can one call with the supplied input do it?** Stay at **L0**. Paste the text directly instead of standing up a vector pipeline. 2. **Does it need live, private, or exact data, or one external action?** Move to **L1** with least-privilege tools. 3. **Do several steps depend on each other?** Move to **L2**. Use a **workflow** when the step graph is known in advance; use an open **agent loop** only when steps depend on unpredictable runtime discoveries. 4. **Do subtasks need separate permissions, isolated context windows, parallel execution, or independent verification?** Move to **L3**. Splitting into role-play personas that share identical tools and context is *agent theater*, not justification for L3. 5. **Must the system author new capabilities at run time?** **L4** is rarely appropriate in production—prefer engineer-built, code-reviewed tools added to L2/L3, and require sandboxed execution plus human approval gates for anything self-generated. ### Scenario: Scoping Four Requests at Spokewise Bike Share *A regional bike-share operator asks for "one AI agent" covering four operational requests:* | Request | Level | Why Not Lower | Why Not Higher | | :--- | :--- | :--- | :--- | | Draft a rider notice about a station closure | **L0** | *(lowest level)* | Input is supplied; no live lookup or mutation needed | | Tell a rider the nearest dock with a charged e-bike | **L1** | Dock availability changes every minute | Single live query; no multi-step planning | | Produce the overnight truck rebalancing plan from demand forecasts, weather, and crew shifts | **L2 (workflow)** | Three dependent data sources must be fetched, reconciled, and checked | Step sequence is fixed; one read-only identity suffices | | Run fleet maintenance: diagnose faults, order replacement parts, verify repairs | **L3** | Diagnostics needs read-only telemetry, ordering needs spend authority, repair sign-off must be independent | Connectors are built and reviewed by engineers, not synthesized at run time | - **Outcome:** One vague "agent" request became three focused builds and one multi-agent system, each evaluated at its own capability tier. - **What the team rejected:** A single L3 "super-assistant" handling all four tasks—which would have exposed purchasing credentials to a public notice drafter and inflated latency on trivial queries. ### Common Mistakes & Bridge to Architecture - **Starting at L3 for an L1 problem:** Multiple persona agents sharing identical tools and context multiply cost and latency without adding capability (*agent theater*). - **Staying at L0 when facts must be current:** Relying on parametric memory for live inventory or policy state guarantees extrinsic hallucinations; attach retrieval or a tool. - **Open loops without stop conditions:** Every L1–L4 loop requires explicit `max_steps`, token/cost budgets, and loop-detection circuit breakers. - **Prompt-only guardrails:** System-prompt wording is a behavioral hint, not a security boundary; irreversible actions require code-level IAM enforcement and confirmation gates. - **Unreviewed self-modification in production:** Allowing an L4 agent to write and register live tools without sandboxing turns any indirect prompt injection into arbitrary code execution. | If You Chose | Continue in Agentic Design Patterns (ADP) With | | :--- | :--- | | **L0 or L1** | [Agent primitives](https://mbettan.github.io/machine-learning-cert/ADP.html#agent-primitives): tool contracts and stop conditions | | **L2 (workflow)** | [Workflow patterns](https://mbettan.github.io/machine-learning-cert/ADP.html#workflow-patterns): prompt chaining, routing, parallelization, orchestrator-workers, evaluator-optimizer | | **L2 (agent loop)** | [Agent primitives](https://mbettan.github.io/machine-learning-cert/ADP.html#agent-primitives) and [Memory and context](https://mbettan.github.io/machine-learning-cert/ADP.html#memory-context) | | **L3** | [Multi-agent topologies](https://mbettan.github.io/machine-learning-cert/ADP.html#multi-agent-topologies): the agent theater test and handoff schemas | | **Any level, pre-launch** | [Anti-patterns and checklist](https://mbettan.github.io/machine-learning-cert/ADP.html#anti-patterns-checklist) | *Credit:* The L0–L4 capability progression builds on *Introduction to Agents* (Blount, Gulli, Saboo, Zimmermann & Vuskovic; Google, 2026), while the workflow-vs.-agent distinction follows Anthropic's [Building effective agents](https://www.anthropic.com/engineering/building-effective-agents); scenarios and decision tables here are original. --- ## 10. The 5-Layer Agentic Interoperability Protocol Stack Modern enterprise agent architectures separate **vertical tool plumbing**, **horizontal peer delegation**, **human UI rendering**, **real-time media streaming**, and **cryptographic value transfer** into five distinct protocol layers. Mixing them up is a classic certification trap. | Layer | Boundary | Standard Protocols | Core Primitives & Mechanics | When to Use | | :--- | :--- | :--- | :--- | :--- | | **Layer 1: Agent ↔ Tools & Data** (*Vertical*) | Single agent calling deterministic APIs, databases, or local dev resources | **MCP (Model Context Protocol)**, OpenAPI | `Tools` (model-controlled actions), `Resources` (app-controlled read context), `Prompts`, structured `isError: true` payloads over `stdio` or **Streamable HTTP** | Exposing enterprise CRUD tools, SQL/search connectors, or IDE context to L1+ agents ([MCP guide](https://mbettan.github.io/machine-learning-cert/MCP.html)) | | **Layer 2: Agent ↔ Peer Agents** (*Horizontal*) | Cross-team, cross-framework, or cross-vendor opaque agent delegation | **A2A (Agent2Agent Protocol)** | Discovery via `/.well-known/agent-card.json` or **Agent Registry**; stateful `Task` lifecycle (`submitted → working → input-required → completed`), `Artifacts`, streaming SSE | Delegating goals across organizational or runtime boundaries without exposing internal prompts, memory, or tool credentials ([A2A guide](https://mbettan.github.io/machine-learning-cert/A2A.html)) | | **Layer 3: Agent ↔ Rich Human UI** (*Generative UI*) | Streaming interactive frontend components and bidirectional user state | **MCP UI**, **AG-UI**, **A2UI** | **AG-UI:** bidirectional client-agent event stream (state diffs, tool progress, HITL interrupts). **A2UI:** declarative JSON widget payloads rendered natively by host UIs. **MCP UI:** sandboxed interactive views returned by MCP tools | Replacing walls of raw Markdown with live seat pickers, approval cards, charts, and editable forms without letting the LLM emit raw unsanitized HTML/JS | | **Layer 4: Agent ↔ Real-Time Streams** (*Live Media*) | Sub-second full-duplex voice, video, and screen sharing with barge-in | **Gemini Live API** over **WebSockets / WebRTC** | Continuous audio/video frame streaming, native **Voice Activity Detection (VAD)** / interruption (*barge-in*), server-side session resumption | Live voice concierges, camera-guided field diagnostics, and real-time tutoring (terminate heavy media at the edge, then invoke L1/L2 backend tools via MCP/A2A) | | **Layer 5: Agent ↔ Commerce & Value** (*Payments*) | Verifiable human spend authorization & autonomous API micropayments | **AP2 (Agent Payments Protocol)** & **x402** | **AP2:** cryptographically signed **Mandates** (`Intent Mandate` for bounded autonomous rules; `Cart Mandate` for explicit human item/price approval) separating shopper, merchant, and credentials provider. **x402:** reviving `HTTP 402 Payment Required` for instant machine-to-machine per-request API/data micropayments | Booking travel, executing B2B procurement with non-repudiable spend proof (`AP2`), or paying per API call/dataset query autonomously (`x402`) | > **Architectural Rule of Thumb:** > - **MCP is vertical** (an agent's *hands* reaching into deterministic tools/data). > - **A2A is horizontal** (peer *colleagues* collaborating as opaque black boxes via `AgentCard`s). > - **A2UI / AG-UI / MCP UI** govern the **human glass**, **Live API / WebRTC** governs **real-time media**, and **AP2 + x402** govern **money and cryptographic consent**. --- ## 11. RAG & Context Augmentation **RAG (Retrieval-Augmented Generation)** bypasses a model's static training cutoff and reduces extrinsic hallucinations by grounding synthesis in external, permission-checked enterprise data at query time. ```text User Query ──▶ Embed / Hybrid Search ──▶ Retrieve & Rerank Chunks (ACL-filtered) ──▶ Inject Grounded Context ──▶ LLM Synthesizes Cited Answer ``` - **How it works:** User submits a query → System runs **hybrid retrieval** (dense vector embeddings for semantic meaning + sparse lexical BM25 for exact SKU/error codes) against an ACL-filtered index → Cross-encoder reranker selects top passages → Retrieved evidence is injected into the prompt → The LLM synthesizes an answer grounded *strictly* in cited passages (see the [RAG guide](https://mbettan.github.io/machine-learning-cert/RAG.html)). - **Vector Database:** Specialized storage indexing high-dimensional embedding vectors (via HNSW or ScaNN approximate nearest-neighbor graphs) so retrieval matches **semantic similarity** rather than brittle exact-keyword overlap alone. --- ## 12. Risks, Ethics & Vulnerabilities ### Core Risks & Attack Surfaces - **Hallucinations:** Plausible-sounding but fabricated or contradicted statements. *Combat via: grounded RAG, citation verification, autoraters, and human-in-the-loop review.* - **Extrinsic hallucination:** The model invents facts absent from the provided source context (e.g., fabricating a clause or API flag that nowhere appears in the document). - **Intrinsic hallucination:** The model directly contradicts facts explicitly stated in the provided source context (e.g., summarizing a stated "24-month contract term" as "12 months"). - **Prompt Injection (Direct & Indirect):** Attackers embed adversarial instructions inside user input (*direct*) or untrusted external content such as emails, web pages, MCP tool outputs, or PDFs (*indirect / zero-click exploit*) to hijack agent tool calls or exfiltrate context. - **Jailbreaking:** Adversarial framing (role-play personas, multi-turn escalation, cipher encodings) aimed at bypassing safety alignment to elicit restricted outputs. - **Data Poisoning & Slopsquatting:** Attackers inject backdoored examples into training/RAG corpora (*poisoning*), or register malicious packages on PyPI/npm matching package names that coding LLMs frequently hallucinate (*slopsquatting*). - **Memorization & Privacy Leakage:** Models can verbatim regurgitate PII, secrets, or copyrighted passages over-represented in training splits. - **Detection & Content Provenance:** - **Watermarking (`SynthID`):** Imperceptibly biases token sampling distributions (text) or frequency-domain pixel/audio signals (images, video, audio) so AI-generated media remains statistically detectable even after cropping or compression—without providing an edit history. - **C2PA (Coalition for Content Provenance and Authenticity):** Attaches **cryptographically signed tamper-evident metadata manifests** (*Content Credentials*) recording capture origin, model edits, and custody chain. - **Automation Bias:** The human cognitive failure of rubber-stamping fluent AI recommendations without independent verification—especially hazardous in clinical, legal, financial, and production deployment workflows. *Rule of thumb: AI drafts and proposes; accountable humans or deterministic policy gates verify.* ### Ethics, Quality & Human Oversight - **Human-in-the-Loop (HITL):** Calibrate human review to **action reversibility and blast radius**—automate low-risk reversible reads, require explicit human approval before irreversible state mutations or external communications. - **Bias & Fairness:** Models inherit demographic and historical skews from pretraining corpora; audit disaggregated subgroup metrics and test counterfactual prompts before deployment. - **Copyright & Plagiarism:** Mitigate verbatim reproduction via deduplicated corpora, recitation filters, grounded citations, and avoiding prompts that request living artists' or authors' proprietary styles. --- ## 13. Glossary - **Tokens:** Sub-word units (roughly ~3–4 English characters per token) processed by tokenizers. Context limits, throughput, and API billing (including hidden System 2 thinking tokens) are metered in tokens. - **Context Window:** Maximum token budget a model can attend over in a single turn (`input + retrieved context + tool schemas + thinking tokens + visible output`). Content evicted past the window is forgotten unless persisted in external state/memory. - **Parameters:** Learned numerical weights defining neural connections inside dense or Sparse MoE layers. - **Temperature:** Sampling hyperparameter scaling logit probabilities before token selection. Low (`0–0.3`) concentrates probability mass on the likeliest tokens for extraction/classification (though GPU floating-point non-determinism and MoE batching mean even `0` is not strictly deterministic); high (`0.8–1.5`) flattens probabilities for creative brainstorming. *(Note: Several 2026 reasoning models lock sampling defaults and expose `thinking_level` or `effort` instead.)* - **Fine-Tuning (SFT / LoRA):** Updating all weights or lightweight Low-Rank Adaptation (`LoRA`) adapter matrices on curated domain examples to specialize style, schema adherence, or task behavior. - **SLMs (Small Language Models):** Compact, parameter-efficient models (typically 1B–30B active parameters, such as `Gemma 4`, `Gemini 3.5 Flash-Lite`, or `Gemini Nano`) engineered for sub-200ms latency, low serving cost, or on-device/edge execution. - **MaaS (Model-as-a-Service):** Consuming open-weight or partner models (e.g., Claude, Llama, DeepSeek) via managed, per-token serverless cloud endpoints without provisioning dedicated GPU nodes. - **`thinking_level` / Adaptive Thinking:** Inference-time System 2 compute controls—`thinking_level` (`MINIMAL`..`HIGH`) + `thought_signatures` on Gemini, and `thinking: {type: "adaptive"}` + `output_config.effort` (`low`..`max`) on Claude. - **Custom GPTs / Gems / Projects:** Lightweight workspace wrappers combining persistent system instructions, curated reference files, and pre-bound tools on top of a foundation model. - **Digital Twins:** Dynamic virtual simulations of physical assets, factories, or operational networks used for scenario testing and predictive maintenance. - **Agent Capability Level (`L0`–`L4`):** Scoping ladder from a single model call (`L0`), tool-augmented call (`L1`), multi-step workflow/agent (`L2`), and multi-agent system (`L3`) to self-extending runtime systems (`L4`). - **Orchestration Loop:** Harness runtime executing *gather context → decide → act → observe* while enforcing step budgets, `thought_signatures`, and guardrails. - **Workflow vs. Agent:** Code-fixed step sequence (*workflow*) vs. model-directed runtime control flow (*agent*). - **5-Layer Agentic Protocol Stack:** `MCP` (L1 tools/data), `A2A` (L2 peer agents), `MCP UI / AG-UI / A2UI` (L3 generative human UI), `Live API / WebRTC` (L4 real-time audio/video streams), and `AP2 + x402` (L5 cryptographic spend mandates & HTTP 402 micropayments). ### Self-Assessment Questions - **Q1.** What is the core difference between pretraining, RAG, and Supervised Fine-Tuning (SFT)? - *Answer:* Pretraining teaches general language representations on massive uncurated corpora; RAG injects live, ACL-filtered external facts at query time without changing weights; SFT updates weights on curated demonstrations to lock in domain formatting, tone, or compact SLM specialization. - **Q2.** What is the difference between Direct and Indirect Prompt Injection? - *Answer:* Direct injection comes from the user's own prompt; indirect injection hides malicious instructions inside untrusted external data (emails, web pages, PDFs, tool responses) that an agent ingests, creating a zero-click hijack risk. - **Q3.** Why doesn't choosing an open-weight model (like Gemma 4 or Llama) automatically require deploying dedicated GPU VMs? - *Answer:* Open-weight models can be consumed via serverless Model-as-a-Service (MaaS) per-token APIs with zero idle GPU cost; dedicated VPC endpoints (billed per node-hour even at 0 QPS) are needed only for strict single-tenant compute residency, custom fine-tuned weights, or sustained high-duty-cycle TCO. - **Q4.** How do System 2 reasoning controls differ between Gemini 4 Argon and Claude Opus 4.7, and when should you keep reasoning effort low? - *Answer:* Gemini 4 uses `thinking_level` (`MINIMAL` to `HIGH`) and requires preserving encrypted `thought_signatures` across tool turns; Claude 4.7 uses `thinking: {type: "adaptive"}` with `output_config.effort` (`low` to `max`). Keep effort minimal/low on simple routing, classification, or real-time voice turns where extra thinking tokens only add latency and output-token cost. - **Q5.** Why should a team pick the lowest agent capability level (`L0`–`L4`) that solves a task? - *Answer:* Each climb adds latency, token cost, nondeterminism, evaluation burden, and security blast radius; climb one level only when evaluation proves the lower level fails on a named capability (live data, dependent steps, isolated permissions). - **Q6.** Contrast `MCP`, `A2A`, `A2UI/AG-UI`, and `AP2 + x402` in the 5-Layer Agentic Protocol Stack. - *Answer:* `MCP` connects an agent vertically to deterministic tools/resources; `A2A` connects opaque peer agents horizontally via `AgentCard`s and stateful `Task`s; `A2UI/AG-UI` stream declarative interactive UI components and state events to humans; `AP2` provides cryptographic human spend Mandates while `x402` handles machine-to-machine `HTTP 402` micropayments. ================================================================================ GUIDE: Prompt Engineering URL: https://mbettan.github.io/machine-learning-cert/PromptEngineering.html Practice quiz: https://mbettan.github.io/machine-learning-cert/quiz.html?topic=prompt-engineering Section: Foundation · AI Fundamentals Exam relevance: Google Professional Agentic Architect (supporting), Claude Certified Architect – Foundations (core) ================================================================================ # Prompt Engineering: Study Guide > **Author:** Michaël Bettan --- ## Overview ### Definition Prompt engineering is the discipline of specifying task contracts, evaluation rubrics, input boundaries, demonstration examples, and output schemas so a language model produces reliable, verifiable results across varied production inputs—paired with programmatic validation, explicit reasoning boundaries, and staged prompt chains where prose instructions alone are insufficient. ### Why production prompting looks different from chat prompting In an interactive chat, a human spots a vague answer and clarifies on the next turn. In an automated code reviewer, document extractor, or customer-operations pipeline, nobody sits in the loop unless your system routes the item there. Production prompt engineering therefore combines **authoring levers** (explicit categorical criteria, contrast examples, input delimiters, normalization rules) with **API enforcement levers** (constrained JSON schemas, strict tool definitions, `tool_choice`, adaptive reasoning effort, bounded validation retries, and multi-instance review). > **Mental model:** Prose steers *judgment*; schemas and code enforce *invariants*. Use prompt instructions and few-shot examples to define ambiguous boundaries and domain semantics, and use API schemas, validators, and fresh reviewer contexts to guarantee syntax, catch cross-field math errors, and eliminate self-review blind spots. ![Production prompt and structured output pipeline across five stages: task contract, contrast examples, strict schema, code validation, and review routing](https://mbettan.github.io/machine-learning-cert/diagrams/prompt-output-pipeline.jpg) * **Scope of this guide:** prompt anatomy and attention placement, explicit criteria and false-positive control, multishot design, reasoning effort vs. sampling parameters, task decomposition and iterative refinement, schema-enforced structured outputs, validation-retry and reviewer-feedback loops, batch transport selection and chained batch stages, chain-of-thought isolation and independent critique passes, caching, injection defense, and LLMOps. * **Vendors covered:** Anthropic Claude (Messages API, `tool_use`, `output_config`, Adaptive Thinking, Message Batches) and Google (Gemini API, Vertex AI `response_schema`, `FunctionCallingConfig`, Thinking levels, Batch Prediction, Vertex AI Prompt Optimizer). * **Related guides:** [Claude Platform & API](https://mbettan.github.io/machine-learning-cert/ClaudeAPI.html#structured-outputs) for Claude-specific endpoint rules, [Context Engineering & Reliability](https://mbettan.github.io/machine-learning-cert/ContextEngineering.html#context-budget) for long-horizon context assembly and provenance, [Agentic Design Patterns](https://mbettan.github.io/machine-learning-cert/ADP.html#workflow-patterns) for workflow topologies, [Claude Code](https://mbettan.github.io/machine-learning-cert/ClaudeCode.html#ci-review) for headless CI review, and [AgentOps](https://mbettan.github.io/machine-learning-cert/AgentOps.html#grading-hierarchy) for evaluation pipelines. * **Official documentation:** Claude [prompt engineering overview](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/overview), [prompting best practices](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices), [structured outputs](https://platform.claude.com/docs/en/build-with-claude/structured-outputs), [strict tool use](https://platform.claude.com/docs/en/agents-and-tools/tool-use/strict-tool-use), [effort](https://platform.claude.com/docs/en/build-with-claude/effort), and [batch processing](https://platform.claude.com/docs/en/build-with-claude/batch-processing); Gemini API [prompt design strategies](https://ai.google.dev/gemini-api/docs/prompting-strategies), [structured output](https://ai.google.dev/gemini-api/docs/structured-output), [function calling](https://ai.google.dev/gemini-api/docs/function-calling), and [thinking](https://ai.google.dev/gemini-api/docs/thinking). * **Further reading:** Positional recall curves in long inputs are analyzed in Liu et al., [Lost in the Middle: How Language Models Use Long Contexts](https://arxiv.org/abs/2307.03172) (TACL 2024). Reasoning-and-acting loops originate in Yao et al., *ReAct* (ICLR 2023), and verbal self-critique loops in Shinn et al., *Reflexion* (NeurIPS 2023). --- ## Prompt Anatomy, Roles & Attention Placement ### Message roles and trust hierarchy | Channel | Claude Messages API | Gemini / Vertex AI | What belongs here | | :--- | :--- | :--- | :--- | | **System instruction** | Top-level `system` parameter | `system_instruction` in config | Role identity, non-negotiable behavioral rules, categorical criteria, global output conventions | | **Reference & examples** | Early `user` content blocks or `system` blocks (with `cache_control`) | Cached prefix contents or early `user` parts | Domain glossary, schema normalization rules, ``, large source documents | | **Untrusted payload** | Delimited block inside `user` (`…`) | Delimited part inside `user` | Raw customer messages, pull-request diffs, OCR text, web pages | | **Task trigger** | Final lines of the `user` turn | Final part of the `user` turn | The specific question, target action, and reminder of edge-case rules | ### Structural delimiters and edge placement * **Separate instructions from data:** wrap variable inputs in unambiguous tags such as ``, ``, or `` rather than loose quotation marks. Clear boundaries stop multi-paragraph inputs from bleeding into your rules and make indirect prompt-injection instructions easier for the model to recognize as data. * **Respect the U-shaped attention curve ("Lost in the Middle"):** across long prompts, recall is highest at the opening and closing edges of the window and weakest in the middle interior. Place large reference documents near the top (where they also form a stable cacheable prefix), put key findings summaries ahead of raw appendices, label sections with explicit headers, and state the exact task instructions and critical constraints at the very end. * **Positive framing vs. hard negative boundaries:** frame style and workflow instructions positively (*"Write a 3-bullet summary citing section numbers"* outperforms *"Don't write long paragraphs or omit citations"*). Reserve strict negative directives (`NEVER`, `DO NOT`, `ONLY`) for safety gates, scope exclusions, and anti-fabrication rules (*"Return `null` when a date is absent; never estimate or infer missing dates"*). * **Authoring checklist (Task · Context · Examples · Evaluate · Iterate):** define a concrete verb, role, and target audience (**Task**); supply the minimum situational facts and source boundaries needed (**Context**); attach 2–5 representative demonstrations (**Examples**); grade outputs on a labeled test set (**Evaluate**); and refine one variable per run (**Iterate**). ### XML tag disambiguation rules Claude has no reserved tag vocabulary: any descriptive tag name works, provided it is used consistently. Gemini prompts benefit from the same discipline (clear prefixes or tags that separate instructions, context, and input). | Rule | What goes wrong without it | Practice | | :--- | :--- | :--- | | **One tag per content type** | Instructions, reference text, and live input blur together | Wrap each kind separately: ``, ``, ``, `` | | **Nest to express hierarchy** | The model cannot tell which passage a rule refers to | `` containing `` with inner `` and `` tags | | **Refer to tags by name** | *"Use the document above"* is ambiguous once several blocks exist | *"Answer only from ``; apply tone rules from ``"* | | **Keep names stable** | Renaming tags between prompt versions or examples confuses the model and breaks cached prefixes | Freeze tag names in the versioned template; change content, not structure | | **Neutralize tag look-alikes in payloads** | Untrusted text containing `` can fake the end of the data block | Escape or strip tag-like strings from the payload before insertion | | **Name the output container** | Free-form replies mix commentary with the result | Ask for the result inside `` for human-facing text; use a JSON schema for machine consumers | > **Retired pattern — assistant prefilling:** Older guides recommended ending a request with a partial `assistant` turn such as `{"role": "assistant", "content": "{"}` to force JSON output. Modern frontier models (Claude 4.6 and later) **reject assistant prefilling with HTTP 400**. Enforce structure with native JSON Schema decoding (`output_config.format` / `response_schema`) or strict tool calls instead. --- ## Explicit Criteria, Rubrics & False-Positive Control ### Why vague instructions and confidence prompts fail In automated code review, compliance triage, and defect detection, **precision** (the share of flagged items that are genuinely actionable) governs whether engineers read or mute your system. Telling a model to *"be conservative"*, *"only flag high-confidence issues"*, or *"check that comments are accurate"* fails because it never defines the decision boundary—the model's internal threshold stays uncalibrated while noisy edge cases still slip through. | Weak / vague phrasing | Failure mode in production | Explicit categorical specification | | :--- | :--- | :--- | | *"Check whether code comments are accurate"* | Flags minor wording preferences, TODOs, and abbreviated docstrings | *"Report a comment defect **only** when a comment's stated behavior directly contradicts the executable code path"* | | *"Only report high-confidence bugs; be conservative"* | Model still emits stylistic nitpicks labeled `"high confidence"` | **Report:** null dereferences, auth bypasses, resource leaks, SQL/command injection. **Skip:** naming conventions, formatting, and idioms already established in the repository | | *"Assign a severity of low, medium, or high"* | Severity labels drift across runs and files | Define each tier with explicit impact rules **plus one concrete code example per severity level** | ### Protecting developer trust in automated pipelines * **One noisy category poisons every category:** if your PR reviewer is 92% accurate on security bugs but 35% accurate on "maintainability" warnings, developers stop reading *all* comments from the bot within days. * **Measure precision per category, not only in aggregate:** track acceptance and dismissal rates separately for each rule family (`security`, `concurrency`, `null-safety`, `comment-drift`). * **Disable noisy categories immediately while fixing them:** when a category produces excessive false positives, **turn that category off** in production right away to restore trust in the high-precision categories, then rewrite its criteria and few-shot contrast examples offline against an evaluation set before re-enabling it. | Policy component | Role in the prompt contract | Concrete specification | | :--- | :--- | :--- | | **Report categories** | Enumerates the only defect classes eligible for comments | **`defect`** (runtime exceptions, off-by-one bounds, lost updates, unhandled promises); **`security`** (broken access checks, unsanitized shell/SQL/HTML sinks, secret exposure); **`comment_contradiction`** (a guarantee such as *"thread-safe"* or *"returns UTC"* violated by the code) | | **Skip categories** | Explicitly suppresses high-noise style feedback | Formatting, variable naming, import ordering, missing docstrings, and established local repository idioms | | **Severity rubric** | Anchors each severity tier to concrete runtime impact | **`critical`:** exploitable security flaw or guaranteed crash/data loss on valid input (e.g., `if is_admin or True:`); **`major`:** wrong business result or unhandled edge case on reachable inputs (e.g., dividing by `len(items)` before checking emptiness) | > **Exam trap:** When a scenario shows developers ignoring an automated PR reviewer due to false positives, avoid answers that *"lower temperature to 0"*, *"tell the model to only report findings above 90% confidence"*, or *"keep all categories live while adding a disclaimer"*. Temporarily disable the noisy category and replace vague prompts with explicit report/skip criteria and severity examples. --- ## Few-Shot & Multishot Design ### When few-shot examples beat longer instructions When prose rules are already clear yet outputs still vary in structure, tone, or boundary judgment, **2–5 targeted few-shot examples** (wrapped in `…`) are the highest-leverage intervention. Demonstrations teach **generalizable judgment** on novel patterns rather than brittle keyword matching on enumerated cases. | Engineering goal | How to construct the `` set | | :--- | :--- | | **Consistent output format** | Every example shows the complete target shape (for a ticket classifier: `intent`, `urgency`, `evidence_span`, `reply_draft`) with identical key order and casing | | **False-positive suppression** | Use **contrast pairs**: one example showing a real defect that *is* reported alongside a plausible look-alike or valid repo idiom that produces `[]` (no finding) | | **Ambiguous routing / tool choice** | Include a brief `` trace inside the example explaining *why* action A beats plausible alternative B | | **Varied document extraction** | Demonstrate extraction across structural layouts (a prose paragraph with inline citations, a formal bibliography table, informal units like `"half a dozen"`, and a document missing an optional field returning `null`) | ### Structuring the example block Keep demonstrations physically and semantically separate from the live request so the model never mistakes a sample input for the one it must process: ```xml Card charged twice for order 5521 {"intent": "billing", "urgency": "high"} {{TICKET_TEXT}} ``` * **Label the role of examples:** state that examples illustrate format and judgment only, and that names, amounts, and wording must come from the live ``. Unlabeled examples get copied verbatim. * **Show the target pattern, not the anti-pattern:** Gemini's prompt design guidance notes that examples of what to do steer better than examples of what to avoid. Contrast pairs still follow this rule: the look-alike input is paired with its *correct* (empty or benign) output. * **Keep formatting identical across examples:** same tags, same key order, same whitespace conventions. Inconsistent demos teach inconsistent output. * **Place examples in the static prefix:** examples before the live input stay cacheable and leave the task trigger at the end of the prompt. ### Preventing few-shot overfitting and bias * **Randomize class order (class mixing):** listing four `"escalate"` examples followed by one `"resolve"` example biases the model toward escalating borderline inputs. Interleave outcomes and categories across examples. * **Vary surface structure:** if every example invoice uses `USD` and three line items, the model may copy `USD` on a `EUR` document or drop a fourth line item. Vary lengths, field presence, and phrasings while keeping the schema invariant. * **Demonstrate `null` explicitly:** to cure hallucination on missing required-looking fields, include at least one example where the source omits a field and the target output sets `"renewal_date": null` rather than guessing. | Sample `` snippet | Target structured extraction | Boundary taught by the example | | :--- | :--- | :--- | | *"Memo (14 May 2026): Pilot renewal approved for 18 months; PO number pending finance sign-off."* | `effective_date`: `"2026-05-14"`, `term_months`: `18`, `po_number`: `null`, `status`: `"approved"` | Normalizes informal prose dates to ISO-8601 while setting an explicitly pending field (`po_number`) to `null` | | *"Verbal discussion notes — terms still under negotiation between legal teams; no dates agreed."* | `effective_date`: `null`, `term_months`: `null`, `po_number`: `null`, `status`: `"unclear"` | Demonstrates full `null` extraction plus the `"unclear"` escape hatch when no binding agreement exists | --- ## Reasoning, Effort & Sampling Controls ### Adaptive thinking and reasoning depth Modern frontier models allocate variable test-time compute ("System 2" reasoning) before emitting visible output or between tool calls. | Capability | Claude 4.6 / 4.7+ (`Messages API`) | Google Gemini (`Gemini API` / `Vertex AI`) | | :--- | :--- | :--- | | **Enable reasoning** | `thinking={"type": "adaptive"}` (interleaves automatically across tool turns) | `thinking_config={"thinking_level": "HIGH"}` (model-dependent levels: `MINIMAL`, `LOW`, `MEDIUM`, `HIGH`) | | **Control compute budget** | `output_config={"effort": "low" \| "medium" \| "high" \| "xhigh" \| "max"}` | `thinking_level` (or `thinking_budget` on earlier Gemini 2.5 models) | | **Legacy trap** | Manual `{"type": "enabled", "budget_tokens": N}` is deprecated on 4.6 and **returns HTTP 400 on Claude 4.7+** | Mixing legacy budget fields with level-based models fails validation | | **Cost & latency rule** | Full internal thinking tokens are billed as output tokens even when `display: "omitted"`; use `low` effort for simple routers/classifiers | Thinking tokens count toward output billing and latency; dial down `thinking_level` on high-throughput classification | ### Sampling parameters and modern API constraints * **Claude 4.7+ rejects non-default sampling parameters:** passing non-default `temperature`, `top_p`, or `top_k` on Claude 4.7+ returns an **HTTP 400** (and on earlier Claude 4.x models, `temperature` and `top_k` are incompatible with thinking). Frequency and presence penalties do not exist on the Claude API. Steer determinism and creativity through explicit criteria, few-shot examples, structured schemas, and `effort`. * **Models that expose sampling (`Gemini`, open weights, older endpoints):** lower `temperature` (`0.0–0.2`) narrows token selection toward higher-probability paths for extraction and classification, while higher values (`0.7–1.0`) broaden diversity for brainstorming. Adjust `temperature` *or* `top_p`, rarely both simultaneously. * **Low temperature is never a determinism guarantee:** floating-point batching, MoE routing, and hardware concurrency still produce run-to-run variation at `temperature=0`. Grade outputs on parsed schema properties and invariants, never exact raw string equality. * **`max_tokens` / `max_output_tokens` is a hard cutoff, not a length instruction:** hitting the ceiling stops generation mid-token (`stop_reason == "max_tokens"` on Claude, `FinishReason.MAX_TOKENS` on Gemini), producing truncated, unparseable JSON. Ask for conciseness in the prompt/schema while sizing `max_tokens` with generous headroom. ### Reasoning prompt patterns (when explicit scaffolding helps) * **Chain-of-Thought (CoT) in structured tasks:** when thinking mode is unavailable or disabled for latency, add a concise `"reasoning"` or `"evidence_quote"` string field *ahead* of classification fields in your schema so the model grounds its decision before committing to a label. * **Contrastive examples:** show both a valid deduction and a common miscalculation (such as confusing fiscal-year dates with calendar-year dates) so the boundary is explicit. * **Step-Back prompting:** ask the model to identify the governing policy principle or physical law first, then apply it to the specific case—useful when dense surface details obscure the rule. * **Self-Consistency (majority voting):** sample multiple independent reasoning paths on hard, verifiable problems and aggregate the final structured answer—effective offline, too costly for low-latency chat. --- ## Decomposition & Prompt Chaining vs. Mega-Prompts ### Choosing the right decomposition strategy Packing extraction, cross-reference verification, policy evaluation, and executive formatting into one giant "mega-prompt" dilutes attention across competing instructions and makes failures impossible to localize. | Pattern | When to pick it | Concrete workflow example | | :--- | :--- | :--- | | **Fixed sequential pipeline (prompt chaining)** | Stages and dependencies are known upfront; predictable multi-step transformation | **Step 1:** Extract normalized claims + dates per file → **Step 2:** Cross-check claims across files → **Step 3:** Render final compliance table | | **Dynamic / adaptive decomposition** | Open-ended investigation where step `k+1` depends on what step `k` uncovers | Legacy test-suite expansion: map module graph → rank high-risk untested modules → inspect dependencies → adapt test plan as hidden couplings emerge | | **Parallel fan-out + synthesis** | Independent inputs that need uniform local depth without cross-contaminating context | Summarize 25 customer-interview transcripts concurrently with one shared rubric, then run 1 synthesis call over their structured outputs | ### Iterative refinement patterns for engineering workflows 1. **Concrete input/output pairs over prose rewrites:** when prose instructions for a data transformation or formatting rule are interpreted inconsistently across edge cases, stop rephrasing adjectives—provide **2–3 concrete before/after input-output pairs** covering the tricky edge cases (such as `null` timestamps or nested arrays). 2. **Test-driven iteration:** write executable unit and edge-case tests first, run the model's implementation, and feed the exact failing test assertions back so each turn closes a verified gap. 3. **The interview pattern (clarify before building):** when designing in an unfamiliar domain or underspecified problem space, instruct the model to **ask targeted architectural questions first** (covering failure modes, cache invalidation, concurrency, state boundaries) before writing code or specs. 4. **Bundled vs. sequential bug feedback:** send **interacting** defects together in a single message so the fix accounts for shared state; fix **independent** defects sequentially so instructions stay focused. --- ## Structured Outputs & Schema Design ### Three ways to get structured data from a model | Mechanism | Claude implementation | Gemini / Vertex AI implementation | Syntax guaranteed? | Best fit | | :--- | :--- | :--- | :---: | :--- | | **1. Prompt-only formatting** | System/user prose or output style | System/user prose instructions | ❌ No | Streaming human-readable Markdown, YAML plans, UI prose | | **2. Native JSON Schema output** | `output_config={"format": {"type": "json_schema", "schema": …}}` (or `claude -p --output-format json --json-schema`) | `GenerateContentConfig(response_mime_type="application/json", response_schema=…)` | ✅ Yes (on normal completion) | Single-step document extraction, classification, CI JSON gates | | **3. Tool-use / function-call extraction** | `tools=[{…, "strict": True, "input_schema": …}]` + `tool_choice` | `tools=[FunctionDeclaration(…)]` + `FunctionCallingConfig` | ✅ Yes (with strict mode) | Multi-schema routing (`tool_choice: "any"`), agent pipelines, mandatory first-step extraction | ### Writing a schema-constrained output contract A schema is part of the prompt: property names, enums, and `description` strings are all read by the model while constrained decoding enforces the shape. ```json {"type": "object", "additionalProperties": false, "required": ["intent", "evidence_span", "urgency"], "properties": { "intent": {"type": "string", "enum": ["billing", "access", "bug", "other"]}, "evidence_span": {"type": ["string", "null"], "description": "Verbatim quote that supports intent"}, "urgency": {"type": "string", "enum": ["low", "high", "unclear"]}}} ``` * **Put field-level rules in `description`:** per-field extraction rules (*"verbatim quote"*, *"ISO-8601 date"*) sit next to the field; global policy stays in the system instruction. * **One contract, not two:** never describe a JSON shape in prose with different key names from the schema. Conflicting contracts produce inconsistent fields when constraints are relaxed or the model changes. * **Control property order where it matters:** Gemini's `response_schema` supports `propertyOrdering`; when a rationale or evidence field should be generated before the decision, order it first. * **Shape is not truth:** a schema-valid object can still hold a wrong label or a quote that is not in the source, so keep the validation layer in place. ### Controlling tool selection with `tool_choice` / `FunctionCallingConfig` | Mode | Claude `tool_choice` | Gemini `tool_config` mode | Behavior & production use case | | :--- | :--- | :--- | :--- | | **Auto** | `{"type": "auto"}` | `mode="AUTO"` | Model may return conversational text *or* call a tool. Default for multi-turn agents so the loop can end with `end_turn`. | | **Any (must call a tool)** | `{"type": "any"}` | `mode="ANY"` (`allowed_function_names` optional) | Model **must** invoke one of the provided tools, choosing which schema fits. Ideal when routing an unknown document across `extract_invoice`, `extract_receipt`, and `extract_w9`. | | **Forced named tool** | `{"type": "tool", "name": "extract_metadata"}` | `mode="ANY"`, `allowed_function_names=["extract_metadata"]` | Model **must** call that exact tool. Use to guarantee a required first extraction step (`extract_metadata`) before downstream enrichment turns. | | **None** | `{"type": "none"}` | `mode="NONE"` | Forbids tool calls while keeping tool definitions visible in context. | > **Compatibility trap:** Forced tool selection (`"any"` or `"tool"`) is incompatible with manual extended thinking and is rejected on certain newer reasoning configurations; when forced mode is unavailable, pair `output_config.format` (for single schemas) or `tool_choice: "auto"` + `strict: true` + explicit prompt routing. Also remember to inspect `stop_reason` (`"end_turn"` / `"tool_use"` vs. `"refusal"` or `"max_tokens"`) before assuming a response satisfies your schema. ### Schema design rules that prevent hallucination | Schema design decision | Anti-pattern | Production pattern | Why it matters | | :--- | :--- | :--- | :--- | | **Optional / missing source data** | Marking every property non-null `required` "for completeness" | Make potentially absent fields **nullable** (`"type": ["string", "null"]`) or optional, and state *"use `null` when not explicitly stated in the source"* | A non-null required field forces constrained decoding to **fabricate** a plausible date, PO number, or name when the document omits it | | **Open-world categories** | Fixed enum `["billing", "technical", "shipping"]` | Fixed enum **plus `"other"`** paired with a nullable `"category_detail"` string | Preserves clean downstream aggregation without forcing novel cases into wrong buckets | | **Genuine source ambiguity** | Binary enum `["approved", "rejected"]` | Add **`"unclear"`** (plus `"ambiguity_note"`) | Gives the model a truthful escape hatch when a contract draft is unsigned or contradictory | | **Messy surface formats** | Relying on `"type": "string"` alone for dates/units | Keep `"type": ["string", "null"]` **and** put explicit normalization rules in the prompt (*"Convert all dates to ISO-8601 `YYYY-MM-DD`; convert weights to integer grams; strip currency symbols"*) | JSON Schema validates types (`string`, `number`), not how `"14/05/26"` vs. `"May 14"` was interpreted | | **Strict-schema constraints** | Putting `minimum` / `maximum` or unbounded dicts into strict schemas | Set `"additionalProperties": false` on every object; enforce numeric ranges, cross-field sums, and regexes in application validator code | Strict constrained-decoding compilers support a subset of JSON Schema keywords and cap optional/union parameter counts; unsupported keywords return HTTP 400 | --- ## Validation, Retry & Dismissal Feedback Loops ### Syntax errors vs. semantic errors vs. absent information Constrained JSON decoding (`output_config.format`, `strict: true`, `response_schema`) **eliminates syntax and type errors**—you will not get trailing commas, unquoted keys, or a string where a number was declared. It **never eliminates semantic errors**: line items that fail to sum to the invoice total, a shipping date copied into `invoice_date`, or two conflicting figures silently resolved by guessing. | Failure class | Concrete example | Caught by strict schema? | Will retry-with-feedback fix it? | Correct system action | | :--- | :--- | :---: | :---: | :--- | | **Syntax / structural** | Truncated JSON, missing required key, string in float slot | ✅ Prevented upfront (unless `max_tokens` / `refusal`) | Yes (if not using strict mode) | Use strict schemas + check `stop_reason` | | **Semantic / format mismatch** | `sum(line_items) == 940.00` while `stated_total == 1040.00`; date written `05/14/2026` instead of `2026-05-14` | ❌ No | ✅ **Yes** (1–2 bounded retries) | Re-prompt with original source + failed JSON + exact validator message | | **Information absent from source** | Contract references *"Exhibit B pricing schedule"*, which was never uploaded | ❌ No | ❌ **No — retrying causes fabrication!** | Stop retrying; return `null`, fetch the missing exhibit, or route to human review | ### Designing self-verifying schemas and retry loops * **Self-check fields in the extraction schema:** * **`stated_total` + `calculated_total`:** ask the model to extract the printed invoice total *and* compute/list individual line items so deterministic Python code checks `abs(sum(items) - stated_total) < 0.01` and flags discrepancies. * **`conflict_detected` (boolean) + `conflict_notes`:** when a multi-page packet states two different effective dates or revenue numbers, require the model to set `conflict_detected: true` and record both values rather than picking one silently. * **Anatomy of a retry-with-error-feedback turn:** include (1) the **original source document**, (2) the **failed JSON extraction**, (3) the **exact programmatic validation error** (*"`line_items` sum to 410.00, which does not match `stated_total` 460.00; check whether a shipping or tax line was omitted"*), and (4) an explicit reminder to return `null` rather than guessing if the source genuinely lacks the figure. Cap retries at **1–2 attempts**. * **Reviewer-override telemetry:** when a human reviewer corrects or rejects a model output, log the schema path of the overridden field, the prompt version, and one reason code from a fixed list (`wrong_field`, `missed_null`, `bad_normalization`, `not_in_source`). Aggregating overrides by field and reason code shows which instruction or example is ambiguous; fix it with a targeted contrast example and re-run the evaluation set before releasing the next prompt version. * **Calibrating human review routing:** never disable human review based on a single aggregate metric (*"97% overall accuracy"* can hide 62% accuracy on handwritten amendments or `tax_id` fields). Slice accuracy by **document type × field**, have the model emit **field-level confidence** alongside `conflict_detected`, calibrate thresholds against a labeled validation set, route low-confidence or conflicting records to reviewers, and maintain **stratified random sampling** of high-confidence auto-approved extractions across every document type to catch drift and novel error modes. --- ## Batch Processing vs. Synchronous Execution ### Matching API transport to workflow latency and interactivity Both Anthropic (**Message Batches API**) and Google Cloud (**Gemini / Vertex AI Batch Prediction**) offer asynchronous batch endpoints priced at **50% of synchronous token rates** with high throughput quotas and up to a **24-hour completion window**, with **no guaranteed per-request latency SLA**. | Dimension | Synchronous API (`messages.create` / `generate_content`) | Asynchronous Batch API (`messages.batches` / Batch Prediction) | | :--- | :--- | :--- | | **Pricing** | Standard input/output rates | **50% cost reduction** on input and output tokens | | **Completion SLA** | Interactive (seconds); streaming supported | Up to **24 hours** (often faster, never guaranteed; unprocessed items expire at 24h) | | **Multi-turn tool execution** | ✅ Supported: your loop executes tools between turns | ❌ **Single model turn per request:** cannot execute tools mid-request and feed `tool_result` back inside the same batch run | | **Single-shot extraction via forced `tool_use`** | ✅ Supported | ✅ **Supported:** because the `tool_use` input block *is* the final structured record, no second turn is needed | | **Result ordering & correlation** | Immediate 1:1 HTTP response | Out-of-order results; correlate every item via required **`custom_id`** | | **Ideal workloads** | Live chat replies, interactive IDE assistance, agents that call tools between turns, any request a person is waiting on | Offline evaluation sweeps, product-catalog enrichment, back-filling labels on archived tickets, overnight log summarization | ### Prompt chaining across batch runs Each batch request is a single model turn, so a multi-stage pipeline runs as a **chain of batches**: parse and validate stage *k* results in code, then embed them in the stage *k+1* requests. | Stage | Request content | Hand-off to the next stage | | :--- | :--- | :--- | | **1. Extract** (batch) | System prompt + `` + JSON schema | Parsed records keyed by `custom_id` | | **2. Validate** (code, no model call) | Schema parse, cross-field checks | Valid records forward; failures to a retry batch or review queue | | **3. Synthesize** (batch or sync) | Validated records inside `` + task instruction | Final summary or report | * **Hand off structured data inside named tags:** pass validated JSON in `` and instruct the next stage to rely only on that block. Re-sending the raw source alongside invites the model to re-extract and contradict stage 1. * **Version each stage prompt separately:** a chain localizes failures only if every stage has its own template, schema, and evaluation set. * **Dry-run request bodies synchronously:** send a handful of representative requests through the Messages API first; a malformed request only shows up as an `errored` result after the batch has run. * **Correlate by meaningful `custom_id`:** results arrive in any order. Use ids that encode record and stage (`tkt-88412-s1`); resend `expired` items as they are, and fix the request before resending `errored` ones. > **Design note:** Batch fits work nobody is actively waiting on. When a person or a downstream gate blocks on the result, stay synchronous (with prompt caching for the shared prefix) regardless of the discount. --- ## Multi-Instance & Multi-Pass Review Architectures ### Chain-of-thought isolation: keep reasoning out of the answer Reasoning improves hard decisions, but downstream code and end users should consume only the final result. Give reasoning its own channel and parse only the answer channel. | Mechanism | How reasoning is separated | What your code consumes | | :--- | :--- | :--- | | **Prompted CoT** (thinking mode off) | Instruct the model to reason inside `` and place the result inside `` | Only the `` contents; log `` for debugging | | **Claude extended / adaptive thinking** | The response carries `thinking` content blocks separate from `text` and `tool_use` blocks | The `text` or `tool_use` blocks, selected by block type | | **Gemini thinking** | Thoughts stay internal; with `include_thoughts` enabled, thought summaries arrive as parts flagged `thought` | Parts not flagged as thoughts | | **Schema field** | A short `rationale` property ordered before the decision fields | The decision fields; `rationale` is audit text, never data | * **Why isolate:** reasoning text leaking ahead of JSON breaks parsers, exposes scratch work to users, and can echo untrusted input back into downstream systems. * **Never demand both "think step by step" and bare JSON:** without a dedicated channel the model either skips the reasoning or pollutes the output. Enable thinking, add a tag pair, or add a schema field. * **Grade answers, not explanations:** a fluent rationale is not proof of a correct decision; evaluate the parsed answer against labeled data. ### Multi-pass critique chains | Pattern | Context given to the checking call | Best fit | | :--- | :--- | :--- | | **Same-session "check your work"** *(weak)* | Full generation history | Catching obvious slips only; the model tends to defend its own earlier reasoning | | **Draft → critique → revise chain** | Critique call receives the draft and rubric; revise call receives draft plus critique | Raising quality while logging every intermediate output | | **Independent grader instance** | Fresh context with only the artifact, source, and rubric | Grounding and policy checks free of the generator's assumptions | | **Multi-sample agreement** | Several independent samples of the same task | Verifiable answers (labels, numbers) where agreement signals reliability | 1. **Draft** with the stage-1 prompt and store the output. 2. **Critique** in a separate call containing ``, ``, and ``; return a schema list of issues (`criterion`, `quote`, `fix`). 3. **Revise** in a third call that receives the draft and the issue list and returns the result inside `` or a schema. 4. **Stop deterministically:** end after 1–2 rounds or when the critique returns an empty list, and keep every stage in the trace for evaluation. --- ## Prompt Caching, Security & Enterprise LLMOps ### Prefix ordering for prompt caching Both Claude (`cache_control`) and Gemini (implicit and explicit context caching) match caches on **exact byte-for-byte prefixes**. Changing a timestamp, user name, or tool schema near the top of a prompt invalidates everything beneath it. | Position in request | Content | Mutability | | :--- | :--- | :--- | | **1. Top (Prefix)** | Tool definitions → system instructions → normalization rubrics → `` → shared reference corpus | Static across calls (**cached**) | | **2. Middle** | Session history / prior turns (with incremental cache breakpoints) | Append-only across turns | | **3. Bottom (Suffix)** | Current document, PR diff, per-request variables, and final task query | Changes every call (**uncached**) | * **Claude cache economics:** 5-minute TTL writes cost **1.25×** base input tokens (break-even after **1** cache read at ~**0.1×**); 1-hour TTL writes cost **2×** base input tokens (break-even after **2** reads). Keep `thinking` / `effort` settings stable within a workload so message-history caches stay warm. ### Defending prompts against untrusted inputs * **Prompts are not security boundaries:** a clever indirect prompt injection inside a parsed PDF or PR description can urge the model to ignore instructions or call a destructive tool. * **Defense-in-depth stack:** (1) isolate untrusted text inside explicit XML tags (`…`) and instruct the model to treat tag contents strictly as data; (2) restrict agent tool sets to least privilege (`Read`, `Grep`, `Glob` only on untrusted review runs); and (3) enforce deterministic policy checks (refund thresholds, allowed file paths, schema validation) in **application code or `PreToolUse` hooks**, never solely in prose. ### Prompt versioning, automated optimization, and when to fine-tune * **Treat prompts and schemas as versioned code:** store prompt templates, JSON schemas, model IDs (pinned snapshots such as `claude-sonnet-4-6`), and `effort` / `thinking_level` configs in version control gated by regression evals ([AgentOps](https://mbettan.github.io/machine-learning-cert/AgentOps.html#grading-hierarchy)). * **Automated prompt optimization (APE / Vertex AI Prompt Optimizer):** once you have a labeled golden dataset and programmatic metric (schema F1, per-category precision), optimizer loops can systematically mutate instructions and select few-shot examples to lift accuracy—especially when migrating between model families. * **Prompting vs. Fine-Tuning (SFT / LoRA):** | Lever | Changes weights? | Reach for it when… | Avoid it when… | | :--- | :---: | :--- | :--- | | **Prompting + Few-Shot + Schema** | ❌ No | Iterating rapidly, enforcing JSON structure, reasoning over fresh context/RAG, low-to-medium volume | Prompt prefix exceeds latency/cost budgets on millions of identical narrow calls | | **Retrieval (RAG) / Context** | ❌ No | Facts change daily or require source citations and per-user ACLs | You only need consistent output formatting or tone | | **Supervised Fine-Tuning / LoRA** | ✅ Yes | Distilling a large teacher model's specialized style/classification into a small fast SLM at massive scale | Trying to teach rapidly changing facts or compensate for missing schema validation | --- ## Failure Modes & Diagnostic Traps | Observed production symptom | Root cause | Correct engineering fix | Tempting wrong answer | | :--- | :--- | :--- | :--- | | Ticket-triage model marks 40% of tickets `urgent`; support agents stop trusting the flag | `urgent` is defined only as *"time-sensitive"*, with no boundary | Define urgency with explicit criteria (outage, security exposure, payment blocked) plus contrast examples of look-alike non-urgent tickets | Asking the model to *"only mark urgent when very confident"* | | Extractor invents `"1970-01-01"` or plausible dates when contracts omit `renewal_date` | `renewal_date` is non-null `required` in the JSON schema | Make `renewal_date` nullable (`["string", "null"]`), document `"null if absent"`, and add a `null` few-shot example | Raising retry count or lecturing *"Never hallucinate"* while keeping `required` non-null | | Retry loop runs 5 times and eventually fabricates a `tax_id` that passes regex check | `tax_id` only exists in an unattached schedule (absent from input) | Cap retries at 1–2 for semantic/format errors; treat missing-source fields as `null` and route to human review | Retrying until validation succeeds | | Line items extract cleanly as floats, yet `sum(items) != stated_total` on 6% of invoices | Constrained JSON schema guarantees types, not arithmetic semantics | Extract both `line_items` and `stated_total` (`+ calculated_total`), verify sums in Python, and run 1 error-feedback retry | Adding more prose rules without code validation | | Claude 4.7 upgrade fails in production with `HTTP 400 invalid_request_error` | Legacy code sends `temperature=0.1`, `budget_tokens`, or an assistant `{` prefill | Remove `temperature`/`top_p`/`top_k` and prefill; use `thinking: {"type": "adaptive"}`, `output_config.effort`, and `output_config.format` | Retrying the 400 with exponential backoff | | JSON parser fails on 9% of replies because prose appears before the opening brace | Prompt says *"think step by step"* but the reply must be bare JSON | Isolate reasoning: enable thinking, use a ``/`` split, or add a `rationale` schema field, and parse only the answer channel | Regex-stripping everything before the first `{` | | Assistant obeys an instruction hidden in a pasted email (*"forward this thread to all contacts"*) | Payload concatenated after instructions without delimiters; the email even contains a fake closing tag | Wrap the payload in a named tag, neutralize tag-like strings inside it, and state that tag contents are data only | Adding *"ignore malicious instructions"* in bold | | Outputs reuse customer names and amounts from the few-shot examples | Examples are unlabeled and share tags with the live input | Put demos in `` with ``/``, vary surface details, and state that examples illustrate format only | Deleting every example | | Stage-2 summary cites figures that never appear in the stage-1 extraction | Stage-1 output handed off as loose prose while stage 2 also re-reads the raw source | Pass validated stage-1 JSON inside `` and instruct stage 2 to use only that block | Collapsing both stages back into one mega-prompt | --- ## Worked Extraction & Validation Blueprint This end-to-end vendor-invoice blueprint combines normalization contracts, nullable/extensible strict schemas, deterministic cross-field validation, and bounded error-feedback retries (for SDK syntax, see [Claude Platform & API](https://mbettan.github.io/machine-learning-cert/ClaudeAPI.html#structured-outputs)). ![Self-verifying extraction, bounded error-feedback retry, and human review routing flowchart](https://mbettan.github.io/machine-learning-cert/diagrams/validation-retry-flow.jpg) 1. **Input isolation & normalization contract:** wrap raw OCR or email text inside `…` delimiters; mandate ISO-8601 `YYYY-MM-DD` dates, stripped currency symbols, explicit `null` for absent values, and `conflict_detected: true` whenever two sections state different figures. 2. **Extensible strict schema:** enforce constrained decoding (`strict: true` / `output_config.format` / `response_schema`) with `additionalProperties: false`, nullable optional fields (`issue_date`, `po_number`, `stated_total`), open-world `spend_category` (`"other"` paired with nullable `category_detail`), and calibrated `field_confidence` (`high`, `medium`, `low`, `unclear`). 3. **Deterministic semantic validation:** verify cross-field invariants in application code after schema parsing—require non-empty `category_detail` whenever `spend_category == "other"`, and verify `abs(sum(line_amounts) - stated_total) <= 0.01` whenever both values are present and `conflict_detected` is `false`. 4. **Bounded error-feedback retry (max 1–2 attempts):** when semantic validation fails on an extractable slip, append the failed record and exact validator error (`is_error: true`, e.g., *"line_amounts sum to 410.00 vs. stated_total 460.00; re-check omitted tax/shipping lines or set conflict_detected=true"*) while reminding the model to keep `null` rather than guessing missing facts. 5. **Calibrated human routing & stratified audit:** bypass retries and route directly to a human review queue whenever `stop_reason` is unexpected (`max_tokens`, `refusal`), `conflict_detected` is `true`, calibrated `field_confidence` is `low`/`unclear`, or the retry cap expires—and continuously audit a stratified random sample of verified records across every document type. --- ## Glossary | Term | Definition | | :--- | :--- | | **Adaptive Thinking** | Dynamic test-time reasoning mode (`thinking: {"type": "adaptive"}`) scaled via `output_config.effort`, replacing fixed `budget_tokens`. | | **Attention dilution** | Recall drop caused by packing too many documents, rules, or tasks into one prompt; cured by splitting work into focused chained or parallel calls. | | **Chain-of-thought isolation** | Giving reasoning its own channel (thinking blocks, `` tags, or a `rationale` field) so code parses only the final answer. | | **Class mixing** | Interleaving target labels across few-shot examples so the model does not over-predict whichever class appeared first or last. | | **Constrained decoding** | Token-level grammar masking (`output_config.format`, `strict: true`, `response_schema`) that guarantees schema-valid JSON syntax. | | **Contrast pair** | Two few-shot examples pairing a genuine positive case with a near-miss acceptable case that should *not* be flagged. | | **`custom_id`** | Per-request correlation string in Batch APIs used to match out-of-order results and resubmit only failed items. | | **Independent review instance** | Separate model call with clean context (artifact + rubric only) that avoids anchoring on the generator's reasoning trace. | | **Interview pattern** | Prompting the model to ask clarifying architectural and edge-case questions before generating an implementation. | | **Lost in the Middle** | U-shaped positional recall effect where models attend strongly to prompt start and end while missing details buried mid-window. | | **Nullable schema field** | Property typed `["string", "null"]` (or optional) so the model outputs `null` rather than fabricating data absent from the source. | | **Prompt chaining** | Splitting a task into sequential calls whose validated outputs are passed to the next stage inside named tags. | | **Retry with error feedback** | Bounded follow-up turn supplying the source, failed extraction, and exact validator error so the model repairs semantic/format slips. | | **Semantic validation error** | Output that parses against the JSON schema yet violates domain invariants (e.g., line items not summing to `stated_total`). | | **Stratified random sampling** | Ongoing human audit sampling across every document type and field stratum—including high-confidence auto-approved outputs—to measure true error rates. | | **`tool_choice`** | Parameter (`auto`, `any`, forced `tool`, `none`) controlling whether the model may or must invoke a tool on the current turn. | | **XML tag delimiting** | Wrapping each content type in consistently named tags so instructions, examples, and untrusted input stay unambiguous. | ================================================================================ GUIDE: Agentic Design Patterns URL: https://mbettan.github.io/machine-learning-cert/ADP.html Practice quiz: https://mbettan.github.io/machine-learning-cert/quiz.html?topic=adp Section: Foundation · Agent Architecture & Protocols Exam relevance: Google Professional Agentic Architect (core), Claude Certified Architect – Foundations (core) ================================================================================ # Agentic Design Patterns (ADP): Study Guide > **Author:** Michaël Bettan > **Version:** 1.0 --- ## 00. Overview & Mental Model **Definition:** Agentic design patterns are reusable, vendor-neutral architectural blueprints that govern how an LLM-based system decomposes work, selects control flow, invokes tools, manages context memory, coordinates across specialized workers, and defers high-stakes decisions to deterministic code or human reviewers. They apply identically across Google ADK, the Claude Agent SDK, LangGraph, or direct model API loops. **What agentic design patterns solve for:** A raw language model is a stateless, probabilistic token predictor—it can *propose* an action, but it cannot *execute* tools, enforce hard business invariants, persist session state, or halt a runaway loop. Architectural patterns define **who owns the control flow** (deterministic application code vs. dynamic model reasoning), **where authority sits** (code gates vs. model judgment vs. human sign-off), and **how context stays clean** as tasks scale. > **Mental model:** > The model is the **probabilistic reasoning engine**; the architecture is the **deterministic harness** (nervous system, brakes, and safety interlocks) that decides when the model thinks, when tools execute, and when a human must approve. > **Rule 0 (The Simplicity Ladder):** > **Start with the simplest architecture that passes your evaluation suite.** Climb from deterministic code $\rightarrow$ a single augmented LLM call $\rightarrow$ a predefined workflow $\rightarrow$ an autonomous agent **only when empirical evaluations prove the simpler tier fails.** ### LLM vs. Workflow vs. Autonomous Agent vs. Harness | Layer | Control Owner | Role & Runtime Behavior | Memory Hook | | :--- | :--- | :--- | :--- | | **Raw LLM** | Caller (1 turn) | Stateless token predictor; reasons over prompt context and emits text or structured tool requests, but never executes code. | *"Brain in a jar"* | | **Augmented LLM** | Caller (1 turn + tools) | Single model call equipped with **Retrieval** (RAG), **Tools** (MCP/functions), and **Memory** (read/write state). Handles most production features. | *"LLM with a backpack"* | | **Composable Workflow** | **Application code** | Multi-step pipeline where LLMs and tools execute along **predefined code paths** with deterministic gates between steps. | *"Railway tracks"* — train cannot leave the rails | | **Autonomous Agent** | **The model** | LLM dynamically directs its own multi-turn process, choosing which tools to call and when to stop based on environmental ground truth. | *"Taxi"* — driver picks the route dynamically | | **Agent Harness / Framework** | Infrastructure | Runtime layer (ADK, Claude Agent SDK, LangGraph) supplying the event loop, state persistence, sandboxing, callbacks, and observability. | *"Chassis, brakes & dashboard"* | > [!WARNING] > **Classic architectural trap:** Calling an LLM five times in a Python script does **not** make the system an "autonomous agent." If application code dictates the sequence, branching, and retry count, it is a **workflow**. It becomes an **agent** only when the model itself decides the next step and action trajectory at runtime. --- ## 01. Architectural Triage & Sequential Decision Framework (`#triage-decomposition`) Before writing a single system prompt or selecting an orchestration framework, an architect must resolve four sequential decisions. Skipping step 1 or 2 is the primary cause of over-engineered, brittle, or unsafe agentic deployments. ### Sequential Architectural Decision Framework ``` ┌────────────────────────────────────────────────────────────────────────────┐ │ Stage 1: Problem Decomposition & Execution Engine Triage │ │ └─► Triage every sub-task across Tripartite Execution Regimes │ ├────────────────────────────────────────────────────────────────────────────┤ │ Stage 2: Work-Shape Selection │ │ └─► Single Augmented Call vs. Composable Workflow vs. Autonomous Agent │ ├────────────────────────────────────────────────────────────────────────────┤ │ Stage 3: Execution Routing, Model Tiering & Hosting Infrastructure │ │ └─► Model tier routing (fast vs. frontier), sync/stream/batch, residency │ ├────────────────────────────────────────────────────────────────────────────┤ │ Stage 4: Guardrail Wrapping, Invariant Verification & Human-in-the-Loop │ │ └─► Pre-LLM epistemic gates, tool-boundary hooks, auditors, HITL gates │ └────────────────────────────────────────────────────────────────────────────┘ ``` 1. **Stage 1 — Problem Decomposition & Execution Engine Triage:** Separate the problem into deterministic rules, probabilistic reasoning, and human accountability using the **Tripartite Execution Regimes** below. 2. **Stage 2 — Work-Shape Selection:** Select the lowest rung on the complexity ladder (Single Call $\rightarrow$ Workflow $\rightarrow$ Autonomous Agent) using the **Architectural Evaluation Criteria** (§02). 3. **Stage 3 — Execution Routing, Model Tiering & Hosting Infrastructure:** Match each step to the right model tier and reasoning effort (e.g., high-capability planner producing a spec for a fast/balanced executor), transport mode (streaming vs. async batch), and regional data boundary. 4. **Stage 4 — Guardrail Wrapping, Invariant Verification & Human-in-the-Loop Controls:** Surround every probabilistic step with deterministic pre-checks, schema validators, tool-boundary hooks, independent critique, and Human-in-the-Loop (HITL) checkpoints. ### Tripartite Execution Regimes: Deterministic Code, Probabilistic Inference & Human Oversight Every enterprise workflow is a hybrid of three distinct execution regimes. Never assign Regime 1 or Regime 3 responsibilities to Regime 2: | Execution Regime | Execution Engine | What Belongs Here | Canonical Examples | Why Other Regimes Fail | | :--- | :--- | :--- | :--- | :--- | | **Regime 1: Deterministic Code** | Plain application code, SQL, regex, type schemas, policy hooks | Exact arithmetic, schema validation, RBAC & permission enforcement, idempotency keys, database writes, PII redaction, hard policy gates | Fee & tax calculations, date math, checking `amount <= 500`, enforcing `user_id` tenant scoping, verifying cited section headers exist in source text | LLMs introduce **deterministic invariant degradation** (silent tail-risk arithmetic or rule violations); humans are too slow and inconsistent at scale | | **Regime 2: Probabilistic Model (LLM)** | Augmented LLM, Workflow node, or Agent loop | Unstructured synthesis, semantic classification, ambiguous entity extraction, fuzzy matching, multi-step planning over open-ended inputs | Summarizing conflicting legal clauses, triaging free-text support tickets, diagnosing novel stack traces across a codebase, drafting code diffs | Rigid code/regex shatters on phrasing variations; humans cannot manually read 50,000 documents per hour | | **Regime 3: Human-in-the-Loop (HITL)** | Human reviewer via durable approval queue / UI checkpoint | High-blast-radius irreversible actions, novel policy exceptions, regulated sign-offs, subjective value trade-offs | Approving a `$25,000` wire refund, executing a production database drop, granting an exception to legal policy, merging security-critical PRs | Code lacks contextual judgment for novel exceptions; models cannot bear legal, financial, or ethical accountability | ### Hard Invariant Enforcement in Code (Preventing Deterministic Drift) **Deterministic drift** occurs when an engineering team delegates exact, rule-bound logic—such as calculating pro-rated subscription refunds, enforcing role-based access control (RBAC), checking date eligibility windows, or redacting Social Security Numbers—to a system prompt instead of application code. - **Why it is dangerous:** Even when a frontier model follows a prompt rule on the vast majority of test runs, probabilistic token sampling guarantees a non-zero tail-risk failure rate—especially under **instruction crowding** (long prompts with 20+ competing rules) or adversarial prompt injection. - **Architectural rule ("Deterministic Handcuffs"):** Let the LLM extract structured parameters or propose an action, but execute all math, eligibility comparisons, permission checks, and compliance gates in **deterministic code** (`PreToolUse` / `before_tool_callback` hooks, Pydantic validators, or SQL constraints). --- ## 02. Work-Shape Spectrum & Architectural Evaluation Criteria (`#work-shape-matrix`) Once Regime 2 (the probabilistic work) is isolated, choose between the three primary work shapes: **Single Augmented LLM Call**, **Composable Workflow**, and **Autonomous Agent**. ### Architectural Evaluation Criteria: Single Call vs. Workflow vs. Autonomous Agent | Evaluation Criterion | Single Augmented LLM Call | Composable Workflow (DAG / Chain) | Autonomous Agent (Open Loop) | | :--- | :--- | :--- | :--- | | **1. Task Path Predictability** | **Fully known (1 step):** Input maps directly to output with retrieved context or a single tool lookup. | **Known decomposition ($K$ steps):** Multi-step process where the sequence, branches, or fan-out categories are fixed in advance. | **Unknown / emergent ($N$ steps):** Step $N+1$ depends entirely on what tool observation $N$ revealed (e.g., debugging an unknown bug). | | **2. Latency Budget** | **Sub-second to ~3 s:** Single model roundtrip; suitable for synchronous inline UI responses. | **~3 s to ~15 s:** Predictable bounded latency ($K$ sequential stages or $1$ parallel fan-out + join). | **Seconds to minutes+:** Open-ended $O(N)$ sequential turns; requires async streaming, progress UI, or background execution. | | **3. Error Blast Radius & Reversibility** | **Low / read-only:** Output is displayed to a user or validated by a caller before any state mutation. | **Medium / gated:** Intermediate code gates catch errors between stages before downstream side-effects run. | **High if ungated:** Multi-turn tool mutations can compound; requires sandboxing, checkpoints, and HITL gates on writes. | | **4. Token Cost Ceiling** | **~1× baseline:** Minimal prompt overhead; highest prompt-cache hit ratio. | **~3–5× baseline:** Bounded multiplier across $K$ focused prompts; easy to route cheap tiers to simple nodes. | **~10–15×+ baseline:** Context accumulates across turns and tool outputs; requires strict token/turn caps and compaction. | | **5. Auditability & Compliance Rigor** | **High:** Single input/output pair; trivial to log and reproduce. | **Highest for multi-step:** Deterministic transitions and explicit code gate pass/fail logs satisfy strict regulatory audits. | **Complex:** Nondeterministic trajectories require full event-ledger tracing (`tool_trajectory`) and replay harnesses. | ### Inherent Operating Characteristics of Probabilistic Systems Every agentic architecture must explicitly defend against four inherent properties of language models: 1. **Non-determinism:** Identical prompts can yield different phrasings or tool trajectories across runs (and modern frontier APIs reject non-default sampling knobs like `temperature: 0` or still exhibit floating-point variance). *Mitigation:* Grade structured properties and behavioral rubrics, never exact string equality; enforce JSON schemas at the token level. 2. **Plausible-sounding errors (Hallucinations & Silent Corruption):** Models are trained to complete patterns helpfully. When fed garbled OCR text, an empty string `""` from a timed-out tool, or a missing policy clause, an unguarded model will invent confident facts or fabricated citations (`Section 99.1`). *Mitigation:* Pre-LLM epistemic quality gates, explicit `is_error: true` tool results, and deterministic citation verification against source text. 3. **Context sensitivity (Attention Dilution & Instruction Crowding):** Packing 30 rules, 15 irrelevant documents, and 40 tool schemas into one call degrades recall in the middle of the window and causes the model to silently drop late audit rules. *Mitigation:* Split generation from verification, keep tool catalogs $\le 16$ tools per agent, and isolate noisy exploration in subagents. 4. **Compounding multi-step error rates:** If a single autonomous step has a $95\%$ reliability rate, a 10-step unverified chain succeeds only $0.95^{10} \approx 60\%$ of the time, because an early misread file or wrong assumption poisons every subsequent reasoning turn. *Mitigation:* Insert **ground-truth environmental feedback** (compilers, unit tests, schema validators, deterministic gates) after every step to reset the error curve. --- ## 03. The 5 Canonical Workflow Patterns When a single call falls short because of instruction crowding, multi-stage dependencies, or latency/cost constraints—but the overall control structure is predictable—use one or more of the **five composable workflow patterns**. In all five, **application code owns the control flow** (or bounds the loop). | Pattern | Control Flow Shape | Primary Optimization | Canonical Use Case | Memory Hook | | :--- | :--- | :--- | :--- | :--- | | **1. Prompt Chaining** | `LLM 1 ──► [Code Gate] ──► LLM 2 ──► LLM 3` | Accuracy via focused sub-steps + early fail-fast | Drafting an outline $\rightarrow$ verifying structure in code $\rightarrow$ generating full localized report | *"Assembly line with quality inspectors"* | | **2. Routing** | `Classifier ──► Specialist Handler A | B | C` | Separation of concerns & per-tier cost optimization | Directing billing, refund, and tech-support tickets to distinct prompts, tools, and model tiers | *"Hospital triage desk"* | | **3. Parallelization** | `Fan-Out (N calls) ──► Programmatic Aggregator` | Wall-clock speed (*Sectioning*) or high confidence (*Voting*) | Running content safety screening in parallel with answer generation (*Sectioning*); multi-prompt code security audit (*Voting*) | *"Sectioning = slice the pizza; Voting = poll 5 judges"* | | **4. Orchestrator–Workers** | `Orchestrator LLM ──► Dynamic Workers (1..K) ──► Synthesizer` | Runtime task decomposition when sub-tasks depend on input | Multi-file refactoring or multi-source research where the target files/sources aren't known upfront | *"Tech lead assigning tickets after reading the spec"* | | **5. Evaluator–Optimizer** | `Generator ──► Critic/Auditor ──► (Pass | Feedback Loop [max M])` | Iterative quality & compliance against a strict rubric | Legal clause extraction + adversarial citation auditor; literary translation refinement | *"Writer + strict editor"* | ### Pattern 1: Prompt Chaining (Sequential Pipeline with Code Gates) Decomposes a complex task into a fixed sequence of narrower LLM calls, where each step consumes the validated output of the previous step. | Pipeline Stage | Component | Functionality | Invariant / Exit Condition | | :--- | :--- | :--- | :--- | | **Step 1: Extraction** | Extraction LLM | Extracts structured clauses and cited line anchors from source text. | Emits structured JSON matching schema. | | **Deterministic Code Gate** | Application Code | Validates mandatory fields (`termination_clause`) and citations list. | If missing or invalid, halts execution immediately ($0 downstream spend). | | **Step 2: Risk Assessment** | Assessment LLM | Analyzes validated clauses against risk rubrics. | Emits final structured risk assessment report. | - **The Gate (`gate_check`):** Always insert a deterministic code check (schema validation, regex citation lookup, length/completeness check) between steps so corrupted intermediate outputs fail fast for `$0` rather than poisoning downstream calls. - **When NOT to use:** Do not chain steps that have no data dependency (use *Parallelization* instead to avoid serial latency) or tasks that a single well-structured prompt already solves cleanly. ### Pattern 2: Routing (Intent & Complexity Dispatch) Classifies an incoming request (via a fast small model, embedding similarity, or deterministic rules) and dispatches it to a specialized prompt, scoped toolset, or appropriate model tier. - **Why routing beats a monolith prompt:** Optimizing a single mega-prompt to handle billing disputes, technical debugging, and account cancellations simultaneously causes instruction clash and forces every call to pay for the largest model tier and full tool catalog. - **Model-tier routing:** Route high-volume, routine intents to a fast, low-cost tier (e.g., Haiku / Gemini Flash-Lite) with minimal reasoning effort, and reserve frontier tiers (Sonnet / Opus / Gemini Pro) for complex multi-step troubleshooting. - **Preventing misrouting cascades:** Include an explicit `unclear / ambiguous` fallback category that asks a clarifying question or routes to a general handler, log confidence scores, and validate that the target handler rejects out-of-domain requests rather than guessing. ### Pattern 3: Parallelization (Sectioning vs. Voting) Executes multiple LLM calls concurrently (`asyncio.gather` / `ParallelAgent`) and merges their outputs programmatically or via a downstream synthesizer. | Variant | Architecture | Goal | Key Engineering Rule | | :--- | :--- | :--- | :--- | | **Sectioning** | Split a task into **distinct, independent subtasks** or personas run concurrently. | **Speed & focused attention:** Total latency equals the slowest branch rather than the sum of branches. | Keep each branch's output key isolated (`output_key="market"`, `output_key="pricing"`) so concurrent workers never overwrite shared mutable state. Running guardrails/PII screening in parallel with response generation avoids adding serial latency. | | **Voting** | Run the **same task multiple times** (either identical prompt at $T>0$ for *Self-Consistency CoT*, or diverse critique prompts) and aggregate via threshold/majority vote. | **High confidence & recall:** Reduces false negatives on high-stakes reviews. | Tune the vote threshold to match asymmetric risk: e.g., in a security vulnerability scan, flag the diff if **any 1 of 3** reviewers finds an exploit (high recall); in noisy classification, require **2 of 3** agreement (high precision). | ### Pattern 4: Orchestrator–Workers (Dynamic Task Decomposition) A central **Orchestrator LLM** inspects the input, dynamically invents a list of subtasks (because the exact number and nature of subtasks cannot be hardcoded in advance), dispatches **Worker LLMs** with scoped context, and synthesizes their returned summaries. - **Orchestrator–Workers vs. Parallelization:** Both fan out work to multiple calls, but in **Parallelization** the branches are **predefined in code** (e.g., always run Financial, Legal, and Technical checks), whereas in **Orchestrator–Workers** the branches are **determined dynamically at runtime by the orchestrator model** (e.g., inspecting a bug report and deciding which 4 out of 2,000 repository files need targeted worker edits). - **Spec-Driven Planner $\rightarrow$ Executor split:** A high-capability model acts as the read-only orchestrator/planner to resolve ambiguities and emit a structured task specification, then dispatches mid-tier or fast-tier workers in fresh context windows to execute each bounded subtask. | Orchestration Phase | Execution Engine | Mechanism | Guardrail | | :--- | :--- | :--- | :--- | | **Phase 1: Dynamic Decomposition** | Orchestrator LLM | Generates structured task plan based on runtime input inspection. | Strict Pydantic task schema (`subtasks: list[SubTask]`). | | **Phase 2: Scoped Worker Fan-Out** | Parallel Worker LLMs | Workers execute bounded subtasks in fresh, isolated context windows. | Bounded concurrency waves (3–5 workers max) with private outputs. | | **Phase 3: Context Synthesis** | Synthesizer LLM | Merges worker `HandoffContract` summaries into unified deliverable. | Artifact pointer pattern: large diffs stay on disk/storage. | ### Pattern 5: Evaluator–Optimizer (Bounded Generator–Critic Loop) Pairs a **Generator** call with an independent **Evaluator / Critic** call that grades the draft against an explicit rubric and returns actionable revision feedback in a loop. - **Two prerequisites for fit:** (1) Output quality measurably improves when a human provides structured feedback, and (2) an LLM (paired with deterministic checks) can reliably articulate that feedback. - **Separation of Authority (Avoiding Self-Grading Bias):** Never ask the same prompt in the same turn to both draft an answer and grade its own compliance—it will rationalize its own mistakes. Use a separate critic call with a cold context (`include_contents='none'`), an adversarial rubric prompt, and deterministic verification tools (e.g., checking that every cited clause actually appears in the source document). - **Mandatory loop hygiene:** Always enforce `max_iterations` (typically 2–3 rounds) and an explicit convergence/exit signal (`escalate=True` / `status="approved"`). If `max_iterations` trips without passing the rubric, do **not** silently ship the unverified draft—mark the state unapproved and escalate to HITL or a safe fallback. | Loop Phase | Agent / Mechanism | Inputs & Context | Exit / Escalation Criteria | | :--- | :--- | :--- | :--- | | **Generation** | Generator LLM | Task brief + prior critic feedback (if any). | Emits structured draft with cited section headers. | | **Pre-Critic Gate** | Deterministic Code | Source text + draft citations. | Deterministically checks that cited headers exist in source text; fails fast if missing. | | **Evaluation** | Cold-Context Critic LLM | Source text + draft + rubric (`include_contents='none'`). | Emits `approved: bool` and actionable critique. Loop terminates if `approved == True`. | | **Termination / Fallback** | Deterministic Guard | Iteration counter (`attempt <= max_iterations`). | If `attempt > max_iterations` without approval, fails closed to HITL escalation. | ### Cross-Vendor Pattern Vocabulary | Architectural Pattern | Anthropic / General Framing | Google Cloud / ADK Primitive | Graph / State Machine Framing | | :--- | :--- | :--- | :--- | | **Single Building Block** | Augmented LLM (Retrieval + Tools + Memory) | `LlmAgent` with `tools` + `MemoryService` | Single node with tool bindings | | **Pattern 1: Sequential** | Prompt Chaining + Programmatic Gate | `SequentialAgent` / `Workflow` + `before_agent_callback` | Linear edges + conditional gate node | | **Pattern 2: Dispatch** | Routing (Intent / Tier Classifier) | Coordinator (`sub_agents`) or custom router node | Conditional edge router from `START` | | **Pattern 3a: Split Work** | Parallelization — Sectioning | `ParallelAgent` (distinct `output_key`s) + synthesizer | Fan-out branches $\rightarrow$ `JoinNode` | | **Pattern 3b: Consensus** | Parallelization — Voting / Self-Consistency | `ParallelAgent` ($N\times$ runs) + vote aggregator | Parallel ensemble $\rightarrow$ majority/threshold reducer | | **Pattern 4: Dynamic** | Orchestrator–Workers | `LlmAgent` invoking specialist `AgentTool`s | Dynamic map-reduce / send API | | **Pattern 5: Refinement** | Evaluator–Optimizer | `LoopAgent(max_iterations=N)` + generator & critic | Cyclic edge bounded by iteration counter | | **Open-Ended Autonomy** | Autonomous Agent (Tools in a Loop) | `LlmAgent` native ReAct loop (`Runner`) | Cyclic `agent` $\rightleftarrows$ `tools` graph | | **Human Checkpoint** | Planned Checkpoint / HITL Approval | `FunctionTool(..., require_confirmation=True)` | `interrupt()` / `RequestInput` with checkpointer | --- ## 04. Autonomous Agent Primitives & Reasoning Loops When the task path is genuinely unpredictable—because step $N+1$ depends on discovering what happens in step $N$ (e.g., diagnosing a production outage, resolving an unfamiliar GitHub issue across a codebase, or navigating dynamic web workflows)—an **Autonomous Agent** takes control of the loop. ### Reasoning Loop Paradigms: ReAct vs. Plan-and-Execute vs. Reflection ``` ReAct (Interleaved): [Think] ──► [Act: Tool Call] ──► [Observe: Ground Truth] ──┐ ▲ │ └─────────────────── (Repeat until done / cap) ─────────┘ Plan-and-Execute / ReWOO: [Planner LLM: Emit DAG / Symbolic Plan] ──► [Workers: Execute Steps/Tools] ──► [Solver / Replanner] ``` | Reasoning Paradigm | Execution Mechanics | Strengths | Failure Mode & When to Avoid | | :--- | :--- | :--- | :--- | | **ReAct (Reason + Act)** | Interleaves **Thought** $\rightarrow$ **Action** (tool call) $\rightarrow$ **Observation** (tool result) turn-by-turn. | Maximum adaptability; each action is grounded in the immediate observation from the previous step. | High sequential latency ($O(N)$ LLM turns); context grows every turn; vulnerable to local rabbit holes without a high-level checklist. | | **Plan-and-Execute (incl. ReWOO & LLMCompiler)** | **Planner** generates an upfront multi-step plan (or DAG of symbolic variables `#E1`, `$1`); **Executor / Dispatcher** runs steps (often in parallel); **Replanner / Solver** synthesizes or adjusts on failure. | Slashes latency and token re-ingestion (ReWOO uses 2 LLM calls + parallel I/O; LLMCompiler eagerly dispatches DAG nodes as dependencies resolve). | Brittle if early tool observations invalidate the entire upfront plan; requires dynamic replanning hook when the environment shifts. | | **Reflection / Self-Correction** | Agent critiques its own draft or observes a deterministic failure (compiler error, test failure, schema error) and revises before emitting the final answer. | Catches syntax, schema, and logic defects automatically when grounded in external test/validator output. | **Ungrounded reflection drift:** Reflecting purely on internal prose without external ground truth (tests, linters, search) often degrades accuracy or reinforces hallucinations. | ### Stopping Conditions & Loop Hygiene An autonomous loop without strict termination controls is a production outage waiting to happen. Enforce **five mandatory loop guardrails**: 1. **Explicit completion criteria:** Define objective ground-truth exit conditions (all unit tests pass, target schema validates, or explicit `finish_task` / `end_turn` reached). 2. **Hard turn & call ceilings (`max_turns` / `max_llm_calls`):** Cap total model iterations per run so a stuck agent exits cleanly (`error_max_turns`) rather than looping infinitely. 3. **Token & cost budget ceilings (`max_budget_usd`):** Track cumulative input/output token spend in session state and halt or escalate when the run approaches its budget ceiling. 4. **Duplicate tool-call detection:** Hash `(tool_name, canonical_args)` across recent turns. If the agent invokes the exact same failing or no-op tool call 2–3 times in a row, intercept the call in the harness and inject a loop-breaker error or terminate. 5. **Structured error feedback (`is_error: true`) & ACI Poka-Yoke:** - **Never swallow tool exceptions as empty strings `""`** (which tricks the model into thinking no records exist) and never let transient tool errors crash the whole process. Return a structured error result (`is_error: true` / `{"status": "error", "message": "..."}`) describing *what* failed and *how* to fix it so the model can self-correct or escalate. - **Craft the Agent-Computer Interface (ACI) with Poka-Yoke (mistake-proofing):** Write tool descriptions like docstrings for a junior engineer (what it does, when to use it, when *not* to use it), constrain inputs with enums/`Literal` and strict JSON schemas, and require unambiguous formats (e.g., **absolute file paths** `/repo/src/app.py` instead of relative paths that break after working-directory changes). | Loop Guardrail Dimension | Detection Mechanism | Runtime Action on Trigger | | :--- | :--- | :--- | | **Turn Budget** | Counter tracking total completed model roundtrips. | Halts execution when `turn > max_turns`; returns `error_max_turns`. | | **Financial Cost Ceiling** | Cumulative token spend tracker (`spent_usd >= max_usd`). | Immediately terminates session; emits `budget_ceiling_exceeded`. | | **Duplicate Call Trap** | Hash map tracking `(tool_name, hash(canonical_args))` across recent turns. | Intercepts 3rd duplicate invocation; returns `is_error: true` loop-breaker feedback. | | **Structured Tool Errors** | Exception handler wrapping tool execution harness. | Emits `{"type": "tool_result", "is_error": true, "content": "..."}` with corrective advice. | | **Completion Signal** | Inspects `stop_reason == "end_turn"` and output schema. | Returns completed payload with final answer and turn count. | ### ACI Design & Poka-Yoke Tool Engineering | Tool Design Dimension | Fragile Prototype Tool | Production Poka-Yoke Tool | | :--- | :--- | :--- | | **Path Arguments** | `path: str` (accepts relative `../config.yaml`; breaks after `cd`) | `absolute_path: str` (requires leading `/`; validated via `Path.resolve().is_relative_to(ROOT)`) | | **Constrained Choices** | `status: str` (free text; model invents `"in_progress_now"`) | `status: Literal["open", "pending", "resolved"]` (enforced via `strict: true` JSON Schema) | | **Boundary Clarity** | `search(query)` (ambiguous across 4 similar search tools) | `orders_search_by_customer_id(customer_id)` — docstring states *what* it queries, *limits*, and *when to use `orders_get_by_id` instead* | | **Output Payload Size** | Dumps all 10,000 matching rows (`250 KB` JSON) into context | Paginates (`limit: int = 25, offset: int = 0`) and returns total count + compact summary fields | | **Error Semantics** | Raises unhandled `KeyError` or returns `""` on timeout | Returns `is_error: true` with error category, retryability, and corrective hint | --- ## 05. Memory Architecture & Context Budget Engineering A 200K–1M token context window is **not** an excuse to dump every file, tool schema, and historical transcript into a single prompt. As context fills, latency and cost scale linearly while instruction adherence and mid-window recall degrade (**Context Rot**). ### The 4-Tier Agent Memory Model (CoALA Taxonomy) | Memory Tier | Cognitive Role | Storage & Lifetime | Production Engineering Rules | | :--- | :--- | :--- | :--- | | **1. Working Memory** | Active scratchpad for the current task (system instructions, recent turns, live tool observations, `temp:` state) | In-window `messages[]` + ephemeral invocation state (`temp:*`); cleared after turn/session | Keep lean; place byte-identical static instructions and tool schemas **first** to maximize prompt-cache hits; externalize live checklists (`TodoWrite` / state keys) so goals survive long runs. | | **2. Episodic Memory** | Record of specific past sessions, interaction trajectories, and checkpoints (`session.events`) | Persistent session store (`DatabaseSessionService`, JSONL transcripts, checkpointer) or consolidated session summaries | Used for session resume (`--resume`), audit trails, trajectory evaluation, and few-shot recall of how similar tasks were solved previously. | | **3. Semantic Memory** | Verified domain knowledge, entity facts, and consolidated user preferences (`user:*`) | Vector/hybrid RAG index or managed Memory Bank (`VertexAiMemoryBankService`) across sessions | **Consolidate, don't just append:** Use background LLM consolidation to update superseded user facts rather than retrieving contradictory raw chat logs. Keep enterprise policy stores **read-only** during inference. | | **4. Procedural Memory** | How the agent operates: system prompt heuristics, tool definitions, and reusable playbooks (**Skills**) | Code repository (`CLAUDE.md`, `.claude/rules/`, `SKILL.md` bundles, MCP prompt templates) | Use **3-level progressive disclosure** for skills: load only compact YAML `name` + `description` (~60–100 tokens) at startup, load the full `SKILL.md` body only when triggered, and execute bundled `scripts/` outside context so only stdout enters the window. | ### Context Rot Taxonomy & The 4 Context Engineering Strategies | Context Rot Failure Mode | Root Mechanism | Correct Remediation | | :--- | :--- | :--- | | **Context Poisoning** | An early hallucination, misread file path, or rejected refusal turn enters `messages[]` and gets repeatedly trusted or reinforced on subsequent turns. | **Do NOT run summarization/compaction** (which bakes the poisoned premise into the summary!). **Rewind / truncate** history to the checkpoint prior to the bad turn, or **clear/reset** into a fresh context window with only verified facts. | | **Context Confusion** | Raw 50 KB JSON tool dumps, full file reads, and 60 unused MCP tool schemas crowd the attention window, causing the model to lose track of core constraints. | **Prune & Isolate:** Truncate/paginate tool outputs (`offset`/`limit`), strip bulky historical tool payloads after extraction, defer unused tool schemas behind on-demand search (`ToolSearch` / `tool_filter`), and delegate noisy searches to isolated subagents. | | **Context Clash** | Contradictory rules accumulate across global instructions, subdirectory rules, and multi-turn user corrections. | Audit instruction hierarchy, scope path-specific rules (`paths:` frontmatter) so they load only when relevant files are touched, and resolve conflicts explicitly in a structured spec before execution. | **The 4 Context Strategies (`Write`, `Select`, `Compress`, `Isolate`) & Compaction Mechanics:** 1. **Write (Externalize State):** Persist plans, intermediate artifacts, and extracted schemas outside the fragile message history (into typed session state keys, scratchpad files on disk, or `ArtifactService`). 2. **Select (Just-in-Time Retrieval & Progressive Disclosure):** Pull in only the exact file ranges (`Grep` $\rightarrow$ paginated `Read`), scoped rules, and active skill bodies needed for the current step. 3. **Compress (Sliding Window, Compaction & Tool-Output Pruning):** - **Tool-output pruning:** Replace multi-kilobyte raw tool outputs from 10 turns ago with a 2-line digest once the relevant fields have been extracted into structured state. - **Summarization checkpoints (Compaction):** When a healthy long-running session approaches a token threshold (e.g., 70–80% of the context budget), trigger a compaction pass that summarizes older turns into a structured checkpoint block (preserving architectural decisions, modified file paths, verified facts, and open checklist items) while keeping the most recent $K$ turns verbatim (**sliding window overlap**). - **Prompt-cache prefix discipline:** Never inject per-request dynamic timestamps, random UUIDs, or changing user metadata at the top of the system prompt—keep the prefix byte-identical so cached system instructions, tool schemas, and prior turns hit cache at ~75–90% lower input cost. 4. **Isolate (Subagent Sandboxing):** Run exploratory search, multi-file analysis, or independent verification inside a separate subagent with its own clean `messages[]` array, returning only a compact structured summary to the coordinator. --- ## 06. Multi-Agent Topologies & Handoff Contracts Multi-agent systems partition complex work across specialized agents—not to engage in anthropomorphic role-play, but to **enforce context isolation, separate conflicting toolsets, parallelize independent I/O, and separate generation from verification**. ### The "Agent Theater" Litmus Test Before splitting a system into multiple agents, apply the **Multi-Agent Justification Rule**: $$\text{Multi-Agent is justified} \iff \text{Subagents have distinct tools/permissions, isolated noisy context, parallel I/O, or adversarial verification.}$$ If three "persona" agents share the exact same tools, read the same short prompt, and run sequentially without independent verification or context isolation, you have built **Agent Theater**—multiplying token spend and latency while introducing cross-agent drift for something a single structured prompt handles better. ### Four Canonical Multi-Agent Topologies ``` 1. Hub-and-Spoke (Coordinator) 2. Sequential Pipeline 3. Mesh / Swarm (Peer Handoff) 4. Hierarchical Teams ┌──► Subagent A [Agent 1] ──► [Gate] ──► [Agent A] ◄───► [Agent B] [Root Coordinator] [Coord] ┼──► Subagent B [Agent 2] ──► [Agent 3] ▲ ▲ ┌───────┴───────┐ └──► Subagent C └───► [Agent C] ┘ [Team Lead 1] [Team Lead 2] ├─► W1 ├─► W2 ├─► W3 ├─► W4 ``` | Topology | Control & Routing Mechanics | Strengths | Key Risks & Required Safeguards | | :--- | :--- | :--- | :--- | | **1. Hub-and-Spoke (Coordinator / Subagents)** | Central coordinator decomposes the request, invokes specialist subagents as tools (`AgentTool` / `Task`), collects compact results, and synthesizes the final response. | Central visibility, strict context isolation per spoke, easy to enforce hub-level guardrails and wave limits (3–5 concurrent workers). | Coordinator becomes a bottleneck if spokes return raw 50 KB dumps instead of structured summaries. Block subagents from spawning recursive subagents unless explicitly architected. | | **2. Sequential Pipeline** | Fixed relay (`Agent 1` $\rightarrow$ `Gate` $\rightarrow$ `Agent 2` $\rightarrow$ `Agent 3`) passing state via explicit keys (`output_key` + `output_schema`). | Zero routing overhead; deterministic transitions; ideal for Staged Extract $\rightarrow$ Redact $\rightarrow$ Analyze $\rightarrow$ Format pipelines. | Rigid; if step 1 misclassifies an input and no gate catches it, steps 2 and 3 compound the error. | | **3. Mesh / Debate / Swarm** | Peer specialists transfer control directly to one another (`transfer_to_agent`) or critique each other's proposals without a central hub. | Useful for adversarial red-teaming, multi-perspective design critique, or seamless specialist conversation handoffs. | Highest risk of infinite ping-pong loops, state corruption, and diffusion of responsibility. Requires strict `max_turns`, explicit exit tools, and a deterministic tie-breaker/judge. | | **4. Hierarchical Teams** | Multi-tier tree where a root orchestrator delegates to domain leads, which in turn manage narrow execution workers. | Scales to massive enterprise workflows with dozens of tools by keeping each node's tool catalog $\le 16$. | High cumulative latency and token cost; trace debugging requires correlation IDs across every parent-child boundary. | ### Vertical (MCP) vs. Horizontal (Subagents / A2A) Integration | Integration Axis | Protocol / Mechanism | Architectural Boundary | Trust & Context Contract | | :--- | :--- | :--- | :--- | | **Vertical (`Agent → Tool/Data`)** | **MCP (Model Context Protocol)** or native `FunctionTool` | Connects an agent to deterministic APIs, databases, filesystems, and prompt/resource templates. | Agent sees tool schema and passes typed arguments; scrub PII and untrusted content at the tool boundary (`after_tool_callback`). | | **Horizontal In-Process (`Parent → Subagent`)** | **`AgentTool` / `Task` / `sub_agents`** | Delegates a cognitive subtask to a specialist agent running inside the same application runtime. | Subagent runs in an isolated context window with scoped tools and returns a typed `HandoffContract` (plus optional state/artifact deltas). | | **Horizontal Cross-Boundary (`Agent → Remote Agent`)** | **A2A (Agent2Agent Protocol)** | Delegates long-running, stateful tasks across team, cloud, or organizational boundaries via an `AgentCard`. | Remote agent is **opaque** (never shares internal prompts, memory, or tools); caller authenticates via card `securitySchemes` and validates all returned artifacts as untrusted input. | ### The Context Isolation Rule & Handoff Contracts The single most important rule in multi-agent engineering is **Context Isolation**: - **Fresh, task-scoped context by default:** When a coordinator spawns a subagent to research a codebase, audit a diff, or query a database, the subagent must start with a **fresh context window** containing *only* its specialist system instructions and the explicit task payload—**not** the coordinator's entire 80-turn noisy conversation history. (Reserve full-history context forking strictly for deliberate cache-sharing continuations where the child needs the exact prior transcript.) - **No shared mutable scratchpads across parallel workers:** When running parallel subagents in waves (e.g., 3–5 concurrent workers), assign each worker a distinct `output_key` or isolated file path / git worktree so concurrent writes never race or clobber one another. - **Explicit `HandoffContract` schema:** Because a subagent's intermediate tool calls stay inside its private window, **only its final response crosses back to the parent**. Enforce a typed `HandoffContract` (via Pydantic `output_schema` / structured JSON) across every agent boundary: ```python class HandoffContract(BaseModel): task_id: str status: Literal["completed", "partial", "blocked", "escalation_required"] confidence_band: Literal["high", "medium", "low"] summary: str = Field(max_length=1200, description="Compact synthesis") artifact_uri: str | None = Field(default=None, description="Artifact path") verified_citations: list[dict] = Field(default_factory=list) ``` > [!TIP] > **Artifact pointer pattern:** If a worker generates a 2,000-line code diff or a 30-page extracted table, do **not** stuff the raw payload into the coordinator's message history. Have the worker write the full payload to disk or an `ArtifactService` (`artifact_uri`) and return only the compact `HandoffContract` summary and URI to the coordinator. --- ## 07. Anti-Patterns, Failure Modes & Architectural Checklist ### Production Trust Architecture: Why Single-Prompt "Happy-Path" Prototypes Fail While a single multimodal LLM call with a JSON schema can easily summarize a clean 1-page demo PDF in 2 seconds, unguarded single-prompt pipelines fail at enterprise scale for five structural reasons: 1. **Token burn & hallucinations on corrupted inputs (Missing Epistemic Gate):** Sending a 120-page unreadable scan or wrong document type directly to a frontier model wastes time and tokens, and the model often guesses to be helpful. *Fix:* Enforce a **$0 deterministic Pre-LLM Epistemic Gate** checking file metadata and a **composite extraction quality score** (OCR token confidence, lexical dictionary ratio, character entropy, structural page completeness); if the score falls below threshold, halt and escalate without calling the LLM. 2. **Instruction crowding & self-grading bias:** Asking one prompt to extract 30 fields, format Markdown, and audit every citation causes the model to prioritize fluent generation and silently skip strict verification rules.Even strong prompts remain probabilistic, whereas a deterministic Python post-check paired with an independent cold-context critic enforces hard invariants reliably. 3. **Unverified citations:** Checking only that the output matches the regex `(Source: Section X)` misses fabricated references (`Section 99.1` on a 4-page contract). *Fix:* **Grounded citation verification** checks deterministically in code that every cited section header or quote exists in the extracted source text. 4. **Cloud PII exposure:** Sending raw binary documents directly to external model APIs leaks unredacted PII before any prompt rule runs. *Fix:* Parse locally (`Docling` / local layout parser) and run deterministic PII scrubbing at the pre-LLM and `after_tool_callback` boundaries. 5. **Rule drift across repositories:** Copy-pasting compliance prompts across 10 microservices guarantees policy drift. *Fix:* Package shared domain validators, prompts, and policy thresholds into a central **MCP server** (*Rules as Infrastructure*). ### Symptom $\rightarrow$ Root Cause $\rightarrow$ Architectural Fix | Production Symptom | Root Architectural Cause | Deterministic Architectural Fix | | :--- | :--- | :--- | | **Agent miscalculates fees, discounts, or date windows on edge cases** | **Deterministic drift:** Delegating Bucket 1 arithmetic or policy thresholds to probabilistic prose instructions. | Move calculations and threshold checks into deterministic Python/SQL tools or `PreToolUse` / `before_tool_callback` hooks. | | **Simple 3-step pipeline takes 45 seconds and costs $0.80 per run** | **Over-agentification:** Using an open-ended autonomous ReAct agent (or Agent Theater) where the task path is known in advance. | Replace the autonomous loop with a **Prompt Chain** or **Parallel Sectioning Workflow** using fast-tier models for routine steps. | | **Coordinator hits `model_context_window_exceeded` after calling 4 research workers** | **Missing context isolation:** Workers return raw 40 KB tool dumps into the parent window, or inherit full parent history. | Enforce the **Context Isolation Rule**: spawn workers in fresh windows, write bulky artifacts to disk/storage, and return a bounded `HandoffContract` summary. | | **Parallel workers intermittently lose data or overwrite each other's findings** | **Shared mutable state race:** Concurrent branches write to the same session state key (`state["result"]`) or edit the same file simultaneously. | Give each parallel branch a unique `output_key` (`result_a`, `result_b`) merged by a downstream `JoinNode`/synthesizer, or isolate file edits in separate worktrees. | | **Generator–Critic loop burns 15 turns and still ships a hallucinated citation** | **Unbounded loop + ungrounded reflection:** No `max_iterations` cap, and the critic grades prose by "vibes" without checking source text in code. | Cap the loop at `max_iterations=3`, verify citations deterministically against source text (regex/substring match), and escalate to HITL if the cap trips without approval. | | **Agent tells the user "You have no active orders" during a database timeout** | **Swallowed tool error:** Tool wrapper caught the timeout exception and returned `""` or `{}` with HTTP 200 success semantics. | Return a structured error block (`is_error: true` or `{"status": "error", "retryable": true, "message": "DB timeout"}`) so the model knows the lookup failed. | | **After compacting a confused session, the agent keeps repeating the same wrong file path** | **Compacting poisoned context:** Summarization compressed the earlier hallucinated file path into the compacted summary block. | Use **rewind / hard context reset (`/clear`)** to discard poisoned turns entirely instead of summarizing them. | | **Agent selects the wrong tool when given 45 enterprise MCP endpoints** | **Tool catalog saturation:** Model attention degrades when forced to discriminate among $>16$ overlapping tool schemas in one prompt. | Group tools by domain across specialist subagents ($\le 16$ tools each) or enable on-demand **progressive tool disclosure / Tool Search** plus `tool_filter` allowlists. | | **Summarizer agent reading an external email is tricked into calling `send_email`** | **Indirect prompt injection + excessive agency:** Untrusted data entered the same context that held high-privilege mutation tools. | Enforce **least privilege** (strip `send_email` from the reader agent), screen tool outputs in `after_tool_callback`, and gate external sends with a **HITL checkpoint**. | ### 8-Point Pre-Production Architectural Checklist - [ ] **1. 3-Bucket Triage Verified:** Are all math, schema validation, RBAC, and hard compliance rules executed in deterministic code (Bucket 1), and all high-blast-radius irreversible actions gated by HITL (Bucket 3)? - [ ] **2. Complexity Ladder Justified:** Have you proven with an evaluation baseline that a single Augmented LLM call or deterministic Workflow cannot meet the bar before deploying an Autonomous Agent? - [ ] **3. Pre-LLM Epistemic Gating Active:** Does deterministic code validate file types, input completeness, and OCR/extraction quality *before* spending model tokens, with `escalation_required = True` as the safe default? - [ ] **4. Loop Hygiene Enforced:** Does every agent and evaluator-optimizer loop enforce `max_turns` / `max_iterations`, token budget ceilings, duplicate tool-call detection, and explicit escalation when the cap trips? - [ ] **5. ACI & Poka-Yoke Hardened:** Are tool catalogs $\le 16$ per agent, arguments constrained via strict schemas/enums/absolute paths, and failures returned as structured `is_error: true` signals? - [ ] **6. Context Budget & Cache Optimized:** Are static instructions and tool definitions placed before dynamic per-request data for cache stability, and is noisy exploration isolated in subagents? - [ ] **7. Handoff Contracts Typed:** Do subagents run in fresh, task-scoped context windows and return compact, schema-validated `HandoffContract` objects (with URIs for large artifacts)? - [ ] **8. Separation of Authority & Trajectory Evals:** Are high-stakes outputs verified by an independent critic + deterministic code checks, and does CI evaluate both final answers and tool trajectories? --- ## 08. Glossary (`#glossary`) - **ACI (Agent-Computer Interface):** The design, naming, schema constraints, error semantics, and documentation of tools exposed to a model—the agentic equivalent of HCI/UX. - **Agent Theater:** Anti-pattern where multiple persona agents with identical tools and no context isolation or independent verification are chained together, inflating cost and latency for zero architectural benefit. - **Architectural Evaluation Criteria:** Decision rubric comparing Single Call, Composable Workflow, and Autonomous Agent across path predictability, latency budget, error blast radius, token cost ceiling (~1× vs. 3–5× vs. ~15×), and auditability. - **Augmented LLM:** The foundational building block of agentic systems: a single LLM equipped with retrieval, tools (often via MCP), and memory. - **Compaction:** Context compression technique that summarizes older conversation turns into a compact block so long sessions stay within token budgets; effective for *context confusion*, but unsafe for *context poisoning*. - **Context Isolation Rule:** Principle that subagents execute in fresh, task-scoped context windows rather than inheriting the coordinator's full noisy history, returning only a structured summary (`HandoffContract`). - **Context Rot (Poisoning / Confusion / Clash):** Degradation of model accuracy as context grows: **poisoning** (hallucinated premise persists in history), **confusion** (bulky tool outputs/schemas dilute attention), and **clash** (contradictory instructions). - **Deterministic Invariant Degradation (Deterministic Drift):** Reliability failure caused by delegating exact arithmetic, schema validation, or policy enforcement to probabilistic prompt instructions instead of deterministic code. - **Epistemic Gating:** Deterministic pre-LLM guardrail that scores input/extraction integrity (OCR confidence, lexical ratio, structural completeness) and blocks or escalates corrupted inputs before spending model tokens. - **Evaluator–Optimizer:** Workflow pattern pairing a generator LLM with an independent critic/auditor LLM (plus deterministic checks) in a rubric-driven loop bounded by `max_iterations`. - **Gate (`gate_check`):** Deterministic plain-code check between workflow steps that validates intermediate schemas, invariants, or citations before allowing execution to proceed. - **HandoffContract:** Strict structured schema (status, confidence band, compact summary, verified citations, and `artifact_uri` pointer) returned by a subagent to its coordinator. - **HITL (Human-in-the-Loop):** Control pattern that pauses execution and persists state before an irreversible or high-stakes action commits, presenting concrete tool arguments for human approval, modification, or rejection. - **LLMCompiler / ReWOO:** Plan-and-execute architectures that decouple upfront planning (emitting symbolic variable plans or task DAGs) from parallel tool execution to eliminate sequential ReAct turn latency. - **Orchestrator–Workers:** Workflow pattern where a central LLM dynamically decomposes an unpredictable task at runtime, dispatches scoped workers, and synthesizes their results. - **Parallelization (Sectioning vs. Voting):** Running concurrent LLM calls either on independent sub-parts of a task for speed/focus (*Sectioning*) or repeating the same task for consensus/high-recall verification (*Voting*). - **Poka-Yoke:** Mistake-proofing tool definitions (e.g., requiring absolute file paths, strict enums, and self-contained schemas) so invalid model calls become structurally impossible. - **Prompt Chaining:** Sequential workflow decomposing a predictable task into fixed LLM steps separated by deterministic code gates. - **ReAct (Reason + Act):** Core autonomous agent loop that interleaves reasoning (*Thought*), tool execution (*Action*), and environmental ground truth (*Observation*) until a stopping condition is met. - **Routing:** Workflow pattern that classifies an input's intent or complexity and dispatches it to a specialized prompt, toolset, or cost-appropriate model tier. - **Separation of Authority:** Decoupling generation from verification by running an independent, cold-context critic/auditor alongside deterministic code checks to eliminate self-grading confirmation bias. - **Sequential Decision Framework:** Ordered architectural checklist run before prompt writing: (1) Problem decomposition & execution engine triage $\rightarrow$ (2) Work-shape selection $\rightarrow$ (3) Routing & hosting $\rightarrow$ (4) Guardrails & verification. - **Tripartite Execution Regimes:** Triage framework dividing system logic into **Regime 1 (Deterministic Code)** for math/validation/RBAC, **Regime 2 (Probabilistic LLM)** for unstructured synthesis/classification/planning, and **Regime 3 (Human-in-the-Loop)** for irreversible, regulated, or accountability-bearing decisions. ================================================================================ GUIDE: Context Engineering & Reliability URL: https://mbettan.github.io/machine-learning-cert/ContextEngineering.html Practice quiz: https://mbettan.github.io/machine-learning-cert/quiz.html?topic=context-engineering Section: Foundation · Agent Architecture & Protocols Exam relevance: Google Professional Agentic Architect (supporting), Claude Certified Architect – Foundations (core) ================================================================================ # Context engineering & reliability study guide > **Author:** Michaël Bettan --- ## Overview ### Definition Context engineering is the per-call discipline of choosing which tokens enter a model's finite context window (instructions, tool definitions, working state, retrieved material, history and tool results), in what form and in what order, and what is moved outside the window so a long-running agent stays coherent after hundreds of turns. ### What context engineering solves for A model is stateless: at each step it sees only what the harness placed in the window for that call. Agents running in a loop keep producing tokens (tool payloads, reasoning, drafts) and most of them stop mattering a few turns later. Left alone, the window fills, recall of the details that still matter drops, and the agent either hits the hard limit or slowly drifts away from its instructions. Context engineering keeps the window small, current and high-signal for the whole run. > **Mental model:** treat the window like a kitchen counter during a long dinner service. Only the ingredients for the dish being plated stay on the counter; finished pans go to the sink (clearing), prep notes go on the whiteboard (scratchpad), the pantry is opened only when a recipe calls for something (just-in-time retrieval), and a second cook works a separate station (subagent). ``` ┌──────────────────────── one model call (finite window) ────────────────────────┐ │ 1. Stable prefix instructions · tool schemas · examples cacheable │ │ 2. Working state task values read by key · scratchpad excerpt exact │ │ 3. Summary prefix compacted digest of older turns rolling │ │ 4. Recent window last N turns · newest tool results verbatim │ │ 5. Current input user message or task brief last │ └───────────────────────────────────────┬────────────────────────────────────────┘ ▼ offload what can wait: notes files · state · memory · fetchable handles ``` * **Scope of this guide:** budget and placement, long-session techniques (sliding windows, summary prefixes, compaction), tool-result clearing, scratchpads and structured note-taking, just-in-time retrieval, sessions and memory, subagent context isolation, strategy selection, caching and security. * **Vendors covered:** Claude (Messages API context management, memory tool, Agent SDK, Claude Code) and Google (ADK compaction and state, Agent Platform Sessions and Memory Bank, Gemini API caching). Mechanisms differ; the design decisions are shared. * **Related guides:** [Claude Agent SDK](https://mbettan.github.io/machine-learning-cert/ClaudeAgentSDK.html#sessions) for session operations, [ADK](https://mbettan.github.io/machine-learning-cert/ADK.html#state-memory) for state prefixes and memory services, [Agentic design patterns](https://mbettan.github.io/machine-learning-cert/ADP.html#memory-context) for the memory taxonomy, [Agent Platform](https://mbettan.github.io/machine-learning-cert/AgentPlatform.html#sessions-memory) for managed Sessions and Memory Bank, [Claude Code](https://mbettan.github.io/machine-learning-cert/ClaudeCode.html#claude-md) for `CLAUDE.md` and auto-compact. * **Official documentation:** Claude [context windows](https://platform.claude.com/docs/en/build-with-claude/context-windows), [context editing](https://platform.claude.com/docs/en/build-with-claude/context-editing), [compaction](https://platform.claude.com/docs/en/build-with-claude/compaction-on-demand), [memory tool](https://platform.claude.com/docs/en/agents-and-tools/tool-use/memory-tool) and [prompt caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching); ADK [state](https://adk.dev/sessions/state/), [memory](https://adk.dev/sessions/memory/), [compaction](https://adk.dev/context/compaction/) and [caching](https://adk.dev/context/caching/); [Memory Bank](https://docs.cloud.google.com/gemini-enterprise-agent-platform/scale/memory-bank); Gemini API [caching](https://ai.google.dev/gemini-api/docs/caching). * **Further reading:** the attention-budget view, just-in-time retrieval, structured note-taking and the compaction, notes and subagent trio are described in Anthropic's engineering post [Effective context engineering for AI agents](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents). The position effect is from Liu et al., [Lost in the middle: how language models use long contexts](https://arxiv.org/abs/2307.03172) (TACL 2024). --- ## Prompt vs. Context vs. Memory | Layer/approach | Type | Role & behavior | | :--- | :--- | :--- | | **Prompt engineering** | Authoring | Wording, structure and examples of one instruction; written once, reused on every call | | **Context engineering** | Per-turn curation | Selects, orders, compresses, offloads and isolates everything in the window on every call | | **State** | Exact working data | Key/value values the harness reads and writes deterministically (IDs, flags, step outputs) | | **Scratchpad** | Agent-written notes | Files the agent writes and re-reads during a task (plan, progress, open questions); outlives a context reset, usually not the task | | **Memory** | Long-term recall | Facts extracted across sessions and fetched by search or a memory tool; recall is probabilistic and can be stale | | **Retrieval (RAG)** | Shared knowledge | Documents valid for every user (policies, manuals), fetched up front or on demand | ### Context levers * **Trim** drops the oldest turns once the window passes a size (a sliding window over history). * **Summarize** replaces a span of older turns with a digest that sits in front of the recent turns (summary prefix, compaction). * **Clear** removes stale tool results or thinking blocks while leaving the rest of the conversation intact (context editing). * **Offload** writes information to a file, state or memory and keeps only a handle (path, key, query) in the window. * **Fetch late** keeps references instead of payloads and loads data with a tool when a step needs it (just-in-time retrieval). * **Isolate** gives a noisy subtask its own window (subagent or agent-as-tool) and brings back a short digest. * **Mapping to the common framing:** offload corresponds to *write*, fetch late to *select*, trim, summarize and clear to *compress*, and isolate to *isolate* (the four strategies in the [Agentic design patterns guide](https://mbettan.github.io/machine-learning-cert/ADP.html#memory-context)). > **Key rule:** every token draws on the same attention budget. A bigger window raises the ceiling, but irrelevant tokens still lower recall of the relevant ones, so the goal is the smallest set of high-signal tokens for the current step. ### Per-turn assembly workflow 1. **Load:** read session events, the working state for this task and, if the task uses one, the current scratchpad. 2. **Reduce:** apply the history policy (sliding window, summary prefix or compaction) and clear tool results that can be re-fetched. 3. **Fetch:** pull in only the documents or memories the next step needs, or expose search tools and let the agent decide. 4. **Place:** stable prefix first, then working state and the summary prefix, then recent turns, then the current input. 5. **Call:** count tokens when close to the limit and reserve room for the output. 6. **Offload:** after the call, write new decisions to state or the scratchpad and replace bulky results with handles. ### When not to engineer context * **Short, single-turn tasks:** one well-written prompt with the needed document is enough; compaction, notes and memory add cost with no benefit. * **Everything fits and stays relevant:** a 20-turn chat with small tool results needs no trimming; measure before adding machinery. * **Shared reference knowledge:** put it in retrieval or the stable prefix, not in per-user memory. --- ## Context budget and placement ### Budget arithmetic * **The window holds input and output:** system prompt, tool definitions, history, tool results and the reply all count; reserve output headroom (`max_tokens` on Claude, `max_output_tokens` on Gemini) before filling the rest. * **Window sizes differ by model:** current Claude models default to a 1M-token window with up to 128k output tokens, while legacy models keep 200k; check the model page rather than assuming. * **Count, don't guess:** Claude exposes a token counting endpoint (`POST /v1/messages/count_tokens`, returning `input_tokens`) and Gemini exposes `count_tokens`; character heuristics (about four characters per token in English) are only rough planning numbers. * **Recount when you switch models:** tokenizers change between model generations (Anthropic documents roughly 30% more tokens for the same text with the tokenizer introduced in Opus 4.7), so a prompt that fit before can overflow or cost more after an upgrade. * **Let the agent see its budget:** recent Claude models receive a context-awareness note after tool calls with tokens used and remaining, which helps them wrap up or write notes before the limit. * **Tool definitions are a fixed tax:** every tool schema is sent on every call. A bloated tool set also creates ambiguous choices; if an engineer cannot say which tool fits a situation, the model cannot either. Keep a minimal set per agent. * **Context rot:** recall of a specific detail degrades gradually as total tokens grow, well before the hard limit. Treat the usable window as smaller than the advertised one. ### Placement * **Position effect:** models use information at the start and end of a long input more reliably than material buried in the middle (Liu et al.). Keep the instruction that governs the step near an edge. * **Long documents first, question last:** for large inputs, place documents above the instructions and the question at the end; Anthropic's long-context guidance reports better answers with this order. * **Label what you place:** wrap documents and notes in named sections or XML-style tags (``, ``) so the model can refer to them precisely. * **Prompt altitude:** write instructions specific enough to steer behavior but general enough to act as heuristics. Hard-coded if-else rules are brittle; vague guidance assumes context the model does not have. * **Examples over rule lists:** a few diverse, canonical examples steer more reliably per token than a long list of edge-case rules. > [!NOTE] > *Directional figures:* the "about four characters per token" heuristic and vendor-reported quality gains from document placement are rules of thumb. Measure on your own prompts with the token counting APIs and an evaluation set. --- ## Long sessions: sliding windows, summary prefixes and compaction ### Symptoms of an overloaded session * **Generic answers:** the agent talks about "the usual approach" instead of the specific file, ticket or decision it found earlier. * **Instruction drift:** a rule stated early stops being applied after many large tool calls. * **Repeated work:** the agent re-runs searches whose results it already has, because those results are buried or were dropped without a trace. ### History policies | Policy | How it works | Keeps | Loses | Fits | | :--- | :--- | :--- | :--- | :--- | | **Sliding window (trimming)** | Send only the last N turns or last K tokens; drop older turns | Recent detail, verbatim | Everything older, abruptly | Chat where old turns rarely matter | | **Summary prefix** | A rolling digest of older turns is placed before the recent window and refreshed as turns age out | Gist of older turns plus recent detail | Exact wording of summarized turns | Assistants with long back-and-forth | | **Compaction (reset with summary)** | Near a threshold, summarize the whole conversation and continue in a fresh window seeded with the summary | Decisions, open issues, key artifacts | Redundant tool output and chatter | Long agentic runs with tool use | | **Overlapping compaction** | Each new summary covers a span that overlaps the previous one | Continuity across summary boundaries | Some extra tokens | Event-based compaction in ADK | * **Sliding windows are cheap but blunt:** they need no extra model call, yet a constraint stated in turn 3 disappears at turn 3 + N. Anything that must survive belongs in the system prompt, state or a notes file, never only in early turns. * **Summary prefixes trade fidelity for continuity:** the digest is generated by a model, so it paraphrases. Keep values that must stay exact (amounts, IDs, deadlines) in state or a scratchpad that the summarizer never rewrites. * **Compaction is the usual first lever for long agent runs:** Anthropic describes Claude Code compaction as summarizing the history while keeping architectural decisions, unresolved bugs and implementation details, discarding redundant tool output, and continuing with the summary plus the five most recently accessed files. ### Tuning a compaction prompt 1. **Collect real traces:** pick long, messy runs where a later step depended on an early detail. 2. **Maximize recall first:** write the summary instructions so nothing that a later step used is dropped. 3. **Then raise precision:** remove categories of content the agent never needed again (raw tool payloads, superseded drafts). 4. **Re-test on new traces:** overly aggressive summaries lose details whose importance appears only later. ### Vendor mechanisms | Mechanism | Identifier and key parameters | Behavior | | :--- | :--- | :--- | | **Claude on-demand compaction** | Top-level `compaction: {"type": "summarize", "instructions": ...}` (beta `compact-2026-09-04`) | Your app decides when; Claude returns a signed `compaction` block that you send first in later `messages`, in place of the summarized turns | | **Claude threshold compaction** | Edit `compact_20260112` (beta `compact-2026-01-12`); `trigger` defaults to 150,000 input tokens; optional `instructions`; `pause_after_compaction` | Server summarizes when input crosses the trigger; pausing returns `stop_reason: "compaction"` so you can inspect the summary or re-insert notes | | **Claude Code** | Auto-compact (`autoCompactWindow` setting), `/compact [focus]`, `/context`, `/clear` | Project-root `CLAUDE.md` is re-read from disk after compaction, so durable rules belong there | | **Agent SDK** | `compact_boundary` system message with `compact_metadata` (`pre_tokens`, `trigger`) | Lets the harness log compactions and re-inject state afterwards | | **ADK event compaction** | `EventsCompactionConfig(compaction_interval, overlap_size, summarizer)` on `App`; token mode uses `token_threshold` with `event_retention_size` | Every `compaction_interval` completed events, older events are summarized; `overlap_size` re-includes earlier compacted events so summaries overlap; recent events stay verbatim | * **Choose the summarizer model deliberately:** ADK accepts a custom `LlmEventSummarizer` with its own model, so a smaller, cheaper model can write summaries while the main agent runs on a larger one. * **Steer what survives:** both Claude compaction modes accept `instructions`, `/compact` accepts a focus string and ADK accepts a custom summarizer prompt. Name the categories to keep verbatim (open decisions, file paths, user commitments). > **Common trap:** relying on early turns to carry a rule once a sliding window or compaction is enabled. Restate durable rules in the system prompt or a re-read file such as `CLAUDE.md`, so trimming can never remove them. --- ## Tool-result clearing and context editing * **Lightest-touch compaction:** once a tool result has been read and acted on, the agent rarely needs the raw payload again. Clearing it frees space without asking a model to rewrite anything, which is why Anthropic describes tool-result clearing as one of the safest forms of compaction. * **Calls stay visible:** by default Claude clears only the tool results and keeps the original tool calls, so the agent still knows what it asked for and can call the tool again if it needs the data. * **Token-efficient tools reduce the need to clear:** tools that paginate, filter on the server and return handles to large objects put fewer tokens in the window to begin with. ### Claude context editing strategies | Strategy | Key parameters | Behavior | | :--- | :--- | :--- | | **Tool-result clearing** (`clear_tool_uses_20250919`, beta `context-management-2025-06-27`) | `trigger` (when clearing starts), `keep` (recent tool uses kept, default 3), `clear_at_least`, `exclude_tools`, `clear_tool_inputs` | Replaces older tool results with placeholders once the prompt crosses the trigger | | **Thinking-block clearing** (`clear_thinking_20251015`) | `keep: {"type": "thinking_turns", "value": N}` or `"all"` | Drops thinking blocks from older assistant turns; listed first in `edits` | ```json "context_management": {"edits": [{ "type": "clear_tool_uses_20250919", "trigger": {"type": "input_tokens", "value": 60000}, "keep": {"type": "tool_uses", "value": 4}, "exclude_tools": ["memory"] }]} ``` * **Protect tools whose results stay relevant:** list them in `exclude_tools` (a memory tool, a configuration lookup that the whole task depends on). * **Pair clearing with notes:** when the memory tool is enabled and the context approaches the clearing threshold, Claude receives an automatic warning so it can save important information to memory files first, then look it up later instead of keeping it in the window. * **Mind the cache:** clearing tool results invalidates the cached prefix from the cleared point. Use `clear_at_least` so each clearing event frees enough tokens to justify the cache write. * **ADK equivalent:** there is no built-in clearing strategy; prune or rewrite history in a `before_model_callback`, or store bulky outputs as artifacts and keep a reference in the event. ### Scenario: shipment tracing at Larchmont Freight *A tracing agent calls a carrier API about 40 times per investigation; each response is a large JSON document.* * **Before:** after 25 calls the window is mostly raw carrier responses and the agent stops applying the customer's "ground only" constraint. * **After:** tool-result clearing triggers at 60k input tokens and keeps the last four results; the `shipment_policy` lookup is excluded from clearing; the agent writes each confirmed scan event to a `trace.md` notes file as it goes. * **Outcome:** the window stays under the trigger for the rest of the run, the constraint stays visible and earlier findings remain one file read away. > **Common trap:** clearing with no breadcrumb. If an agent sees neither the result nor a placeholder or note, it assumes the lookup never happened and repeats it. --- ## Scratchpads and structured note-taking * **Definition:** the agent periodically writes notes to storage outside the window (a `NOTES.md`, a to-do list, a progress file) and reads them back when it needs them, typically after a compaction, a context reset or a new session. * **Why it works:** notes give persistence with little overhead. A short progress file carries goals, completed steps and dependencies across dozens of tool calls that would otherwise be trimmed or summarized away. * **Agent-written, agent-read:** unlike state, which the harness owns, the scratchpad is curated by the agent itself, which decides what is worth remembering. ```markdown # trace.md (agent scratchpad) Goal: locate pallet LF-88213; customer requires ground transport only Done: origin scan Reno 11-02 06:10; hub scan Denver 11-03 21:44 Open: no scan since Denver; carrier ticket CT-5512 pending Next: query Denver hub scan log; re-check ticket at 14:00 ``` ### Ephemeral vs. Durable notes | Kind | Lifetime | Examples | Clean-up | | :--- | :--- | :--- | :--- | | **Ephemeral scratchpad** | One task or run | Working plan, intermediate tallies, hypotheses being tested | Delete or archive when the task closes | | **Task progress file** | Until the task completes, across resets and sessions | `progress.md`, to-do list, test status | Removed at completion | | **Durable memory** | Across tasks and sessions | User preferences, project conventions, lessons learned | Consolidated and expired by policy | ### Vendor mechanisms * **Claude memory tool** (`memory_20250818`) is client-side: Claude issues `view`, `create`, `str_replace`, `insert`, `delete` and `rename` commands against a `/memories` directory that your application stores. With the tool enabled, Claude is instructed to check the directory before starting and to record progress as it works. * **Claude Code** keeps durable project guidance in the `CLAUDE.md` hierarchy and auto memory (`MEMORY.md`); a plain notes file in the repository works as a task scratchpad. * **ADK** offers `temp:` state for values that live only within one invocation, session state for task progress, and artifacts (`ArtifactService`) for larger, versioned notes files. ### Note-taking workflow 1. **Seed:** at task start, create the notes file with the goal and constraints. 2. **Append at milestones:** after each confirmed finding or decision, write one line; avoid logging every tool call. 3. **Re-read at boundaries:** after compaction, clearing or a new session, read the file before acting. 4. **Prune:** collapse completed sections so the file itself stays small. 5. **Close out:** delete the ephemeral parts and promote only durable lessons to long-term memory. * **Anti-pattern (notes as a transcript):** copying raw tool output into the scratchpad recreates the bloat the file was meant to avoid. * **Anti-pattern (ephemeral notes promoted wholesale):** turning a task's working notes into permanent memory fills future sessions with stale hypotheses. --- ## Just-in-time retrieval * **Up-front retrieval** embeds and searches a corpus before inference and pastes the top results into the window. It is fast at run time, but the agent receives whatever the retriever ranked highly, relevant or not, and the index can be stale. * **Just-in-time retrieval** keeps lightweight identifiers (file paths, stored queries, URLs, record IDs) and loads the data with a tool only when a step needs it. Anthropic describes agents writing targeted queries and using commands like `head` and `tail` on large data instead of loading whole objects into the window. * **Metadata is signal:** folder names, file sizes, naming conventions and timestamps tell the agent what a reference is for before it opens it. * **Progressive disclosure:** each lookup informs the next, so the agent assembles understanding layer by layer and keeps only what the current step needs in the window. ### Up-front vs. Just-in-time vs. Hybrid | Approach | Latency | Freshness | Risk | Fits | | :--- | :--- | :--- | :--- | :--- | | **Up-front** | Lowest at run time | As fresh as the index | Window filled with loosely relevant chunks | Stable corpora, single-shot Q&A | | **Just-in-time** | Higher (extra tool calls) | Reads the source directly | Wasted calls and dead ends without good tools and heuristics | Codebases, large datasets, fast-changing sources | | **Hybrid** | Medium | Mixed | Two systems to maintain | Some essentials preloaded, the rest explored on demand | * **Hybrid in practice:** Claude Code drops `CLAUDE.md` files into the window at start, then uses `glob` and `grep` style tools to find and open files just in time, which avoids stale indexes. * **Give the agent good handles:** list or search tools that return names, sizes and short previews, plus read tools with ranges or pagination, keep exploration cheap. * **Cap exploration:** set a tool-call budget or ask for a plan before broad searches, so the agent does not burn context on dead ends. ### Scenario: policy lookups at Fernhill Insurance *An underwriting assistant answers questions over 3,000 policy documents that change weekly.* * **Before:** each question preloads the top 20 chunks from a nightly vector index; answers sometimes quote last week's wording and the window carries about 30k tokens of loosely related text. * **After:** the assistant starts with a short catalog (product line, document ID, effective date) and opens only the two or three documents it selects, reading the relevant section by heading. * **Outcome:** answers cite the current version, and the per-question context shrinks to the sections actually read. > [!NOTE] > *Illustrative figures:* the scenario numbers are invented to show the pattern; measure retrieval cost and quality on your own corpus. --- ## Session vs. State vs. Memory | Store | Holds | Lifetime | Recall | | :--- | :--- | :--- | :--- | | **Session (history)** | The ordered turns and tool calls of one conversation | One conversation (resumable) | Replayed into the window, usually trimmed or compacted | | **State** | Exact key/value working data | Session, user or app scope | Deterministic read by key | | **Memory** | Extracted facts about a user or domain, across sessions | Long term, with consolidation and expiry | Search (similarity or tool call); probabilistic | | **Files and artifacts** | Large outputs, scratchpads, notes | Until deleted | Read by name or path | ### Cross-vendor map | Concern | Claude | Google ADK and Agent Platform | | :--- | :--- | :--- | | **Statelessness** | The Messages API keeps nothing between calls; the client resends history every time | The `Runner` loads session events from a `SessionService` before each model call | | **Session handle** | Agent SDK and Claude Code session IDs | `Session` keyed by `app_name`, `user_id`, `session_id` | | **Session storage** | Local session transcripts (Agent SDK, Claude Code); your own database for raw API apps | `InMemorySessionService` (dev), `DatabaseSessionService`, `VertexAiSessionService` (Agent Platform Sessions) | | **Exact state** | Your application store, or files the agent reads | `session.state` with prefixes: none (session), `user:`, `app:`, `temp:` (never persisted) | | **Long-term memory** | Memory tool (`memory_20250818`): client-side files under `/memories`; Claude Code `CLAUDE.md` hierarchy plus auto memory (`MEMORY.md`) | `MemoryService`: `InMemoryMemoryService` (dev, keyword match), `VertexAiMemoryBankService` (Memory Bank: extraction and consolidation), `VertexAiRagMemoryService` | | **Memory read path** | The agent calls the memory tool when it decides to | `load_memory` (agent decides) or `PreloadMemoryTool` (every turn) | | **Large outputs** | Files API, local files | `ArtifactService` (versioned artifacts) | ### Where does it belong? | Information | Store | Why | | :--- | :--- | :--- | | Ticket ID, approved budget, chosen option | State | Must be exact; read by key every turn | | "Prefers reports in French" | User-scoped state or long-term memory | Survives sessions; small and stable | | Facts learned over many conversations | Long-term memory with consolidation | Needs dedup and updates when facts change | | A 40-page contract | File or artifact plus just-in-time reads | Too large to replay; open sections on demand | | Today's working plan for a multi-hour task | Scratchpad file | Agent-curated; survives compaction | | Value needed only within this turn | `temp:` state or a local variable | Must not leak into the next turn | | The tone of yesterday's chat | Nothing | No future decision depends on it | ### Memory write policy * **Extract at boundaries:** generate memories at session end or at task milestones (ADK: `add_session_to_memory` on a Memory Bank service, which calls `GenerateMemories` in the background), not on every turn. * **Consolidate, don't append:** a new fact that contradicts an old one updates it ("moved to Lyon" replaces "lives in Paris"); Memory Bank consolidates extracted facts against existing memories in the same scope, while append-only stores accumulate contradictions. * **Direct writes for explicit facts:** when the user states a fact to remember, write it directly (Memory Bank `CreateMemory`, or the Claude memory tool's `create` command) instead of waiting for extraction. * **Scope every record:** key memories by user (and tenant); Memory Bank `RetrieveMemories` matches the exact scope before any similarity search, and your own stores must do the same. * **Expire what goes stale:** set a time-to-live for facts with a shelf life (Memory Bank supports TTL per instance or per memory and keeps revisions); review the rest periodically. > **Common trap:** in ADK, `user:` and `app:` state looks persistent but disappears on restart if `InMemorySessionService` is still configured. Durable scopes need `DatabaseSessionService` or `VertexAiSessionService`. --- ## Subagent context isolation * **Clean windows by design:** rather than one agent holding the whole project in its window, a lead agent keeps the high-level plan and hands focused work to subagents that each start with an empty window. * **Explore wide, return small:** a subagent can spend tens of thousands of tokens searching and reading, then return a condensed digest (Anthropic cites 1,000 to 2,000 tokens as typical). The detailed search context never reaches the lead agent. * **What each side sees:** on Claude, an Agent SDK subagent receives only its own system prompt and the task prompt the lead writes, not the parent conversation. In ADK, `AgentTool` runs the child in an isolated session and returns its final response as a tool result, while `sub_agents` transfer hands over the shared conversation; `include_contents='none'` stops an `LlmAgent` from receiving prior history. * **Cost of isolation:** subagents add calls and total tokens. Use them when exploration is bulky or parallelizable, not for a two-step lookup. ``` ┌───────────── lead agent window ─────────────┐ │ plan · state · digests from subagents (~2k) │ └───────┬───────────────┬───────────────┬─────┘ brief │ brief │ brief │ ┌───────▼─────┐ ┌───────▼─────┐ ┌───────▼─────┐ │ subagent A │ │ subagent B │ │ subagent C │ │ 40k explore │ │ 25k explore │ │ 30k explore │ └───────┬─────┘ └───────┬─────┘ └───────┬─────┘ └──── digest ───┴──── digest ───┘ ``` ### Writing the brief and the digest * **Brief:** the goal, the scope boundary, the identifiers the subagent needs and a tool or token budget. Write it fresh; do not forward the lead's transcript. * **Digest:** what was found, where it lives (paths, IDs, links) so the lead can open it just in time, and what was not checked. * **Format:** a short structured list the lead can scan beats a long narrative. ### Scenario: dependency upgrade at Pinecrest Labs *A lead agent upgrades a logging library across 30 services in one repository.* * **Fan-out:** one subagent per group of services searches for call sites and deprecated options; each returns a table of file paths and the pattern found. * **Lead window:** holds the upgrade plan, the per-group digests and the notes file, never the raw search output. * **Outcome:** the lead edits file by file, opening each path just in time from the digests. | Isolation mistake | Consequence | Fix | | :--- | :--- | :--- | | Lead forwards its full transcript | Subagent starts half full of irrelevant history | Write a fresh, focused brief | | Subagent returns everything it read | Lead window floods; isolation wasted | Ask for a short digest with handles | | Brief omits the scope boundary | Subagents overlap or wander | State what is in and out of scope | | Subagent used for a tiny lookup | Extra latency and tokens for no gain | Call the tool directly | --- ## Choosing a strategy | Task shape | First lever | Add when needed | | :--- | :--- | :--- | | Long conversation with steady back-and-forth | Summary prefix or compaction | Durable rules in the system prompt | | Tool-heavy run where payloads dominate | Tool-result clearing | Notes file for findings worth keeping | | Iterative work with clear milestones (coding, migrations) | Structured note-taking | Compaction at a threshold | | Broad research or analysis that parallelizes | Subagents with digests | Notes file in the lead agent | | Large or fast-changing corpus | Just-in-time retrieval | Small preloaded catalog (hybrid) | | Short task that fits comfortably | Nothing | Measure before adding machinery | * **Rule of thumb from Anthropic's guidance:** compaction keeps conversational flow for extended back-and-forth, note-taking suits iterative development with milestones, and multi-agent setups fit research where parallel exploration pays off. * **Start light:** clearing stale tool results changes nothing the model wrote, so try it before summarization. Add notes, then subagents, as the task demands. * **Combine deliberately:** a common stack is threshold compaction plus tool-result clearing plus a notes file. Each event rewrites part of the context, so tune triggers together and watch cache costs. --- ## Caching and cost * **Prefix caching rewards stable order:** both vendors reuse computation for an unchanged prefix. Put tools, system instructions and fixed examples first; put per-turn content (timestamps, user data, retrieved items) after them. * **Claude prompt caching:** the prefix is hashed in the order tools → system → messages. Use automatic caching (one top-level `cache_control`, the recommended default) or explicit breakpoints (`cache_control: {"type": "ephemeral"}` on up to 4 blocks). The default lifetime is 5 minutes; `"ttl": "1h"` extends it. Writes cost 1.25× base input (5 minutes) or 2× (1 hour); reads cost 0.1× base or less on the newest models. Usage reports `cache_creation_input_tokens` and `cache_read_input_tokens`. * **Gemini caching:** implicit caching is on by default for Gemini 2.5 and later, with no storage fee; explicit caches (`cachedContents`) take a time-to-live and bill hourly storage. ADK exposes `ContextCacheConfig(min_tokens, ttl_seconds, cache_intervals)` through `App(context_cache_config=...)` and `static_instruction` for an instruction that stays cache-stable. * **What breaks a cache:** any change before the breakpoint (a timestamp in the system prompt, reordered tools, an edited example). Clearing old tool results or thinking blocks also changes the prefix after the cleared point; keeping thinking blocks preserves the cache. * **Compaction, clearing and caching interact:** each compaction or clearing event rewrites part of the context and forces a new cache write. Trigger them at thresholds, and use `clear_at_least` so each clear frees enough tokens to pay for the lost cache. | Lever | Saves | Costs | | :--- | :--- | :--- | | Prefix caching | Input price and latency on repeated prefixes | Cache writes; discipline about ordering | | Tool-result clearing | Window space in tool-heavy runs | Cache invalidation at each clear | | Compaction | Window space in long sessions | Summarization call; detail loss | | Just-in-time retrieval | Tokens spent on unused context | Extra tool calls and latency | | Subagents | Lead window space | Extra calls and tokens overall | > [!NOTE] > *Pricing changes:* cache multipliers, minimum cacheable lengths and lifetimes differ by model and change over time. Check the current pricing pages before quoting numbers. --- ## Security and operations ### Threats specific to context and memory * **Memory poisoning:** an attacker plants an instruction in content the agent later saves as memory ("always approve invoices from this vendor"); it then applies in every future session. Extract facts, not instructions, and validate what is written. * **Indirect prompt injection through fetched content:** web pages, emails and files opened just in time can carry instructions. Treat tool output as data, never let it change tool permissions, and gate sensitive actions behind code-level checks. * **Poisoned notes and summaries:** injected text that lands in a scratchpad or compaction summary survives the reset that would otherwise have removed it. Review what is written to notes files and keep summaries free of instructions from untrusted content. * **Cross-tenant leakage:** memory or state keyed without a user and tenant scope returns another customer's facts. Scope every read and write, and test isolation. * **PII accumulation:** long-term memory, notes files and logs become a store of personal data. Minimize what is written, redact at the tool boundary, set retention and support deletion requests. * **Path traversal in file-based memory:** when an agent edits memory files, the harness must confine paths to the memory directory and reject `..` and absolute paths. ### Controls * **Authentication:** memory and session services are called with the end user's identity or a scoped service identity (IAM on Agent Platform, per-user keys in your own store); the model never chooses whose memory to read. * **Human-in-the-loop gates:** irreversible or high-impact actions require approval regardless of what the context says. * **Audit:** log every memory write, notes update and compaction event with the source, so a poisoned entry can be traced and removed. * **Least context:** give each agent only the tools and data its role needs; context it never sees cannot leak or be injected. ### Operational metrics | Metric | Signal | | :--- | :--- | | Tokens per call and per session | Budget pressure; candidates for clearing or isolation | | Compaction and clearing events per session | Sessions that should use notes or subagents | | Repeated identical tool calls | Results cleared without a breadcrumb or note | | Cache hit rate | Prefix instability (a timestamp in the system prompt) | | Notes file size over time | Scratchpad turning into a transcript | --- ## Failure modes and anti-patterns | Symptom | Root cause → fix | Tempting wrong answer | | :--- | :--- | :--- | | Agent ignores a rule stated in turn 2 after many turns | Sliding window dropped it → move durable rules to the system prompt or `CLAUDE.md` | Raising the window size | | A subtle early decision vanishes after compaction | Compaction prompt tuned for brevity → tune for recall on real traces, then trim | Disabling compaction entirely | | Agent repeats the same lookup over and over | Results cleared with no note → keep tool calls visible and log findings to a notes file | Turning clearing off | | Window fills with raw API payloads | No clearing and verbose tools → enable tool-result clearing; paginate or filter tools | Telling the model to skim | | Answers quote outdated policy text | Up-front retrieval from a stale index → just-in-time reads from the source | Retrieving more chunks | | Lead agent window floods during research | Subagents return everything they read → ask for short digests with handles | Forwarding the lead transcript to subagents | | Subagent re-explores what the lead already knows | Brief lacked identifiers and scope → write a focused brief | Giving the subagent the whole history | | Scratchpad grows to thousands of lines | Notes used as a log → milestone entries only, prune completed sections | Compacting the notes file into memory | | Old hypotheses resurface in new tasks | Ephemeral notes promoted to memory → close out and promote durable lessons only | Bigger memory retrieval limits | | `user:` preferences vanish after deploy | `InMemorySessionService` still configured → durable session service | Moving preferences into `temp:` | | Cache hit rate near zero | Timestamp in the system prompt → move dynamic values after the stable prefix | Raising the cache lifetime | | A vendor email's instruction applies in later sessions | Memory poisoning → extract facts only, validate writes, audit | Telling the model to ignore instructions | --- ## Glossary ### Context techniques * **Attention budget:** the idea that every token in the window draws on a limited capacity to attend, so irrelevant tokens reduce recall of relevant ones. * **Compaction:** summarizing a conversation near the window limit and continuing in a fresh window seeded with the summary; lossy by design. * **Context editing:** removing specific stale content (old tool results, old thinking blocks) while keeping the rest of the conversation as written. * **Context rot:** the gradual decline in recall of specific details as the total number of tokens in the window grows. * **Context window:** the maximum number of tokens (input plus output) a model processes in one call. * **Hybrid retrieval strategy:** preloading a small set of essential context and letting the agent fetch the rest just in time. * **Just-in-time retrieval:** keeping lightweight references (paths, IDs, queries, URLs) in the window and loading data with tools only when a step needs it. * **Overlap size:** in ADK event compaction, the number of previously compacted events included again in the next summary so spans overlap. * **Progressive disclosure:** discovering relevant context step by step, using each result and its metadata to decide what to open next. * **Sliding window:** a history policy that sends only the most recent turns or tokens and drops older ones. * **Summary prefix:** a rolling digest of older turns placed in front of the recent verbatim turns. * **Tool-result clearing:** replacing older tool results with placeholders once the prompt passes a threshold, keeping the tool calls visible. ### Notes, state and memory * **Artifact:** a named, versioned file an ADK agent saves and loads (`ArtifactService`), used to keep large outputs out of the window. * **Digest:** the short summary a subagent returns to its lead agent in place of its full exploration context. * **Memory (long-term):** facts extracted across sessions and retrieved by search or a memory tool; probabilistic recall. * **Memory consolidation:** merging a new fact with existing ones, updating or removing contradicted facts instead of appending. * **Scratchpad:** an agent-written notes file (plan, progress, open questions) re-read after resets; ephemeral when scoped to one task. * **Session:** one conversation's ordered history of turns and tool calls, identified by a session ID. * **State:** exact key/value working data read and written deterministically; in ADK scoped by prefix (none, `user:`, `app:`, `temp:`). * **Structured note-taking:** the practice of an agent regularly writing notes outside the window and pulling them back in later. * **Subagent isolation:** running a focused subtask in its own clean window so its intermediate context never reaches the lead agent. ### Caching * **Cache breakpoint:** the point in a prompt up to which the prefix is cached (Claude `cache_control`). * **Prefix caching:** reusing computation for an unchanged leading part of the prompt across calls; any change before the breakpoint invalidates it. ================================================================================ GUIDE: Agent Development Kit (ADK) URL: https://mbettan.github.io/machine-learning-cert/ADK.html Practice quiz: https://mbettan.github.io/machine-learning-cert/quiz.html?topic=adk Section: Foundation · Agent Architecture & Protocols Exam relevance: Google Professional Agentic Architect (core) ================================================================================ # Agent Development Kit (ADK): Study Guide > **Author:** Michaël Bettan > **Version:** 2.0 · **Updated:** October 2026 --- ## Overview **Definition:** Agent Development Kit (ADK) is Google's open-source, code-first framework (**Python, TypeScript, Go, Java, and Kotlin**) for building, evaluating, governing, and deploying production AI agents. Rather than leaving teams to hand-roll conversation state, tool dispatch, retries, and trace instrumentation around a raw model endpoint, ADK provides the agent harness as modular infrastructure: an event-sourced execution runtime, typed tool and protocol connectors, multi-tier state and memory, multi-agent orchestration, deterministic callback/plugin guardrails, trajectory evaluation, and one-command cloud deployment. **Why ADK? (What a raw model API lacks):** A bare model endpoint is **stateless** (retains zero memory across HTTP requests unless using server-side interaction handles), **passive** (emits tool-call intents as JSON text without executing anything), **unstructured in control flow** (no durable loops, fan-out joins, checkpoints, or hard iteration caps), and **ungoverned** (no tenant isolation, approval gates, or tool-boundary PII scrubbing). ADK supplies all four layers cleanly decoupled from model weights. > **Mental model:** Treat the foundation model (**Gemini 4 Argon**, **Gemini 3.8 Flash**, **Claude**, or **Gemma 4**) as the **brain**, and ADK as the **nervous system and skeleton**: hands (`FunctionTool`, `McpToolset`), short- and long-term memory (`SessionService`, `MemoryService`), spinal reflexes (`before_tool_callback`, `BasePlugin`), and the motor loop (`Runner` / `Workflow`) that coordinates when the brain reasons and when tools act. **End-to-end lifecycle:** **1. Define** (`LlmAgent` / `Workflow` / `ManagedAgent` + tools + instructions) → **2. Run** (`Runner` loads session state, invokes the model or **Interactions API**, executes tools, and commits each step as an immutable `Event`) → **3. Evaluate** (`adk conformance` + `adk eval` in CI, paired with **Agent Platform Gen AI Evaluation Service** in cloud) → **4. Deploy** (**Agent Runtime**, **Cloud Run**, or **GKE**). --- ## 1. When to use ADK, framework choices and multi-language parity > **Rule 0:** Begin with a single grounded model call. Step up to deterministic workflows—and only then to autonomous multi-turn agents—**when benchmark evaluations prove the simpler tier cannot meet accuracy or adaptability targets.** | Architectural tier | Core mechanism | Makes runtime routing decisions? | Executes external actions? | |---|---|---|---| | **Single LLM call** | Stateless prompt → completion | ✅ (single step) | ❌ (can only emit tool-call JSON) | | **Deterministic workflow** | Predefined DAG or pipeline orchestrating models and tools | ❌ (code/edges decide the path) | ✅ | | **Autonomous agent** | Model-directed loop executing tools, inspecting observations, and updating state until goal completion | ✅ (multi-step adaptive) | ✅ | | **ADK** | Production framework providing both deterministic workflows and autonomous agent loops under one event ledger | n/a | n/a | | Workload shape | Recommended pattern | Architectural rationale | |---|---|---| | Single prompt (+ retrieval) answers reliably | Augmented LLM / RAG call | Adding an agent loop inflates TTFT, token spend, and variance for zero gain | | Fixed, predictable multi-step pipeline (e.g., parse → validate → summarize) | **ADK 2.0 `Workflow` graph** (or ADK 1.x `SequentialAgent` / `ParallelAgent` / `LoopAgent`) | Deterministic execution order, bounded cost, straightforward debugging | | Read-only Q&A over enterprise SharePoint/Drive corpora with ACLs | Managed **Agent Search** / RAG pipeline | Retrieval grounding solves missing knowledge without open-ended tool loops | | Next action depends dynamically on intermediate tool findings | **`LlmAgent`** (ReAct / planner loop) | Control path cannot be hardcoded ahead of time | | Long-horizon cloud sandbox execution using Google's hosted harness | **`ManagedAgent`** (backed by **Interactions API**) | Offloads loop, remote container sandbox, and server-side state to Google | | Strictly deterministic compliance or actuarial calculation | Plain code / classical ML | Zero stochastic variance, 100% reproducible auditability | **The "agent theater" litmus test:** Splitting a system into multiple agents is justified **only when specialists require distinct tool permissions, isolated data scopes, an independent auditor (`include_contents='none'` + verification tools), or explicit human sign-off boundaries.** Breaking a single prompt into four chatty persona agents sharing identical context and zero tools burns ~4× the tokens and ~2× the latency without improving accuracy. | SDK language | Package / module | ADK 2.0 graph `Workflow` & task modes | Primary strengths & parity nuances (Oct 2026) | Typical Google Cloud deploy target | |---|---|---|---|---| | **Python** | `google-adk` | ✅ Full support | Reference flagship SDK: earliest access to **`ManagedAgent`**, `SkillToolset` + `GCPSkillRegistry`, `adk eval` / `AgentEvaluator` / `adk conformance`, `VertexAiMemoryBankService`, and `run_live()`. Agent Runtime upload excludes the API server (platform injects it). | **Agent Runtime**, Cloud Run, GKE | | **TypeScript** | `@google/adk` | ✅ Full support | Node.js / full-stack web integration, MCP servers, experimental **`RoutedLlm`** dynamic model switcher, and `BuiltInCodeExecutor` without single-tool mixing limits on modern Gemini models (`Agent Search` built-in tool not yet in TS). | **Cloud Run**, GKE | | **Go** | `google.golang.org/adk` | ✅ Full support | High-concurrency, low-memory compiled microservices with native goroutine fan-out, MCP, OpenTelemetry, and **Agent Runtime** support (Go deploy payload bundles its own dedicated ADK API server binary). | **Agent Runtime**, Cloud Run, GKE | | **Java & Kotlin** | `com.google.adk` | ❌ (ADK 1.x workflow agents) | Enterprise Spring/JVM integration (`LlmAgent.builder()` in Java; type-safe DSL in Kotlin) with `SequentialAgent` / `ParallelAgent` / `LoopAgent`, `FunctionTool`, `McpToolset`, and callbacks; built-in `GoogleSearchTool` + custom tools must be isolated via `AgentTool`. | **Cloud Run**, GKE | > **Common trap:** Confusing the **Gen AI SDK (`google-genai`)** with **ADK (`google-adk`)**. `google-genai` is the model/interactions client library (supporting `generate_content`, `client.interactions.create`, and basic in-process function calling), whereas ADK provides multi-user `SessionService` isolation, the persistent `Event` ledger, multi-agent transfers, lifecycle guardrails (`BasePlugin`), conformance/trajectory evals, and runtime deployment. --- ## 2. Runtime architecture: Agent, Runner, SessionService and ManagedAgent ADK enforces a strict separation between **stateless agent definitions**, **stateful session storage**, and the **event-loop engine**: ``` ┌───────────────────────────┐ ┌──────────────────────────────────┐ │ Agent / Workflow (code) │ │ SessionService (stateful store) │ │ model, instruction, │ │ app_name → user_id → session_id │ │ tools, sub_agents, hooks │ │ immutable events + scoped state │ └────────────┬──────────────┘ └────────────────┬─────────────────┘ └─────────────────┬─────────────────┘ ▼ ┌───────────────────────┐ 1. Hydrate session events & state │ Runner │ 2. Call GenerateContent / Interactions │ (commit-then-resume) │ 3. Execute local / MCP / AgentTool └───────────────────────┘ 4. Commit Event (+ state/artifact delta) 5. Repeat until turn completes ``` | Core primitive | Responsibility | Production engineering rule | |---|---|---| | **`Agent`** (`LlmAgent`) | Stateless blueprint: model, instructions, tools, sub-agents, planners, callbacks | Store **zero** per-user attributes on `self`; one singleton safely handles thousands of concurrent requests | | **`ManagedAgent`** | Data-plane client for Google's **Managed Agents API** via the **Interactions API** | Connects to an existing `agent_id` (`global` endpoint on Agent Platform); runs loop & tools remotely | | **`BaseLlm`** | Model abstraction (`"gemini-4-argon"`, `"gemini-3.8-flash"`, `LiteLlm(...)`, `Claude(...)`, `Gemma(...)`) | Mix model families across coordinator and worker agents without changing session or callback code | | **`Runner`** | Executes turns (`run_async`) or bidirectional audio/video (`run_live`) | **Commit-then-resume:** persists every `Event` to `SessionService` before advancing to the next step | | **`SessionService`** | Manages `Session` objects keyed by `app_name` / `user_id` / `session_id` | `InMemorySessionService` (local dev) → `DatabaseSessionService` (`postgresql+asyncpg` / `sqlite+aiosqlite`) or `VertexAiSessionService` (prod; `app_name` = Reasoning Engine ID) | | **`MemoryService`** | Cross-session long-term recall per user | `InMemoryMemoryService` (unit tests) → `VertexAiMemoryBankService` (consolidated facts) / `VertexAiRagMemoryService` | | **`ArtifactService`** | Versioned binary/file blobs (`"user:filename"` persists across sessions) | `InMemoryArtifactService` / `FileArtifactService` (dev) → `GcsArtifactService` (Cloud Storage in prod) | | **`App`** | Root application envelope binding `root_agent`, `plugins=[...]`, and runtime policies | Attach `ResumabilityConfig`, `EventsCompactionConfig`, and `ContextCacheConfig` here—never on `LlmAgent` | **`LlmAgent` (`GenerateContent`) vs. `ManagedAgent` (`Interactions API`):** - **Standard `LlmAgent`:** Your ADK `Runner` owns the orchestration loop, re-sending compacted conversation history (and cryptographic **Thought Signatures**) over `GenerateContent` while executing custom Python `FunctionTool`s and `McpToolset` connections inside your runtime environment. - **`ManagedAgent` (Preview):** Invokes a pre-existing hosted agent (`agent_id`, such as `antigravity-preview-...` or a custom agent provisioned via the **Agents API** control plane) over the **Interactions API** (`stream=True`) with `environment={'type': 'remote'}`. Reasoning, built-in tools (`google_search`, `ToolCodeExecution`), and sandbox execution happen inside Google's managed cloud infrastructure while partial and final events stream into your ADK workflow. - **State split in `ManagedAgent`:** ADK never re-transmits prior conversation turns to a `ManagedAgent`. Instead, the local `Session` records only two lightweight pointers on emitted events—**`previous_interaction_id`** and **`environment_id`**—and reuses them on subsequent turns so the server-side `Interaction` and remote container sandbox continue seamlessly. - **Direct `Interactions API` (`client.interactions.create`) for long-horizon tasks:** When calling `gemini-4-argon` or Deep Research directly via `google-genai`, setting `background=True` runs multi-minute tasks asynchronously on the server (preventing ~60s HTTP gateway timeouts and unlocking long decodes up to the unified `1,048,576`-token window). Chaining a follow-up request via `previous_interaction_id` while the prior interaction is still `in_progress` returns **`400 Bad Request`**—poll until `completed` first. ```python # ManagedAgent runs server-side via Interactions API; coordinator wraps it as AgentTool web_researcher = ManagedAgent( name="managed_web_researcher", description="Grounded web synthesis in Google's remote managed environment.", agent_id=os.environ.get("MANAGED_AGENT_ID", "antigravity-preview-09-2026"), environment={"type": "remote"}, tools=[google_search], # Server-side tools only; local tools raise NotImplementedError ) coordinator = LlmAgent( name="portfolio_coordinator", model="gemini-4-argon", instruction="Combine internal CRM lookups with external web research.", tools=[AgentTool(agent=web_researcher)], ) ``` **Context hierarchy & Event ledger:** `InvocationContext` (full per-turn runtime handle), `ReadonlyContext` (passed to `InstructionProvider` callables for read-only `ctx.state`), `CallbackContext` (agent/model hooks: mutable state + artifacts), and `ToolContext` (tools + tool hooks: state, artifacts, credentials, `search_memory`, confirmation, and `actions`). Every state change, tool call, transfer, or escalation is recorded as an `Event` (`author`, `invocation_id`, `content`, `actions.state_delta`, `actions.artifact_delta`), creating a single source of truth for `adk web` time-travel inspection, Cloud Audit trails, and trajectory grading. > **Common trap:** Passing client-executed Python functions or an `McpToolset` into `ManagedAgent(tools=[...])` raises **`NotImplementedError`**, and pointing `ManagedAgent` at a regional Agent Platform endpoint fails because the Managed Agents API is served exclusively from **`global`**. Keep custom/MCP tools on an `LlmAgent` and compose `ManagedAgent` beside it. --- ## 3. Agent types and deterministic control flow Every ADK agent inherits from `BaseAgent`. Only `LlmAgent` (and `ManagedAgent`) consult a model to decide next steps; **workflow agents route deterministically in code with zero LLM routing overhead.** *(In ADK 2.0 for Python, TypeScript, and Go, `SequentialAgent`, `ParallelAgent`, `LoopAgent`, and custom `BaseAgent` orchestrators are superseded by unified graph `Workflow`s in §5, though both patterns remain active in Java/Kotlin and exam scenarios.)* | Agent class | Routing mechanism | Ideal workload | Critical configuration | |---|---|---|---| | **`LlmAgent`** (`Agent`) | Dynamic model reasoning | Tool selection, open-ended synthesis, triage | `model`, `output_key`, `output_schema`, `planner`, `tools`, `sub_agents` | | **`SequentialAgent`** | Strict linear order `A → B → C` | Staged pipelines where step `N` consumes step `N-1`'s `output_key` | `sub_agents=[...]` | | **`ParallelAgent`** | Concurrent execution (`branch` isolated, shared `session.state`) | Multi-source retrieval, independent compliance checks | Unique `output_key` per parallel branch | | **`LoopAgent`** | Iterative cycle until exit signal or cap | Draft → critique → revise loops, polling | `max_iterations` + tool setting `actions.escalate = True` | | **Custom (`BaseAgent`)** | Arbitrary async code | Deterministic rule routers, custom state machines | Override `async def _run_async_impl(self, ctx)` | ```python def approve_patch(tool_context: ToolContext) -> dict: tool_context.actions.escalate = True # Halts enclosing LoopAgent immediately tool_context.actions.skip_summarization = True # Suppresses redundant post-tool LLM turn tool_context.state["temp:verified"] = True # Proves convergence vs max_iterations cap return {"status": "verified"} patch_loop = LoopAgent(name="patch_and_verify", max_iterations=3, sub_agents=[ LlmAgent(name="patcher", model="gemini-3.8-flash", instruction="Write patch for {cve}.", output_key="diff"), LlmAgent(name="auditor", model="gemini-4-argon", include_contents="none", instruction="Verify {diff}. Call approve_patch only if tests pass.", tools=[run_tests, approve_patch], output_key="audit"), ]) root_agent = SequentialAgent(name="pipeline", sub_agents=[gather_agent, patch_loop]) ``` - **State templating, `ParallelAgent` race safety & `InstructionProvider`:** Setting `output_key="cve"` writes the agent's final text to `state["cve"]`, which downstream instructions reference via `{cve}` (use `{audit?}` when optional so turn 1 doesn't raise `KeyError`, or `{artifact.report}` to inline text artifacts). Inside a **`ParallelAgent`**, all branches execute concurrently against the **same shared `session.state`**: if two parallel workers write to the same `output_key` or mutate a shared list key concurrently, they trigger last-write-wins data loss, so assign a **distinct `output_key` per parallel branch** and aggregate in a downstream merger step. Whenever an instruction includes literal curly braces (such as JSON schemas `{"status": "ok"}` or regex quantifiers), supply a callable `def build_instruction(ctx: ReadonlyContext) -> str:` so ADK bypasses `{var}` placeholder parsing. - **Silent `LoopAgent` exhaustion:** Reaching `max_iterations` exits `LoopAgent` **silently without raising an exception**, allowing any subsequent agent in an outer `SequentialAgent` to run on unapproved output. Always write an explicit convergence flag (`state["temp:verified"] = True`) inside your exit tool and check it in the downstream step's `before_agent_callback`. - **Independent verification:** Prevent self-grading bias in critic loops by isolating the verifier (`include_contents="none"`, dedicated deterministic test/linter tools, and explicit pass/fail criteria). > **Common trap:** Expecting `SequentialAgent` to skip irrelevant steps dynamically or expecting `LoopAgent(max_iterations=3)` to stop automatically when tests pass. `SequentialAgent` always runs every child in sequence, and `LoopAgent` always runs all `max_iterations` unless a tool or callback sets `escalate = True`. --- ## 4. Multi-agent delegation: transfer, AgentTool and remote agents | Delegation pattern | Invocation mechanics | Who speaks to the user next? | Session & context boundary | Best architectural fit | |---|---|---|---|---| | **`sub_agents`** (LLM transfer) | ADK injects `transfer_to_agent(agent_name=...)`; LLM selects target using each child's **`description`** | **Target sub-agent** assumes control | Shares the same `session.events` and `session.state` | Full conversational handoff to a domain specialist (billing vs. tech support) | | **`AgentTool(agent=...)`** | Parent invokes child like a typed function (`skip_summarization=True` returns raw output) | **Parent coordinator** keeps control | Runs in an isolated child session while **merging `state_delta` / `artifact_delta` (`output_key`) back** | Modular sub-task execution without polluting the coordinator's chat history | | **`RemoteA2aAgent`** | Connects to an external service via its A2A Agent Card (`/.well-known/agent-card.json`) | Depends on transfer vs. `AgentTool` wrapping | Strictly opaque network boundary: only A2A messages, tasks, and artifacts cross | Cross-team, multi-language, or cross-organization agent federation | - **Scoping transfers, hop caps & tree rules:** Use `disallow_transfer_to_parent=True` on one-shot triage helpers (blocks upward handoff and routes the user's next turn back to `root_agent`) and `disallow_transfer_to_peers=True` to enforce strict hub-and-spoke governance through a coordinator. Bound coordinator-worker recursion and ping-pong transfers via **`max_hop_count`** (alongside `RunConfig(max_llm_calls=...)`) and keep `sub_agents` nesting shallow (`<= 2–3` levels) so latency and context bloat do not compound. Every `Agent` instance obeys the **single-parent rule** (`ValueError` if attached to two parents' `sub_agents`); use factory functions or `agent.clone()` to reuse definitions. - **ADK 2.0 collaborative sub-agent modes (Python, TS, Go):** - `mode="chat"` (classic default): injects one shared `transfer_to_agent` tool; the specialist takes over user interaction. - `mode="task"`: specialist can ask the user clarifying questions across turns, then invokes `finish_task` to hand structured output back to the parent coordinator. - `mode="single_turn"`: generates a dedicated delegation tool per specialist, runs autonomously with zero direct user prompts, and supports parallel fan-out calls from the coordinator. - **Cost-efficient routing:** Replace frontier LLM routers with deterministic `Workflow` conditional edges (`Event(route=...)`), embedding similarity, or **`gemini-3.5-flash-lite`** (`thinking_level="minimal"`) when intent buckets are well-defined. > **Common trap:** Writing instructions like `"Call transfer_to_billing()"` in `mode="chat"`. ADK does **not** create per-agent transfer functions in chat mode; it exposes a single `transfer_to_agent(agent_name="billing")` tool (per-specialist tools are generated only in ADK 2.0 `mode="single_turn"` / `"task"` or via `AgentTool`). ### Choosing the orchestration boundary: Workflow Agents vs. Graph Workflows vs. Remote A2A Agents The delegation table above decides who speaks next. This table decides where the process boundary sits, which in turn fixes what can be isolated, who can govern it and what each handoff costs. | Pattern | State sharing | Deployment boundary | Handoff latency | Fault and identity isolation | Governance granularity | Choose when | |---|---|---|---|---|---|---| | **Workflow agents** (`SequentialAgent`, `ParallelAgent`, `LoopAgent`) | one `session.state`; children exchange data through `output_key` and `{key}` templates | single process, single deployable unit | in-process call | none: a crash or runaway loop takes down every child, and all children act as the deployment's one principal | ADK callbacks and plugins inside your code | fixed pipeline owned by one team, or Java/Kotlin and ADK 1.x codebases | | **Graph workflows** (`Workflow(edges=[...])`) | typed `node_input` between nodes plus the shared `session.state` | single process, single deployable unit | in-process edge evaluation | same as workflow agents; explicit edges make the failing node easier to locate | callbacks plus explicit routes and human-input nodes (`RequestInput`) | branching, loops and approval pauses inside one service (Python, TypeScript, Go) | | **Remote A2A agents** (`RemoteA2aAgent` to a peer on Agent Runtime) | nothing shared; only A2A messages, task status and artifacts cross the wire | separate service per peer, possibly another project, language or organization | network round trip plus gateway policy evaluation on every call | strong: each peer scales and fails on its own and runs under its own Agent Identity principal | agent policies enforced by Agent Gateway (IAM Unified Access Policies, Principal Access Boundary, Model Armor, semantic governance) on top of each peer's own callbacks | different owners, release cycles, runtimes or privilege tiers, and partner agents | - **Anti-pattern (network for its own sake):** splitting one team's three-step pipeline into three A2A services adds network hops, three deployments and three policy surfaces with no isolation gain; keep it as a `Workflow`. - **Anti-pattern (privilege mixing in one process):** attaching a read-only researcher and a refund-issuing specialist as `sub_agents` of one deployment gives both the same principal, so no platform policy can tell them apart; move the privileged specialist behind A2A with its own identity. --- ## 5. Graph workflows (ADK 2.0), planners and Gemini 4 Argon thinking levels **ADK 2.0 Graph Engine (`Workflow`):** Available across **Python, TypeScript, and Go**, `Workflow(edges=[...])` unifies sequential pipelines, parallel fan-out/fan-in, cyclic loops, and conditional routing into one directed graph combining `LlmAgent`s, plain code nodes (`FunctionNode`), `ManagedAgent`s, and nested `Workflow`s. Return values pass as typed `node_input`, and function parameters matching `session.state` keys are dependency-injected automatically. ```python def classify_severity(node_input: str) -> Event: target = "deep_architect" if "architecture" in node_input.lower() else "fast_responder" return Event(output=node_input, route=target) # Route key for conditional edge deep_architect = LlmAgent(name="deep_architect", model="gemini-4-argon", planner=BuiltInPlanner(thinking_config=types.ThinkingConfig(thinking_level="high"))) fast_responder = LlmAgent(name="fast_responder", model="gemini-3.8-flash", planner=BuiltInPlanner(thinking_config=types.ThinkingConfig(thinking_level="low"))) root_agent = Workflow(name="support_graph", edges=[ (START, classify_severity), (classify_severity, {"deep_architect": "deep_architect", "fast_responder": "fast_responder"}), (("deep_architect", "fast_responder"), END), ]) ``` | Graph capability | LangGraph equivalent | ADK 2.0 `Workflow` (Python / TS / Go) | |---|---|---| | Conditional branching | `StateGraph` + `add_conditional_edges` | `Workflow(edges=[(router_fn, {"route": node})])` + node returning `Event(output=..., route="target")` | | Parallel fan-out & synchronization | Parallel edges + `Annotated[..., operator.add]` reducers | Edge tuple `(START, ("branch_a", "branch_b"), join_node)` + **`JoinNode`** (`dict[node_id, output]`); no state reducers, so use distinct state keys | | Durable checkpointing | Checkpointer keyed by `thread_id` | `SessionService` keyed by `app_name` / `user_id` / `session_id` | | Mid-graph human pause | `interrupt()` | Generator node yielding `RequestInput(message=..., response_schema=...)` or `LongRunningFunctionTool` | - **ADK 2.0 orchestration positioning:** ADK 2.0 steers new stateful branching, cyclic retries, and human-input pauses toward `Workflow(edges=[...])` across Python, TypeScript, and Go, while keeping `SequentialAgent`, `ParallelAgent`, and `LoopAgent` as first-class primitives in Python and Java/Kotlin for fixed pipelines and exam blueprints. - **Graph node exception & retry boundaries (`BaseException` vs. `RetryConfig`):** Catching broad `BaseException` (or bare `except:`) without re-raising traps `NodeInterruptedError` and breaks `RequestInput` HITL pauses, whereas broad `except Exception:` inside ADK 2.0 tools or nodes masks failures from `RetryConfig(max_attempts=...)` automatic retries. Let transient `Exception`s propagate to the framework, catching only narrow recoverable domain errors when intentionally returning a structured result. **Planners & October 2026 `thinking_level` Matrix:** - **`BuiltInPlanner(thinking_config=types.ThinkingConfig(...))`:** Activates native Gemini System-2 reasoning (`part.thought == True` when `include_thoughts=True`) and preserves latent **Thought Signatures** across multi-turn tool loops. - **`PlanReActPlanner()`:** Prompts non-thinking or third-party models to emit explicit `/*PLANNING*/` → `/*ACTION*/` → `/*REASONING*/` → `/*FINAL_ANSWER*/` scaffolding tags. - **`thinking_level` (`low` | `medium` | `high` | `minimal`) vs. legacy `thinking_budget`:** `thinking_level` is **mandatory on Gemini 4 Argon** and **recommended across Gemini 3.x** (whereas integer `thinking_budget` is a legacy Gemini 2.5 parameter). | `thinking_level` | **`gemini-4-argon`** | **`gemini-3.8-flash`** | **`gemini-3.5-flash-lite`** | Architectural sweet spot | |---|---|---|---|---| | **`"minimal"`** | ❌ **HTTP 400 Error** | ❌ **HTTP 400 Error** | ✅ **Supported (Default)** | Ultra-low-latency intent routers, schema extractors, inline classifiers | | **`"low"`** | ✅ Supported | ✅ Supported | ✅ Supported | Routine MCP tool execution, straightforward RAG synthesis, cost-sensitive turns | | **`"medium"`** | ✅ **Supported (Default)** | ✅ **Supported (Default)** | ✅ Supported | Balanced production default for multi-step agentic tool loops | | **`"high"`** | ✅ Supported (Dynamic) | ✅ Supported (Dynamic) | ✅ Supported (Dynamic) | Ambiguous planning, repository-scale code synthesis, conflict resolution (higher TTFT) | > **Common trap:** Passing `thinking_level="minimal"` on **`gemini-4-argon`** or **`gemini-3.8-flash`**, OR passing legacy integer `thinking_budget=...` on **`gemini-4-argon`**, fails immediately with **HTTP `400 INVALID_ARGUMENT`**. Note also that `gemini-4-argon` ignores custom `temperature`, `top_p`, and `top_k` values—control reasoning depth and determinism via `thinking_level`, structured schemas, and explicit rubric instructions. --- ## 6. State, sessions and memory | Storage tier | Data shape | Lifecycle scope | ADK interface | |---|---|---|---| | **Session events** | Chronological immutable turn & tool transcript | Single conversation thread | `SessionService` (`session.events`) | | **Scoped state** | Exact structured key/value scratchpad | Invocation, session, user, or app (by prefix) | `output_key`, `tool_context.state`, `EventActions(state_delta=...)` | | **Semantic memory** | LLM-distilled, consolidated user facts retrieved by relevance | Cross-session per `user_id` | `MemoryService` (`VertexAiMemoryBankService`) + `load_memory` / `PreloadMemoryTool` | | **Artifacts** | Versioned binary blobs (PDFs, images, CSVs; `"user:..."` = cross-session) | Session or user | `ArtifactService` (`GcsArtifactService` in prod) | | State key prefix | Visibility scope | Survives process restart? | |---|---|---| | *(unprefixed)* | Current `session_id` only | Only with `DatabaseSessionService` (**Spanner**, **AlloyDB**, **Cloud SQL**) or `VertexAiSessionService` | | `user:` | All sessions belonging to `user_id` under `app_name` | Only with `DatabaseSessionService` (**Spanner**, **AlloyDB**, **Cloud SQL**) or `VertexAiSessionService` | | `app:` | Global across all users/sessions of `app_name` | Only with `DatabaseSessionService` (**Spanner**, **AlloyDB**, **Cloud SQL**) or `VertexAiSessionService` | | `temp:` | Current single invocation (shared across sub-agents in that turn) | **Never persisted** (stripped prior to commit) | - **Memory Bank vs. RAG Memory vs. Enterprise Grounding:** - **`VertexAiMemoryBankService` (Managed Memory Bank):** Runs LLM fact extraction and **contradiction consolidation** (`await memory_service.add_session_to_memory(session)` -> `GenerateMemories`, typically triggered in `after_agent_callback`). If a user changes preferences in Session 5, Memory Bank revises the stored memory rather than returning stale facts (`add_memory` / `CreateMemory` writes verbatim entries unless `enable_consolidation=True`). - **`VertexAiRagMemoryService`:** Indexes raw session chunks into a RAG corpus (`similarity_top_k`, `vector_distance_threshold` where **smaller = closer**); does **not** reconcile conflicting historical statements. - **Shared enterprise grounding (not per-user memory):** Use `VertexAiSearchTool` (**Agent Search** with native SharePoint/Drive ACL propagation), `VertexAiRagRetrieval` (**RAG Engine**), or `google_search`. - **`load_memory` vs. `PreloadMemoryTool`:** `load_memory` lets the model query memory on demand (cheaper, risk of skipped recall); `PreloadMemoryTool` injects relevant memories automatically at the start of every turn (consistent personalization at higher per-turn latency/tokens). **State Hygiene & Backing Store Quotas:** | Sensitive or oversized payload | Why it must NEVER enter `session.state` or Memory Bank | Mandated Google Cloud store & ADK pattern | |---|---|---| | **OAuth 3LO / 2LO tokens** | Plaintext tokens leak into `session.events`, traces, and model prompts | **Auth Manager** token vault (`GcpAuthProvider` + regional `GcpAuthProviderScheme` on `locations//authProviders/...` emitting `adk_request_credential` with PKCE, `continue_uri`, and auto-generated anti-fixation `consent_nonce` verified on `:finalize`) | | **Static API keys, DB passwords & SA keys** | Triggers Security Command Center secret findings and audit exposure | **Secret Manager** (or Auth Manager API Key provider) injected at the tool/proxy boundary | | **Unredacted PII / PHI / PCI** | Persists across session history and distills into cross-session memories | **Sensitive Data Protection (SDP)** deterministic surrogates (`CryptoDeterministicConfig`) via **Model Armor** or `before_model_callback` / `after_tool_callback` | | **Multi-MB binary blobs (PDFs, images, CSVs)** | Bloats JSON state payloads and hits document/gateway size limits | **`GcsArtifactService`** (`{app}/{user}/{session}/{filename}/{version}` or `"user:..."`); leaving `Runner(artifact_service=None)` raises **`ValueError`** on `save_artifact()` | ```python jira_auth = GcpAuthProviderScheme( name="projects/my-sec-proj/locations/us-central1/authProviders/jira-3lo", # Regional-only (never global) scopes=["read:jira-work"], # Minimized read-only scope (never broad admin scopes) continue_uri="https://app.internal.example/oauth/callback", # Handler posts consentNonce to :finalize ) ``` - **Firestore vs. Memorystore persistence boundaries:** - **Firestore:** 99.999% multi-region strong consistency and document TTLs, with a hard **`1 MiB` max document size** and ~**`1` sustained write/sec per document** (persist turn events in **subcollections**, never by appending unbounded arrays inside a single parent document). - **Memorystore for Redis / Valkey:** **`< 1 ms`** HNSW/FLAT semantic cache, sliding-window RPM/TPM rate limiters, and **Pub/Sub `messageId` idempotency keys** (scales to 250 nodes / 25+ TB), with mutually exclusive `RDB` snapshot or `AOF` (`appendfsync`) persistence and async replication (still an in-memory tier; never use as a sole compliance audit ledger). > **Common trap:** Mutating `session.state["user:plan"] = "pro"` directly on the object returned by `await session_service.get_session(...)`. That mutates a detached snapshot, skips `append_event()`, leaves zero audit trail, and drops the write in Spanner / AlloyDB / PostgreSQL / Agent Runtime. Always mutate state through `tool_context.state`, `callback_context.state`, `output_key`, or `EventActions(state_delta={...})`. --- ## 7. Tools and structured output Passing a callable in `tools=[lookup_invoice]` wraps it as a `FunctionTool` whose **docstring** defines when the LLM invokes it and whose **type annotations** compile into the JSON Schema. Poor docstrings directly cause tool-selection hallucinations. | Tool design rule | Architectural reason | Implementation pattern | |---|---|---| | **Atomic single responsibility** | Monolithic tools hide unobservable logic and bloat failure blast radius | Separate `crm_get_customer` from `crm_issue_refund` | | **Namespace prefixes** | Prevents collisions when combining multiple MCP servers and toolsets | `jira_create_issue`, `bq_execute_sql` | | **Document constraints & units** | Models plan accurately when limits are explicit in the schema description | `"Read-only SQL; max 500 rows; 15 s timeout; dates in ISO-8601."` | | **Return structured errors, don't raise** | Uncaught exceptions abort the turn; error dicts let the LLM self-correct or escalate | `{"status": "error", "error_code": "NOT_FOUND", "retryable": False}` | | **Enforce security inside tool code** | Prompts cannot act as security boundaries against injection | Canonical path `resolve().is_relative_to(ROOT)` + `WriteMode.BLOCKED` | | **Idempotent mutations** | `ResumabilityConfig` and transport retries have at-least-once semantics | Require `idempotency_key` before charging cards or creating tickets | | **Cap active tools (~16 max)** | Tool selection precision degrades sharply when schemas crowd context | Partition across specialist `AgentTool`s or progressive `SkillToolset`s | | Enterprise integration layer | ADK class & transport | Target backend & primary strengths | Operational & auth constraint | |---|---|---|---| | **Integration Connectors + Application Integration** (`GA`) | `ApplicationIntegrationToolset` | **100+ SaaS & ERP systems** (SAP, Salesforce, ServiceNow, Workday) with schema mapping | Pair with **Auth Manager 3LO** for user-scoped mutations; higher hop latency/cost than direct DB pooling | | **MCP Toolbox for Databases** (`GA`) | `ToolboxToolset` / `McpToolset` (`googleapis/genai-toolbox`) | **30+ relational, graph & NoSQL DBs** via parameterized `tools.yaml` SQL/Cypher queries | Sub-50 ms connection pooling, IAM auth, and OTel spans; no SaaS/ERP entity adapters | | **Apigee + API Hub** (`GA`) | `APIHubToolset` | **Cataloged OpenAPI REST microservices** with rate/token quotas, mTLS, semantic caching, and Model Armor | Filter operations explicitly rather than importing hundreds of raw endpoints onto one agent | | **Custom Remote MCP on Cloud Run** (`GA`) | `McpToolset` + `StreamableHTTPConnectionParams` | **Bespoke domain tools** deployed with `--functional-type=mcp-server` and `--no-allow-unauthenticated` behind IAP | Send Cloud Run OIDC in **`X-Serverless-Authorization`** when forwarding a user `Authorization` OAuth token, or delegate to **Agent Gateway + Auth Manager** | - **Built-in tool mixing rules across SDK languages:** Gemini server-side tools (`google_search`, `VertexAiSearchTool`, `BuiltInCodeExecutor`) execute inside the model API call. - In **ADK Python (v1.16+)**, set `bypass_multi_tools_limit=True` on `GoogleSearchTool` or `VertexAiSearchTool` to mix them alongside custom `FunctionTool`s. - Combining **`BuiltInCodeExecutor`** with custom tools (in Python or TypeScript), OR combining **`GoogleSearchTool` with custom tools in Java/Kotlin**, is rejected unless you isolate each built-in tool inside its own specialist `LlmAgent` wrapped in **`AgentTool(agent=...)`**. - **Additional native ADK toolsets:** `BigQueryToolset` (`BigQueryToolConfig(write_mode=WriteMode.BLOCKED)`), `SpannerToolset`, `OpenAPIToolset`, and `SkillToolset`. - **Structured output (`output_schema`):** Pass a Pydantic model (`output_schema=TriageVerdict`) with `output_key="verdict"` for schema-guaranteed JSON. When `output_schema` conflicts with multi-turn tool calling on a given model/SDK configuration, decouple them in a 2-step `SequentialAgent` / `Workflow`: Step 1 runs tools and writes raw findings to state; Step 2 runs a stateless formatter (`output_schema=TriageVerdict`, `include_contents="none"`). --- ## 8. MCP, A2A and Agent Registry integration **MCP is vertical (`Agent → Tools/Resources`); A2A is horizontal (`Agent → Peer Agent`).** Enterprise architectures combine both under **Agent Registry** governance. ```python # Vertical: remote Streamable HTTP MCP server with explicit allowlist & namespace prefix erp_mcp = McpToolset( connection_params=StreamableHTTPConnectionParams(url="https://erp-mcp.internal.example/mcp"), tool_filter=["get_order", "list_shipments"], # Exact list or ReadonlyContext predicate (never globs) tool_name_prefix="erp", ) # Horizontal: cross-org or cross-language specialist discovered via A2A Agent Card customs_agent = RemoteA2aAgent( name="customs_clearance_agent", agent_card="https://partner.example/.well-known/agent-card.json", ) ``` | Integration layer | Production rules & Google Cloud integration | |---|---| | **MCP Transports & Lifecycle** | Use **`StdioConnectionParams`** only for local dev subprocesses; use **`StreamableHTTPConnectionParams`** for remote stateless/stateful production services (legacy SSE-only transport is deprecated). Instantiate `McpToolset` **synchronously at module scope** in `agent.py` for `adk deploy`, and call `await toolset.close()` on custom FastAPI shutdown. | | **Google Cloud Managed MCP & Registry** | Discover governed enterprise tools and agents through **Agent Registry** (`AgentRegistry`), first-party remote `StreamableHTTP` MCP servers auto-registered in `global` Registry (`bigquery.googleapis.com/mcp`, `container.googleapis.com/mcp`, `compute.googleapis.com/mcp`, `run.googleapis.com/mcp`, `logging.googleapis.com/mcp`, `developerknowledge.googleapis.com/mcp` via `private.googleapis.com` in VPC-SC, and Agent Retrieval MCP), and **MCP Toolbox for Databases** (`ToolboxToolset`). Authorize calls using **Agent Identity** (`principal://...`) and **Auth Manager** OAuth 2.0 tokens. | | **A2A Consumption & Publishing** | Consume peers via `RemoteA2aAgent` (supports Go/Java/Python/LangGraph interoperability); expose an ADK agent via `to_a2a(agent, port=8001)` or `adk api_server --a2a`. Supports JSON-RPC 2.0 / gRPC / HTTP+JSON, SSE streaming (`message/stream`), and async webhook push notifications across task states (`submitted` → `working` → `input-required` / `auth-required` → `completed` / `failed` / `canceled` / `rejected`). | | **Adjacent Agentic Protocols** | **A2UI:** declarative JSON UI payloads rendered only from the client's trusted component catalog. **AP2:** A2A payment extension enforcing cryptographically signed **Intent → Cart → Payment** mandates. | > **Common trap:** Assuming local ADK `before_tool_callback` hooks inspect a `RemoteA2aAgent`'s internal database queries. Remote A2A agents are **opaque peers** executing outside your process—never expose internal prompts/tools over A2A, authenticate via the card's `securitySchemes`, and validate every returned A2A artifact at your boundary. ### Platform path: expose, deploy, register and discover | Step | Action | Documented mechanism | |---|---|---| | **1. Expose** | Serve the ADK agent over A2A with an Agent Card | `to_a2a(root_agent, port=...)` builds the card from the agent's metadata and skills and publishes it at `/.well-known/agent-card.json` once served with `uvicorn`; pass `agent_card=` (an `AgentCard` object or a JSON file path) to control what is advertised. `adk api_server --a2a` exposes only agent folders that contain an `agent.json` card. | | **2. Deploy** | Host on Agent Runtime with a per-agent identity | `agents-cli deploy --agent-identity`; or `adk deploy agent_engine` with `.agent_engine_config.json` set to `{"identity_type": "AGENT_IDENTITY"}`; or the Agent Platform SDK `client.runtimes.create(...)` with `identity_type` in the config. A2A-native hosting on Agent Runtime uses the `A2aAgent` template (`AgentCard` + `AgentExecutor`), currently Preview. | | **3. Register** | Make the agent discoverable in Agent Registry | Automatic or manual registration (next table). | | **4. Discover** | Resolve peers at runtime instead of hardcoding URLs | ADK `AgentRegistry` client (`google-adk[a2a,agent-identity,mcp]`, version `1.29.0` or later): `get_remote_a2a_agent()` returns a remote agent you can attach as a sub-agent; `get_mcp_toolset()` returns a registered MCP server as a toolset. | | **5. Authorize** | Allow only approved caller-to-peer paths | Agent policies at Agent Gateway (see the next subsection). | | Where the agent runs | Registration mode | How it happens | What gets indexed | |---|---|---|---| | **Agent Runtime** | automatic, on by default | deploying registers it; later updates and deletions sync | A2A skills read from the Agent Card when the agent speaks A2A | | **Cloud Run** | automatic, opt-in | deploy with `--functional-type=agent`, which requires `--identity-type=agent-identity` | agent name and `AGENT` type, plus card skills for A2A agents | | **GKE** | automatic, opt-in | label `registry.gke.io/functional-type: "AGENT"` plus an `a2a-protocol.org/agent-card` annotation on the Deployment | card skills from the annotated endpoint | | **Built-in Google agents** (Google Workspace, Gemini Enterprise) | automatic | no action | ingested by the platform | | **External clouds, on-premises, unsupported runtimes, other projects outside a management boundary** | manual | console, `gcloud agent-registry services create --agent-spec-type=a2a-agent-card --agent-spec-content=agent-card.json`, or Terraform `google_agent_registry_service`; needs `roles/agentregistry.editor` on the governance project | `A2A_AGENT_CARD` agents get their card synced; plain REST agents use `NO_SPEC` and carry no skills | - **Discovery scope:** automatic registration only sees the governance project unless you configure a management boundary across projects; everything else must be registered manually. - **Non-A2A agents:** the registry records them but cannot capture endpoint, identity or skills on its own; add those through manual registration or skill registration. - **Location limit:** manual registration is not available in the `us` and `eu` multi-regions; use a single region or `global`. ```python from google.adk.a2a.utils.agent_to_a2a import to_a2a # Card is generated from the agent; port must match the uvicorn --port value # or the card advertises an unreachable URL a2a_app = to_a2a(root_agent, port=8001) ``` ```python from google.adk.integrations.agent_registry import AgentRegistry registry = AgentRegistry(project_id=GOVERNANCE_PROJECT, location="global") # Resolve once at startup, not on every invocation fx_agent = registry.get_remote_a2a_agent( agent_name="agents/AGENT_ID", # short form inherits project and location httpx_client=authed_httpx_client, # attaches the caller's credentials ) coordinator = LlmAgent( name="treasury_coordinator", model="gemini-3.8-flash", instruction="Send currency questions to the FX specialist.", sub_agents=[fx_agent], # fx_agent.clone() for a second parent ) ``` - **Startup resolution & `.clone()` across branches:** resolve `AgentRegistry.get_remote_a2a_agent()` and `get_mcp_toolset()` once at startup rather than on every turn; because every ADK agent instance obeys the single-parent rule, call `.clone()` whenever reusing a resolved remote A2A agent across multiple workflow branches or parent orchestrators. - **A2A large binary payload handling:** pass Cloud Storage `gs://` URIs via `GcsArtifactService` rather than inline base64 A2A `Part` blobs, which bloat JSON-RPC payloads and choke Agent Gateway `CONTENT_AUTHZ` inspection. - **Anti-pattern (hardcoded peer URLs):** pasting a partner's card URL into `RemoteA2aAgent` skips the registry, so endpoint changes break silently and the gateway cannot apply per-resource rules to that peer. ### Cross-agent authorization with agent policies Agent policies are the controls that Agent Gateway enforces on Agent-to-Anywhere (egress) traffic. Each control answers a different question, and a call proceeds only when all of them pass. ``` coordinator on Agent Runtime (principal A) │ A2A request, mTLS + certificate-bound token ▼ Agent Gateway, Agent-to-Anywhere mode 1. IAP: UAP grants A egress to the target? PAB includes the target? 2. Agent Registry: target registered, or an explicit rule for its URL? 3. Inspection: Model Armor, semantic governance, custom authz engine ▼ specialist on Agent Runtime (principal B, its own egress policies) ``` | Control | Question it answers | How it is enforced | Exam detail & quotas | |---|---|---|---| | **Agent Identity** (GA) | Who is calling? | SPIFFE-based `principal://` identifier per agent, tied to the Agent Runtime resource; X.509 certificate and certificate-bound tokens (~24 h rotation) checked with mTLS and DPoP under a Google-managed Context-Aware Access policy | not shared across workloads by default, cannot be impersonated, no long-lived keys | | **Agent Registry check** | Is the destination approved? | gateway confirms the target is registered (`destination.agent_registry.agent.name` / `.mcp_server.name`); an unregistered target needs an explicit access rule on `destination.unregistered.host` / `.path` | no matching policy means the request is denied (`roles/agentregistry.viewer` grants discovery only, not invocation) | | **IAM Unified Access Policies** (via IAP, `REQUEST_AUTHZ`) | May principal A reach this agent, tool or endpoint? | allow/deny rules (`roles/iap.egressor` / `iap.googleapis.com/resources.egressViaIAP`) plus CEL keys `api.getAttribute('iap.googleapis.com/mcp.toolName', '')`, `api.getAttribute('iap.googleapis.com/mcp.tool.isReadOnly', false)`, and `destination.agent_registry.*` | **max `4` custom `AuthzPolicy` per egress gateway**; deny overrides allow; dry-run via `iamEnforcementMode: "DRY_RUN"` (enforce by removing field); match `global` vs. regional registry location to avoid `403 IAP_AUTHZ = DENIED` | | **Principal Access Boundary** (GA) | What is the most A could ever reach? | PAB policy bound to the Agent Identity principal set (**max `500` rules per policy**, **`500` resources per rule**), evaluated before allow policies | acts strictly as an identity ceiling (never grants access alone); an over-granted allow rule still cannot reach resources outside the boundary | | **Model Armor** (`CONTENT_AUTHZ`) | Is the payload safe? | inline `FULL_DUPLEX_STREAMED` content filter on prompts, tool responses, and the **first `256` URIs** per payload (consumes quota in the template's project) | dry-run via `INSPECT_ONLY` → enforce via `INSPECT_AND_BLOCK`; org/folder Floor Settings enforce a minimum floor that project templates cannot weaken | | **Semantic governance policies** (Preview, `CONTENT_AUTHZ`) | Does this tool call match the user's intent and business rules? | intent-aware LLM policy firewall evaluating chat history, tool calls, and `load_skill` actions against Natural Language Constraints (**max `5,000` chars per NLC**; **max `1,000` chars per tool description**) | **strict deny-overrides** (`any DENY` blocks); dry-run via `sgpEnforcementMode: "DRY_RUN"` (enforce by removing flag); scope CEL predicates away from raw gRPC health traffic; supplements IAM, never replaces it | - **Availability:** the Google-managed gateway covers Agent Runtime and Gemini Enterprise; GKE, Cloud Run and cross-cloud deployment models are listed as early preview. - **Anti-pattern (enforcing on day one):** enforcing a new egress policy without a dry run can cut off working handoffs in production; validate in `DRY_RUN` against audit logs first. - **Anti-pattern (semantic governance as the only gate):** relying on natural-language constraints for access control leaves a probabilistic check as the sole barrier; keep deterministic UAP deny rules and PAB underneath it. ### Handoff controls and approval pauses by pattern The orchestration-boundary table in section 4 picks the boundary. This table shows what each choice gives you at the handoff itself: whose identity the next agent uses, whether the registry knows about it, what can sit between the two agents and where a person can stop the flow. | Pattern | Where it runs | Sub-agent identity | Agent Registry discovery | Controls between agents | Where a human approval pauses it | |---|---|---|---|---|---| | **Sequential** (`SequentialAgent`) | inside the parent's process, children one after another in the same invocation | the deployment's single principal, shared by every child | children are not registered; only the deployed agent as a whole is | `output_key` and `{key}` templates carry data; `before_agent_callback` can gate the next child | ADK tool confirmation inside a child's tool (`require_confirmation` or `tool_context.request_confirmation(hint=..., payload=...)`) | | **Parallel** (`ParallelAgent`) | same process; branches start concurrently and see no history or state from sibling branches while running | same single principal | same as sequential | one distinct `output_key` per branch, then a merge step that reads them after all branches finish | same tool-level confirmation, raised inside a branch's tool | | **Graph workflow** (`Workflow(edges=[...])`) | same process (Python, TypeScript, Go) | same single principal | nodes are not registered | typed `node_input`, route keys on edges, and plain function nodes that validate or redact before the next agent node | a node yields `RequestInput(message=..., payload=..., response_schema=...)` (from `google.adk.events`); the reply becomes the next node's input, and `rerun_on_resume` decides whether the pausing node runs again | | **Remote A2A** (`RemoteA2aAgent` to a peer on Agent Runtime) | separate service, possibly another project or organization | the peer's own Agent Identity principal | `AgentRegistry.get_remote_a2a_agent()`; Agent Runtime peers register automatically, others manually from their card | Agent Gateway policies (registry check, UAP, PAB, Model Armor, semantic governance) plus artifact validation at your boundary | the peer returns its task in `TASK_STATE_INPUT_REQUIRED` (v0.3 JSON: `input-required`), an interrupted state that is not terminal; the caller answers with another message on the same task | - **Tool confirmation limit:** ADK documents that tool confirmation does not support `DatabaseSessionService` or `VertexAiSessionService`; for durable approvals use `LongRunningFunctionTool` or a graph `RequestInput` node (section 11). - **Confirmation with resume:** when resumability is on, the confirmation response must carry the same `invocation_id` that raised the request, otherwise ADK starts a new invocation. - **Confirmation vs. RequestInput:** tool confirmation lives inside an `LlmAgent`'s tool call and asks for a yes/no answer or a small payload; `RequestInput` is a standalone graph node that can ask for structured data against a schema. - **Detective control after the handoff:** Agent Anomaly Detection (Preview, allowlisted access) reads OpenTelemetry logs and traces asynchronously, so it adds no latency and never blocks a call. Its categories include agentic cascading failures (OWASP ASI08: fault propagation, infinite loops, oscillating retries), tool misuse (ASI02), identity privilege abuse (ASI03) and rogue agents (ASI10), with findings sent to Security Command Center. - **Anti-pattern (anomaly detection as the loop breaker):** an asynchronous detector reports a ping-pong handoff after it happened; the hard stops remain `RunConfig(max_llm_calls=...)`, `LoopAgent(max_iterations=...)` and hub-and-spoke transfer rules. --- ## 9. Skills, Skill Registry and Agents CLI A **tool** executes an action; a **skill** packages domain procedural knowledge (`SKILL.md` instructions + reference docs + helper scripts) using **3-tier progressive disclosure** so a 50-skill agent doesn't burn 80,000 prompt tokens on turn 1. | Disclosure tier | Payload loaded into context | Triggered by ADK tool | |---|---|---| | **L1: Routing metadata** | YAML frontmatter `name` (kebab-case matching folder) + trigger-phrased `description` (~60 tokens) | `list_skills` / `search_skills` (or eager catalog summary) | | **L2: Playbook body** | Complete `SKILL.md` workflow instructions (< 500 lines) | `load_skill` | | **L3: Deep assets & scripts** | `references/`, `assets/`, and executable `scripts/` | `load_skill_resource`, `run_skill_script` (requires sandboxed `code_executor`) | - **Wiring local skills & `GCPSkillRegistry`:** Load local folders via `load_skill_from_dir(Path("./skills/incident-triage"))` into `SkillToolset(skills=[...], code_executor=AgentEngineSandboxCodeExecutor())`, or attach **`GCPSkillRegistry`** (**Agent Platform Skill Registry**) so agents dynamically discover versioned, organization-approved skills via `search_skills` without redeployment. - **Configuring Skills & Plugins with `Agents CLI`:** - **Skills & plugins management:** Install reusable skills (`npx skills add google/agents-cli` or `agents-cli` skill/plugin commands) and scaffold telemetry, Terraform, CI/CD, and security plugins across ADK workspaces. - **Human mode vs. Agent mode:** In **human mode** (interactive TTY), `agents-cli` renders rich interactive wizards, colorized diffs, and confirmation prompts for developers. When invoked by autonomous coding harnesses (**Google Antigravity**, **Claude Code**, or CI runners), run `agents-cli` in **agent / non-interactive mode** (explicit CLI flags, no interactive stdin prompts, deterministic exit codes, and machine-readable JSON output) so automated workflows never hang waiting on terminal input. Same rule applies to L3 `SKILL.md` scripts: **non-interactive flags/env vars only**, `--help` support, and structured JSON on `stdout`. > **Common trap:** Treating an unreviewed third-party `SKILL.md` as harmless documentation. Skill instructions enter the model's privileged context and `run_skill_script` executes code—sync skills only from governed **Skill Registry** or code-reviewed repos, and never attach `UnsafeLocalCodeExecutor` to `SkillToolset`. --- ## 10. Callbacks, plugins and guardrails **Deterministic harness principle:** Never rely on system-prompt wording (`"Please redact SSNs and never refund over $500"`) for security, compliance, or budget caps. Enforce hard invariants in deterministic ADK callbacks and `BasePlugin`s surrounding the model. | Lifecycle hook | Return `None` | Return a typed override object | |---|---|---| | `before_agent_callback(callback_context)` | Agent executes normally | `types.Content` → **skips agent run**; returns `Content` as the reply | | `after_agent_callback(callback_context)` | Final output kept as-is | `types.Content` → **appends** an additional event (does **not** overwrite reply) | | `before_model_callback(callback_context, llm_request)` | Calls LLM (`llm_request` mutable in place) | `LlmResponse` → **skips LLM call** (cache hit, fast policy block) | | `after_model_callback(callback_context, llm_response)` | Keeps LLM output | `LlmResponse` → **replaces** model response (final output redaction/disclaimer) | | `on_model_error_callback(callback_context, llm_request, error)` | Re-raises exception | `LlmResponse` → **suppresses error** and substitutes fallback (`after_model_callback` still runs) | | `before_tool_callback(tool, args, tool_context)` | Executes tool (`args` mutable in place) | `dict` → **skips tool execution**; returns `dict` to the model as the tool result | | `after_tool_callback(tool, args, tool_context, tool_response)` | Keeps raw tool result | `dict` → **replaces tool result** before it enters `session.events` or LLM context | | `on_tool_error_callback(tool, args, tool_context, error)` | Re-raises tool exception | `dict` → **suppresses exception** and returns recovery `dict` (`after_tool_callback` still runs) | - **App-wide `BasePlugin` precedence:** Register cross-cutting guardrails (**Model Armor** screening, **Sensitive Data Protection** PII redaction, token budgets, audit logging) once on `App(name="...", root_agent=..., plugins=[...])`. **Plugin callbacks always execute before agent-level callbacks**, and the first hook returning a non-`None` override short-circuits remaining hooks in that phase. - **Defense-in-depth ordering:** Screen user prompts in `before_model_callback` (fast regex → lightweight classifier / **Model Armor**) → validate tool parameters & spending caps in `before_tool_callback` → strip indirect prompt injections and **redact PII in `after_tool_callback`** → keep `after_model_callback` as the final outbound safety net. > **Common trap:** Scrubbing PII exclusively in `after_model_callback`. By that stage, unredacted customer records returned by a tool have already been committed to `session.events`, exported to traces (if content capture is on), and transmitted to the model context. Always scrub at **`after_tool_callback`** and **`before_tool_callback`** first. --- ## 11. Human-in-the-loop, resilience and durability | Operational requirement | ADK implementation pattern | |---|---| | Conditional approval on sensitive tools | `FunctionTool(fn, require_confirmation=lambda amount, **_: amount > 500)` (note: built-in `require_confirmation` is **`InMemorySessionService`-only**) | | Production durable HITL across DB / Agent Runtime sessions | **`LongRunningFunctionTool`** (returns pending ticket/status immediately; external approval webhook resumes with matching `function_call.id`) or ADK 2.0 `Workflow` generator yielding `RequestInput(message=..., response_schema=...)` | | Crash-resilient invocation replay | `App(..., resumability_config=ResumabilityConfig(is_resumable=True))` resumed with `invocation_id` (**at-least-once** execution—tools must be idempotent!) | | Runaway reasoning loop protection | `RunConfig(max_llm_calls=...)` (default **500**), `LoopAgent(max_iterations=...)`, plus per-session failure counters in `state` | **3-Layer Fault Recovery Ladder:** 1. **Transport layer (HTTP `429` / `503`):** Configure `HttpRetryOptions(attempts=3, initial_delay=1.0, exp_base=2.0)` on the `Gemini` model wrapper (and route stateless calls to the `global` endpoint or **Priority PayGo** / **Provisioned Throughput**). 2. **Tool argument self-correction:** Attach `ReflectAndRetryToolPlugin(max_retries=3, throw_exception_if_retry_exceeded=False)`. **Critical nuance:** out of the box it hooks raised exceptions (`on_tool_error_callback`); when your tools follow best practice and return `{"status": "error", ...}` dicts instead of raising, **override `extract_error_from_result`** on the plugin so structured error dicts trigger reflection. 3. **Model failover & graceful escalation:** Catch regional/model outages in `on_model_error_callback` (Python) or `RoutedLlm` (TypeScript) to fail over (e.g., `gemini-4-argon` → `gemini-3.8-flash`), and escalate deterministically when consecutive tool retries exhaust budget. --- ## 12. Context engineering, Gemini 4 Argon model routing and caching Stuffing raw multi-turn transcripts and 50 KB JSON payloads into every specialist degrades accuracy ("lost in the middle"), inflates p95 latency, and multiplies token spend ~15× across multi-agent trees. **3-Tier Agentic Model Routing:** | Routing tier | Recommended model & `thinking_level` | Workload & architectural role | |---|---|---| | **Tier 1: High-QPS Router & Guardrail** | **`gemini-3.5-flash-lite`** (`thinking_level="minimal"`) or fine-tuned **Gemma 4** SLM | Sub-200ms intent classification, slot extraction, schema normalization, and pre-LLM guardrail triage | | **Tier 2: Primary Agentic Workhorse** | **`gemini-3.8-flash`** (`thinking_level="low"` or `"medium"`) | Default execution loop for MCP tool calling, code generation, RAG synthesis, and `AgentTool` specialists | | **Tier 3: Frontier Planner & Deep Synthesis** | **`gemini-4-argon`** (`thinking_level="high"` or `"medium"`) | Ambiguous multi-step task decomposition, cross-document legal/financial conflict resolution, repo-scale refactoring, and unified 1M-token decodes | | **Real-Time Voice / Video Front-Door** | **`gemini-3.8-live`** (`runner.run_live`) + **`gemini-4-argon`** (`AgentTool`) | `gemini-4-argon` does not support Live API WebSockets; handle sub-second voice + barge-in on `3.8-live` and delegate heavy backend reasoning via `AgentTool` | **Context Compaction & Prefix Caching Mechanics (Gemini 4 Argon vs. Gemini 3.x):** | Mechanism | Configured on | How it works & version-specific rules | |---|---|---| | **`static_instruction` vs. `instruction`** | `LlmAgent` | Keeps unchanging system rules, rubrics, and tool schemas at a byte-identical prefix (`static_instruction`) while appending dynamic `{state_key}` values afterward (`instruction`) | | **Implicit Context Caching (Gemini 4 Argon & 3.x)** | Automatic (model API) | **`gemini-4-argon` uses implicit caching exclusively** (triggers automatically on shared prefixes `>= 4,096` tokens with zero hourly storage fee; **explicit `cachedContents` creation fails on `gemini-4-argon`**). Prefix stability via `static_instruction` is essential! | | **`ContextCacheConfig(cache_intervals, ttl_seconds, min_tokens)`** | `App` | Manages explicit TTL `cachedContents` handles on **Gemini 3.x / 2.5** base models when deterministic cache retention across sparse traffic justifies storage fees | | **`EventsCompactionConfig`** | `App` | Summarizes older `session.events` slices (`token_threshold=12000, event_retention_size=6` or `compaction_interval, overlap_size`) into compact summary events so long-running sessions avoid context degradation | | **`include_contents='none'` + `{state_key}`** | `LlmAgent` | Strips prior chat history completely for stateless extractors, formatters, and independent auditors | | **Observation masking (`after_tool_callback`)** | `LlmAgent` / `BasePlugin` | Trims verbose 40-column API responses down to the 4 needed fields before the event enters `session.events` | ```python app = App( name="support_hub", root_agent=root_agent, events_compaction_config=EventsCompactionConfig(token_threshold=12000, event_retention_size=6), ) ``` > **Common trap (prefix-cache invalidation):** Trying to provision explicit `cachedContents` (`ContextCacheConfig`) for **`gemini-4-argon`**, or injecting `{current_timestamp}` or `{user_id}` at line 1 of `instruction`. `gemini-4-argon` rejects explicit `cachedContents` and relies on **automatic implicit caching (`>= 4,096` tokens)**, while even on Gemini 3.x `ContextCacheConfig`, mutating the leading prompt prefix invalidates every cached token block (0% cache hit rate)! --- ## 13. Security and sandboxing | Threat vector | Attack scenario | Deterministic ADK & platform control | |---|---|---| | **Direct prompt injection** | User attempts to override system instructions | `before_model_callback` + **Vertex AI Model Armor** prompt screening | | **Indirect prompt injection & skill poisoning** | Malicious instructions hidden inside fetched web pages, PDFs, or dynamic `SKILL.md` loads | Sanitize untrusted payloads in `after_tool_callback`, isolate untrusted reader subagents (`include_contents="none"`, zero write tools), and enforce **Skill Governance (`govern-agent-skills`, Preview)** on `load_skill` at **Agent Gateway (`CONTENT_AUTHZ`)** | | **Confused deputy / excessive agency** | Read-only support bot tricked into invoking `delete_account` | Least-privilege `tool_filter` allowlist, `WriteMode.BLOCKED`, and per-agent **Agent Identity** | | **Cross-agent taint propagation** | Compromised sub-agent poisons downstream publisher | Enforce strict Pydantic `output_schema` + boundary validation callbacks between handoffs | | **Host container escape via generated code** | LLM-generated Python reads `/proc/self/environ` or metadata credentials | Never use `UnsafeLocalCodeExecutor` in prod; isolate in **Agent Runtime Code Execution** or **GKE Agent Sandbox (gVisor)** | | `code_executor` implementation | Isolation boundary & statefulness | Recommended environment | |---|---|---| | `BuiltInCodeExecutor` | Gemini server-side sandbox (stateless, 30 s max, zero network egress; cannot mix directly with custom tools on one agent) | Zero-infra math, data wrangling, charts | | `AgentEngineSandboxCodeExecutor` | Managed **Agent Runtime** isolated container (stateful files across turns, up to 14-day TTL, 100 MB I/O) | Default production choice on Google Cloud | | `GkeCodeExecutor(executor_type="sandbox")` | Pre-warmed **GKE Agent Sandbox** backed by **gVisor** kernel isolation (sub-second startup; replaces slow per-run K8s `"job"` mode) | GKE clusters needing custom VPC/hardware control | | `ContainerCodeExecutor` | Dedicated Docker container | Local staging or custom system binary testing | | `UnsafeLocalCodeExecutor` | Runs `exec()` inside the ADK host process with full IAM/env access | Throwaway local debugging only, **never** staging or prod | **Three identity authorities:** **(1) Google Cloud Agent Identity** (`principal://...` / `spiffe://...` per-agent cryptographic identity bound to **Principal Access Boundary [PAB]** policies and **Dual-Attribution Cloud Audit Logs**), **(2) Workload Service Account** (coarse infrastructure identity), and **(3) User OAuth 2.0 Delegation** (`ToolContext.request_credential` / **Auth Manager** `GcpAuthProviderScheme` when acting strictly under the end user's own entitlements). --- ## 14. Evaluation: ADK `evalset` vs. Gen AI Evaluation Service & observability **Core principle:** Grade **both** the final response and the **tool trajectory**. An agent that guesses the right refund figure while skipping mandatory `verify_return_window` compliance checks is a production defect. **Architectural Decision Matrix: Which Evaluation Engine Should You Use?** | Evaluation tier | Primary tool / API | Executes where & against what? | Key strengths & when to choose | |---|---|---|---| | **1. Deterministic Harness Replay** | `adk conformance record` + `adk conformance test --mode=replay` | Local / PR CI in `<5s` using recorded LLM responses (zero live model calls) | Verifies Python/Go/TS callback wiring, `state_delta` routing, and guardrail refactors with **zero token cost and zero sampling variance** on every commit | | **2. Local & CI Golden Regression** | `adk eval ` / `pytest` `AgentEvaluator.evaluate(...)` / `adk web` Eval tab | In-repo CI runner executing your ADK agent against `*.test.json` (unit) and `*.evalset.json` (multi-turn) | Fast developer inner-loop and PR merge gate grading `tool_trajectory_avg_score` (`EXACT`, `IN_ORDER`, `ANY_ORDER`), `final_response_match_v2`, `hallucinations_v1`, `UserSimulatorProvider`, and Python `custom_metrics` (`CodeExecutionMetric`) | | **3. Managed Cloud Fleet Evaluation & Simulation** | **Agent Platform Gen AI Evaluation Service** (`client.evals` / `EvalTask` / `agents-cli eval compare --fail-on-regression`) | Managed Google Cloud service grading **any** agent (ADK on Agent Runtime, Go/TS/Java Cloud Run endpoints, LangGraph, or custom callables) | **Choose when you need:** **Adaptive Rubrics**, `GROUNDING` vs. `HALLUCINATION` vs. `PairwiseMetric` (A/B order swap), **2-Step Agent Simulation** (from immutable **Agent Registry** snapshots), **Environment Simulation** (injecting HTTP 503/429/timeout tool faults safely), **Online Evaluation Monitors** (`evaluate-online`), **Auto-Loss Analysis** (`client.evals.generate_loss_clusters()`), or **`adk optimize`** (`optimize-agent`) | | ADK `test_config.json` criterion | What it grades | Rater type & exam nuances | |---|---|---| | `tool_trajectory_avg_score` | Predicted vs. reference tool sequence (`match_type`: `EXACT` default, `IN_ORDER` allows extra helper calls between required steps, `ANY_ORDER` for parallel lookups) | Deterministic (mirrors cloud `trajectory_exact_match` / `trajectory_in_order_match` / `trajectory_any_order_match` / `trajectory_precision` / `trajectory_recall`) | | `response_match_score` vs. `final_response_match_v2` | Lexical ROUGE-1 overlap vs. semantic equivalence | Deterministic vs. LLM judge (prefer `final_response_match_v2` whenever valid phrasing varies) | | `rubric_based_final_response_quality_v1` / `rubric_based_tool_use_quality_v1` | Domain-specific quality rubrics | LLM judge | | `hallucinations_v1` & `safety_v1` | Context faithfulness (catches invented numbers despite calling the right tool; requires tool-output spans in context to avoid false positives) & harmlessness | LLM judge (`safety_v1` / `safety_v3` delegates to Gen AI Eval Service) | | `multi_turn_task_success_v1` / `multi_turn_trajectory_quality_v1` / `per_turn_user_simulator_quality_v1` | End-to-end goal completion and red-teaming with `UserSimulator` (`UserPersona`, `starting_prompt`, `conversation_scenario`, and adversarial **`violation_rubrics`**) plus simulator persona-fidelity meta-scoring (`per_turn_user_simulator_quality_v1`) | LLM judge | | `tool_call_count_v1`, `token_usage_v1`, `invocation_duration_v1` | Resource efficiency telemetry | **`INFORMATIONAL` only**: setting thresholds on these in `test_config.json` throws an error; enforce hard call/cost gates via Python `custom_metrics` (`code_config.name` / `CodeExecutionMetric`) | **Observability, Loss Clustering & `adk optimize`:** - **OpenTelemetry GenAI spans:** ADK emits hierarchical spans (`invoke_agent` -> `call_llm` -> `execute_tool`). Export via `--otel_to_cloud` locally or `GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY=true` on Agent Runtime (powers the 8 **Agent Registry Observability** tabs: Overview, Evaluation, Models, Tools, Usage, Logs, Traces, Topology). - **Privacy-by-default:** Keep `ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS=false` (and `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` unset/false) in production unless paired with **Sensitive Data Protection** de-identification and restricted log buckets. - **Closed-loop flywheel & `adk optimize` (`optimize-agent`):** Export low-scoring **Online Evaluation Monitor** traces and human escalations -> cluster root causes via **`client.evals.generate_loss_clusters()`** -> de-identify PII with SDP (`ReplaceWithInfoTypeConfig` / `CryptoDeterministicConfig`) -> promote into **Example Store** and permanent `*.evalset.json` suites -> run **`adk optimize`** (GEPA data-driven prompt optimizer) with a strict **`train` vs. held-out `validation` split** so mutated instructions generalize without overfitting. > **Common trap:** Testing agent resilience against rate limits or ERP outages by firing tests against live staging/production APIs, or running `adk optimize` on a single unsplit test file (overfitting prompt edits to test questions). Always use **Environment Simulation** + **2-Step Agent Simulation** against an immutable **Agent Registry** snapshot and hold out a separate `validation` split during prompt optimization. --- ## 15. Deployment, live streaming and developer tooling | Deployment target | CLI workflow | Infrastructure you manage | Choose when | |---|---|---|---| | **Agent Runtime** (managed `reasoningEngines`) | `adk deploy agent_engine` or `agents-cli deploy` | Agent code & dependencies only (**Python** payload omits API server; **Go** payload bundles its dedicated ADK API server) | **Default on Google Cloud:** turnkey `VertexAiSessionService`, **Memory Bank**, **Code Execution** sandbox, **Agent Identity**, and **Gemini Enterprise** registration | | **Cloud Run** | `adk deploy cloud_run [--trigger_sources="pubsub,eventarc"]` | Container image, session DB (`DatabaseSessionService` / `agentengine://`), scaling | **TypeScript / Java / Kotlin** ADK containers, custom FastAPI/HTTP gateways, scale-to-zero, or **ambient event-driven agents** triggered by Pub/Sub & Eventarc | | **GKE** | `adk deploy gke` / custom Helm | Cluster, node pools, networking | Strict VPC/data-residency isolation, co-located self-hosted **Gemma 4** GPUs/TPUs, or **gVisor Agent Sandbox** (`GkeCodeExecutor`) | | Event-driven or async trigger | Delivery & timeout limits | Reliability & rate-control pattern | Best workload fit | |---|---|---|---| | **Pub/Sub Push / Eventarc → Cloud Run or Agent Runtime** | At-least-once delivery; **`600 s` (10 min) max ack deadline** | Enforce an **atomic `messageId` idempotency check** in **Memorystore** or **Firestore** before mutative tools, attach a **Dead-Letter Topic (`maxDeliveryAttempts >= 5`)**, and tune `container_concurrency` for async I/O | 1-to-N ambient event fan-out (GCS uploads, Cloud Audit events, CRM webhooks) | | **Cloud Tasks → Cloud Run / HTTP endpoint** | Point-to-point dispatch; **`30 min`** HTTP timeout | Throttle bursty workloads via **`maxDispatchesPerSecond`** and `scheduleTime` backoff (no 1-to-N topic fan-out) | Protecting fragile downstream ERPs and Gemini RPM/TPM quotas from spiky agent bursts | | **Cloud Run Jobs** | Headless batch execution; up to **`24 h`** max duration & `10,000` parallel tasks | Shard batch input in Cloud Storage / BigQuery with task-index checkpoints; zero interactive HTTP/SSE surface | Nightly corpus audits, bulk document triage, and offline batch agent evaluations | > **Platform deep dive:** See **Agent Platform: Build, Retrieve & Run** for capacity planning, regional compute exhaustion vs. project quotas, Provisioned Throughput GSUs, and Agent Registry governance. ```python runner = Runner(agent=agent, app_name="hub", session_service=session_service) msg = types.Content(role="user", parts=[types.Part(text=prompt)]) # Drain runner stream; filter out intermediate thought summaries (p.thought == True) async for event in runner.run_async(user_id="u_42", session_id=session.id, new_message=msg): if event.is_final_response() and event.content and event.content.parts: final_answer = "".join( p.text or "" for p in event.content.parts if not getattr(p, "thought", False) ) ``` - **Real-time bidirectional voice & video (`runner.run_live`):** Open low-latency **Gemini 3.8 Live (`gemini-3.8-live`)** sessions via `runner.run_live(user_id=..., session_id=..., live_request_queue=LiveRequestQueue(), run_config=RunConfig(response_modalities=["AUDIO"]))` (`send_realtime(Blob)` for PCM audio/video frames, halt playback immediately when `event.interrupted == True` on user barge-in). Remember: setting `StreamingMode.BIDI` on `runner.run_async()` is a **silent no-op** in Python ADK (`run_async` only supports `StreamingMode.SSE`), and **`gemini-4-argon` does not support Live API WebSockets**; pair `gemini-3.8-live` at the voice front-door with a `gemini-4-argon` `AgentTool` for deep backend reasoning. --- ## 16. Failure modes and exam traps | Symptom or production failure | Root cause → architectural fix | Tempting distractor (wrong) | |---|---|---| | `HTTP 400 INVALID_ARGUMENT` when configuring thinking on `gemini-4-argon` or `gemini-3.8-flash` | Passed `thinking_level="minimal"` (unsupported on `4-argon` & `3.8-flash`) or legacy integer `thinking_budget` on `4-argon` → use `thinking_level` in `{"low", "medium", "high"}` (reserve `"minimal"` for `gemini-3.5-flash-lite`) | Setting `temperature=0.0` to disable thinking | | Explicit `cachedContents` creation fails when migrating an ADK app to `gemini-4-argon` | `gemini-4-argon` supports **implicit caching only** (`>= 4,096` tokens, zero storage fee) → remove explicit cache creation and place static prompt content in `static_instruction` | Increasing `ContextCacheConfig.ttl_seconds` | | `ManagedAgent` raises `NotImplementedError` or regional connection error | Attached local Python `FunctionTool` / `McpToolset` or targeted `us-central1` → `ManagedAgent` supports **server-side tools only** and requires the **`global`** location on Agent Platform; keep local/MCP tools on a companion `LlmAgent` | Wrapping `McpToolset` in `StdioConnectionParams` | | Chaining `previous_interaction_id` on `Interactions API` (`background=True`) returns `400 Bad Request` | Previous background interaction is still `in_progress` → poll interaction status until `completed` before sending the follow-up turn | Resending the full chat history without `previous_interaction_id` | | Java/Kotlin ADK team tries to import ADK 2.0 `Workflow(edges=[...])` or mix `GoogleSearchTool` + custom tools on one agent | ADK 2.0 graph `Workflow` is currently **Python, TypeScript, and Go**; Java/Kotlin use ADK 1.x workflow agents (`SequentialAgent`/`ParallelAgent`/`LoopAgent`) and must isolate `GoogleSearchTool` inside an `AgentTool` specialist | Enabling `bypass_multi_tools_limit=True` in Java | | Coding agent hangs indefinitely running `agents-cli` or a `SKILL.md` helper script in CI | Executed in interactive human TTY mode waiting on stdin confirmation → invoke `agents-cli` and skill scripts in **non-interactive agent mode** with explicit CLI flags and JSON stdout | Extending the Cloud Build timeout to 60 minutes | | Cross-user data leak under load | State stored on `self` inside a singleton `Agent` → move all mutable state to `SessionService` (`app_name` / `user_id` / `session_id`) | Instantiating a new MCP server per HTTP request | | `LoopAgent(max_iterations=3)` finishes without converging, yet downstream `publisher` still runs | `LoopAgent` exits **silently** at `max_iterations` → set `state["temp:approved"] = True` in the exit tool alongside `escalate = True` and gate `publisher` via `before_agent_callback` | Catching `TimeoutError` around `LoopAgent` | | Specialist hijacks user conversation instead of returning a structured sub-result | Placed specialist in `sub_agents` (`mode="chat"` transfer) → wrap in `AgentTool(agent=...)` or use ADK 2.0 `mode="single_turn"` | Prompting `"Do not talk to the user"` | | `KeyError` before LLM call when instruction contains `{"id": 1}` | ADK parses `{id}` as a `session.state` placeholder → pass an `InstructionProvider` callable `def prompt(ctx: ReadonlyContext) -> str:` | Escaping braces as `{{id}}` | | Early `return` on `event.is_final_response()` returns step 1's draft or raw thoughts | Every sub-agent in a pipeline emits `is_final_response()` → drain the full `run_async` generator and filter `not getattr(p, "thought", False)` | Switching from `run_async` to `run_live` | | `ReflectAndRetryToolPlugin` ignores failing tools | Tools return `{"status": "error"}` dicts instead of raising exceptions → override `extract_error_from_result` on the plugin | Raising `max_retries` | | Built-in `require_confirmation=True` breaks after switching to `DatabaseSessionService` / `VertexAiSessionService` | Built-in tool confirmation is `InMemorySessionService`-only → use `LongRunningFunctionTool` or ADK 2.0 `RequestInput` | Switching SQLAlchemy drivers | | Raw SSNs appear in `session.events` and traces despite `after_model_callback` redaction | Tool results are persisted and sent to the LLM *before* `after_model_callback` → scrub PII at the tool boundary (`after_tool_callback` & `before_tool_callback`) and keep `ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS=false` | Strengthening system prompt privacy rules | | `adk eval` fails when setting a threshold on `tool_call_count_v1` or `token_usage_v1` | Built-in efficiency metrics are `INFORMATIONAL` only → enforce call/token budgets via Python `custom_metrics` (`code_config.name`) | Lowering threshold values to `0.5` | | Team wants to test 503 tool outages and auto-generate multi-turn personas across a deployed Go Cloud Run agent | `adk eval` is designed for local/CI golden files → use **Agent Platform Gen AI Evaluation Service** (**2-Step Agent Simulation** + **Environment Simulation** + **Adaptive Rubrics**) | Running chaos faults against live production ERP APIs | | Audit logs show one principal for every specialist, and a research sub-agent can reach the refund MCP server | Shared identity across sub-agents: every in-process `sub_agents` child runs under its deployment's single Agent Identity, so policy cannot separate them → deploy the privileged specialist as its own Agent Runtime agent with its own identity, call it over A2A, and scope UAP allow rules to that principal | Adding "only the billing agent may issue refunds" to the coordinator's instruction | | Coordinator's A2A call to a partner agent is denied at Agent Gateway even though the URL responds | Unregistered remote agent: the destination is not in Agent Registry and no access rule targets its URL, so the gateway denies by default → register it manually from its Agent Card, grant `iap.googleapis.com/resources.egressViaIAP` through a UAP, and resolve it with `get_remote_a2a_agent()` | Exporting a service account key so the coordinator can call the partner directly | | Two agents pass the same task back and forth until the LLM call budget runs out | Delegation loop: peers can hand work to each other with no termination rule → enforce hub-and-spoke (`disallow_transfer_to_peers=True`, specialists as `AgentTool` or `mode="single_turn"` returning to the coordinator), keep `RunConfig(max_llm_calls=...)` as a backstop, and add a UAP deny rule so the specialist's identity cannot call back into the coordinator | Raising `max_llm_calls` | | Handoffs to a partner agent start failing with connection or schema errors after the partner redeploys, while the coordinator's config has not changed | Stale Agent Card or endpoint drift: the peer URL or card is pinned in code, or a manual registry entry was never updated, so the coordinator works from an old snapshot → resolve the peer with `get_remote_a2a_agent()` (Agent Runtime registrations sync on update), update manual registrations from the new card in the partner's release pipeline, re-resolve on connection or schema errors, and compare the card's `version` and `protocol_version` | Raising `HttpRetryOptions(attempts=...)` | | Latency and token cost climb with every handoff, and specialists answer about topics from earlier in the conversation | Context bloat: `sub_agents` transfer shares the full `session.events`, so each specialist reads the whole transcript → call specialists through `AgentTool` (isolated child session) or set `include_contents="none"` and pass explicit state keys, enable `EventsCompactionConfig(compaction_interval=..., overlap_size=...)` on the `App` to summarize older events, and send a short task message over A2A instead of the transcript | Switching to a model with a larger context window | | One of several parallel researchers errors or hangs, and the merge step returns an incomplete answer or the whole turn fails | Partial branch failure in parallel fan-out: branches share no state, and the merger assumes every branch's key exists → have branch tools return structured error dicts with timeouts set in tool code, write one `output_key` per branch, make the merger handle missing or error keys and label the result as partial, and in a graph route on the failing node's output | Rewriting the fan-out as a `SequentialAgent` | | Remote tasks pile up unanswered, users never see the peer's question, and coordinator connections stay open | Orphaned `input-required` task: the peer returned `TASK_STATE_INPUT_REQUIRED`, which is interrupted but not terminal, and nothing surfaced or closed it → if the card declares the `pushNotifications` capability, register a webhook with `CreateTaskPushNotificationConfig` (`url`, `token`) instead of holding a stream open, store the task ID with a deadline in session state, and call `CancelTask` when it expires (the task ends in `TASK_STATE_CANCELLED` when the peer supports cancellation) | Raising the HTTP client timeout | | ADK 2.0 `Workflow` graph ignores `RequestInput` approval pauses or `RetryConfig` automatic retries never fire | Catching broad `BaseException` (or bare `except:`) traps `NodeInterruptedError` and breaks `RequestInput` pauses, while broad `except Exception:` inside tools/nodes masks errors from `RetryConfig(max_attempts=...)` → let `BaseException` and transient `Exception`s propagate, catching only specific recoverable domain errors | Replacing `RequestInput` with `require_confirmation=True` on `VertexAiSessionService` | | Eventarc / Pub/Sub-triggered Cloud Run agent issues duplicate refunds or ERP tickets on slow turns | Pub/Sub at-least-once delivery (`600 s` max ack deadline) redelivers unacknowledged events while the turn is still running → perform an atomic `messageId` idempotency check in Memorystore or Firestore before executing mutative tools, configure a Dead-Letter Topic (`maxDeliveryAttempts >= 5`), or throttle via Cloud Tasks (`maxDispatchesPerSecond`) | Setting Pub/Sub `ackDeadlineSeconds` to `3600` | | Multi-MB PDFs sent over A2A fail at Agent Gateway `CONTENT_AUTHZ`, or attaching `AgentRegistry.get_remote_a2a_agent()` to two workflow branches raises `ValueError` | Inline base64 A2A `Part` blobs saturate gateway payload inspection and every ADK agent instance obeys the single-parent rule → persist binaries in Cloud Storage via `GcsArtifactService` (`Runner` raises `ValueError` if `artifact_service=None`), pass `gs://` URIs across A2A, and resolve `get_remote_a2a_agent()` once at startup calling `.clone()` per additional branch or parent | Disabling `CONTENT_AUTHZ` on Agent Gateway and resolving the registry inside every turn | --- ## 17. Glossary - **`Runner`:** ADK's stateless execution engine (`run_async`, `run_live`) implementing the commit-then-resume event loop across `SessionService`, `ArtifactService`, and `MemoryService`. - **`LlmAgent` vs. `Workflow` (ADK 2.0):** probabilistic model-driven agent (`mode="chat"` or `mode="single_turn"`) vs. ADK 2.0's deterministic/conditional graph engine (`edges`, `BaseNode`, `RequestInput`, `RetryConfig`) across Python, TypeScript, and Go. - **`ManagedAgent`:** ADK wrapper around the Agent Platform Managed Agents API (`Interactions API` in `global`) supporting server-side tools only (`NotImplementedError` on client `FunctionTool` or `McpToolset`). - **`Session` / `Event` / `State`:** single-conversation append-only event stream and prefix-scoped scratchpad (`none` session, `user:`, `app:`, `temp:` current-invocation-only). - **`VertexAiMemoryBankService`:** managed cross-session long-term semantic memory service that asynchronously extracts, consolidates, and revises durable user facts from completed sessions. - **`GcsArtifactService`:** Cloud Storage-backed binary artifact store (`{app}/{user}/{session}/{filename}/{version}` and `user:` scope) used to keep multi-MB blobs out of `session.state` and inline A2A `Part` payloads. - **`AgentTool` vs. `sub_agents`:** explicit tool-like specialist invocation in an isolated child session vs. control-flow handoff (`transfer_to_agent`) sharing `session.events` under the single-parent rule (`.clone()` required to reuse an instance). - **`ApplicationIntegrationToolset` / `ToolboxToolset` / `APIHubToolset` / `McpToolset`:** ADK's four enterprise tool connectors for Integration Connectors (100+ SaaS/ERPs with Auth Manager 3LO), MCP Toolbox pooled SQL databases (`tools.yaml`), Apigee API Hub OpenAPI specs, and Streamable HTTP / stdio MCP servers. - **`RequestInput` & `NodeInterruptedError`:** ADK 2.0 durable human-in-the-loop pause primitive that raises a `BaseException` subclass to checkpoint graph execution cleanly across restarts. - **`before_tool_callback` / `after_tool_callback` & `BasePlugin`:** deterministic single-agent interception hooks and `Runner`-wide cross-cutting guardrails (such as `ReflectAndRetryToolPlugin` and tool-boundary SDP redaction). - **`AgentRegistry.get_remote_a2a_agent()`:** startup-time discovery helper resolving governed A2A Agent Cards from Agent Registry for cross-runtime delegation through Agent Gateway. - **`adk eval` vs. `agents-cli eval`:** local/CI trajectory grading on `*.evalset.json` files vs. non-interactive production dataset evaluation (`EvaluationDataset` + `eval_config.yaml`) wired into Agent Platform CI/CD pipelines. ================================================================================ GUIDE: Model Context Protocol (MCP) URL: https://mbettan.github.io/machine-learning-cert/MCP.html Practice quiz: https://mbettan.github.io/machine-learning-cert/quiz.html?topic=mcp Section: Foundation · Agent Architecture & Protocols Exam relevance: Google Professional Agentic Architect (core), Claude Certified Architect – Foundations (core) ================================================================================ # Model Context Protocol (MCP) study guide > **Author:** Michaël Bettan > **Spec baseline:** MCP revision **2026-07-28**. Older behavior (2024-11-05 -> 2025-11-25) is called out where it differs, because SDKs and production deployments still reference both revisions. --- ## Table of contents 0. [Overview: what MCP is and why it exists](#0-overview-what-mcp-is-and-why-it-exists) 1. [Architecture: host, client, server](#1-architecture-host-client-server) 2. [Primitives: tools, resources, prompts](#2-primitives-tools-resources-prompts) 3. [Discovery & tool invocation](#3-discovery--tool-invocation) 4. [Tool descriptions & interface design](#4-tool-descriptions--interface-design) 5. [Structured error handling](#5-structured-error-handling) 6. [Elicitation & multi-round-trip input](#6-elicitation--multi-round-trip-input) 7. [Transports](#7-transports) 8. [Server deployment & configuration scoping](#8-server-deployment--configuration-scoping) 9. [Authentication & authorization](#9-authentication--authorization) 10. [Security threats & mitigations](#10-security-threats--mitigations) 11. [Enterprise integration](#11-enterprise-integration) 12. [MCP vs. API, function calling, frameworks & A2A](#12-mcp-vs-api-function-calling-frameworks--a2a) 13. [Spec evolution & deprecations](#13-spec-evolution--deprecations) 14. [Quick review & glossary](#14-quick-review--glossary) --- ## 0. Overview: what MCP is and why it exists **Definition.** The Model Context Protocol (MCP) is an **open, vendor-neutral protocol** that standardizes how AI applications discover and use external **tools**, **data (resources)** and **prompt templates**. Messages are **JSON-RPC 2.0** over a pluggable transport (stdio or Streamable HTTP). > **The clean interview line:** MCP standardizes how AI applications connect to tools and context. It doesn't determine how your agent reasons or orchestrates those tools. The common analogy is a **"USB-C port for AI"**. Any compliant server plugs into any compliant host. Like USB-C, it defines the connector, not what the device does with it. ### Why it was created | Problem before MCP | What MCP changes | |---|---| | Models "trapped behind information silos and legacy systems" | One protocol to reach any data source or tool | | Every new data source needed its **own custom integration** per AI app | Write a server once; every MCP host can use it | | **M × N** integration matrix (M AI apps × N tools) | **M + N**: each app implements a client once, each tool implements a server once | | Tool schemas, auth and transport reinvented per vendor | Shared discovery (`*/list`), invocation (`tools/call`), transport and OAuth profile | | Context about the user's world (files, tickets, DBs) hard to bring to the model safely | Standard resources and consent hooks owned by the host | **Timeline (know the shape, not the dates to the day):** | When | Event | |---|---| | Nov 2024 | Anthropic open-sources MCP: spec, SDKs and reference servers (Google Drive, Slack, GitHub, Git, Postgres, Puppeteer) | | 2025 | Rapid adoption across vendors and IDEs. Streamable HTTP replaces HTTP+SSE (2025-03-26). OAuth 2.1 authorization profile matures (2025-06-18). Tasks, server identity and official extensions arrive (2025-11-25) | | Late 2025 | MCP is donated to the **Agentic AI Foundation (AAIF)**, a directed fund under the **Linux Foundation** | | 2026-07-28 | Spec revision makes the protocol **stateless**: no `initialize` handshake, per-request metadata, and `server/discover`. Roots, sampling and logging are deprecated | > **Common trap:** "MCP is a proprietary plug-in format for one model." No. It started at Anthropic, but it's an open spec governed under the Linux Foundation and implemented by many hosts, frameworks and cloud providers. ### Value by audience | Audience | Value | |---|---| | Tool/data owners | Expose a capability once; reach every MCP host | | Agent developers | Plug in an ecosystem of servers without writing glue code | | Enterprises | One place to apply auth, allow-listing, audit and policy for AI-to-system access | | End users | Assistants that can safely act on their own data (calendar, docs, tickets) with consent | --- ## 1. Architecture: host, client, server ``` ┌──────────────────────── HOST (AI application) ────────────────────────┐ │ LLM + agent logic + UI + consent/policy │ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │ │ Client A │ │ Client B │ │ Client C │ (1 per server)│ │ └────┬─────┘ └────┬─────┘ └────┬─────┘ │ └────────┼───────────────────┼───────────────────┼──────────────────────┘ │ stdio │ Streamable HTTP │ Streamable HTTP ┌─────▼─────┐ ┌─────▼─────┐ ┌─────▼──────────┐ │ Server A │ │ Server B │ │ Server C │ │ local FS │ │ Git host │ │ Cloud database │ └───────────┘ └───────────┘ └────────────────┘ ``` | Role | What it is | Responsibilities | Examples | |---|---|---|---| | **Host** | The AI application the user interacts with | Runs the model and agent loop. Creates and manages clients. **Owns consent, policy and what the model sees.** Aggregates context across servers | Desktop chat apps, coding CLIs, IDEs, an agent built with a framework | | **Client** | A connector component **inside the host**, one per server connection | Speaks the protocol to exactly one server. Handles transport, version/capability metadata, request IDs and routing results back | ADK's `McpToolset` connection; an IDE's per-server connection | | **Server** | A program exposing capabilities over MCP | Advertises and executes **tools**, serves **resources**, provides **prompts**. Validates input, enforces access control | Filesystem, Git hosting, Postgres, chat/ticketing SaaS, managed cloud data services | **Key relationships:** - **1 host → many clients; 1 client ↔ 1 server.** Servers are isolated from each other. A server never sees the full conversation or other servers' data unless the host passes it in. - Tools from **all connected servers** are discovered when the host connects and are available to the agent **at the same time**. The host must handle name collisions (see §4). - Servers are **LLM-agnostic**. They don't know or care which model the host uses. - The **host is the trust and consent authority**. It decides which tools are exposed to the model, which calls need human approval, and what results are shown. > **Rule of thumb:** If the question asks "who decides whether this tool call runs?", the answer is the **host** (often via a human-in-the-loop prompt), not the server and not the model. > **Common trap:** "The client is the end-user app." In MCP vocabulary, the **host** is the app. The **client** is the per-server protocol connector living inside it. --- ## 2. Primitives: tools, resources, prompts Servers expose three primitives. The key distinction is **who controls each one**. | Primitive | Control model | Purpose | Discovery / use | Typical UI | |---|---|---|---|---| | **Tools** | **Model-controlled** (host-gated) | Executable actions with side effects or computation | `tools/list` → `tools/call` | Model decides to call; host may ask the user to approve | | **Resources** | **Application-controlled** | Context/data identified by URI (files, schemas, records, catalogs) | `resources/list`, `resources/templates/list` → `resources/read`; optional subscribe | App attaches them (file picker, @-mention, auto-include) | | **Prompts** | **User-controlled** | Reusable, parameterized message templates and workflows | `prompts/list` → `prompts/get` | Slash commands, menu items | ### When to use which primitive | You want to… | Use | Why | |---|---|---| | Let the model **take an action** or run a parameterized query (create ticket, run SQL, send message) | **Tool** | The model decides when; side effects are gated by the host | | Give the model **read-only context** the app or user chooses (a file, a schema, a policy doc) | **Resource** | Addressable by URI, no side effects, the app decides when to attach it | | Show the agent **what content exists** before it starts exploring | **Resource catalog** | One read replaces many exploratory list/search tool calls | | Offer a **tested, repeatable workflow** a human triggers | **Prompt** | Server-owned template, invoked explicitly (e.g. `/review-pr 42`) | > **Rule of thumb:** Tools for **actions**, resources for **context**, prompts for **human-triggered workflows**. Client-side features (things the **server** can ask the **client/host** for): | Feature | Status (2026-07-28) | Notes | |---|---|---| | **Elicitation** | Active | Server asks the user for structured input (form) or sends them to a URL (OAuth, payment). See §6 | | Roots | **Deprecated** | Pass paths via tool parameters, resource URIs or config instead | | Sampling | **Deprecated** | Servers call LLM APIs directly instead of borrowing the host's model | ### Tools - Each tool has a `name`, `description`, `inputSchema` (JSON Schema) and optionally a `title`, `outputSchema`, `annotations` and `icons`. - Results return `content` (text, image, audio, resource links or embedded resources) and/or `structuredContent` (JSON matching `outputSchema`), plus `isError`. - **Annotations** (`readOnlyHint`, `destructiveHint`, `idempotentHint`, `openWorldHint`) help the host decide on confirmation UX. They are **hints only**: clients should never make tool-use decisions based on annotations from untrusted servers. ### Resources and resource catalogs A **resource catalog** is the server's browsable inventory of context the host can pull in. | Mechanism | Method / capability | Use | |---|---|---| | Direct resources | `resources/list` (paginated) | Concrete URIs such as `file:///repo/README.md` or `postgres://prod/schema` | | Resource templates | `resources/templates/list` | **URI templates** (RFC 6570), e.g. `file:///{path}` or `tickets://{project}/{id}` | | Read | `resources/read` | Returns text (UTF-8) or binary (base64) with a MIME type | | Change of catalog | capability `resources.listChanged` | Catalog itself changed | | Change of a resource | capability `resources.subscribe` | Content of a subscribed URI changed | | Annotations | `audience`, `priority` (0–1), `lastModified` | Hints about who the content is for and how important it is | **Resources as content catalogs (reduce exploratory tool calls).** Without a catalog, an agent typically burns several turns calling `list_*`/`search_*` tools just to learn what exists. Exposing a compact index as a resource gives the agent that map up front: | Catalog resource | Replaces | |---|---| | `docs://index` (documentation hierarchy with titles and URIs) | Repeated `search_docs` calls to discover page names | | `db://warehouse/schema` (tables, columns, descriptions) | Trial-and-error `run_query` calls against unknown tables | | `tickets://open/summary` (open issues with one-line summaries) | Paginating `list_tickets` to find a relevant issue | The agent reads the catalog, then calls the **one** tool it actually needs (`get_page`, `run_query`, `get_ticket`) with a precise argument. > **Common trap:** "Resources are static data." They're **addressable** data. They can be dynamic (templated, generated on read) and **subscribable** for change notifications. What makes them resources is that the **application** decides when to include them, not the model. ### Prompts Prompts move prompt engineering into the server. The domain owner ships a tested template (for example `/review-pr {pr_number}`) and the server fills context in server-side. `prompts/get` returns an array of messages ready to inject. --- ## 3. Discovery & tool invocation ### End-to-end flow (2026-07-28) ``` Host/Client Server │ server/discover (optional first call) │ │ ───────────────────────────────────────────────▶ │ │ ◀──────── identity, capabilities, versions ───── │ │ tools/list (+ resources/list, prompts/list) │ │ ───────────────────────────────────────────────▶ │ │ ◀── tools[], nextCursor, ttlMs, cacheScope ───── │ │ host filters tools, adds them to model context │ │ model emits a function call ─▶ host approves? │ │ tools/call {name, arguments} │ │ ───────────────────────────────────────────────▶ │ │ ◀──── result: content / structuredContent ────── │ │ or resultType "input_required" (see §6) │ │ host feeds result back to the model │ ``` 1. **Discover the server.** Servers MUST implement `server/discover`, which returns supported protocol versions, capabilities, server identity and optional natural-language `instructions`. 2. **Per-request metadata, no handshake.** Every request carries `_meta` with the protocol version (`io.modelcontextprotocol/protocolVersion`), client capabilities and client info. Results carry server info. A version mismatch returns `UnsupportedProtocolVersionError`. 3. **List capabilities.** `tools/list` is **paginated** (`cursor` → `nextCursor`) and can include caching hints (`ttlMs`, `cacheScope`). Servers that declare `tools.listChanged` notify clients when the tool set changes. 4. **Host selects and exposes.** The host decides which tools reach the model (filtering, namespacing, per-user policy) and converts them into the model provider's function-calling format. 5. **Invoke.** When the model emits a function call, the host (optionally after user approval) sends `tools/call`. 6. **Return.** The server returns a result (`resultType: "complete"`), or asks for more input (`resultType: "input_required"`). **Example `tools/call` request and result (trimmed):** ```json { "jsonrpc": "2.0", "id": 42, "method": "tools/call", "params": { "name": "get_order_status", "arguments": { "order_id": "A-1001" }, "_meta": { "io.modelcontextprotocol/protocolVersion": "2026-07-28" } } } ``` ```json { "jsonrpc": "2.0", "id": 42, "result": { "resultType": "complete", "content": [{ "type": "text", "text": "Order A-1001 shipped on 2026-09-30." }], "structuredContent": { "order_id": "A-1001", "status": "SHIPPED" }, "isError": false } } ``` > **Common trap:** "MCP's dynamic discovery means tool schemas don't consume tokens." Whatever the host exposes still enters the model's context. Token savings come from **host-side choices**: filtering, lazy loading / tool search, and caching schemas with provider prompt caching. They don't come from the protocol itself. ### Pre-2026 lifecycle (still in many SDKs) From 2024-11-05 through 2025-11-25, a connection started with a **three-step handshake**: 1. Client → `initialize` (protocol version, client capabilities). 2. Server → response (server capabilities: tools/resources/prompts). 3. Client → `notifications/initialized`. No normal requests were allowed before this. The session then stayed stateful until shutdown. The 2026-07-28 revision removed this handshake: capability and version info travel with every request, and `server/discover` replaces the probe. --- ## 4. Tool descriptions & interface design ### Descriptions are the main selection signal The model picks a tool almost entirely from its **name, description and input schema**. It never sees your implementation. A one-line description ("Gets data") gives the model nothing to separate two similar tools, so selection becomes a coin flip. A good description states: | Element | Example | |---|---| | **What it does and returns** | "Returns current stock level, reserved quantity and warehouse for one SKU." | | **When to use it** | "Use when the user asks whether an item is available or how many are left." | | **When NOT to use it (boundary)** | "For historical sales volumes, use `get_sales_history` instead." | | **Input formats** | "`sku`: 8-character code like `AB-12345`. Not the product name." | | **Example queries** | "'Do we still have the blue kettle?' → resolve SKU first, then call this." | | **Edge cases** | "Returns `quantity: 0` with `isError: false` when the SKU exists but is out of stock." | ### Disambiguating similar or overlapping tools | Symptom | Fix | |---|---| | Two tools with near-identical descriptions get confused | Rewrite both descriptions to state **purpose, inputs, outputs and "use X instead when…"** boundaries | | A generic tool does several unrelated jobs (`process_document`) | **Split** it into purpose-specific tools (`extract_invoice_fields`, `summarize_contract`, `check_signature_present`) | | Tool names don't convey intent (`tool_a`, `query2`) | **Rename** with a clear verb + object; namespace by domain (`inventory_get_stock`) | | Several tools overlap heavily | **Consolidate** into one tool with a constrained parameter (`report_type: "daily" \| "weekly"`) | | Agent prefers a built-in generic tool (shell search, file read) over a more capable MCP tool | Expand the MCP tool's description to explain **what it can do that the generic tool can't** and what it returns | | A system-prompt instruction ("always search first") keeps overriding good descriptions | Review the system prompt for **keyword-sensitive instructions** that bias selection, and align them with the tool descriptions | Spec naming guidance: tool names SHOULD be 1–128 characters, case-sensitive, using only `A–Z a–z 0–9 _ - .`, and unique **within a server**. When a host aggregates servers, two servers may both expose `search`; the host SHOULD disambiguate (for example by prefixing a server identifier). ### Interface design rules (agent-first, not API-first) | Rule | Why | |---|---| | **Don't mirror REST 1:1.** Design task-level tools (`find_customer_orders`), not one tool per endpoint | Fewer calls, less reasoning burden, fewer errors | | **Typed, constrained inputs** (enums, formats, required fields) | Reduces malformed calls; server still validates | | **Paginate and filter server-side** (cursor/limit, query params) | Unbounded results blow the context window. Some hosts cap or offload large tool outputs, so oversized results silently lose detail | | **Truncate intelligently.** Return a summary plus how to refine the query | Keeps the model on track | | **Enrich results (semantic layer).** Resolve IDs to names, drop noise, return `structuredContent` | Model reasons over meaning, not raw keys | | **Structured, actionable errors** (see §5) | Lets the model choose the right recovery | | **Mark read-only vs. destructive** with annotations; make writes idempotent where possible | Enables safe auto-approval and retries | | **Keep each agent's toolset small and role-scoped** | Selection accuracy drops as the toolset grows | | **Version tool contracts.** Treat description changes as prompt changes | Silent behavior shifts otherwise | ### SDK options | Option | Notes | |---|---| | **Official SDKs** (Python, TypeScript, Java, C#, Go, Kotlin, …) | High-level decorators (Python **FastMCP**: `@mcp.tool`, `@mcp.resource`, `@mcp.prompt`) generate JSON Schemas from type hints and docstrings. Low-level handlers are available for full control | | **Agent frameworks** | Consume servers as toolsets (e.g. ADK `McpToolset`, LangChain `MultiServerMCPClient`); some can also expose their own tools as an MCP server | | **Managed** | Use vendor-hosted remote servers instead of running your own | ```python from mcp.server.fastmcp import FastMCP mcp = FastMCP("inventory") @mcp.tool() def inventory_get_stock(sku: str) -> dict: """Return current stock level, reserved quantity and warehouse for ONE SKU. Use when the user asks if an item is available. For past sales, use inventory_get_sales_history instead. sku format: 'AB-12345' (not a name).""" return {"sku": sku, "quantity": 12, "reserved": 3, "warehouse": "LYS-1"} @mcp.resource("inventory://catalog") def catalog() -> str: """Compact index of product families and SKU ranges (content catalog).""" return '{"kettles": "AB-10000..AB-19999", "toasters": "AB-20000..AB-29999"}' ``` Call `mcp.run()` to serve it (stdio by default; HTTP transport is configurable). > **Common trap:** Output schemas must resolve to a JSON **object**. Untyped `*args/**kwargs` or non-serializable return types break schema generation at startup. --- ## 5. Structured error handling ### Two error channels in the spec | Channel | Signal | Meaning | Who reacts | |---|---|---|---| | **Protocol error** | JSON-RPC `error` object with a numeric `code` and a `message` (unknown tool, malformed request, unsupported method, server fault) | The request could not be processed as a valid MCP call | Host/client code; the model usually never sees it | | **Tool execution error** | `CallToolResult` with `isError: true` and an explanation in `content` | The tool ran but failed (upstream API failure, invalid argument values, business rule) | **Model**, which reads the message and adjusts | The spec says errors that originate **inside the tool** SHOULD be reported in the result with `isError: true`, not as a protocol error, otherwise the model can't see what went wrong. Clients SHOULD pass tool execution errors to the model. Recent revisions also classify **input validation failures** (for example a date in the past) as tool execution errors, so the model gets a chance to fix its arguments. ### JSON-RPC 2.0 error codes MCP reuses the standard JSON-RPC 2.0 codes. Don't invent extra fields to carry meaning: use the `code`, a clear `message` and, if needed, the optional `data` member. | Code | Name | Typical MCP trigger | |---|---|---| | `-32700` | Parse error | The body is not valid JSON | | `-32600` | Invalid request | Valid JSON, but not a valid JSON-RPC request (e.g. missing `jsonrpc` or `method`) | | `-32601` | Method not found | The server does not implement the method, e.g. a capability it never advertised | | `-32602` | Invalid params | Wrong parameter structure, or `tools/call` names a tool that does not exist | | `-32603` | Internal error | Unexpected failure inside the server while handling the request | | `-32000` … `-32099` | Server error | Reserved for implementation-defined server errors; document any code you use | ```json { "jsonrpc": "2.0", "id": 3, "error": { "code": -32602, "message": "Unknown tool: ship_parcel_v1" } } ``` ### Tool execution errors the model can act on A tool execution error is an ordinary result. Everything the model needs is in the text it reads: ```json { "jsonrpc": "2.0", "id": 7, "result": { "resultType": "complete", "isError": true, "content": [{ "type": "text", "text": "Pickup date 2026-10-02 is in the past. Use a date on or after 2026-11-02 (YYYY-MM-DD)." }] } } ``` A useful error message answers four questions in plain language: 1. **What failed?** Name the argument, record or upstream system. 2. **Why?** The rule or condition that was violated. 3. **What would succeed?** The valid format, range or next step. 4. **Is repeating the same call pointless?** Say so explicitly when only the user or an administrator can unblock it. If the tool declares an `outputSchema`, `structuredContent` must conform to it; keep error explanations in `content` rather than bending the schema. > **Common trap:** One generic error string for every failure. The model can't tell a typo it could fix from a condition only a human can resolve, so it either loops on hopeless calls or gives up on recoverable ones. ### Failure vs. a valid empty answer | Situation | Correct response | |---|---| | The lookup ran and matched **nothing** | Normal result (`isError` absent or `false`) with an empty list and a short note such as `"0 shipments for customer C-88"` | | The backend was unreachable or refused access | `isError: true` with a message explaining that the lookup **could not be performed** | > **Common trap:** Catching an exception and returning `[]`. The model then states "no records exist" with confidence when the truth is "I couldn't check". ### Retries are host policy, informed by annotations The protocol does not define retry semantics. Retry decisions live in the host or agent framework: | Signal | How it informs retries | |---|---| | `idempotentHint: true` | Repeating the call with the same arguments has no extra effect, so an automatic retry after a timeout is low risk | | `idempotentHint: false` / `destructiveHint: true` | Don't auto-retry blindly. Prefer a server-supported idempotency key argument, or confirm with the user first | | `readOnlyHint: true` | No side effects; retries only cost latency and quota | Annotations are **hints**: treat them as untrusted unless the server itself is trusted. Bound every retry loop (attempts, backoff, overall timeout) so a failing tool cannot burn the session budget. --- ## 6. Elicitation & multi-round-trip input Sometimes a tool needs more information from the human mid-call (a missing date, a confirmation, a third-party login). **Multi-round-trip input (2026-07-28):** 1. Client sends `tools/call`. 2. Server returns `resultType: "input_required"` with `inputRequests` (for example an `elicitation/create` request) and an opaque `requestState`. 3. Host renders the request to the user and collects the answer. 4. Client **retries** the call with `inputResponses` and the same `requestState`, using a **new JSON-RPC id**. The server can stay stateless because the state travels with the request. **Elicitation modes:** | Mode | What happens | Use for | |---|---|---| | **Form** | Server sends a `requestedSchema` (JSON Schema). Host renders a form. Data is exposed to the client. User can accept, decline or cancel | Missing parameters, confirmations, choices | | **URL** | Server gives a URL; the user completes the flow **out-of-band** in the browser. Data (other than the URL) is not exposed to the client | OAuth to a third-party service, payments, entering API keys | > **Common trap:** Collecting a third-party password or API key via **form** elicitation. Sensitive credentials should use **URL mode**, so they never transit the MCP client. The server then stores the resulting third-party tokens itself, and must **not** reuse the client's MCP access token for upstream calls. --- ## 7. Transports | Transport | How it works | When | Status | |---|---|---|---| | **stdio** | Host spawns the server as a **subprocess**. Newline-delimited JSON-RPC on stdin/stdout. **stderr is for logs only** | Local tools, dev machines, IDE plugins | Current | | **Streamable HTTP** | Single endpoint (e.g. `/mcp`). Each message is an HTTP **POST**. Server replies with plain JSON or an SSE stream | Remote and shared servers, cloud, multi-tenant | Current (standard for remote) | | HTTP+SSE (two endpoints) | Separate SSE GET stream plus POST endpoint | Legacy clients only | **Deprecated** since 2025-03-26 | ### Streamable HTTP: 2026-07-28 vs. earlier | Aspect | 2025-03-26 -> 2025-11-25 | **2026-07-28** | |---|---|---| | Session | `Mcp-Session-Id` assigned at `initialize`, echoed on later requests | **No sessions.** Each request is self-describing via `_meta` | | Headers | `Mcp-Session-Id`, `MCP-Protocol-Version` | `MCP-Protocol-Version`, `Mcp-Method`, `Mcp-Name` (tool/resource/prompt name for `tools/call`, `resources/read`, `prompts/get`), and cloud gateway routing headers such as `Mcp-Param-Region`, so gateways can route and authorize (`REQUEST_AUTHZ`) **without parsing the JSON-RPC body** | | Server -> client notifications | GET SSE stream | `subscriptions/listen` opens a long-lived stream for change notifications | > **Rule of thumb:** stdio = "one user, one machine, the host owns the process." Streamable HTTP = "a network service." The stateless 2026 design lets remote servers scale horizontally behind ordinary load balancers, with no sticky sessions. > **Common trap:** "SSE is the remote transport." The standalone HTTP+SSE transport is **deprecated**. Streamable HTTP may *use* SSE to stream a response, but it's a different transport. --- ## 8. Server deployment & configuration scoping ### Local vs. remote servers | Dimension | **Local** (stdio subprocess) | **Remote** (Streamable HTTP) | |---|---|---| | Where it runs | User's machine, spawned by the host | Cloud/service endpoint (vendor-hosted or self-hosted) | | Trust boundary | Runs with the **user's OS privileges** | Network service, so treat it as an external API | | Credentials | From **environment variables** / local config; stdio **should not** use the HTTP OAuth flow | **OAuth 2.1** bearer tokens (or API keys/IAM in managed offerings) | | Tenancy | Single user | Multi-tenant; must isolate users | | Distribution | `npx`, `uvx`, container image, binary | URL; nothing installed client-side | | Updates | User must update the package | Provider updates centrally | | Main risks | Supply-chain (malicious package), local file/command access | Token theft/misuse, confused deputy, SSRF, tenant leakage | | Mitigations | Pin versions, containerize, run non-root, restrict FS paths | Audience validation, least-privilege scopes, gateway policy, audit logs | ### Configuration scopes: shared vs. personal Most hosts let you declare servers at more than one scope. The pattern is the same everywhere: | Scope | Typical location | Shared? | Use for | |---|---|---|---| | **Project** | A config file committed at the repo root (e.g. `.mcp.json`) | **Yes**, via version control | Team tooling everyone on the project needs (issue tracker, internal docs, staging DB) | | **User** | A file in the user's home directory | No | Personal or experimental servers you use across projects | | **Local / per-project private** | User config keyed by project path | No | Personal overrides for one project | In Claude Code, for example: project scope = `.mcp.json` at the project root; user and local scopes live in `~/.claude.json`; on name clashes **local > project > user**; and project-scoped servers trigger an **approval prompt** the first time, so a cloned repo cannot launch processes without consent. Servers from every scope are merged: the host connects to **all** configured servers and their tools are offered to the model **together** in the same session (Claude Code defers full schemas behind tool search by default; `alwaysLoad: true` preloads one server's tools). ### Secrets: environment-variable expansion A committed project config must **never contain secrets**. Reference them instead: ```json { "mcpServers": { "tracker": { "type": "http", "url": "${TRACKER_URL:-https://tracker.example.com}/mcp", "headers": { "Authorization": "Bearer ${TRACKER_TOKEN}" } }, "inventory": { "type": "stdio", "command": "uvx", "args": ["inventory-mcp-server==1.4.2"], "env": { "INVENTORY_API_KEY": "${INVENTORY_API_KEY}" } } } } ``` - `${VAR}` expands from each developer's environment; `${VAR:-default}` adds a fallback (Claude Code supports both in `command`, `args`, `env`, `url` and `headers`). - Exact keys vary by host (`url`, `httpUrl`, `type: "http"`). The principles don't: **pin versions** and **inject secrets from the environment or a secret manager**. ### Community vs. custom servers | Choose | When | |---|---| | **Existing community / vendor server** | Standard integrations (issue trackers, Git hosting, chat, common databases). Faster, maintained by others. Still vet it: review the tool definitions, pin the version, prefer official or registry-listed servers | | **Custom server** | Team-specific workflows, internal systems, or when you need tighter tool design, filtering or policy than a generic server provides | > **Common traps:** (1) Hard-coding a token in a committed config "because the repo is private". (2) Putting a team-wide server in one person's user config, so nobody else gets it. (3) Building a custom server for a mainstream SaaS integration "for control" instead of starting with a vetted existing one. --- ## 9. Authentication & authorization Keep three layers distinct: | Layer | Question | Mechanism | |---|---|---| | **Authentication** | Who is the client/user? | OAuth 2.1 access token (remote); OS user + env credentials (local); cloud IAM for managed servers | | **Authorization** | What may this identity do on this server? | Token **scopes** and **audience**, server-side access control (RBAC/ABAC), per-tool policy | | **Permissions / consent** | Should *this particular* call happen now? | **Host** confirmation (HITL), tool allow-lists, annotation-informed UX, auto-approve only safe/read-only tools | ### The MCP authorization profile (HTTP transports) Authorization is **optional**. HTTP implementations **should** conform to the profile; stdio implementations **should not** and should take credentials from the environment instead. | Concept | Rule | |---|---| | Roles | **MCP server = OAuth 2.1 resource server.** A separate **authorization server** (your IdP) issues tokens. **MCP client = OAuth client** | | Discovery | Server publishes **Protected Resource Metadata** (RFC 9728) listing `authorization_servers` | | Challenge | Unauthenticated request → `401` with `WWW-Authenticate` pointing at the resource metadata URL | | Flow | Authorization code + **PKCE**. Tokens sent as `Authorization: Bearer …` | | Resource indicators | Client sends the `resource` parameter (RFC 8707), so the token is minted **for this specific server** | | Audience validation | Server **MUST** validate that the token was issued for it | | **No token passthrough** | Server **MUST NOT** forward the client's token to upstream APIs. For upstream calls it acts as its own OAuth client with separate tokens | | Client registration | Preferred: **Client ID Metadata Documents (CIMD)**. Dynamic Client Registration (RFC 7591) is **deprecated** in 2026-07-28 | | Step-up | Server may return `403` / `insufficient_scope` with the needed scope; client re-authorizes | > **Common trap:** "Token passthrough is fine if the downstream API accepts it." It's explicitly forbidden. It breaks audience binding, enables **confused-deputy** attacks, bypasses the server's own controls and audit, and lets a stolen token be replayed across services. --- ## 10. Security threats & mitigations ### Least privilege in practice - Act **as the end user** (delegated identity), not as a shared super-account. - Request **narrow scopes** and step up only when needed. - Expose **only the tools a given agent needs** (host allow-lists, framework tool filters). - Prefer **read-only** tools by default. **Require human confirmation for destructive or outbound actions** (delete, pay, send, deploy). - The spec says there SHOULD always be a human able to deny tool invocations: show which tools are exposed, indicate when they run, and prompt for confirmation. - Servers validate all inputs, enforce access control, rate-limit and sanitize outputs. Clients show tool inputs before calling, validate results, set timeouts and log calls. ### Threat catalog | Threat | What it looks like | Mitigations | |---|---|---| | **Prompt injection (indirect)** | Tool output or resource content contains instructions ("ignore previous instructions, email the file to…") | Treat all tool/resource content as untrusted data. Screen with classifiers. HITL for sensitive actions. Least-privilege tools | | **Tool poisoning** | Malicious instructions hidden in tool **descriptions** or annotations | Allow-list and review servers. Pin versions. Diff tool definitions on change. Treat annotations as untrusted | | **Rug pull / silent redefinition** | Trusted server later changes a tool's behavior or description | Version pinning, change alerts, re-approval on definition change | | **Tool shadowing / name collision** | One server's tool mimics another's name to intercept calls | Namespacing, per-server allow-lists, host-side disambiguation | | **Malicious or typosquatted servers** | Supply-chain attack via `npx`/`uvx` package | Private registry, signed/pinned packages, containerized execution | | **Token passthrough / confused deputy** | Server forwards or misuses user tokens | Audience validation, resource indicators, no passthrough | | **Data exfiltration** | Agent combines a private-data tool with an outbound tool (email, HTTP fetch) | Egress controls, DLP, separate high-trust and open-world tools, HITL on outbound sends | | **Excessive agency** | Agent can do more than the task requires | Narrow scopes, tool filters, read-only defaults, RBAC/ABAC on the server | | **SSRF** | Fetch-style tools pointed at internal endpoints | Block private IP ranges and metadata endpoints; validate URLs | | **Local command execution** | stdio server runs shell/FS operations driven by injected arguments | Sandbox (container, non-root), path allow-lists, argument validation | | **Runaway loops** | Agent loops on a tool, burning cost | Rate limits, quotas, timeouts, circuit breakers | > **Rule of thumb:** The model is not a security boundary. Enforce access control in the **server** and consent in the **host**. Assume every description, argument and result could be adversarial. --- ## 11. Enterprise integration ``` Agents / IDEs / chat apps (MCP hosts) │ OAuth / IAM identity of the end user ▼ ┌──────────────────────────────┐ │ MCP gateway / proxy │ allow-list, routing (Mcp-Method/Mcp-Name/Mcp-Param-Region), │ │ authN/Z, rate limits, content screening, audit (OTel) └──────┬──────────┬────────────┘ ▼ ▼ ▼ Internal MCP Vendor remote Managed cloud MCP servers MCP (SaaS) servers ``` | Concern | Enterprise pattern | |---|---| | **Inventory / discovery** | Private **registry** of approved servers (**Agent Registry**). Block unregistered servers at the gateway | | **Allow-listing** | Approved server URLs/packages with **pinned versions**; per-agent `tool_filter` allow-lists | | **Identity** | SSO-backed OAuth (**Auth Manager** 3LO), per-user delegated tokens, SPIFFE **Agent Identity** for service-to-service calls | | **Policy enforcement** | Central **Agent Gateway**: header-based `REQUEST_AUTHZ` (`Mcp-Name`, `Mcp-Param-Region`), `CONTENT_AUTHZ` (**Model Armor** + **Semantic Governance**), egress controls, rate limits | | **Content safety** | Prompt-injection / sensitive-data screening (**SDP**) of tool inputs and outputs | | **Observability** | Log every `tools/call` with dual attribution (agent + human user), tool, redacted arguments, latency and outcome via **OpenTelemetry** traces (replaces deprecated protocol logging) | | **Isolation** | Containerize servers; network policies; separate tenants; sandbox untrusted connectors | | **Lifecycle** | Versioned servers, schema change review (`toolspec.json` <= 10 KB for manual external registration), staged rollout | **Framework example (ADK, open-source):** consume a remote server with least privilege. ```python from google.adk.agents import LlmAgent from google.adk.tools.mcp_tool import McpToolset, StreamableHTTPConnectionParams inventory = McpToolset( connection_params=StreamableHTTPConnectionParams(url="https://inventory.internal/mcp"), tool_filter=["inventory_get_stock", "inventory_list_warehouses"], # read-only subset ) agent = LlmAgent(model="gemini-3.8-flash", name="stock_agent", instruction="Answer stock questions using the tools.", tools=[inventory]) ``` ### Google Cloud enterprise MCP stack | Integration surface | Mechanism, quotas & operational discriminators | |---|---| | **Google-hosted remote MCP servers** | Stateless `2026-07-28` `StreamableHTTP` endpoints (`bigquery.googleapis.com/mcp`, `container.googleapis.com/mcp`, `compute.googleapis.com/mcp`, `run.googleapis.com/mcp`, `logging.googleapis.com/mcp`), auto-registered in **`global` Agent Registry**, gated by `roles/mcp.toolUser` + target resource IAM, and routed by `Mcp-Name` / `Mcp-Param-Region` headers (**VPC-SC exception:** `developerknowledge.googleapis.com/mcp` does not support `restricted.googleapis.com` `199.36.153.4/30` and must egress via `private.googleapis.com` `199.36.153.8/30`) | | **Custom remote MCP on Cloud Run** | Deploy with `gcloud run deploy --functional-type=mcp-server --no-allow-unauthenticated` behind **Cloud IAP** and **Agent Gateway** (`roles/iap.egressor`); pass Cloud Run OIDC in `X-Serverless-Authorization` when forwarding a user `Authorization` OAuth token | | **Gemini Enterprise MCP connectors** | Enforces a **`100`-action cap** per custom MCP server registration; inspects tool annotations (`readOnlyHint: true` tools can execute without repetitive prompts when policy allows, whereas mutative/`destructiveHint: true` actions trigger explicit user confirmation) | | **Databases, SaaS & OpenAPI bridges** | **MCP Toolbox for Databases** (`googleapis/genai-toolbox` / `ToolboxToolset`) provides sub-50 ms pooled, parameterized `tools.yaml` SQL/Cypher queries across 30+ DBs; **`ApplicationIntegrationToolset`** connects 100+ SaaS/ERP apps with Auth Manager 3LO; **`ApiHubToolset` / Apigee MCP Discovery Proxy** exposes cataloged OpenAPI REST APIs as governed MCP tools with rate/token quotas and Model Armor | > **Rule of thumb:** In an enterprise answer, mention four things: **registry/allow-list, per-user OAuth with audience-bound tokens, a gateway for policy and audit, and host-side HITL for destructive tools.** --- ## 12. MCP vs. API, function calling, frameworks & A2A ### MCP vs. a plain API | | REST/gRPC API | MCP server | |---|---|---| | Audience | Deterministic software written by developers | **AI applications and models** | | Discovery | Docs / OpenAPI, read by humans at build time | **Runtime, self-describing** (`tools/list`, `resources/list`, `prompts/list`) with descriptions aimed at the model | | Shape | Resource-oriented, many fine-grained endpoints | **Task-oriented tools**; context as resources; templates as prompts | | Integration cost | Custom client per app | One generic MCP client works with every server | | Relationship | N/A | MCP usually **wraps** existing APIs; it does not replace them | ### MCP vs. function calling | | Function (tool) calling | MCP | |---|---|---| | What it is | A **model API feature**: you pass function declarations, the model returns "call X with args Y" | A **protocol between the host and external servers** for discovering and executing tools (and resources/prompts) | | Where it lives | Between your code and the **LLM** | Between your host and **tool providers** | | Who executes | Your application code | The MCP server (the host sends `tools/call`) | | Portability | Declaration format differs per provider | Same server works with any host and any model | **They are complementary.** The host fetches tools via MCP, translates them into the provider's function declarations, the model emits a function call, and the host routes it back as `tools/call`. > **Common trap:** "MCP replaces function calling." No. The model still uses function calling to *choose* a tool. MCP standardizes *where tool definitions come from and how the call is executed*. ### MCP vs. agent framework | | Agent framework (ADK, LangGraph, Claude Agent SDK…) | MCP | |---|---|---| | Concern | **Reasoning and orchestration**: agent loop, planning, routing, state, memory, evaluation | **Connectivity**: how capabilities and context are exposed and invoked | | Decides *which* tool and *when* | Yes (with the model) | No | | Holds conversation state / memory | Yes | No (stateless protocol in 2026-07-28) | | Relationship | Frameworks act as **MCP hosts/clients** and can also expose tools as MCP servers | Pluggable layer under any framework | ### MCP vs. A2A | | MCP | A2A (Agent2Agent) | |---|---|---| | Connects | Agent <-> **tools/data** (vertical) | Agent <-> **agent** (horizontal) | | Remote side is | A capability provider without its own goals | An autonomous agent, treated as **opaque** | | Discovery | `server/discover`, `tools/list` | **Agent Card** (`/.well-known/agent-card.json` / Agent Registry) | | Unit of work | Tool call / resource read | **Task** with lifecycle states, messages and artifacts | > **Rule of thumb:** **MCP = agent-to-tool. A2A = agent-to-agent.** Real systems use both. ### An interoperability protocol, not an orchestration framework MCP defines message format, methods, transports, capability metadata/versioning and an authorization profile. It does **not** define planning, tool ordering/retries, memory, multi-agent routing, guardrail policy or which model you use. That narrow contract is what makes the **M + N** ecosystem possible and keeps concerns separated: tool owners ship capabilities, agent builders own reasoning. --- ## 13. Spec evolution & deprecations | Revision | Headline changes | |---|---| | **2024-11-05** | Initial release. stdio + HTTP+SSE. Tools, resources, prompts, sampling, roots, logging | | **2025-03-26** | **Streamable HTTP** replaces HTTP+SSE. First OAuth 2.1 authorization framework. Tool annotations | | **2025-06-18** | Server = OAuth **resource server** with PRM (RFC 9728). RFC 8707 resource indicators. **Elicitation**. Structured tool output (`outputSchema`/`structuredContent`). `MCP-Protocol-Version` header | | **2025-11-25** | Tasks (long-running), server identity, official **extensions** mechanism, URL-mode elicitation, CIMD | | **2026-07-28** | **Stateless protocol**: no `initialize`, per-request `_meta`, `server/discover`, `input_required` multi-round-trip, session-free Streamable HTTP (`Mcp-Method`/`Mcp-Name` headers, `subscriptions/listen`) | **Deprecated in 2026-07-28** (removable in the first revision on or after 2027-07-28): | Deprecated | Replacement | |---|---| | **Roots** | Pass paths via tool parameters, resource URIs or server config | | **Sampling** (server borrows the host's LLM) | Server calls an LLM provider API directly | | **Protocol logging** | stderr (stdio) or **OpenTelemetry** | | **Dynamic Client Registration** (RFC 7591) | **Client ID Metadata Documents** | | HTTP+SSE transport (deprecated since 2025-03-26) | Streamable HTTP | > **Common trap:** Answering a "current spec" question with the `initialize` -> `initialized` handshake or `Mcp-Session-Id`. Both are correct for **2025-11-25 and earlier**, but were removed in 2026-07-28. If a question names a revision, answer for that revision. --- ## 14. Quick review & glossary ### Quick review | If the question says… | Think… | |---|---| | "Standard way for AI apps to connect to tools and data" | **MCP** | | "Agent delegating to another autonomous agent" | **A2A**, not MCP | | "Who approves a destructive tool call?" | **Host** (HITL), not the server or model | | "Component inside the app that talks to one server" | **MCP client** (1:1 with a server) | | "Model decides to invoke it" / "App attaches context" / "User picks a slash command" | **Tool** / **Resource** / **Prompt** | | "Agent wastes turns listing things to find what exists" | Expose a **resource catalog** | | "Agent keeps picking the wrong one of two similar tools" | Improve **descriptions** (purpose, inputs, boundaries); rename/split overlapping tools | | "Tool failed; how does the model find out?" | `isError: true` in the result, not a JSON-RPC error | | "Should the agent retry?" | Host policy: retry idempotent/read-only tools with bounded backoff; confirm before repeating destructive calls | | "`tools/call` names a tool that doesn't exist" | Protocol error, JSON-RPC `-32602` | | "Query matched nothing" | Valid empty result, `isError: false` | | "Team needs the same server; secrets must stay out of git" | Project-scoped config + `${VAR}` expansion | | "Personal experimental server" | User-scoped config | | "Standard SaaS integration" | Vetted community/vendor server first; custom for team-specific workflows | | "Local server, user's machine" / "Remote, multi-tenant" | **stdio** + env credentials / **Streamable HTTP** + OAuth 2.1 | | "SSE transport" | **Deprecated** (HTTP+SSE); Streamable HTTP may stream via SSE | | "Bind token to this MCP server" | **RFC 8707** `resource` parameter + audience validation | | "Server forwards user's token upstream" | **Token passthrough**, forbidden | | "Malicious instructions in a tool description" | **Tool poisoning**: allow-list, pin, diff, treat annotations as untrusted | | "Third-party OAuth / API key during a tool call" | **URL-mode elicitation** (or **Auth Manager** 3LO) | ### Glossary | Term | Definition | |---|---| | **Annotations (tool)** | `readOnlyHint`, `destructiveHint`, `idempotentHint`, `openWorldHint`. Hints only; untrusted unless the server is trusted | | **CIMD** | Client ID Metadata Document: OAuth `client_id` is an HTTPS URL to the client's metadata; replaces DCR | | **Client** | Protocol connector inside the host, one per server | | **Confused deputy** | A privileged component tricked into acting for an unauthorized party | | **Elicitation** | Server-initiated request for user input (form or URL mode) | | **Host** | The AI application: runs the model, manages clients, owns consent and policy | | **`input_required`** | 2026-07-28 result type asking the client to gather input and retry with `requestState` | | **`isError`** | Tool-result flag signalling a tool execution error the model should see | | **JSON-RPC error code** | Numeric `code` in a protocol error: `-32700`, `-32600`, `-32601`, `-32602`, `-32603`, or `-32000`…`-32099` for implementation-defined server errors | | **`Mcp-Name` / `Mcp-Param-Region`** | Stateless Streamable HTTP routing headers exposing target tool/resource name and cloud region for header-only gateway authorization (`REQUEST_AUTHZ`) | | **Project scope** | Server config committed with the repo (e.g. `.mcp.json`), shared by the team | | **PRM** | Protected Resource Metadata (RFC 9728): tells clients which authorization servers protect the server | | **Prompt (MCP)** | User-controlled, server-provided message template | | **Resource** | Application-controlled, URI-addressable context; may be templated and subscribable | | **Resource catalog** | A resource listing what content exists, so the agent skips exploratory tool calls | | **`server/discover`** | Required 2026-07-28 method returning versions, capabilities and identity | | **stdio / Streamable HTTP** | Local subprocess transport / remote POST-based transport | | **Token passthrough** | Forbidden anti-pattern of forwarding a client's token to downstream APIs | | **Tool** | Model-controlled executable function with a JSON Schema input | | **Tool poisoning** | Hiding malicious instructions in tool metadata | | **User scope** | Personal server config available across all of a user's projects | ================================================================================ GUIDE: Agent2Agent (A2A) URL: https://mbettan.github.io/machine-learning-cert/A2A.html Practice quiz: https://mbettan.github.io/machine-learning-cert/quiz.html?topic=a2a Section: Foundation · Agent Architecture & Protocols Exam relevance: Google Professional Agentic Architect (core) ================================================================================ # Agent2Agent (A2A) & Agentic Interoperability: Study Guide > **Author:** Michaël Bettan > **Version:** 2.0 > **Updated:** October 2026 Comprehensive architectural guide to the **Agent2Agent (A2A)** open standard (v0.3 → v1.0 specification) governed under the Linux Foundation, how to choose between **intra-app `AgentTool` / sub-agents** and **inter-app A2A delegation**, and how A2A composes with the broader **5-layer agentic interoperability stack**: **MCP** (tools), **A2A** (peer agents), **MCP UI / AG-UI / A2UI** (declarative generative UI), **Live Streaming** (real-time voice/video), and **AP2 + x402** (agentic payments & commerce). --- ## 01. Definition & Scope ### What is A2A? The **Agent2Agent (A2A)** protocol is an open, vendor-neutral standard initiated by Google with 50+ enterprise partners and governed by the **Linux Foundation**. It defines how autonomous agents **discover capabilities**, **negotiate modalities**, **delegate stateful tasks**, **stream incremental artifacts**, and **enforce enterprise authentication** across organizational, cloud, and framework boundaries. > **What A2A Solves — The Multi-Agent Silo Problem:** > Enterprise departments inevitably build specialized agents on heterogeneous stacks (Google ADK on Agent Runtime, LangGraph on GKE, CrewAI on Cloud Run, SaaS agents inside Workday, ServiceNow, Salesforce, or SAP). Without a wire-level peer protocol, integrating $N$ agents requires brittle custom glue adapters and exposes internal prompts or memory structures. A2A lets **opaque agents** collaborate as peers without sharing internal code, prompts, session databases, or model providers. ### MCP vs. A2A: Vertical Tools vs. Horizontal Peers **MCP** and **A2A** are **complementary, orthogonal layers** — never mutually exclusive replacements: | Dimension | **Model Context Protocol (MCP)** | **Agent2Agent Protocol (A2A)** | | :--- | :--- | :--- | | **Topology** | **Vertical** (`Agent ↔ Tool / Resource / Database`) | **Horizontal** (`Client Agent ↔ Remote Specialist Agent`) | | **Target abstraction** | Deterministic, schema-bound functions (`tools/call`), static/dynamic `resources`, prompt templates | Autonomous, opaque cognitive peers that plan, reason, ask follow-up questions, and run multi-step workflows | | **State & duration** | Primarily stateless/transactional RPC per tool invocation | **Stateful `Task` lifecycle** (`submitted` → `working` → `input-required` → `completed`) spanning seconds to days | | **Discovery artifact** | `tools/list` JSON Schema over stdio or Streamable HTTP | **`AgentCard`** at `/.well-known/agent-card.json` (plus authenticated extended cards & enterprise registries) | > **Canonical Production Pattern:** A front-door orchestrator in **Gemini Enterprise** delegates domain goals over **A2A** to specialized department agents (Finance, Legal, SecOps), and each specialist agent invokes its own private databases and SaaS APIs over **MCP**. --- ## 02. Core & Wire Protocol Architecture (v0.3 → v1.0) ### Three Architectural Roles - **User (Human or Trigger):** Initiates the high-level objective and holds primary identity/authorization context. - **A2A Client (Client / Host Agent):** Decomposes the user goal, inspects discovered `AgentCard` skills, dispatches tasks, handles `input-required` / `auth-required` interruptions, and synthesizes multi-agent outputs. - **A2A Server (Remote / Opaque Agent):** Exposes an HTTPS A2A endpoint, executes tasks autonomously using its own private tools/memory/LLM, and emits `Task` state updates and `Artifact` outputs **without exposing its internal chain-of-thought, system prompt, or tool credentials**. ### Three-Layer Protocol Stack (`a2a.proto`) 1. **Canonical Data Model:** Protobuf-defined domain objects (`AgentCard`, `Task`, `Message`, `Part` [`TextPart`, `DataPart`, `FilePart`], `Artifact`, `TaskStatusUpdateEvent`, `TaskArtifactUpdateEvent`) that neutralize framework mismatch. 2. **Abstract Operations (v1.0 PascalCase vs. v0.3 slash names):** - `SendMessage` *(v0.3: `message/send`)* — synchronous response or initial async `Task` creation. - `SendStreamingMessage` *(v0.3: `message/stream`)* — opens an SSE stream of status + artifact chunks. - `GetTask` *(v0.3: `tasks/get`)* — polls current `Task` status and artifacts by `taskId`. - `CancelTask` *(v0.3: `tasks/cancel`)* — requests graceful cancellation of an in-flight `Task`. - `SubscribeToTask` *(v0.3: `tasks/resubscribe`)* — **reconnects an SSE stream** to an existing in-flight `Task` after client network drop without restarting work. - `GetExtendedAgentCard` *(REST: `GET /extendedAgentCard`)* — fetches privileged/internal skills after authentication. 3. **Protocol Bindings (`supportedInterfaces`):** A2A v1.0 supports three official wire bindings declared in `AgentCard.supportedInterfaces[]` (`url` + `protocolBinding`): **`JSONRPC`** (JSON-RPC 2.0 over HTTPS), **`HTTP+JSON`** (RESTful HTTP), and **`GRPC`** (high-throughput binary gRPC). --- ## 03. Standard Task Lifecycle & State Machine Every delegated unit of work is tracked via two distinct identifiers: - **`contextId`:** Groups multiple related `Task` objects and follow-up turns into one logical multi-turn conversation thread. - **`taskId`:** Uniquely identifies one specific execution lifecycle through three state categories: | State Category | v1.0 Protobuf Enum *(v0.3 JSON)* | Wire & Client Behavior | | :--- | :--- | :--- | | **Active (Non-Terminal)** | `TASK_STATE_SUBMITTED` (`submitted`), `TASK_STATE_WORKING` (`working`) | Task is queued or actively running; `/message:stream` (`SendStreamingMessage`) SSE streams stay open; polling continues | | **Interrupted (Paused)** | `TASK_STATE_INPUT_REQUIRED` (`input-required`), `TASK_STATE_AUTH_REQUIRED` (`auth-required`) | **SSE stream closes cleanly**; client must either prompt the human/orchestrator for missing slot data (`TASK_STATE_INPUT_REQUIRED` using the same `taskId` + `contextId`) or complete an out-of-band OAuth 3LO / step-up consent flow (`TASK_STATE_AUTH_REQUIRED`) before resuming | | **Terminal (Final)** | `TASK_STATE_COMPLETED` (`completed`), `TASK_STATE_FAILED` (`failed`), `TASK_STATE_CANCELED` (`canceled`), `TASK_STATE_REJECTED` (`rejected`) | Execution is permanently finished (`TASK_STATE_REJECTED` indicates the remote agent declined the task based on policy or capability); further mutations require creating a new `Task` (optionally sharing `contextId`) | - **`Message` vs. `Artifact` & `gs://` URI handoff:** A **`Message`** carries conversational turns, instructions, and clarification prompts between client and server (`role: "user"` or `"agent"`), whereas an **`Artifact`** represents a durable task deliverable (generated report, SQL extract, PDF, or structured JSON). Never base64-encode multi-megabyte PDFs, CSVs, or media inside inline `Part` / `Artifact` JSON payloads (which bloat JSON-RPC serialization and choke **Agent Gateway `CONTENT_AUTHZ`** stream inspection); persist binary blobs in Cloud Storage via **`GcsArtifactService`** and pass **`gs://` URIs** in `FilePart` (`fileWithUri`). **Standard 5-step execution flow:** 1. **Capability Discovery:** Fetch `/.well-known/agent-card.json` (or query **Agent Registry**). 2. **Skill & Modality Matching:** Match goal against `skills[]`, `defaultInputModes` / `defaultOutputModes` (MIME types), and `securityRequirements`. 3. **Dispatch:** Call `SendMessage` or `SendStreamingMessage` (`/message:stream`) with typed `Part` payloads (`TextPart`, `DataPart`, `FilePart` with `gs://` URI) and HTTP `Authorization` headers. 4. **Incremental Collaboration:** Receive `TaskStatusUpdateEvent` transitions and chunked `TaskArtifactUpdateEvent` outputs (or handle `TASK_STATE_INPUT_REQUIRED` / `TASK_STATE_AUTH_REQUIRED` pauses). 5. **Artifact Consumption:** Pass completed `Artifact` outputs downstream (artifact chaining) or render to the user. --- ## 04. Transports, Discovery & Agent Cards ### Selecting the Right Transport Pattern | Pattern | A2A Operation(s) | Task Duration | When to Choose | | :--- | :--- | :--- | :--- | | **1. Synchronous Request/Response** | `SendMessage` | `< 5–15 seconds` | Fast deterministic lookups or single-step specialist replies | | **2. Streaming Updates (SSE)** | `SendStreamingMessage` (`/message:stream`, + `SubscribeToTask` on drop) | `5 seconds – several minutes` | Interactive UX needing live progress stages and incremental chunked artifacts (`append: false/true`, `lastChunk: true`) | | **3. Asynchronous Polling** | `SendMessage` -> periodic `GetTask` | `Minutes` (or environments blocking SSE/inbound webhooks) | Batch synthesis where client sits behind strict NAT/egress-only firewalls | | **4. Push Notifications (Webhooks)** | `SendMessage` + `TaskPushNotificationConfig` (`CreateTaskPushNotificationConfig`) | `Minutes to hours / days` | Deep research, human approval queues, or batch workflows where holding open HTTP/SSE connections will time out | > **Exam trap:** Never use synchronous `SendMessage` without webhooks or polling for a multi-hour workflow: load balancers and Cloud Run (`60 min` max HTTP timeout) will sever the connection and orphan client state. For `TaskPushNotificationConfig` webhooks, validate callback domain ownership and verify cryptographic **HMAC / JWT signatures** + replay nonces on incoming callbacks so attackers cannot spoof fake `TASK_STATE_COMPLETED` events! ### Agent Cards & Four Discovery Patterns An **`AgentCard`** advertises: identity (`name`, `description`, `version`, `provider`), endpoints (`supportedInterfaces` with `url` + `protocolBinding`), `capabilities` (`streaming`, `pushNotifications`, `extendedAgentCard`, `extensions`), `securitySchemes` + `securityRequirements` (OpenAPI-style OAuth2, OIDC, API key, mTLS), `defaultInputModes` / `defaultOutputModes`, and `skills[]` (`id`, `name`, `description`, `tags`, `examples`, per-skill MIME modes). 1. **Well-Known URI (RFC 8615):** `GET https:///.well-known/agent-card.json` for open/federated discovery. *(Version trap: early drafts showed `agent.json` and `tasks/send`; v0.3 standardized `agent-card.json` with `message/send`; v1.0 keeps `agent-card.json` and uses PascalCase `SendMessage` / `GetExtendedAgentCard`.)* 2. **Curated Enterprise Registry (**Google Cloud Agent Registry**):** Central governed catalog with IAM visibility controls, lifecycle snapshots, and security posture metadata — **the enterprise default** to prevent rogue agent sprawl. 3. **Authenticated Extended Agent Card:** Public card exposes safe metadata + `capabilities.extendedAgentCard: true` *(v0.3: `supportsAuthenticatedExtendedCard`)*; authenticated callers invoke `GetExtendedAgentCard` to unlock sensitive internal skills. 4. **Direct Configuration:** Pinned card URLs/configs injected via Terraform/env vars for tightly coupled internal microservices. --- ## 05. Intra-App `AgentTool` vs. Inter-App `A2A` Decision Matrix One of the highest-frequency **Professional Agentic Architect (PAA)** design questions is deciding when to compose specialists **inside one application** (ADK `sub_agents` transfer or `AgentTool` / agent-as-a-tool) vs. **across a network boundary with A2A**: | Architectural Dimension | **Intra-App (`sub_agents` / `AgentTool`)** | **Inter-App (`A2A Protocol`)** | | :--- | :--- | :--- | | **Deployment & team boundary** | Single codebase, single repo/CI pipeline, same runtime stack (e.g., all Python ADK in one Agent Runtime service) | Separate repositories, independent release cadences, **polyglot frameworks** (ADK ↔ LangGraph ↔ CrewAI ↔ SaaS vendor) or cross-org partners | | **Session & memory ownership** | **Shared `InvocationContext` & `session.state`** in-process (`sub_agents` share full conversation history; `AgentTool` runs a scoped child call in-process) | **Strict state opacity:** remote agent owns an isolated session store; sees **only** the explicit `Message` parts sent for that `contextId` / `taskId` | | **IAM & blast-radius isolation** | Runs under the **same container & service/agent identity** — one compromised dependency affects the whole agent | **Independent security boundary:** distinct **SPIFFE Agent Identity**, separate VPC-SC perimeter, independent IAM roles, mTLS, and OAuth token exchange | | **Latency & serialization cost** | Zero network hop (in-memory Python/Go call); lowest TTFT | HTTPS/gRPC serialization + auth verification + independent remote reasoning loop | | **Live audio / modality boundary** | Can keep native streaming context inside one process where supported | **Modality normalization boundary:** live voice front-doors (e.g., **CX Agent Studio** or **Gemini 3.8 Live** WebSockets) terminate real-time audio locally and delegate normalized `TextPart` / `DataPart` / `FilePart` payloads over A2A | **Decision rule:** Start **in-process (`sub_agents` / `AgentTool`)** when one team owns tightly coupled steps sharing the same data permissions. Cross into **A2A** the moment you need **independent scaling/deployment, polyglot frameworks, third-party SaaS agents, or least-privilege IAM / data-perimeter isolation** (e.g., preventing a customer-support front-door agent from directly holding HR or PCI database credentials). --- ## 06. The 5-Layer Agentic Interoperability Ecosystem Modern production architectures combine five specialized open protocols — **know what each layer owns so you never force one protocol to do another's job**: | Layer | Protocol(s) | Direction & Payload | Architectural Purpose | | :--- | :--- | :--- | :--- | | **1. Agent ↔ Tool / Data** | **MCP** & **OpenAPI** | Agent → Server (`tools/call`, JSON Schema) | Deterministic invocation of databases, APIs, files, and SaaS connectors | | **2. Agent ↔ Agent** | **A2A** | Client Agent ↔ Remote Agent (`Task`, `Message`, `Artifact`) | Stateful horizontal delegation, discovery, and multi-turn collaboration across opaque peers | | **3. Agent ↔ User Generative UI** | **A2UI (v0.8 / v0.9)**, **AG-UI**, **MCP UI** | Agent/Tool → Host Client UI (declarative JSON / event stream) | Safe, interactive rich UI rendering (forms, tables, approval cards, charts) instead of walls of Markdown text | | **4. Real-Time Multimodal Stream** | **Gemini Live API** (WebRTC / WSS) | Human ↔ Voice/Video Front-Door (PCM audio + video frames) | Sub-second full-duplex speech & camera interaction with server-side VAD barge-in | | **5. Agent ↔ Commerce / Money** | **AP2 (Agent Payments Protocol)** & **x402** | User → Agent → Merchant / API (`Mandates` / HTTP `402`) | Cryptographically verifiable human-authorized purchases (**AP2**) and frictionless M2M micropayments (**x402**) | ### Deep Dive: Generative UI Protocols (`MCP UI` vs. `AG-UI` vs. `A2UI`) Why not let a remote agent return raw HTML/JavaScript `