Production engineering for non-deterministic AI systems (PAA Sec 4 Core · CCAR-F D5): the 3-tier grading hierarchy, Gen AI Evaluation Service autoraters, Adaptive Rubrics & 2-Step Agent Simulation, pass@k vs. pass^k math, the 4-level test pyramid, OpenTelemetry gen_ai.* tracing & dual-attribution audit logs, the LM Feedback Flywheel & Agent Optimizer, Agent Runtime vs. Cloud Run vs. GKE selection, the 4-layer resilience stack, shadow/A/B rollouts, and 4-step enterprise FinOps ROI.
AgentOps (AI Evaluation, Observability, Runtime Engineering, and Production FinOps) is the operational control plane that transforms a probabilistic language model and multi-step tool loop into a reliable, auditable, and economically viable production service. Classical binary unit tests and HTTP 200 OK uptime checks cannot detect silent quality regressions, "right answer, wrong trajectory" policy violations, or heavy-tail cost explosions.
200 OK.p50) tokens causes 4×–15× budget overruns when multi-agent fan-out, thinking_level=HIGH loops, retries, and human review labor hit production scale.Treat prompts and model weights as probabilistic components, and treat your evaluation harness, simulation sandbox, OpenTelemetry gen_ai.* trace pipeline, runtime boundary, resilience stack, and FinOps ledger as the deterministic containment vessel around them. You cannot eliminate sampling variance inside the model, so you bound, measure, optimize, and price it at every boundary.
Enforce the 3-tier grading ladder, Adaptive Rubrics, 2-Step Agent Simulation, and Environment Simulation.
Gate releases on worst-case consistency (pass^k), trajectory_in_order_match, and L3 seam tests.
Stream OTel gen_ai.* + dual-attribution logs, Online Monitors, Failure Clusters, and Agent Optimizer.
Match runtimes (Agent Runtime / Cloud Run / GKE), isolate faults, run Shadow → A/B, and subtract Step 4 Review Tax.
p95 latency, and cost per task first.| Tier | Grader Class | What It Verifies | Speed & Cost | Reproducibility | Architectural Rule |
|---|---|---|---|---|---|
| Tier 1 | Deterministic Code Graders | Exact match, regex/forbidden strings, JSON Schema & Pydantic validation, AST/compile checks, execution unit tests, tool-call signature & sequence invariants | Milliseconds · $0 | 100% deterministic | Always prefer first. Never use an LLM judge to check syntax, field presence, or math that code can verify. |
| Tier 2 | Calibrated LLM-as-Judge | Open-ended semantic properties: factual faithfulness to source context, completeness, tone, reasoning coherence, policy adherence | Seconds · Token cost per eval | High (when anchored & calibrated) | Run only after Tier 1 passes; use anchored rubrics with CoT critique before score, calibrated against Tier 3. |
| Tier 3 | Human Domain Calibration | High-stakes clinical/legal/financial accuracy, ambiguous domain edge cases, golden-set labeling, and calibrating Tier 2 judges | Hours/days · $50–$200+/hr | Subject to inter-rater variance | Use to build the golden set, audit novel failure slices, and compute agreement ($\kappa \ge 0.75$) with Tier 2 judges. |
| LLM-as-Judge Bias | How It Distorts Evals | Production Engineering Mitigation |
|---|---|---|
| Verbosity (Length) Bias | Judges systematically reward longer, polite, multi-paragraph answers even when padded with fluff or subtle hallucinations. | Grade atomic factual claims (precision/recall) rather than holistic prose; add explicit conciseness penalties to the rubric. |
| Positional (Order) Bias | In pairwise A/B grading, judges favor whichever candidate appears in Slot A (primacy bias) or Slot B (recency bias). | Swap A/B order on every pair: evaluate both (A, B) and (B, A); award a win only if the same candidate wins both passes (otherwise record a tie). |
| Self-Preference Bias | A model favors outputs that match its own phrasing distribution and overlooks reasoning flaws it would make itself. | Use a different model family or higher capability tier as judge, and always execute the judge in a clean, isolated messages[] context. |
| Score Inflation & Scale Drift | Unanchored 1–10 or 1–100 scales cluster arbitrarily around 7–9 with high run-to-run variance. | Use decomposed binary (Pass/Fail) criteria or low-granularity 1–3 or 1–5 anchored rubrics with explicit behavioral definitions per integer. |
| Post-Hoc Rationalization | Emitting the numeric score first locks the judge's subsequent explanation into justifying the chosen number. | Require structured output where the step-by-step critique / reasoning field is generated before the score field. |
| Field | Type | Autorater Target |
|---|---|---|
extracted_claims | list[str] | Atomic factual claims in candidate answer. |
unsupported_claims | list[str] | Claims missing from source context. |
critique | str | Evidence check executed BEFORE scoring. |
score | int (1-3) | 1=Hallucination; 2=Incomplete; 3=Faithful. |
Never ask the same conversation thread that generated an artifact to grade its own work in the next turn—prior reasoning tokens stay in messages[] and induce severe self-confirmation bias. Equally important: never pay a Tier 2 LLM judge to check whether an output is valid JSON, contains 3 bullet points, or calls the right tool name; enforce those in Tier 1 deterministic code at $0 cost and 100% reliability.
| Evaluation Tooling Tier | Primary Mechanism | Speed & Variance | Best Fit & Architectural Boundary |
|---|---|---|---|
1. Deterministic Harness Replayadk conformance test --mode=replay |
Replays recorded LLM responses from adk conformance record baselines against live agent code, callbacks, and state transitions. |
< 5 seconds · $0 · Zero variance | Every git commit / pre-commit hook verifying that callback guardrails, state_delta logic, and routing code did not regress. |
2. In-Repo CI Gateadk eval · agents-cli eval run |
ADK runs *.test.json files and *.evalset.json evalsets against live models, grading test_config.json criteria such as tool_trajectory_avg_score (EXACT, IN_ORDER, ANY_ORDER), rubric criteria, and custom metrics. agents-cli now reads EvaluationDataset files (tests/eval/datasets/*-dataset.json) and grades traces with the metrics named in tests/eval/eval_config.yaml. |
1–5 minutes · Low cost | Pull-request blocking gates inside GitHub Actions or Cloud Build for code-first ADK repositories (tests/eval/). |
| 3. Cloud-Scale Managed Evaluation Gen AI Evaluation Service ( client.evals) |
Framework-agnostic managed service providing Adaptive Rubrics, Custom Autoraters, 2-Step Agent Simulation, Environment Simulation, and Online Evaluation Monitors. | Minutes to continuous · Managed scale | Cross-language/cross-framework agents (ADK, LangGraph, Managed Agents, Cloud Run), model migrations (e.g., to gemini-4-argon), synthetic stress-testing, and live production trace scoring. |
| Gen AI Evaluation Metric Family | Reference Needed? | How It Works | Canonical Built-In Metrics & Use Cases |
|---|---|---|---|
| Adaptive Rubrics (Recommended Default) |
No (Reference-free) | Dynamically generates a tailored set of pass/fail unit-test rubrics unique to each individual prompt in the dataset, then scores the candidate response against every check. | GENERAL_QUALITY, multi-turn instruction adherence across diverse open-ended prompts where one static checklist would be vague or brittle. |
| Static Rubrics | No | Applies an identical organization-wide pass/fail rubric across every prompt in the suite. | Mandatory regulatory disclaimers, brand voice rules, PII refusal formatting, policy invariants. |
| Agent Outcome & Grounding Autoraters | No (Context-grounded) | Specialized LLM-as-judge autoraters inspecting full conversation traces and retrieved/tool context. | FINAL_RESPONSE_QUALITY, HALLUCINATION (faithfulness to tool/RAG context), TOOL_USE_QUALITY, SAFETY. |
| Trajectory Metrics | Yes (Golden tool trace) | Deterministic comparison between the agent's executed tool sequence/arguments and a reference trajectory. | trajectory_exact_match (strict compliance order), trajectory_in_order_match (required milestones in order, extra read calls allowed), trajectory_any_order_match, trajectory_precision / recall. |
| Computation & Custom Autoraters | Configurable | Deterministic code metrics ($0 judge spend) or custom calibrated LLM autoraters registered in client.evals. |
EXACT_MATCH, ROUGE, JSON Schema / SQL AST checks, plus proprietary clinical, legal, or underwriting domain autoraters. |
Requires an immutable Agent Registry snapshot so every benchmark is reproducible—even when zero historical production chat logs exist:
Starting Prompts paired with hidden Conversation Plans (user persona, goal, constraints).max_turns.Intercepts outbound tool calls during evaluation to inject deterministic faults and edge cases without touching live production backends:
HTTP 503 Service Unavailable, 429 RESOURCE_EXHAUSTED, 12-second latency spikes, malformed JSON, and empty rows.Grading a refund or clinical agent solely with FINAL_RESPONSE_QUALITY or multi_turn_task_success will pass an agent that issued a refund without calling verify_customer_identity or check_return_eligibility first. Whenever policy mandates verification steps, always pair outcome autoraters with trajectory_in_order_match or trajectory_exact_match plus a forbidden-tool custom check.
The tiers above say when to evaluate. The table below says which tool to reach for, since the four options overlap but differ in dataset format, metrics and where the grading happens.
| Criterion | ADK eval tooling | agents-cli eval |
Gen AI evaluation service | Custom autoraters |
|---|---|---|---|---|
| Use case | Unit and integration tests inside an ADK repo; web UI capture of sessions into evals | Inner eval-fix loop and PR gates for agents scaffolded with agents-cli; regression diff and failure clustering from the terminal |
Cross-framework batch evaluation, user simulation, online monitors on live traces, Automatic Loss Analysis | Domain rules that built-in metrics do not encode (clinical, legal, financial, format or length policies) |
| Dataset format | *.test.json (single session) and *.evalset.json (multi-session, can hold a conversation scenario for user simulation); criteria in test_config.json |
EvaluationDataset JSON in tests/eval/datasets/ (default basic-dataset.json); metrics in tests/eval/eval_config.yaml |
EvaluationDataset built in the SDK, from files, or from sampled Cloud Trace data |
Defined alongside whichever harness runs them |
| Metrics | tool_trajectory_avg_score, response_match_score, final_response_match_v2, rubric-based quality criteria, hallucinations_v1, custom metric functions |
Built-ins such as tool_use_quality, multi_turn_task_success, multi_turn_trajectory_quality, final_response_match, hallucination, safety (agents-cli eval metric list prints all) |
Adaptive and static rubrics, agent autoraters, trajectory and computation metrics (see the taxonomy above) | Code metrics (custom_function) or LLM-as-judge metrics (prompt_template plus optional judge_model) |
| Where it runs | Developer machine or CI runner (adk eval, pytest with AgentEvaluator); judge-based criteria call a model |
Agent inference runs locally; grading goes through the Gen AI evaluation SDK | Managed Google Cloud backend through SDK, console or scheduled online monitor | Code metrics run in the CLI process by default; add execution: remote in eval_config.yaml for a sandboxed server-side code metric |
| Scale | Tens to low hundreds of cases per run | Same order as ADK per CI run; eval dataset synthesize grows coverage |
Large batch datasets plus continuous sampling of production traffic | Bound by the host harness and, for judges, by model quota |
| Setup | Low: capture sessions in adk web, save as evalset, set thresholds |
Low: scaffolded tests/eval/ folder; edit dataset and config |
Low to medium: point the SDK or console at a dataset or deployed agent | Medium to high: write, version and calibrate against human labels |
| Cost driver | Agent tokens plus judge tokens for LLM criteria | Agent tokens plus judge tokens | Judge model usage; check the Agent Platform pricing page | Free compute for code metrics; judge tokens multiplied by judge_model_sampling_count (1 to 32) |
agents-cli eval generate and eval run no longer read tests/eval/evalsets/*.evalset.json. Run agents-cli scaffold upgrade to convert them into tests/eval/datasets/, then hand-port any seeded session state, which the migrator does not carry over. ADK's own adk eval and AgentEvaluator still use the evalset schema, so the two formats coexist across repos. The scale and setup rows above are rules of thumb, not published limits.
execution: remote or review the code like any other dependency.Each gate is cheaper and faster than the next, so most regressions die early and only release candidates pay for simulation and judge-heavy grading.
agents-cli eval run (or adk eval / pytest with AgentEvaluator in an ADK-only repo) in CI, for example a pipeline created with agents-cli infra cicd. Cases replay recorded or mocked tool outputs, and the metrics are trajectory checks plus cheap code metrics, so the gate stays fast and deterministic enough to block merges.agents-cli eval dataset synthesize or the Gen AI evaluation service user simulation) with environment simulation for tool faults. Grade with multi_turn_task_success, multi_turn_trajectory_quality, hallucination, safety and any calibrated custom autoraters, and enforce an absolute floor per metric.agents-cli eval compare prod_results.json rc_results.json. Fail the build when a critical slice drops beyond the agreed tolerance, even if the averages still clear their floors, and run agents-cli eval analyze --eval-result rc_results.json to cluster what broke.Metric floors, regression tolerances and canary percentages are team choices, not Google defaults, and the result file names above are placeholders.
A golden set is the frozen, versioned dataset every gate grades against. Cover each case type below and tag every case with its slice and origin so regressions can be traced.
| Case type | What it catches | How to express it | Typical metrics |
|---|---|---|---|
| Trajectory cases | Skipped verification steps, wrong tool, bad arguments | Single prompt (prompt field in agents-cli, one turn in an ADK evalset) plus expected tool calls; assert required milestones in order, not full transcripts |
tool_trajectory_avg_score (IN_ORDER), tool_use_quality, forbidden-tool code check |
| Continued-conversation cases | Mid-session regressions after earlier turns set context | agent_data turns ending with a user message (agents-cli) or a multi-turn evalset session (ADK) |
multi_turn_tool_use_quality, multi_turn_trajectory_quality |
| Simulated multi-turn scenarios | Clarification loops, users who change their mind, goal abandonment | Starting prompt plus conversation plan and persona (eval dataset synthesize --instruction ..., ADK conversation scenario, service user simulation) |
multi_turn_task_success |
| Abstention cases | Hallucinating when retrieval is empty, answering out-of-scope or unauthorized requests | Prompts with no supporting data or outside the agent's remit; expected outcome is a decline or a clarifying question | hallucination, custom refusal check |
| Adversarial cases | Prompt injection in tool output, jailbreaks, PII extraction | Injected strings in mocked tool results and user turns; expected outcome is refusal and no forbidden tool call | safety, forbidden-tool code check |
| Fault cases | Broken retry, fallback and escalation paths | Environment simulation returning 429, 503, timeouts, empty rows |
Trajectory milestones on the fallback path |
| Loss-analysis backfill | Production failure classes the set never covered | Representative traces from Automatic Loss Analysis or agents-cli eval analyze clusters, scrubbed of PII and converted to cases |
Whatever metric flagged the cluster |
agents-cli eval optimize see during tuning, so prompt fixes generalize instead of memorizing the set.eval_config.yaml next to the agent so a revision and its gate always travel together.pass@k vs. pass^kpass@k — At-Least-One SuccessFormula: $\text{pass}@k = 1 - (1 - p)^k$ (where $p$ is single-trial success rate).
pass^k — Consistent Every-Run ReliabilityFormula: $\text{pass}^k = p^k$ (chance that every run across $k$ trials succeeds).
| Single-Run Accuracy ($p$) | pass@1 / pass^1 |
pass@5 ($1 - (1-p)^5$) |
pass^5 ($p^5$) |
pass^10 ($p^{10}$) |
Production Readiness Verdict |
|---|---|---|---|---|---|
| 0.70 (70%) | 70.0% | 99.8% | 16.8% | 2.8% | Viable only when a compiler/test runner picks 1 of 5 samples; fails 83% of 5-run live suites. |
| 0.85 (85%) | 85.0% | 99.99% | 44.4% | 19.7% | Fails more than half the time across 5 consecutive runs. |
| 0.90 (90%) | 90.0% | 99.999% | 59.0% | 34.9% | Classic trap: looks like "90% accurate," yet 41% of 5-run batches contain a failure. |
| 0.95 (95%) | 95.0% | >99.999% | 77.4% | 59.9% | Minimum single-turn baseline before adding deterministic validation gates. |
| 0.99 (99%) | 99.0% | >99.999% | 95.1% | 90.4% | Production-grade step consistency (achieved by combining strict schemas + code guardrails). |
The exact same exponential law governs a single run of a multi-step sequential workflow. If an agent must complete $M$ sequential steps (e.g., Classify Intent → Rewrite Query → Select Tool → Validate Payload → Synthesize Answer) with step reliabilities $p_1, p_2, \dots, p_M$, overall trajectory reliability is the product $\prod_{i=1}^M p_i$:
SequentialAgent) and strict schema parsers, the probabilistic chain shrinks from $0.92^6$ (60.6%) to $0.92^3$ (77.9%).pass@5 to Justify Launching an Autonomous AgentIf an engineering team reports "Our support agent achieved 99.9% pass@5 on the benchmark" when single-run accuracy is 85%, do not approve launch. A live customer does not see the best of 5 parallel timelines, and you cannot execute mutating tools (like issuing refunds or canceling subscriptions) 5 times hoping one succeeds. Gate unattended agents strictly on pass^k.
Schemas, parsers, callbacks, and adk conformance test --mode=replay (ms, $0, every commit).
Isolated router, extractor, rewriter, or RAG synthesis call against a frozen golden dataset (per PR / model bump).
Multi-turn tool loops, state handoffs, and Environment Simulation fault injection (pre-release / nightly).
Shadow runs, canary slices, and sampled async OnlineEvaluator autoraters on live traffic (continuous).
| Pyramid Level | Target Under Test | Dependencies & Mocks | Cadence & Cost | Primary Defects Caught |
|---|---|---|---|---|
| Level 1: Deterministic Unit & Replay Tests | Tool JSON schemas, Pydantic validators, prompt builders, PII scrubbers, adk conformance test --mode=replay |
LLM 100% mocked / replayed from recorded fixtures; pure deterministic execution | Every save / pre-commit · Milliseconds · $0 | Missing template variables, broken callbacks/state_delta, parser crashes on malformed JSON, off-by-one retry logic. |
| Level 2: Single-Step Component Evals | A single LLM hop in isolation (intent classifier, entity extractor, query rewriter, answer generator) | Live LLM + frozen golden input and pre-recorded context snapshot | Every PR touching prompts/models · Seconds · Low cost | Classification drift, extraction regressions, prompt instruction conflicts, single-turn hallucination. |
| Level 3: Trajectory & Seam Integration | End-to-end multi-turn loop across 2–8 steps: tool selection, state handoffs, subagent coordination, error recovery | Live LLM + Environment Simulation / sandbox tools (injected 503, 429, and empty-result fixtures) |
Pre-release gate / nightly · Minutes · Moderate cost | Integration seam bugs, infinite tool loops, swallowed tool errors, observation flooding, context collapse. |
| Level 4: Live Canary & Online Monitors | Full deployed production system serving real customer traffic | 100% live stack (canary cohort, shadow fork, or 1–5% sampled OnlineEvaluator) |
Continuous in production · Fixed % of prod budget | Input/data drift, unseen user phrasing, third-party API latency/schema shifts, cache invalidation bugs. |
An integration seam is the contract boundary where the output of Component A becomes the input of Component B. In production, every component can score >95% on isolated component evals while the end-to-end system fails on >40% of real workflows because Level 2 tests feed clean, hand-crafted inputs to each component rather than the messy, truncated, or slightly shifted outputs actually emitted by upstream components.
| Integration Seam Failure Mode | Why Component A Passes Level 2 | How Component B Fails Across the Seam | Architectural Remediation |
|---|---|---|---|
| Implicit Format & Schema Drift | Entity extractor returns dates as "Oct 26, 2026" or wraps JSON in Markdown fences (marked correct by a lenient L2 grader). |
Downstream SQL/API tool requires strict ISO-8601 2026-10-26 or raw JSON and throws a hard validation error. |
Enforce strict structured outputs (output_schema / JSON Schema) at every seam and verify with L1/L3 contract assertions. |
| Unbounded Tool Output & Silent Truncation | Log-fetching tool dumps 40,000 raw tokens into context (diluting model attention), or middleware naively slices the first 4,000 characters. | Attention degrades across turns (context rot) or the stack trace at the bottom is cut off, causing a confident "No errors found." | Truncate structurally (keep head + tail + error lines) and attach explicit metadata (truncated: true, omitted_lines: 850). |
| State Loss During Context Compaction | Sliding-window summarizer compresses 12 prior turns into a concise 3-sentence prose summary. | Summarizer drops an exact UUID, booking ID, or policy flag read in Turn 2 that Turn 14 needs to execute a tool safely. | Persist exact IDs, constraints, and slot values in structured Session state (session.state), never solely in summarized prose. |
| Swallowed Tool Errors (Empty vs. Failure) | Tool wrapper catches an upstream 504 Gateway Timeout and returns [] or "" so the Python process doesn't raise. |
The LLM treats [] as factual ground truth and tells the customer "You have no active subscriptions." |
Return explicit structured error blocks (is_error: true, error type, retryability) so the model distinguishes outage from empty data. |
| Coupled Stage Distribution Shift | Query rewriter expands a 6-word user question into a 90-word keyword-stuffed paragraph (looks thorough in L2). | Dense embedding model was tuned on short questions; top-5 retrieval recall drops by 30% on 90-word inputs. | Test Query Rewriter + Retriever as a coupled pair in Level 3 seam evals measuring end-to-end Recall@K. |
Do not assert an exact character-for-character transcript or rigid read-tool ordering when multiple valid search paths exist. Instead grade three invariants: (1) Final sandbox state (did the database/ticket reach the right state?), (2) Safety & ordering milestones (trajectory_in_order_match: e.g., verify_caller_identity MUST execute before issue_refund, and delete_tenant is NEVER called), and (3) Fault-injection recovery (how the agent behaves when Environment Simulation returns is_error: true or HTTP 503).
gen_ai.* Distributed TracingInstrument every workflow with OpenTelemetry Semantic Conventions for Generative AI (gen_ai.* attributes across invoke_agent, chat, and execute_tool spans) so traces render uniformly across Google Cloud Trace, Agent Registry Observability, and vendor-neutral APMs:
trace_id, span_id, session_id, provider request-id, workflow_name.gen_ai.request.model (pinned ID), prompt_template_version, tool_schema_hash, rag_index_version.input_tokens, output_tokens, thinking_tokens, thinking_level, cached_tokens, finish_reason / stop_reason, retry_count, fallback_used.Production observability must diagnose failures and prove actor provenance without turning logs into a compliance liability:
roles/cloudtrace.user for debugging without read access to conversation content.OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=EVENT_ONLY (never true, which collects nothing under the latest gen_ai conventions) and ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS=false; redact upstream with Sensitive Data Protection templates or Model Armor and restrict the log bucket (roles/logging.privateLogViewer).is_agentic=true): Cloud Audit Logs tag agent actions with is_agentic=true and record both the cryptographic SPIFFE Agent Identity (spiffe://...) and the initiating human principal—eliminating "confused deputy" blind spots.Platform signals (request logs, container logs, resource metrics) arrive without any agent code change. Agent-level gen_ai.* spans and prompt content do not: they appear only after you turn on ADK's built-in OpenTelemetry instrumentation (ADK 1.17.0 or later) and export it to Cloud Trace through the Telemetry (OTLP) API. An empty Traces tab on a healthy agent almost always means this step was skipped.
| Runtime | Available without extra setup | Requires explicit opt-in |
|---|---|---|
| Agent Runtime | Agent logs in Cloud Logging; the Observability tab's Usage view (container CPU and memory allocation) | Agent traces: set GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY=true, pass --otel_to_cloud to adk deploy agent_engine, or build the app with AdkApp(..., enable_tracing=True) |
| Cloud Run | Request logs, container stdout/stderr, request latency, instance count, CPU and memory metrics | ADK spans: initialize ADK's telemetry module with Google Cloud exporters (get_gcp_exporters(enable_cloud_tracing=True)) or an OTLP exporter targeting telemetry.googleapis.com; adk web --otel_to_cloud covers the local dev server |
| GKE | Container logs and system metrics through the default Cloud Logging and Cloud Monitoring integration | Same ADK telemetry module or OTLP exporter as Cloud Run; GKE agents also need an Agent Registry entry before they appear in Topology |
The writer identity (usually the runtime service account) needs roles/telemetry.tracesWriter, roles/logging.logWriter and roles/monitoring.metricWriter. The ADK instrumentation guide recommends:
GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY=true (Agent Runtime only)OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental (latest gen_ai conventions)OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=EVENT_ONLY (messages as events, never true)ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS=false (no prompt text or PII in span attributes)OTEL_INSTRUMENTATION_GENAI_COMPLETION_HOOK=upload, OTEL_INSTRUMENTATION_GENAI_UPLOAD_FORMAT=jsonl and OTEL_INSTRUMENTATION_GENAI_UPLOAD_BASE_PATH=gs://... so images and large documents land in a bucket instead of inside spans.gen_ai.input.messages, gen_ai.output.messages, gen_ai.system_instructions and gen_ai.tool.definitions from the inference event. Switching content capture off also switches off online evaluation, so address PII by redacting upstream instead.The agent's Traces tab offers a Session view (multi-turn conversations with session token totals and evaluation scores), a Trace view (one end-to-end request) and a Span view (individual model calls, API requests and tool executions). Open one trace for a single turn and walk it top-down.
| Order | Span to inspect | Healthy pattern | Investigate when |
|---|---|---|---|
| 1 | Root invoke_agent |
One trace per user turn; total duration roughly equals the sum of child spans | A gap before the first child (cold start or blocking initialization), or one turn split across several traces (context propagation broken) |
| 2 | LLM spans (call_llm wrapping generate_content {model}) |
A small, stable number of calls per turn; input tokens grow modestly turn over turn; each call ends in a final answer or a well-formed tool call | Input tokens climbing every turn (history bloat), output cut off at the token limit, repeated retries after 429 |
| 3 | Tool spans (execute_tool; MCP calls carry mcp.method.name = tools/call and gen_ai.tool.name) |
Independent tools overlap in time; compact results; no error.type |
Same tool with the same arguments three or more times in a row, error.type set, or a serial waterfall of multi-second calls |
| 4 | Sub-agent handoffs (nested invoke_agent spans) |
A shallow parent-to-specialist tree that matches the intended design and the Topology tab's edges | A to B to A ping-pong, unexpected specialists, or depth that grows with every turn |
| 5 | Retrieval spans (search and RAG tools appear as execute_tool) |
Fast, non-empty results whose content shows up in the final answer | Zero or irrelevant results followed by a confident answer (hallucination risk) |
| 6 | Guardrail spans (Model Armor, Preview) | Screening adds little latency and allows benign traffic | An empty or refused answer that a Model Armor interception explains, which rules out a retrieval miss |
| 7 | Evaluation tab on the trace | Online monitor scores and rationales agree with what you just read | Low scores on a trace that looks clean (check the rubric) or high scores on a broken one (check the autorater) |
Agent Registry's Observability tab (Overview, Evaluation, Models, Tools, Usage, Logs) plus the agent's Traces and Topology tabs map production symptoms to root causes. The Models view isolates p95 latency, errors, quota failures and tokens per model; the Tools view shows p95 latency, call counts and error rates per tool, plus how often a turn called no tool at all.
| Symptom | Likely cause | Where to look | Fix |
|---|---|---|---|
Tool latency spike (turn p95 jumps while LLM time stays flat) |
Unpooled database connections, cross-region hops, or independent reads executed one after another | Tools view p95 per tool; Traces waterfall with long or serial execute_tool spans |
Co-locate runtime and data, use MCP Toolbox connection pooling, fan out independent reads (ParallelAgent), set per-tool timeouts, cache hot lookups |
| Reasoning or tool-call loops | Ambiguous tool errors (an empty string instead of actionable feedback) or cyclic sub-agent routing | Traces: 8 or more consecutive call_llm and execute_tool pairs with identical or oscillating arguments; Topology for A to B to A edges |
Return structured error hints (ReflectAndRetryToolPlugin), cap iterations (max_llm_calls, max_iterations), break cycles with deterministic state flags (per-cause fixes in the runaway loop triage table below) |
| Ignored tools ("no-tool interactions") | Vague or overlapping tool descriptions, so the model answers from parametric memory | Tools view no-tool rate alongside rising HALLUCINATION scores on domain questions |
Sharpen tool descriptions and schemas; add trajectory_in_order_match CI gates |
| Prefill or thinking bottleneck | thinking_level="HIGH" on routine hops, or volatile text at the top of the prompt defeating prefix caching |
Models and Traces views: thinking_tokens dominating generation time, high TTFT with cached_tokens = 0 |
Route simple steps to gemini-3.5-flash-lite (minimal) or gemini-3.8-flash (low); move volatile state after static_instruction |
| Rising token cost | Unbounded conversation history, large tool payloads, many tool schemas sent on every turn | Overview token usage (input vs. output); per-span input tokens growing turn over turn | ADK context compaction (token-based or sliding window), context caching, pruned tool results, tools and skills loaded on demand |
| Hallucinated answers | Empty or weak grounding, or the agent answering despite an empty retrieval | Evaluation view hallucination rate; retrieval span output compared with the final response | Ground on Agent Search or RAG Engine, add reranking, instruct the agent to abstain when retrieval returns nothing |
| Retrieval misses | Dense-only search missing exact codes and acronyms, or chunks stripped of their section headings | Retrieval execute_tool spans returning irrelevant or zero results |
Hybrid (dense plus keyword) retrieval, chunking that keeps ancestor headings, reranking, corpus refresh |
| Quality drift after a model or prompt change | New model version or prompt edit changes tool selection, formatting or refusal behavior | Evaluation dashboard trends across agent versions; Automatic Loss Analysis clusters for the new version | Roll back to the prior pinned version, add failing traces to the golden set, re-run the optimizer before re-promoting (see feedback flywheel) |
429 / quota errors |
Burst traffic or runaway loops exhausting model quota (RESOURCE_EXHAUSTED) |
Models view quota failures; Cloud Logging entries with 429 |
Full-jitter backoff (see resilience stack), loop caps, concurrency limits, Provisioned Throughput for predictable capacity |
| Cold starts | No idle instance when a request arrives; heavy imports and client setup at startup | Gap before the first child span on the first turn; instance count rising during spikes | Raise min_instances (Agent Runtime) or --min-instances (Cloud Run) to cover baseline load, lazy-load heavy libraries, slim images; for async ADK agents on Agent Runtime start container_concurrency at a multiple of 9 (for example 36) and watch memory |
| Traces tab or online monitor empty | Telemetry never enabled, or content capture disabled | Cloud Trace for gen_ai.* attributes; Logs Explorer filtered on resource.labels.online_evaluator |
Apply the opt-in settings above; confirm monitor filters with Initial Inspection |
The loop row above names the shared symptom. Five different causes produce it, and each one needs its own fix. The common rule: a loop must never end on model judgment alone. An iteration cap, a terminal route or a budget has to end it.
| Loop cause | Trace signal | Deterministic fix |
|---|---|---|
LoopAgent with no exit path |
The same child agent spans repeat in the same order until something external stops the run; LoopAgent never decides to stop by itself |
Set max_iterations on every LoopAgent and give the checker an exit tool that sets tool_context.actions.escalate = True (a custom agent can instead yield an event with EventActions(escalate=True)) |
| Graph back-edge with no terminal branch | Two graph nodes alternate in the trace and the run ends only on a timeout or budget | Keep an iteration counter in state and have the router node emit a terminal route (for example DONE) once the counter reaches its cap; point DEFAULT_ROUTE, which fires when no other route on that node matches, at a terminal or fallback node rather than at the back-edge |
| Tool error retried forever | execute_tool spans with error.type set, each followed by an LLM call that reissues the same tool with the same arguments |
ReflectAndRetryToolPlugin(max_retries=...) caps retries per tool (default 3, tracked per invocation) and returns structured guidance to the model; for errors that cannot self-heal (401, 403, 404), a before_tool_callback that counts calls per tool-and-arguments key returns a dict once the limit is hit, which skips the tool and hands the model a "do not retry" result |
| A2A ping-pong delegation | Nested spans alternate between two remote agents inside one distributed trace; Topology shows edges in both directions | Keep delegation acyclic (a specialist never delegates back to its caller), carry a hop or depth counter in the delegated task and refuse past a fixed depth, and give each task a deadline and token budget |
| Over-strict critic | Every run uses the full max_iterations even on drafts a reviewer would accept; the escalate event never appears |
Calibrate the critic rubric against golden examples labeled acceptable, and when the cap is hit return the best draft flagged as partial or hand off to a human instead of failing silently |
tool_context.actions.escalate = True when the draft passes, and always construct the loop with a hard cap, for example LoopAgent(name="refine", sub_agents=[critic, refiner], max_iterations=5), so the loop stops even if escalation never fires.max_iterations or max_llm_calls so the agent "has room to finish" leaves the missing exit signal in place and makes each runaway cost more.DEFAULT_ROUTE to the loop's back-edge means any unexpected route value, including a typo, re-enters the cycle.| SLI Dimension | Key Production Metrics | Typical SLO Target | Why Mean/Average Hides Problems |
|---|---|---|---|
| 1. Responsiveness | Streaming TTFT (p50, p95, p99); End-to-End Turn Latency; Tool Execution Latency |
p95 TTFT < 800 msp95 Turn < 5.0 s |
Fast 1-turn cache hits hide 25-second p99 multi-tool stalls that cause users to abandon sessions. |
| 2. Pipeline Reliability | Upstream 429/5xx rate; Tool error rate (is_error: true); Fallback trigger rate; Truncation rate (stop_reason: "max_tokens") |
< 0.5% unhandled error< 2.0% fallback rate |
HTTP 200 OK responses with stop_reason: "refusal" or "max_tokens" look healthy to standard web monitors. |
| 3. Cost & Cache Efficiency | Prompt Cache Hit Ratio (cached_tokens / total_input); Token Cost per Resolved Task; p95 Turns per Session |
Cache Hit > 75%Cost/Task < $0.08 |
A broken cache breakpoint (0% hit rate) 5×–10×s input token spend without raising a single functional error. |
| 4. Semantic Quality | Sampled Online Evaluation Monitor pass rates (HALLUCINATION, TOOL_USE_QUALITY) on 1%–5% live traces; Implicit UX signals |
Faithfulness ≥ 96%Escalation < 12% |
Semantic accuracy can degrade silently over weeks due to data drift while latency and uptime remain 100% green. |
| Production Drift Type | Underlying Cause | Diagnostic Signature in Telemetry | Correct Engineering Fix |
|---|---|---|---|
| Input / Data Drift (The World Changed) |
Real-world customer queries, new product SKUs, updated regulations, or external API schemas shift away from your static golden set and RAG index. | Pinned model ID and prompt hash are unchanged, infrastructure SLIs are green, but retrieval similarity drops, out-of-domain queries rise, and Online Monitor faithfulness declines gradually. | 1. Track query embedding distribution shift & retrieval score histograms. 2. Refresh the RAG / Agent Search corpus. 3. Feed SDP-scrubbed failing traces into the LM Feedback Flywheel to expand the golden set. |
| Model / Provider Drift (The System Changed) |
Pointing production at a floating model alias (e.g., -latest), an unannounced provider update, or an unvalidated prompt/tool-schema deployment. |
Sudden step-change inflection in output token length, refusal rate, JSON parse failures, or tool-call frequency across all query categories at once. | 1. Pin immutable model snapshot IDs and Agent Registry versions. 2. Gate cutovers on L2/L3 evals (using Data-Driven Prompt Optimizer for model family migrations). 3. Keep the prior pinned ID ready for instant rollback. |
Every row maps a signal that already exists in Agent Platform or Cloud Monitoring to an example condition. Tune each threshold to your own SLOs and baseline.
| Signal | Where it comes from | Example alert condition (directional) |
|---|---|---|
Turn latency p95 |
Overview latency percentiles (p50, p95, p99) | p95 above the latency SLO (for example 8 s for chat) for 10 minutes |
| Agent error rate | Overview error rate | Above 2% of invocations over 5 minutes |
| Tool error rate | Tools view per-tool error rate; execute_tool spans with error.type |
Any single tool above 5% errors for 15 minutes |
| Quota failures | Models view quota failures; 429 log entries |
Sustained 429 responses for 5 minutes, or more than 1% of model calls |
| Input tokens per turn | Overview token usage; per-span token counts | p95 input tokens at 2× the trailing 7-day baseline |
| No-tool interaction rate | Tools view | A 50% rise against the pre-deploy baseline |
| Online evaluation score | aiplatform.googleapis.com/online_evaluator/scores, filtered by metric.labels.evaluation_metric_name |
Mean task success below 0.8 over a 30-minute window (the threshold used in the public sample policy) |
The numbers above are starting points, not Google-published limits. Only the metric type and its evaluation_metric_name label come from the documentation. Set each threshold from your own SLOs and two to four weeks of baseline data, then revisit after every model or prompt change.
Offline evals prove a release was good on the day it shipped. Online Monitors keep scoring production traffic afterwards, and quality alerts turn a falling score into a page, so both drift types from the table above surface without a manual spot check.
OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental, OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=EVENT_ONLY) so each trace carries gen_ai.agent.name, gen_ai.conversation.id and the inference messages.OnlineEvaluator can attach it to any agent in the project. Restrict that permission to authorized administrators.gen_ai.* attributes, re-check filters with Initial Inspection, and search Logs Explorer for resource.labels.online_evaluator error entries.Drift detection tells you that quality moved; the LM Feedback Flywheel ensures every production failure class is discovered automatically, grouped by root cause, fixed programmatically without "prompt whack-a-mole," and permanently guarded against in CI/CD:
OnlineEvaluator) sampling 1%–5% of live Cloud Trace spans asynchronously with reference-free autoraters (HALLUCINATION, TOOL_USE_QUALITY, GENERAL_QUALITY). IAM rule: restrict monitor creation/editing IAM permissions because online monitors read trace payloads and incur judge token spend.trace_id payloads to BigQuery via Cloud Logging sinks after passing through Sensitive Data Protection (SDP) de-identification templates so fixtures never leak PII or secrets.Frequency × Severity).evalset cases with recorded tool fixtures, asserting end states and required safety milestones (trajectory_in_order_match) rather than exact prose transcripts.| Optimizer Engine | Inputs Required | Optimization Modes & Mechanism | When to Choose |
|---|---|---|---|
| Zero-Shot Optimizer ( client.prompt_optimizer.optimize_prompt) |
Only the raw prompt / system instruction (zero labeled examples or eval dataset required). | optimization_mode="zero_shot": Analyzes prompt structure against model best practices and immediately returns suggested_prompt plus applicable_guidelines (fixing ambiguous constraints, missing role boundaries, and redundant rules). |
Cold-start prototyping, cleaning up ad-hoc developer prompts in seconds, or preparing an initial baseline before collecting golden data. |
Data-Driven Optimizer & Agent Optimizer(Custom Optimization Job) |
System instructions + prompt template + labeled/scored sample dataset + Gen AI Evaluation metrics + target model ID. | Runs an iterative closed-loop job (evaluate → analyze failure clusters → mutate prompt/demos → re-evaluate on target model) across three modes:• instruction: Rewrites system instructions only.• demonstration: Selects optimal few-shot examples (or populates Example Store).• instruction_and_demo: Jointly optimizes instructions and demonstrations. |
1. Fixing ranked Failure Clusters without regressing existing evals. 2. Model migrations (adapting prompts to gemini-4-argon or distilling a frontier prompt down to gemini-3.5-flash-lite / Gemma 4). |
Apply Rule 0 (Managed by Default) when choosing Google Cloud compute runtimes: start at the most managed runtime that satisfies your language, protocol, accelerator, and isolation constraints—escalating to GKE only when self-hosted open-weight models on GPUs/TPUs or custom Kubernetes kernel sandboxes require it.
| Architectural Dimension | Agent Runtime (formerly Agent Engine) | Cloud Run (Services & Jobs) | Google Kubernetes Engine (GKE) | Managed Agents API (Preview) |
|---|---|---|---|---|
| Primary Workload Fit | Stateful Python/ADK, LangGraph, CrewAI, or A2A orchestrators (reasoningEngines) needing turnkey agent ops |
Custom Streamable HTTP MCP servers, polyglot agents (TS, Java, Go, Python), webhook/Pub/Sub event-driven agents | Self-hosted open-weight models (Gemma 4, Llama) on GPUs/TPUs, dense multi-agent meshes, strict kernel-isolated sandboxes |
Declarative autonomous coding, research, and file-processing agents (Agents API + Interactions API) |
| Operational Overhead | Lowest (fully managed agent container lifecycle) | Low (serverless container-to-URL, scales to zero) | Highest (cluster, node pools, networking, controller upgrades) | Zero (Google runs the loop and no-egress Linux container) |
| State & Memory Model | Built-in: Native Agent Platform Sessions, Memory Bank, Example Store, and Code Execution sandbox | Stateless / Ephemeral RAM: Local memory vanishes on scale-in; attach external Sessions (agentengine://), Memorystore, Firestore, or Cloud SQL |
StatefulSets, Agent Substrate (<1s auto-snapshot/restore of memory & files on GKE Standard), or external managed DBs |
Server-side state chained via previous_interaction_id and environment_id (global endpoint) |
| Scaling & Duration | Managed autoscaling; min_instances for zero cold start; supports long-running / multi-day async workflows |
Scales 0 → N rapidly; HTTP timeout up to 60 minutes; Cloud Run Jobs for batch backfills |
Granular HPA/VPA, custom queue metrics, GPU time-slicing, Spot VMs, Committed Use Discounts (CUDs) | Synchronous streaming or background=True multi-hour async execution |
| Hosts Custom MCP Servers? | No (consumes MCP/A2A, does not host standalone MCP servers) | Yes — Primary serverless choice for Streamable HTTP /mcp servers |
Yes — High-density, stateful, or private in-cluster MCP fleets | No (supports server-side tools only) |
| Custom GPUs / Self-Hosted Models? | No (calls managed model APIs) | Limited (serverless GPU inference for lightweight bursts) | Yes — Primary choice for vLLM/TGI serving of Gemma 4 on GPUs/TPUs (never proprietary Gemini weights!) | No |
| Code Sandbox Isolation | Managed Code Execution micro-container (<1s startup, 100 MB I/O, up to 14-day TTL) |
Call external Agent Platform Code Execution sandbox API | Agent Sandbox on GKE (SandboxTemplate + SandboxWarmPool pre-warmed gVisor pods; blocks hostNetwork/hostPath) |
Built-in isolated Linux container (outbound network disabled by default) |
The matrix above names the best fit. The table below compares the three general-purpose runtimes on the dimensions that usually decide an architecture review.
| Dimension | Agent Runtime | Cloud Run | GKE |
|---|---|---|---|
| Managed agent primitives | Sessions, Memory Bank, Code Execution sandbox and Agent Identity available to the deployed agent; bidirectional streaming in Preview | None built in; call Agent Platform Sessions or Memory Bank through the SDK, or bring Firestore, Memorystore or Cloud SQL | None built in; wire state yourself or use Agent Sandbox and Agent Substrate |
| Scale-to-zero | min_instances defaults to 1, so one warm instance stays up |
Scales to zero when idle unless you set minimum instances | Pods scale with HPA and nodes with the cluster autoscaler; the cluster itself stays provisioned |
| Cold start and concurrency | Raise min_instances to absorb baseline load; container_concurrency defaults to 9 (one per agent process), and async ADK agents should start at a multiple of 9 such as 36 (too high risks OOM) |
Default 80 concurrent requests per instance for console deploys (80 × vCPU with gcloud or Terraform), configurable up to 1,000; minimum instances and startup CPU boost cut cold starts | Warm pods have no cold start; new pods pay image pull and readiness time, which you tune |
| Accelerators | None exposed; calls hosted models | NVIDIA L4 (24 GB) and NVIDIA RTX PRO 6000 Blackwell (96 GB) GPUs, with scale to zero; L4 needs at least 4 CPU and 16 GiB memory | Full GPU and Cloud TPU node pools for self-hosted models |
| Runtime contract | Custom containers listen on 0.0.0.0:8080; /api/reasoning_engine (unary query) and /api/stream_reasoning_engine (streamQuery) are optional but required for SDK use, and the streaming one for the console playground; AdkApp is the reference implementation |
Any container that serves on $PORT (HTTP, gRPC, WebSockets) |
Standard Kubernetes Pod spec: sidecars, DaemonSets, StatefulSets |
| Sandboxing untrusted code | Managed Code Execution sandbox | Per-instance sandboxed execution environment; call the managed Code Execution sandbox for model-written code | GKE Sandbox (gVisor) and Agent Sandbox with warm pools |
| Networking | Private Service Connect interface for private egress into your VPC (needs at least a /28 subnet); VPC Service Controls supported; with Agent Gateway it needs VPC network egress through the agent connectivity template (gateways created after 2026-09-08) |
Direct VPC egress, internal ingress, VPC Service Controls, IAP | Native VPC subnets, private clusters, NetworkPolicy, service mesh |
| Identity | Agent Identity (GA) per agent, or a service account | Per-service service account; IAP for callers | Workload Identity Federation for GKE maps Kubernetes service accounts to IAM principals |
| Ops burden | Lowest: packaging, revisions and telemetry hooks managed | Low: no nodes, you own the container | Highest: cluster upgrades, node pools, networking, controllers |
| Billing shape | Compute and memory of running instances, including the warm baseline; see Agent Platform pricing | Request-based or instance-based billing; GPUs billed per GPU second | Cluster fee plus nodes (Standard) or requested Pod resources (Autopilot) |
The Agent Runtime scaling guide shows one benchmark (300 concurrent requests, default min_instances=1) averaging about 4.7 seconds on a cold run versus about 0.4 seconds warm, and roughly 1.4 seconds once min_instances=10. Cloud Run documents about 5 seconds to start a GPU instance with drivers preinstalled. Treat these as examples and load test your own agent.
container_concurrency for a synchronous agent: on Agent Runtime the default of 9 assumes blocking code; only async agents benefit, and memory limits still apply.< 1 second, maximizing node utilization.Sandbox / SandboxClaim CRDs, SandboxWarmPool for sub-second startup), full PodSpec sidecars, GPUs/TPUs, and mandatory gVisor kernel isolation enforced via Validating Admission Policy."Region temporarily at capacity" vs. 429 RESOURCE_EXHAUSTED: "Region temporarily at capacity" on Agent Runtime deploy means physical regional compute pool exhaustion (filing an IAM/quota increase ticket will NOT fix it—deploy to an alternate region or right-size vCPU/RAM). By contrast, 429 RESOURCE_EXHAUSTED on Gemini inference is TPM/PayGo contention—route stateless calls to the global endpoint, enable Priority PayGo, or purchase Provisioned Throughput (GSUs).GOOGLE_CLOUD_PROJECT, PORT, or custom GOOGLE_CLOUD_AGENT_ENGINE* vars. Cloud Run auto-registers in Agent Registry only when deployed with --functional-type=agent + --identity-type=agent-identity.Call-site retry owned by ONE layer; honor Retry-After; require idempotency keys on mutating tools.
Service-boundary state machine (Closed → Open → Half-Open) preventing thread-pool starvation.
Orchestration failover across global/secondary regions, pre-validated model tiers, or deterministic search.
Structured handoff to a human queue with trace_id, state summary, and verified entities intact.
| Resilience Layer | Architectural Placement | Trigger Condition | Exact Mechanism & Design Rules |
|---|---|---|---|
| Layer 1: Exponential Backoff with Full Jitter | RPC / SDK Client Boundary (owned by exactly one layer—never stack custom loops on top of SDK default retries) | Transient capacity or transport errors: HTTP 408, 409, 429 (rate limit), 500, 502, 503, 504, 529 (overloaded) |
1. Respect Retry-After header first.2. Otherwise sleep $t = \text{Uniform}(0, \min(T_{\text{cap}}, T_{\text{base}} \cdot 2^{\text{attempt}}))$ to prevent synchronized retry storms. 3. Fail fast on terminal 400, 401, 403, 404, spend-cap 429 (no Retry-After), and HTTP 200 refusals.4. Enforce deterministic idempotency_key on all mutating tool retries. |
| Layer 2: Circuit Breaker | Service / Dependency Boundary (shared state across worker processes per upstream endpoint) | Sliding-window failure rate or slow-call rate exceeds threshold (e.g., >50% 5xx/529 over 20 calls) |
Transitions Closed → Open and rejects calls in < 1 ms without opening sockets, protecting application threads/memory. After cooldown, enters Half-Open to test 1–3 probe requests before resetting to Closed. |
| Layer 3: Model & Routing Fallbacks | Orchestration Layer (invoked when Layer 1 exhausts retries or Layer 2 circuit is Open) |
Primary model/region unavailable, circuit breaker tripped, or latency budget exceeded | 3a. Global / Cross-Region: Same model snapshot via global endpoint or secondary region.3b. Cross-Tier Fallback: Route to a faster/higher-quota model tier (e.g., gemini-4-argon → gemini-3.8-flash or Opus → Sonnet) using a prompt & parameter config specifically validated on that fallback model's eval suite.3c. Deterministic Fallback: Return lexical/vector search results or cached FAQ answers. |
| Layer 4: Graceful Degradation & Escalation | Workflow / UX Boundary (terminal safety net when automated recovery is exhausted) | All fallbacks exhausted, max_turns budget reached, or high-risk policy trigger |
Persist full session state and route to a human operator queue with a structured bundle: trace_id, user intent, verified entities, tool outputs already gathered, and exact failure reason. |
If an agent calls issue_refund(order_id="991", amount=50) and receives an HTTP 504 Gateway Timeout, your code does not know whether the payment processor committed the refund before the connection dropped. Retrying blindly risks double-refunding the customer. Always pass a deterministic idempotency_key = sha256(session_id + turn_idx + tool_name + canonical_json(args)) so Memorystore/Redis or the downstream API safely returns the cached receipt on duplicate attempts.
Official LLM SDKs (google-genai, anthropic) automatically retry transient errors 2 times by default (3 total attempts). If a developer wraps the SDK call in a custom 5-attempt retry loop AND an agent orchestration framework retries failed steps 3 times, a single user request generates $3 \times 5 \times 3 = \mathbf{45\text{ upstream calls}}$ during a rate-limit event—turning a minor 10-second 429 blip into a total self-inflicted denial of service.
p95/p99 latency under live concurrency, crash/exception rates, token consumption, cache hit ratios, single-turn tool selection, and Tier 2 autorater parity on real long-tail inputs.user_id or session_id (never per HTTP request, which mixes models mid-conversation and ruins prefix caching).p95 latency, cost per resolved task, refusal rate, HALLUCINATION monitor rate).| Evaluation Dimension | Offline & 2-Step Sim | Shadow Mode (Dark Launch) | Canary Release (1%–5%) | 4-Element A/B Experiment |
|---|---|---|---|---|
| User Exposure Risk | Zero (synthetic/historical data) | Zero (candidate output discarded) | Low (bounded 1%–5% blast radius) | Moderate (full treatment cohort) |
| Measures Real P99 Latency & Load? | No | Yes (at live traffic volume) | Partial (small slice) | Yes |
| Measures Multi-Turn User Reaction? | Simulated personas only | No (user reacts to baseline) | Yes (closed-loop) | Yes (statistically powered) |
| Mutating Tool Execution | Environment Simulation / mocks | MUST be stubbed / read-only | Live (with circuit breakers) | Live |
p95/p99 vs. p50)Token consumption in production AI systems follows a heavy-tailed (Pareto / log-normal) distribution, never a symmetric normal curve:
p50) budgeting fails: A typical p50 chat turn might use 3,000 tokens ($0.01), while the p95–p99 tail consists of 15-turn agent debugging loops, 50-page PDF uploads, large MCP JSON payloads, thinking_level="HIGH" blocks, and retries consuming 150,000–500,000+ tokens.p50/p90/p99 weighted buckets, and enforce hard per-session caps (max_turns, observation pruning) to clip runaway tails.Autonomous multi-agent workflows consume ~15× more tokens than single-turn chat (~4× more than a single-agent tool loop). Bend the cost curve with six architectural levers:
gemini-3.5-flash-lite (Haiku) for guardrails/extraction → gemini-3.8-flash (Sonnet) for tool loops → gemini-4-argon (Opus) for complex synthesis (50%–80% savings).gemini-4-argon & 3.x (≥4,096 tokens, zero storage fee, ~90–95% read discount) vs. explicit cachedContents / cache_control (TTL storage/write cost; gemini-4-argon supports implicit only). Keep static prefixes at byte 0!thinking_level="minimal" / "low" (effort: "low") on routine hops—thinking tokens bill at output token rates.after_tool_callback and store exact IDs in session.state before compacting prose.idleAction=SUSPEND) and undeploy unused Vector Search 1.0 index endpoints.| ROI Step | Formula | Worked Example (50,000 Insurance Claims / Month) | Common FinOps Mistake |
|---|---|---|---|
| Step 1: Baseline Human Task Cost | $\text{Volume} \times t_{\text{manual}} \times R_{\text{labor}}$ | $50,000 \times 0.20\text{ hr (12 min)} \times \$60/\text{hr} = \mathbf{\$600,000/\text{mo}}$ | Using base salary instead of fully loaded hourly labor cost. |
| Step 2: Gross Automation Value | $\text{Baseline} \times f_{\text{auto}} \times s_{\text{saved}}$ | 80% automated (40k claims $\times \$12.00$) = $\mathbf{\$480,000/\text{mo}}$ gross savings | Assuming 100% of queries are automated on day one. |
| Step 3: Total AI Operating Cost | $C_{\text{tokens (p95)}} + C_{\text{infra}} + C_{\text{eng}}$ | $\$9,000\text{ (tail tokens + Online Monitors)} + \$6,000\text{ (infra)} + \$15,000\text{ (eng)} = \mathbf{\$30,000/\text{mo}}$ | Pricing only p50 happy-path tokens and ignoring online judge/retry overhead. |
| Step 4: Mandatory Human-Review & Exception Tax | $(N_{\text{audited}} \times t_{\text{audit}} \times R_{\text{auditor}}) + (N_{\text{err}} \times t_{\text{fix}} \times R_{\text{spec}})$ | 40k audited $\times 6\text{ min ($\$6$)} = \$240\text{k}$ + 2,500 errors $\times 20\text{ min ($\$30$)} = \$75\text{k} \rightarrow \mathbf{\$315,000/\text{mo}}$ | Ignoring human verification and error-rework labor (#1 cause of negative ROI in production). |
| Net Monthly Benefit | $\text{Step 2} - \text{Step 3} - \text{Step 4}$ | $\$480,000 - \$30,000 - \$315,000 = \mathbf{+\$135,000/\text{mo}}$ (vs. $\$450\text{k}$ if Step 4 was omitted!) | Deploying low-precision workflows where human audit + rework exceeds manual baseline cost. |
| Architectural Readiness Classification | Diagnostic Criteria | Canonical Scenario | Required Engineering Action |
|---|---|---|---|
| 1. Feasible as Scoped | Accuracy ceiling, latency physics (p95), unit economics, and governance rules all align with current capabilities. |
Internal IT helpdesk agent grounded on Agent Search with 4s streaming TTFT and cited sources. | Build Level 1–3 eval suite and proceed to MVP rollout. |
| 2. Feasible with Constraints | Core task is solvable, but synchronous latency or zero-review autonomy conflicts with physics/risk. | Automated legal contract redlining across 60 clauses requested in < 800 ms with autonomous client transmission. |
Re-scope constraints: Shift to async background execution (background=True, 30–60s) and add a mandatory human approval gate (HITL). |
| 3. Not Feasible | Fundamental physics or economic conflict: requires 100.00% zero-defect determinism without human review, sub-10ms control loops, or token cost exceeds task value by 10×. | Sub-5ms high-frequency trading execution or unreviewed autonomous clinical dosing from raw audio. | Decline LLM for the critical path; use deterministic code or classical tabular ML instead. |
| Observed Production Symptom | Root Cause | Correct Engineering Remediation | Trap / Wrong Fix |
|---|---|---|---|
Compliance agent scores 97% on FINAL_RESPONSE_QUALITY yet issues refunds without checking eligibility |
"Right answer, wrong path" blind spot in outcome-only grading | Pair outcome autoraters with trajectory_in_order_match and a forbidden-tool check |
Raising the FINAL_RESPONSE_QUALITY threshold from 0.90 to 0.99 |
Team has zero historical chat logs and cannot test multi-turn edge cases or 503 tool outages safely |
Relying solely on static single-turn prompts or live prod backends | Run 2-Step Agent Simulation (from an Agent Registry snapshot) + Environment Simulation fault injection | Running load/fault tests directly against live production payment APIs |
| Manual prompt edits to fix one bug cluster keep breaking three unrelated instructions | Manual "prompt whack-a-mole" without closed-loop optimization | Cluster traces via Failure Cluster Analysis, keep a held-out split, and run Data-Driven Prompt Optimizer / Agent Optimizer | Appending contradictory ALL-CAPS warnings to the bottom of the system prompt |
| Deploying a custom Streamable HTTP MCP server onto Agent Runtime fails | Agent Runtime hosts stateful orchestrators (reasoningEngines), not standalone MCP servers |
Host custom Streamable HTTP MCP servers on Cloud Run (or GKE) and register them in Agent Registry | Embedding external MCP server daemons inside session.state |
Architect proposes self-hosting gemini-4-argon on GKE GPUs for VPC isolation |
Proprietary Gemini weights cannot be self-hosted on customer clusters | Call managed Gemini endpoints inside VPC Service Controls (+ GSUs if needed), or self-host open-weight Gemma 4 on GKE | Creating a GKE GPU node pool to pull gemini-4-argon container weights |
Agent Runtime deployment fails with "Region temporarily at capacity" |
Physical regional compute pool exhaustion (not an IAM or project quota cap) | Deploy to an alternate supported region, right-size vCPU/RAM, or retry off-peak | Filing a Gemini TPM quota increase ticket and blocking launch in that region |
| LLM judge scores padded, wordy answers higher than concise accurate ones | Verbosity bias & unanchored 1–10 holistic scale | Use binary or 1–3 anchored rubrics, Adaptive Rubrics, and require CoT critique before the score | Expanding the scale to 1–100 or setting judge temperature to 0.9 |
| Pairwise judge picks Candidate A 68% of the time regardless of content | Positional (primacy) bias in pairwise evaluation | Run every comparison twice ((A,B) and (B,A)); require consistent victory across both passes |
Keeping the baseline permanently in Slot A |
Agent passes benchmark with 99% pass@5 but fails 41% of live 5-run batches |
Evaluating with pass@k instead of pass^k ($0.90^5 = 59\%$) |
Gate production agents on pass^k and replace probabilistic hops with deterministic code gates |
Executing mutating tools 5 times in parallel in production |
| Every component scores >95% in Level 2 evals, but end-to-end tasks fail 45% of the time | Integration seam mismatch (schema drift, context collapse, or swallowed "" errors) |
Add Level 3 Trajectory & Seam Integration Tests with strict schemas, session.state slots, and is_error: true |
Tuning isolated Level 2 prompts further |
During a 2-minute 429/529 event, worker threads hit 100% and the web tier crashes |
Stacked retry loops (SDK $\times$ custom wrapper) + missing Circuit Breaker | Enforce a single retry owner with full jitter + Retry-After, and trip a Layer 2 Circuit Breaker into Layer 3 fallback |
Increasing retry attempts from 5 to 20 |
Customer is billed twice after a payment tool times out (504) and is retried |
Missing idempotency key on a mutating tool call | Attach deterministic idempotency_key (hash(session_id + turn + args)) and deduplicate downstream |
Disabling all retries on read-only tools as well |
| Shadow mode test accidentally sends real cancellation emails to live customers | Shadow pipeline invoked live mutating tools without a sandbox | Intercept/mock all state-mutating tools in shadow mode; permit only read-only retrieval or stubbed writes | Reducing shadow traffic from 100% to 10% |
| AI saves 75% of drafting time, yet total department operating cost increases | Omitted Step 4 Human-Review & Exception Tax (manual audit + specialist error rework) | Include human verification and rework labor in Step 4 ROI; use calibrated confidence + Tier 1 gates to focus human review | Treating gross time saved as net financial savings |
instruction, demonstration, or instruction_and_demo) tuning instructions and Example Store shots against Gen AI Evaluation metrics.is_agentic=true recording both the cryptographic SPIFFE Agent Identity and the initiating human user principal.<1s snapshot/restore; Agent Sandbox isolates pods with gVisor via SandboxTemplate / SandboxWarmPool.pass@k ($1-(1-p)^k$) measures at least one success across $k$ tries; pass^k ($p^k$) measures all $k$ trials succeeding—mandatory for unattended production agents.Closed → Open → Half-Open) that fails fast in <1 ms when error thresholds breach, preventing thread-pool exhaustion.p99 latency, errors, and autorater scores at zero user risk.aiplatform.googleapis.com/online_evaluator/scores) that notifies owners through Slack, email or Pub/Sub when quality drifts below a threshold.escalate)tool_context.actions.escalate = True or EventActions(escalate=True)) that ends a LoopAgent before max_iterations is reached.DEFAULT_ROUTE--otel_to_cloudadk web, adk deploy agent_engine) that exports the built-in OpenTelemetry instrumentation to Google Cloud Observability.OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=EVENT_ONLY records prompts and responses as events (required by online monitors), while ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS=false keeps that text out of span attributes.EvaluationDataset (agents-cli)agents-cli eval reads from tests/eval/datasets/*-dataset.json; each case is a single prompt or an agent_data conversation ending in a user turn. Replaced the ADK evalset format in agents-cli projects.eval_config.yamlagents-cli file under tests/eval/ that lists metrics_to_run and defines custom code metrics (custom_function) or LLM-as-judge metrics (prompt_template).*.evalset.json file holding one or more sessions (or a conversation scenario for user simulation), graded by adk eval or AgentEvaluator with criteria from test_config.json.0.0.0.0:8080, and implement /api/reasoning_engine and /api/stream_reasoning_engine to support the Python SDK and console playground.container_concurrencyadk eval, conformance replay tests, workflow agents, and Gemini 4 Argon routing.stop_reason, prompt caching, and Message Batches.Listen to the 65-minute AgentOps audio deep dive, open the 2-column study guide PDF, 21-slide deck, and sketchnote infographic in the Hub Viewer, or drill 38 scenario-based questions on the 3-tier grading ladder, autoraters, pass@k vs. pass^k, OTel gen_ai.* tracing, and heavy-tail FinOps.
Q1. A regulated refunds agent passes 98% of cases on FINAL_RESPONSE_QUALITY, yet an audit discovers it occasionally issues refunds without calling check_refund_eligibility first. Why did outcome grading miss this, and what Gen AI Evaluation configuration fixes it?
FINAL_RESPONSE_QUALITY + HALLUCINATION with process/trajectory metrics: specifically trajectory_in_order_match (requiring verify_identity → check_refund_eligibility → issue_refund in chronological order while allowing extra read-only lookups) plus a Tier 1 check blocking forbidden destructive tools.
Q2. An autonomous refund-processing agent has a single-run accuracy of $p = 0.90$. The engineering lead reports a pass@5 score of $99.999\%$ and argues the agent is ready for unattended production launch. Why is this metric misleading, and what is its actual 5-run consistency?
pass@5 ($1 - 0.10^5 = 99.999\%$) measures the probability that at least one of 5 attempts succeeds, which is only valid when a deterministic verifier can select a single working candidate before any side effects occur. In unattended production, every run impacts a real customer, so the required metric is pass^5 ($0.90^5 = \mathbf{59.0\%}$)—meaning 41% of 5-run batches will contain at least one failure.
Q3. You are launching a brand-new healthcare scheduling agent with zero historical production chat logs and cannot run stress or fault-injection tests against the live hospital EHR API. How do you evaluate multi-turn edge cases and HTTP 503 tool outages pre-deployment?
503 timeouts and malformed responses.
Q4. Online Evaluation Monitors flag 600 failing traces this week. Whenever an engineer manually edits the 4,500-token system prompt to fix one bug, two older capabilities regress. How should you systematize remediation?
Agent Optimizer): (1) scrub traces with SDP, (2) group them via Failure Cluster Analysis ranked by frequency × severity, (3) codify top clusters as outcome-graded evalset cases while holding out a separate validation slice to prevent overfitting, and (4) run a Data-Driven Prompt Optimizer job (instruction_and_demo mode) to tune instructions and Example Store demonstrations without regressing the suite.
Q5. Map three Google Cloud workloads to the optimal runtime (Agent Runtime, Cloud Run, or GKE): (a) a stateful Python ADK orchestrator needing built-in Sessions and Memory Bank; (b) a stateless TypeScript Streamable HTTP MCP server scaling to zero; (c) serving fine-tuned open-weight Gemma 4 on L4 GPUs alongside gVisor-sandboxed code execution pods.
Gemma 4 on GPU/TPU node pools and running pod-isolated Agent Sandbox on GKE (SandboxWarmPool + gVisor).
Q6. During a regional capacity event returning HTTP 529 Overloaded, your service wraps the default SDK client inside a custom 5-attempt retry loop with a fixed 2-second sleep. Within 60 seconds, all application web workers freeze. What two mechanisms failed, and where should each fix be placed?
Retry-After. (2) Missing Layer 2 Circuit Breaker: Holding threads open across 15 retries exhausts the worker pool; place a Circuit Breaker at the service boundary that trips Open when error rates breach threshold and routes immediately to Layer 3 global/cross-region or cross-tier model fallbacks.
Q7. You want to validate a new customer-support agent prompt that can call lookup_order (read) and issue_credit_refund (write). What can Shadow Mode safely test, what must be sandboxed during shadow mode, and why do you still need a 4-Element A/B Test afterward?
p95 latency, token spend, crash rates, first-turn tool selection, and async autorater scores on real traffic, provided issue_credit_refund (and all mutating tools) are intercepted/mocked in a read-only sandbox so customers aren't double-refunded. You still need a live 4-Element A/B test (sticky session randomization, single-variable isolation, guardrail metrics, and pre-committed sample size) because shadow mode cannot measure multi-turn user reactions to the candidate's replies or true ticket resolution rates.
Q8. A pilot agent shows a median (p50) cost of $0.03 per conversation. The team multiplies $0.03 by 1,000,000 monthly sessions ($30,000/mo) and claims $200,000/mo in gross labor savings without human review costs. Why will this business case fail in production?
p95/p99 tail where the true mean cost per session is $3\times$–$10\times$ higher than the p50 median. (2) Omitting Step 4 (Human-Review & Exception Tax): Subtracting the mandatory labor cost of human auditors verifying flagged outputs and specialists fixing false positives is essential to compute net enterprise ROI.