Gemini Enterprise Agent Platform end-to-end engineering reference: the three build paths (Agent Studio, Managed Agents API + Skill Registry, and code-first ADK / LangGraph with agent-starter-pack & agents-cli), Agent Runtime vs. Cloud Run vs. GKE, Context Engineering (Sessions, Memory Bank, Example Store), Code Execution sandboxing, enterprise grounding & MCP servers, 2-step user simulation & environment simulation, Agent Registry observability & dual-attribution audit logs, failure cluster analysis & prompt optimization, and FinOps quota controls.
Gemini Enterprise Agent Platform is the evolution of Vertex AI for building, scaling, governing, and optimizing production agents on Google Cloud. This guide covers the build, retrieve, run, and optimize half: choosing among Agent Studio, the Managed Agents API, and code-first ADK / open frameworks, hosting agents on Agent Runtime (formerly Agent Engine), managed Sessions, Memory Bank, Example Store, and the Code Execution sandbox, choosing between Agent Runtime, Cloud Run, and GKE, enterprise retrieval and MCP tooling, 2-step user simulation and environment simulation, Agent Registry observability, programmatic prompt optimization, and FinOps quota management.
Cryptographic Agent Identity, IAM, Principal Access Boundary (PAB), VPC Service Controls, Model Armor, Sensitive Data Protection, and Agent Gateway policies live in Agent Platform Security & Governance. Code-level ADK patterns live in the ADK guide; cross-agent wire protocols and A2UI/AP2 in A2A & Interoperability; vendor-neutral RAG in RAG; Gemini 4 Argon and Model Garden in Gemini & Google Models.
The model (Gemini 4 Argon, Gemini 3.8 Flash…) is the reasoning brain; the orchestration layer (ADK, LangGraph, Managed Agents harness, or Agent Studio) wires the Think → Act → Observe loop; Agent Runtime is the managed body keeping long-running tasks alive; Sessions are short-term working memory for one conversation while Memory Bank is consolidated long-term memory across conversations; MCP servers, retrieval services, and the Code Execution sandbox are the hands and library; Evaluation + Observability + Agent Optimizer form the continuous AgentOps quality flywheel.
Choose the most managed component that meets the requirement. Move down the stack (Managed Agents API / Agent Studio → Agent Runtime → Cloud Run → GKE) only for a concrete engineering need: custom orchestration code, hosting stateless Streamable HTTP MCP servers, non-Python stacks, GPUs/self-hosted open weights, or kernel-level isolation.
Product names, preview status, quotas, and regional availability change frequently. Items marked Preview are Pre-GA without GA SLAs. Verify current limits and pricing in official Google Cloud documentation before production commitment.
| Pillar | What it covers | Key components |
|---|---|---|
| Build | Create agents, mount skills/tools, and connect to enterprise data | ADK (Python, Go, Java, TS), Agent Studio (low-code), Managed Agents API + Interactions API Preview, Skill Registry, Agent Garden, Model Garden (200+ models), RAG Engine, Agent Search, Agent Retrieval, agent-starter-pack, agents-cli |
| Scale | Run stateful, multi-step agents reliably in production | Agent Runtime (reasoningEngines), Sessions, Memory Bank, Example Store Preview, Code Execution sandbox, A2A Agent Hosting Preview |
| Govern | Control identity, discovery, egress, and security policy | Agent Registry, Agent Identity (SPIFFE), Agent Gateway, Model Armor, governance policies → see Security guide |
| Optimize | Simulate, evaluate, observe, and continuously improve quality | Gen AI Evaluation Service (2-step user simulation, environment simulation, adaptive rubrics, trajectory metrics, Online Monitors), Agent Registry Observability, Failure Cluster Analysis, Prompt Optimization / Agent Optimizer |
Billing model: Agent Platform uses pay-as-you-go developer consumption (tokens, runtime vCPU/GiB-seconds, deployed index node-hours) with optional Provisioned Throughput (GSUs); the Gemini Enterprise app (formerly Agentspace) is a separate per-seat subscription workspace where business users discover and invoke published agents.
| Build Path | Builder & Control Level | Loop, State & Sandbox Ownership | Best Fit |
|---|---|---|---|
| 1. Agent Studio (low-code) | Domain experts & rapid prototypers | Visual canvas configuring Gemini model, system prompt, grounding data stores, MCP tools, and sub-agents; 1-click deploy to Agent Runtime or Gemini Enterprise Workflow Builder | Departmental assistants, rapid enterprise RAG validation before pro-code handoff |
| 2. Managed Agents API Preview | Developers wanting a declarative REST/SDK harness | Pairs Agents API + mounted Skill Registry bundles with stateful Interactions API (previous_interaction_id, background=true). Google runs the loop, restarts, artifacts, and an isolated Linux sandbox with no network egress by default (powers Antigravity Agent & Deep Research). | Autonomous coding, file/spreadsheet processing, and multi-hour synthesis with zero harness ops |
| 3. Code-first ADK & Open Frameworks | Software & AI engineers needing deterministic code control | Author custom control flow and callbacks in ADK, LangGraph, CrewAI, AG2, or LlamaIndex; scaffold Terraform + CI/CD eval gates via agent-starter-pack & agents-cli; deploy to Agent Runtime, Cloud Run, or GKE | Complex multi-agent graphs, strict compliance guardrails, custom hybrid A2A meshes |
| Current name | Former name | Exam / API note |
|---|---|---|
| Gemini Enterprise Agent Platform | Vertex AI | Evolution of the same platform; IAM permissions still use aiplatform.* |
| Gemini Enterprise (end-user app) | Google Agentspace | Unified enterprise search + multi-agent hub supporting A2A and A2UI |
| Agent Runtime | Agent Engine / Reasoning Engine | The REST/gRPC resource remains projects/…/locations/…/reasoningEngines/<id> |
| Agent Search | Vertex AI Search | Turnkey managed search/RAG (discoveryengine) with connectors and native ACLs |
| Agent Retrieval | Vector Search 2.0 | Collections-based, self-tuning vector store with schema & auto-embeddings |
| Vector Search 1.0 | Matching Engine | High-scale ScaNN ANN index-as-a-service |
| Workflow Builder (Gemini Enterprise) | Agent Designer | No-code builder for chat agents and scheduled workflows in Gemini Enterprise |
| Gemini 4 Argon / Gemini 3.x | Gemini 1.5 / PaLM 2 | Active frontier (gemini-4-argon) and agentic workhorse (gemini-3.8-flash) |
Model Garden catalogs Google first-party (Gemini 4 Argon, Gemini 3.8 Flash, Nano Banana 2.1, Gemini Omni 1.1 Flash, Veo 3.1, Lyria 3.5, Chirp 3), partner MaaS (Anthropic Claude, Mistral, DeepSeek), and open-weight models (Gemma 4, Gemma 3n, Llama). Model selection and tuning live in the Google guide. Gemini itself is served exclusively through managed Google endpoints — you cannot self-host Gemini on Cloud Run or GKE; open models such as Gemma 4 can be self-hosted there.
Agent Runtime is a fully managed, opinionated compute runtime for stateful agents: sub-second cold starts, long-running and multi-day asynchronous workflows, optional min_instances for warm capacity, IAM and VPC Service Controls enforcement, private VPC/on-prem egress through a Private Service Connect (PSC) interface network attachment, cryptographic Agent Identity, and built-in integration with Sessions, Memory Bank, Example Store, Code Execution, A2A, and OpenTelemetry.
adk deploy agent_engine or agents-cli; Go via adkgo; Java & TypeScript via containers)query / async_query / stream_queryA2aAgent wrapper Preview exposing an AgentCard and task store on Agent Runtimeagent-starter-pack & agents-cligoogle-cloud-agentplatform exposes agentplatform.Client(project=…, location=…) with client.runtimes, client.sessions, client.sandboxes, and client.memory_banks (replacing legacy agent_engines).agent-starter-pack & agents-cli: scaffolds production Terraform IaC, Cloud Build / GitHub Actions CI/CD with automated agents-cli eval regression gates, OpenTelemetry spans, and BigQuery feedback sinks.| Deployment method | Languages | Best for |
|---|---|---|
| Developer Connect (Git repository) | Python | GitOps CI/CD — builds and deploys directly from a connected Git repo |
| Source files (inline directory) | Python | Automated CI/CD pipelines and Terraform; no Cloud Storage staging bucket needed |
| Dockerfile | Any | Custom OS packages or non-Python agents (Go, Java, TypeScript) |
| Container image (Artifact Registry) | Any | Pre-built, vulnerability-scanned images promoted across environments; lowest deployment latency |
| SDK object (in-memory) | Python | Notebooks/Colab prototyping; the agent object must be serializable |
Do not set platform-managed variables such as GOOGLE_CLOUD_PROJECT or PORT in your deployment config, and avoid the GOOGLE_CLOUD_AGENT_ENGINE prefix for custom variables (except documented flags like GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY=true).
Agent Runtime hosts stateful agent orchestrators, not standalone MCP tool servers. Deploy custom Streamable HTTP MCP servers on Cloud Run or GKE, register them in Agent Registry, and call them from your agent.
| Criterion | Agent Runtime | Cloud Run | GKE |
|---|---|---|---|
| Best fit | Stateful Python/ADK/LangGraph/A2A agents; fastest path to production | Polyglot containerized agents, webhook services, and Streamable HTTP MCP servers | Self-hosted open weights (Gemma 4 on GPUs/TPUs), dense K8s fleets, custom kernel isolation |
| Ops overhead | Lowest (fully managed agent harness) | Low (serverless containers) | Highest (you manage cluster & node lifecycle) |
| State | Built-in Sessions, Memory Bank, Example Store, Code Execution | Stateless — externalize to Agent Platform Sessions, Memorystore, Firestore, or Cloud SQL | StatefulSets, local SSDs, or external managed stores |
| Scaling & duration | Managed autoscaling; min_instances for warm capacity; multi-day workflows | Scale to zero; HTTP timeout up to 60 minutes; Services, Jobs, Worker pools | Full control (HPA/VPA, node pools, GPU sharing) |
| Isolation & security | IAM, VPC-SC, PSC interface, Agent Identity, Code Execution sandbox | IAM, Direct VPC egress, built-in Agent Identity & Agent Registry integration | Workload Identity Federation, GKE Sandbox (gVisor), NetworkPolicies |
| Hosts MCP servers? | No | Yes (Streamable HTTP) | Yes |
| Weak spot | No custom GPUs; non-Python stacks need a custom container | Ephemeral in-process memory lost on scale-to-zero | Operational & platform engineering burden |
→ Agent Runtime (or Managed Agents API for zero-code harness).
→ Reuse existing platform investments where ops tooling is mature.
→ Cloud Run (Streamable HTTP MCP servers & event-driven webhooks).
→ GKE with accelerator node pools and fine-grained NetworkPolicies.
Route easy steps to Gemini 3.5 Flash-Lite, loops to 3.8 Flash, hard plans to 4 Argon.
A Cloud Run instance can be replaced or scaled to zero at any moment. Conversation state kept in process RAM vanishes. Persist it in Agent Platform Sessions (--session_service_uri agentengine://<id>), Memorystore, Firestore, or Cloud SQL.
Production agents apply context engineering — assembling only relevant recent events, compacted summaries, scoped long-term user facts, and top-k dynamic examples per turn to prevent context rot and runaway token spend.
| Abstraction | Scope | Lifetime |
|---|---|---|
| Session | Chronological append-only sequence of events (messages, thoughts, tool calls/outputs) for one interaction | One conversation (apply sliding-window compaction on long threads) |
| State | Mutable key-value scratchpad attached to a session (temp:, session, user:, app: prefixes in ADK) | Current turn or current conversation |
| Memory Bank | LLM-extracted, consolidated semantic facts & preferences about a user/entity | Persistent across sessions (governed by TTL & revisions) |
Sessions: ADK agents deployed to Agent Runtime get managed sessions automatically. Outside Agent Runtime (Cloud Run, GKE, local), point ADK at the managed service with --session_service_uri agentengine://<id>.
GenerateMemories extracts durable facts from session events in the background and consolidates them with existing memories (merging duplicates, updating changed preferences, resolving contradictions).CreateMemory writes an explicit fact immediately when the agent decides to remember on command.RetrieveMemories fetches by scope or semantic similarity search against the active user turn.{user_id, app_name} — never share unscoped namespaces across users.roles/aiplatform.memoryViewer / roles/aiplatform.memoryEditor, CMEK, and regional residency (gcp.resourceLocations).VertexAiMemoryBankService; trigger incremental extraction in callbacks via add_events_to_memory (preferred over full-session add_session_to_memory).Untrusted external tool outputs (emails, web pages, tickets) processed during memory extraction can plant malicious instructions into Memory Bank that replay across future sessions. Screen inputs with Model Armor, constrain Memory Topics, and audit memory revisions (Security guide).
Stores curated few-shot input→trajectory/output demonstrations and injects only the most semantically relevant ones dynamically at inference time. The embedding model (e.g., text-embedding-005, text-multilingual-embedding-002, or gemini-embedding-001) is immutable after instance creation; changing embedding models requires creating a new instance. Default quota: 50 instances per project per region.
| Need | Right Component |
|---|---|
| Remember what this user said three turns ago in the chat | Session events / scratchpad state |
| Remember user preferences and recurring project context next month | Memory Bank |
| Steer complex tool trajectories or output schemas with dynamic few-shot exemplars | Example Store Preview |
| Answer grounded in 50,000 permissioned enterprise documents | Agent Search / Retrieval (Section 06) |
The Code Execution sandbox runs untrusted, model-generated Python inside an isolated container: sub-second startup/execution, file I/O up to 100 MB, and stateful sandbox files/variables persisting up to 14 days (configurable TTL). ADK's AgentEngineSandboxCodeExecutor binds one sandbox per session so DataFrames and generated charts carry across turns — and can be invoked even when your agent runs on Cloud Run or GKE. Similarly, the Managed Agents API Linux sandbox blocks outbound network egress by default while mounting approved Skill Registry assets.
The runtime executes trusted orchestration code you wrote and reviewed (holding IAM/MCP credentials). The sandbox executes untrusted code synthesized by the model at runtime. Never run LLM-generated code via exec() inside the agent's own container.
Injecting service-account tokens or opening unrestricted internet egress inside a code sandbox defeats containment. Pass input data strictly as scoped files and read sanitized outputs back.
| Service | Type | Choose when | Watch out for |
|---|---|---|---|
| Agent Search (formerly Vertex AI Search) | Turnkey managed search/RAG | Enterprise corpora (Drive, SharePoint, Confluence, Jira, GCS, BigQuery) needing built-in parsing, hybrid search, reranking, and native document-level ACLs | Less low-level control over custom chunking or bespoke embeddings |
| RAG Engine | Code-first managed RAG pipeline | You want explicit control over parsers (Document AI Layout Parser, LLM parser), chunking, embeddings, and vector DB (RagManagedDb, Agent Retrieval, Vector Search 1.0, Pinecone, Weaviate; the choice depends on the deployment mode) | Check regional & model matrix (gemini-4-argon is not directly bound as a RAG Engine generator; Agent Retrieval corpora require supported regions + cosine metric) |
| Vector Search 1.0 | ScaNN ANN index-as-a-service | Tens of millions to billions of custom vectors (text or multimodal embeddings you compute yourself) at single-digit-ms latency | Deployed index endpoints bill continuously even when idle |
| Agent Retrieval (formerly Vector Search 2.0) | Self-tuning collection store | Schema-aware collections of Data Objects with managed auto-embeddings or BYO embeddings | Exact KNN up to ~100k objects; create an ANN Collection Index beyond ~100k |
| Grounding with Google Search / Maps / Web | Live public-index grounding | Fresh public facts, places/geospatial context, or enterprise web grounding (Parallel, Exa) | Must render required Search Suggestions & attribution metadata |
BigQuery (VECTOR_SEARCH) | In-warehouse SQL vectors | Analytical batch enrichment, warehouse joins, and offline evaluation pipelines | Multi-second startup latency — not a sub-100ms interactive chat store |
| AlloyDB / Cloud SQL (pgvector / ScaNN) | Operational OLTP vectors | Low-latency vector similarity filtered alongside live transactional SQL rows | You manage sizing, pooling (MCP Toolbox), and index tuning |
Ingestion (local files, Cloud Storage, Google Drive, Slack/Jira connectors) → transformation (chunking via default parser, LLM parser, or Document AI Layout Parser) → embedding → corpus index → retrieval → generation. Vector store options depend on the deployment mode: Spanner mode offers RagManagedDb (default), Agent Retrieval, Vector Search 1.0, Pinecone, and Weaviate; Serverless mode defaults to an Agent Retrieval collection (Pinecone and Weaviate as alternatives). Both modes support VPC-SC; only Spanner mode supports CMEK.
semantic-ranker-default@latest): stateless cross-encoder reranker applied to top-N candidates from any retriever.webSearchQueries, searchEntryPoint, groundingChunks, groundingSupports — render verifiable citations.roles/discoveryengine.viewer / editor.ADK built-in retrieval tools: google_search (web grounding), VertexAiSearchTool (Agent Search data store), VertexAiRagRetrieval (RAG Engine corpus), VertexAiRagMemoryService (RAG-backed memory).
https://<service>.googleapis.com/mcp or regional .rep.googleapis.com/mcp) that become available when you enable the service API and are registered in Agent Registry automatically, across BigQuery, Cloud SQL, AlloyDB, Spanner / Bigtable / Firestore / Memorystore, Cloud Storage, GKE, Cloud Run, Compute Engine, Pub/Sub, Cloud Logging / Monitoring / Trace, Apigee API hub, Knowledge Catalog, and Agent Retrieval Preview. Requires MCP Tool User (roles/mcp.toolUser → mcp.tools.call) plus underlying service IAM roles.github.com/googleapis/mcp-toolbox, formerly genai-toolbox): open-source MCP server that manages connection pools and auth, ships prebuilt tools per database (--prebuilt=<database>), and lets you define parameterized, SQL-injection-safe custom tools in tools.yaml for Cloud SQL, AlloyDB, Spanner, BigQuery, and many non-Google databases (PostgreSQL, MySQL, MongoDB, Neo4j, Snowflake and others).ApplicationIntegrationToolset); govern discovery in Agent Registry and egress via Agent Gateway (Security guide).When an Agent Retrieval / Vector Search index backs a RAG Engine corpus, configure the distance metric as COSINE. RAG Engine queries using cosine distance, so leaving the index at default DOT_PRODUCT fails with FAILED_PRECONDITION.
When the matrix above leaves two vector options close, compare them on these dimensions.
| Dimension | Vector Search 1.0 | Agent Retrieval | RAG Engine |
|---|---|---|---|
| Object model | Datapoints (ID, embedding, optional filter metadata) in an index deployed to an endpoint; source records live in a separate store | JSON Data Objects (payload fields plus one or more vector fields) inside a schema-validated Collection | RagCorpus and RagFile resources; vectors go to a backing vector database |
| Embedding pipeline | Bring your own dense (and, for hybrid, sparse) vectors before upserting | Your own vectors, or automatic fill from a vertex_embedding_config (model ID, text template over payload fields, task type) when a Data Object arrives without that vector | Parses, chunks, and embeds files for you at import time |
| Hybrid search and reranking | Dense plus sparse index fused with Reciprocal Rank Fusion (RRF), weighted by rrf_ranking_alpha; you build sparse vectors (for example TF-IDF) | Semantic plus built-in keyword text search through RRF; optional Ranking API rerank | Reranking in RagRetrievalConfig (rank_service or llm_ranker); hybrid_search.alpha applies only with Weaviate |
| Agent and MCP integration | Custom function tool or your own MCP server | Google-managed remote MCP server (Preview, read-only tools) | Corpus attached to Gemini requests as a vertex_rag_store retrieval tool |
| Scaling and cost | Deployed index endpoints bill by node-hour, even when idle | Brute-force KNN with no index; add a Collection Index near 100,000 Data Objects or when KNN latency is too high; usage-based or resource-based pricing | Depends on the deployment mode (see below) |
One RAG API routes to whichever mode the project has active; corpora created under the other mode stay stored but hidden until you switch back.
--kms-key). An existing collection cannot be switched; export its Data Objects to Cloud Storage and import them into a new CMEK collection.Serverless mode has no CMEK option, so a key-management policy rules it out. Pick Spanner mode before importing data, because corpora do not carry over between modes.
Google-hosted HTTPS endpoints; callers need roles/mcp.toolUser (mcp.tools.call) plus IAM on the underlying resources.
| Server (endpoint host) | What the agent can do | Auth notes |
|---|---|---|
BigQuery (bigquery.googleapis.com) | Discover datasets and tables, read schemas, run SQL over warehouse data without copying tables into the prompt | IAM principal; BigQuery row/column controls still apply |
Cloud SQL (sqladmin.googleapis.com) | Create, manage, and query MySQL, PostgreSQL, and SQL Server instances (execute_sql, execute_sql_readonly) | IAM principal; prefer the read-only tool for analysis agents |
| AlloyDB / Spanner / Firestore | Manage resources and run queries or document operations against operational data | IAM principal; AlloyDB regional endpoints are Preview |
Cloud Storage (storage.googleapis.com/storage/mcp) | List buckets and read or write objects such as reports and artifacts | IAM principal plus bucket IAM |
Agent Retrieval (vectorsearch.googleapis.com) Preview | Inspect collections and indexes; run vector, semantic, hybrid, and text search (nine read-only tools, no write tools) | IAM principal; the client sends routing headers (such as the target project) on every tools/call |
| Cloud Logging / Monitoring / Trace | Query logs, metrics, and traces while troubleshooting agents | IAM principal with observability viewer roles |
| Developer Knowledge and Maps Grounding Lite | Search Google developer documentation; look up places and location facts | Google (non-Cloud) servers: API key or OAuth, no roles/mcp.toolUser, and not controllable through IAM policies |
The catalog grows quickly and launch stages differ per server and per endpoint. Check the public Supported products list for Google Cloud MCP servers before you design around a specific server.
Pick the thinnest integration layer that still gives the agent narrow, governed tools.
| Option | Custom logic | Ops burden | Typical auth | Choose when |
|---|---|---|---|---|
| Managed remote MCP server | None (Google-defined tool schemas) | None (Google-hosted) | Agent identity, service account, or user OAuth; IAM on roles/mcp.toolUser + resource roles | Standard access to Google Cloud data and infrastructure with no server to run |
| MCP Toolbox for Databases | Declarative (tools.yaml parameterized SQL) | Low to medium (you run the container on Cloud Run or GKE) | IAM database auth or per-tool OAuth/OIDC | Agents must run only vetted, parameterized queries against your databases |
| Custom MCP server on Cloud Run | Full (any language, multi-API composition) | Medium (image, scaling, patching) | Service account or agent identity; IAP in front | Proprietary business workflows or internal services with no managed server |
OpenAPI tool (ADK OpenAPIToolset) | Lives in the existing REST service | Low when a spec already exists | API key, OAuth, or service account | Wrapping a well-documented REST API without standing up an MCP server |
| Application Integration connectors | Visual flows and data mapping | Low (managed connectors) | Managed connector auth profiles | SaaS ERPs/CRMs (SAP, Workday, ServiceNow) without a native MCP endpoint |
run_any_sql toolHanding the model unrestricted SQL turns every prompt injection into a database incident. Prefer read-only tools, parameterized Toolbox tools, or narrow custom tools.
McpServer resource exists only in global; regional endpoints appear in its interfaces field, so query with --location=global (a regional query returns NOT_FOUND).gcloud beta run deploy --functional-type=mcp-server (services only; Cloud Run jobs can't be MCP servers). --identity-type (beta) selects agent-identity or service-account (default).registry.gke.io/functional-type: "MCP_SERVER" label plus modelcontextprotocol.info/* annotations; GKE fetches the tool spec from the running server.gcloud agent-registry services create … --mcp-server-spec-type=tool-spec) with an uploaded toolspec.json (MCP tool schema, 10 KB max) listing tool names, descriptions, and annotations such as readOnlyHint or destructiveHint.roles/agentregistry.editor on the governance project.stdio runs the server as a local child process on the same machine (IDEs, CLIs, local Toolbox); Streamable HTTP over HTTPS is required for remote and production servers on Google infrastructure, Cloud Run, or GKE.2026-07-28, which drops the initialize handshake and Mcp-Session-Id; each request carries what it needs in HTTP headers or _meta, so load balancers route without parsing bodies.Skill Registry is the Agent Platform repository for agent skills in the open Agent Skills layout: a SKILL.md file plus optional scripts/, references/, and assets/ folders. Agents load a skill only when the task calls for it. Agent Registry can also hold the same packages as standalone skills, with its own API, roles, and tighter payload limits, so keep the two paths apart.
| Stage | Skill Registry (Agent Platform API) | Agent Registry standalone skills |
|---|---|---|
| Author | SKILL.md front matter: name (required, max 64 characters, lowercase letters, digits, and hyphens, no leading or trailing hyphen), description (required, max 1,024 characters), optional license. Body max 500,000 characters. The skill ID (1 to 63 characters, starts with a letter) stays reserved after deletion; the gcp- prefix is reserved for built-in skills | Same SKILL.md at the zip root with valid front matter per the Agent Skills specification |
| Publish and validate | CreateSkill long-running operation (REST v1beta1 or client.skills.create) with a base64 zip. Validation rejects a zip over 10 MB, over 500 MB unzipped, over 10,000 entries, nested deeper than 8 folders, compression ratio over 100, any symbolic link, duplicate names, .. or absolute paths, or a missing SKILL.md | gcloud alpha agent-registry skills create with --payload or --gcs-source-uri. Limits: 500 KB compressed, 10 MB uncompressed, 1 MB per file, 8 folder levels. The ID gets a private- prefix; a Cloud Storage import needs the Agent Registry service agent to hold storage.objects.get |
| Revisions | Every create or update stores an immutable skill revision; GetSkill returns the latest, and ListSkillRevisions / GetSkillRevision expose history | New skills start DISABLED; set the default revision and switch the target state to Active before agents can discover it |
| Discover | ListSkills returns name, display name, and description; RetrieveSkills runs semantic search over them. The built-in gcp-skill-registry skill lets an agent create, search, and manage skills itself | Agent Registry search across agents, MCP servers, and skills |
| Authorize | Inherits project IAM: roles/aiplatform.user to manage skills, roles/aiplatform.viewer for read-only access | roles/agentregistry.viewer (discover and read), roles/agentregistry.user (plus create, update, delete skills and revisions), roles/agentregistry.editor (plus agents, MCP servers, endpoints), roles/agentregistry.admin (plus bindings and skill IAM policies) |
| Govern at runtime | Semantic governance policies (Preview) intercept the ADK skill tools and judge each call against natural language constraints (NLCs) | Same policies apply, because loading a skill is itself a tool call |
list_skills shows the model only the catalog of skill names and descriptions.load_skill pulls one skill's full instructions and tools into context.load_skill_resource fetches a referenced file or asset on demand.run_skill_script executes the skill's bundled code. Each tool is a point where a policy can return a verdict.us-central1, europe-west4, us-east5.load_skill for any skill that sends messages or writes externally once the history contains web, email, or uploaded content.Granting roles/agentregistry.admin or roles/agentregistry.editor to an agent identity lets a compromised agent alter tool definitions or enroll a malicious agent. On the standalone-skills path, give consuming agents roles/agentregistry.viewer and publishers or CI pipelines roles/agentregistry.user; on the Skill Registry path, use roles/aiplatform.viewer for consumers and roles/aiplatform.user for publishers.
IAM decides who may publish or read a package, but an authorized agent can still be steered by injected content into loading a powerful skill. Pair IAM with semantic governance policies.
| Store | Role in an agent system |
|---|---|
| Gemini Context Caching | Implicit caching (on by default, zero storage fee; the only cache mode supported on gemini-4-argon) and Explicit caching (cachedContents with TTL + hourly storage on eligible Gemini 2.5/3.x base models) for large repeated instructions/corpora |
| Memorystore for Redis / Valkey | Sub-millisecond hot cache, rate-limit/budget counters, idempotency locks, and semantic response caching via Redis HNSW/FLAT vector search |
| Firestore | Serverless document store for real-time client state sync, HITL approval queues, and user profiles |
| Cloud SQL / AlloyDB | ACID relational state; backs ADK DatabaseSessionService when SQL querying over sessions is required |
| Agent Platform Sessions | Zero-ops managed session/event persistence for ADK and Agent Runtime |
| BigQuery | Long-term analytics, audit/trace sinks, failure-cluster mining, and offline evaluation datasets — never hot-path turn state |
Semantic cache pattern: embed incoming query → vector lookup in Memorystore → if similarity ≥ threshold within the caller's ACL partition, return cached answer; otherwise invoke the agent and store with TTL.
A global semantic cache shared across users can leak answers synthesized from restricted HR/finance documents to unauthorized callers. Always partition semantic cache keys by tenant/role/ACL scope or skip response caching for permission-sensitive corpora.
Split agent data by access pattern rather than putting everything in one database. Each row names the default pick and the usual fallback.
| Data need | Default service | Common alternative | Why the default fits |
|---|---|---|---|
| Conversation sessions (events + state) | Agent Platform Sessions via VertexAiSessionService | Cloud SQL or AlloyDB via DatabaseSessionService | No schema to own; reachable from Agent Runtime, Cloud Run, GKE, or a laptop; feeds Memory Bank generation |
| Long-term user memory | Memory Bank | Your own facts table with vectors in AlloyDB/Cloud SQL (pgvector) or Firestore vector search | Managed LLM extraction, consolidation, identity-scoped similarity retrieval, TTL, and revisions |
| Semantic cache | Memorystore for Redis vector search (Redis 7.2; HNSW approximate or FLAT exact) | AlloyDB vector index | In-memory lookups with native key expiry, so repeated or paraphrased prompts skip the model call |
| Rate-limit and budget counters | Memorystore (Redis/Valkey atomic increments with TTL) | Firestore counters (higher latency, lower write rate per document) | Atomic in-memory operations suit per-user or per-agent throttling on the hot path |
| Transactional data that tools read and write | Cloud SQL or AlloyDB | Spanner (global scale) or Firestore (document model) | ACID transactions, joins, and vector columns next to live rows; reachable through managed MCP servers or MCP Toolbox |
| Analytics and agent telemetry | BigQuery | Cloud Logging Log Analytics | Serverless SQL over prompt/response logs, traces, eval results, and token cost; never the per-turn state store |
| Documents, media, and artifacts | Cloud Storage (ADK GcsArtifactService for agent outputs) | BigQuery object tables (SQL metadata over Cloud Storage objects) | Durable, cheap object storage that also feeds Agent Search, RAG Engine, and Agent Retrieval ingestion |
| Vector embeddings | Agent Retrieval or Vector Search 1.0 | AlloyDB/Cloud SQL pgvector, Firestore vector search, BigQuery VECTOR_SEARCH | Purpose-built ANN and hybrid search; pick the operational database when vectors must be filtered with live transactional rows |
The SessionService backend decides whether conversation state survives restarts and scale-out.
SessionService | Where data lives | Survives restart / scale-out | Use for |
|---|---|---|---|
InMemorySessionService | Process memory | No (lost on restart; not shared across replicas) | Local development, unit tests, adk web demos |
DatabaseSessionService | SQL database you operate (SQLite locally; PostgreSQL or MySQL such as Cloud SQL or AlloyDB in production) | Yes | Self-hosted agents on Cloud Run or GKE that need sessions in your own database for residency, private networking, or custom SQL reporting |
VertexAiSessionService | Agent Platform Sessions (managed, tied to an Agent Runtime resource) | Yes | Default on Agent Runtime; also usable from Cloud Run, GKE, or local code; pairs with VertexAiMemoryBankService |
Use InMemorySessionService only until the first deployment. Any multi-replica or serverless deployment needs DatabaseSessionService or VertexAiSessionService.
Latency, consistency, cost model, and operational burden for the stores agents use most.
| Service | Latency profile | Consistency | Cost model | Ops burden |
|---|---|---|---|---|
| Agent Platform Sessions + Memory Bank | API calls on the hot path; memory generation runs asynchronously | Session reads/writes are immediate; memories converge after background consolidation | Managed usage-based pricing | Lowest |
| Firestore | Low (serverless document reads/writes) | Strong, with ACID transactions | Per operation + storage | Very low (scales to zero) |
| Memorystore | Sub-millisecond in-memory | Strong on the primary; replicas are asynchronous | Provisioned capacity per hour | Low to medium (capacity sizing) |
| Cloud SQL | Low within the VPC | Strong relational ACID | Provisioned vCPU/RAM + storage | Medium (instances, backups, indexes) |
| AlloyDB | Low, with accelerated vector and analytical queries | Strong relational ACID | Provisioned primary and read pool nodes | Low to medium |
| BigQuery | Seconds per query | Strong after commit | Per data scanned or reserved slots + storage | Very low |
| Cloud Storage | Tens of milliseconds to first byte | Strong read-after-write | Per GB-month by class + operations + egress | Lowest |
| Agent Retrieval / Vector Search | Low-latency ANN | Near real time for streaming updates | Usage-based (Agent Retrieval) or deployed node-hours (Vector Search 1.0) | Low |
Benchmark with your own payload sizes and regions before committing to an SLO.
Four agent shapes and the services each one typically combines.
| Component | Customer support agent | Data analysis agent | High-QPS public FAQ agent | Regulated-industry agent |
|---|---|---|---|---|
| Runtime | Agent Runtime | Agent Runtime | Cloud Run (autoscaled) | GKE private cluster inside a VPC-SC perimeter |
| Sessions | VertexAiSessionService | VertexAiSessionService | Firestore or Memorystore (short-lived state) | DatabaseSessionService on AlloyDB with CMEK and private connectivity |
| Memory | Memory Bank (device history, preferences) | Memory Bank (analyst preferences) | None (anonymous traffic) | Memory Bank with strict topics and TTL, or none |
| Knowledge and vectors | Policy PDFs in Cloud Storage indexed by Agent Search or Agent Retrieval | BigQuery tables through the BigQuery MCP server | Agent Retrieval behind a Memorystore semantic cache | AlloyDB vectors with row-level security next to transactional rows |
| Tools and transactions | Order and refund lookups in Cloud SQL via MCP Toolbox | Code Execution sandbox for Python; charts saved with GcsArtifactService | Memorystore rate limits per IP or API key | Narrow custom MCP tools behind Agent Gateway |
| Telemetry | Cloud Trace + BigQuery eval sink | BigQuery cost and eval analytics | Cloud Monitoring cache hit rate + BigQuery logs | Audit logs to BigQuery; locked Cloud Storage retention bucket |
A cache or counter store can evict or lose keys during failover. Keep authoritative orders, approvals, and audit data in Cloud SQL, AlloyDB, Spanner, or Firestore, and treat Memorystore entries as rebuildable.
The Gen AI Evaluation Service (client.evals.evaluate() / agents-cli eval) grades both final response quality and multi-step tool trajectories.
| Metric family | What it does | Representative metrics |
|---|---|---|
| Adaptive rubrics (recommended) | Autorater dynamically generates prompt-specific pass/fail tests per eval case, then scores the agent response against them | GENERAL_QUALITY |
| Static rubrics | Applies fixed organization-wide pass/fail rules to every prompt | Custom brand tone / regulatory disclaimer rubrics |
| Agent & faithfulness metrics | Specialized LLM-as-judge autoraters for grounded agent turns | FINAL_RESPONSE_QUALITY, HALLUCINATION, TOOL_USE_QUALITY |
| Trajectory metrics | Deterministically compares actual tool-call sequence/args against expected reference steps | trajectory_exact_match, trajectory_in_order_match, trajectory_any_order_match, precision/recall |
| Computation metrics | Fast, deterministic programmatic checks | JSON Schema validation, SQL AST check, exact match, ROUGE/BLEU |
Starting Prompt + hidden Conversation Plan (persona, goal, edge-case constraints); (2) runs a simulated user LLM across multi-turn sessions to produce full traces.503, 429, timeout spikes, mocked edge-case payloads) without mutating live production APIs.OnlineEvaluator): continuously samples live production OpenTelemetry GenAI traces and runs async autoraters to alert on quality regressions.agents-cli eval).Final-answer metrics alone miss "right answer, wrong path" compliance bugs (e.g., issuing a refund without calling verify_eligibility first). Always pair outcome rubrics with trajectory metrics (trajectory_in_order_match / trajectory_exact_match) when tool ordering matters.
Online evaluation monitors run under a platform service agent with access to trace payloads and judge quota; restrict IAM permissions to create or edit monitors to prevent unauthorized trace inspection.
GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY=true (exports traces, metrics, and logs without raw prompt text) + OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental. Raw payload capture is opt-in: set OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=EVENT_ONLY and keep ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS=false (the value true is invalid with the latest conventions and collects nothing), paired with SDP de-identification and restricted log buckets (roles/logging.privateLogViewer).is_agentic=true): records both the cryptographic SPIFFE Agent Identity and the initiating human user principal to eliminate confused-deputy audit blind spots.Agent Optimizer): closed-loop programmatic optimizer that mutates system prompts/examples and validates candidates against your golden evalset via client.evals before rollout.| Production Symptom | First Diagnostic View & Fix |
|---|---|
| Slow p95 turn responses | Traces waterfall → check whether thinking_level=HIGH decode, serial MCP calls, or retrieval dominates |
| Agent hallucinates instead of calling a tool | Tools tab ("no-tool interactions") → sharpen tool docstring/schema + add trajectory eval |
| Recurring errors or quality drop after model migration | Failure Cluster Analysis + Evaluation tab → run programmatic Prompt Optimization against golden evals |
| Unexpected token spend spike | Models / Usage tabs → compact long session histories and align static prompt prefixes for implicit caching |
Setting OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=EVENT_ONLY writes raw user prompts and tool payloads into logs, risking PII exposure. Keep content capture disabled by default or pair it with SDP de-identification and restricted log buckets (Security guide).
| Error / situation | Meaning | Architectural fix |
|---|---|---|
429 RESOURCE_EXHAUSTED on Standard PayGo | Shared regional pool contention or project TPM/QPM quota exceeded | Route stateless calls to the global endpoint, use exponential backoff with jitter, enable Priority PayGo for critical bursts, or reserve Provisioned Throughput (GSUs; spills over to PayGo by default unless X-Vertex-AI-LLM-Request-Type: dedicated is set) |
| "Region temporarily at capacity" deploying to Agent Runtime | Physical regional compute pool saturation — not a project quota cap | Deploy to an alternate region, request smaller per-instance CPU/memory, or retry off-peak |
Agent Runtime representative default quotas (per project per region; verify current docs): query/streamQuery 90/min, session writes 100/min, append events 300/min, session reads 10,000/min, max 100 agent resources, Memory Bank writes 100/min and reads 300/min, sandbox executions 1,000/min.
Six FinOps levers: (1) tiered routing (3.5 Flash-Lite → 3.8 Flash → 4 Argon); (2) right-size thinking_level; (3) byte-identical static prompt prefixes for zero-storage-cost implicit caching; (4) session compaction & tool-output masking; (5) right-size min_instances and undeploy idle Vector Search 1.0 endpoints; (6) sample 1–5% of live traffic for OnlineEvaluator and run bulk offline evals on Flex PayGo or Batch inference.
Filing an IAM/project quota increase ticket will never resolve "Region temporarily at capacity". Switch region, reduce instance sizing, or retry instead.
| Trap | Why it's wrong | Correct move |
|---|---|---|
| Hosting a custom MCP server on Agent Runtime | Agent Runtime hosts stateful agents (reasoningEngines) | Deploy Streamable HTTP MCP servers on Cloud Run or GKE |
| Keeping chat history in Cloud Run memory | Serverless instances scale to zero or recycle | Externalize to Agent Platform Sessions, Memorystore, Firestore, or Cloud SQL |
| Self-hosting Gemini 4 Argon / 3.8 Flash on GKE | Proprietary Gemini weights are served only via managed endpoints | Call managed Gemini APIs; self-host open models (Gemma 4) if weights are needed |
| Running generated Python inside the agent container | Exposes IAM credentials and VPC access | Use the Code Execution sandbox or GKE Sandbox (gVisor) |
| Building custom RAG when SharePoint/Drive ACL sync is required | Re-implements brittle connector & permission logic | Use Agent Search with native ACL enforcement |
Default DOT_PRODUCT index under RAG Engine | RAG Engine queries cosine distance → FAILED_PRECONDITION | Configure index metric as COSINE |
| Quota increase ticket for "region at capacity" | Physical pool exhaustion, not a project quota cap | Alternate region, smaller CPU/RAM, or retry |
| Grading compliance agents only on final-answer quality | Misses skipped verification tools ("right answer, wrong path") | Add trajectory metrics (trajectory_in_order_match) |
| Fault-testing resilience against live production APIs | Triggers real side effects & non-reproducible states | Use Environment Simulation + 2-Step Agent Simulation |
| Global semantic cache over ACL-protected documents | Leaks confidential answers across users | Partition cache keys by ACL/tenant scope |
Querying an auto-registered Google Cloud MCP server with --location=us-central1 | Auto-registered McpServer resources exist only in global, so the call returns NOT_FOUND | Query with --location=global, or register the server manually in the region when metadata residency matters |
Registering an external MCP server without a toolspec.json | Manual registration records only the endpoint; the registry does not inspect the server, so no tools are discoverable | Upload a toolspec.json (MCP tool schema, 10 KB max) during registration |
Shipping with InMemorySessionService on Cloud Run | Sessions vanish on restart and are not shared across replicas | Switch to VertexAiSessionService or DatabaseSessionService before the first deployment |
reasoningEngines.Starting Prompt + Conversation Plan) and deterministic tool-fault injector.is_agentic=true) recording both SPIFFE Agent Identity and human principal.roles/mcp.toolUser plus resource IAM.googleapis/mcp-toolbox) with prebuilt database tools and parameterized custom tools defined in tools.yaml.toolspec.jsonVertexAiSessionService (managed Agent Platform Sessions), DatabaseSessionService (SQL database you operate), and InMemorySessionService (non-persistent process memory).SKILL.md-based skill packages; inherits project IAM (roles/aiplatform.*).gcp-skill-registryload_skill and run_skill_script.Listen to the 71-minute Agent Platform audio deep dive, open the 21-slide architectural deck and sketchnote infographic in the Hub Viewer, or drill scenario-based questions on build paths (Agent Studio vs. Managed Agents API vs. ADK), Agent Runtime, Memory Bank, 2-step simulation, and prompt optimization.