Production engineering on the Claude Messages API: choosing a model and effort level, request shape and roles, explicit criteria and few-shot examples, the tool-use loop and its pairing rules, stop_reason handling, streaming events, structured outputs and schema design, validation and retry loops, adaptive thinking, context windows, compaction and prompt caching, preserving facts in long sessions, Message Batches, multi-pass review, retriable vs. terminal errors, evals and cost levers, human review and confidence calibration, deployment platforms and data residency, and security for apps that read untrusted content.
The Claude API is a stateless HTTP interface. The core endpoint, POST /v1/messages, takes the whole conversation (system prompt, alternating user/assistant turns, tool definitions) and returns one assistant message made of typed content blocks plus a stop_reason and token usage. Official SDKs (Python, TypeScript, Go, Java and others) wrap the same API and add retries, streaming helpers and types.
The API remembers nothing between calls and never runs your tools. Your application owns history (what to resend, prune or compact), tool execution (and authorization), branching on stop_reason, retries, validation of outputs, and cost/latency instrumentation. Almost every exam-style defect is one of these responsibilities done wrong.
Treat every response as a state transition: read stop_reason first, then decide whether to show text, run tools, continue, compact, fall back or stop. Code that reads content[0].text without checking stop_reason is the root of most production bugs.
Model ID, max_tokens, system, messages, tools, optional thinking/effort and output format.
Pick the transport by who is waiting and how much work there is.
end_turn, tool_use, max_tokens, refusal, and the rest.
Append the assistant turn unchanged, add tool results, repeat until done.
| Tier | Profile | Typical fit | Watch out |
|---|---|---|---|
| Haiku | Fastest, lowest cost per token | High-volume classification, routing, extraction, real-time triage, sub-agent workers | Validate on your eval set before trusting it on nuanced steps |
| Sonnet | Balanced capability, speed and cost | Default starting point for most apps and agents, coding, tool use | Don't upgrade the whole pipeline when only one step fails |
| Opus | Most capable, slowest, highest cost | Hardest reasoning, long multi-step agent work, planning, review of critical output | Paying top-tier prices on steps a smaller tier passes |
output_config.effort (low, medium, high, xhigh, max; the accepted set varies by model).On Claude 4.7 and later, setting temperature, top_p or top_k to any non-default value returns HTTP 400. Steer behavior with prompting, examples, structured outputs and effort instead. Even on older models where temperature: 0 is accepted, output was never fully deterministic: test properties, not exact strings.
"Use the best model everywhere" is still a tier decision, just the most expensive one. And "set temperature to 0 for consistency" fails twice: it 400s on the newest models and never guaranteed identical text anyway.
| Field | Required? | What it does |
|---|---|---|
model | Yes | Model ID. Pin a fixed ID in production (see §17). |
max_tokens | Yes | Cap on output tokens for this response. Does not create context room or fix validation errors. |
messages | Yes | Conversation as user / assistant turns; content is a string or an array of blocks. |
system | No | Top-level system prompt (string or text blocks) for instructions that apply to the whole conversation. Newer models also accept mid-conversation role: "system" messages inside messages, placed after a user turn, to change instructions later without invalidating the cached prefix (check per-model support). |
tools / tool_choice | No | Tool definitions and how Claude may use them (§4). |
thinking / output_config | No | Adaptive thinking, effort, and format for structured outputs (§7, §9). |
stream | No | Server-sent events instead of one JSON body (§6). |
stop_sequences | No | Custom strings that end generation (stop_reason: "stop_sequence"). |
inference_geo | No | First-party data residency: "global" (default) or "us" (§17). |
text, image, document, tool_result (in user turns); prior text, tool_use, thinking, redacted_thinking (in assistant turns you echo back).id, role: "assistant", content[], stop_reason, stop_sequence, usage (input_tokens, output_tokens, cache_creation_input_tokens, cache_read_input_tokens).| System Prompt Altitude | What it looks like | Failure mode / why |
|---|---|---|
| Too specific (brittle state machine) | Hardcoded if/else trees, exact regexes, rigid step-by-step tool sequences in prose | Breaks on the first edge case; move deterministic rules into code, hooks, or schema validators |
| Goldilocks zone (heuristics & 4-step framework) | Clear role/goal, tool-selection heuristics, and a 4-step operating loop (Explore → Plan → Confirm on ambiguity → Act & Verify) | Gives the model strong guardrails while preserving flexibility to recover from unexpected tool outputs |
| Too vague (assumed shared context) | "Be a helpful assistant and follow best practices" | Forces the model to guess domain rules, tool boundaries, and stopping criteria |
import anthropic
client = anthropic.Anthropic() # reads ANTHROPIC_API_KEY from the environment
resp = client.messages.create(
model=PINNED_MODEL_ID,
max_tokens=1024,
system="You are a support assistant for Acme. Answer only from the provided policy.",
messages=[
{"role": "user", "content": "Can I return an opened item?"},
],
)
if resp.stop_reason == "end_turn":
answer = next(b.text for b in resp.content if b.type == "text")
Prefilling the final assistant turn to force a format is rejected with a 400 on Claude 4.6 and later. Use structured outputs (§7) instead. Also, content[0] is not always text: with thinking on, the first block is usually a thinking block. Select blocks by type.
In review, triage and extraction pipelines, precision (how many reported items are real) decides whether people trust the output. Vague instructions produce noisy output; confidence wording does not fix it. Define what qualifies, then show it.
| Vague instruction | Why it fails | Explicit replacement |
|---|---|---|
| "Check that comments are accurate" | Everything slightly imprecise gets flagged, so false positives pile up | "Flag a comment only when the behavior it claims contradicts what the code does" |
| "Be conservative" / "Only report high-confidence findings" | Doesn't define the boundary; the model's notion of "confident" is unchanged, so precision barely moves | Categorical report / skip lists: report bugs, security flaws, data loss; skip style, naming, and patterns used consistently in the repo |
| "Rate the severity" | Labels drift between runs and reviewers | A severity rubric with one concrete code example per level (critical / major / minor) |
<example> tags (inside <examples>); 3–5 examples usually work best.| Use few-shot to… | What the examples show |
|---|---|
| Handle ambiguous cases (which tool, escalate or not, is this a real gap) | The chosen action and why it beats the plausible alternative |
| Fix output format | The exact finding shape: location, issue, severity, suggested fix |
| Cut false positives | Contrast pairs: an acceptable pattern (not reported) next to a genuine issue (reported) |
| Reduce extraction hallucination | Correct extraction from varied layouts (inline citations vs. bibliography, informal amounts like "a couple of cups"), including an explicit null when a field is absent |
<examples>
<example>
<code>if user.is_admin or True:</code>
<finding>{"location": "auth.py:42", "issue": "condition is always true, so the admin check is bypassed",
"severity": "critical", "suggested_fix": "remove 'or True'"}</finding>
</example>
<example>
<code>for i in range(len(items)): # idiom used across this repo</code>
<finding>none: style preference consistent with the codebase, not reported</finding>
</example>
</examples>
Fixing a noisy reviewer by adding "only report findings you are very confident about". It doesn't define what counts. Write categorical criteria, add contrast examples, and measure precision per category.
name, description (what it does, when to use it and when not to), input_schema.
Response has tool_use blocks (each with an id) and stop_reason: "tool_use".
Authorize, validate inputs, run. Add a human gate before irreversible actions.
Append the assistant turn unchanged, then one user turn with a tool_result per call.
| Pairing rule | What breaks if you ignore it |
|---|---|
Every tool_use gets a tool_result whose tool_use_id equals the call's id | 400: result references an unknown ID, or a call is left unanswered |
| Results go in the very next user message; nothing in between | 400 on the next request |
In that user message, tool_result blocks come first; any text goes after them | Validation error, or the turn ends early |
| Parallel calls in one turn are all answered in one user message | Orphaned calls; splitting results across turns fails |
Failures are returned as a tool_result with is_error: true and a short message | An empty string reads as "no data", so Claude confidently reports a false result |
messages.append({"role": "assistant", "content": resp.content}) # echo unchanged
results = []
for block in resp.content:
if block.type == "tool_use":
try:
out = run_tool(block.name, block.input) # your authz + validation inside
results.append({"type": "tool_result", "tool_use_id": block.id, "content": out})
except ToolError as e:
results.append({"type": "tool_result", "tool_use_id": block.id,
"content": f"lookup failed: {e.short}", "is_error": True})
messages.append({"role": "user", "content": results}) # all results, one turn, first
tool_choice | Behavior | Notes |
|---|---|---|
{"type": "auto"} (default) | Claude decides whether to call a tool | Add "disable_parallel_tool_use": true inside this object for at most one call per response |
{"type": "any"} | Must call some tool | Forced use: not allowed with manual extended thinking, and some of the newest models return 400 |
{"type": "tool", "name": "..."} | Must call that tool | Same restrictions as any; prefer auto + strict: true or structured outputs where forcing is rejected |
{"type": "none"} | No tool calls | Tools stay defined (keeps the cache prefix stable) |
tool_resultrequired; use enums for closed sets.strict: true guarantees tool inputs match the schema.tool_result: content can be an array of text and image blocks (e.g. Playwright browser screenshots), enabling closed-loop visual UI verification.tool_result.pause_turn: send the assistant content back as-is to continue.| Architecture | How tools & context are handled | Scaling limits & best fit |
|---|---|---|
| Workflow / Hybrid RAG | Fixed code path; static chunking + embeddings + reranking injected before calling Claude | Predictable latency; fails on multi-hop code navigation (under-retrieval, over-retrieval, and misaligned retrieval across imports) |
| Shallow ReAct Loop / Standard MCP | Single messages[] loop calling JSON tools one turn at a time | 4 scaling bottlenecks: (1) upfront tool-schema bloat, (2) K-turn sequential ping-pong latency & repeated prefix billing, (3) pretraining mismatch (LLMs are trained on millions of real TypeScript/Python repos vs. synthetic JSON tool_use traces), and (4) JSON schemas cannot express loops, conditionals, or error-handling chains |
| Deep Agent Harness (e.g., Claude Code) | Explicit planning tool (TodoWrite) + isolated subagents + virtual/real filesystem (Glob/Grep/paginated Read) | Solves context crowding and goal drift over 10–100+ step tasks |
| Code-orchestrated tool calling (Anthropic programmatic tool calling; Cloudflare Code Mode) | Anthropic programmatic tool calling: Claude writes Python that runs in Anthropic's code execution container; tools opt in via allowed_callers, and intermediate results stay out of context. Cloudflare Code Mode (separate third-party variant): converts MCP tool schemas into a typed TypeScript API (declare namespace); LLM emits one TypeScript script run inside a network-isolated V8 isolate (~ms startup, no filesystem/env access) where tools are exposed as outbound RPC bindings (await env.ServerName.toolName(...)) | Collapses K sequential tool turns into 1 LLM inference pass and keeps intermediate tool payloads inside V8 memory (out of the LLM context window) until console.log outputs the final distilled result |
"Unknown tool_use_id" is a wiring bug, not a schema or description bug. And disable_parallel_tool_use is not a top-level request field: it lives inside tool_choice.
stop_reason | Meaning | Correct handling |
|---|---|---|
end_turn | Natural finish | Use the response |
tool_use | Claude is calling one or more tools | Run them, return paired tool_results, call again |
max_tokens | Hit your max_tokens cap (or the model maximum) | Raise the cap or continue the response; never parse truncated JSON as complete |
model_context_window_exceeded | Generation filled the context window | Treat as truncated; prune/compact history before the next turn |
stop_sequence | One of your stop_sequences fired | Read stop_sequence to see which |
pause_turn | A long server-tool loop was paused | Send the assistant content back unchanged to continue |
refusal | Claude declined (safety classifiers) | Returned as HTTP 200. Read stop_details; don't blind-retry the same request; reset or rephrase the refused turn, or retry on a fallback model where offered |
stop_details can carry a policy category (may be null) and an explanation that isn't guaranteed stable.message_delta event; discard the partial turn.max_tokens exceeds the window → newer models accept it and stop with model_context_window_exceeded if they run out.A refusal is a content decision, not a transient error: exponential backoff on the same request just repeats it. And max_tokens limits output length only; raising it does nothing for an oversized prompt or a crowded context.
| Event (in order) | Carries | Handler action |
|---|---|---|
message_start | Message shell, initial usage | Open a buffer for the turn |
content_block_start | Block index and type (text, tool_use, thinking) | Start a per-index accumulator |
content_block_delta | text_delta, input_json_delta (partial tool JSON), thinking deltas | Render text; append JSON fragments, don't parse yet |
content_block_stop | Block complete | Now parse tool input JSON and validate |
message_delta | Top-level changes: stop_reason, cumulative output usage | Branch on stop_reason (refusal, tool_use, max_tokens…) |
message_stop | Message complete | Only now commit the assistant turn to history |
ping | Keep-alive | Ignore |
max_tokens.message_stop means complete.Saving a half-streamed assistant turn leaves a malformed tool_use in history, and every later request fails validation, often surfacing only on retries. Parse tool input at content_block_stop; commit the turn at message_stop.
| Approach | Guarantee | Use when |
|---|---|---|
System-prompt output styles (format/persona instructions in system, e.g. YAML/HTML/tutor style) | None: best effort (steers tone, verbosity, or non-JSON formats like YAML/Markdown) | Human-facing presentation or custom personas; never rely on prompt prose for downstream JSON parsers |
JSON outputs: output_config.format = {"type": "json_schema", "schema": {...}} | The response text parses and matches the schema (constrained decoding) | The answer itself must be machine-readable (extraction, classification, reports) |
Strict tool use: "strict": true on a tool | Tool name and inputs match input_schema | Agents whose tool calls feed real systems |
resp = client.messages.create(
model=PINNED_MODEL_ID, max_tokens=1024,
messages=[{"role": "user", "content": email_text}],
output_config={"format": {"type": "json_schema", "schema": {
"type": "object",
"properties": {"name": {"type": "string"}, "demo_requested": {"type": "boolean"}},
"required": ["name", "demo_requested"],
"additionalProperties": False}}},
)
if resp.stop_reason == "end_turn": # check before parsing
data = json.loads(next(b.text for b in resp.content if b.type == "text"))
stop_reason. A refusal or max_tokens stop can leave output that doesn't conform.When a parser breaks on "almost JSON", don't add a third paragraph of formatting instructions or a regex repair step. Move format enforcement into the API with output_config.format or strict: true.
Define a tool whose input_schema is the record you want, then read the tool_use block's input (already a parsed object). Nothing has to be executed: the tool call is the output.
tool_choice for extraction | Guarantee | Use it when |
|---|---|---|
{"type": "auto"} | Claude may answer in text instead of calling the tool | You can tolerate a text reply, or the prompt states clearly when the tool applies |
{"type": "any"} | Must call one of the tools; Claude picks which | Several extraction schemas (invoice, receipt, contract) and the document type is unknown |
{"type": "tool", "name": "extract_metadata"} | Must call that tool | A specific extraction must run first (e.g. metadata before enrichment steps) |
any / tool) errors with manual extended thinking, and some of the newest models reject it with a 400. There, use auto + strict: true + a prompt that says when to call the tool, or output_config.format.thinking block.| Schema design choice | Why | Example |
|---|---|---|
Put in required only fields every source contains | A required field the document lacks pushes the model to invent a value | invoice_number required; po_number optional |
Make may-be-missing fields nullable ("type": ["string", "null"]) | "Not stated" becomes a valid answer instead of a fabrication | due_date nullable + "use null if not stated". Required-but-nullable keeps the key always present |
Enum + "other" + a detail string | Extensible categories without losing information | category: [..., "other"] + category_detail |
Add an "unclear" enum value | The model can flag ambiguity instead of guessing | sentiment: [positive, negative, neutral, unclear] |
| Put format-normalization rules in the prompt | The schema fixes types, not how messy source formats are rendered | "Dates as YYYY-MM-DD; amounts as numbers; weights in grams" |
record_invoice = {
"name": "record_invoice", "strict": True,
"description": "Record fields from ONE invoice. Use null when a field is not stated.",
"input_schema": {"type": "object", "additionalProperties": False,
"properties": {
"invoice_number": {"type": "string"},
"due_date": {"type": ["string", "null"], "description": "YYYY-MM-DD, or null if absent"},
"category": {"type": "string", "enum": ["hardware", "software", "services", "other"]},
"category_detail": {"type": ["string", "null"]},
"stated_total": {"type": "number"}},
"required": ["invoice_number", "due_date", "category", "category_detail", "stated_total"]}}
resp = client.messages.create(model=PINNED_MODEL_ID, max_tokens=1024, tools=[record_invoice],
tool_choice={"type": "tool", "name": "record_invoice"},
messages=[{"role": "user", "content": invoice_text}])
record = next(b for b in resp.content if b.type == "tool_use").input # dict, schema-valid
Objects need additionalProperties: false; numeric (minimum, maximum) and string-length constraints aren't supported (400), so enforce ranges in your validator; complexity caps include 20 strict tools per request, 24 optional parameters and 16 union-type (e.g. nullable) parameters across strict schemas. Enum capitalization isn't guaranteed: compare case-insensitively.
Marking every field required "for completeness". When the source lacks the value, a schema-valid but invented value comes back. Make it nullable or optional and say "null if not stated".
| Error class | Example | Removed by strict schema? | Fix |
|---|---|---|---|
| Syntax / shape | Broken JSON, wrong type, extra key | Yes (strict tools, JSON outputs) | Nothing more to do |
| Semantic | Line items don't sum to the total; value in the wrong field; impossible date | No | Validate in code → retry with error feedback → human review |
| Information absent | The total is only in an attachment you didn't send | No, and retries can't help | Return null / flag it, fetch the missing source, or route to a human |
stated_total is 1,240.00; check for a missing line or misread digit").calculated_total vs. stated_total: the model sums the line items and copies the printed total; your code compares them and flags mismatches.conflict_detected (boolean + note): the source contradicts itself (two different dates for one event). Report it; don't let the model pick one silently.detected_pattern on review findings: name the construct that triggered the finding (e.g. string-built-sql, unchecked-return). When developers dismiss findings, group dismissals by pattern to see which criteria cause false positives, then rewrite those criteria (§3).def extract_with_retry(doc, max_retries=2):
messages = [{"role": "user", "content": doc}]
for attempt in range(max_retries + 1):
resp = client.messages.create(model=PINNED_MODEL_ID, max_tokens=2048, tools=[record_invoice],
tool_choice={"type": "tool", "name": "record_invoice"}, messages=messages)
call = next(b for b in resp.content if b.type == "tool_use")
errors = validate(call.input) # sums, ranges, cross-field business rules
if not errors:
return call.input
messages += [{"role": "assistant", "content": resp.content},
{"role": "user", "content": [{"type": "tool_result", "tool_use_id": call.id, "is_error": True,
"content": "Validation failed:\n" + "\n".join(errors) +
"\nRe-read the document and fix only these fields. Use null if the value is not in it."}]}]
return route_to_review(doc, call.input, errors) # don't loop forever on absent data
"Retry up to five times until validation passes." If the value isn't in the source, the loop eventually produces a fabricated value that passes. Absent data is an input problem: flag it or fetch the source.
The older extended thinking mode made you allocate a fixed token budget. Adaptive thinking lets the model judge how much to reason per request; you steer depth with effort.
thinking={"type": "adaptive"} plus output_config={"effort": "high"}.{"type": "enabled", "budget_tokens": N} is deprecated on Claude 4.6 and returns 400 on 4.7 and later.budget_tokens, switch the type to adaptive, control depth with effort.thinking.display: "summarized" returns readable summaries; "omitted" returns an empty thinking field.| Rule in tool-use loops | Why |
|---|---|
Echo the assistant message exactly, including thinking and redacted_thinking blocks, when you return tool results | Thinking blocks carry a cryptographic signature; editing, summarizing or filtering them gives a 400 |
| Keep one thinking configuration per assistant turn | A tool-use loop is one assistant turn; change settings only between turns |
Manual extended thinking allows only tool_choice auto or none | Forced tool use is incompatible with manual mode |
| To save context, prune or compact older completed turns | Never "save tokens" by stripping blocks from the active turn |
Changing thinking or effort between requests invalidates the messages prompt cache (and on some models the system and tool caches too). Pick a setting per workload and keep it stable.
| Context strategy (Write / Select / Compress / Isolate) | How | Use when |
|---|---|---|
Prune (Compress) | Drop or truncate old tool results and stale turns in your code | Tool outputs dominate the history |
Context editing (Compress) | Beta context_management edits that clear old tool uses (and thinking) server-side past a trigger | Long agent loops with bulky, one-time tool output |
Compaction (Compress) | Beta context_management edit compact_20260112: older history is summarized into a compaction block (default trigger 150k input tokens; optional pause_after_compaction returns stop_reason: "compaction") | Persistent agents that run for hours or days |
Progressive disclosure & tool scoping (Select) | Pass only relevant tools per turn or load skills/docs on demand via retrieval instead of dumping 50+ schemas up front | Large tool catalogs where schema tokens bloat every turn and degrade tool selection |
Virtual filesystem offloading (Write + Select) | Persist plans, scratchpads, and bulky intermediate results to external state/files; read back targeted slices | Multi-step tasks that must survive compaction without losing exact specs or artifacts |
Sub-agent handoff (Isolate) | Give a bounded sub-task its own fresh messages[] array; pass an explicit context contract and return only the synthesized answer + required raw artifacts | Exploration, deep search, or independent verification that would flood the main context |
| Measure | Token-counting endpoint before sending; watch usage | Always; production payloads are larger than test fixtures |
An agent that picks the wrong tool on Turn 1 has a schema/description collision or tool bloat problem (too many overlapping tools; fix with Select / progressive disclosure or sharper descriptions). An agent that picks the right tool on turns 1–4 and the wrong one from Turn 5+ does not have a schema problem: accumulated tool_result output is crowding out instructions (fix with Compress / prune / edit / compact, not by rewriting descriptions or raising max_tokens).
messages[] array—never assume it inherits parent conversation history.messages.create() call with a fresh messages[] array containing only the spec and candidate artifact—never ask the same context that generated the artifact to grade its own work.cache_control: {"type": "ephemeral"}; up to 4 breakpoints per request.tools → system → messages. A change at one level invalidates that level and everything after it."ttl": "1h" is optional.cache_creation_input_tokens vs. cache_read_input_tokens in usage.tool_choice, toggling images, changing thinking/effort, or swapping system output styles mid-session (invalidates both system and downstream messages caches).A 0% hit rate with a timestamp, user ID or ticket text placed before the breakpoint. Each request writes a new entry and never reads one. A longer TTL only makes those writes cost more; move the dynamic text after the breakpoint.
The API is stateless, so every request must carry the conversation. As sessions grow, the tempting fixes (summarize, compact, drop turns) quietly lose the exact facts the task depends on.
| Risk | Symptom | Fix |
|---|---|---|
| Progressive summarization | "Refund of 1,240.18 for order A-7731, promised by 14 March" becomes "customer wants a refund" | Keep a persistent case facts block outside the summarized history |
| Tool-output bloat | An order lookup returns 40+ fields when 5 matter; resent on every later turn | Trim tool output to the relevant fields in your wrapper before appending the tool_result |
| Lost in the middle | Findings buried in the middle of a long aggregated input get skipped | Put a key-findings summary first; organize details under explicit section headers or XML tags |
| Partial history | Only the latest user message is sent; the model loses commitments and context | Send the complete (or deliberately managed) history every request |
<document> tags with source metadata.case_facts = {"customer_id": "C-1029",
"issues": {"A-7731": {"amount": 1240.18, "ask": "refund", "promised_by": "2026-03-14",
"status": "return label sent"}}}
system = [{"type": "text", "text": STATIC_POLICY, "cache_control": {"type": "ephemeral"}},
{"type": "text", "text": "<case_facts>\n" + json.dumps(case_facts) + "\n</case_facts>"}]
Trusting compaction or a rolling summary to keep amounts, dates and IDs. Summaries paraphrase. Keep authoritative facts in a structured block your code maintains, and trim tool output at the source.
custom_id_, -). Results can come back in any order, so match on custom_id, never on position.processing_status is ended → fetch results (each succeeded, errored, canceled or expired). Results stay downloadable for 29 days.tool_use, the next step needs a new request. Batches suit single-shot work; forced extraction tools (§7) fit because the tool_use input is the output.| Workload | Best transport | Why |
|---|---|---|
| Nightly classification of 40k tickets, read next morning | Message Batches | Half price, no rate-limit fight, latency doesn't matter |
| Pre-merge PR check a developer is waiting on | Synchronous API | Blocking workflow; batch has no latency guarantee |
| Agent that must query your database between steps | Synchronous tool loop | Batch can't run your tools mid-request |
| Chat UI with long answers | Streaming | A person is waiting; perceived latency |
| Backend fan-out of a few hundred calls per user action | Async client with bounded concurrency | Needs results in seconds |
| Same 5k-token instructions on every batched item | Batches + prompt caching | The discounts can be combined (batch cache hits are best effort) |
errored / expired results by custom_id and resubmit just those, with fixes (e.g. split documents that exceeded the context window). Errored, canceled and expired requests aren't billed.Splitting a list into chunks and calling messages.create() per item is not batching: it is serial synchronous traffic that hits rate limits. Smaller chunks or more backoff don't change that; the Batches API does. And the reverse trap: moving a blocking check (pre-merge, user waiting) to batches for the 50% saving, which removes any latency guarantee.
The session that generated an artifact keeps its own reasoning in context and tends to defend it. "Now review your work critically" in the same thread, or more thinking, doesn't remove that anchor.
| Pattern | How | Use it for |
|---|---|---|
| Independent reviewer instance | A separate messages.create with a fresh messages list holding only the spec/rubric and the artifact, without the generator's reasoning | Catching subtle bugs in generated code, contracts, extractions |
| Per-file local pass | One focused request per file: local bugs, security, logic | Large PRs where a single prompt dilutes attention |
| Cross-file integration pass | A separate pass over the per-file structured findings plus interfaces: data flow, contracts, consistency | Issues no single file shows; resolving contradictory local findings |
| Confidence-annotated verification | Each finding carries a self-reported confidence; route by threshold | Sending low-confidence findings to humans; thresholds must be calibrated (§16) |
Builder session produces the artifact (code, extraction, report).
One fresh request per file; structured findings with detected_pattern and confidence.
New request over the findings and interfaces: cross-file data flow and contradictions.
High-confidence findings auto-comment; low-confidence ones go to a human.
Reviewing a 30-file PR in one prompt, or asking the generating session to "double-check". The first dilutes attention and yields contradictory findings; the second inherits the generator's blind spots. Split into local + integration passes and use an independent instance.
| Status / type | Class | Action |
|---|---|---|
400 invalid_request_error (bad shape, prompt too long, unsupported param, spend limit you set) | Terminal | Fix the request; retrying the same body fails identically |
| 401 authentication · 403 permission · 404 not found | Terminal | Fix key, permissions or model ID; fail fast |
402 billing_error · 413 request_too_large | Terminal | Fix billing; shrink the payload below the endpoint's size limit |
429 rate_limit_error | Retriable | Wait retry-after, then backoff + jitter. A spend-cap 429 has no retry-after and keeps failing |
500 api_error · 504 timeout_error | Retriable | Exponential backoff; for 504 on long requests, stream instead |
529 overloaded_error | Retriable | Backoff with jitter; consider a fallback model in the orchestration layer |
HTTP 200 + stop_reason: "refusal" | Not an HTTP error | Content decision: handle as in §5, don't backoff-retry |
retry-after.max_retries (client-wide or per request); 0 disables.request-id response header for support.Wrapping SDK calls in your own 5-attempt loop with default SDK settings gives up to 15 attempts per request against a rate limit. One layer owns retries. And a bare except Exception: pass retries 400s pointlessly while hiding the real error.
| Output shape | Grader | Catches / misses |
|---|---|---|
| Label, number, enum, JSON field | Exact match / code assertion | Cheap, deterministic; can't judge free text |
| Structured object, tool trajectory | Code checks on properties (fields, ranges, expected/forbidden tools, call counts) | Robust to wording; misses tone and nuance |
| Free-text answer, summary, tone | LLM judge with a rubric | Scales; must be calibrated against human labels |
| High-stakes or novel behavior | Human review | Gold standard; slow and costly |
| Cost lever | Effect | Number / rule |
|---|---|---|
| Model tier per step | Largest price driver | Start balanced; move steps on eval evidence |
| Effort / thinking | Thinking billed as output | Low or off for classifiers and routers |
| Prompt caching | Cheaper repeated prefixes, faster first token | Write 1.25× / 2×, read ≈0.1× |
| Message Batches | Half price for offline work | 50%, ≤24 h |
| Context size | Every resent token is billed again | Prune, edit, compact; count tokens |
| Call count / fan-out | Multi-agent runs multiply tokens | Fan out only for truly independent subtasks |
| Streaming | Perceived latency only | No cost change |
| US-only residency (first party) | Premium | 1.1× on Claude 4.6+ with inference_geo: "us" |
| Workspace spend caps & rate limits | Hard blast-radius control per team/env | Separate workspaces + monthly spend limits in Console (settings/limits); spend-cap 429s have no retry-after |
Quoting the per-call cost of a pilot. Instrument tokens, latency and errors per call from day one, multiply by production volume, size for the heavy tail and p95 rather than the median, and set the reliability floor before tuning cost.
An aggregate number hides segment failures. "97% field accuracy" can mean 99% on typed invoices and 70% on handwritten receipts, or one field (say tax_id) failing everywhere. Slice accuracy by document type × field before you reduce human review.
| Practice | How | What it prevents |
|---|---|---|
| Segmented accuracy | Report accuracy per document type, per field, per source | Automating a segment that quietly fails |
| Field-level confidence | The model outputs a confidence per field (alongside "unclear" / conflict_detected flags) | One document-level score hiding one bad field |
| Calibration on labeled data | On a labeled validation set, measure real accuracy per confidence band; set thresholds per field / doc type where accuracy meets the target | Trusting raw self-reported confidence, which isn't calibrated until checked |
| Risk-based routing | Low confidence, conflicts, ambiguous or contradictory sources → human queue, highest-risk first | Spending scarce reviewer time on easy items |
| Stratified random sampling | Keep sampling auto-approved, high-confidence items within each stratum (doc type, field, source) | Unmeasured error rates and missed new error patterns or drift; plain random sampling under-samples rare types |
Reviewer corrections become labeled data → recalibrate thresholds, add few-shot examples for the new error patterns (§3), extend the eval set (§15).
"Overall accuracy is 97%, so remove human review", or "review only low-confidence items". The first ignores segments; the second never measures errors among auto-approved items. Validate per segment, calibrate thresholds on labeled data, and keep stratified sampling running.
| Platform | Operated by / identity | Residency control | Pick it when |
|---|---|---|---|
| Claude API (first party) | Anthropic; API keys and workspaces | inference_geo: "global" (default) or "us" (1.1×), Claude 4.6+ only. No EU-only value; older models 400 on the param | No binding cloud or EU constraint; newest features and betas first |
| Claude Platform on AWS | Anthropic-operated, AWS billing | Same global / us geographies | First-party features with AWS procurement |
| Amazon Bedrock | AWS-operated; IAM, AWS bill | Global endpoint, single-region endpoints, or geographic inference profiles (e.g. eu.-prefixed IDs); regional costs 10% more | Workload and compliance posture already live in AWS |
| Google Cloud (Vertex AI) | Google-operated; Cloud IAM and billing | Global, multi-region (us / eu) or regional (some models); 10% premium off global | Workload and compliance posture already in Google Cloud; EU residency |
| Microsoft Foundry | Anthropic-operated, on Azure | Varies by deployment type | Customer standardised on Azure |
claude-sonnet-4-6) is a fixed snapshot; updates ship under new IDs.claude-haiku-4-5-20251001); the short alias moves to the newest dated snapshot.anthropic.claude-… (dated models end in -v1:0; optionally global. / us. / eu. prefixed), Google Cloud …@YYYYMMDD for dated models and the API format from 4.6 on."Inference must stay in the EU" cannot be met with first-party inference_geo (global or US only). Use Bedrock regional/EU routing or the Google Cloud eu multi-region, decided at scoping because residency is pass/fail. Encryption at rest does not change where inference runs.
| Risk | Control that actually bounds it | Not enough on its own |
|---|---|---|
| Indirect prompt injection via fetched pages, emails, files or tool output | Treat tool/fetched content as data; delimit it; scan it before it enters context; isolate readers from action tools | Filtering only the user's prompt |
| Excessive agency (agent sends, deletes, pays) | Least privilege at the tool registry: remove tools the task doesn't need; read-only credentials | "Never send email" in the system prompt |
| Side-effecting tool misuse | Deterministic authorization in your code before execution (allowlist, caller identity, amount scope), failing closed; human approval for irreversible actions | An output filter that runs after the tool already executed |
| Forged identity ("I am an admin") | Inject verified role and tenant server-side from your auth layer | Asking the model to verify the claim |
| Malformed or hostile tool inputs | strict: true plus server-side validation; parameterised queries; fixed write paths | Trusting model-generated arguments |
| Leaked secrets and PII | Keys in env/secret manager, never in prompts; pass reference IDs, not raw PII; per-tenant keys or workspaces | "Don't reveal secrets" instructions |
| Unsafe rendering of model output | Escape or sanitize before display; never execute model HTML/JS | Assuming model output is trusted |
| Programmatic tool calling / LLM-generated code | Run generated code inside a capability-restricted sandbox (e.g., V8 isolate with zero ambient network, filesystem, or env-var access; only explicit outbound RPC bindings) | Running LLM-generated Python/JS on the host or passing raw API credentials into the sandbox |
"All our users are trusted employees, so injection isn't a risk." The attacker is whoever wrote the web page, email or document the agent reads. Prompt rules and bigger models are probabilistic; capability removal and pre-action authorization are not.
| Symptom / scenario | Root cause → correct fix | Tempting wrong answer |
|---|---|---|
| "tool_result references unknown tool_use_id" | Result ID ≠ call ID, or result not in the next user turn → echo block.id; results first in the next message | Rewriting the tool description |
| 400 after "trimming" history to save tokens with thinking on | Thinking block edited/dropped, signature mismatch → echo turns unchanged; compact older turns | Raising max_tokens |
| Validation error only on retries after a dropped stream | Half-streamed turn saved → commit at message_stop; discard partial turns | Changing the tool schema |
| Correct tool on turns 1–4, wrong tool from turn 5 | Context crowded by tool output → prune, edit or compact; measure tokens | Clarifying descriptions or raising max_tokens |
| Agent repeats a hallucinated API path even after context compaction | Context poisoning survived summarization → truncate/reset history to before the poisoned turn | Running compaction again or lowering temperature |
| Orchestrator spawns a subagent to generate a config file, gets "Completed!" back, and lacks the actual config | Subagent returned a summary instead of the raw artifact → require verbatim output or write to a shared store/file | Increasing the subagent's max_tokens |
| Agent with 40 MCP tools suffers 12-turn sequential tool ping-pong and 80k schema tokens per turn | Standard ReAct loop bottleneck → use progressive tool disclosure (Select) or code-orchestrated tool calling (Anthropic programmatic tool calling, or Code Mode in a V8 isolate) | Switching to a larger model tier |
| Model says "no orders found" during a DB outage | Failure returned as "" → tool_result with is_error: true | "Never say no orders found" in the prompt |
| Answer silently missing; logs show HTTP 200 | stop_reason: "refusal" or max_tokens not checked → branch on stop_reason | Retrying on HTTP status only |
| Same refusal repeats every turn | Refused turn kept in history → remove/rephrase it or reset | Exponential backoff |
| Nightly job hits 429 every night | Sync loop over items → Message Batches + custom_id | Smaller chunks |
| Batch results attached to the wrong tickets | Matched by position → match on custom_id | Sorting results by timestamp |
| 429 storms despite "retries" | Own loop stacked on SDK retries → one retry owner, honor retry-after, jitter | More attempts |
400 after model upgrade on temperature / budget_tokens | Unsupported on 4.7+ → omit sampling params; adaptive thinking + effort | Lowering the values |
| 0% cache hit rate | Dynamic text before the breakpoint, or tool_choice/effort changing per call → static prefix first, stable settings | 1-hour TTL |
| Parser breaks after an upstream release; can't roll back | Moving alias → pin the full ID, keep prior pin, eval-gate promotion | Adding parser retries |
| EU-only residency requirement on first-party API | inference_geo has no EU value → Bedrock/Google Cloud EU routing | inference_geo: "us" + encryption |
| Summarizer agent forwards a thread externally | Injection via fetched email + unneeded send tool → treat content as data; remove the tool | Stricter user-input regex |
| Code reviewer floods PRs with style nits; developers start ignoring real bugs | Vague criteria → categorical report/skip rules + contrast few-shot examples; disable the noisiest category while fixing it | "Only report high-confidence findings" |
Extractor returns plausible but invented po_number values | Field is required but often absent → nullable/optional + "null if not stated" | More retries |
Retry loop never fixes total; the value is only in an attachment that wasn't sent | Information absent → stop retrying; flag, fetch the source or route to review | Raising the retry cap |
| Support agent forgets the refund amount and deadline after 30 turns | Rolling summary dropped exact facts → persistent case-facts block; trim tool outputs | Switching to a bigger model |
| 97% overall accuracy, but handwritten receipts are often wrong | Aggregate hides segments → accuracy per doc type × field; stratified sampling; calibrated thresholds | Removing human review |
| Pre-merge check moved to Message Batches; developers wait hours | Batch has no latency SLA → sync API for blocking checks, batch for overnight | Polling the batch more often |
This code ran on an older model. After moving to a Claude 4.7+ pinned ID it fails or misbehaves. Find at least three defects before revealing.
resp = client.messages.create(
model=NEW_PINNED_ID, max_tokens=4096,
temperature=0, # (a)
thinking={"type": "enabled", "budget_tokens": 8000}, # (b)
tools=TOOLS, messages=messages,
)
messages.append({"role": "assistant", # (c)
"content": [b for b in resp.content if b.type != "thinking"]})
call = next(b for b in resp.content if b.type == "tool_use")
try:
out = run_tool(call.name, call.input)
except Exception:
out = "" # (d)
messages.append({"role": "user", "content": [
{"type": "text", "text": "Here is the result:"}, # (e)
{"type": "tool_result", "tool_use_id": call.id, "content": out},
]})
Fixes. What must change, and why?
temperature: non-default sampling values return 400 on Claude 4.7+. Steer with the prompt instead. (b) Replace the manual budget with thinking={"type": "adaptive"} and output_config={"effort": "high"}; budget_tokens returns 400 on 4.7+. (c) Append resp.content unchanged: filtering thinking blocks breaks the signature check on the next call. (d) Return failures as a tool_result with is_error: true and a short message, never "". (e) Put tool_result blocks first; text after. Bonus: check resp.stop_reason == "tool_use" before looking for a call, and loop over all tool_use blocks in case of parallel calls.
text, image, document, tool_use, tool_result, thinking, redacted_thinking.end_turn, tool_use, max_tokens, stop_sequence, pause_turn, refusal, model_context_window_exceeded.tool_result that must equal the id of the tool_use block it answers.tool_result telling Claude the tool failed, so it can retry, ask, or stop instead of trusting empty data.auto (default), any, tool, none; holds disable_parallel_tool_use.strict: true on a tool: inputs are guaranteed to match its input_schema.thinking: {type: "adaptive"}; the model sizes its own reasoning, steered by output_config.effort.low → max (set varies by model). Higher = slower and costlier.content_block_stop.cache_control marker ending a reusable prefix; up to 4 per request; TTL 5m (default) or 1h.Write (persist outside context), Select (progressive disclosure/scoped tools), Compress (prune/edit/compact), and Isolate (subagents with clean messages[]).allowed_callers), so chained calls and intermediate results stay out of the context window. Cloudflare Code Mode is a separate third-party variant (TypeScript in a V8 isolate with RPC bindings).global or us (Claude 4.6+).["string", "null"] so "not stated" is a valid answer instead of a fabricated value.A visual walkthrough of the Messages API request/response lifecycle, tool-use loops, adaptive thinking & effort, SSE streaming, prompt caching, server-side compaction, and multi-cloud deployment.
Listen to the Claude Platform & API audio deep dive, or open the 2-column study guide PDF, the 23-slide Architecting Claude deck, and the AI API Architecture Overview infographic in the Hub Viewer.
Q1. Claude returns two tool_use blocks in one response. How must the next request be structured?
tool_result blocks, each with the matching tool_use_id (any text goes after them). Failed calls use is_error: true.
Q2. How do max_tokens and model_context_window_exceeded differ, and what happens when the input alone is too large?
max_tokens means the output hit your cap; raise it or continue. model_context_window_exceeded means generation filled the context window; treat it as truncated and shrink history. An input that alone exceeds the window is rejected with a 400 before generation.
Q3. Why is a refusal not handled by your retry logic, and what should the app do?
stop_reason: "refusal", so status-based retry never sees it, and resending the same request repeats it. Read stop_details, remove or rephrase the refused turn (or reset history), or retry on a fallback model where offered.
Q4. In a streaming handler, when is it safe to parse tool input and when is it safe to commit the turn?
input_json_delta fragments and parse at that block's content_block_stop. Commit the assistant turn to history only after message_stop; on a dropped stream, discard the partial turn.
Q5. A team migrates to a Claude 4.7+ model and gets 400s. The request sets temperature: 0.2 and thinking: {type: "enabled", budget_tokens: 6000}. What changes?
thinking: {type: "adaptive"} plus output_config: {effort: ...}.
Q6. A prompt with a 6k-token static system prompt shows 0% cache reads. The breakpoint is on a system block that begins with today's date. Fix?
tool_choice, tools and effort stable too. A longer TTL would only raise write costs.
Q7. An EU insurer requires inference to stay in the EU. Which deployment choices satisfy it?
inference_geo, which supports only global or us. Use Amazon Bedrock regional/EU routing or the Google Cloud eu multi-region (both carry a regional premium), pin the model ID, and verify feature availability on that platform.
Q8. Your code wraps each SDK call in its own five-attempt retry loop. What's wrong?
retry-after, so attempts multiply against a rate limit. Let one layer own retries (or set max_retries), fail fast on 4xx terminal errors, and put fallbacks in orchestration.