Your AI Agent Is Not Failing at Answers — It Is Failing at State

The Agent Can Find the Right Website and Still Be Wrong

Most agent evaluations ask whether the final answer is correct. That is necessary, but it is not enough: an agent can reach the right source, produce a plausible response, and fail because it queried the source in the wrong state.

This is not a hypothetical edge case. Specialized websites hide evidence behind filters, views, hierarchies, date ranges, permissions, and query scopes. A regulatory database may return a valid response for the wrong jurisdiction. A commerce system may show the right product in the wrong inventory state. A cloud console may expose the right resource while the agent silently ignores the account or region constraint. In each case, the final prose is downstream of a state error.

The most useful recent evidence comes from SGR-Bench, a benchmark from Peking University and Beijing University of Technology that names and isolates this capability: state-gated retrieval, or SGR. Its 100 expert-curated tasks span 12 public data ecosystems. The benchmark’s strongest evaluated system reached 66.18% item-level F1, while an audit of 156 analyzable failed trajectories found retrieval-scope drift accounted for 37.2% of failures and criterion mismatch for 27.6%. Final answer composition accounted for only 10.3%.

That distribution changes the engineering priority. If most failures happen before synthesis, adding a better answer grader or a longer response prompt attacks the wrong layer. Production agents need evaluation that records and verifies state transitions—not merely the words they eventually return.

Why Final-Answer Accuracy Hides the Real Failure

A final answer compresses a long execution into a small artifact. That compression is convenient for a user and destructive for debugging.

Consider an agent asked to list all regulatory enforcement actions in a date range for a particular institution type. The agent might discover the correct regulator, open its search interface, select a date range, choose a category, and extract rows. If it leaves the category at its default value, the resulting table can contain real records with real fields. A text grader may find the entities and dates internally coherent. The answer is still wrong because the retrieval state did not represent the user’s constraints.

This is the difference between content correctness and query-state correctness:

Layer	Question	Typical signal	Failure that slips through
Intent	Did the agent preserve what the user asked?	Constraint extraction	“All subsidiaries” becomes the parent company
State	Did it configure the source correctly?	Filter, scope, view, and parameter trace	Correct database, wrong jurisdiction
Evidence	Did it read the answer-bearing material?	Cited rows and source spans	Relevant page, but default result set
Composition	Did it assemble the requested output?	Item and row-level metrics	Correct fields, wrong population
Operations	Did execution stay safe and economical?	Tool calls, retries, latency, cost	Twelve calls and a duplicate side effect

These layers are coupled, but they should not be scored as one number. A perfect final sentence cannot compensate for a wrong retrieval scope. Conversely, an agent can take the right path but phrase its conclusion poorly; that is a composition defect, not a retrieval defect.

SGR-Bench makes this separation visible. Its benchmark compares item-level and row-level F1 and audits trajectories by error type. Criterion mismatch can preserve a moderate item score while collapsing row correctness: the agent may retrieve many locally valid values, but use the wrong field definition or decision rule to assemble complete rows. That is exactly the kind of failure a single answer-level score hides.

State Is a First-Class Artifact, Not an Implementation Detail

Treat the agent’s effective retrieval state as a versioned artifact that can be inspected, replayed, and evaluated independently of the final answer.

A useful state record is not a screenshot and not a model’s hidden reasoning. It is a structured record of the externally meaningful choices that determine what evidence becomes available. For a web or API retrieval task, that can look like this:

{
  "source": "regulatory_actions_api",
  "scope": {
    "jurisdiction": "US",
    "institution_type": "credit_union",
    "date_start": "2025-01-01",
    "date_end": "2025-12-31"
  },
  "view": "enforcement_actions",
  "filters": {
    "status": ["final"],
    "product": ["consumer_credit"]
  },
  "evidence_request": {
    "fields": ["institution", "action_date", "action_type"],
    "completeness": "all_matching_rows"
  }
}

The format is less important than the contract. Before retrieval, the agent should produce or update an explicit state object. Every tool call should declare which part of that state it reads or changes. The system should then be able to answer four questions:

1. Which user constraint created this state field?
2. Which tool call changed it?
3. Which evidence was retrieved under that exact state?
4. Did the final rows satisfy the state’s scope and completeness requirements?

This is a practical extension of the trace-first advice in Google’s October 2026 agent evaluation guide. Google recommends combining unit tests, deterministic metrics, LLM-judge rubrics, and integration tests, while inspecting tool calls rather than only the final response. It also recommends frozen golden datasets, boolean checks where possible, and tracking latency, token use, API cost, and tool-call counts beside answer quality.

The important synthesis is that “inspect the trace” must include semantic state, not just chronology. A timestamped list of browser clicks is insufficient if the evaluator cannot tell whether a click changed the active jurisdiction, whether a filter was overwritten, or whether a later call silently reset the scope.

Two Independent Benchmarks Point to the Same Design

SGR-Bench identifies state errors in specialized retrieval. Q2D-Web shows why retrieval evaluation must also resemble the traffic and scale of deployed agentic search. The two efforts measure different layers, but together they expose a common mistake: evaluating a simplified surrogate instead of the system’s real operating conditions.

Q2D-Web, from researchers at Perplexity AI, pairs a roughly 190-million-document web corpus with approximately 70,000 agent-reformulated queries in ten languages, sampled from nine months of PII-free production traffic. The queries are not ordinary user strings; they are the machine-written reformulations that first-stage retrievers serve in production. The benchmark evaluates 13 retrievers and uses three relevance-judgment sets to reduce dependence on any one labeling source.

Its result that a one-third subcorpus can preserve full-corpus model ordering is useful operationally, but the deeper lesson is methodological: the benchmark preserves the distribution that matters. It evaluates what the retrieval layer actually sees, including machine reformulation, multilingual queries, and deep candidate pools.

SGR-Bench contributes the missing state dimension. It asks whether the agent can configure the source so that answer-bearing evidence becomes accessible. Q2D-Web asks whether a retrieval system can surface relevant evidence at production scale. A robust agent evaluation program needs both:

Evaluation surface	What it catches	What a final-answer test misses
Production-shaped retrieval	Query reformulation, recall, language and domain variance	Evidence never entering the candidate set
State-gated tasks	Scope drift, wrong filters, criterion mismatch	Correct source queried under wrong conditions
Trajectory checks	Tool choice, parameter changes, retries, side effects	A plausible summary of a bad path
Output checks	Missing items, incorrect fields, unsupported claims	—
Operating metrics	Latency, cost, rate limits, duplicate actions	Quality that looks fine but cannot scale

This is why an “agent accuracy” dashboard is usually a category error. It combines retrieval, state management, reasoning, and composition into one value that is too coarse to guide a fix.

The Production Pattern: Evaluate at Three Boundaries

Use three evaluation boundaries, each with its own golden cases and promotion rule: state construction, state execution, and state-to-answer fidelity.

1. State construction: did the agent translate intent correctly?

Start with cases where the user’s constraints are explicit and cases where they are distributed across turns. Score the extracted state with deterministic assertions. If a user says “final actions against credit unions in 2025,” the test should verify the institution type, status, and date range—not ask a judge whether the extracted JSON “looks reasonable.”

Test negative constraints too. “Exclude preliminary actions” is a state requirement even if the user never uses the word filter. A useful golden set should contain adjacent values that are easy to confuse: parent versus subsidiary, fiscal versus calendar year, state versus federal jurisdiction, active versus archived records.

2. State execution: did tools preserve the contract?

Every state-mutating tool should emit before-and-after state, not just a success code. The evaluator should fail a trace when a tool resets a filter, drops a scope field, or issues a query whose parameters disagree with the declared state.

A minimal event schema can be implemented without coupling the evaluator to a particular agent framework:

{
  "trace_id": "run-1842",
  "turn": 7,
  "tool": "search_actions",
  "state_before_hash": "sha256:…",
  "arguments": {"status": "final", "jurisdiction": "US"},
  "state_after_hash": "sha256:…",
  "evidence_ids": ["action-884", "action-901"],
  "assertions": {
    "scope_preserved": true,
    "criteria_preserved": true
  }
}

Do not treat a hash as proof of semantic correctness; it only makes changes observable and replayable. The evaluator still needs to compare normalized state fields with the user-derived contract. But hashes, event IDs, and immutable evidence references make regressions reproducible when a prompt, model, tool schema, or upstream dataset changes.

3. State-to-answer fidelity: did the answer stay inside the state?

Finally, verify that every output row belongs to the declared population and that completeness claims are justified. For structured tasks, item-level F1 is not enough. Report row-level correctness, missing-row rate, extra-row rate, and field-definition errors separately. For narrative answers, require citations or evidence IDs that can be resolved under the same state version.

This is where Google Cloud’s evaluation engineering lessons are especially practical. Google Cloud reports that an initial 11-prompt evaluation of its developer plugin produced false failures when a grader judged only the final summary: the agent had performed the required command, but did not mention it in the closing text. The recommendation is to grade execution steps, preserve turn structure for concurrent calls, run live and plan-only conditions side by side, and maintain durable golden sets.

The same rule applies to retrieval state. Grade the operation that makes evidence available, not only the sentence that cites it.

What Teams Should Build This Week

Do not begin by buying a larger judge model. Begin by making state inspectable.

Do not create a single “answer quality” score from a final-response grader. Do publish separate metrics for intent preservation, state validity, evidence recall, output correctness, safety, latency, and cost.

Do not flatten tool calls into a text transcript. Do preserve turn boundaries, concurrent dispatch, arguments, responses, state mutations, and evidence IDs. Google Cloud’s report specifically warns that flattening concurrent operations can obscure why an agent behaved incorrectly.

Do not let the golden set change whenever a test is inconvenient. Do version it, freeze it during tuning, and retire cases only when the underlying requirement changes. Google’s evaluation guidance makes the same point: changing the test suite moves the baseline rather than proving the agent improved.

Do not use benchmark accuracy as a deployment verdict. Do pair reliability with escalation and oversight cost. Scale’s READY framework measures reliability, human oversight, and cost together. Its reported findings show why two systems with nearly identical autonomous accuracy can require materially different review rates to meet the same reliability target.

A practical rollout can fit into three iterations:

1. Instrument: define a normalized state object, capture state-changing tool events, and attach evidence IDs to outputs.
2. Diagnose: manually inspect 20 representative traces, label state, retrieval, composition, and operational failures, then convert repeated labels into deterministic assertions.
3. Gate: run the frozen suite on every prompt, model, tool-schema, and data change; block releases on critical state regressions even when the aggregate answer score improves.

This approach also gives product teams a clearer escalation policy. If the agent cannot verify a required state field, it should ask for clarification or route to a human rather than confidently search a default view. Uncertainty at the state boundary is often cheaper than a polished answer built on the wrong population.

The New Unit of Agent Reliability Is a Verified State Transition

Agent evaluation is moving from “did it answer?” to “did it establish and preserve the conditions under which an answer could be true?” SGR-Bench quantifies the cost of ignoring that question: the strongest tested system still reached only 66.18% item-level F1, and most audited failures occurred in scope or criteria management rather than final composition. Q2D-Web demonstrates the complementary need for production-shaped retrieval data. Google’s evaluation guidance turns the principle into an engineering workflow: frozen scenarios, trace-level checks, deterministic assertions, and operational metrics. AWS’s nine-stage production evaluation framework adds lifecycle gates, canary scoring, drift detection, and multi-level evaluation from tools to sub-agents to orchestration.

The recommendation is straightforward: make state a first-class artifact, score it at every boundary, and keep final-answer quality as one layer—not the whole evaluation system. Agents will continue to become more fluent. That makes the hidden state problem more dangerous, not less. When the prose is persuasive, only a verifiable trace can tell you whether the system actually earned the user’s trust.

The teams that ship reliable agents will not be the ones with the most impressive demo transcripts. They will be the ones that can replay a failure, identify the exact state transition that went wrong, and prove that the fix survives the next release.