Agent Traces Are the Missing Control Plane for Production AI

A Successful HTTP Response Can Hide a Failed Agent

Production agents need a control plane for behavior, not another log viewer. An agent can return HTTP 200 while choosing the wrong tool, retrying a slow call, passing stale context to a subagent, or spending an unacceptable number of tokens.

That distinction is becoming operationally important. GitHub’s 2026 Octoverse report says more than 40% of pull requests began with an agent in September, up from 17% in March. More agent-generated work means more decisions to validate after generation. The bottleneck is no longer merely whether a model can produce an answer or a patch; it is whether the surrounding system can explain, evaluate, and safely improve the path that produced it.

My recommendation is simple: treat an agent trace as a product artifact. It should connect intent, model calls, tool calls, approvals, retries, subagents, infrastructure, cost, and outcome in one trace context. OpenTelemetry gives teams a portable vocabulary for the first layer; production platforms such as Cloudflare show how that vocabulary can become an operational feedback loop.

Why Traditional Application Telemetry Stops Too Early

Traditional telemetry answers whether a request reached a service and how long that service spent in a database or downstream API. An agent needs additional boundaries because its most consequential decisions happen between those infrastructure spans.

OpenTelemetry’s GenAI observability guidance describes a useful span tree: an invoke_agent operation containing model chat spans and execute_tool spans. The associated attributes include the requested model, input and output token usage, and finish reasons. Those details turn “the agent was slow” into testable alternatives: the model was slow, a tool timed out, the prompt grew, or the agent entered a retry loop.

Cloudflare makes the same argument from a hosted-agent perspective. Its agent tracing announcement says a normal application trace can show fetches, KV reads, or database queries while omitting the agent operation that caused them. Its agent-aware traces add agent invocations, model calls, tool execution, approval events, and supported subagent calls alongside Workers infrastructure spans.

The practical difference looks like this:

Question	Request telemetry	Agent telemetry
Where did time go?	Service, network, or database span	Model, tool, approval wait, subagent, or infrastructure span
Why did the result change?	Request and response metadata	Model version, prompt state, tool choice, arguments, and returned context
What did the run cost?	Compute or platform usage	Input/output tokens, model mix, retries, and tool-side usage
Can we reproduce it?	Request ID and logs	Trace-linked state, tool results, approvals, and deployment version
Can the system improve?	Alert and manual investigation	Structured evidence for evaluation, routing, and workflow changes

This does not mean every prompt should be stored forever. It means the system must distinguish between metadata that is safe to retain, content that requires controls, and fields that are necessary to evaluate behavior.

The Trace Must Represent Decisions, Not Just Events

A timestamped sequence of spans is useful, but chronology alone is not enough. The trace must expose the decisions that change the agent’s available actions and the evidence it sees.

At minimum, record these boundaries:

1. Intent boundary: the request, tenant or session context, and the constraints the agent must preserve.
2. Planning boundary: the selected model, route, policy, and planned tool or subagent operation.
3. Tool boundary: tool name, version, arguments, authorization result, duration, status, and returned context metadata.
4. Approval boundary: whether a human or policy gate paused the run, what action required approval, and what was approved.
5. Outcome boundary: the final response or side effect, evaluator result, cost, and whether the run can be replayed.

A compact event model can stay framework-neutral:

{
  "trace_id": "run-1842",
  "agent": {"name": "support-router", "version": "2026.10.3"},
  "turn": 4,
  "operation": "execute_tool",
  "tool": {
    "name": "refund_order",
    "version": "2.1.0",
    "read_only": false,
    "approval": "required"
  },
  "input": {"order_id_hash": "…", "amount_minor": 4200},
  "result": {"status": "approved", "external_request_id": "…"},
  "latency_ms": 382,
  "token_usage": {"input": 1840, "output": 96}
}

The example intentionally hashes the order identifier and omits raw customer content. Observability is not a license to copy sensitive prompts into every vendor dashboard. OpenTelemetry’s guidance notes that prompt content and tool arguments are not captured by default because they may contain sensitive data; content capture is an explicit configuration choice. Cloudflare similarly documents controls for turning message and tool-payload recording off when data may contain personal information or secrets.

The design rule is minimum sufficient evidence: retain the identifiers and structured metadata needed to explain and evaluate a run, then make content capture scoped, redacted, time-limited, and access-controlled.

The Dashboard Is Not the Control Loop

A dashboard helps a person inspect a failure. A control plane turns the inspection into a bounded action.

A related distinction is made in the state-first evaluation pattern for agents: a correct final answer can still conceal a wrong retrieval state. Agent traces make that state inspectable alongside the operation that produced it.

Cloudflare’s real-time issue detection workflow demonstrates the missing handoff. Its Issues feature groups repeated exceptions, 5xx responses, and error logs; attaches stack traces, leading and trailing telemetry, the Worker version, and application context; and can send that package to an agent after an occurrence threshold or quiet period. The agent can query related logs and traces, propose a fix, and open a pull request. A human still reviews and deploys the change.

That architecture is better than asking an agent to search an unstructured dashboard because it defines an event contract:

production signal
  -> grouped issue with scope and trend
  -> trace-linked diagnostic context
  -> bounded agent investigation
  -> tests and pull request
  -> gradual release
  -> new telemetry

The important phrase is bounded investigation. The trigger should include scope, threshold, deployment version, relevant trace IDs, and permissions. The agent should be able to inspect and test, but not silently turn an error alert into an unrestricted production mutation.

This is also why push-based workflows matter. Cloudflare’s Agent Development Lifecycle argues that systems built for agents must be programmatic, horizontally scalable, reproducible, real-time, atomic, permissioned, and self-improving. Those are not just deployment properties. They describe what a trace-backed control plane needs in order to move from observation to action without relying on a human to manually copy context between tools.

Measure the Agent at Four Separate Layers

Do not collapse agent quality into one “success rate.” A single number cannot tell you whether a failure came from intent interpretation, retrieval, tool execution, or answer composition.

Use four score families and keep them linked to the same trace ID:

Layer	Example question	Deterministic signals
Intent	Did the run preserve the user’s constraints?	Required fields, exclusions, tenant and scope checks
Decision	Did it choose the permitted model and tool?	Route policy, tool allowlist, approval requirement
Execution	Did each operation complete under its contract?	Tool status, schema validation, retries, timeout, side effects
Outcome	Did the result solve the task safely?	Golden assertions, evaluator rubric, user correction, rollback

Add operating measures beside quality: p50 and p95 latency, model and tool duration, input/output tokens, retry count, approval wait time, tool-call count, and estimated cost. OpenTelemetry specifically identifies model duration and token-usage metrics as useful for comparing models, detecting token-heavy prompts, and monitoring cost and latency regressions.

The split matters operationally. If outcome quality falls while tool execution remains stable, investigate routing or model behavior. If quality is unchanged but cost doubles, inspect context growth and retry policy. If successful answers increase while approval waits dominate, the system may need a better risk-based policy rather than a larger model.

Build a Small Tracing Contract Before Buying a Bigger Platform

Start with a contract that every agent framework can emit. A vendor dashboard can be replaced; an undocumented event shape cannot be migrated cheaply.

Define these fields first:

• trace_id, session_id, agent.name, and agent.version.
• operation.name: invoke, chat, execute tool, approval, handoff, or evaluation.
• Model provider and model identifier, with input and output token counts.
• Tool name, tool version, read-only or mutating classification, arguments hash, status, and external request ID.
• Parent-child relationships for subagents and downstream infrastructure.
• Deployment version, tenant or account identifier, and redaction classification.
• Outcome status, evaluator result, estimated cost, and retry count.

Then enforce three gates:

1. Instrumentation gate: no production agent version ships without a trace that covers model and tool operations.
2. Policy gate: mutating tools, sensitive data, and cross-tenant access require explicit policy attributes and approval behavior.
3. Evaluation gate: a release needs regression results for outcome quality, latency, cost, and tool correctness—not only a passing smoke test.

A local collector is enough to begin. The OpenTelemetry article demonstrates exporting to an OTLP endpoint and inspecting traces in a local dashboard, while Cloudflare describes exporting agent traces to any OTLP-compatible destination. That portability lets a team validate its event model before committing to a long-term backend.

The Counterargument: Tracing Can Become Expensive and Dangerous

The case for traces is not a case for recording everything. Full prompts, tool arguments, and results can contain credentials, personal data, customer secrets, or proprietary source code. They also increase storage and processing costs.

Three controls keep the system useful:

• Separate metadata from content. Keep model, duration, tokens, tool names, hashes, statuses, and identifiers in the default trace. Capture content only for selected environments, tenants, or sampled failures.
• Redact at the producer. Do not rely on a dashboard to hide secrets after they have already crossed the network. Apply field-level filtering before export.
• Sample by risk and value. Keep all mutating operations, policy denials, retries, and failed evaluations; sample routine read-only successes. Retain a complete trace when an issue crosses a defined threshold.

There is also a maturity limit. A trace cannot tell you whether a product decision was wise when the system has no explicit success criteria. Instrumentation exposes behavior; it does not create judgment. Teams still need golden tasks, human review for high-impact actions, and a rollback mechanism.

Make Observability the Agent’s Feedback Surface

The next step after tracing is not a prettier waterfall. It is a feedback surface that an agent can query safely and that engineers can use to improve the system.

Cloudflare’s agent development model points toward this loop: traces power evaluations, analytics, and token reporting; production issues can carry diagnostic context into a bounded coding workflow; and the resulting change can be tested and rolled out gradually. GitHub’s Octoverse data supplies the pressure behind the shift: when agent-started pull requests rise from 17% to more than 40%, review and operational feedback must scale with generation.

For a new production agent, do this—not that:

• Do define an OpenTelemetry-compatible trace contract before choosing a vendor. Do not start with a dashboard whose event schema is impossible to export.
• Do record tool versions, approval decisions, retries, tokens, and deployment versions. Do not treat the final answer as the entire execution record.
• Do route grouped, trace-linked issues into bounded investigations. Do not give an alert-triggered agent unrestricted production credentials.
• Do sample content while retaining high-value metadata. Do not export customer prompts and tool arguments by default.

Agent systems will become more autonomous by first becoming more inspectable. The production advantage will not go to the team with the most elaborate chat transcript. It will go to the team that can answer, quickly and reproducibly: what did the agent decide, what did it touch, what did it cost, what failed, and what changed after the failure?