How Do We Know an Agent’s Action Worked?

A trace explains what an agent attempted, but only a check against the responsible system shows whether the intended result actually happened.

An incident agent calls a rollback tool and receives “accepted.” Its trace shows the call succeeded. Can it tell the incident commander that payments have recovered? That claim needs further evidence: the earlier version must be running, and payment checks must show the intended recovery.

The deployment may still be in progress. It may complete while failures continue for another reason. Even if the error rate drops, that does not automatically establish that the rollback caused the recovery. These are separate questions, and the agent’s report should distinguish them.

Logs record events, metrics summarize measurements, and traces connect operations across services. Together they help a team understand the execution path. An evidence record connects those observations to a specific claim, such as which version was deployed or whether payments passed a health check. Outcome verification checks that claim against the responsible system.

This is Post 20 in The Abstraction Shift: How Software Keeps Moving Up.

Why a final answer and trace ID leave gaps

The first production system usually has logs. Developers add messages around request handling, errors, database writes, and external calls. An operator searches those messages during an incident and reconstructs a timeline.

The next system adds metrics and traces. Metrics show rates, counts, and latency. Traces correlate a request across services. Span attributes record the operation, resource, and status. Distributed trace context links work across process and network boundaries.

These foundations remain necessary. They become insufficient when the system can make a different decision on every run.

An agent-mediated request may involve:

• the context selected for the model;
• evidence retrieved and evidence rejected;
• a model proposal and its uncertainty;
• a policy or guardrail decision;
• a capability call and its arguments;
• a retry, approval, clarification, or cancellation;
• an external state change; and
• a final response that may overstate what happened.

Storing only the final answer and a trace ID drops the most important parts of the control loop. Storing every prompt and response without structure makes the evidence expensive to search and difficult to interpret. A complete trace that never checks the source of truth can explain an incorrect result with impressive detail.

The operational question changes from “What happened in this request?” to “What claims does the system need to support, and what evidence would make each claim credible?”

What logs, metrics, and audit records each tell us

Software engineering has several mature evidence practices.

Logs, metrics, traces, audit records, tests, evaluations, verification, and validation each answer a different question. Scientific and operational workflows add provenance, so another person can inspect how a conclusion was produced. Each practice has its own grain and purpose:

Signal or practice	Best question
Log or event	What happened at this point in time?
Metric	How often, how much, or how quickly did something happen?
Trace	Which operations participated in this execution path?
Audit record	Which principal or policy decision allowed or denied an action?
Test	Did the implementation behave as expected for a known case?
Evaluation	How well did the system perform against a criterion or dataset?
Verification	Did the result satisfy a specified condition?
Validation	Does the result meet the intended use or need?
Provenance	Which source, version, transformation, or actor produced this artifact?

The old application boundary often made the relationship easier to infer. A request entered a service, passed through known code paths, wrote a record, and returned a response. The path was not always simple, but the set of possible transitions was designed in advance.

Model-mediated software adds open interpretation to that path. It needs the old signals plus explicit records for proposals, context selection, authority, uncertainty, and outcome claims.

What Is Evidence, Observability, and Outcome Verification?

Suppose the rollback trace records a successful request. To support the claim that the service recovered, the team also needs the deployment status and the later health checks. Evidence connects those records to the claim they support, while retaining any gaps or contradictions.

Observability is the ability to understand system behavior from signals such as logs, metrics, and traces. OpenTelemetry supplies shared conventions for those signals and their relationships. It helps teams connect operations; it does not decide whether a business claim is true.

Outcome verification checks a specified result against the responsible system. In this example, that means checking the deployed version and payment health after rollback. A tool acknowledgement belongs in the record, but it cannot substitute for those checks. The table shows what each kind of record contributes.

Layer	Question	Example record
Context evidence	What information did the system use?	Account snapshot, source IDs, freshness, retrieval decision
Proposal evidence	What did the model or planner propose?	Structured action, alternatives, uncertainty, model route
Authority evidence	Why was the action permitted?	Principal chain, policy version, scope, approval, credential
Execution evidence	What did the runtime attempt?	Tool name, arguments, attempt, idempotency key, response
State evidence	What changed in the environment?	Database event, provider job, deployment record, resource version
Evaluation evidence	How was behavior judged?	Test result, rubric, judge output, human review
Outcome evidence	What proves completion or failure?	Source-of-truth read, postcondition, recipient acknowledgement

The evidence model keeps these layers connected while preserving their different meanings.

Telemetry becomes evidence when signals are connected to the claims, decisions, actions, and outcomes they are meant to explain.

Connecting recorded operations to the claims they support

Distributed systems have needed correlation and instrumentation for a long time, so better logging alone is old news. What is new is that an agent’s execution path is a variable control loop, and the evidence must explain decisions as well as operations.

A conventional trace may look like:

request span -> service span -> database span -> response

An agent evidence graph may look like:

task -> context projection -> model proposal -> policy decision
  -> tool attempt -> external state change -> verification -> user claim

The graph contains operations, but it also contains relationships between claims and records. Which retrieved source supported the proposal? Which policy rule authorized the capability? Which approval covered the exact target? Which state read established completion? Which parts remain uncertain?

This makes trace semantics more important than trace volume. A hundred spans with inconsistent names may be less useful than a smaller set of correlated, semantically defined events.

Missing records matter too. If the billing check never ran, the absence of a completed-credit record cannot establish that no credit exists. Record which check was attempted and whether it returned an answer. Likewise, retain a denied action or an expired approval as a meaningful result, rather than leaving the operator to infer it from an incomplete trace.

The system should be able to say:

observed -> proposed -> authorized -> attempted -> changed -> verified

or:

observed -> proposed -> denied

or:

observed -> proposed -> attempted -> outcome unknown

Each path carries a different claim and should produce a different user-facing result.

Where the familiar comparisons stop helping

Observability and evidence sound familiar, but several shortcuts become dangerous in variable control loops.

A complete trace can still record a wrong result

A trace can faithfully record a wrong proposal, an unauthorized action, or a failed outcome. Completeness helps explain behavior. It does not establish that the behavior met the intended goal.

Verification needs an assertion and a source of truth in addition to a timeline.

A model explanation is not decision evidence

The model may provide a plausible reason for choosing a tool or answer. That explanation is an output to evaluate, not an authoritative account of what the system knew or why policy allowed the action.

Preserve the model explanation, retrieved evidence, policy result, and execution record as separate evidence types.

A metric can hide the path

An average success rate may improve while a high-risk segment degrades. Aggregate metrics need dimensions that preserve task type, resource, route, policy decision, user impact, and outcome state where privacy and cardinality allow.

A tool acknowledgement is not a business outcome

The provider may acknowledge a request while processing it asynchronously, rejecting it later, or applying only part of the requested change. Read the authoritative state or obtain a domain-specific postcondition.

Sampling can remove the decisive event

Tail sampling, redaction, retention limits, and cost controls are necessary. They can also remove the proposal, denial, approval, or outcome evidence needed to investigate a high-consequence action.

Sampling policy should be risk-aware. A low-latency read and a production deployment do not need identical evidence treatment.

Offline evaluation cannot speak for a production task

An offline benchmark can show that a model usually selects the right tool. It cannot prove that this production task used current evidence, had authority, or achieved the requested result. Evaluation evidence and runtime evidence should connect without being confused.

Evidence can become an attack surface

Logs, traces, retrieved documents, tool results, and screenshots may contain untrusted instructions or sensitive data. The system must preserve provenance and trust level instead of treating every recorded string as a trusted explanation.

Collecting evidence where decisions and state change

A practical evidence path makes each control-loop boundary observable and reviewable:

1. Define the claims. Start with what the system must be able to establish: context freshness, authorization, tool execution, state change, user notification, and verified outcome.
2. Name the entities. Give tasks, principals, resources, proposals, capabilities, external jobs, and artifacts stable identifiers.
3. Propagate correlation. Carry trace and task context across requests, queues, workers, model calls, tool calls, and downstream services. Use semantic conventions for shared fields.
4. Record context selection. Preserve source identifiers, versions, timestamps, freshness, filters, and inclusion or exclusion decisions without storing unnecessary sensitive content.
5. Record proposals separately. Store the structured output, model and route, schema version, uncertainty, refusal, and alternatives as a proposal record.
6. Record policy and authority. Link the principal chain, policy version, decision, scope, approval, credential, and denial reason to the proposed operation.
7. Record execution. Capture the capability, sanitized arguments, attempt number, idempotency key, start and end, response class, and external reference.
8. Record state transitions. Capture the source-of-truth event or resource version that shows what changed, including partial or ambiguous states.
9. Verify the outcome. Define a postcondition and read it from the authoritative system. Record the check, timestamp, evaluator, and result.
10. Assemble the evidence record. Make the claim, supporting signals, gaps, confidence, access policy, retention, and reviewer or automation explicit.

The data model should support both human investigation and automated checks. A verifier may reject a task whose tool result lacks an external reference. An operator may inspect all production actions where approval preceded a resource change by more than an hour. An evaluator may compare the model proposal with the verified outcome without treating the proposal as ground truth.

An evidence path follows a control loop from what it saw and proposed to what it changed and what proves the result.

Evidence should be queryable by claim, not only by request ID. “Show me everything about trace 123” is useful. “Show me actions that were authorized but never verified” is more useful for governance.

Following the example through the system

Consider an incident-response agent with this request:

Investigate the increase in payment failures. If the new release is responsible, roll back the payments service and tell me what changed.

The agent may inspect dashboards, traces, recent deployments, error samples, feature flags, and service health. It may ask for approval before rollback, call a deployment capability, and then wait for recovery signals.

The response needs more than a final paragraph. The system should preserve an evidence record like this:

Evidence	Example
Task and trace	Task T-2048, trace context propagated across agent, observability, deployment, and health services
Context	Error-rate window, selected dashboards, deployment commit, feature-flag state, source freshness
Proposal	Model proposed rollback because failures clustered after commit 91c2; confidence and alternatives recorded
Authority	Incident commander principal, rollback capability, approval or emergency policy, scope and expiry
Execution	Deployment tool call, sanitized arguments, idempotency key, provider response, external operation ID
State change	Payments service version changed from 91c2 to 90af; rollout status recorded by deployment system
Verification	Error rate, latency, and synthetic payment checks after rollback; time window and thresholds recorded
User claim	”Rollback completed and payment failures returned below threshold,” with links to supporting evidence

The runtime path is:

1. Create the task and trace context before gathering evidence.
2. Record which dashboards, traces, deployment records, and feature flags were selected and their freshness.
3. Ask the model for a structured incident hypothesis and candidate actions.
4. Preserve the hypothesis as a proposal, not as an incident fact.
5. Apply identity, authority, policy, and approval checks before exposing rollback capability.
6. Execute the rollback with an idempotency key and capture the external operation reference.
7. Observe deployment state and health signals while the task remains open.
8. Verify the postcondition using independent source-of-truth checks rather than the tool acknowledgement alone.
9. Assemble a result that distinguishes observed facts, model interpretation, authorized action, state change, and verified outcome.

Three variations show where the record earns its keep. First, the rollback tool returns “accepted” and the agent immediately tells the incident commander that the service recovered. The evidence is incomplete. The task should remain in a reconciling state until the deployment system and payment checks establish the postcondition.

In a second case, the error rate drops for unrelated reasons. The outcome may be good while the causal explanation remains uncertain. The evidence record should separate “payments recovered” from “rollback caused recovery.” Verification of the requested state does not automatically verify the model’s explanation.

In a third, the agent’s context includes a malicious dashboard annotation instructing it to disable audit logging. The annotation is evidence about an untrusted source, not a policy instruction. Provenance and trust labels keep it from silently changing the control loop.

The value of the evidence model is that it lets the system be useful without making a stronger claim than the records support.

What the team needs to design and operate

Architecture

Evidence has to connect the records kept by several parts of the application. In the rollback example, the task identifies the request, the policy record establishes permission, and the deployment and health systems establish the result. A tracing service helps join operations, but the team must decide which records support each claim and which gaps prevent it from reporting success.

Define an evidence registry or schema for claims, signals, relationships, source-of-truth records, and retention. Keep domain evidence close to the systems that can verify it, then link it into the task and trace graph.

Engineering

Instrument the boundaries where meaning changes: context selection, proposal creation, policy decision, capability invocation, state transition, and outcome verification. Use semantic conventions for common fields, but define application-specific claim and postcondition schemas.

Test missing and contradictory evidence. Test a tool that returns success without changing state, a state change without a trace, a model proposal with no provenance, an approval for a stale target, and an evaluator that disagrees with the source of truth.

Product and UX

Users should see the result at the right certainty level. “I found evidence of,” “I proposed,” “I requested,” “the provider accepted,” “the state changed,” and “the outcome is verified” are different product states.

For high-consequence actions, show the target, decision, evidence summary, approval, current status, and unresolved uncertainty. For low-risk work, keep the detail available without forcing every user to read an execution trace.

Operations and governance

Operate on claims and gaps as well as error counts. Useful queries include unverified completions, actions without current approval, tasks with stale context, repeated ambiguous outcomes, and high-risk operations missing external references.

Retention, privacy, redaction, and access control are part of evidence design. The most useful trace may contain sensitive prompts, customer data, credentials, or retrieved documents. Preserve enough to support the claim while limiting who can inspect it and how long it remains.

Failure Modes

The final answer becomes the only record

The system stores the assistant’s summary and discards the context, proposal, policy result, and outcome check. Keep the answer as one derived artifact linked to the evidence graph.

Trace ID without semantic relationships

Every service emits spans, but no record identifies which span represents the proposal, approval, side effect, or verification. Define names, attributes, events, and links for the control-loop boundaries.

Tool success treated as outcome

An API returns an acknowledgement and the system reports completion. Add a domain-specific postcondition and reconciliation read.

A plausible rationale passes for provenance

The explanation sounds credible, so the team treats it as evidence. Preserve source IDs, retrieved content metadata, policy decisions, and runtime observations separately.

Sampling removes high-risk evidence

Cost controls discard the exact event needed to investigate a consequential action. Use risk-aware retention and tail-sampling rules, with explicit redaction for sensitive content.

Evaluation disconnected from production

Offline evals measure tool selection, while production telemetry cannot show whether the task had current context or achieved its outcome. Link evaluation cases and runtime evidence without treating either as a replacement for the other.

Evidence store becomes untrusted data

An operator or model reads a log entry that contains injected instructions and treats it as policy. Preserve trust level, provenance, and data-versus-instruction classification.

Retention destroys the chain

The system keeps a final audit record after deleting the task, credential, source snapshot, or outcome reference needed to interpret it. Define retention by claim and reconstructability, not only by storage cost.

An evidence record should state what claim it supports, which signals support it, and what remains uncertain.

Reviewing what each success claim needs

Before a model-mediated control loop reaches production, complete this evidence card:

Question	Record
What claim must the system support?	User, operator, policy, or business outcome claim
What context was used?	Source IDs, versions, freshness, filters, and trust labels
What did the model or planner propose?	Structured proposal, alternatives, uncertainty, model route, and schema version
Who authorized the action?	Principal chain, policy version, scope, approval, and credential reference
What did the runtime attempt?	Capability, sanitized arguments, attempt, idempotency key, and response
What changed?	Source-of-truth event, resource version, external operation, or partial state
How is completion verified?	Assertion, authoritative read, evaluator, timestamp, and result
What remains uncertain?	Missing evidence, conflicting signals, causal ambiguity, or stale context
Who may inspect the evidence?	Access policy, redaction, retention, and deletion behavior
How is the evidence queried?	Claim, task, trace, principal, resource, risk, and outcome dimensions

This card turns observability into an evidence practice: it tells the team what to collect before the incident instead of leaving them to search for it afterward. The vocabulary will keep moving between observability, auditability, provenance, and assurance cases, and the telemetry vendor may change. The application will still need to distinguish a rollback request from a deployed version, a healthy payment check, and a causal explanation of recovery, because those distinctions keep the report proportionate to the evidence.

Sources

• OpenTelemetry Concepts: introduces traces, metrics, logs, baggage, context propagation, instrumentation, and semantic conventions as distinct observability concepts.
• OpenTelemetry Tracing API: defines spans, span context, attributes, links, status, and timestamped events within traces.
• OpenTelemetry GenAI semantic conventions: provides current common attributes for generative AI operations, tools, models, usage, and results.
• W3C Trace Context: standardizes HTTP trace context propagation for distributed tracing across services and vendors.
• NIST TEVV-Athlon Framework: distinguishes testing, evaluation, verification, and validation as evidence-producing activities for AI systems.