AI-Agent Evaluations: A Successful Outcome Can Hide an Unacceptable Route

Completion is necessary but not sufficient

An agent finishes a requested change and the final screen looks correct. That is a useful outcome, but it leaves important questions unanswered. Did the agent preserve unrelated work? Did it contact an unapproved service? Did it create duplicate resources while retrying? Did it spend an unreasonable amount of time or money reaching the result? Agents act through sequences, so evaluation needs to examine the route as well as the destination. A polished final response can conceal a process the user would not have accepted.

Anthropic’s public guidance on agent evaluations discusses task outcomes, transcripts, graders, and evaluation maintenance. Supabase’s evaluation material provides another primary example of checking agent work through concrete artifacts and environment behavior. These sources establish current industry approaches, not a universal score that predicts performance for every application. This article develops an original evaluation design for a hypothetical business assistant. It does not publish a benchmark, claim a vendor ranking, or imply that an automated judge can replace every form of human review.

Define the task as a state change

A vague task such as improve this project is difficult to score. A bounded task can specify a starting state, permitted resources, required outcome, and constraints. For example, the assistant might need to add a field to a test database, update one form, and preserve existing records. The evaluation can then inspect the resulting schema, application behavior, and data. This makes success less dependent on whether the final explanation sounds persuasive. It also helps distinguish a legitimate alternative solution from a failure.

Include constraints that matter to the user. If the task permits local edits but not deployment, a deployed result should not receive an unqualified success score. If a support assistant must prepare a draft without sending it, a sent message is a material failure even when its wording is excellent. These constraints belong in the test specification and the grader. Otherwise, the evaluation rewards the agent for reaching a visible goal while ignoring the scope that made the task acceptable in the first place.

Use the environment as evidence where possible

Some outcomes can be checked directly. A database query can establish whether a record exists. A test can exercise a requested behavior. A filesystem comparison can reveal unrelated edits. These checks are often stronger than asking another language model whether the agent’s final message seems correct. They still need thoughtful design. A test that mirrors an implementation detail can reject a valid solution, while a superficial check can accept a result that only resembles the requested behavior.

Use independent evidence suited to the task. For a hypothetical scheduling assistant, inspect the actual calendar entry, time zone, participants, and duplicate count. For a code change, exercise the user-visible behavior and review the diff. For a research brief, verify the cited claims against accessible sources. The grader should know what it can establish and what remains judgment-based. A combination of deterministic checks and targeted human review can be more informative than one broad score produced by a single method.

The transcript reveals the cost of success

An action trace can show whether the agent used excessive retries, repeated the same unsuccessful search, or broadened permissions unnecessarily. These patterns may not change the final artifact, but they affect operating cost and trust. Define process measures that connect to the application’s goals: elapsed time, tool calls, rejected actions, recovery attempts, and external effects. Avoid optimizing a single count blindly. A few additional checks may be worthwhile if they prevent an expensive mistake, while repeated identical checks may add little value.

Trace review should also identify uncertain outcomes. If a remote write times out, a responsible agent may inspect the resulting state before retrying. An agent that immediately repeats the write may create duplicates. The final state can reveal the duplicate, while the trace explains how it happened. This distinction matters for remediation. The solution might be a better tool contract, a stable operation identifier, or clearer uncertainty handling rather than a general instruction to be more careful.

Keep cases that expose regressions

An evaluation suite should evolve as the product encounters real problems. Preserve a minimized version of a failure that can be rerun without exposing personal information. Label the capability or constraint it tests. When the agent or tool changes, check whether the correction survives. A suite made only of easy demonstrations can improve steadily while the product repeats the same difficult mistakes. Regression cases provide continuity between a support incident and the next release decision.

Do not let the suite become a collection nobody understands. Remove redundant cases, document ambiguous judgments, and review whether the tasks still represent the product. If an evaluation depends on an external page that changes frequently, distinguish source drift from agent behavior. If a tool becomes unavailable, the test may need a controlled fixture or a different objective. Evaluation maintenance is part of product maintenance because an unreliable test can send the team toward the wrong engineering conclusion.

Use repeated trials when behavior is variable and the decision justifies the cost. One success does not establish dependable performance, and one failure does not fully characterize a system. Report the trial conditions and the range of outcomes. For a consequential workflow, consistency may matter more than an impressive best attempt. The evaluation should make that tradeoff visible instead of selecting the most flattering run for a demonstration.

Make the release decision explicit

Before running an evaluation, decide what result would permit a rollout, require more work, or trigger a limited pilot. Otherwise, a team can reinterpret the score after seeing it. Different task classes may require different thresholds and review. A low-risk drafting feature and a tool that changes shared infrastructure should not inherit the same acceptance rule merely because both use an agent. The decision should reflect the effect of failure and the recovery options available.

Agent evaluation is ultimately a way to make delegated work inspectable. Define the requested state change, preserve the constraints, verify outcomes through the environment, and inspect the route when it matters. Keep the difficult cases alive as the product evolves. A successful agent is not simply one that can reach a goal. It reaches the intended goal through an acceptable process, with evidence strong enough for the people relying on it to understand what happened.