How to Evaluate AI Agents Beyond Task Completion

Evaluating an AI agent cannot stop at "did it finish?" A sound evaluation asks eight questions: was the outcome correct, was the route sensible, were tools called properly, was the cost acceptable, was it timely, could it recover from failure, did it stay inside safety boundaries, and would a person sign off on the result? Miss any one and a lucky success can be mistaken for production capability.

An agent differs from an ordinary question-answering system because it plans and acts repeatedly; What Is an AI Agent? explains that loop. The evaluation target is therefore not the final paragraph alone. It is the full trajectory from receiving a goal and choosing tools to changing external state and delivering evidence. A business is testing a repeatable work system, not a demo.

Start by turning "done" into a task contract

Before writing tests, write the task contract. State the input, the deliverable, which facts must be right, which data and tools may be used, which actions may never be automatic, when the agent must stop and ask a person, and what evidence proves completion. "Organise customer inquiries" is too loose. A testable version says: read tickets from a specified period, classify them using the approved labels, preserve ticket IDs as provenance, never alter the source records, and queue cases with missing product information for confirmation.

The contract separates three outcomes that teams often blur. Full pass means every mandatory condition holds. Partial pass means the main deliverable is usable but has an allowed, repairable omission. Failure includes factual error, a missing critical step, an unauthorised action, or a claim of completion without evidence. Count only full passes in the task success rate and report partial passes separately. Otherwise, producing something quietly becomes the definition of success.

A complete evaluation examines eight separate dimensions

The eight dimensions are not ingredients for a decorative radar chart. They isolate different failure types. Keep them as separate measures and give high-risk dimensions hard vetoes:

• Task outcome: whether facts, calculations, format, completeness and provenance satisfy the contract.
• Execution process: whether planning missed required steps, made unsupported assumptions, looped uselessly or stopped too early.
• Tool calls: whether tool choice, parameters, order, response checking and permission scope were correct.
• Cost: what each accepted task actually costs across model calls, external tools, retries and human review.
• Latency: how long ordinary work takes and whether the slowest group still fits the business deadline.
• Recoverability: whether the agent can resume safely after timeouts, malformed responses, disconnections or human takeover without repeating side effects.
• Safety: whether it resists hostile input, refuses excess authority, protects sensitive data and waits for approval before consequential actions.
• Human evaluation: whether a business reviewer finds the result usable, the evidence clear and the risk acceptable enough to sign off.

Do not average the eight into one comforting score. If an agent sends without authorisation or reads outside its scope, strong output, low cost and fast execution cannot compensate. Set hard gates for security, permissions and critical facts first. Compare success, cost and latency only among versions that clear every gate.

Grow the test set from real work, not ideal prompts

An actionable test set begins with a task inventory. Choose a bounded workflow and collect sanitised historical inputs, human deliverables, common rejection reasons and operational failures. Do not treat the human wording as the only answer: many tasks allow several good expressions. What should stay fixed are the facts, invariants and forbidden actions. Each case should record an ID, input snapshot, mandatory outcome, acceptable variation, available tools, permissions, time and cost limits, injected failure and human rubric.

Cover at least four families of cases. Normal cases exercise the happy path. Edge cases include missing fields, ambiguous requests, duplicate records and unusual formats. Adversarial cases place instructions in a webpage, document or user input that try to expand authority, extract secrets or replace the goal. Failure cases deliberately trigger tool timeouts, expired sessions, empty responses and interrupted execution. Every frequent task needs representatives from these families if evaluation is to see beyond the demo path.

Separate development cases from a held-out release set. The team may use the development set to improve prompts, tools and orchestration. The held-out set stays out of routine tuning and runs only at release, reducing the chance that the system merely memorises familiar tests. When production exposes a new failure, add a sanitised, reproducible version to regression and tag the incident and regression it is meant to prevent. The test set then matures with the operation instead of being archived after launch.

Score outcomes, trajectories and tool calls separately

At the outcome layer, check hard conditions before softer quality. Structured work can be compared field by field. Research and summarisation should be checked for coverage of required facts, whether citations support the claims, and whether the agent introduced assertions absent from the input. Where several answers can be correct, combine required facts, forbidden content and a business rubric instead of demanding exact wording. Every pass should remain explainable rather than ending as an opaque model-generated score.

Trajectory review asks whether the agent confirmed critical assumptions, guessed when evidence was missing, repeated the same failed route, or kept acting after the task was already complete. There need not be one canonical path, but every acceptable path must respect the same invariants. Evaluators can allow different plans as long as none skips approval, mutates source data or loses the connection between the deliverable and its inputs.

At the tool layer, inspect the selected tool, parameters, identity, returned status and what happened next for every call. The dangerous case is not a visible tool error; it is an error the agent fails to check before reporting success. Test idempotency too: if a retry executes the same step twice, does it send two emails, create two tickets or decrement stock twice? If the process never needed dynamic tool choice, revisit AI Workflow or AI Agent: How to Choose rather than paying extra risk for autonomy.

Cost, latency and recovery decide whether it can keep running

Cost is larger than the model bill. Total cost per task includes search, browser or third-party services, failed retries, and the time people spend reviewing and repairing the work. Report both cost per attempt and cost per accepted result; the second brings failure and rework into the ledger and better reflects business spending. An agent that plans dozens of steps for a simple task may have no production value even when it eventually succeeds.

Latency should not be reduced to an average either. Look separately at typical cases and the slowest group, and split queueing, tool waits and human approval. Customer-service assistance may need a prompt response, while an overnight batch mainly needs to finish before its deadline. The measure should come from the operating window, not from one universal number imposed on every scenario.

Recovery tests deliberately push the system off the happy path: disconnect after a tool call, expire the session, return a malformed payload or make a dependency briefly unavailable. Observe whether the agent recognises state, retries safely or resumes from a checkpoint. If it cannot recover, it should hand a person the completed actions, failure point and remaining work without ambiguity. For the wider production transition, see From AI Pilot to Production.

Treat untrusted content as part of the safety environment

Agents read webpages, emails and documents. Content does not become an instruction merely because it looks like reference material. Add prompt injection, forged administrator requests, attempts to obtain secrets, requests for unrelated customer records and demands to bypass approval. The evaluation should show that the system distinguishes the task goal and trusted system policy from untrusted external content. Record refusals as well, so reviewers can tell a working safeguard from a tool that happened to fail.

Permission testing asks not only whether the agent can act, but whether it truly cannot act when it should not. Repeat a case across roles, data scopes and action levels, checking that reading, generating, modifying, submitting and external sending are controlled in stages. Approval for an irreversible or high-impact act must happen before execution and bind to the exact object and parameters; one broad consent must not authorise every later action. Who Owns AI Agent Errors? continues with the responsibility and evidence chain.

Give human review a rubric, then make every result regressible

Human evaluation is not asking a colleague whether the output feels good. The rubric should separate correctness, completeness, readability, evidence clarity, business usability and risk, with observable descriptions for each score. Hide model and version names during review to reduce brand and novelty bias. When reviewers disagree, record the point of contention and let the business owner decide whether the task contract needs clarification instead of merely averaging the scores. High-risk cases require sign-off from someone with the corresponding business authority.

A reliable release sequence starts with the fixed offline set, moves into shadow mode without real write-back, opens next to controlled users and low-risk tasks, and continues with production sampling. Rerun the held-out set whenever the model, prompt, tool, permission or knowledge source changes; any regression on a hard gate blocks release. The final product of evaluation is not a scorecard. It is evidence showing which version may run, inside which boundaries, and why.