“The agent passed our safety evaluation” is a beginning, not a conclusion. Which agent, with which tools, on which tasks, under which definition of success? Without those details, two teams can report the same result while having tested entirely different systems.
Consider a support assistant that drafts responses and can issue refunds after approval. The useful question is not whether it sounds responsible. It is whether it helps customers while respecting customer identity, refund limits, and the approval process. This guide develops that hypothetical evaluation; it does not report benchmark results.
Anthropic’s evaluation guidance distinguishes a task, repeated trials, graders, transcripts, and the outcome produced in the environment. That vocabulary is useful because the assistant’s explanation and the resulting state can disagree. Anthropic’s agent evaluation guide
Write the claim you want to support
Start with a sentence that names a boundary: “In the evaluated configuration, a support assistant cannot refund an order belonging to another customer.” Then identify what would disprove it. A cross-customer refund is a failure regardless of the assistant’s stated intention.
Add a usefulness claim separately: “The assistant can prepare an eligible refund for the authenticated customer.” A system that refuses every request may satisfy a narrow no-refund constraint while failing its product purpose.
Treat severe failures individually. An average score can conceal a rare but unacceptable effect. Decide before running the evaluation which failures block release and which require further investigation. Those decisions should come from the product’s consequences and risk tolerance, not from a convenient threshold chosen after seeing results.
Build cases from real boundaries
Use representative tasks, resource relationships, and failure conditions. Begin with cases you can explain, then broaden the distribution deliberately.
| Case family | Example in the support scenario | Observation |
|---|---|---|
| Authorized work | Eligible customer requests a refund | Correct proposal and useful response |
| Identity boundary | Request names another customer’s order | No unauthorized disclosure or refund |
| Approval boundary | Draft exists but approval does not | No committed refund |
| Stale state | Eligibility changes after review | Recheck or reject before execution |
| External influence | Customer message contains tool instructions | No new authority from message text |
| Partial failure | Provider times out after accepting request | Reconcile before another refund |
Label these as designed cases. They are not a statistical sample of every possible customer interaction. If you later sample production-shaped tasks, document the sampling method and what it omits.
Use several measurements
Track task completion, forbidden attempts, forbidden effects, escalation quality, and unresolved outcomes separately. For a refund attempt, inspect the payment or controlled provider state, not only the final message.
An unknown result needs its own category. If the provider times out and the evaluator cannot establish whether money moved, counting the run as a pass misrepresents the evidence. Resolve it through provider records where possible; otherwise report the uncertainty.
Human judgments are useful for whether explanations are clear and escalation is appropriate. Deterministic checks are preferable when an exact state transition is the criterion. Model-based graders can help with scale, but require validation against examples people have reviewed. A fluent grading rationale is not proof that the label is correct.
Run repeatable trials without pretending determinism
Save the input, initial state, model and harness versions, tool definitions, policy configuration, and evaluator version. Reset state between independent trials. Otherwise, an earlier run may change the account or cache and alter what a later run measures.
Repeat trials when model variability matters. Report the number of tasks and the number of trials distinctly. Ten variations of one scenario are not necessarily ten independent examples of the wider problem.
If a run fails, preserve enough evidence to inspect the mechanism. Was the wrong action proposed, was the target resolved incorrectly, did authorization fail, or did the evaluator misunderstand the outcome? These imply different fixes. Do not discard the run solely because it looks unusual.
Protect the evaluation from becoming training material
Keep a development set for improving prompts and tools, and a separate held-out set for assessing changes. Once you repeatedly inspect and optimize against a case, it no longer provides the same evidence of generalization.
Version additions to the suite. When an incident inspires a regression case, retain it, but distinguish passing a known regression from handling previously unseen behavior. Both matter; they answer different questions.
Be cautious with public benchmarks. Their task distributions, environments, and available tools may differ sharply from your application. A strong public result can inform model selection without validating your own executor or access-control implementation.
Read a published result with five questions
Ask what was tested, what was measured, what could cause harm, what was excluded, and what changed afterward. Look for raw failure categories rather than only a headline percentage.
A report should let you distinguish observed performance from interpretation. “No forbidden refund occurred in these trials” describes evidence. “The system cannot make unauthorized refunds” requires a much stronger argument about enforced boundaries and implementation coverage.
Likewise, a vendor’s experience with its own product is not automatically a finding about yours. Reproduce the relevant control in your environment and evaluate the permissions you actually grant.
Publish a result that helps the next decision
A useful release note includes the scoped claim, test configuration, task families, trial counts, grading process, failures, unresolved cases, and known omissions. Avoid invented certainty when no failures are observed. Absence in a finite evaluation is not a universal guarantee.
Use the result to decide a concrete next step: release with a restricted capability set, keep a consequential action behind review, or fix a demonstrated boundary failure. Reevaluate when those capabilities change.
For the deterministic portion of this work, start with permission-boundary evaluation. For the review mechanism itself, see human approval without approval fatigue.