Evaluating AI agents: from evals to reliable production
How to evaluate autonomous agents you can trust — LLM-as-a-judge done right, the metrics that matter, and the CI/CD quality gates that keep agents from regressing.
Definition
Agent evaluation (evals) is the systematic measurement of an agent's quality against defined criteria — task success, tool use, faithfulness, cost and latency. It combines automated scoring, often with an LLM judge, and human review, run as a repeatable gate rather than a one-off test.
A demo proves an agent can work once. Evaluation proves it works reliably, and keeps proving it as the model, prompts and tools change underneath you. Without evals, every deploy is a guess.
Why are agents hard to evaluate?
A traditional test asserts a fixed output. Agents are non-deterministic and open-ended: there are many acceptable answers, the path varies, and “correct” often means faithful, useful and safe rather than string-equal. So evaluation shifts from assertions to scoring — grading outputs against criteria, at scale.
LLM-as-a-judge, done right
The practical way to score at scale is LLM-as-a-judge: a model rates or compares outputs. It is cheap and fast, and it is wrong if you skip calibration. Judges carry predictable biases:
- Position bias (order of options)30%
- Verbosity bias (longer = better)28%
- Self-enhancement (prefers own style)24%
- Formatting / authority cues18%
Calibration is not optional. Label 100–200 examples by hand, run the judge against them, and measure agreement — aim for Cohen’s kappa above roughly 0.6. Randomize option order to kill position bias, and use rubrics rather than vague “rate 1–10” prompts.
| Judge bias | What it is | How to mitigate |
|---|---|---|
| Position bias | Favours the option shown first or last | Randomize order; average over swaps |
| Verbosity bias | Rates longer answers higher | Score against a rubric, not length |
| Self-enhancement | Prefers its own style or model | Use a different judge; spot-check with humans |
| Formatting cues | Swayed by confident formatting | Judge substance; strip styling |
The metrics that matter
- Task completion rate — did the agent finish the job?
- Faithfulness / hallucination rate — is the output grounded?
- Tool-selection accuracy — did it pick and call the right tools?
- Cost and latency per task — is it affordable and fast enough?
Track these on a fixed test set so a change in score means a change in the agent, not the data.
Evals belong in CI/CD
The highest-leverage move is to treat a curated set of cases as a golden dataset and run it on every change. A prompt tweak that lifts one case can sink five others; only a regression suite catches that. Set thresholds — completion rate, faithfulness — and fail the build when they drop. This is eval-driven development: you change the agent against a measurable bar, not a vibe.
A starter loop
- Build a small golden set from real, varied cases.
- Score with a calibrated judge plus spot human review.
- Gate deploys on completion rate and faithfulness.
- Grow the set every time you find a new failure in production.
Reliable agents are evaluated agents. The reliability you prove here is also what makes human oversight and governance credible. Terms are defined in the glossary.
Frequently asked questions
Can an LLM reliably grade another LLM?
Yes, but only once it is calibrated against human judgement, and an uncalibrated judge is worse than useless because it measures its own preferences while looking authoritative. The practical procedure is to hand-label a representative set of 100 to 200 examples, run the judge over them, and measure agreement with the human labels — Cohen's kappa above roughly 0.6 is a reasonable bar before you trust the judge for anything that gates a deploy. The reason this matters is that judges carry well-documented, predictable biases: they favour the option presented first or last, rate longer answers higher regardless of quality, prefer their own style, and are swayed by confident formatting. Calibration plus a few defensive habits — randomising option order, scoring against an explicit rubric rather than a vague rate-it-out-of-ten prompt, and occasionally re-checking against fresh human labels — turn a judge from a liability into a fast, cheap scorer. Treat the judge itself as something you evaluate, not a ground truth you assume.
What is the single most useful agent metric?
Task completion rate measured against a fixed test set, because it answers the question that actually determines whether the agent is useful — does it finish the job, end to end — and most other metrics are refinements once you can answer that. The fixed-set part is what gives it teeth: scored against the same curated cases every time, a drop in completion rate means the agent changed, not that the day's traffic was harder, which is exactly what turns a silent regression into a visible one. It is also the metric that survives a model upgrade or a prompt edit as a stable point of comparison, so you can tell whether a change helped or quietly broke a class of tasks. That said, completion alone can mislead if failures are silent and damaging, so pair it with faithfulness and, where the agent acts, with whether the failures are caught. Start with completion rate as your headline, then add the others as the supporting cast.
How do evals fit into CI/CD?
Treat a curated set of representative cases as a golden dataset and run it automatically on every change to the model, prompts or tools, exactly as you would run unit tests on code. Set explicit thresholds — a minimum completion rate, a maximum hallucination rate — and fail the build when a change pushes a metric past them, so a regression is blocked before it ships rather than discovered by users. This is what people mean by eval-driven development: you change the agent against a measurable bar instead of a subjective impression that it seems better. The crucial discipline is that a prompt tweak which improves one case routinely degrades several others, and only a regression suite run as a gate catches that trade-off. Keep the golden set small enough to run on every commit but representative enough to be meaningful, and grow it every time production surfaces a new failure mode, so the suite gets stronger precisely where the agent has proven weak.
How big does a golden eval set need to be?
Smaller than most teams expect to start, and the right size is driven by coverage and stability rather than a raw count. Begin with a few dozen to a couple of hundred cases chosen to span the real distribution of work — the common happy paths, the important edge cases, and the specific failures you have already seen — because a small set that covers the ways the agent actually fails is far more useful than a large set of near-duplicates. The practical test is statistical stability: if re-running the same agent on the set produces a wildly different score each time, the set is too small or too noisy to gate on; if the score is steady and a real regression moves it, it is doing its job. Keep it small enough to run on every change without slowing the team, and grow it deliberately, adding a case each time production reveals a new failure mode, so the suite compounds in value over time. Quality and coverage of cases beat sheer quantity, and a curated few hundred well-chosen examples will outperform thousands of redundant ones.