The production metrics that tell you an agent is working

Task completion rate measured against a fixed set, because it answers the single question that matters most — does the agent actually finish the job, end to end — and everything else is secondary until you can answer that. Measured against a stable test set rather than whatever traffic happened to arrive, it also turns silent regressions into visible ones: when a model upgrade or prompt change quietly breaks a class of tasks, the completion number drops and you see it before users do. Start there, then add cost per task and latency, which together tell you what that completion costs in money and time, and only then layer in the harder quality metrics like accuracy and hallucination rate. The sequencing reflects effort versus signal: completion, cost and latency are cheap to capture and immediately actionable, while the quality metrics need sampling and judging and are worth the investment once the basics are in place. The mistake is to start with an elaborate quality dashboard before you can even say whether the agent finishes its work.

You sample, because grading every live response is neither feasible nor affordable at production volume. Take a representative sample of responses and score them with a calibrated judge — often an LLM-as-a-judge tuned against human labels — backed by periodic spot human review to keep the judge honest, then track the resulting rate over time. The crucial point is that the trend matters more than any single number: an absolute hallucination percentage is hard to interpret in isolation, but a rate that is climbing week over week is an early warning that something — a model change, a data shift, a prompt regression — has degraded grounding. Sample on a stable basis so the movement reflects the agent, not the sampling. And weight your attention by stakes: a hallucination an agent merely says is bad, but one it acts on is an incident, so sample acted-upon outputs more heavily. Treat hallucination as a monitored trend with human calibration, not a one-off measurement.

Yes, and they are seductive precisely because they are easy to produce and always go up. Counting runs, tokens consumed or tool calls made feels like measurement and fills a dashboard, but it tracks activity rather than value — an agent can run more, spend more and call more tools while delivering less. The discipline is to anchor every metric to a task outcome: completed, correct, affordable, fast, satisfying. If a number does not connect to whether the agent actually helped someone at acceptable cost, it does not belong on the dashboard you use to make decisions. This matters more for agents than for traditional software because their activity is so visible and so easy to quantify that it invites measuring motion instead of progress. A useful test for any proposed metric is to ask what decision it would change; if the honest answer is none, it is probably vanity. Measure outcomes, and let the activity numbers stay in the traces where they belong as debugging detail.

There is no universal target, and anyone quoting one without context is selling a number rather than an answer. A good completion rate depends entirely on the task's difficulty, its tolerance for failure, and what the fallback is when the agent does not finish: a low-stakes drafting assistant might be perfectly useful at seventy percent because a human easily catches the rest, while an agent taking irreversible actions might be unacceptable below the high nineties. So the right way to use the metric is relative, not absolute. Establish a baseline for your specific tasks, then watch for movement — a drop signals a regression, a rise signals an improvement — and set a threshold tied to the cost of a failure in your context rather than an industry figure. Pair the rate with what happens on the misses: a high completion rate hides risk if the failures are silent and damaging, while a lower rate can be fine if failures are caught and handled. Judge the number against your own task and your own safety net, not a benchmark.