Agent observability: seeing what your AI actually does

Traditional application performance monitoring assumes deterministic services: the same request follows the same path, so you watch latency, throughput and error rates and that largely tells the story. Agents break that assumption. The same prompt can branch differently on each run, call a different set of tools, loop, retry, and cost ten times as much on a bad day as on a good one, so averages hide exactly the failures that matter. Agent observability therefore has to capture the reasoning trace — the full tree of prompts, model calls, tool invocations and sub-agent steps, each with its tokens, timing and outcome — not just request-level metrics. It also has to treat cost and correctness as first-class signals, because a run can succeed technically while being wrong or absurdly expensive. The mental shift is from monitoring a service that behaves predictably to reconstructing the behaviour of a system that does not, so you can debug, cost and audit any individual run after the fact.

Start with three signals that together tell you whether the agent works, what it costs and whether it is fast enough: task completion rate, cost per task and latency. Completion rate is your headline quality measure — what fraction of runs actually achieve the goal — and it is what surfaces a regression before users complain. Cost per task, tracked as tokens and money, catches the failure mode unique to agents, where a looping or runaway run does not error but quietly bills, so it pays for the instrumentation by itself. Latency keeps the experience usable and flags timeouts. These three are cheap to capture and high in signal, which is why they belong from day one. As the system matures, add the harder, more valuable metrics — tool-selection accuracy and hallucination rate — which most teams instrument late precisely because they are harder, even though they catch the failures that damage trust. Measure the easy three now, then close the gap to the hard two deliberately.

No, you can start with OpenTelemetry and its GenAI semantic conventions, which most platforms now support, and that is the right first move because it keeps your agent traces in the same backend as the rest of your stack and avoids vendor lock-in. The 2025 convergence on a shared span schema means you no longer have to adopt a proprietary format to get structured agent traces. Dedicated observability platforms do add real value on top — session replay, time-travel debugging through a run, and dashboards built around agent concepts like tool calls and sub-agents — and many teams adopt one as they scale. The key is to build on the open standard rather than around it, so that switching or adding a tool later does not mean re-instrumenting everything. In short: standard first for portability, dedicated tooling second for depth, and never a proprietary format that traps your traces where only one vendor can read them.

They are closely related but not identical, and the practical insight is that your observability traces are the raw material your compliance audit logs are made from. Observability exists to help you operate the agent — debug failures, control cost, spot regressions — and is typically optimised for engineers, with sampling, retention limits and mutable storage. The audit logs governance regimes expect, including under the EU AI Act, have stricter requirements: they must attribute each consequential action to an identity, be retained for defined periods, and be tamper-evident, because a log a defendant could quietly alter is worthless as evidence. So the two overlap in content — both record what the agent did — but differ in guarantees. The efficient path is to design tracing once and derive both from it: keep rich traces for operations, and promote the decision-and-action records you need for accountability into signed, append-only, retained logs. Treat governance logging as a hardened projection of your observability data, not a separate system you bolt on later.