How is agent observability different from classic APM?

APM assumes a deterministic service: same request, same path, so latency and error rate tell the story. An agent can branch differently on every run, loop, retry, and cost ten times more on a bad day, so averages hide the failures that matter. You need the full trace of model calls and tool invocations per run, with cost and outcome as first-class signals.

What should I measure first if I have nothing?

Outcome rate and cost per run, from day one. They need only a success flag and token counts on each run, no tracing infrastructure, and they catch the two failure modes that hurt most: the agent quietly getting worse and the agent quietly getting expensive. Add loop depth, tool error rate and a liveness signal next; full traces complete the picture.

Do I need an LLM-specific tool, or will my existing stack do?

Start with what you have. The OpenTelemetry GenAI conventions mean an agent emits ordinary OTLP data your existing backend can store. LLM-specific platforms such as Langfuse or Arize Phoenix add prompt-aware views — token costs per step, input and output rendering, eval scores on traces — which earn their place once humans start debugging runs daily.

Is observability enough for AI Act compliance?

It is the raw material, not the compliance. High-risk systems under the EU AI Act need record-keeping and effective human oversight, and both assume you can reconstruct what the system did. Traces and metrics make that possible; the obligations themselves — retention, oversight procedures, documentation — are governance work on top of the telemetry.

Agent observability: the five signals and the alert rules that watch them

You cannot debug, cost or audit what you cannot see. An agent gives you more not to see than any service you have run before: it plans, branches, retries, and calls tools in an order nobody wrote down. Agent observability is the discipline of capturing enough telemetry — traces, metrics, events — to reconstruct any single run after the fact. The good news is that the data model has standardised. The OpenTelemetry GenAI conventions define spans for agent and tool operations, with attributes like gen_ai.usage.input_tokens — so the telemetry is portable across backends instead of locked to one vendor.

This article gives you the five signals worth alerting on, rules you can copy, and the tooling layers in the order you should add them. The deep dive into trace structure lives in agent tracing with OpenTelemetry; this is the operating view.

Five signals, and the questions they answer

Everything an agent does eventually shows up in one of five numbers. Each answers one operational question:

Signal	Question it answers	Source
Outcome rate	Is the agent still doing its job?	Success flag per run — human label, eval, or downstream acceptance
Cost per run	What does one unit of work cost, and is that drifting?	`gen_ai.usage.*` token counts × price, summed over the trace
Loop depth	Is the agent iterating or thrashing?	Steps per run; retries per tool
Tool error rate	Is the world the agent acts on healthy?	Failed tool calls / total, per tool
Liveness	Is a “running” agent actually alive?	Time since last span or heartbeat

The fifth is the one most teams skip, and it bit me this week. Two API calls from an automation in this portfolio hung for 36 minutes with no response and no error. The watchdog’s abort was the only signal anything was wrong; a third call to the same provider returned instantly. A dashboard of averages showed nothing, because a hung call produces no data points at all. Silence looks identical to “still running” unless you alert on the absence of telemetry, not just on bad values in it.

Alert rules you can copy

Thresholds belong to your workload, but the shape of the rules does not. This is the starting set I use, in plain YAML you can translate to any alerting stack:

# Agent alert rules — starting points, tune to your baseline.
- alert: outcome_rate_drop
  expr: success_runs / total_runs < 0.90        # vs your calibrated baseline
  for: 1h                                        # trends, not single runs
- alert: cost_per_run_drift
  expr: p95(cost_per_run) > 2 * baseline_p95     # catches loops and prompt bloat
  for: 30m
- alert: loop_depth
  expr: max(steps_per_run) > 25                  # a thrashing agent, not a hard one
  for: 5m
- alert: tool_error_rate
  expr: tool_errors / tool_calls > 0.10          # per tool, not aggregated
  for: 15m
- alert: liveness
  expr: time() - last_span_timestamp > 600       # silence IS the signal
  for: 0m                                        # page immediately

Two design choices matter more than the numbers. Alert on percentiles and windows, not single runs — one expensive run is an agent doing a hard task, a drifting p95 is a regression. And keep tool_error_rate per tool: an aggregate hides the one integration that broke, which is usually the whole story.

The standard: OpenTelemetry GenAI conventions

Until recently every observability vendor invented its own schema for LLM telemetry, and switching backends meant re-instrumenting. The GenAI semantic conventions fix the vocabulary: operations like create_agent, invoke_agent and execute_tool, attributes like gen_ai.operation.name, gen_ai.agent.name, gen_ai.request.model and the gen_ai.usage.* token counters. The conventions are still marked Development: names can change between releases, so pin your instrumentation library versions and expect a migration or two. They are still the only vocabulary with multi-vendor momentum, and emitting them today keeps your data portable.

The practical consequence: your agent telemetry is ordinary OTLP. A general backend (Grafana, Datadog, anything OTLP-capable) can store and alert on it. LLM-specific platforms — Langfuse, Arize Phoenix — sit on the same data and add what generic tools lack: rendered prompts and outputs per step, cost breakdowns, and judge scores attached to traces. Add the specialised layer when humans start reading individual traces daily, not before.

Cost is an observability problem

Cost failures are behavioural, which is why they belong here and not only in the finance review. A prompt change that doubles context, a tool that returns bloated payloads, a retry loop on a flaky endpoint. Each shows up as token counts long before it shows up on an invoice. Summing gen_ai.usage.input_tokens and output_tokens over the trace gives you cost per run in near real time, and the cost_per_run_drift alert above turns the invoice surprise into a same-day page. The budget-and-breaker side of this — what should stop the agent when cost runs away — is covered in agent cost and FinOps and the kill switch design.

Where to start, in order

A success flag and token counts per run. No infrastructure required beyond logging two numbers. This is outcome rate and cost per run — the two alerts that matter most.
Structured traces with the GenAI conventions. One span per model call and per tool call, so any run can be reconstructed. The tracing article has the code.
The alert set above, tuned for two weeks against your baseline before anyone gets paged.
An LLM-aware viewer once people debug traces daily.
Dashboards last. A dashboard nobody acts on is decoration; every panel should map to an alert or a decision. The metrics that deserve panels are in agent production metrics.

Observability does not make an agent good. It makes an agent’s behaviour a fact you can check. That is the precondition for every other discipline on this site, from evals to the audit trail your governance work will eventually ask for.