Tracing AI agents with OpenTelemetry: spans, attributes and working code

When an agent fails, the first question is always the same: what did it actually do? An agent that plans, retries and calls five tools over thirty seconds outruns anyone’s memory of what it was supposed to do. A trace answers the question with data: one tree of timed spans per run, one span per model call and tool call, each carrying the attributes that let you replay the decision. Since the OpenTelemetry GenAI conventions standardised the vocabulary, that tree is portable — the same instrumentation feeds Grafana today and an LLM-specific viewer tomorrow.

The vocabulary: three operations, one namespace

The GenAI conventions model agent work as three span-producing operations, named in gen_ai.operation.name:

• create_agent — an agent instance is configured (system instructions, tools, model).
• invoke_agent — one agent run; usually the root span of the trace.
• execute_tool — one tool call inside a run; a child of the invocation.

Around them sits one attribute namespace. The ones you will use daily: gen_ai.agent.name, gen_ai.provider.name, gen_ai.request.model, and gen_ai.conversation.id to tie multi-turn work together. Add gen_ai.tool.definitions for what the agent could call, plus the gen_ai.usage.input_tokens / output_tokens counters that make cost per run computable from the trace alone. The conventions are marked Development: emit them, but pin your semconv and instrumentation versions, because attribute names have changed before and can change again.

Working code

The pattern is two nested spans and honest attributes. This is provider-neutral Python with the OpenTelemetry SDK — swap the client calls for your own:

from opentelemetry import trace

tracer = trace.get_tracer("agent")

def run_agent(task: str, run_id: str) -> str:
    with tracer.start_as_current_span(
        "invoke_agent support-triage",
        attributes={
            "gen_ai.operation.name": "invoke_agent",
            "gen_ai.agent.name": "support-triage",
            "gen_ai.provider.name": "anthropic",
            "gen_ai.request.model": "claude-sonnet-5",
            "gen_ai.conversation.id": run_id,
        },
    ) as run_span:
        plan = call_model(task)                      # model call -> its own span
        for step in plan.tool_calls:
            with tracer.start_as_current_span(
                f"execute_tool {step.name}",
                attributes={
                    "gen_ai.operation.name": "execute_tool",
                    "gen_ai.tool.name": step.name,
                },
            ) as tool_span:
                result = execute(step)
                tool_span.set_attribute("tool.outcome",
                                        "success" if result.ok else "error")
        answer = call_model(plan.synthesize())
        run_span.set_attribute("gen_ai.usage.input_tokens", usage.input)
        run_span.set_attribute("gen_ai.usage.output_tokens", usage.output)
        return answer

Three habits make the data worth having. Put the run id on every span (gen_ai.conversation.id), because the id is what turns “an error somewhere” into “this run, this step”. Record outcome on tool spans explicitly — a tool that returns HTTP 200 with garbage is a success to the transport and a failure to the agent, and only your attribute knows the difference. And treat prompts and outputs as sensitive payload. The conventions define input and output message capture, but opt in deliberately, with redaction — do not ship customer data to your telemetry backend by default.

A correlation id is a trace you did not have to build

The habit pays off outside your own code too. In August a deploy pipeline in this portfolio failed with a bare 500 Request failed from the hosting provider’s API. That is the same symptom an expired credential produces, and the token had a history of expiring. The response carried one thing of value: a correlation_id. The pipeline’s logs showed the preceding step had authenticated fine and resolved the account, which localised the fault to the provider’s side of that one call. A rerun succeeded twenty minutes later. Without those two trace fragments, the obvious move was rotating a healthy credential and rewriting a working workflow — an afternoon of self-inflicted work to fix a fault that was never ours. The trace is what tells you where the failure is not.

From trace to operations

The trace tree is raw material; the operating value comes from what you derive from it:

• Metrics — outcome rate, cost per run, loop depth and tool error rate aggregate straight off spans; the alert rules are in agent observability, the panel set in production metrics.
• Debugging — a failed run is a tree you read, not a log you grep. Keep every failed, slow or expensive trace even if you sample the routine ones.
• Evidence — regression evals replay real traced cases, and the audit trail your governance obligations expect is the trace store with retention applied.
• Stop conditions — the kill switch trips on the same signals the trace emits; an untraced agent cannot even tell you it needs stopping.

Instrument the loop once, with the standard vocabulary, and every later discipline — alerting, evaluation, audit, control — reads from the same tree.