Observability for Agents: OpenTelemetry GenAI Conventions in Practice

One user request to a production agent is not one call. It fans out to somewhere between 4 and 12 model calls, 3 to 8 tool calls, and a handful of retrieval hops before it returns an answer. Run 10 of those agents at once and the only number most teams have is a request duration. That number tells you the run was slow. It does not tell you the third retrieval hop timed out, the router quietly downgraded you to a cheaper model, or the tool call that burned 40 seconds never returned. You are debugging a distributed system with a stopwatch.

OpenTelemetry's GenAI semantic conventions are the trace-first alternative, and they run on the collector and backend your platform team already operates. This piece covers what the gen_ai.* conventions capture today, the parts you still instrument yourself, and a checklist that gets you to span-level signal in about two hours. The payoff: the conventions now define far more than an LLM call, but "in the spec" and "emitted by your installed instrumentation" are different facts, and the gap between them is where your incident review goes dark.

What the conventions actually cover

The gen_ai.* attribute namespace extends the core OpenTelemetry semantic conventions (stable release 1.43.0 as of this writing). The GenAI conventions themselves live in a dedicated repository, open-telemetry/semantic-conventions-genai, and carry a document status of Development. Read that as a warning label: the names are usable and shipping in real instrumentation, but they are not frozen, and core stability guarantees do not cover them yet.

A single model call gets a span named {gen_ai.operation.name} {gen_ai.request.model} (for example, chat gpt-5.1), per the spans spec. The attributes that carry the signal:

Attribute	What it captures	What it does not tell you
`gen_ai.provider.name`	Which provider served the call (`openai`, `anthropic`, `aws.bedrock`)	Which model ran, or what it cost
`gen_ai.request.model` / `gen_ai.response.model`	Model asked for vs. model that answered	Whether a router downgraded you, unless you compare the two
`gen_ai.operation.name`	The operation (`chat`, `embeddings`, `execute_tool`)	Where in the agent loop the call sits
`gen_ai.usage.input_tokens` / `gen_ai.usage.output_tokens`	Token counts for the call	Dollar cost (you multiply by your rate); nothing at all if the call streamed and never completed

Note the attribute name: the current spec uses gen_ai.provider.name. The older gen_ai.system is not in the current attribute registry, yet plenty of installed instrumentation still emits it. Both appear in real trace stores today, a failure mode in its own right (see below).

What they leave to you, and the gap you instrument

Here is what changed, and what most 2026 write-ups get wrong: tool calls, retrieval, memory, and agent operations are no longer missing from the spec. The same repo now defines an execute_tool span with gen_ai.tool.name and gen_ai.tool.call.id, retrieval spans (gen_ai.retrieval.*), and a separate agent-spans document with invoke_agent and create_agent operations and gen_ai.agent.name. Do not reinvent tool.name as a custom attribute. It exists.

The real gap is two-sided. First, everything above is Development status, so coverage in your instrumentation library lags the document. Second, the spec deliberately stops at the model boundary. Three things it does not define, which you add yourself:

Loop position. Nothing tells you this chat span is iteration 7 of a runaway loop. Add agent.loop.iteration.
Business context. Which tenant, which feature flag, which user tier triggered the run. Add app.tenant.id and friends.
A derived success flag. The spec records error.type (a stable core attribute) but no tool-success boolean. Derive one so you can alert on tool failure rate.

A minimal Python instrumentation, using the core SDK so it works regardless of your framework:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

trace.set_tracer_provider(TracerProvider())
trace.get_tracer_provider().add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="http://collector:4317"))
)
tracer = trace.get_tracer("agent.runtime")

with tracer.start_as_current_span(f"{op} {model}") as span:
    span.set_attribute("gen_ai.operation.name", op)          # "chat"
    span.set_attribute("gen_ai.provider.name", "openai")
    span.set_attribute("gen_ai.request.model", model)
    # ... call the model ...
    span.set_attribute("gen_ai.response.model", resp.model)
    span.set_attribute("gen_ai.usage.input_tokens", resp.usage.input_tokens)
    span.set_attribute("gen_ai.usage.output_tokens", resp.usage.output_tokens)

    # attributes the spec does not define - you own these
    span.set_attribute("agent.loop.iteration", step)
    span.set_attribute("app.tenant.id", tenant_id)

The with block matters as much as the attributes: nesting your tool and retrieval spans inside the parent model span is what keeps the trace connected. Break the context and you get orphans.

Where to send the traces

Any OTLP backend ingests these spans. The choice is whether the backend understands them. Pricing below is the vendor's published model, fetched 2026-07-30; confirm current numbers before you commit budget.

	Purpose-built (Phoenix / LangSmith / Braintrust)	Plain OTel (Honeycomb / Grafana Tempo)	Self-hosted Jaeger
GenAI attributes out of the box	Native: parses `gen_ai.*`, renders prompts and tool calls	Generic spans; `gen_ai.*` stored as raw attributes, no GenAI views	Same as plain OTel; you build every view
Pricing model	Phoenix self-host free; LangSmith 1 seat free then $39/seat/mo plus per-trace; Braintrust free then $249/mo plus usage	Honeycomb free tier then Pro from $150/mo (event volume); Grafana Cloud free then $19/mo plus usage	Free software; you pay for the host and storage
PII-in-traces risk	High: prompt and completion capture is surfaced prominently	Medium: you decide what to export; no prompt UI	Lowest external exposure; highest ops burden
Token-budget alerting	Built on `gen_ai.usage.*` natively	Yes, if you export the token attributes and write the query	Yes, you build the alert
Eval integration	Native: evals run off the stored traces	None built in; export to a separate harness	None; wire your own

Which column, by org profile: a solo or early team wanting evals in the same pane picks Phoenix self-hosted (free) or LangSmith's free tier. A regulated shop that cannot let prompt text leave its boundary runs Jaeger or Honeycomb inside its own VPC with redaction at the collector. A team already standardized on Grafana or Honeycomb keeps it and adds one eval export rather than onboarding a new vendor. Phoenix ingests OTLP directly, so "purpose-built" and "self-hosted" are not exclusive.

What your security team will ask

Forward this section as-is.

Is prompt text in spans PII? It can be. Content capture (gen_ai.input.messages, gen_ai.output.messages) is marked Opt-In in the spec and is off by default. Turn it on and your spans carry raw user input and model output, which for most products means regulated data. Leave content off in production, or gate it behind redaction; the token, model, and finish_reasons attributes give you the debugging signal without the content.

Where do traces land and who can read them? Wherever your OTLP exporter points. If that is a vendor cloud and you enabled content capture, prompt text left your boundary. For regulated data, self-host the collector and backend in your own network, or strip content before it is exported.

Can we redact completion text without breaking the eval pipeline? Yes. Redact in an OpenTelemetry Collector processor sitting between your app and the backend, so metadata (tokens, tool names, finish reasons) survives while content is stripped. Run evals on a separate, access-controlled content path rather than off the shared trace store.

What breaks first

Ranked by how often it bites, drawn from the spec's own caveats:

Attribute-name drift. Your installed instrumentation emits gen_ai.system; your dashboard queries gen_ai.provider.name (the current registry name). Neither errors. Half your data is silently invisible until someone notices the provider facet is empty.
Streaming token counts vanish. Usage attributes are recorded when the call completes. A streaming call that errors or is cancelled mid-stream records no tokens, so your cost dashboard undercounts exactly the failures you most want to see.
Orphaned child spans. Trace context is not propagated through the agent loop, so execute_tool and retrieval spans detach from the parent chat span. The trace reads as unrelated fragments and you cannot see which tool the model called.
Sampling hides the incident. Head sampling decides up front and keeps a fixed fraction of whole traces, so set it low and rare failures are dropped at the same rate as routine runs. Use tail-based sampling, which sees the finished trace before deciding, to keep every error trace plus a small sample of clean ones, or you will review the wrong runs.

Two-hour setup checklist

Install the OpenTelemetry SDK, an OTLP exporter, and your framework's GenAI instrumentation (reference implementations live in the genai repo).
Point the exporter at an OpenTelemetry Collector, not directly at the backend, so redaction and sampling have a home.
Run one agent request and confirm gen_ai.provider.name, gen_ai.request.model, and both gen_ai.usage.*_tokens attributes actually land.
Add the attributes the spec omits: agent.loop.iteration, app.tenant.id, a derived tool-success flag.
Verify context propagation: tool and retrieval spans are children of the parent model span, not siblings.
Add a Collector redaction processor so prompt and completion text is stripped before anything leaves your boundary.
Configure tail-based sampling: keep 100 percent of error traces, sample clean ones.
Wire one alert on token budget: sum gen_ai.usage.input_tokens and gen_ai.usage.output_tokens per window against a threshold.

Those traces are also the input your eval harness reads. Once they are flowing, the next move is gating merges on eval scores so a regression cannot ship, covered in CI for Agents, and handing the redaction plan to the reviewers in the agent-skill security review.