LLM observability: tracing, metrics and evals for AI features in production

Key points

• Record the model requested and returned, the prompt name and version, input, output, cached and reasoning tokens, time to first token, total latency, finish reason, tool calls, retrieval IDs and errors.
• Model one request or agent run as a trace, with a span for every model call, tool execution and retrieval, so you can see where the time and tokens went.
• The OpenTelemetry GenAI semantic conventions give shared names such as gen_ai.request.model and gen_ai.usage.input_tokens, but they are still in Development: pin a version.
• Prompts and outputs are user data. Leave them out by default, redact before export, and keep any captured content behind its own access controls.
• Alert on latency, error, refusal and truncation rates and on token usage, and sample so that every trace with an error or bad feedback is kept.
• Stamp every call with a prompt version and score live traffic with online evals and user feedback, so a regression shows up as a difference between versions.

Why AI features need their own telemetry

Standard application monitoring tells you that a request to a model API took 2.3 seconds and returned HTTP 200. For an AI feature that is the least interesting part. The answer can be wrong, empty, a refusal, cut off at the token limit or built on the wrong documents, and every one of those is a successful request. Three things make LLM features different to operate:

• Failures are often silent. Quality problems do not raise exceptions, so you need signals about the content: finish reasons, validation results, eval scores and user feedback.
• Tokens are the unit of work and spend, not requests. One request can use many times the tokens of another, depending on the size of the context and the number of tool-loop steps.
• One user action is many calls. An agent run can include several model calls, tool executions and retrievals. Without a trace that ties them together, a slow or wrong answer cannot be explained.

This post covers what to record, how to structure it as traces, how to handle the privacy problem, and how to turn the data into dashboards, alerts and evals.

What to record on every model call

Store spend as units, not money. Token counts per model are the durable record; apply your current rates at query time, so a rate change does not rewrite history and cached or reasoning tokens can be weighted correctly.

Read the provider’s usage fields carefully, because they do not mean the same thing. OpenAI’s Responses API reports usage.input_tokens and usage.output_tokens, with cached input in input_tokens_details.cached_tokens and reasoning tokens, which count as output, in output_tokens_details.reasoning_tokens. Anthropic’s usage.input_tokens counts only the tokens after the last cache breakpoint, and total input is that plus cache_read_input_tokens and cache_creation_input_tokens. The OpenTelemetry conventions say gen_ai.usage.input_tokens should include cached tokens, so for Anthropic you add the three fields up.

If you are not ready for tracing, one structured log line per call with the same names gets you most of the way, as long as it carries the trace ID:

{
  "ts": "2026-11-03T10:42:17.381Z",
  "event": "llm.call",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "feature": "ticket-triage",
  "gen_ai.operation.name": "chat",
  "gen_ai.provider.name": "openai",
  "gen_ai.request.model": "example-model",
  "gen_ai.response.model": "example-model-2026-10-01",
  "gen_ai.prompt.name": "triage",
  "gen_ai.prompt.version": "14",
  "gen_ai.usage.input_tokens": 1840,
  "gen_ai.usage.cache_read.input_tokens": 1536,
  "gen_ai.usage.output_tokens": 212,
  "gen_ai.response.finish_reasons": ["stop"],
  "ttft_ms": 412,
  "duration_ms": 2380,
  "tools_called": ["lookup_order"],
  "retrieval_doc_ids": ["kb-118", "kb-342"],
  "schema_valid": true,
  "user": "u_7f3a9c",
  "content": "not captured"
}

Traces and spans for agent runs

In OpenTelemetry terms a trace is the path of one request through your system, and a span is one unit of work inside it, with a start, an end, attributes, events and a status. Spans form a tree: each records its parent, and the root span has none. A span’s kind says what it represents: CLIENT for an outgoing call such as a request to a model API, SERVER for an incoming request, INTERNAL for work that stays inside the process.

For an AI feature the natural tree has one root span for the user’s request, one span for the agent run, and a child span for every model call, tool execution and retrieval:

POST /api/answer                       SERVER    1.92 s
└─ invoke_agent support-assistant      INTERNAL  1.86 s  steps=2
   ├─ retrieval kb-articles            CLIENT    0.12 s  docs=5
   ├─ chat example-model               CLIENT    0.64 s  in=2410 out=38
   │    finish_reasons=["tool_calls"]
   ├─ execute_tool lookup_order        INTERNAL  0.21 s  ok
   └─ chat example-model               CLIENT    0.79 s  in=2690 out=164
        finish_reasons=["stop"]

Read top to bottom, this trace answers the first questions of any incident. The second model call took longest and read the most input tokens, because the tool result was added to the context. The tool ran between the two model calls. If the answer was wrong, the retrieval span says which documents it was built on.

• Name spans by convention: {operation} {model} for model calls, execute_tool {tool} for tools, retrieval {data source} for search and invoke_agent {agent} for the run.
• Keep one trace across process boundaries. When an agent hands work to a queue or a background job, send the trace context with the message so the job’s spans join the same trace; see background jobs and queues.
• Put retries inside the logical span. The conventions say a model-call span should cover the whole logical operation, automatic retries included.
• Record loop counters on the agent span: steps taken, tool calls made, and whether a limit stopped the run. The loop itself is explained in how LLM tool calling works.

The OpenTelemetry GenAI semantic conventions

You can name these attributes anything, but shared names let any backend understand them. The OpenTelemetry GenAI semantic conventions define spans, metrics and events for model calls, embeddings, retrievals, tool executions and agents, with provider-specific pages for OpenAI, Anthropic, AWS Bedrock and Azure AI Inference. Two facts shape how you use them today:

• They are marked Development, not Stable, and names change between drafts. The current draft, for example, replaces a single token-usage histogram with separate input and output token metrics, and adds inference metrics such as gen_ai.client.inference.duration and gen_ai.client.inference.time_to_first_chunk. Pin the version your instrumentation follows, and expect renames when you upgrade.
• They now live in their own repository. The GenAI pages on opentelemetry.io point to open-telemetry/semantic-conventions-genai on GitHub, which has the current text.

The model-call span is the core. Its name should be {gen_ai.operation.name} {gen_ai.request.model}, such as chat example-model, its kind should be CLIENT, and gen_ai.operation.name and gen_ai.provider.name are required. Most of the rest, model names, token counts and finish reasons included, is recommended. Prompts and responses are opt-in, for reasons the next section explains. A hand-written sketch with the OpenTelemetry Python API:

from opentelemetry import trace
from opentelemetry.trace import SpanKind

tracer = trace.get_tracer("support-assistant")

def traced_chat(call_model, provider, model, messages, prompt):
    with tracer.start_as_current_span(f"chat {model}", kind=SpanKind.CLIENT) as span:
        span.set_attribute("gen_ai.operation.name", "chat")
        span.set_attribute("gen_ai.provider.name", provider)  # e.g. "anthropic"
        span.set_attribute("gen_ai.request.model", model)
        span.set_attribute("gen_ai.prompt.name", prompt.name)
        span.set_attribute("gen_ai.prompt.version", prompt.version)
        try:
            r = call_model(model, prompt.render(messages))  # your SDK, normalized
        except Exception as err:
            span.set_attribute("error.type", type(err).__name__)
            raise  # the context manager records it and sets the status to ERROR
        span.set_attribute("gen_ai.response.model", r.model)
        span.set_attribute("gen_ai.response.finish_reasons", [r.finish_reason])
        span.set_attribute("gen_ai.usage.input_tokens", r.input_tokens)  # incl. cached
        span.set_attribute("gen_ai.usage.cache_read.input_tokens", r.cached_tokens)
        span.set_attribute("gen_ai.usage.output_tokens", r.output_tokens)
        return r

In practice an instrumentation library usually writes the model-call span for you by wrapping the provider’s SDK, and your code adds what only it knows: the prompt version, the feature, the retrieval IDs. Whatever writes the spans, check which version of the conventions it follows.

Prompts are user data: privacy and redaction

The most useful debugging data, the actual prompts and responses, is also the most sensitive. Users paste personal details, documents and credentials into chat boxes, and retrieved documents and tool results carry more. OWASP lists sensitive information disclosure as LLM02 in its 2025 Top 10 for LLM applications, and recommends least-privilege access to sensitive data and pattern matching to detect and redact confidential content.

The GenAI conventions take the same line. Content attributes such as gen_ai.input.messages, gen_ai.output.messages, gen_ai.system_instructions and gen_ai.tool.call.arguments are opt-in, the spec says instrumentations should not capture them by default, and it describes three patterns:

1. The default: record no instructions, inputs or outputs. Metrics and metadata still flow.
2. Record content on the spans. The spec suggests this where telemetry volume is manageable and privacy rules do not apply or the telemetry store meets them, for example before production.
3. Store content elsewhere and put a reference on the span, with attributes such as gen_ai.input.messages_ref. The spec recommends this in production, because the content store can have its own access controls.

• Redact before export, not in the dashboard. Scrub emails, phone numbers, card-like numbers, API keys and tokens in the process or in a collector, so raw values never reach the telemetry backend.
• Pseudonymize user IDs. You need to group by user; you rarely need to know who the user is.
• Set retention for captured content separately from metrics, and keep it short.
• Restrict who can open traces with content, and log that access.
• Treat tool arguments and results like prompts. They often carry the same data.

Sampling without losing the interesting traces

Full traces are large, and most of them show a normal request. Sampling keeps a share of them. OpenTelemetry describes two kinds:

For AI features, a useful policy keeps every trace with an error, a refusal, a truncated output, a failed schema validation, a negative rating or a low eval score, every trace slower than your latency target, and a small random share of the rest. Two details keep it honest:

• Compute metrics before sampling. Token counts, latency histograms and error rates should come from every call, or the dashboards move with the sampling rate.
• Feedback arrives late. A user clicks thumbs-down after the trace has ended and the sampler has decided. Record feedback against the trace ID and keep the conversation for feedback cases separately, or the trace you need may be gone.

Dashboards and alerts

Alert on rates and percentiles against a baseline, not on single events: one refusal is normal, a doubled refusal rate after a release is not. Break every chart down by model, prompt version and feature, because an average across features hides the one that broke. Once token usage is visible, how to reduce LLM costs lists the levers for bringing it down.

Online evals, feedback and prompt versions

Metrics catch broken; evals catch wrong. Offline evals run a fixed dataset before you ship, as described in how to evaluate RAG and how to evaluate AI agents. Online evals score live traffic after the fact:

1. Sample production traces, more heavily from the cases your sampling policy flags.
2. Score them off the request path: code checks for format and policy, and an LLM judge for faithfulness to the retrieved context, relevance or tone, calibrated against human labels.
3. Attach every score to the trace ID and the prompt version, the same way you attach user feedback.
4. Review the low scorers by hand, and add the real failures to the offline dataset so the next change is tested against them.

Prompt versioning ties it together. Keep prompts as named, versioned artifacts, or at least give each one an explicit version string, and stamp every call with gen_ai.prompt.name and gen_ai.prompt.version. A regression then becomes a comparison: version 14 against version 13 on refusal rate, tokens per request, eval scores and thumbs-down rate. Roll a new version out to a slice of traffic first, and keep the previous one ready to switch back to.

Watch the model side too. If you call a model through an alias, a change in gen_ai.response.model with no deploy of your own is a model change, and it deserves the same comparison.

Instrument first, then pick a backend

Spans written with OpenTelemetry can go to any compatible backend. Several open-source projects specialize in LLM traces; three examples, described neutrally:

• OpenLLMetry, from Traceloop, is a set of OpenTelemetry instrumentations for LLM providers and vector databases under the Apache 2.0 license. Its output is standard OpenTelemetry, so it can feed an existing observability stack.
• Langfuse is an open-source platform for tracing, prompt management and evals that can be self-hosted. It accepts traces over OTLP and maps gen_ai.* attributes, and its docs note that the conventions are still evolving.
• Arize Phoenix covers tracing, evals, datasets and experiments, and prompt management. Its docs say it is built on OpenTelemetry and uses OpenInference instrumentation, which has its own attribute names.

Whichever you choose, instrument with OpenTelemetry where you can, so the spans are not tied to one tool, and check which conventions and which version the backend expects.

If you need an AI feature built so that it can be measured and debugged, BYD Tech builds AI agents: the code lives in your GitHub from day one, and AI-assisted work is reviewed by an engineer.