Tracking Agent Latency: P50, P95, and P99 in Production

An agent dashboard reporting a 2.1 second average response time is not telling you whether the product works. Two very different systems can produce that number: one where every run takes 2.1 seconds, and one where most runs finish in 400 ms while a slow tail drags a handful out past 30 seconds. The first is a system you can tune; the second is a system that is quietly training users to abandon it. The average cannot tell them apart, and neither can p50 alone.

Percentile discipline is the practice of measuring latency as a distribution and holding yourself to the percentiles users actually feel. This post is about applying it to agent systems, where the problem is sharper than in ordinary request/response services: agents are multi-step, the steps have different cost profiles, and the tail of a run is the product of the tails of its parts.

Why averages lie about agent latency

Latency distributions in LLM-backed systems are long-tailed. A model call might have a median of 800 ms, but a 1-in-50 request takes 12 seconds because of a cold cache, a retry against a saturated provider, a long tool, or a context window that crossed a batch boundary. A mean folds that 12-second outlier into the average across thousands of fast requests, where it moves the number by a few milliseconds and disappears.

Worse, agent runs multiply these tails. A single user request that fans out into five LLM turns and three tool calls does not experience the p95 of any individual step — it experiences something closer to the sum of them, and its own p95 is worse than the p95 of any component. If each step has a 5% chance of hitting a slow tail, and the run makes ten independent steps, the probability that at least one step is slow is 1 - 0.95^10, roughly 40%. That is the arithmetic that turns a “fast” per-step p95 into a bad run-level p95.

The practical consequence: measure at the run boundary, and decompose underneath it. A per-tool or per-call percentile is a diagnostic; the run-level percentile is the thing your users experience and the thing your SLO should describe.

What p50, p95, and p99 actually mean for agent UX

• p50 is the run a typical user gets. It tells you about your happy path and your baseline cost, and almost nothing about whether the product feels reliable.
• p95 is the run a frustrated-but-tolerant user gets. This is usually the right SLO target for interactive agents — it is tight enough to force you to fix real problems, loose enough that you are not paging on noise.
• p99 is the run someone screenshots and posts. At this percentile you are no longer debugging your code; you are debugging provider rate limits, autoscaling lag, and queue depth. It is worth tracking even if you do not set an SLO on it, because drift here is your early warning that the p95 is about to move.

For a rough sense of scale: if your run-level p95 is 8 seconds and you want it under 4, and you have already removed the obvious tool bottlenecks, the remaining lever is almost always reducing the number of sequential steps or streaming the first one, not shaving milliseconds off a single call.

Span structure: give the distribution something to decompose

Before you can explain a slow run you need spans that mirror the run’s real structure. A useful agent trace has four levels:

agent.run                    (one user request; the SLO boundary)
├── agent.queue_wait         (time before the first model call)
├── llm.chat (turn 1)        (gen_ai spans, per provider call)
├── tool.search              (gen_ai.execute_tool span)
├── llm.chat (turn 2)
└── tool.fetch

The queue_wait span is the one teams forget and the one that explains most “mystery” p99. If your run p99 spikes while every model-call p99 is flat, the time is being spent waiting for a semaphore, a GPU shard, or a worker to pick up the task — none of which appears in model latency.

Instrument with the GenAI semantic conventions

The OpenTelemetry GenAI semantic conventions now live in their own repository, open-telemetry/semantic-conventions-genai, having moved off the main semconv site. The conventions define histogram instruments with the bucket boundaries already tuned for this domain, so you should use them rather than inventing your own buckets.

The two that matter most:

Instrument	Type	What it measures
gen_ai.client.operation.duration	Histogram (seconds)	A single provider-facing operation
gen_ai.invoke_agent.duration	Histogram (seconds)	End-to-end agent run — your SLO boundary

gen_ai.client.operation.duration ships explicit bucket boundaries of 0.01, 0.02, 0.04, ... 40.96, 81.92 seconds — geometric, doubling, spanning two orders of magnitude, which is exactly the range an LLM call occupies. gen_ai.server.time_to_first_token has its own finer set (0.001 up to 10.0) because first-token latency lives in a different range from total latency.

The attributes are deliberately low-cardinality. gen_ai.operation.name is required and takes values from a fixed set (chat, embeddings, execute_tool, invoke_agent, retrieve, …). gen_ai.provider.name is conditionally required. gen_ai.request.model and gen_ai.response.model are recommended. error.type becomes conditionally required the moment the operation fails.

Here is that shape in Python using the OpenTelemetry SDK:

from opentelemetry import metrics
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter

reader = PeriodicExportingMetricReader(OTLPMetricExporter(), export_interval_millis=15_000)
meter_provider = MeterProvider(metric_readers=[reader])
meter = metrics.get_meter("agent.runtime")

# gen_ai.client.operation.duration — per model call
op_duration = meter.create_histogram(
    name="gen_ai.client.operation.duration",
    unit="s",
    description="GenAI operation duration",
)

# gen_ai.invoke_agent.duration — per run (the SLO boundary)
run_duration = meter.create_histogram(
    name="gen_ai.invoke_agent.duration",
    unit="s",
    description="End-to-end agent run duration",
)

LOW_CARDINALITY_MODELS = {"gpt-5", "gemini-3-pro", "claude-4-sonnet"}

def record_call(model: str, seconds: float, *, failed: bool = False) -> None:
    # Never emit the raw model string if users can influence it.
    attrs = {
        "gen_ai.operation.name": "chat",
        "gen_ai.request.model": model if model in LOW_CARDINALITY_MODELS else "other",
    }
    if failed:
        attrs["error.type"] = "timeout"
    op_duration.record(seconds, attrs)

Histograms, not summaries, for percentiles across replicas

This is the single most common implementation mistake. A summary computes quantiles at the point of measurement and exports the resulting numbers. It works on one process. It breaks the moment you run more than one replica, because quantiles are not additive: you cannot average the p95 of eight replicas to get the fleet p95. Each replica will report a different p95, and any fleet-level “p95” you compute is a fiction.

A histogram exports bucket counts instead. Bucket counts are additive — you sum the buckets across every replica, then compute the quantile once from the merged distribution. That is why the OpenTelemetry metrics documentation names the histogram as the instrument for value statistics (“how many requests take fewer than 1s?”), and why the GenAI conventions specify explicit bucket boundaries rather than leaving it to you.

In PromQL, if your histogram metric is gen_ai_invoke_agent_duration_seconds with the standard le buckets:

# Run-level p95 across every replica
histogram_quantile(0.95,
  sum by (le) (rate(gen_ai_invoke_agent_duration_seconds_bucket[5m]))
)

# p99 by model, same merge-then-quantile shape
histogram_quantile(0.99,
  sum by (le, gen_ai_request_model) (
    rate(gen_ai_client_operation_duration_seconds_bucket[5m])
  )
)

The sum by (le) before the histogram_quantile is the part people get wrong. Quantile computation must happen after aggregation, never before.

Measure the thing users feel

Total latency is the honest metric, but it hides a distinction that matters enormously for streaming agents: time-to-first-token (TTFT) versus time-to-complete. A user watching a stream feels TTFT almost exclusively; a downstream system consuming the full response feels time-to-complete. They fail for different reasons — TTFT is dominated by queue wait and prefill, time-to-complete by decode rate and output length.

Emit both. The conventions give you gen_ai.server.time_to_first_token for the first, and operation duration for the second. An agent that streams its first token in 400 ms but takes 25 seconds to finish feels fast to a human and slow to an API client; one number cannot serve both audiences.

Decompose, do not just observe

When the run p95 moves, you want to know which component moved. Model the run as three additive parts and record each:

run_duration = queue_wait + Σ(model_latency) + Σ(tool_latency) + overhead

A run-level histogram plus per-span histograms gives you this for free — you can compare gen_ai.invoke_agent.duration against the sum of its children to see whether the gap (scheduling, retries, serialization) is growing. If the run p95 rises while every child p95 is flat, the time is in the gaps, and no amount of model optimization will fix it.

SLOs, error budgets, and alerting on drift

A defensible agent SLO looks something like: p95 of gen_ai.invoke_agent.duration < 8 s, measured over 30 days, excluding runs that the user cancelled. Two properties make it work:

1. It is percentile-based, not average-based. An SLO on the mean will be green while your worst 5% of users are having an unusable experience.
2. It has a stated exclusion for user cancellation. Otherwise a user who abandons a long research run becomes a latency violation you cannot act on.

Error budget is the inverse: with a p95 target, the budget is the fraction of runs you are allowed to let exceed it. Track budget burn rate rather than raw violations, because a fast burn at the start of a window is a genuine emergency while the same number of violations spread across a month is noise.

Alert on percentile drift, not on absolute thresholds alone. A p95 that climbs from 3 s to 6 s is a problem even if 6 s is still inside your 8 s SLO — it means something changed. The complementary alert is the one on error-budget burn rate: it fires when you are consuming budget fast enough to breach before the window closes.

# Multi-window, multi-burn-rate alert on run-level p95
(
  histogram_quantile(0.95, sum by (le) (rate(gen_ai_invoke_agent_duration_seconds_bucket[5m])))
  /
  histogram_quantile(0.95, sum by (le) (rate(gen_ai_invoke_agent_duration_seconds_bucket[1h])))
) > 1.3

That ratio — short window p95 divided by long-window p95 — catches a regression the moment the recent distribution shifts, without you having to hard-code a threshold that goes stale the next time the product gets faster.

Cardinality: the pitfall that eats the metric

The OpenTelemetry metrics SDK enforces a cardinality limit — a maximum number of unique attribute combinations per metric stream, defaulting to 2000. Because the SDK keeps separate aggregation state in memory for each combination, high-cardinality attributes drive memory cost directly.

Two failure modes matter here:

• Per-tool or per-model labels with unbounded values. If a tool name can include a record ID, or a model attribute can carry a fine-tuned model name per tenant, you will hit the limit. When you do, the SDK does not drop measurements — it folds them into a single overflow data point tagged otel.metric.overflow=true. Memory stays bounded and totals stay correct, but any query that groups by the overflowed attribute now undercounts silently. An error-rate alert built on a dropped attribute can stop firing while the system is on fire.
• The overflow replaces the entire attribute combination, not just the high-cardinality part. A {model, success} metric that overflows loses success too, so a success=false query misses those measurements.

The defense is boring: put process-lifetime context (service name, environment, region) on the Resource, where the cardinality limit does not apply and where it survives overflow. Keep measurement attributes to a small, enumerated set — operation name, provider, a curated model list, error class. If you genuinely need per-tool drill-down, put it in spans, where cardinality is cheap, and aggregate tools into a handful of classes for the metric.

Checklist

• Instrument at the run boundary (gen_ai.invoke_agent.duration) and decompose underneath with per-operation spans and histograms.
• Emit TTFT and time-to-complete separately; they serve different audiences and fail for different reasons.
• Use histograms, never summaries, for any percentile you intend to aggregate across replicas — merge buckets first, compute the quantile once.
• Reuse the GenAI conventions’ explicit bucket boundaries instead of inventing your own; they are tuned for the LLM latency range.
• Keep metric attributes low-cardinality and enumerated; put drill-down detail in spans.
• Set SLOs on p95 (or p99), explicitly excluding user-cancelled runs.
• Alert on percentile drift and error-budget burn rate, not on static thresholds alone.
• Watch for otel.metric.overflow=true in your metric streams; its presence means a query somewhere is undercounting.

Further reading

• OpenTelemetry: Metrics concepts
• OpenTelemetry GenAI semantic conventions
• GenAI metrics conventions (operation duration, TTFT, invoke_agent)
• OpenTelemetry: Histogram instrument
• Google SRE Workbook: Service Level Objectives
• Prometheus: histogram_quantile