Dashboarding Agent Fleets: From Spans to Business Metrics

Your tracing backend is full of agent spans. Your VP of Operations wants to know whether the support agent is resolving tickets. Those two facts are not in conflict, but they are not the same artifact, and most agent dashboards fail because they try to serve one audience with the other’s data.

Traces are engineering data: dense, per-request, perfect for the 2 a.m. incident and useless in a quarterly business review. Business metrics are aggregate, slow, and answer “how are we doing.” The work is building the pipeline between them without throwing away the ability to drill from a KPI back to the exact trace that hurt it.

The pipeline: instrument → collect → aggregate → visualize

Instrument your agent runtime against a shared vocabulary. Microsoft’s Agent Framework does this for you: its observability guide states that it “emits traces, logs, and metrics according to the OpenTelemetry GenAI Semantic Conventions.” It produces a predictable span hierarchy — invoke_agent <agent_name> at the top, chat <model_name> for each model call, execute_tool <function_name> for each tool — and histogram metrics such as gen_ai.client.operation.duration, gen_ai.client.token.usage, and agent_framework.function.invocation.duration. In Python, enabling it is a one-liner:

from agent_framework.observability import configure_otel_providers

# Reads OTEL_EXPORTER_OTLP_* environment variables automatically
configure_otel_providers(
    service_name="customer-support-agent",
    resource_attributes={"deployment.environment.name": "production"},
    otlp_endpoint="http://otel-collector:4317",
    otlp_protocol="grpc",
)

Collect everything through an OpenTelemetry Collector over OTLP rather than pointing every service at your vendor backend. The collector is where you drop noisy attributes, batch, and re-aggregate before storage — the metrics data model explicitly supports spatial reaggregation and delta-to-cumulative conversion as collector-side transformations. For local development, the Agent Framework docs show how to run the Aspire Dashboard in Docker with an OTLP endpoint on port 4317, which is a cheap way to validate your instrumentation before it matters.

Aggregate deliberately. This is where spans become metrics, and where most teams silently lose information.

Visualize in layers, not in one screen.

The mapping layers

Each layer has a different audience, a different time range, and a different cardinality budget.

Layer 1 — LLM call spans

Per model call, the GenAI span conventions give you the raw material: gen_ai.operation.name, gen_ai.request.model, gen_ai.response.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.response.finish_reasons, and error.type when the operation fails. Finish reasons are the quiet killer here — a run that ends in length instead of stop is a truncated answer, and it never shows up as an error.

Layer 2 — Tool spans

execute_tool spans aggregate into tool error rate and duration. The semconv defines gen_ai.execute_tool.duration as a histogram, and Agent Framework adds its own agent_framework.function.invocation.duration. A tool that succeeds slowly is a different problem from a tool that fails fast; you need both views.

Layer 3 — Run and session level

One user request is one invoke_agent span containing everything beneath it. The GenAI conventions define gen_ai.invoke_agent.duration, gen_ai.invoke_agent.inference_calls, and gen_ai.invoke_agent.tool_calls, plus gen_ai.invoke_workflow.duration for end-to-end workflows. This is the level where “retries,” “human escalations,” and “run success” live, because a run is the unit a user actually experiences.

Layer 4 — Business metrics

Now you can define outcomes in terms executives recognize:

Metric	Definition	Built from
Task completion rate	Runs reaching a successful terminal state ÷ runs started	Run spans + application outcome event
Cost per successful task	Σ token spend ÷ completed tasks	gen_ai.client.token.usage + outcome
Tool error rate	Failed execute_tool spans ÷ tool calls	Tool spans
Containment / escalation rate	Runs ended without human handoff ÷ runs	Run spans + handoff events
P50 / P95 latency per agent step	Histogram quantiles per span kind	Duration histograms

The honest part of this table: business outcomes are not in the span data. “Ticket resolved” is an application fact. Emit it yourself as a counter at the moment you know it, tagged with the same service.name, agent name, and environment as your traces so the layers join.

from agent_framework.observability import get_meter

meter = get_meter()
runs = meter.create_counter("agent.runs", unit="{run}", description="Agent runs by outcome")
# called when the application knows the real outcome
runs.add(1, {"agent": "support_triage", "outcome": "resolved"})

Exemplars: the bridge back to the trace

A dashboard that cannot answer “show me an example” is a dead end. The OpenTelemetry metrics data model defines exemplars as a recorded value carrying trace_id and span_id alongside the observation, with the stated use case being “to allow users to link Trace signals w/ Metrics.” Enable exemplar recording in your SDK and a backend that supports it lets you click a spike in agent.runs{outcome="failed"} and land on a real trace.

That single feature is the difference between a dashboard and a debugging tool. Build the layout around it.

Suggested dashboard layout

1. Fleet overview. Runs per agent, success rate, P95 run latency, token spend, escalation rate — one row per agent, time range at the top, environment as a filter.
2. Per-agent drill-down. Latency split by span kind (invoke_agent vs chat vs execute_tool), tool error rate, inference calls and tool calls per run, finish-reason distribution, cost trend.
3. Single-trace view. Exemplar click-through to the waterfall, then the session’s event history for full context.

Never make the waterfall the landing page. Engineers go straight there; everyone else bounces.

Metrics worth tracking

• P50 and P95 latency per agent step, not per request. Users feel the slow tool, not the average.
• Token spend per successful task. Spend alone rewards failure: a loop that retries forever looks like engagement.
• Tool error rate by tool name, with error.type kept low-cardinality.
• Containment and escalation rate, because deflection without quality is a vanity metric.
• Truncated completions (gen_ai.response.finish_reasons containing length).

Pitfalls

Counting tokens but not successes. Token spend going up while completion rate goes flat means you are buying retries. Always pair a spend chart with an outcome chart on the same screen.

Averaging away P95. The metrics concepts explain that a Histogram is the instrument for value statistics — “how many requests take fewer than 1s?” A mean over a histogram hides the tail where your worst runs live. The GenAI conventions even ship explicit bucket boundaries for gen_ai.client.operation.duration (0.01 through 81.92 seconds) so percentiles resolve where agent operations actually land.

Cardinality blowups from per-user labels. The same OTel page warns that high-cardinality attributes such as user IDs “can cause unbounded memory growth,” and that the SDK enforces a cardinality limit — default 2000 unique attribute combinations per metric stream — beyond which measurements are folded into an overflow data point tagged otel.metric.overflow=true. The insidious part: when a stream overflows, the entire attribute combination is replaced, so a success=false dimension can disappear from your error-rate query while the total still looks correct. Keep user IDs in traces and baggage (the fourth signal), where cardinality is cheap, and keep metrics labeled by agent, model, environment, and outcome only.

Sensitive data in span attributes. Agent Framework gates prompt and tool-argument capture behind ENABLE_SENSITIVE_DATA, and its docs warn to enable it only in development or test. Cost and latency dashboards do not need content; your traces do not need PII in production.

Closing

Spans were never the wrong raw material — they were just never the finished product. Instrument against the GenAI semantic conventions, collect through a collector, aggregate into outcome counters you emit yourself, and wire exemplars so every business metric can be traced back to the run that produced it.

Primary sources: Microsoft’s Agent Framework documentation and its observability guide, plus the OpenTelemetry signals, metrics, and metrics data model documentation with the GenAI semantic conventions.