Agent Evaluation with OpenTelemetry GenAI Semantic Conventions

You cannot improve what you cannot measure. AI agents make tool calls, chain reasoning steps, and produce outputs — but without observability, you are flying blind. Did the agent call the right tool? How many LLM tokens did this interaction cost? Which tool call failed and why?

OpenTelemetry provides the instrumentation framework, and the GenAI semantic conventions define the standard attributes for AI workloads. Together, they give you a unified way to trace, measure, and evaluate agent behavior across your entire stack.

Why This Matters

Agent evaluation is fundamentally different from evaluating a single LLM call. An agent interaction is a chain of operations:

1. User input processing
2. Tool selection (which tool, what arguments)
3. Tool execution (API calls, database queries)
4. Result processing
5. Response generation

Each step can fail, introduce latency, or consume unexpected resources. Without tracing, you see only the final output. With tracing, you see the entire decision tree.

OpenTelemetry GenAI semantic conventions standardize how you record this data:

• gen_ai.system — The AI provider (openai, anthropic, etc.)
• gen_ai.request.model — The model used for each step
• gen_ai.usage.input_tokens — Token count per request
• gen_ai.usage.output_tokens — Token count per response
• gen_ai.operation.name — The operation type (chat, completion)
• gen_ai.tool.name — Which tool was called
• gen_ai.tool.input — What arguments were passed

Setting Up OpenTelemetry for Agents

Installation

pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp \
    opentelemetry-instrumentation-openai opentelemetry-instrumentation-httpx

Configure the Tracer Provider

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource

resource = Resource.create({
    "service.name": "my-ai-agent",
    "service.version": "1.0.0",
    "deployment.environment": "production",
})

provider = TracerProvider(resource=resource)
processor = BatchSpanProcessor(
    OTLPSpanExporter(endpoint="http://localhost:4317"),
)
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)

tracer = trace.get_tracer("agent-tracer")

Instrument LLM Calls

The OpenAI instrumentation automatically captures LLM calls:

from opentelemetry.instrumentation.openai import OpenAIInstrumentor

# Auto-instrument OpenAI calls
OpenAIInstrumentor().instrument()

# Now every OpenAI call is automatically traced
from openai import OpenAI
client = OpenAI()

# This call produces a span with gen_ai.* attributes
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "What is MCP?"}],
)

Instrument Tool Calls

For MCP tool calls, create custom spans with GenAI semantic conventions:

from opentelemetry import trace
from opentelemetry.semconv.trace import SpanAttributes

tracer = trace.get_tracer("agent-tools")

async def call_mcp_tool(session, tool_name: str, arguments: dict) -> dict:
    """Call an MCP tool with full tracing."""
    with tracer.start_as_current_span(
        f"mcp.tool.{tool_name}",
        attributes={
            "gen_ai.tool.name": tool_name,
            "gen_ai.tool.input": json.dumps(arguments),
            "gen_ai.system": "mcp",
        },
    ) as span:
        try:
            result = await session.call_tool(tool_name, arguments)

            # Record success
            span.set_status(trace.StatusCode.OK)
            span.set_attribute("gen_ai.tool.output", json.dumps(result.content)[:500])
            span.set_attribute("gen_ai.tool.success", True)

            return result

        except Exception as e:
            span.set_status(trace.StatusCode.ERROR, str(e))
            span.set_attribute("gen_ai.tool.success", False)
            span.set_attribute("gen_ai.tool.error", str(e))
            raise

Building an Agent Evaluation Pipeline

Instrument the Full Agent Loop

Wrap your agent with comprehensive tracing:

from opentelemetry import trace

tracer = trace.get_tracer("agent-evaluation")

class EvaluatedAgent:
    def __init__(self, agent, llm_client):
        self.agent = agent
        self.llm = llm_client

    async def process(self, user_message: str) -> dict:
        """Process a user message with full evaluation tracing."""
        with tracer.start_as_current_span("agent.process") as root_span:
            root_span.set_attribute("gen_ai.operation.name", "agent_process")
            root_span.set_attribute("user.message", user_message[:200])

            # Step 1: Understand intent
            with tracer.start_as_current_span("agent.understand_intent") as intent_span:
                intent = await self._understand_intent(user_message)
                intent_span.set_attribute("agent.intent", intent)
                intent_span.set_attribute("gen_ai.request.model", "gpt-4o")

            # Step 2: Select and call tools
            with tracer.start_as_current_span("agent.execute_tools") as tools_span:
                tool_results = await self._execute_tools(intent, user_message)
                tools_span.set_attribute("agent.tools_called", len(tool_results))
                tools_span.set_attribute(
                    "agent.tool_names",
                    [r["tool"] for r in tool_results],
                )

            # Step 3: Generate response
            with tracer.start_as_current_span("agent.generate_response") as response_span:
                response = await self._generate_response(
                    user_message, intent, tool_results
                )
                response_span.set_attribute("gen_ai.usage.output_tokens", len(response.split()))
                response_span.set_attribute("response.length", len(response))

            # Record overall metrics
            root_span.set_attribute("agent.total_tools_called", len(tool_results))
            root_span.set_attribute("agent.success", True)

            return {
                "response": response,
                "intent": intent,
                "tools_called": tool_results,
                "trace_id": root_span.get_span_context().trace_id,
            }

Track Cost and Performance

from dataclasses import dataclass
import time

@dataclass
class AgentMetrics:
    total_input_tokens: int = 0
    total_output_tokens: int = 0
    tool_calls: int = 0
    tool_failures: int = 0
    total_latency_ms: float = 0
    cost_usd: float = 0.0

class CostTracker:
    # Pricing per 1K tokens (example rates)
    PRICING = {
        "gpt-4o": {"input": 0.0025, "output": 0.01},
        "gpt-4o-mini": {"input": 0.00015, "output": 0.0006},
    }

    def calculate_cost(self, model: str, input_tokens: int, output_tokens: int) -> float:
        rates = self.PRICING.get(model, self.PRICING["gpt-4o"])
        return (
            (input_tokens / 1000) * rates["input"]
            + (output_tokens / 1000) * rates["output"]
        )

async def track_agent_interaction(agent, user_message: str) -> AgentMetrics:
    """Track metrics for a single agent interaction."""
    metrics = AgentMetrics()
    start_time = time.time()

    with tracer.start_as_current_span("agent.interaction") as span:
        # Track each LLM call
        for step in ["understand", "plan", "execute", "respond"]:
            with tracer.start_as_current_span(f"agent.{step}") as step_span:
                # Simulate LLM call with token tracking
                input_tokens = estimate_tokens(user_message)
                output_tokens = estimate_tokens(f"response for {step}")

                metrics.total_input_tokens += input_tokens
                metrics.total_output_tokens += output_tokens

                step_span.set_attribute("gen_ai.usage.input_tokens", input_tokens)
                step_span.set_attribute("gen_ai.usage.output_tokens", output_tokens)

        metrics.total_latency_ms = (time.time() - start_time) * 1000
        metrics.cost_usd = CostTracker().calculate_cost(
            "gpt-4o", metrics.total_input_tokens, metrics.total_output_tokens
        )

        span.set_attribute("agent.cost_usd", metrics.cost_usd)
        span.set_attribute("agent.latency_ms", metrics.total_latency_ms)
        span.set_attribute("agent.total_tokens", metrics.total_input_tokens + metrics.total_output_tokens)

    return metrics

Evaluation Metrics

Automated Quality Metrics

Track these metrics per agent interaction:

from enum import Enum

class EvaluationDimension(Enum):
    TOOL_SELECTION = "tool_selection"       # Did the agent choose the right tool?
    ARGUMENT_QUALITY = "argument_quality"   # Were tool arguments correct?
    RESPONSE_QUALITY = "response_quality"   # Was the final response helpful?
    EFFICIENCY = "efficiency"               # Minimum tool calls for the task?
    ERROR_HANDLING = "error_handling"       # Did failures get handled gracefully?

def evaluate_interaction(interaction: dict) -> dict:
    """Score an agent interaction across evaluation dimensions."""
    scores = {}

    # Tool selection: did it use tools relevant to the query?
    tools_used = interaction.get("tools_called", [])
    relevant_tools = interaction.get("expected_tools", [])
    if relevant_tools:
        scores["tool_selection"] = len(
            set(t["tool"] for t in tools_used) & set(relevant_tools)
        ) / len(relevant_tools)

    # Efficiency: fewer tool calls for same result is better
    min_expected_calls = interaction.get("min_expected_calls", 1)
    actual_calls = len(tools_used)
    scores["efficiency"] = min(min_expected_calls / actual_calls, 1.0)

    # Error handling: did it retry or fail gracefully?
    errors = [t for t in tools_used if not t.get("success", True)]
    scores["error_handling"] = 1.0 if not errors else max(0, 1.0 - len(errors) * 0.3)

    return scores

Exporting to Observability Backends

from opentelemetry.metrics import MeterProvider
from opentelemetry.sdk.metrics import MeterProvider as SdkMeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader

# Set up metrics
reader = PeriodicExportingMetricReader(
    OTLPMetricExporter(endpoint="http://localhost:4317"),
    export_interval_millis=30000,
)
meter_provider = SdkMeterProvider(resource=resource, metric_readers=[reader])
meter = meter_provider.get_meter("agent-metrics")

# Create counters and histograms
tool_call_counter = meter.create_counter(
    "agent.tool.calls",
    description="Number of tool calls made by agents",
    unit="1",
)

latency_histogram = meter.create_histogram(
    "agent.interaction.latency",
    description="Agent interaction latency in milliseconds",
    unit="ms",
)

cost_counter = meter.create_counter(
    "agent.cost.usd",
    description="Total cost of agent interactions in USD",
    unit="USD",
)

# Record metrics
tool_call_counter.add(1, {"tool.name": "get_order", "agent.name": "support-agent"})
latency_histogram.record(1250.0, {"agent.name": "support-agent"})
cost_counter.add(0.032, {"model": "gpt-4o", "agent.name": "support-agent"})

Getting Started Tutorial

Step 1: Set Up a Collector

Run an OpenTelemetry Collector to receive and export traces:

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

exporters:
  otlp/jaeger:
    endpoint: jaeger:4317
  prometheus:
    endpoint: 0.0.0.0:8889

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp/jaeger]
    metrics:
      receivers: [otlp]
      exporters: [prometheus]

docker run -p 4317:4317 -p 4318:4318 \
    -v $(pwd)/otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml \
    otel/opentelemetry-collector-contrib

Step 2: Instrument Your Agent

Add tracing to your agent’s core loop:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

# Initialize
provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("my-agent")

# Instrument
with tracer.start_as_current_span("agent.query") as span:
    span.set_attribute("gen_ai.operation.name", "agent_query")
    # Your agent logic here

Step 3: Build a Dashboard

Use Grafana with Jaeger and Prometheus backends to visualize:

• Trace view: See the full span tree for each agent interaction
• Metrics view: Track latency, cost, and error rates over time
• Comparison view: Compare performance across model versions

Best Practices

• Sample intelligently: Do not trace every single interaction in production. Sample 10% of successful requests and 100% of failures.
• Use semantic conventions: Stick to the gen_ai.* attribute namespace. Custom attributes are fine, but standard ones enable cross-tool comparison.
• Correlate traces with quality scores: Link trace IDs to human feedback scores. This lets you find traces that produced poor responses and debug the decision chain.
• Set cost budgets: Alert when agent cost per interaction exceeds a threshold. This catches runaway tool loops before they burn budget.
• Track model versions: Record gen_ai.request.model and gen_ai.response.model so you can compare performance across model updates.

Common Pitfalls

• Tracing everything at verbose level: Detailed traces are useful for debugging but expensive in production. Use sampling and log levels wisely.
• Ignoring tool call latency: A fast LLM call followed by a slow tool call is still a slow interaction. Trace the tool calls, not just the LLM.
• Not correlating cost with quality: High cost is acceptable if quality is high. Track both together to find the cost-quality sweet spot.
• Forgetting about context propagation: Traces must propagate across service boundaries. Ensure MCP HTTP clients forward trace headers.

Conclusion and Next Steps

OpenTelemetry GenAI semantic conventions provide the foundation for evaluating AI agents at scale. Instrument your agent, track tool calls and costs, and build dashboards that show you exactly what is happening in every interaction.

The next step is building automated evaluation pipelines that score agent responses against expected behavior, flag regressions, and trigger alerts when quality drops. Start with the tracing foundation here, and the evaluation layer builds naturally on top.