End-to-End OpenTelemetry Tracing for ADK Agents

When an AI agent fails or responds unexpectedly in production, standard print() statements and linear logs are inadequate. Agent execution is non-deterministic, involving multi-step reasoning, tool invocations, state mutations, and dynamic callbacks. Effective debugging requires full-trajectory observability.

Google ADK natively exports telemetry following OpenTelemetry (OTel) semantic conventions for GenAI applications. This guide demonstrates how to instrument your agent harness with OpenTelemetry to capture detailed execution traces.

The Agent Trajectory

A single user request turn generates a chain of distinct, traceable spans:

flowchart LR
    A["harness.user_turn"]
    B["llm.reasoning<br/>(Model processing &<br/>Time-To-First-Token)"]
    C["tool.check_server_status<br/>(Tool execution & latency)"]
    D["llm.synthesis<br/>(Final response generation)"]
    E((End))

    A --> B
    B --> C
    C --> D
    D --> E

    classDef orchestrator fill:#e0f2fe,stroke:#0284c7,stroke-width:2px,color:#0f172a;
    classDef llm fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#0f172a;
    classDef tool fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#0f172a;
    classDef terminal fill:#f3f4f6,stroke:#6b7280,stroke-width:1.5px,color:#0f172a;

    class A orchestrator;
    class B,D llm;
    class C tool;
    class E terminal;

1. User Prompt Ingestion: Session context setup and prompt preprocessing.
2. LLM Reasoning: Initial model invocation and Time-To-First-Token (TTFT).
3. Tool Dispatch & Arguments: Capturing tool function names and serialized inputs.
4. Tool Execution Duration: Measuring exact execution time and return values.
5. Secondary Synthesis: Final response formulation and token usage aggregation.

Setting Up OpenTelemetry Instrumentation

Below is a complete script demonstrating how to instrument an ADK agent using OpenTelemetry SDK components:

import asyncio
from google.adk.agents import Agent
from google.adk.tools import FunctionTool
from google.adk.runners import Runner
from google.adk.sessions import InMemorySessionService

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import ConsoleSpanExporter, BatchSpanProcessor

# 1. Initialize OpenTelemetry Tracer Provider
provider = TracerProvider()
processor = BatchSpanProcessor(ConsoleSpanExporter())
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)

tracer = trace.get_tracer("adk.observability.demo")

# 2. Define Instrumented Tool
def check_server_status(server_id: str) -> dict:
    """Checks the health and uptime metrics of a target server.

    Args:
        server_id: The infrastructure server ID (e.g. 'srv-01').
    """
    with tracer.start_as_current_span("tool.check_server_status") as span:
        span.set_attribute("server.id", server_id)
        # Simulated server status check logic
        return {"server_id": server_id, "status": "HEALTHY", "cpu_load": "12%"}

server_tool = FunctionTool(func=check_server_status)

# 3. Agent Definition
ops_agent = Agent(
    name="devops_agent",
    model="gemini-2.5-flash",
    instruction="Use `check_server_status` to report infrastructure health.",
    tools=[server_tool]
)

# 4. Harness Execution & Trace Generation
async def main():
    session_service = InMemorySessionService()
    runner = Runner(agent=ops_agent, session_service=session_service)

    print("--- Executing Instrumented Agent Harness ---")

    with tracer.start_as_current_span("harness.user_turn") as root_span:
        root_span.set_attribute("session.id", "trace-session-100")

        response = await runner.run_async(
            session_id="trace-session-100",
            message="Check the health status of server srv-01."
        )
        print("\nAgent Response:\n", response.text)

    # Flush pending spans to exporter
    provider.shutdown()

if __name__ == "__main__":
    asyncio.run(main())

What the Execution Trace Reveals

Running an instrumented harness outputs structured JSON spans containing actionable telemetry:

• Root Span (harness.user_turn): Tracks total turn latency and session metadata.
• Tool Span (tool.check_server_status): Logs execution duration alongside custom attributes like server.id = srv-01.
• ADK Internal Spans: Automatically captures model generation time, token counts (prompt + completion tokens), and API request IDs.

Production Setup: Cloud Trace Exporter

For production workloads on Cloud Run, GKE, or GCP, swap out ConsoleSpanExporter for CloudTraceSpanExporter:

from opentelemetry.exporter.cloud_trace import CloudTraceSpanExporter

exporter = CloudTraceSpanExporter()
processor = BatchSpanProcessor(exporter)
provider.add_span_processor(processor)

When deployed to Google Cloud environments, telemetry flows directly into Cloud Trace and Cloud Logging without code alterations.

Payload Auditing for Compliance

While OpenTelemetry handles span durations and call graphs, security compliance often requires payload auditing — recording sanitized tool inputs and outputs:

from datetime import datetime

async def audit_tool_payload(tool_name: str, args: dict, result: dict) -> None:
    """Logs sanitized tool call inputs and outputs to a security audit ledger."""
    safe_args = redact_pii(args)
    safe_result = redact_pii(result)

    await audit_logger.log({
        "tool": tool_name,
        "input": safe_args,
        "output": safe_result,
        "timestamp": datetime.utcnow().isoformat()
    })

Key Takeaway

Instrumenting your Google ADK agent with OpenTelemetry from day one provides full transparency into non-deterministic execution paths. Capturing span hierarchies, tool latencies, and token counts guarantees rapid root-cause analysis and reliable production performance.