Monitoring ADK Agent Fleets with BigQuery Analytics

Individual trace inspection is essential for debugging single execution paths. However, at enterprise scale—where millions of agent interactions occur across dozens of specialized agent fleets—you need macro-level, fleet-wide analytics. Streaming agent telemetry into BigQuery enables flexible, SQL-based analysis across your entire agent deployment.

Why BigQuery for Agent Analytics

BigQuery provides serverless, petabyte-scale analytics with real-time streaming capabilities. By streaming agent telemetry into partitioned BigQuery tables, engineering and product teams can dynamically answer critical operational questions:

• Tool Reliability: Which tools exhibit the highest failure rates or timeout ratios?
• Cost Allocation: What is the exact token consumption and estimated cost per customer tenant?
• Performance Profiling: Which agent pipelines or sub-agent invocations are latency bottlenecks?
• Regression Detection: Are model upgrades or prompt changes causing subtle drops in response quality?

What to Stream: Structured Telemetry Schema

For every turn executed by an agent, emit a structured JSON event containing execution metadata, token usage metrics, tool calls, and tenant identifiers:

{
  "session_id": "sess-001",
  "agent_name": "customer_dispatcher",
  "model": "gemini-2.5-flash",
  "input_tokens": 450,
  "output_tokens": 120,
  "tool_calls": [
    {
      "tool": "process_refund",
      "duration_ms": 340,
      "success": true
    }
  ],
  "total_latency_ms": 2800,
  "timestamp": "2026-07-30T12:00:00Z",
  "tenant_id": "customer-abc"
}

Key Analytics Queries

Once your event stream lands in BigQuery, you can execute powerful analytical queries to monitor operational health and efficiency.

1. Cost Tracking per Tenant

Track token usage and compute estimated billing costs per tenant over a rolling 30-day window:

SELECT
  tenant_id,
  SUM(input_tokens + output_tokens) AS total_tokens,
  ROUND(SUM(input_tokens + output_tokens) * 0.0001, 2) AS estimated_cost
FROM `agent_telemetry.turns`
WHERE timestamp >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)
GROUP BY tenant_id
ORDER BY total_tokens DESC;

2. Tool Failure & Latency Analysis

Identify tools with high failure rates or unexpected latency spikes over the past week:

SELECT
  tool_name,
  COUNT(*) AS total_calls,
  SUM(CASE WHEN success = FALSE THEN 1 ELSE 0 END) AS failures,
  ROUND(AVG(duration_ms), 0) AS avg_duration_ms
FROM `agent_telemetry.tool_calls`
WHERE timestamp >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY)
GROUP BY tool_name
HAVING failures > 0
ORDER BY failures DESC;

3. Pipeline Latency Percentiles

Analyze latency distribution (P50, P95, P99) across different agent roles to detect edge-case bottlenecks:

SELECT
  agent_name,
  APPROX_QUANTILES(total_latency_ms, 100)[OFFSET(50)] AS p50_latency,
  APPROX_QUANTILES(total_latency_ms, 100)[OFFSET(95)] AS p95_latency,
  APPROX_QUANTILES(total_latency_ms, 100)[OFFSET(99)] AS p99_latency
FROM `agent_telemetry.turns`
WHERE timestamp >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 DAY)
GROUP BY agent_name;

Setting Up the Streaming Pipeline

Setting up a telemetry pipeline requires three steps:

1. Instrument your harness: Hook into the agent completion callbacks to format telemetry events after each turn.
2. Stream to BigQuery: Write events asynchronously using the BigQuery Storage Write API or via GCP Pub/Sub for decoupling.
3. Visualize & Alert: Build dashboards in Looker Studio or set up scheduled queries for real-time monitoring and alerting.

Python Instrumentation Example

Here is how you can stream telemetry directly using the BigQuery Python SDK:

from datetime import datetime
from google.cloud import bigquery

client = bigquery.Client()
table = client.get_table("project.agent_telemetry.turns")

def emit_turn_event(session_id: str, agent_name: str, metrics: object, tenant_id: str = "default"):
    """Emits agent execution telemetry directly to BigQuery."""
    row = {
        "session_id": session_id,
        "agent_name": agent_name,
        "tenant_id": tenant_id,
        "input_tokens": metrics.input_tokens,
        "output_tokens": metrics.output_tokens,
        "total_latency_ms": metrics.latency_ms,
        "timestamp": datetime.utcnow().isoformat()
    }
    errors = client.insert_rows_json(table, [row])
    if errors:
        print(f"Failed to insert telemetry rows: {errors}")

Beyond Basics: Quality Tracking & Trend Analysis

Historical telemetry stored in BigQuery enables deep correlation analysis when updating models, prompt instructions, or tool definitions:

• Model Regressions: Did upgrading to a newer foundation model version increase tool call errors?
• Prompt Drift: Did adding extra system instructions increase average prompt tokens and end-to-end latency?
• Tenant Outliers: Is a specific customer tenant triggering anomalous tool retry loops?

Key Takeaway

While trace-level debugging solves individual execution errors, BigQuery fleet analytics reveals systemic performance, cost, and reliability trends. Instrumenting your agent harness early ensures that historical data is readily available for capacity planning, cost management, and quality control as your agent deployments scale.