A Year of Agents: What We Learned Building Production Agent Systems

After spending over a year deploying AI agents in production — handling everything from customer support to code review to data pipeline orchestration — we have accumulated a significant body of knowledge about what works, what fails, and what most teams get wrong in their first attempt. This post distills those hard-won lessons into actionable guidance for teams building their own agent systems.

Why This Matter

The gap between a working demo and a production-ready agent system is enormous. Demos hide the hard problems: reliability at scale, cost control, observability, and graceful degradation. Teams that skip straight to building without understanding these challenges often abandon their agent initiatives after costly failures.

The Agent Development Kit (ADK) from Google has emerged as a leading framework for production agent development. It provides built-in solutions for many of the problems teams encounter, but understanding the underlying principles is essential for any team building agents, regardless of framework.

Lesson 1: Agents Are Distributed Systems

The biggest misconception about agents is that they are simple request-response systems. They are not. A production agent is a distributed system with asynchronous tool calls, parallel sub-agents, state management, and complex error recovery paths.

from google.adk import Agent
from google.adk.tools import FunctionTool
import asyncio

# Define tools that may fail, timeout, or return unexpected results
@FunctionTool
async def fetch_customer_data(customer_id: str) -> dict:
    """Fetch customer data from CRM. May timeout under load."""
    try:
        async with aiohttp.ClientSession() as session:
            async with session.get(
                f"https://crm.internal/api/customers/{customer_id}",
                timeout=aiohttp.ClientTimeout(total=5)
            ) as resp:
                return await resp.json()
    except asyncio.TimeoutError:
        return {"error": "CRM_TIMEOUT", "retryable": True}
    except Exception as e:
        return {"error": "CRM_UNAVAILABLE", "retryable": False}

# Build agent with explicit error handling
support_agent = Agent(
    name="support_agent",
    model="gemini-2.0-flash",
    instruction="""You are a customer support agent.
    
When fetching customer data fails:
- If retryable, inform the customer you are checking again
- If not retryable, ask the customer for the information directly
- Never guess customer data""",
    tools=[fetch_customer_data],
)

Key takeaway: Treat every tool call as a potential failure point. Design your agents to handle timeouts, partial results, and service outages gracefully.

Lesson 2: Evaluation Is Not Optional

The number one reason agent projects fail in production is insufficient evaluation. Without a robust eval suite, you cannot catch regressions, measure quality, or justify continued investment.

from google.adk.evaluation import EvalRunner, EvalCase

# Define evaluation cases that cover real production scenarios
eval_cases = [
    EvalCase(
        name="happy_path_refund",
        user_message="I want a refund for order #12345",
        expected_tool_calls=["lookup_order", "process_refund"],
        expected_output_contains="refund processed",
        max_latency_ms=3000,
    ),
    EvalCase(
        name="ambiguous_request",
        user_message="The thing I bought is broken",
        expected_tool_calls=["search_orders", "ask_clarification"],
        expected_output_contains="Could you provide",
        max_latency_ms=3000,
    ),
    EvalCase(
        name="tool_failure_recovery",
        user_message="What's my account balance?",
        mock_tool_responses={"get_balance": {"error": "SERVICE_DOWN"}},
        expected_output_contains="unable to retrieve",
        max_latency_ms=5000,
    ),
]

eval_runner = EvalRunner(
    agent=support_agent,
    cases=eval_cases,
    metrics=["accuracy", "latency", "tool_accuracy", "cost_per_call"],
)

results = eval_runner.run()
print(f"Pass rate: {results.pass_rate:.1%}")
print(f"Average latency: {results.avg_latency_ms:.0f}ms")
print(f"Average cost: ${results.avg_cost:.4f}")

Run evaluations on every pull request. Track results over time. Set quality gates that block deployments when metrics degrade.

Lesson 3: Cost Management Is a First-Class Concern

Agents can burn through API budgets faster than you expect. A single agent run might trigger 5-20 model calls and multiple tool invocations. At scale, costs compound quickly.

from google.adk import Agent, RunConfig
from google.adk.models.lite_llm import LiteLlm

# Configure cost-aware routing
cost_router = LiteLlm(
    model="gemini-2.0-flash",  # Default to fast, cheap model
    fallbacks=[
        {"model": "gemini-2.5-pro", "condition": "complex_reasoning"},
    ]
)

run_config = RunConfig(
    max_tool_calls=10,
    max_model_calls=5,
    token_budget=50000,
    cost_budget_usd=0.10,
    timeout_seconds=30,
)

agent = Agent(
    name="cost_aware_agent",
    model=cost_router,
    run_config=run_config,
    instruction="Complete the task efficiently. Prefer quick answers over exhaustive searches."
)

Key strategies:

• Set token budgets per agent run
• Use faster, cheaper models as the default and escalate only when needed
• Cache tool results aggressively
• Monitor cost per task and set alerts at budget thresholds

Lesson 4: Observability Makes or Breaks Debugging

When an agent makes a wrong decision in production, you need to trace exactly why. Without detailed logging of every model call, tool invocation, and decision point, debugging is guesswork.

import logging
from google.adk.evaluation.traces import TraceCollector

# Configure tracing for production
trace_collector = TraceCollector(
    export_to="cloud_trace",  # or your preferred backend
    sample_rate=1.0,  # Start with 100%, reduce later
)

agent = Agent(
    name="traceable_agent",
    model="gemini-2.0-flash",
    instruction="You are a helpful assistant.",
    tools=[my_tools],
    trace_collector=trace_collector,
)

# Each trace captures:
# - Full conversation history
# - Every tool call and its result
# - Model reasoning at each step
# - Token usage and latency per call
# - Final output

Invest in observability from day one. You will need it the first time something goes wrong at 2 AM.

Lesson 5: Humans Are Part of the Loop

Purely autonomous agents are rare in production. Most successful deployments use human-in-the-loop patterns for critical decisions, ambiguous situations, or high-stakes actions.

from google.adk.workflows import WorkflowAgent, HumanInputGate

# Define a workflow with human approval for sensitive actions
workflow = WorkflowAgent(
    name="expense_approval",
    model="gemini-2.0-flash",
)

workflow.add_step(
    name="categorize_expense",
    instruction="Categorize this expense and determine the approval threshold."
)

workflow.add_step(
    name="human_approval_gate",
    gate=HumanInputGate(
        prompt="This expense requires manager approval. Approve?",
        approvers=["finance_team"],
        timeout_hours=24,
        default_on_timeout="reject",
    )
)

workflow.add_step(
    name="process_approved_expense",
    instruction="Process the approved expense and notify the employee."
)

Design your agents knowing that humans will intervene. Make intervention easy, fast, and informative.

Best Practices

1. Start with a narrow scope: Build one agent for one task. Get it to production quality before expanding.

2. Write eval cases before you write agent code: This forces you to define what “good” looks like upfront.

3. Log everything in production: You cannot debug what you cannot see.

4. Set hard limits on agent behavior: Max tool calls, max tokens, max cost per run. Agents without limits are liability.

5. Test failure modes deliberately: Tool timeouts, malformed responses, unexpected inputs. Your agent should handle all of them.

6. Version your agent configurations: Treat prompts, tool definitions, and run configs as code.

Common Pitfalls

• Over-automating too fast: Start with human-in-the-loop, then reduce human involvement as you build confidence.
• Ignoring latency: Users will not wait 30 seconds for an agent response. Profile and optimize your slowest tool calls.
• Treating all tasks equally: Simple classification tasks should use small models. Complex reasoning needs larger models. Route accordingly.
• Skipping load testing: An agent that works for one user may collapse under concurrent load. Test with realistic traffic patterns.

Conclusion

Building production agent systems is fundamentally different from building demos. The challenges are distributed systems challenges: reliability, observability, cost management, and graceful degradation. The teams that succeed treat agents as production software from day one — with proper testing, monitoring, and incremental rollout.

The tools and frameworks are maturing rapidly. Google’s ADK, with its built-in evaluation, tracing, and workflow capabilities, has significantly lowered the barrier to production-ready agents. But the principles remain the same: start narrow, evaluate rigorously, observe deeply, and keep humans in the loop where it matters.

Next steps:

• Set up an evaluation suite for your existing agent using ADK’s evaluation tools
• Implement structured tracing on your next agent build
• Define hard limits (tokens, cost, tool calls) for every agent in production
• Review your agent’s error handling paths — they probably need work