Testing Tools in Isolation: Unit Tests Before Agent Evals
Agent evals are slow and expensive. Cover the testing pyramid first: pytest for tool functions, contract tests for schemas, property tests for parsers, and fake models for loop tests.
Published on • November 16, 2026
AI Assistant

You change one line in a tool’s docstring. CI is green. Two hours later the agent in production starts calling search_orders with an empty string in a loop and burns forty dollars before anyone notices.
The problem isn’t that evals failed — it’s that you never gave them a chance. Evals measure emergent behavior in a nondeterministic system; they are the wrong instrument for a missing None check. Most agent bugs are ordinary software bugs wearing a model-shaped hat, and ordinary software bugs die in milliseconds under pytest.
In this post, you will learn how to build the base of the testing pyramid for agents using LangChain/LCEL-style tools, pytest for schema validation, error paths, and mocked APIs, contract tests that assert the schema the model is shown matches the code that runs, property-based tests for parsers, and fake/stub chat models for deterministic loop tests — plus how each layer maps to a CI stage. The LangChain testing docs are the reference for the fake-model layer (the current mirror lives at docs.langchain.com).
Prerequisites
- Python 3.10+ with
pip install langchain langgraph pytest hypothesis respx httpx. - An OpenAI/Anthropic/etc. key is not required — the whole point is that these tests run offline.
- A tool or two of your own written with
@tool(LangChain) or plain functions with type hints. - Familiarity with pytest fixtures and
monkeypatch.
The Testing Pyramid for Agents
The LangChain docs split agent testing into three tiers: unit tests (“small, deterministic pieces of your agent in isolation using in-memory fakes”), integration tests (“real network calls to confirm that components work together”), and evals (“evaluators to assess your agent’s execution trajectory”). Their note that agentic apps “lean more on integration” describes where people end up, not where to start.
The pyramid for agents looks like this:
/ LLM evals \ slow, costly, nondeterministic — weekly
/ integration \ real APIs, schema + latency checks — nightly
/ contract tests \ model-visible schema == real schema — per PR
/ property tests \ parsers survive weird inputs — per PR
/ tool unit tests \ logic, errors, edge cases — per commit, <1s
Everything above a layer inherits the defects below it. If a unit test would have caught it, an eval catching it is a waste of a run.
Level 1: Unit Test the Tool Function
Start with the tool as a plain function. Here is the subject — an order lookup that parses a messy upstream API response:
# tools/orders.py
import json
import httpx
from langchain.tools import tool
class ToolInputError(ValueError):
"""Raised for bad input; the agent runtime turns this into a tool error
the model can read and self-correct from."""
def parse_order_payload(raw: str) -> dict:
"""Parse the vendor's order payload, tolerating their stray trailing comma."""
return json.loads(raw.replace(",}", "}").replace(",]", "]"))
@tool
def lookup_order(order_id: str) -> str:
"""Look up an order by id. `order_id` must look like ORD-12345.
Returns JSON with id, status, and total. Raises ToolInputError with an
actionable message if the id is malformed or the order is missing.
"""
if not order_id.startswith("ORD-") or not order_id[4:].isdigit():
raise ToolInputError(
f"Invalid order_id '{order_id}'. Expected format: ORD-12345."
)
resp = httpx.get(f"https://api.example.com/orders/{order_id}", timeout=5.0)
if resp.status_code == 404:
raise ToolInputError(f"Order '{order_id}' not found. Check the id.")
resp.raise_for_status()
order = parse_order_payload(resp.text)
return json.dumps({
"id": order["id"],
"status": order["status"],
"total": order["total"],
})
Now the tests, one concern per test:
# tests/test_lookup_order.py
import json
import httpx
import pytest
import respx
from langchain_core.tools import ToolInvocation
from tools.orders import ToolInputError, lookup_order, parse_order_payload
# --- happy path ---------------------------------------------------------
@respx.mock
def test_lookup_order_happy_path():
respx.get("https://api.example.com/orders/ORD-12345").mock(
return_value=httpx.Response(200, text='{"id":"ORD-12345","status":"shipped","total":42.00}')
)
result = lookup_order.invoke({"order_id": "ORD-12345"})
payload = json.loads(result)
assert payload["status"] == "shipped"
assert payload["total"] == 42.00
# --- schema / validation ----------------------------------------------
def test_rejects_malformed_id_with_actionable_message():
with pytest.raises(ToolInputError) as exc:
lookup_order.invoke({"order_id": "banana"})
# The message goes to the model — it must tell the model what to do instead.
assert "ORD-12345" in str(exc.value)
# --- error paths --------------------------------------------------------
@respx.mock
def test_404_becomes_recoverable_tool_error():
respx.get("https://api.example.com/orders/ORD-99999").mock(
return_value=httpx.Response(404)
)
with pytest.raises(ToolInputError, match="not found"):
lookup_order.invoke({"order_id": "ORD-99999"})
@respx.mock
def test_5xx_propagates_as_infrastructure_failure():
respx.get("https://api.example.com/orders/ORD-12345").mock(
return_value=httpx.Response(500)
)
with pytest.raises(httpx.HTTPStatusError):
lookup_order.invoke({"order_id": "ORD-12345"})
# --- edge cases ---------------------------------------------------------
def test_parser_tolerates_vendor_trailing_commas():
assert parse_order_payload('{"id":"ORD-1","total":1.50,}')["id"] == "ORD-1"
def test_missing_optional_fields_do_not_crash():
# A KeyError here would surface as an opaque tool failure in production.
with pytest.raises(KeyError):
parse_order_payload('{"id":"ORD-1"}') # documents current behavior
The respx mock keeps the network out entirely — no keys, no flakiness, under a second. Note what these catch that an eval never reliably will: the exact wording of a self-correction message, the 404-vs-500 distinction (recoverable vs. infra), and parser edge cases a model might paper over once and never trigger again.
One subtlety: assert on the string the model will read, not just the exception type. assert "ORD-12345" in str(exc.value) is asserting on the model’s instructions.
Level 2: Contract Tests — Schema vs. What the Model Is Told
Your tool’s code and the schema the runtime advertises to the model can drift independently — someone renames a parameter or edits the description without touching the registry. Contract tests assert that what the model is shown is exactly what will execute:
# tests/test_contract.py
import jsonschema
from langchain.tools import tool
@tool("sum_totals")
def sum_totals(values: list[float], round_to: int = 2) -> str:
"""Sum a list of numbers, rounded."""
return str(round(sum(values), round_to))
def test_tool_name_and_description_match_registry():
# Registry entry written by hand (or exported from your MCP catalog).
registry_entry = {
"name": "sum_totals",
"description": "Sum a list of numbers, rounded.",
}
assert sum_totals.name == registry_entry["name"]
assert sum_totals.description == registry_entry["description"]
def test_advertised_schema_accepts_declared_arguments():
schema = sum_totals.args_schema.model_json_schema()
# What the model is told:
assert set(schema["properties"]) == {"values", "round_to"}
assert schema["required"] == ["values"]
# And the schema itself must be valid JSON Schema — some providers
# reject the whole tool list if it isn't.
jsonschema.Draft202012Validator.check_schema(schema)
def test_schema_matches_actual_callable_signature():
import inspect
params = inspect.signature(sum_totals.func).parameters
assert list(params) == ["values", "round_to"]
assert params["round_to"].default == 2
Run this against every tool with a parametrized test over your registry (from post 2 of this series) and you have fleet-wide protection: a breaking schema change fails CI before it reaches a model. If you expose tools over MCP, validate inputSchema and outputSchema the same way — the spec requires outputSchema conformance from servers and asks clients to validate against it.
Level 3: Property-Based Tests for Parsers
Tools that parse model-generated or vendor-generated strings are where subtle bugs live. hypothesis explores the input space for you:
# tests/test_parsers_property.py
import json
from hypothesis import given, settings, strategies as st
from tools.orders import parse_order_payload
json_text = st.recursive(
st.none() | st.booleans() | st.integers() | st.text(),
lambda children: st.lists(children) | st.dictionaries(st.text(), children),
max_leaves=50,
)
@given(json_text)
@settings(max_examples=200)
def test_parser_round_trips_valid_json(value):
"""Anything json.dumps accepts must survive the parser unchanged."""
assert parse_order_payload(json.dumps(value)) == value
@given(st.text())
@settings(max_examples=200)
def test_parser_never_hangs_or_segfaults_on_arbitrary_text(value):
"""The parser must either return a value or raise JSONDecodeError —
never IndexError, RecursionError, or a bare Exception."""
try:
parse_order_payload(value)
except json.JSONDecodeError:
pass
The second test is the important one. Your agent runtime probably catches Exception and reports it to the model; if a weird input raises RecursionError or returns None, that contract breaks in a way no example-based test will find. Property tests are cheap, deterministic, and belong in the same PR stage as unit tests.
Level 4: Deterministic Loop Tests With a Fake Model
Once the tool is solid, test the loop — still with zero API calls. LangChain ships GenericFakeChatModel, which returns scripted responses from an iterator, one per invocation. Because create_agent calls bind_tools when you pass tools, subclass it to return itself, exactly as the unit-testing guide shows:
# tests/test_agent_loop.py
from langchain.agents import create_agent
from langchain.messages import AIMessage, HumanMessage, ToolCall
from langchain_core.language_models.fake_chat_models import GenericFakeChatModel
from tools.orders import lookup_order
class FakeToolCallingModel(GenericFakeChatModel):
"""GenericFakeChatModel has no bind_tools; create_agent needs it."""
def bind_tools(self, tools, **kwargs):
return self
def make_model():
return FakeToolCallingModel(messages=iter([
# Turn 1: ask for the tool
AIMessage(content="", tool_calls=[
ToolCall(name="lookup_order", args={"order_id": "ORD-12345"}, id="call_1")
]),
# Turn 2: answer with the tool result
AIMessage(content="Order ORD-12345 has shipped and totals 42.00."),
]))
def test_agent_calls_tool_then_answers():
agent = create_agent(make_model(), tools=[lookup_order])
result = agent.invoke({"messages": [HumanMessage(content="Where's my order?")]})
calls = [m for m in result["messages"] if isinstance(m, AIMessage) and m.tool_calls]
assert len(calls) == 1
assert calls[0].tool_calls[0]["name"] == "lookup_order"
assert "shipped" in result["messages"][-1].text
def test_tool_error_is_visible_to_the_model():
"""A failing tool doesn't raise — it comes back as a ToolMessage the model reads."""
model = FakeToolCallingModel(messages=iter([
AIMessage(content="", tool_calls=[
ToolCall(name="lookup_order", args={"order_id": "banana"}, id="call_1")
]),
AIMessage(content="That id looks wrong — the format is ORD-12345."),
]))
agent = create_agent(model, tools=[lookup_order])
result = agent.invoke({"messages": [HumanMessage(content="Look up banana")]})
tool_messages = [m for m in result["messages"] if m.type == "tool"]
assert tool_messages and "ORD-12345" in tool_messages[0].text
Two gotchas from the docs: keep the fixture at default function scope, because an exhausted iterator cannot be reused across tests; and for persistence across turns, add an InMemorySaver checkpointer instead of mocking the graph. These catch loop bugs — wrong tool selected, tool result never reaching the model, malformed tool-call ids — in about 30 milliseconds each.
Mapping the Layers to CI Stages
| Stage | What runs | Catches | Budget |
|---|---|---|---|
| Per commit | Tool unit tests + property tests | Logic bugs, bad messages, parser crashes | < 5 s |
| PR | Contract tests + fake-model loop tests | Schema drift, loop regressions | < 30 s |
| Nightly | Integration tests with a real model API | Credentials, provider schema changes, latency | minutes |
| Weekly | Trajectory evals (match + LLM judge) | Strategy quality, prompt regressions | dollars |
The ordering matters: if a PR-stage contract test fails, there is no point paying for the nightly run. And when an eval does fail, you have a clean bisect path — if the loop test passes but the eval fails, the defect is genuinely in model behavior, the only case where spending money on more evals is the right response.
Putting It All Together
A change to lookup_order’s docstring now flows like this:
1. Developer edits tools/orders.py, runs `pytest -q`
→ 6 unit tests + 2 property tests pass in 0.4s
→ contract test fails: description no longer matches registry/tools.yaml
2. They update the registry entry in the same PR (schema hash changes,
re-approval triggers for a high-risk tool).
3. Fake-model loop test confirms the model still reaches for the right tool
and a 404 surfaces as an actionable ToolMessage.
4. Merge. Nightly integration runs one real `lookup_order` call; the weekly
eval checks the agent still resolves "where's my order?".
Cost for the first three steps: zero tokens, zero network calls, under a second. The eval still runs — it just no longer carries the burden of catching bugs a fixture could have caught for free.
Conclusion and Next Steps
You built four layers beneath the eval: unit tests for tool logic and the messages models read, contract tests tying advertised schemas to real signatures, property tests hardening parsers against hostile input, and fake-model loop tests exercising the agent graph deterministically. Each runs offline in milliseconds, and together they turn your evals from a bug detector into what they are good at: measuring behavior.
Homework:
- Parametrize the contract test over every tool in your registry; measure how many drift today. The answer is rarely zero.
- Add a
FakeToolCallingModeltest scripting three turns (call → error → corrected call) to prove self-correction works. - Track flakiness: any test failing more than 1% of CI runs without a code change gets quarantined and fixed — flaky tests train the team to ignore red.