Pagination and Timeouts in Long-Running Tool Responses
Huge or slow tool results break agent loops. Learn to return cursors as tool output, hand the model async job handles to poll, and truncate without losing structure.
Published on • November 16, 2026
AI Assistant

Your agent works beautifully in the demo. Then someone points it at the production data source, the model calls generate_quarterly_report(), and you get back 400 KB of markdown. One of three bad things happens: the provider rejects the request for exceeding the context limit, the tool call hits a gateway timeout and the run dies, or — worst — it “works” and the model drowns in the dump and starts hallucinating rows it never saw.
The same failure shows up in slow tools: a 90-second ETL job, a big retrieval over a 200k-document index, a report builder that renders charts. Your agent loop is synchronous; the model waits; everything upstream times out.
In this post, you will learn how to design tool responses that fit the agent loop instead of fighting it: cursor pagination returned as tool output, page tokens the model follows across turns, timeboxed async job handles the model polls, and structure-preserving truncation — all in runnable Python, with a look at how LlamaIndex query engines and workflows expose streamed, citable sources (LlamaIndex docs).
Prerequisites
- Python 3.10+ and a virtual environment.
- An agent framework that supports function tools. The examples are plain Python functions with type hints and docstrings, so they drop into LlamaIndex (
FunctionTool), LangChain (@tool), or your own loop unchanged. - Comfort with tool calling: the model emits a tool call, your runtime executes it, you send the result back as a tool message.
Why a Giant Tool Result Breaks the Loop
An agent turn is a request/response cycle. Every tool result is re-sent to the model on the next call, and often re-sent on every call after that until it is evicted. That means a 400 KB result is not a one-time cost — it is a recurring tax paid on every subsequent turn.
Three rules follow:
- A tool result should be a decision point, not a data dump. Give the model enough to decide what to do next, plus a way to ask for more.
- A tool call must complete inside your loop’s timeout budget. If the work takes longer than that budget, it is not one tool call — it is a job.
- When you must cut data, cut it without breaking its shape. A half-JSON blob is worse than no blob.
Everything below is a variation on those three rules.
Cursor Pagination Returned as Tool Output
Follow the same model MCP uses for its own list operations: an opaque cursor, not numbered pages, with a server-chosen page size and a missing nextCursor meaning “end of results.” The MCP pagination spec is explicit that clients must treat cursors as opaque tokens — never parse them, never assume a page size. Borrow that discipline for your own tools.
import json
import time
import uuid
from dataclasses import dataclass, field
PAGE_SIZE = 20
@dataclass
class ReportStore:
rows: list[dict] = field(default_factory=list)
cursors: dict[str, int] = field(default_factory=dict)
STORE = ReportStore(
rows=[
{"id": i, "region": f"region-{i % 7}", "revenue": 1000 + i * 13}
for i in range(1, 501)
]
)
def search_report(cursor: str | None = None, page_size: int = PAGE_SIZE) -> str:
"""Search the quarterly revenue report, one page at a time.
Returns JSON with `items`, `next_cursor`, and `total_items`.
Call again with `cursor` set to `next_cursor` for the next page;
a null `next_cursor` means there are no more results.
"""
page_size = max(1, min(page_size, PAGE_SIZE))
if cursor is None:
offset = 0
else:
offset = STORE.cursors.get(cursor)
if offset is None:
return json.dumps({
"error": f"Unknown cursor '{cursor}'. Restart with cursor omitted.",
})
page = STORE.rows[offset:offset + page_size]
next_offset = offset + len(page)
next_cursor = None
if next_offset < len(STORE.rows):
next_cursor = uuid.uuid4().hex
STORE.cursors[next_cursor] = next_offset
return json.dumps({
"items": page,
"count": len(page),
"total_items": len(STORE.rows),
"next_cursor": next_cursor,
"hint": "Pass next_cursor as cursor to continue. null means done.",
}, default=str)
Three deliberate choices: the cursor is opaque (uuid4().hex) so the model only echoes it while offsets stay server-side; the envelope is stable — items, count, next_cursor, hint on every page, so the shape holds across turns; and errors are recoverable, with an unknown cursor returning a plain string the model can act on rather than an exception that kills the loop. That last one mirrors MCP’s split between protocol errors and tool execution errors: the latter exist for the model to self-correct with.
Teaching the Model to Iterate
Pagination only works if the model knows to keep going. Do not rely on vibes; write it into the tool description, because the description is the only part the model reads before choosing the tool:
search_report: Search the quarterly revenue report.
IMPORTANT: results are paginated. Each response contains at most 20 rows
and a `next_cursor` field. If `next_cursor` is not null, call this tool
again with that cursor until it is null, then summarize. Never claim the
result set is complete while next_cursor is non-null.
Then add the stop condition to your system prompt:
When a tool returns `next_cursor`, you must continue calling that tool
with the returned cursor before answering. Prefer 3-5 pages per request;
if the full result would exceed that, tell the user how many total items
exist and ask whether to continue.
That second sentence matters. Without a stop condition, a diligent model will happily page through 500 rows — 25 turns of pure waste. Bound the loop in the prompt, and let total_items tell the user how much is left.
Timeboxing Long Jobs with an Async Job Handle
Pagination solves volume. It does not solve latency. When work genuinely takes longer than your tool timeout (say 30 seconds), split the operation into start and poll:
JOBS: dict[str, dict] = {}
MAX_SYNC_SECONDS = 20
def start_etl_job(dataset: str, since: str = "2026-01-01") -> str:
"""Start a long-running ETL extraction and return a job_id immediately.
Poll with poll_etl_job(job_id). Jobs expire 1 hour after finishing;
polling an expired job returns an error saying so.
"""
job_id = f"job_{uuid.uuid4().hex[:12]}"
JOBS[job_id] = {
"status": "running",
"dataset": dataset,
"started_at": time.time(),
"progress": 0.0,
"result": None,
"expires_at": None,
}
# In production: enqueue to Celery/RQ/Arq and return the id.
_simulate_background_work(job_id, dataset, since)
return json.dumps({
"job_id": job_id,
"status": "running",
"next_action": "Call poll_etl_job with this job_id after a short wait.",
})
def poll_etl_job(job_id: str) -> str:
"""Status of a job started with start_etl_job: status, progress, result."""
job = JOBS.get(job_id)
if job is None:
return json.dumps({
"error": f"Job '{job_id}' not found or expired.",
"next_action": "Call start_etl_job again to restart the work.",
})
return json.dumps({
"job_id": job_id,
"status": job["status"],
"progress": round(job["progress"], 2),
"result": job["result"] if job["status"] == "done" else None,
})
This is exactly the pattern the MCP tools spec calls stateful tools: the protocol has no session, so a server that needs state across calls “should do so by returning an explicit handle from a creation tool and accepting that handle as an argument on subsequent calls.” Its handle guidance is worth internalizing — keep ids opaque, treat a handle as a name rather than a capability, state the expiry rule in the tool description, and return a tool execution error for a dead handle so the model recovers by starting fresh.
The polling tool never blocks: each call returns in milliseconds, and the model decides whether to wait or poll again. You have converted a 90-second timeout into a series of cheap turns.
Truncation Policies That Preserve Structure
Sometimes the answer genuinely is “return less.” Truncate, but truncate structurally: keep the envelope, keep the counts, keep head and tail, and mark clearly what was cut.
def truncate_structured(rows: list[dict], keep_head: int = 10,
keep_tail: int = 5, max_chars: int = 4000) -> str:
"""Return a structure-preserving view of rows that fits max_chars."""
if len(json.dumps(rows, default=str)) <= max_chars:
payload = {"items": rows, "truncated": False}
return json.dumps(payload, default=str)
middle_dropped = max(0, len(rows) - keep_head - keep_tail)
payload = {
"items": rows[:keep_head] + rows[-keep_tail:],
"truncated": True,
"total_items": len(rows),
"items_omitted": middle_dropped,
"omission": (
f"[{middle_dropped} middle rows omitted] "
"use search_report with a cursor to page through them"
),
}
return json.dumps(payload, default=str)
Why head and tail? Head rows carry the schema and the strongest matches; tail rows carry totals and the “last page” markers models look for. The omission field is a real string in the output, not a comment — models reliably respond to explicit markers of missing data, and it hands them the cursor as the next action.
How LlamaIndex Exposes Sources and Long-Running Work
The same principles show up in the framework layer. In the LlamaIndex docs, a query engine “takes in a natural language query, and returns a rich response,” and every response carries response.source_nodes — the retrieved chunks behind the answer. CitationQueryEngine goes further, labeling each chunk as a numbered Source N inline: instead of dumping all nodes, you hand the model addressable references it can ask about individually.
For slowness, LlamaIndex solves it at the workflow layer rather than the tool layer:
# Workflows take an explicit timeout — fail fast instead of hanging a turn.
workflow = MyFlow(timeout=60)
# Long runs become handlers you can stream and resume.
handler = workflow.run(user_msg="Build the report")
async for event in handler.stream_events():
print(event.type, event.value)
Deployed as a service, the WorkflowClient goes further: run_workflow_nowait() returns a handler_id, and get_workflow_events(handler_id, after_sequence=...) takes an explicit cursor — -1 replays everything, "now" skips history, an integer resumes from the last event you saw. A job handle and a pagination cursor in one API: the same two primitives this post built by hand.
Putting It All Together
A single user request — “Summarize revenue for all regions and flag outliers” — now runs like this:
user → Summarize revenue for all regions and flag outliers.
assistant → tool_call: search_report()
tool → {"items": [20 rows], "total_items": 500, "next_cursor": "9f3a…"}
assistant → tool_call: search_report(cursor="9f3a…")
tool → {"items": [20 rows], "next_cursor": "c71d…"}
assistant → "I've read 40 of 500 rows. Continue, or should I sample?"
user → Continue.
assistant → tool_call: start_etl_job(dataset="regional_revenue")
tool → {"job_id": "job_1a9f3c77e0b2", "status": "running"}
assistant → tool_call: poll_etl_job(job_id="job_1a9f3c77e0b2")
tool → {"status": "done", "progress": 1.0, "result": {"rows": 441}}
assistant → "ETL complete: 441 rows. Outliers: region-3 (+38%), region-6 (-22%)."
Every tool call returned in milliseconds, the model carried the cursor and the job id across turns, and nothing ever landed in context as an unbounded blob — token spend stayed flat even though the underlying result set was 500 rows plus an ETL artifact.
Conclusion and Next Steps
You started with a loop that died on big, slow tools and ended with three reusable contracts: an opaque-cursor page envelope the model echoes until next_cursor is null, a job handle split so no call ever blocks, and a truncation shape that keeps counts and markers when you must cut. LlamaIndex and MCP have converged on the same primitives — cursors, handles, citable sources — a good sign you are not over-engineering.
Homework:
- Add a
max_pagesguard tosearch_reportso a runaway loop returns a “page limit reached” tool error instead of streaming forever. - Persist
STORE.cursorsandJOBSto Redis, with TTLs matching the expiry text in your tool descriptions. - Instrument it: log per-call latency, page depth, and token size of each tool result — the numbers will tell you which other tools need the same treatment.