Streaming Tool Results Back into the Agent Loop

The tool returns in five minutes. For four minutes and fifty-nine seconds, nothing happens on screen: no spinner text, no partial result, no step indicator. The user assumes it hung, hits refresh, and you now have two runs of the expensive job. Meanwhile your timeout fires at 60 seconds, the retry fires at 90, and the third copy of the job starts.

Long-running tools are normal in agent systems — research sweeps, database migrations, video transcoding, batch code fixes — and the standard trick of token-streaming the LLM’s text does nothing for them, because the silence happens between model messages, while a tool is executing. The fix is to treat the tool’s execution itself as a stream: intermediate events that flow from inside the tool back out through the agent loop to the UI.

This post uses LangGraph because its streaming model is explicit about exactly this problem, but the shape — a stream channel distinct from state updates, emitted from inside tool execution — is the same in LlamaIndex workflows and in hand-rolled loops.

The problem, precisely

Three failures come from a silent long tool:

1. UX: the interface can’t distinguish “working” from “stuck”, so users cancel valid runs or duplicate them.
2. Operations: you can’t measure progress, so you can’t alert on a tool that is 3× slower than yesterday until the whole run fails.
3. Resumability: when the run is interrupted, you have no idea where it was.

Token streaming solves none of these, because tokens come from the model and the model is waiting.

Streaming as intermediate events

LangGraph exposes graph execution through stream modes: values, updates, messages, custom, checkpoints, tasks, debug. Each answers a different question:

Mode	What it carries	Useful for
updates	State changes after each step, with the node name	”Which step am I on?” timelines
values	Full state snapshot after each step	Re-rendering from state, resumable clients
messages	(message_chunk, metadata) token tuples from LLM calls	Streaming model text
custom	Arbitrary payloads emitted from nodes and tools	Tool progress, sub-steps, log lines
checkpoints / tasks	Checkpoint and task lifecycle events (needs a checkpointer)	Durability, debugging stalls

Passing version="v2" (LangGraph 1.1+) normalizes everything into one StreamPart shape — {"type": ..., "ns": ..., "data": ...} — so one consumer handles every mode instead of special-casing tuple formats:

for part in graph.astream(
    inputs,
    stream_mode=["updates", "custom"],
    version="v2",
):
    if part["type"] == "updates":
        for node_name, delta in part["data"].items():
            await ui.step_started(node_name, delta)
    elif part["type"] == "custom":
        await ui.progress(part["data"])

The mode list is not cosmetic: custom events only flow if you ask for them. Requesting ["updates", "custom"] is the minimum for “show the step list and the progress inside each step.”

Emitting progress from inside a tool

The mechanism is get_stream_writer, available inside nodes and tools, which pushes a payload into the custom channel:

from langchain.tools import tool
from langgraph.config import get_stream_writer


@tool
def backfill_records(batch_size: int = 500) -> str:
    """Rewrite legacy records into the new schema, in batches."""
    writer = get_stream_writer()
    total = count_pending()

    done = 0
    while done < total:
        process_batch(limit=batch_size)
        done = min(done + batch_size, total)
        writer({"kind": "progress", "tool": "backfill_records",
                "done": done, "total": total})

    writer({"kind": "progress", "tool": "backfill_records",
            "done": total, "total": total, "status": "complete"})
    return f"backfilled {total} records"

Two practical rules:

• Emit at a human cadence, not per iteration. A batch loop writing 20,000 events floods the consumer and the transport. Throttle inside the tool (every N items or every 500 ms) — the tool owns that policy because only the tool knows its granularity.
• Keep payloads small and structured. {"kind", "tool", "done", "total"} is renderable by any client. Dumping raw records into the stream turns your progress channel into a data-exfiltration path with extra steps.

One version caveat: on Python < 3.11 in async code, get_stream_writer() can’t reach the ambient context, so declare writer: StreamWriter as a parameter of the node/tool and LangGraph passes it in. On 3.11+ the contextvar works as documented.

Tool call chunks and partial model output

Progress events are half the story; the other half is what the model is doing between tool calls:

• stream_mode="messages" yields (message_chunk, metadata) for every LLM invocation in the run — including, notably, invocations made with .invoke() rather than .stream(). Filter with metadata["langgraph_node"] to render only the agent node’s tokens, or by tags to separate multiple models.
• Wrapping an agent as a node hides its tokens. If you add a compiled agent (for example from create_agent) as a node in a parent graph, the parent’s messages stream will not emit the inner model’s chunks unless you pass subgraphs=True. The inner ns namespace tells you which subgraph a chunk came from. This is the single most common “why is my stream empty” bug when people compose agents.
• stream_mode="updates" is what turns tool activity into a step list: each node name plus its delta is one row in the activity panel, in execution order.
• nostream tag suppresses specific invocations from the messages channel while still running them — the right tool for internal/structured calls whose output you don’t want double-rendered or leaked to the client.

UI patterns that work

• Progressive rendering. Don’t buffer the stream and repaint at the end. Append custom events to a bounded list (last ~100), and render the current tool’s done/total as a determinate bar. Determinate beats a spinner the moment you have counts.
• Separate the lanes. Model tokens belong in the transcript; tool steps and progress belong in an activity panel. Mixing them makes both harder to read and makes resuming a run confusing.
• Coalesce before the wire. Between the tool and the browser, throttle emissions (e.g. flush at most every 100 ms) and drop superseded progress events — sending “12%” then “14%” then “16%” to a mobile client is wasted bandwidth if the UI only paints once per frame.
• Timeouts keyed to liveness, not duration. With progress flowing, a “no event for 60 s” watchdog is a much better failure detector than “run exceeded 5 minutes.” A five-minute job emitting heartbeats is healthy; a thirty-second job that stops emitting is not.

Ordering and backpressure

What you can rely on: a single .stream()/.astream() iterator delivers parts in the order the runtime produced them, per stream — so updates for step n arrive before step n+1, and your tool’s progress events interleave in emission order. What you cannot rely on: delivery semantics across a network hop or across multiple consumers. If you fan the same run out to an SSE endpoint and a websocket, each transport needs its own sequence handling.

Backpressure is the consumer’s responsibility. The producer (graph execution) runs at its own pace; if your consumer awaits a slow database write for every event, you simply build a queue in memory. Practical mitigations:

• Buffer and flush on a timer rather than per event.
• If the consumer is persisting events, batch writes; never block the graph on UI persistence.
• For HTTP streaming, disable response buffering on the proxy — a common cause of “streaming works locally, arrives in one burst in production.”

Resumability: what survives a checkpoint

Checkpoints give durability: with a checkpointer, stream_mode="checkpoints" (or tasks) shows checkpoint and task events as they’re written, and an interrupted run resumes from the last checkpoint with Command(resume=...). In version="v2", interrupts surface on values parts as an interrupts field instead of hiding in a __interrupt__ state key, which makes a client able to render “waiting for approval” without string-matching state.

The honest caveat: custom events are not state. A progress event emitted mid-tool is an observation of a run in flight; it is not written to the checkpoint. If the process dies halfway through the backfill and resumes, the client’s progress bar resets to whatever the tool re-emits after the checkpoint — unless the tool also writes progress into state (returning {"progress": ...} or updating a channel at a coarse cadence), which is then checkpointed and replayed. Rule of thumb: stream the fine-grained detail for UX, checkpoint the coarse milestones for correctness. If you need ordered, replayable history of every event, that’s event sourcing of your own — write to a durable log, don’t expect the stream to be one.

A complete loop

from typing import TypedDict, Annotated
import operator
from langgraph.graph import StateGraph, START, END


class State(TypedDict):
    messages: Annotated[list, operator.add]
    progress: list[dict]


def run(state: State):
    return {"progress": []}


graph = StateGraph(State)
graph.add_node("backfill", run)
graph.add_edge(START, "backfill")
graph.add_edge("backfill", END)
app = graph.compile(checkpointer=MemorySaver())

async def drive(client, config):
    async for part in app.astream(
        {"messages": [...], "progress": []},
        config=config,
        stream_mode=["updates", "custom", "messages"],
        subgraphs=True,
        version="v2",
    ):
        kind = part["type"]
        if kind == "updates":
            await client.send_step(list(part["data"]))
        elif kind == "custom":
            await client.send_progress(part["data"])   # throttled inside
        elif kind == "messages":
            chunk, meta = part["data"]
            if chunk.content and meta.get("langgraph_node") == "backfill":
                await client.send_tokens(chunk.content)

One iterator, three lanes, each rendered where it belongs.

When not to stream

• Short tools. If a call completes in 200 ms, progress events are pure noise and connection overhead. Stream only what crosses a perceptual threshold — the usual working number is “will the user wonder if it broke?”
• Tools whose partial results mislead. A search tool that has found 3 of 10 results renders a UI implying completion; emit milestones instead of raw partials when the intermediate state is not meaningful.
• Events that bypass redaction. Whatever your pipeline strips from final output (secrets, PII, private channels), the stream must strip too. A raw values stream carries the whole state, so choose modes deliberately when the state holds sensitive material.
• High-frequency machine consumers. If the consumer is another service that only needs the final result, values once (or invoke()) is cheaper than thousands of parts.
• When it creates duplicate work. If cancel/resume semantics are immature in your deployment, a stream that visibly “hangs” may still be safer than one that invites the refresh-and-rerun behavior. Fix the liveness signal first.

Further reading

• Streaming — LangGraph docs
• Event streaming (typed projections, LangGraph v1.2) — LangGraph docs
• LangGraph documentation root
• Persistence, checkpointing and interrupts — LangGraph docs
• LlamaIndex agents and workflows