Session Replay for Agents: Reproducing a Full Run

Every team that puts an agent into production hits the same wall eventually: a user reports that the agent “did something weird,” you open the logs, and you get a scatter of disconnected lines — a prompt here, a tool call there, a final answer somewhere else. You can tell what happened. You can’t tell why, because the exact inputs the model saw at decision time are gone.

Session replay fixes that asymmetry. Instead of logging outcomes, you record the run itself: the input, each model turn, every tool invocation with its arguments and result, each handoff, and the guardrail decisions along the way. Once you have that, “reproduce this run” becomes a mechanical operation instead of an archaeology project.

Why logs alone aren’t enough

Traditional application logs are fine when code is deterministic. Given the same state, the same function produces the same output, so a log line is a sufficient record of a decision.

Agents break that assumption in three ways:

1. Model calls are non-deterministic. Same prompt, different completion — sampling, provider-side model revisions, and prompt caching all move the result.
2. Tool outputs drift. A get_weather() call made at 09:00 returns something different at 14:00. Re-running the agent later doesn’t re-create the original conditions.
3. Control flow is emergent. Handoffs, loops, and guardrails mean the path through the system is itself an output of the model, not something you wrote in an if statement.

A log that says tool=send_email args={...} records the effect. It doesn’t record the retrieval results, the system instructions, the prior turns, or the alternative the model rejected. Replay requires the full decision context.

What to record: the run envelope

The unit of replay is a single run. For each one, capture an ordered, append-only sequence of events. A practical envelope looks like this:

from pydantic import BaseModel
from typing import Any, Literal

class RunEvent(BaseModel):
    run_id: str            # ties every event in this run together
    seq: int               # strictly monotonic ordering within the run
    ts: str                # ISO 8601 timestamp
    kind: Literal[
        "input", "generation", "tool_call", "tool_result",
        "handoff", "guardrail", "output",
    ]
    agent: str | None = None
    model: str | None = None
    payload: dict[str, Any]  # kind-specific: prompt, args, response, verdict
    parent_seq: int | None = None  # nesting: tool_call -> tool_result
    config: dict[str, Any] = {}     # temperature, prompt version, SDK options

Three fields do most of the heavy lifting later:

• seq — replay must restore original ordering even when events are written concurrently.
• config — model id, temperature, and prompt/agent version. Without it you can re-run but not compare.
• parent_seq — lets you rebuild the tree of turns and tool calls, not just a flat timeline.

Building it on the SDK’s tracing layer

If you’re on the OpenAI Agents SDK, you already have most of the capture machinery. The SDK ships with built-in tracing that records LLM generations, tool calls, handoffs, guardrails, and custom events, organized as traces (one end-to-end workflow, with a workflow_name and a trace_id in the form trace_<32_alphanumeric>) composed of spans (start/end operations with a parent_id linkage).

Default instrumentation wraps each Runner.{run, run_sync, run_streamed} call in a trace, and nests agent_span(), generation_span(), function_span(), guardrail_span(), and handoff_span() underneath. That is exactly the event stream replay needs — the shape is there; you have to persist it somewhere you control.

Two hooks matter:

• Custom trace processors. add_trace_processor() forwards spans to an additional backend while keeping the default exporter; set_trace_processors() replaces the pipeline entirely. For replay you usually want set_trace_processors() with your own processor that writes each span to durable storage keyed by trace_id.
• flush_traces(). The default BatchTraceProcessor exports in the background every few seconds. In a worker or background task, call flush_traces() after the trace context closes so the record exists the moment the run ends — not seconds later, and never lost on process exit.

from agents import Runner, trace, flush_traces
from replay.store import ReplayingProcessor, append_only_sink

processor = ReplayingProcessor(sink=append_only_sink("runs/"))
set_trace_processors([processor])

def run_and_record(prompt: str) -> str:
    try:
        with trace("support_agent"):
            result = Runner.run_sync(agent, prompt)
        return result.final_output
    finally:
        flush_traces()  # block until spans are on disk

Note the ordering: flush_traces() after the trace() block exits. Flushing mid-trace exports a partial record, and a partial record is worse than none — it silently looks complete.

Replay: two modes, two questions

“Replay” means two different things, and conflating them is the most common design mistake.

Mode 1: re-visualize (read-only)

Rebuild the timeline from stored events and walk it. No model calls, no side effects. This answers what did the model see? — you click a generation event and inspect the exact messages, then click the preceding tool_result events to see where bad data entered.

This mode is cheap, safe, and should be your first milestone. It’s also what the SDK’s trace dashboard gives you out of the box, provided you’ve kept sensitive payloads.

Mode 2: re-execute (deterministic replay)

Re-run the agent, but pin every source of variation so execution follows the original path:

class ReplayModel(Model):
    """Serves recorded generations instead of calling the API."""
    def __init__(self, transcript: dict[int, Any]):
        self.transcript = transcript  # keyed by turn index

    async def complete(self, params):
        turn = params.response_format and self._next_turn()
        recorded = self.transcript[turn]
        # Fail loudly on divergence instead of drifting silently.
        if params.input != recorded.request:
            raise ReplayDiverged(turn, expected=recorded.request, got=params.input)
        return recorded.response

class ReplayTool:
    """Returns recorded outputs; never hits the real system."""
    def __init__(self, events: list[RunEvent]):
        self.results = {e.payload["call_id"]: e.payload["result"]
                        for e in events if e.kind == "tool_result"}

Wire the ReplayModel in via the SDK’s model interface, and swap side-effecting tools for their recorded results. Now Runner.run_sync() produces the original trajectory, and any divergence raises immediately at the point of divergence instead of producing a plausible-but-wrong run.

The ReplayDiverged check is the whole value. If your replay silently completes with a different answer, you’ve built an expensive way to generate new bugs.

Practical pitfalls and gotchas

Sensitive data is captured by default. Generation and function spans store inputs and outputs. That’s what makes replay possible, and it’s also how you leak PII into your observability backend. RunConfig.trace_include_sensitive_data (or OPENAI_AGENTS_TRACE_INCLUDE_SENSITIVE_DATA) controls it. For regulated payloads, redact inside your own exporter before it leaves the process — a redaction processor registered as a separate observer doesn’t guarantee the default exporter won’t receive the raw data first.

Tracing is unavailable under Zero Data Retention. If your org uses OpenAI’s APIs under a ZDR policy, the hosted tracing path isn’t available. You need to capture spans through your own processor and store them in your own infrastructure — plan for this before promising replay.

Replay without a pinned prompt version is a lie. Store the agent instructions (or a content hash of them) with the run. Prompt changed + replayed run ≠ original run.

Tool replay breaks on non-idempotent tools. Recorded-output substitution only works if you replace the tool before it executes. Discovering at replay time that delete_row() already ran for real is a bad outcome. Keep a registry of side-effecting tools and force them through the replay shim.

Unbounded storage grows fast. Full generation payloads are large. Decide retention per event kind — keep input and output for weeks, keep config and digests for months.

Ordering under concurrency. Parallel tool calls complete out of sequence. Persist seq at emission time, not at write time, or your replayed timeline won’t match the original.

Where to start

Start with read-only replay: one custom trace processor, one append-only store keyed by trace_id, one UI or CLI that renders the ordered events. You’ll get most of the debugging value within a day. Deterministic re-execution is a second project — build it only for the workflows where reproducing a failure actually matters (refund flows, data mutations, customer-facing writes).

The rule of thumb: logs tell you what the system printed. Replay tells you what the system knew when it decided.

Sources

• OpenAI Agents SDK documentation
• OpenAI Agents SDK — Tracing
• Python contextvars — deterministic context propagation