Event Sourcing Agent Decisions: Replaying Past Runs for Debugging

An agent ships a wrong answer at 2 a.m. and by morning nobody can say why: the prompt changed, the tool returned something odd, the model picked plan B. Logs show fragments, a dashboard shows latency, neither shows the decision sequence. Event sourcing flips the model: instead of mutating state and keeping a trail, you append every decision - plan chosen, tool called, output accepted - as an immutable event, and the run’s state becomes whatever you get when you fold that log. The payoff is replay: you can rebuild the exact conditions of a failure on your laptop and watch it happen again.

What counts as a decision event

A decision event is a record that something was chosen, not that something finished. Useful types:

• plan.selected - the planner picked an approach (or a route in a workflow)
• tool.requested / tool.completed - the model asked for a call, and what came back
• guardrail.tripwire - an input, output, or tool guardrail blocked the run
• output.accepted / output.rejected - a reviewer, evaluator, or human signed off or refused
• handoff - control moved between agents

The test for inclusion: if losing it would make the failure unreproducible, it belongs in the log.

Event log versus tracing spans

OpenTelemetry already gives you a decision-shaped primitive, so do not confuse the two layers. In OTel, a span is a unit of work with a name, parent span ID, start and end timestamps, a span context (trace ID and span ID), attributes, span events, span links, and a status of Unset, Error, or Ok. Spans sharing a trace ID with a parent_id chain form a trace: the big picture of one request’s path through your system.

Span events are the point-in-time annotations inside a span: “a structured log message (or annotation) on a Span, typically used to denote a meaningful, singular point in time”. The OTel guidance for choosing between them is mechanical: if the timestamp of the occurrence is meaningful, attach a span event; if it is not, attach a span attribute.

So the division of labor is:

• Traces answer “where and when”. Latency, the hierarchy of calls, which model request sat inside which tool call. GenAI conventions for this live in the OpenTelemetry GenAI semantic conventions repo, which covers spans, metrics, and events for GenAI clients, MCP, and provider-specific conventions.
• The decision log answers “why, and what exactly”. The chosen option, the arguments, the rejected alternative, the acceptance verdict - durable, queryable records you can fold and replay.

A trace is sampled, trimmed, and eventually aged out. A decision store is your replay substrate. Emit both: the span carries a reference to the event, the event carries the trace ID.

A schema you can actually replay

Replay needs order, identity, and enough payload to reconstruct inputs. This shape works:

{
  "event_id": "01J9Z8Q2M4",
  "seq": 17,
  "run_id": "run_2026_10_15_0042",
  "trace_id": "5b8aa5a2d2c872e8321cf37308d69df2",
  "ts": "2026-10-15T03:12:09.412Z",
  "actor": "billing_agent",
  "type": "tool.requested",
  "payload": {
    "tool": "issue_refund",
    "args": {"order_id": "A-4471", "amount_cents": 2500},
    "decision": "chosen_from": ["issue_refund", "escalate_to_human"],
    "model": "gpt-5.6",
    "prompt_hash": "sha256:9f2c..."
  },
  "idempotency_key": "run_0042:issue_refund:A-4471",
  "prev_hash": "sha256:3b71..."
}

Field notes: seq is monotonic per run and is your ordering guarantee; trace_id joins the record to the OTel trace; prompt_hash ties it to the exact prompt version without storing the whole prompt inline; idempotency_key is what makes replay safe (more below); prev_hash gives you tamper evidence. Keep values to strings, numbers, booleans, and arrays of those - the same restriction OTel imposes on span attributes, which keeps payloads serializable in every backend you might export to.

The matching span looks like the ones in the OTel docs: a span with an events array where each entry has a name, a timestamp, and attributes.

Recording: one write to two places

In your tool wrapper or workflow node, append the decision event first, then emit the observability signal:

import json, time, uuid
from opentelemetry import trace

tracer = trace.get_tracer("agent.decisions")

def record_decision(log, run_id, trace_ctx, **event):
    event = {
        "event_id": uuid.uuid4().hex,
        "seq": len(log) + 1,
        "run_id": run_id,
        "ts": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
        **event,
    }
    log.append(event)                      # durable, append-only
    span = trace.get_current_span()
    span.add_event(event["type"], attributes={   # point-in-time annotation
        "agent.seq": event["seq"],
        "agent.event_id": event["event_id"],
    })
    return event

def issue_refund(order_id: str, amount_cents: int, *, ctx) -> dict:
    record_decision(ctx.decision_log, ctx.run_id, ctx.trace,
                    actor=ctx.agent_name, type="tool.requested",
                    payload={"tool": "issue_refund",
                             "args": {"order_id": order_id,
                                      "amount_cents": amount_cents}})
    result = payments_api.refund(order_id, amount_cents)
    record_decision(ctx.decision_log, ctx.run_id, ctx.trace,
                    actor=ctx.agent_name, type="tool.completed",
                    payload={"refund_id": result["id"], "status": result["status"]})
    return result

Note the ordering: the request event is appended before the side effect. If the process dies mid-refund, the log shows an intent with no completion, which is exactly the state you want to investigate.

Replay workflow: reproduce before you fix

Deterministic replay means running the same decision function over the recorded inputs and stubbing everything non-deterministic.

1. Load and verify. Fetch events for run_id, sort by seq, check prev_hash links.
2. Reconstruct. Fold the log into the state the agent saw: plan.selected sets the plan, tool.completed populates observations.
3. Stub the world. Replace model calls with the recorded response and tools with recorded outputs, keyed by idempotency_key.
4. Re-run the decision code. Execute your router, planner, or guardrail against the reconstructed inputs.
5. Diff. Compare new decisions to the recorded ones. The first divergence index is your bug.

class RecordingStub:
    def __init__(self, events):
        self.responses = {
            e["idempotency_key"]: e["payload"]
            for e in events
            if e["type"] in ("tool.completed", "model.response")
        }

    def call(self, tool, args, run_id, seq_hint):
        key = f"{run_id}:{tool}:{args.get('order_id', '')}"
        if key not in self.responses:
            raise KeyError(f"no recorded response for {key}; replay is not deterministic")
        return self.responses[key]          # never touches the real API

def replay(run_id, events, decision_fn):
    stub = RecordingStub(events)
    state = fold(events)                    # reconstruct at seq N
    for event in sorted(events, key=lambda e: e["seq"]):
        state = decision_fn(state, event, stub)
        if state.get("diverged"):
            return event["seq"], state      # first divergence wins
    return None, state

Debugging replay versus regression replay

The two uses share machinery but not goals. Debugging replay targets one run, asks “what did the code decide given these inputs”, and is allowed to stop at the first divergence; it is interactive, and the output is an explanation. Regression replay takes a corpus of saved runs (especially the ones that failed), replays them in CI with the new build, and fails the pipeline if a previously-correct decision changes or a previously-wrong one survives; it is batch, headless, and its output is a pass/fail. Store failed runs as fixtures: the event log is already the fixture format.

The hazards: duplicates and side effects

Replay re-executes code, and code that talks to the outside world does not rewind with you.

• Non-idempotent tools. Replaying issue_refund against the real payments API refunds twice. Either serve the recorded response (as above), or require the tool to accept an idempotency key and pass the original one, so the upstream API deduplicates.
• Dry-run mode. For tools you cannot stub (rare external writes), give every tool a dry_run flag that logs the intent and returns a synthesized success. Replays default to dry-run; production defaults to live.
• Deduplicate on ingest. Network retries can append the same decision twice. Key on event_id, or on (run_id, seq), and reject the second write.
• Environment drift. Record model version, prompt hash, and tool schema version in the event. If the deployed model differs from the recorded one, the replay is a simulation, not a reproduction - label it as such.

Retention and PII: your log is a transcript

Because decision events carry arguments and outputs, the store is often more sensitive than the trace. Practical posture: hash or redact personal data at write time (keep a stable hash so replay still matches), scope access by tenant and purpose, set a retention window per event type, and keep raw model outputs shorter-lived than the decisions derived from them. A useful rule: enough payload to replay, never more. If a field is not needed to reconstruct a decision, it does not go in the log; it goes in a blob with its own TTL.

Key Takeaways

• Treat decisions as immutable, ordered events (seq, run_id, trace_id, payload, idempotency key); the folded log is the run’s state.
• Traces and the decision log are complementary: spans give hierarchy and timing, span events mark point-in-time moments, the event store gives replayable semantics.
• Append the intent event before the side effect so crashes leave evidence instead of gaps.
• Replay is deterministic only when model and tool outputs are stubbed from the record; the first divergence point is the bug.
• Use one corpus of runs for interactive debugging and for CI regression replay; failed runs double as test fixtures.
• Guard against duplicate appends and non-idempotent tools, and redact PII at write time with a retention policy per event type.

References:

• https://opentelemetry.io/docs/concepts/signals/traces/
• https://opentelemetry.io/docs/specs/semconv/gen-ai/
• https://github.com/open-telemetry/semantic-conventions-genai