Decoupling Agent Triggers from Agent Execution with Task Queues

Synchronous agent invocation inside a webhook handler breaks under real traffic. Put a task queue between 'something happened' and 'the agent runs', and get the queue semantics, payload design, and retry split right.

Published on • October 31, 2026

AI Assistant

The fastest way to ruin a working agent demo is to wire it directly into a webhook handler. A customer uploads a document, your endpoint awaits a five-minute multi-tool agent run, and everything looks fine until the first burst of traffic, the first provider timeout, or the first deploy that kills in-flight runs. The bug is architectural: the process that observed the event is the same process that executes the work.

The fix is an old one that transfers cleanly to LLM agents: a task queue between the trigger and the executor. The webhook handler validates, deduplicates, and enqueues; a separate worker pool runs the agent. Pydantic AI’s own docs describe agents running everywhere “behind a web frontend, in the terminal, on a voice call, [or] on a durable background queue” — the durable queue is the one that survives contact with production. This post walks through why synchronous invocation fails, the queue semantics that actually matter for agent workloads, and a working FastAPI + arq + PydanticAI example.

Why synchronous invocation breaks

Three failure modes, in the order they usually arrive:

Webhook timeouts. Most event providers (Stripe, GitHub, Shopify, internal buses) expect an ACK in a few seconds and retry aggressively when they do not get one. Your agent run takes 40 seconds; the provider times out at 10 and redelivers. Now you have two agent runs for one event — and the trigger side, not the agent side, is where the duplication was created.
Burst loads. A single event is fine. Fifty events in ten seconds means fifty concurrent agent runs, each fanning out into LLM calls and tool executions, all inside your web tier. Memory, connection pools, and provider rate limits all fail at once, and the HTTP layer has no vocabulary for “queued”.
Long agent runs. Agent runs are not database queries. A research agent with retrieval, three tool calls, and a reflection loop can run for minutes. Holding an HTTP connection, a worker slot, and a request-scoped timeout open for that long couples the agent’s latency budget to the transport’s latency budget — and during a rolling deploy, in-flight runs die with the process.

A queue separates these budgets: the trigger path answers in milliseconds, the executor path takes as long as the work takes.

The trigger/executor split

The shape is the same regardless of framework:

webhook / event bus / cron
        │  1. validate + authenticate
        │  2. compute idempotency key
        │  3. enqueue (reference, not payload)
        ▼
   task queue  ──►  worker pool  ──►  agent run (LLM calls + tools)
        │                │
        │                └── retries, timeouts, DLQ
        └── visibility timeout, dedup, backpressure

The trigger endpoint owns three jobs and only three: authenticate the source, turn the event into a reference (an ID and a small payload), and write that reference durably before ACKing. Everything else — model calls, retries, tool execution — belongs to the worker.

Queue semantics that matter for agents

Not all queues are equal here, and the differences show up under failure:

At-least-once delivery. Every mainstream option (Redis streams, SQS, Kafka, Postgres-backed queues) gives you at-least-once, not exactly-once. A worker that crashes after the agent’s side effects but before acking will see the job again. Design for duplicate runs, not for their impossibility.
Visibility timeout. When a worker picks up a job, the queue hides it for a visibility window; if the worker dies without acking, the job reappears. This window must exceed your maximum agent run time, or you get concurrent duplicate runs. If your agent can run five minutes, a 30-second default visibility timeout is a bug. Long jobs generally want per-job timeouts and a heartbeat, not one giant global window.
Dead-letter queues (DLQ). After N attempts, a failing job moves to a DLQ instead of cycling forever. This matters more for agents than for CRUD jobs, because an agent failure often means a poisoned prompt or a tool that returns garbage the model loops on — retries do not heal that, they just multiply your LLM bill.

Payload design: reference, don’t embed

The most common queue mistake with agents is stuffing the entire event payload (a 2 MB document, full conversation history, base64 attachments) into the queue message. Instead, enqueue a reference: the event or object ID, a schema version, and whatever the worker needs to fetch the rest.

Queues are optimized for small messages; large payloads bloat Redis memory, slow replication, and make DLQ inspection painful.
A reference forces the worker to load current state, which is usually what you want anyway — a five-minute-old snapshot in a queue message is stale input.
Version the message schema ({"v": 2, "ticket_id": 123, "mode": "full"}) so you can run old and new worker versions side by side during a deploy.

Idempotency keys on enqueue

Since delivery is at-least-once and providers redeliver, the trigger side needs its own dedup. Derive a stable key from the source’s event ID when it has one (evt_3f8...), or from a natural key (order_id + action) when it does not, and pass it as the queue’s job ID. With arq, re-enqueuing the same job_id while the job is pending or its result is retained is a no-op; SQS’s MessageDeduplicationId and Celery’s task_id serve the same role. Belt and braces: put a unique constraint on the key in Postgres at enqueue time, so a duplicate enqueue from a retried webhook fails visibly instead of silently.

Worker concurrency vs LLM rate limits

Worker concurrency is a rate-limit decision wearing an infrastructure costume. If you run 64 workers and each agent run makes 6 model calls, you can issue ~400 concurrent requests to your provider — a guaranteed 429 storm, and the retry traffic makes it worse. Set max_jobs (arq), worker_concurrency (Celery), or a semaphore to a number derived from your provider’s RPM and your per-run call count, not from your CPU count. Model-level gateways help: LiteLLM’s router tracks per-deployment cooldowns (a 429 cools a deployment immediately by default) and can reject over-parallel requests with a fast 429 instead of queuing them into a latency death spiral, which is a better failure than silently serializing everything.

Retries: queue layer vs inside the agent

Split retries by what failed:

Queue-level retries handle infrastructure: worker crash, deploy kill, network blip to the database. Configure max_tries low (3–4) with exponential backoff. These retries re-run the whole agent.
In-agent retries handle model-level transients: one LLM call times out, one tool call 503s. These are cheap and scoped — retry the call, not the run — with at most 1–2 attempts and a fallback model.

The trap is nesting them. Queue retries of 5 multiplied by in-agent retries of 3 and 3 model calls per step turns one poisoned job into dozens of runs. Cap the outer loop hard, and make the agent bail out fast on non-retryable errors (bad request, context overflow, content filter) rather than burning attempts.

Observability: propagate trace context through the queue

The moment the trigger and the executor split, so does your trace — unless you carry the context. Inject the W3C traceparent header value into the job at enqueue time and extract it in the worker, so the agent run spans appear as children of the webhook span instead of as orphan traces. PydanticAI’s instrumentation is plain OpenTelemetry, so the extracted context flows straight into its model and tool spans with no glue code. Also record queue-native timing separately: enqueue-to-start (queue wait) is a metric your HTTP-path tracing never sees, and it is usually the first thing to degrade under load.

A minimal example: FastAPI + arq + PydanticAI

# api.py — the trigger side: validate, dedupe, enqueue, ACK fast.
from fastapi import FastAPI, Header, HTTPException
from pydantic import BaseModel
from arq import create_pool
from arq.connections import RedisSettings
from opentelemetry import trace

app = FastAPI()
tracer = trace.get_tracer("triggers")

class TicketEvent(BaseModel):
    ticket_id: int
    prompt: str

@app.post("/webhooks/ticket", status_code=202)
async def ticket_webhook(
    event: TicketEvent,
    x_trigger_id: str = Header(...),  # provider's event id = idempotency key
    traceparent: str | None = Header(None),
):
    with tracer.start_as_current_span("enqueue_agent_run"):
        redis = await create_pool(RedisSettings())
        # job_id dedupes: a redelivered webhook is a no-op while the job is live.
        await redis.enqueue_job(
            "run_agent",
            event.ticket_id,
            event.prompt,
            traceparent or "",
            job_id=x_trigger_id,
        )
    return {"queued": True, "trigger_id": x_trigger_id}

# worker.py — the executor side: run the agent, retry only what fails.
from arq.connections import Retry, RedisSettings
from opentelemetry import trace
from opentelemetry.propagate import extract
from pydantic_ai import Agent

agent = Agent("openai:gpt-6.1-sol", output_type=str, retries=2)

async def run_agent(ctx, ticket_id: int, prompt: str, traceparent: str):
    parent = extract({"traceparent": traceparent})
    tracer = trace.get_tracer("agent-worker")
    with tracer.start_as_current_span(
        "agent.run", context=parent, attributes={"ticket.id": ticket_id}
    ):
        result = await agent.run(prompt)
        await save_result(ticket_id, result.output)

class WorkerSettings:
    functions = [run_agent]
    redis_settings = RedisSettings()
    max_jobs = 8          # derive from provider RPM / calls-per-run, not CPU count
    job_timeout = 600     # must exceed the longest possible agent run
    max_tries = 3         # outer loop: infra failures only
    keep_result = 3600    # retains dedupe window and result for the trigger side

The endpoint’s latency budget is now independent of the agent’s, redelivered webhooks collapse onto one job_id, and a rolling deploy only needs workers to finish or release their current jobs.

Failure modes to design for explicitly

Poison messages. A payload that makes the agent loop or crash on every attempt. Without a DLQ and max_tries, it retriggers forever; with them, it parks in the DLQ with its error for inspection. Cap in-agent spend (max turns, token budget) so each attempt fails fast.
Duplicate runs. At-least-once means the same ticket can run twice: once from a redelivered webhook that beat the dedup window, once from a visibility timeout on a slow run. The queue cannot prevent this; your side effects can — idempotency keys on the downstream write, or a runs table with a unique constraint on the trigger ID that the worker checks-and-inserts transactionally before doing anything expensive.
Lost triggers. The ugly gap between “responded 202” and “job durable”. If you ACK before the enqueue is persisted, a crash loses the event and nobody retries it. Enqueue first, ACK second. For environments where even that window is unacceptable, write the trigger row and the queue message via a transactional outbox: one Postgres transaction writes the intent, a relay pushes it to the queue.

Checklist

Never await a full agent run inside a request handler; enqueue a small, versioned reference and return 202.
Choose queue semantics deliberately: at-least-once delivery, a visibility timeout longer than your longest run, and a DLQ with alerting.
Dedupe on enqueue with the provider’s event ID as job ID, backed by a unique DB constraint.
Size worker concurrency from provider rate limits, not hardware.
Split retries: few and coarse at the queue level, one or two and fine-grained inside the agent; never multiply them.
Propagate traceparent through the job payload so queue wait and agent execution live in one trace.
Make side effects idempotent so duplicate runs are boring.