Batching Agent Workloads to Maximize Throughput per Dollar
Trade latency for cost on offline agent work: when to batch vs self-coalesce, how to size batches against token limits, checkpointed queue workers, and a runnable Gemini Batch API pipeline.
Published on • November 16, 2026
AI Assistant

Every organization has two kinds of LLM work: work that must return in 800 milliseconds, and work that must return by morning. Most teams process both identically — one request at a time, standard pricing, hitting the same endpoint as their chat product — which means they pay full price for patience they don’t need. Eval runs, backfills, enrichment, and nightly digests don’t care about p95 latency. They care about finishing before the coffee is ready, for as few dollars as possible.
In this post, you will learn how to batch offline agent workloads to maximize throughput per dollar on the Gemini API: when a provider Batch API beats self-coalescing, how to size batches against token limits, why batching raises throughput but destroys per-request latency guarantees, when not to batch, how to design queues with checkpointed workers, and how to submit, poll, and retrieve Gemini Batch API jobs with google-genai — plus the arithmetic that shows exactly what batching saves.
Prerequisites
- Python 3.10+ and
pip install google-genai. - A paid-tier
GEMINI_API_KEY— the Batch API is a paid-tier feature. - A backlog of genuinely offline work: an eval set, a backfill table, or a digest’s worth of documents.
- A queue in production (SQS, Redis Streams, Kafka). The walkthrough uses Python’s
queue.Queueso it runs anywhere. - Comfort with JSONL — it’s the batch input format.
Why Batching Raises Throughput per Dollar
Throughput per dollar is tokens processed per second divided by cost. Batching attacks the denominator: the Gemini Batch API processes large volumes asynchronously at 50% of the standard cost, with a target turnaround of 24 hours — “much quicker” in the majority of cases, per the batch documentation. Look at what the halving does to the economics on gemini-3.5-flash-lite (prices per 1M tokens, standard vs batch: $0.30/$2.50 vs $0.15/$1.25):
Nightly job: 10M input tokens + 2M output tokens, every night.
standard: 10 * $0.30 + 2 * $2.50 = $3.00 + $5.00 = $8.00 / night
batch: 10 * $0.15 + 2 * $1.25 = $1.50 + $2.50 = $4.00 / night
saved: $4.00 / night
$1,460 / year
Even if batching halved your effective processing speed, tokens per dollar would stay flat — you’d move the same tokens for the same money in twice the wall-clock time. In practice you usually gain throughput too, because the provider packs many requests into shared compute that interactive traffic can’t monopolize. That asymmetry — same or better aggregate throughput, half the price — is the entire pitch. It only holds for work whose deadline is measured in hours.
Batch APIs vs Self-Coalescing
There are three ways to trade latency for cost, and they’re worth ranking by how much they change your code:
- Provider Batch API (deepest discount, most structure). You submit a job — inline requests under 20MB, or a JSONL file up to 2GB uploaded through the File API — and poll (or get webhooks for) completion. Best for: backfills, eval runs, nightly digests.
- Flex tier (middle ground). Gemini’s pricing page lists Flex at the same reduced rates as batch for latency-tolerant work that still needs an interactive-shaped request/response loop.
- Self-coalescing (you do the work). Pack k items into a single prompt: “Classify these 40 tickets and return a JSON array.” This amortizes the system prompt across items and cuts request count by k, but you pay standard rates, and it’s bounded by output limits — see the next section.
Self-coalescing is not a discount; it’s an overhead optimization. Keep it in your toolkit for moderate savings, but don’t confuse it with the 50% you get by letting the provider schedule your work.
Batch Size Selection vs Token Limits
Batch sizing is a constraint-satisfaction problem with four walls: the 20MB inline ceiling (use the file path above it), the 2GB file ceiling, the model’s context window, and — the one teams forget — the max output tokens. If you coalesce k items that each produce ~400 output tokens and the model caps output at 8k, then k ≤ 20 before results start getting truncated mid-array.
def plan_chunks(items: list[dict], est_input: int, est_output: int,
max_input: int = 80_000, max_output: int = 16_000,
max_per_chunk: int = 40) -> list[list[dict]]:
"""Group items so each chunk fits input and output token ceilings."""
chunks, current, current_in = [], [], 0
for item in items:
room_by_output = (max_output - sum(i.get("out", est_output) for i in current)) // est_output
room_by_input = (max_input - current_in) // max(est_input, 1)
if current and min(room_by_output, room_by_input, max_per_chunk) <= 0:
chunks.append(current)
current, current_in = [], 0
current.append(item)
current_in += est_input
if current:
chunks.append(current)
return chunks
chunks = plan_chunks(backlog, est_input=1_200, est_output=400)
For file-based batches (one request per line, the normal case), chunking serves a different purpose: keeping individual jobs small enough that they finish well inside the 48-hour expiry window. A job that sits pending for two days and then expires — which the API does to any job pending or running past 48 hours — is work you have to detect and resubmit. Split by item count or total bytes, checkpoint every submission, and a dead job becomes a retry, not a disaster.
When NOT to Batch
Batching is a deliberate sacrifice of latency guarantees. Per-request latency stops being a property you control — a batch response arrives when the whole job completes, not when your item finishes. So keep these paths off batch entirely:
- Interactive chat and voice — users notice seconds; the Live API exists for this.
- Agent tool loops inside a live run — a plan step waiting 6 hours breaks the agent, not just the SLA.
- Real-time moderation, fraud, and RAG lookups — staleness is a correctness issue.
- Anything you’d retry on a 5-second timeout — batch has no timeout to retry against; it has a 24-hour target.
Also note the API boundary: Batch API currently runs on the generateContent surface only — the Interactions API doesn’t support it yet. That’s fine for offline pipelines, but it means your batch worker is a separate code path from your interactive one. Design for that explicitly rather than discovering it at 2 a.m.
Queue Design and Checkpointed Workers
The shape is a classic producer/consumer with one non-negotiable addition: a checkpoint store that makes submission idempotent. Without it, a crashed worker resubmits a job that already ran — you pay twice for the same tokens.
# worker.py — producer, drain window, idempotent submission.
import hashlib, json, queue, time
from pathlib import Path
from google import genai
from google.genai import types
client = genai.Client()
in_q: queue.Queue = queue.Queue(maxsize=5_000)
CHECKPOINT = Path("batch_checkpoints.json")
def load_ckpt() -> dict:
return json.loads(CHECKPOINT.read_text()) if CHECKPOINT.exists() else {}
def save_ckpt(ckpt: dict) -> None:
CHECKPOINT.write_text(json.dumps(ckpt, indent=2))
def producer(rows: list[dict]) -> None:
for row in rows:
in_q.put(row)
in_q.put(None) # poison pill closes the drain
def drain(max_items: int = 400, max_wait: int = 30) -> list[dict]:
"""Self-coalescing window: gather items until full or the wait expires."""
items, deadline = [], time.time() + max_wait
while len(items) < max_items and time.time() < deadline:
try:
item = in_q.get(timeout=0.5)
except queue.Empty:
continue
if item is None:
break
items.append(item)
return items
def submit(items: list[dict], model: str = "gemini-3.5-flash-lite") -> str:
# Deterministic batch id: re-running after a crash can't double-submit.
batch_id = hashlib.sha256(
json.dumps(sorted(i["id"] for i in items)).encode()
).hexdigest()[:12]
ckpt = load_ckpt()
if ckpt.get(batch_id, {}).get("state") == "SUCCEEDED":
return ckpt[batch_id]["job"] # already paid for — just return it
path = Path(f"batch_{batch_id}.jsonl")
path.write_text("\n".join(
json.dumps({"key": i["id"], "request": build_request(i)}) for i in items
))
uploaded = client.files.upload(
file=str(path),
config=types.UploadFileConfig(display_name=path.name, mime_type="jsonl"),
)
job = client.batches.create(
model=model, src=uploaded.name, config={"display_name": path.name}
)
ckpt[batch_id] = {"job": job.name, "state": "PENDING", "items": len(items)}
save_ckpt(ckpt)
return job.name
The drain window is where batching actually happens operationally: items arrive continuously, but a job only ships when the count or the clock says so. The deterministic batch_id — a hash of the sorted item ids — is what makes retries safe: if the worker died after batches.create but before the checkpoint write, the resubmission hashes identically and you reconcile against client.batches.list() instead of paying for a duplicate. Checkpoint states move PENDING → SUCCEEDED/FAILED/EXPIRED as you poll; anything left in PENDING after the 48-hour expiry window gets re-chunked smaller and resubmitted.
Submitting, Polling, and Retrieving Gemini Batch API
Submission is two calls — upload the JSONL file, then create the job — followed by polling until a terminal state:
TERMINAL = {"JOB_STATE_SUCCEEDED", "JOB_STATE_FAILED",
"JOB_STATE_CANCELLED", "JOB_STATE_EXPIRED"}
def await_results(job_name: str) -> list[dict]:
job = client.batches.get(name=job_name)
while job.state.name not in TERMINAL:
time.sleep(30)
job = client.batches.get(name=job_name)
ckpt = load_ckpt()
for batch_id, entry in ckpt.items():
if entry["job"] == job_name:
entry["state"] = job.state.name
save_ckpt(ckpt)
if job.state.name == "JOB_STATE_SUCCEEDED":
raw = client.files.download(file=job.dest.file_name).decode("utf-8")
# Each line is a response keyed by the "key" you set on the request.
return [json.loads(line) for line in raw.splitlines() if line.strip()]
if job.state.name == "JOB_STATE_EXPIRED":
raise RuntimeError(f"{job_name} expired after 48h — resubmit smaller")
raise RuntimeError(f"{job_name} failed: {job.error}")
If you’d rather not poll at all, the docs offer webhooks: subscribe to batch.succeeded and batch.failed events and let the API push completions to your endpoint. That converts your worker from a poller into an event handler — the cleaner design once the pipeline is in production. Results for file-based jobs come back as a downloadable JSONL file where each line pairs your request key with a GenerateContentResponse; inline jobs return dest.inlined_responses instead, which is convenient for small structured-output runs like the recipe-style examples in the docs.
Putting It All Together
The full nightly pipeline: the producer reads yesterday’s backlog (say, 50,000 documents to enrich) and pushes rows into the queue; the worker drains 400-item windows, writes each window to a deterministic JSONL batch, uploads, creates the job, and checkpoints the job name; a poller (or webhook handler) flips checkpoints to terminal states and merges downloaded results back onto the source rows by key; anything FAILED or EXPIRED is re-chunked at half size and resubmitted. One evening’s report:
batch_id items state turnaround input_tok output_tok std_cost batch_cost
9f2c11a4 400 SUCCEEDED 3h 12m 8,412,900 1,004,331 $5.03 $2.52
b71e0d55 400 SUCCEEDED 2h 47m 8,399,112 998,402 $5.02 $2.51
3aa9f70c 400 EXPIRED — — — — — -> re-chunk at 200
Total for the night: ~$10 in batch spend for work that would have cost ~$20 standard — the same $1,460/year arithmetic from the top of this post, now attached to job ids you can audit.
Conclusion and Next Steps
You now have the full batching discipline: the throughput-per-dollar math that justifies it, a clear line between provider batch, Flex, and self-coalescing, token-aware chunking that respects output ceilings, an explicit list of paths that must never batch, a checkpointed producer/consumer that can’t double-submit, and a working google-genai submission-poll-retrieve loop with expiry handling.
Homework, in order of leverage:
- Measure your real turnaround distribution for a week — the docs say 24-hour target, “much quicker” in most cases; your percentile decides whether batch can absorb your eval runs.
- Replace polling with
batch.succeededwebhooks and delete the sleep loop. - Route interactive failures to batch as a degraded mode: when the online path times out, enqueue instead of erroring.
- Try
client.batches.create_embeddings(...)for embedding backfills — same 50% economics, and vectorizing a corpus is the classic offline job. - Add per-batch cost reconciliation (batch cost vs the standard-price estimate) to your nightly report so the savings stay a measured fact, not a hope.