Scaling Beyond the Single Agent: Distributed Gemini 3 Clusters on Kubernetes

Single-agent demos hit a wall: one Gemini 3 agent serves one conversation. Production needs a workforce — hundreds of concurrent sessions, each an agent with a growing context, pausing for tool calls, resuming, and holding a KV cache in memory. That is a distributed-systems problem, and Kubernetes is where it gets solved.

The core tension: agentic inference is stateful and bursty, while Kubernetes wants stateless, evenly-scheduled workloads. Every agent pause for a tool call leaves a KV cache idle in GPU memory. Naive round-robin routing across replicas throws away cached context. Scaling is no longer “add another replica” — it’s orchestrating prefill, decode, routing, and cache together. Here’s how.

Why Agentic Serving Breaks Simple Deployments

Traditional LLM serving treats each request as independent: route it to any replica, generate, return. Agentic serving breaks that assumption:

• Sessions are long-lived — one agent conversation is many calls with a shared, growing context.
• Tool calls pause sessions — while an agent waits on a database query, its KV cache sits idle (a “memory tax”).
• Cache locality matters — resuming on a node without the cached context means recomputing the whole prefix.

Existing engines schedule at the request level, where each LLM call is an independent unit with no awareness that it belongs to a longer, multi-turn workflow. — ThunderAgent (https://www.together.ai/blog/thunderagent)

The Disaggregated Architecture

The standard fix is disaggregated prefill and decode — separate the compute-heavy prompt processing from the memory-heavy token generation:

• Prefill nodes — high-compute machines (H100s, TPUs) that ingest prompts and build the KV cache tensor quickly.
• Decode nodes — high-memory, scalable nodes optimized for sequential generation.
• Router — stateful routing that sends each request to the node holding its warm cache.

flowchart LR
    C["Agent clients"] --> R["Inference Gateway"]
    R --> P["Prefill workers"]
    P --> D["Decode workers"]
    D --> T["External tools"]
    T --> D
    D --> C

Step 1 — Deploy with a KV-cache-aware router

The router’s job is affinity, not balance. Instead of round-robin, a router like llm-d’s Endpoint Picker (EPP) inspects each request and routes it to the specific pod holding the warm cache for that session. It maintains a real-time, globally consistent view of which token blocks live on which replicas (https://github.com/llm-d/llm-d).

An Envoy ext_proc filter delegates stateful routing intelligence to the EPP without adding data-plane overhead (https://medium.com/google-cloud/a-deep-dive-into-high-efficiency-agentic-serving-on-gke-with-vllm-and-llm-d-298379d93106).

Step 2 — Tier the KV cache

The “memory tax” fix is KV cache tiering. While the agent waits on an external tool call, vLLM offloads the session’s KV cache from GPU VRAM into cheap host CPU RAM (or a distributed tier), freeing the GPU to serve other requests. When the tool returns, the router finds the session, and vLLM swaps the context back and resumes seamlessly.

args:
  - "--swap-space=64"    # host CPU RAM for KV cache offloading during tool calls
  - "--kv-transfer-config={'kv_role':'kv_consumer','kv_connector':'PyNcclConnector'}"

For huge contexts that outgrow even CPU RAM, offload the shared, prefilled KV cache to a high-performance parallel filesystem (e.g., Managed Lustre) as a cluster-wide cache tier — one 2026 deployment cut GPU-hours for 70B inference on a six-node cluster by ~60% (https://cloud.google.com/blog/topics/developers-practitioners/scaling-llm-inference-multi-node-kv-cache-offloading-with-gke-managed-lustre).

Step 3 — Schedule like an application, not a bag of pods

Multi-node inference has completion constraints: a decode instance spanning four pods serves nothing until all four run together. Plain Kubernetes scheduling wastes GPUs on partial deployments.

The modern answer is workload-level abstractions that translate application intent into scheduling constraints:

• LeaderWorkerSet (LWS) — a leader pod coordinating worker pods as one logical unit (multi-node tensor parallel).
• NVIDIA Grove PodCliques — group pods by role; PodCliqueSet describes the full inference service (router + prefill + decode) with startup order; PodCliqueScalingGroup keeps role ratios when scaling (https://blog.aks.azure.com/2026/06/02/dynamo-on-aks-part-4).
• Gang scheduling (KAI Scheduler) — all-or-nothing placement: a full TP group schedules together or not at all. Hierarchical gang scheduling extends it to the service level — at least one prefill + one decode + router must be schedulable before anything is ready.

Topology matters too: pack tightly-coupled pods on the same rack with high-bandwidth interconnects (NVLink, RoCE) so KV transfer between prefill and decode happens at wire speed.

Step 4 — Autoscale per role, not per replica

Prefill and decode bottleneck differently and should scale independently:

• Scale prefill on Time-To-First-Token (TTFT).
• Scale decode on Inter-Token Latency (ITL).
• Scale the router when throughput grows.

Naive HPA on individual Deployments can’t preserve the prefill:decode ratio, so application-level autoscalers coordinate. llm-d’s Workload Variant Autoscaler monitors per-pod KV cache utilization and queue depth, emitting target replica counts that KEDA/HPA actuate. NVIDIA Dynamo’s planner runs separate TTFT/ITL loops, predicts demand with time-series models, and enforces a global GPU budget across roles.

Program-Aware Scheduling: The Agentic Frontier

Even disaggregated serving has a blind spot: it schedules requests, not workflows. A new wave of research treats each agent workflow as a schedulable program, tracking its execution phase, KV footprint, and node placement:

• Admission control — when a node is under memory pressure, pause low-priority workflows instead of letting everyone thrash.
• Global waiting queue — resumed workflows route to the node with the most capacity, balancing cache locality against cluster balance.

ThunderAgent reports >2x single-node throughput and near-linear multi-node scaling (2.4x on 8 nodes) with this approach — a single program_id field on each request is the only client change (https://www.together.ai/blog/thunderagent).

Putting It All Together

To run distributed Gemini 3 agents on Kubernetes:

1. Disaggregate — separate prefill nodes (compute) from decode nodes (memory).
2. Route by affinity — KV-cache-aware routing to the warm-cache replica.
3. Tier the cache — offload idle session state to CPU RAM or a parallel filesystem.
4. Schedule as workloads — LWS/PodCliques + hierarchical gang scheduling + topology-aware placement.
5. Autoscale per role — TTFT/ITL loops with a coordinated global GPU budget.
6. Think in programs — treat the agent workflow, not the request, as the scheduling unit.

This composes with standard primitives — Gateway API, Custom Resources, HPA — so it fits any existing Kubernetes platform.

Conclusion & Next Steps

You’ve learned to scale beyond a single Gemini 3 agent: disaggregate prefill and decode, route by KV-cache affinity, tier memory through tool-call pauses, gang-schedule multi-node instances, and autoscale per role under SLOs.

To go further:

• Prefix caching — hash and reuse shared prefixes across sessions to eliminate redundant prefill.
• Multi-tenant fairness — protect interactive traffic from noisy neighbors with priority queues.
• Observability — trace tool calls and reasoning per session so you can see where tokens and GPU time go.

One agent is a proof. A cluster is a product. When agentic traffic is the workload, the old “stateless replicas behind a load balancer” model silently burns GPU hours. Disaggregation, cache-aware routing, and workload-level scheduling are how you make the workforce real — at Gemini 3 scale.