Running LLM Inference Services in Containers

An LLM server is an unusual container: it can take minutes to become ready, it holds gigabytes of state in GPU memory before it answers its first request, and its CPU usage stays flat while its queue grows. Everything Kubernetes does well by default — fast startup, CPU-based readiness, CPU-based autoscaling — works against you here. This post walks the workload concepts you need, then the practical details of running vLLM, Ollama, and llama.cpp servers in production containers.

Grounding: what kind of workload is this?

The Kubernetes workloads concepts split naturally: pods are the unit, Deployments manage stateless replicas, StatefulSets manage stable identity, Jobs run once. An LLM service is a stateless Deployment in shape but stateful in behavior — each replica holds the same loaded weights (state you restore, not store) and must not be considered healthy until loading finishes. Two consequences:

1. The pod’s lifecycle has three distinct phases: pull image → load model (long) → serve. The workload docs’ startup probe exists exactly for this: give it a generous failureThreshold × periodSeconds window (e.g. 60 × 5s = 5 minutes) so a slow model load is not mistaken for a dead server.
2. Replica count is not capacity. A Deployment with 4 replicas of a 70B model needs 4 GPUs worth of memory whether traffic is zero or full. Scaling decisions are queue decisions, not CPU decisions.

Choosing an image

Three ecosystems cover most deployments:

# vLLM — OpenAI-compatible API, continuous batching, PagedAttention
docker run --gpus all -p 8000:8000 \
  -v /models:/models \
  vllm/vllm-openai:latest \
  --model /models/Qwen2.5-72B-Instruct-AWQ \
  --gpu-memory-utilization 0.92 \
  --max-model-len 16384 \
  --tensor-parallel-size 2

# Ollama — single-binary, model registry, good for internal tools
docker run -d --gpus all -p 11434:11434 \
  -v ollama:/root/.ollama \
  ollama/ollama

# llama.cpp server — CPU/edge-friendly, GGUF quantization
docker run -d -p 8080:8080 \
  -v /models:/models \
  ghcr.io/ggml-org/llama.cpp:server \
  -m /models/llama-3.1-8b-instruct.Q4_K_M.gguf \
  --host 0.0.0.0 --port 8080 -ngl 99

The choice maps to hardware: vLLM when you have datacenter GPUs and need throughput; llama.cpp when the target is a CPU box, an edge node, or a cheap dev environment; Ollama when the consumer is humans and internal tools rather than a production API. NVIDIA NIM and Hugging Face TGI containers occupy the same middle ground as vLLM with different tuning knobs.

Batching and the KV cache

The throughput of an LLM server is decided by how well it batches. Static batching waits for a fixed group of requests — simple, but one slow request stalls everyone. Continuous batching (vLLM’s core contribution with PagedAttention) admits a new request the moment a slot frees, treating sequences as variable-length pages in a shared KV cache rather than pre-allocated slots.

You control this with flags, not YAML:

• --max-num-seqs — concurrent sequences (batch width).
• --max-num-batched-tokens — tokens processed per step (batch depth; this is the real memory knob).
• --gpu-memory-utilization — the fraction of device memory the engine may claim.

Sizing the KV cache is arithmetic you should do before picking a replica count. Per token, per sequence:

KV bytes = 2 (K and V) × num_layers × num_kv_heads × head_dim × dtype_bytes

For Llama-3-8B (32 layers, 8 KV heads, head_dim 128, FP16): 2 × 32 × 8 × 128 × 2 = 131,072 bytes ≈ 128 KiB per token. A 8,192-token context therefore costs ≈ 1 GiB per concurrent sequence — and that is on top of the weights. Budget:

GPU memory = weights + (max_model_len × max_num_seqs × KV per token) + activations/workspace

A 70B model in FP16 is ~140 GB of weights alone (hence --tensor-parallel-size 2 on 80 GB cards); with 32 concurrent 16K sequences you have just added ~5 GiB of KV. Get this wrong and the server either refuses to start (gpu-memory-utilization exhausted) or dies mid-serving when an outlier request hits the context limit.

Quantization trade-offs

Quantization trades quality for the two scarce resources: memory and bandwidth.

Format	Bits/weight (8B model)	Quality	Where it wins
FP16/BF16	16 (~16 GB)	Baseline	Max throughput on A100/H100 (tensor cores)
INT8 (SmoothQuant)	8 (~8 GB)	Very good	Large-batch serving, near-free on newer GPUs
AWQ / GPTQ 4-bit	4 (~4–5 GB)	Good, task-dependent	Single-GPU serving of 30–70B models
GGUF Q4_K_M (llama.cpp)	~4.5 bpw	Good	CPU/edge, Apple Silicon, mixed hardware

Two rules of thumb: on NVIDIA GPUs, prefer AWQ/GPTQ 4-bit over GGUF for serving — the kernels (Marlin, exllama-class) keep 4-bit throughput surprisingly close to FP16 at large batch, while GGUF’s strength is portability, not datacenter speed. And always benchmark on your task (retrieval QA, code completion) rather than trusting a generic eval: quantization degrades long-tail reasoning and factual recall more than it degrades perplexity.

Health checks for models

Standard liveness probes will lie to you. A liveness HTTP check against a vLLM server that is 30 seconds into loading a 70B model either fails (killing a process that was about to work) or, if you point it at a port that answers before the model is ready, passes while the service cannot serve.

The correct pattern uses all three probes:

startupProbe:            # absorbs the model-load window
  httpGet: { path: /health, port: 8000 }
  failureThreshold: 60
  periodSeconds: 5
readinessProbe:          # gates traffic only when generation works
  httpGet: { path: /health, port: 8000 }
  periodSeconds: 5
livenessProbe:           # conservative; only restarts a wedged process
  httpGet: { path: /health, port: 8000 }
  periodSeconds: 30
  failureThreshold: 3

Each server exposes something usable: vLLM’s /health returns 200 once the engine is up, llama.cpp server has a /health endpoint that reports ready only when the model is loaded, and Ollama is often probed via ollama list in an exec probe or its /api/tags endpoint. The deeper point: readiness should mean “can accept a completion request,” not “process is running.”

Autoscaling on queue depth

CPU is useless as a scaling signal for an LLM server — a saturated GPU looks like a mostly-idle pod. The signal that works is the engine’s own queue:

# KEDA ScaledObject: scale on vLLM's waiting-request gauge
- type: prometheus
  metadata:
    query: avg(vllm:num_requests_waiting{namespace="llm"})
    threshold: "4"

vLLM exports vllm:num_requests_waiting and vllm:gpu_cache_usage_perc on its /metrics endpoint; Ollama and llama.cpp can be wrapped with a queue-depth exporter or scaled on request concurrency through a gateway metric. Pair this with a minReplicaCount of at least 2 (a cold replica costs a model load — minutes, not milliseconds) and a scale-down cooldown long enough that a traffic dip does not immediately evict warm weights.

Resource requests and limits that actually work

resources:
  requests:
    cpu: "4"
    memory: 32Gi
    nvidia.com/gpu: "1"   # limits-only for GPUs: add under limits too
  limits:
    nvidia.com/gpu: "1"
    memory: 40Gi          # CPU/memory limit; leave headroom for KV spikes

Details that separate a working deployment from a debugging session:

• GPU requests follow the limits-only rule — requests-only is rejected by the API, and there is no fractional GPU at the scheduler level.
• CPU/memory requests matter even with a GPU. Tokenization, HTTP serving, and detokenization run on CPU; under-provisioned CPU manifests as inexplicable tail latency, not crashes.
• Multi-GPU tensor parallel needs shared memory. Set shm_size (or an emptyDir: medium: Memory volume at /dev/shm) sized for NCCL buffers, or rendezvous will hang or fall back to slow paths.
• Load weights through an init container or a PVC, not at first request: pull a 40 GB model from a registry inside the serving container and your startup probe budget doubles.
• Keep model downloads out of liveness windows — an init container that fetches the weights from an internal mirror (or a PVC populated by a Job) makes pod startup deterministic.

An LLM service is a normal Kubernetes workload with three non-normal properties: startup measured in minutes, memory measured in weights-plus-KV arithmetic, and scaling measured in queue depth. Get those three right and the rest — Deployments, probes, services, rollouts — behaves exactly as the docs promise.

References

• Kubernetes — Workloads (cited source)
• Kubernetes — Liveness, Readiness, and Startup Probes
• vLLM — Quickstart and metrics (vllm:num_requests_waiting, vllm:gpu_cache_usage_perc)
• llama.cpp server · Ollama
• KEDA — Prometheus scaler
• NVIDIA — AWQ and quantized inference on TensorRT-LLM / vLLM