AI Workloads on Kubernetes: GPUs, Inference, and Serving

CPU pods are fungible; GPU pods are not. One mis-scheduled training job can strand an $8,000 accelerator for six hours, and one inference deployment that grabs a whole A100 when it needs an eighth of one burns money on every replica. Getting AI workloads onto Kubernetes is therefore mostly a resource-expression problem: how do you tell the scheduler what a workload needs from a device that has attributes, partitions, and failure modes no CPU has ever had?

That question has been answered twice. First by the device plugin framework, which is what most clusters still run, and then — as of Kubernetes v1.34, where the core of Dynamic Resource Allocation graduated to GA — by a proper claim-based API. This post walks both, then covers slicing, node-pool hygiene, and the serving stacks on top.

Device plugins: the installed base

The device plugin framework has been stable since v1.26 and works by extending the kubelet, not the scheduler: a vendor plugin (NVIDIA GPU Operator, AMD, Intel) registers over gRPC, advertises an extended resource such as nvidia.com/gpu, and the kubelet publishes it as node capacity. From a pod spec’s point of view it is just a resource:

resources:
  limits:
    nvidia.com/gpu: 1   # integer, no overcommit, limits-only rule applies

Two rules bite in practice: GPUs are integer extended resources that cannot be overcommitted, and you may only specify them in limits (a limits-equals-requests pair is allowed; requests-only is rejected). Fractional allocation is therefore impossible at the scheduler level — you either get whole devices or you get nothing.

DRA: claims, device classes, and CEL selectors

Dynamic Resource Allocation replaces “count of opaque devices” with a declarative claim model. The stable resource.k8s.io/v1 group (KEP-4381, GA in 1.34 and enabled by default) introduces four kinds:

Kind	Role
ResourceSlice	Driver publishes device inventory per node/pool
DeviceClass	Named category of devices plus selection policy (CEL expressions)
ResourceClaim	A workload’s request for allocated devices
ResourceClaimTemplate	Per-pod claims generated automatically

A pod stops naming a count and starts referencing claims:

apiVersion: resource.k8s.io/v1
kind: ResourceClaim
metadata:
  name: inference-gpu
spec:
  devices:
    requests:
      - name: compute
        exactly:
          deviceClassName: nvidia-gpu
          allocationMode: ExactCount
          count: 1
---
apiVersion: v1
kind: Pod
metadata:
  name: qwen-inference
spec:
  resourceClaims:
    - name: compute
      resourceClaimName: inference-gpu
  containers:
    - name: server
      image: vllm/vllm-openai:latest

What you gain over integer resources is expressiveness: CEL selectors can filter on device attributes (memory capacity, compute capability, MIG profile), firstAvailable (DRAPrioritizedList) lets a job say “one H100, or failing that two L4s,” and device taints let admins drain problematic accelerators without cordoning whole nodes. For AI fleets this is the difference between “schedule anywhere with a GPU” and “schedule on a GPU with 80 GB free of MIG partitions.”

GPU sharing: MIG, time-slicing, MPS

When one accelerator must serve several consumers, three mechanisms exist with very different guarantees:

• MIG (Multi-Instance GPU). Hardware partitioning supported on A100, A10, H100/H200, and L40S. Profiles like 1g.10gb, 3g.40gb, 7g.80gb give each slice its own memory, L2 cache, and compute — real isolation. The NVIDIA device plugin exposes them as nvidia.com/mig-1g.10gb, and the limits-only integer rule applies to each slice. Cost: the GPU must be put into MIG mode, and partitions are fixed until reconfigured.
• Time-slicing. The device plugin’s sharing.strategy: timeSlicing config advertises one GPU as N replicas (optionally renamed, e.g. nvidia.com/gpu.shared). Contexts switch rapidly, so all replicas share memory bandwidth and — critically — share device memory with no isolation. A runaway replica can OOM the others. Cheap, flexible, unsafe for untrusted tenants.
• CUDA MPS. A middle path: multiple processes submit kernels to one GPU concurrently with better isolation than time-slicing and higher utilization than MIG for small kernels. Still no hard memory partition.

Rule of thumb: MIG for multi-tenant or bursty latency-sensitive inference, time-slicing for homogeneous batch replicas you control, MPS for latency-hiding on well-behaved CUDA services.

Node pools, taints, and separating training from inference

Accelerators deserve their own node pools, and the reason is lifecycle, not capacity:

# node label + taint: only workloads that opt in land here
nvidia.com/gpu.present: "true"
taint: accelerator=nvidia:NoSchedule

Inference pods tolerate the taint and request claims; training jobs do the same but land on a different pool with bigger GPUs and looser consolidation policies. Cloud tooling makes this explicit — EKS managed node groups or Karpenter nodepools with nvidia.com/gpu instance families, GKE autoscaled GPU node pools — and autoscalers must be told not to aggressively consolidate GPU nodes (a Karpenter consolidation that evicts a warm inference replica costs more in lost throughput than it saves).

The operational details that matter: install the NVIDIA driver stack as a DaemonSet (GPU Operator), keep CUDA-forward-compatibility images pinned to the node driver, and give GPU nodes their own disruption budgets — a cluster upgrade that drains eight inference replicas at once is an outage.

Scheduling: inference vs training are different workloads

Inference wants spread, fast scale-up, and graceful degradation:

• Topology spread across nodes/zones so one node failure does not drop 40% of capacity.
• PriorityClass so interactive traffic preempts batch embeddings.
• Autoscaling on queue depth, not CPU: vllm:num_requests_waiting or gpu_cache_usage_perc exported by the server and consumed by KEDA.
• PodDisruptionBudgets that actually reflect replica count (minAvailable: 2 on a 3-replica deployment).

Training wants the opposite: all-or-nothing placement. A 64-GPU job that schedules 63 GPUs and waits is a 63-GPU idling bill. This is gang scheduling, and the current answer in-tree-adjacent is Kueue: jobs enter a queue, a cohort’s quota gates admission, and — with provisioning support — the autoscaler is triggered only when the whole gang can fit. The scheduling-eviction docs’ PodGroup APIs and topology-aware workload scheduling extend the same idea to gang-aware preemption and NUMA/GPU-local placement, which matters when NVLink locality decides whether your job trains at 90% or 40% efficiency.

Multi-node training additionally needs pod-level coordination for NCCL rendezvous (gloo/etcd or the elastic training operator’s rendezvous endpoints), shared /dev/shm sized for collectives, and node affinity pinning the job to a homogeneous instance type — NCCL across mixed GPU generations is a debugging nightmare.

Serving stacks: KServe and vLLM

Serving is where the two workload profiles converge. KServe provides the InferenceService CRD with a predictor/transformer/explainer composition, traffic-splitting revisions for canary rollouts, and pluggable runtimes — its vLLM adapter manages model loading, GPU allocation, and autoscaling for you. Serving modes include RawDeployment (plain Deployments + HPA, simplest to operate) and Serverless/Knative (scale-to-zero, request-based concurrency autoscaling). ModelMesh complements it when you need many models per GPU with smart loading.

Underneath sits the engine: vLLM with PagedAttention and continuous batching typically delivers several times the throughput of naive one-request-at-a-time servers on the same hardware, and its --tensor-parallel-size maps onto a multi-GPU pod directly. The Kubernetes-specific lesson is that the engine’s memory flags (--gpu-memory-utilization, --max-model-len) are scheduling inputs: they determine how much of the device you actually occupy, and therefore how many replicas fit on a node.

The stack, bottom to top: DRA or device plugins expose accelerators → node pools and taints partition the fleet → Kueue/gang scheduling admits training, spread + PDBs protect inference → KServe and vLLM turn pods into endpoints that scale on the metric that matters.

References

• Kubernetes — Dynamic Resource Allocation (cited tutorial path has moved; concept docs are canonical)
• Kubernetes — Install Drivers and Allocate Devices with DRA (tutorial)
• Kubernetes Blog — Kubernetes v1.34: DRA has graduated to GA
• Kubernetes — Schedule GPUs (task)
• NVIDIA — GPU Operator and time-slicing configuration
• KServe · vLLM · Kueue