GPU Scheduling in Kubernetes: How Device Plugins Actually Work

You deploy an inference service that requests one GPU, and the pod sits Pending forever. The node clearly has GPUs - nvidia-smi works, the drivers are loaded - but the scheduler behaves as if the machine is empty.

The gap is almost never in the scheduler itself. It sits one layer down: nothing has told the kubelet that those GPUs exist as schedulable resources, so nothing ever reaches the API server’s view of node capacity. Understanding that chain - plugin registration, resource advertisement, then pod admission - is the whole job of GPU scheduling on Kubernetes.

This post walks the chain end to end using the official device plugin and GPU scheduling documentation, then works through the failure modes that come up in practice.

What the device plugin framework is

Kubernetes provides a device plugin framework that lets you advertise system hardware resources to the kubelet. Instead of modifying Kubernetes itself, vendors implement a device plugin that you deploy either manually or as a DaemonSet. The targeted devices include GPUs, high-performance NICs, FPGAs, InfiniBand adapters, and other computing resources that need vendor-specific initialization.

The framework has been stable since Kubernetes v1.26, for both the GPU scheduling story and the plugin framework itself. Two things follow from that:

1. The kubelet, not the scheduler, is the source of truth. The scheduler only ever sees what the kubelet reports during its node status update.
2. The plugin API itself is versioned separately and, as the docs are careful to point out, is not stable even though the Device Manager component is. Plugin authors are explicitly told to watch for API changes and to support multiple versions.

So the resource name you request in a pod spec - nvidia.com/gpu, amd.com/gpu - is an extended resource naming scheme of the form vendor-domain/resourcetype, invented by a plugin process and published by the kubelet.

Registration: the gRPC handshake

The kubelet exports a small gRPC service for plugins to introduce themselves:

service Registration {
	rpc Register(RegisterRequest) returns (Empty) {}
}

During registration, the plugin sends three things: the name of its Unix socket, the Device Plugin API version it was built against, and the ResourceName it wants to advertise. For an NVIDIA GPU that last one is nvidia.com/gpu.

After a successful registration, the plugin sends the kubelet the list of devices it manages, and the kubelet takes charge of advertising those resources to the API server as part of the kubelet node status update. The docs’ example is concrete: if a plugin registers hardware-vendor.example/foo and reports two healthy devices, the node status is updated to advertise two available “Foo” devices.

The runtime side of the plugin is a second gRPC service on a Unix socket under /var/lib/kubelet/device-plugins/. That path is hardcoded and is not affected by the kubelet’s --root-dir or any other configuration. The interface has five methods, and only two are load-bearing for daily operation:

service DevicePlugin {
      rpc GetDevicePluginOptions(Empty) returns (DevicePluginOptions) {}
      rpc ListAndWatch(Empty) returns (stream ListAndWatchResponse) {}
      rpc Allocate(AllocateRequest) returns (AllocateResponse) {}
      rpc GetPreferredAllocation(PreferredAllocationRequest) returns (PreferredAllocationResponse) {}
      rpc PreStartContainer(PreStartContainerRequest) returns (PreStartContainerResponse) {}
}

ListAndWatch is a stream: whenever a device changes state or disappears, the plugin pushes a new list to the kubelet. Allocate is called during container creation so the plugin can run device-specific preparation - GPU cleanup, for example - and return the container runtime configuration needed to access the device: annotations, device nodes, environment variables, mounts, and fully-qualified CDI device names.

The ordering rule matters when you write your own plugin or debug someone else’s: a plugin MUST start serving its gRPC service before registering with the kubelet. Serve first, then register at /var/lib/kubelet/device-plugins/kubelet.sock.

What you can and cannot request

Once the plugin is up, GPUs look like any other resource in a pod spec - with restrictions. Extended resources are only supported as integer resources and cannot be overcommitted, and devices cannot be shared between containers. For GPUs specifically, the scheduling docs add a rule that surprises people coming from CPU and memory:

What you write	Result
GPU in limits only	Valid. Kubernetes uses the limit as the request value by default.
GPU in both limits and requests	Valid only if the two values are equal.
GPU in requests only	Rejected. You cannot specify GPU requests without limits.

The practical reading: a GPU request is all-or-nothing per container. There is no fractional GPU admission at the scheduler level, and asking for limits: 2 means two whole devices are reserved for that container.

A minimal pod requesting one GPU:

apiVersion: v1
kind: Pod
metadata:
  name: example-vector-add
spec:
  restartPolicy: OnFailure
  containers:
    - name: example-vector-add
      image: "registry.example/example-vector-add:v42"
      resources:
        limits:
          gpu-vendor.example/example-gpu: 1 # requesting 1 GPU

Deploying the plugin itself

As an administrator, you install GPU drivers from the hardware vendor on the nodes and run the vendor’s device plugin. The docs link to the AMD, Intel, and NVIDIA implementations; once installed, the cluster exposes a custom schedulable resource such as amd.com/gpu or nvidia.com/gpu.

The deployment shape that wins in practice is the DaemonSet. You can deploy a device plugin as a DaemonSet, as a package for your node’s operating system, or manually, and the DaemonSet gives you placement, restart after failure, and automated upgrades for free. Two requirements come with it: the canonical directory needs privileged access, so the plugin must run in a privileged security context, and /var/lib/kubelet/device-plugins must be mounted as a volume in the plugin’s pod spec.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nvidia-device-plugin
spec:
  selector:
    matchLabels:
      app: nvidia-device-plugin
  template:
    metadata:
      labels:
        app: nvidia-device-plugin
    spec:
      containers:
        - name: nvidia-device-plugin
          image: nvcr.io/nvidia/k8s-device-plugin:v0.17.0
          securityContext:
            privileged: true
          volumeMounts:
            - name: device-plugin
              mountPath: /var/lib/kubelet/device-plugins
      volumes:
        - name: device-plugin
          hostPath:
            path: /var/lib/kubelet/device-plugins

One restart behavior is worth memorizing: a new kubelet instance deletes all existing Unix sockets under /var/lib/kubelet/device-plugins when it starts. A plugin detects its own socket disappearing and re-registers with the new kubelet. If your GPUs vanish from capacity right after a kubelet upgrade or restart, you are watching this mechanism, and a plugin that does not handle it is your bug.

Pinning pods to the right GPU type

Heterogeneous clusters - some A100 nodes, some L4 nodes - need a second mechanism on top of the extended resource. If different nodes have different types of GPUs, use node labels and node selectors to schedule pods to appropriate nodes:

# Label your nodes with the accelerator type they have.
kubectl label nodes node1 accelerator=example-gpu-x100
kubectl label nodes node2 accelerator=other-gpu-k915

For fleets large enough that hand-labeling rots, deploy Node Feature Discovery (NFD). NFD detects the hardware features available on each node and typically advertises them as node labels, but it can also add extended resources, annotations, and node taints, so only pods that request those features land on those nodes.

When a label selector is not expressive enough, node affinity covers it. This example from the docs combines a memory threshold with an NFD PCI feature label:

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: "gpu.gpu-vendor.example/installed-memory"
            operator: Gt # (greater than)
            values: ["40535"]
          - key: "feature.node.kubernetes.io/pci-10.present" # NFD Feature label
            values: ["true"]
  containers:
    - name: example-vector-add
      image: "registry.example/example-vector-add:v42"
      resources:
        limits:
          gpu-vendor.example/example-gpu: 1

Node selection and taints solve different problems. Node affinity - including the simple nodeSelector field - attracts a pod to nodes with certain labels. Taints are the opposite: they let a node repel a set of pods, applied with kubectl taint nodes node1 key1=value1:NoSchedule, with tolerations written into the PodSpec of the workloads allowed to ignore them. A common GPU node setup is a NoSchedule taint on accelerator nodes so that CPU-only system pods never consume an expensive machine, plus a toleration in the inference workload.

Be aware of one bypass: if you manually set .spec.nodeName on a pod, that action bypasses the scheduler entirely - the pod binds even to a node with NoSchedule taints. If that node also carries a NoExecute taint, the kubelet will eject the pod anyway.

Failure modes and what they look like

Most GPU paging incidents land in one of these buckets:

Symptom	Likely cause	First check
Pod Pending, no node matches	Plugin never registered, or driver install failed	kubectl describe node for nvidia.com/gpu in capacity
Node shows GPU capacity but allocatable is lower	Devices marked unhealthy by ListAndWatch	Node allocatable vs. capacity; plugin logs
Pod scheduled, container crashes on start	Allocation or driver mismatch inside the container	Container events, Allocate errors in plugin logs
GPUs disappear after node upgrade	Kubelet restarted, plugin did not re-register	Socket deletion handling in the plugin
Works on one node type, Pending on another	Heterogeneous cluster without labels/selectors	Node labels vs. pod affinity rules

The allocatable-vs-capacity row is the subtle one. When a device fails or shuts down, the plugin notifies the kubelet through ListAndWatchResponse; the kubelet then decreases the allocatable count for the resource to reflect how many devices can be used for scheduling new pods, while the capacity count does not change. Pods already assigned to the failed device stay assigned to it - the scheduler does not rescue them. The workload’s code simply starts failing, and the pod may enter the Failed phase if the restart policy is not Always, or crash-loop otherwise.

To find out whether a pod is associated with a failed device, before v1.31 you had to use the PodResources API - a kubelet gRPC service with List, GetAllocatableResources, and Get endpoints, stable since v1.28 and the intended interface for monitoring agents. With the ResourceHealthStatus feature gate - beta and enabled by default since v1.36 - each container status gains an allocatedResourcesStatus field reporting health information for every device assigned to that container, including an optional human-readable message. An accelerator reporting an over-temperature event can show up right there in the pod status.

On the API compatibility side, kubelet upgrades are supposed to be seamless since the version-matching requirement was dropped at Beta in v1.12, but the device plugin API is not stable, so the guidance is to upgrade device plugins to support both versions before upgrading nodes that run them.

Where Dynamic Resource Allocation fits

The device plugin path is one of two ways to get devices into pods. Dynamic Resource Allocation (DRA) is the newer one: stable since Kubernetes v1.35, it lets workload operators claim devices by creating ResourceClaims or ResourceClaimTemplates, filtered by DeviceClasses using Common Expression Language selectors. Kubernetes and the device drivers then find available devices, allocate them to pods, and place the pods on nodes that can access them.

The trade-off is operational. nvidia.com/gpu in a pod spec is one line and everyone on the team already reads it. A ResourceClaim is a second object with its own lifecycle: if you directly reference a specific ResourceClaim in a pod, that claim must already exist in the cluster, and if it does not, the pod remains Pending until the claim is created. For new clusters it is worth evaluating; for existing GPU fleets the device plugin model remains the path with the least surprise.

Checklist

• Install vendor GPU drivers and the vendor device plugin before writing any pod spec; the resource does not exist until the plugin registers.
• Verify nvidia.com/gpu (or amd.com/gpu) appears in node capacity and allocatable after plugin rollout.
• Specify GPUs only under limits, or under both limits and requests with equal values; requests alone are invalid.
• Deploy the plugin as a DaemonSet with a privileged security context and the /var/lib/kubelet/device-plugins hostPath mounted.
• Label heterogeneous GPU nodes - by hand or with Node Feature Discovery - and pair labels with node selectors or node affinity.
• Taint accelerator nodes with NoSchedule and tolerate explicitly in GPU workloads; avoid setting .spec.nodeName by hand.
• Treat a drop in allocatable (not capacity) as unhealthy devices, and read allocatedResourcesStatus on v1.36+ clusters when debugging pod failures.
• Re-check plugin re-registration after every kubelet restart or node upgrade, and keep plugin API versions aligned with the nodes they run on.

Sources: Device Plugins - Kubernetes Documentation, Schedule GPUs - Kubernetes Documentation, Assigning Pods to Nodes - Kubernetes Documentation, Taints and Tolerations - Kubernetes Documentation, Allocate Devices to Workloads with DRA - Kubernetes Documentation