Kubernetes GPU scheduling that understands fairness, topology, and office politics

What it does KAI Scheduler is a Kubernetes scheduler that plugs into clusters to optimize GPU and hardware resource allocation for AI and machine learning workloads. It targets large-scale environments—thousands of nodes—and covers the full AI lifecycle, from small interactive jobs to distributed training and inference, while keeping resource allocation fair across different consumers. It can coexist with other schedulers already installed on the cluster.

The interesting bit Instead of treating fairness and topology as afterthoughts, the scheduler builds them into core placement decisions. It uses Dominant Resource Fairness paired with time-decay algorithms to stop any single team from monopolizing GPUs indefinitely, and Topology-Aware Scheduling places workloads with awareness of physical hardware layout—something the project highlights as critical for disaggregated serving stacks like NVIDIA Grove and Dynamo. Hierarchical PodGroups extend this to multi-level gang scheduling, letting complex distributed workloads schedule as atomic units across the cluster.

Key highlights

• Gang scheduling via batch and hierarchical PodGroups ensures multi-pod AI workloads land simultaneously or not at all.
• Topology-Aware Scheduling optimizes placement based on node topology, targeting disaggregated serving architectures such as Grove and Dynamo.
• Fairness policies include Dominant Resource Fairness and time-based fairshare with historical usage decay to balance quotas across organizational queues.
• Supports Dynamic Resource Allocation for vendor-specific hardware, including NVIDIA ComputeResources (GB200/GB300) and AMD GPUs.
• Elastic workloads, GPU sharing, and workload consolidation let operators trade off utilization against fragmentation and guaranteed runtimes.
• Native integrations include KubeRay and the Grove/Dynamo inference stack.

Caveats

• GPU scheduling currently requires the NVIDIA GPU Operator, with extra configuration flags needed for older OpenShift versions.
• The project maintains an explicit breaking-changes document and LTS policy, so upgrades may require migration planning.
• The kai-scheduler namespace is reserved for the control plane; submitting user workloads there is an operational anti-pattern.

Verdict Worth a look for platform teams managing large GPU clusters shared across multiple AI teams. Skip it if you are running CPU-only or small-scale workloads where the default scheduler suffices.