Building a Multi-Tenant AI Factory on Kubernetes

A GPU cluster can look healthy while quietly wasting the asset everyone thinks is scarce. One team holds eight GPUs for a training run stalled on data input. Another needs two MIG slices for evaluation. A third cannot deploy inference because a namespace-level quota says “no,” although physical capacity is idle. This is the failure mode a multi-tenant AI factory on Kubernetes is meant to eliminate.

The answer isn’t a larger pile of accelerators. It is a production line: explicit device claims, isolated tenant control planes, topology-aware placement, admission queues, and workload orchestration that agree on who gets capacity, when, and under what policy. Kubernetes v1.37 “Garhwal,” released on August 26, 2026, provides a notably strong control-plane base for that design.

Here is the practical architecture. NVIDIA DRA owns physical GPU discovery and allocation; vCluster gives each team a Kubernetes control plane; HAMi virtualizes heterogeneous accelerator capacity; KAI Scheduler places work with physical topology in mind; Kueue admits, borrows, and preempts capacity; Kubeflow turns the result into a coherent ML operating layer. Each component has a strict boundary. Blur those boundaries and a shared GPU platform becomes expensive folklore.

The control plane for a multi-tenant AI factory on Kubernetes

Kubernetes v1.37 “Garhwal” is not interesting because it adds one dramatic AI feature. It matters because 67 enhancements tighten the operational foundations that AI platforms tend to punish. The metrics.k8s.io API is Stable after roughly nine years in Beta. HPA scale-to-zero is Beta and enabled by default. StorageVersionMigration is Stable and enabled by default. Resilient watch-cache initialization is complete. Sober improvements. Exactly what an over-subscribed GPU platform needs.

Scale-to-zero deserves special attention. Inference endpoints and evaluation services often reserve GPU capacity long after useful traffic disappears. With HPA scale-to-zero, a service can contract to nothing and return when demand resumes. Don’t mistake this for instant GPU availability: model loading, image pulls, claim allocation, and scheduler delay remain on the critical path. It is still a decisive cost-control primitive when paired with warm-image caches and explicit startup SLOs.

Run containerd v2.3 LTS or evaluate the 2.4 release targeted for August 26, 2026. Containerd’s April, August, and December minor-release cadence makes version planning less opaque. More important, the runtime must support Container Device Interface (CDI) and be configured to use CDI for DRA-allocated devices. GPU injection cannot be an undocumented node tweak. It is part of the platform contract.

Think of the physical arrangement as a vertical stack. Bare-metal or cloud nodes sit at the base, grouped by GPU model, memory, NVLink domain, and failure zone. Above them, the host Kubernetes v1.37 cluster runs NVIDIA GPU Operator, Node Feature Discovery, DCGM Exporter, HAMi, KAI Scheduler, and Kueue. vClusters sit beside tenant namespaces as separate API-server-shaped control planes. Kubeflow workloads enter through those tenant APIs, then become synchronized host-cluster workloads subject to queue admission and device allocation.

A tenant should experience a cluster. The platform must experience a carefully governed pool of finite, topologically uneven devices.

NVIDIA DRA: make GPU requests precise, not hopeful

Dynamic Resource Allocation changes the unit of conversation from “I need four nvidia.com/gpu devices” to a typed, inspectable claim. With the NVIDIA DRA Driver for NVIDIA GPUs v0.5.0, workloads can use DeviceClass and ResourceClaim objects to select GPUs by model, architecture, memory size, compute capability, UUID, and topology through CEL expressions. Large-model training needs that precision. Four arbitrary GPUs are not interchangeable with four devices on the same NVLink island.

The GPU Operator manages DRA through a cluster-scoped singleton GPUCluster custom resource named gpu-cluster. It also coordinates DCGM, DCGM Exporter, the DRA validator, and, where required, ComputeDomains. Enable draDriver.computeDomains only for NVIDIA Grace Blackwell multi-node NVLink fabrics such as NVIDIA HGX GB200 NVL72. Enabling fabric-oriented machinery on ordinary PCIe nodes is the wrong default: it adds operational surface without creating locality.

The architectural fork is non-negotiable. NVIDIA’s older Device Plugin path uses ClusterPolicy; the DRA path uses GPUCluster. They cannot coexist. A fresh AI factory should use GPUCluster and DRA from day one, provided its nodes satisfy the prerequisites: Kubernetes v1.34.2 or later with resource.k8s.io DeviceClass support, NVIDIA drivers version 580 or later, and CDI-capable container runtime configuration.

Claims, MIG, and sharing policy

DRA supports full GPUs and preconfigured MIG devices. DynamicMIG is Alpha and can create MIG devices when a claim arrives instead of requiring the operator to carve every GPU in advance. That flexibility is attractive, but it has a price: repartitioning is a control-plane event with hardware implications, not a cheap scheduler label change. Reserve DynamicMIG for pools with repeatable profiles and a tested drain procedure. For latency-sensitive inference, fixed MIG profiles are usually easier to reason about.

ConsumableShares, also Alpha, introduces scheduler-accounted sharing among ResourceClaim objects. TimeSlicingSettings can define custom slice intervals for shared claims. These capabilities solve a real utilization problem, yet they should not be handed to every workload class. A distributed fine-tuning job that depends on predictable collective communication and a bursty notebook session should not receive identical sharing semantics. I’d rather strand a modest amount of capacity than turn critical training latency into an untraceable fairness experiment.

apiVersion: resource.k8s.io/v1
kind: ResourceClaim
metadata:
  name: trainer-gpu-claim
spec:
  devices:
    requests:
    - name: accelerators
      deviceClassName: nvidia-gpu-80gb
      allocationMode: ExactCount
      count: 4

The claim is intentionally only one layer of intent. The DeviceClass should encode the admissible device family and topology constraints, while Kueue and KAI Scheduler decide whether the work may enter the cluster now and where it can run. Keep policy outside individual notebook manifests whenever possible. Otherwise, every team becomes a part-time capacity planner. That never ends elegantly.

vCluster, HAMi, and Kueue: separate isolation from sharing

Isolation, quotas, and scheduling turn one accelerator pool into fair shared capacity.

Namespaces are useful administrative boundaries. They are not independent Kubernetes control planes. vCluster provides each tenant with its own API server, controller manager, data store, syncer, scheduler, CRDs, and RBAC configuration while sharing host nodes. NVIDIA’s tenant pattern uses a host GPU cluster plus KAI Scheduler and a vCluster agent layer, with clusters such as team-nlp, team-vision, and team-recommender. It is a clean fit for trusted internal groups.

Shared nodes don’t satisfy every threat model. Where tenants are untrusted or compliance boundaries are strict, use vCluster private nodes with node, network, and storage separation. Treat isolation as four distinct controls: tenant control-plane boundaries, pod and runtime boundaries, network and data boundaries, and allocation boundaries. A separate API server does not prevent permissive network paths; a ResourceQuota does not prevent a vulnerable image from becoming a lateral-movement route.

HAMi adds the device virtualization layer. As an incubating CNCF project, it supports GPUs, NPUs, MLUs, and other accelerators through Kubernetes APIs, DRA, CDI, and extended resources. It can enforce memory and core quotas with runtime isolation, then apply binpack, spread, or topology-aware placement. This is useful in a mixed fleet, but don’t treat all accelerators as fungible. A workload image, kernel set, numerical behavior, and communication library may be vendor-specific even when the scheduler sees “accelerator capacity.”

Kueue supplies the admission control that vCluster and HAMi deliberately do not. It manages standard Job, JobSet, KubeRay, Kubeflow Trainer, and LeaderWorkerSet workloads using multi-tenant quotas, dynamic capacity sharing, topology-aware placement, resource pools and preemption, plus multi-cluster scaling. A ClusterQueue establishes the shared contract; LocalQueues expose it to tenant namespaces; cohort membership permits borrowing when peers are idle.

apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: gpu-research
spec:
  cohort: shared-gpu
  namespaceSelector: {}
  resourceGroups:
  - coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
    flavors:
    - name: h100-80gb
      resources:
      - name: nvidia.com/gpu
        nominalQuota: 16
        borrowingLimit: 8

The resource name above illustrates a migration concern, not a recommendation to bypass DRA. Many queue policies still need to account for familiar resource dimensions while actual device placement moves toward ResourceClaims. Model the relationship explicitly and test it under contention. Kueue admission, DRA claims, HAMi slices, and KAI Scheduler decisions must converge on one capacity view, or the factory will accept work it cannot place.

Scheduling Kubeflow workloads through the GPU factory

Kubeflow’s CNCF graduation on August 17, 2026 matters because the project now provides a mature operating layer across data processing, interactive development, distributed training, fine-tuning, inference, and model serving. It is where users should express ML work. It should not become another scheduler hidden behind a friendly user interface.

A disciplined execution path looks like this. A Kubeflow Trainer or pipeline creates a queued workload in the tenant vCluster. The syncer reflects the relevant workload to the host cluster. Kueue evaluates quota, cohort borrowing, priority, flavor, and preemption rules. Once admitted, KAI Scheduler selects nodes based on GPU locality and sharing policy. NVIDIA DRA resolves the ResourceClaim; HAMi applies the required slice or accelerator virtualization; CDI injects the allocated devices into the container. DCGM Exporter and OpenTelemetry then expose what actually happened.

KAI Scheduler is the placement specialist in this arrangement. It understands GPU sharing, queue semantics, and physical topology. Use it to co-locate communication-heavy ranks within an NVLink domain, while spreading independent inference replicas across devices and failure domains. Kueue decides whether a job deserves admission. KAI Scheduler decides its best physical landing point. Combining those responsibilities in one blob of custom admission logic is tempting and difficult to operate.

Benchmark the seams, not only model throughput

A credible validation suite measures more than tokens per second. Capture queue wait time by LocalQueue and priority; ResourceClaim allocation latency; pod start time after admission; GPU memory and utilization from DCGM; MIG-slice contention; training all-reduce latency by topology; inference cold-start duration after HPA scale-to-zero; and preemption recovery time. Seven measurements. Each identifies a different failure boundary.

Instrument Kubeflow components with OpenTelemetry as code is written, using traces, logs, and metrics. Emit dataset version, training epoch, hyperparameter revision, claim name, queue name, and model artifact identifier as structured attributes where security policy permits. Correlate those signals with DCGM GPU-seconds and OpenCost records. Without correlation, chargeback becomes arguments over dashboards, and utilization becomes a vanity metric.

Policy belongs in the same operating model. Enforce Pod Security Standards, NetworkPolicy, hardened container images, dedicated GPU nodes, ResourceQuota, and explicit queue policies. Add Kyverno or OPA for admission rules, OpenBao for secrets, Falco or Trivy for runtime and image signals, Keycloak for identity, and Prometheus for platform metrics. CNCF’s AI factory composition also points to Gateway API and Envoy for traffic, Cilium and SR-IOV for networking, Rook and CSI for storage, and KServe or vLLM for serving. Don’t install every project. Make interfaces and owners explicit instead.

The operating standard: govern capacity before demand arrives

Build the first production slice with a narrow hardware matrix, fixed MIG profiles, two or four tenant vClusters, one Kueue cohort, its LocalQueues, and a small set of approved Kubeflow workload templates. Prove admission, borrowing, preemption, tracing, and GPU billing under synthetic contention before adding multiple accelerator vendors or cross-cluster dispatch. It’s fine for a prototype, not for a fleet, to add that complexity before controller boundaries have been tested.

A mature AI factory balances utilization, resilience, and governance at every layer.

Kubernetes v1.37, NVIDIA DRA Driver for NVIDIA GPUs v0.5.0, vCluster, HAMi, KAI Scheduler, Kueue, and graduated Kubeflow provide the materials for a serious shared GPU platform. The remaining test is architectural discipline: when four teams ask for the same GPU pool, can the platform explain every allocation, every denial, every borrowed slice, and every millisecond of waiting? If it can, you have an AI factory. If it cannot, you have hardware with a queue in front of it.