Simulating multi-tenant GPU sharing with Kamaji and HAMi

Platform engineers often face a difficult trade-off between resource efficiency and tenant isolation when managing GPU workloads. A recent demonstration by Anish Bista shows how combining Kamaji and HAMi allows teams to share physical GPUs safely while maintaining distinct control planes. This approach uses simulated hardware to validate the architecture before committing to expensive infrastructure.

What happened

The article details a reproducible lab environment built on a single Linux host using Docker containers. The goal was to prove that two open-source projects, Kamaji and HAMi, can work together to solve the problem of GPU underutilization in multi-tenant Kubernetes clusters. Traditionally, assigning a full GPU card to a pod that only needs a fraction of its memory leads to significant waste. Conversely, sharing a cluster among multiple teams often results in weak isolation and security risks.

Bista constructed a management cluster using kind (Kubernetes in Docker) to host Kamaji. Within this management layer, two separate tenant control planes were created as pods. Each tenant was given its own worker node, also running as a container, which joined the network and ran HAMi components. The setup included two simulated tenants: tenant-a with one simulated NVIDIA A100 and tenant-b with two simulated NVIDIA L4 cards. The lab successfully passed eight tests, confirming that each tenant could schedule fractional GPU workloads without seeing resources or pods belonging to the other.

How it works

Kamaji addresses the control plane overhead by running each tenant’s Kubernetes API server and etcd datastore as pods inside a central management cluster. This eliminates the need for dedicated control plane machines for every team. HAMi, on the other hand, operates at the scheduling and device level. It allows pods to request specific slices of a GPU, defined by memory in megabytes and compute percentage, rather than claiming an entire card. This enables multiple pods from different applications to coexist on a single physical GPU.

The critical integration point involves the communication path between the tenant’s API server and the HAMi webhook. Since the API server runs in the management cluster but the webhook resides on the tenant’s worker nodes, the connection must traverse a Konnectivity tunnel. The lab confirmed that this reverse proxy mechanism works correctly, allowing the API server to call back into the tenant environment for scheduling decisions. To test this without physical hardware, the author used HAMi’s mock device plugin, which skips driver interaction and registers fake GPU capacity via node annotations.

Key details

• The lab ran on a single host with 4 CPU cores, 8 GB of RAM, and 20 GB of disk space.
• Tenant-a was configured with one simulated 80 GB NVIDIA A100, while tenant-b had two simulated 24 GB NVIDIA L4 cards.
• IPv6 networking in kind caused MetalLB pool failures, requiring manual inspection of the IPv4 subnet to fix the gateway configuration.
• Mock GPU pods initially failed due to node lock contention because the mock plugin does not automatically release locks after allocation.
• Manual annotation of pods and nodes was required to simulate the device plugin’s completion step in the mock environment.
• The setup proves the wiring and scheduling logic but does not validate real-world memory limits or CUDA performance.

Why it matters

For software developers and platform engineers, this architecture offers a path to significantly reduce cloud costs associated with GPU instances. By enabling fractional allocation, organizations can pack more workloads onto fewer cards, improving utilization rates without sacrificing the security benefits of isolated control planes. The ability to test this entire stack using simulated hardware means teams can validate their platform design and CI/CD pipelines before purchasing expensive GPU nodes.

Furthermore, the successful integration of Kamaji and HAMi demonstrates that complex multi-tenant requirements do not necessarily require custom glue code or proprietary solutions. The use of standard tunnels like Konnectivity ensures that the control plane can securely manage distributed worker nodes. This modularity allows teams to adopt these tools incrementally, starting with simulation and moving to production hardware with confidence in the underlying scheduling mechanics.

What you can do

• Replicate the lab environment using kind and Kamaji to test multi-tenant control plane isolation on local hardware.
• Use HAMi’s mock device plugin to validate scheduling policies and quota logic without needing physical NVIDIA GPUs.
• Inspect Docker network subnets manually if you encounter IPv6-related issues with MetalLB in kind clusters.
• Implement manual lock release steps in testing environments to bypass mock plugin limitations during development.
• Verify Konnectivity tunnel connectivity between your management cluster API servers and tenant worker webhooks.
• Plan for real-device plugin integration early, as mock setups do not enforce actual memory or compute limits.