Volcano v1.16 adds namespace queues and fair-share scheduling

The Cloud Native Computing Foundation has released Volcano v1.16.0, a significant update to the Kubernetes-native batch scheduler. This version introduces new mechanisms for multi-tenant resource management, including namespace-scoped queue hierarchies and a fair-share scheduling plugin. It also decouples topology discovery from the core scheduler, allowing broader integration with other scheduling systems.

What happened

Volcano v1.16.0 focuses on improving fairness and scalability for clusters running heavy batch or AI workloads. The release adds two major features for resource governance: NamespaceQueue and the fairshare plugin. NamespaceQueue allows tenants to manage their own hierarchical queues and quotas within specific namespaces, rather than relying solely on cluster-wide administrators. The fairshare plugin introduces historical resource consumption into scheduling decisions, ensuring that teams using shared queues do not monopolize resources over time.

The update also expands the HyperNode ecosystem by making the HyperNode controller independently deployable. Previously, using HyperNode for topology discovery required running the full Volcano scheduler suite. Now, platforms using different schedulers can deploy only the HyperNode controller to manage supernodes and hierarchy, exposing this data via an API for external consumption. Additionally, the release includes performance enhancements such as an Unschedulable Job Cache, which reduces redundant scheduling checks for jobs that cannot currently run.

How it works

The new NamespaceQueue feature aligns queue management with Kubernetes namespace boundaries. Administrators authorize specific namespaces to attach their queue hierarchies to a parent cluster Queue. Within that boundary, tenants can define their own sub-queues with capabilities like guaranteed resources and reclaimable limits. This separation allows teams to manage their internal resource distribution without needing cluster-admin privileges for every change. The system ensures that different namespaces can use identical queue names without conflict, as identity is scoped to the namespace.

Fairness across shared queues is handled by the new fairshare plugin, which draws inspiration from Slurm’s fair-share scheduling. Instead of looking only at current allocation, the plugin tracks cumulative resource-seconds for each namespace. It applies an exponential decay to this history, with a default half-life of four hours. Namespaces with lower recent consumption are prioritized when competing for resources. This prevents a single team from dominating a shared queue by continuously submitting jobs, as their historical usage weight increases and eventually lowers their priority until their usage decays.

Performance improvements center on the Unschedulable Job Cache. When a job fails to schedule due to full queues or missing resources, the scheduler previously re-evaluated it in every cycle. The new cache records the specific reason for rejection and skips re-evaluation until a relevant event occurs, such as a node label change or resource release. A watchdog timer ensures cached jobs are retried periodically, preventing them from being stuck indefinitely if event hints fail. This reduces CPU overhead in clusters with large pending backlogs.

Key details

• NamespaceQueue: Enables tenants to manage hierarchical queues and quotas within authorized namespaces, separating resource pool authorization from daily tenant management.
• Fairshare plugin: Tracks historical resource consumption per namespace with exponential decay (default four-hour half-life) to improve long-term fairness in shared queues.
• Unschedulable Job Cache: Caches rejection reasons for pending jobs and skips redundant checks until triggering events occur, reducing scheduler load.
• Standalone HyperNode controller: Can be deployed independently of the Volcano scheduler, allowing other scheduling systems to use Volcano’s topology discovery and supernode management.
• PodGroup topology anti-affinity: New rules allow inference replicas to spread across different supernodes, reducing the risk of simultaneous failure in a single topology domain.
• Kubernetes 1.36 support: The release updates compatibility to Kubernetes 1.36 and includes improvements for Dynamic Resource Allocation (DRA) scoring and accelerator handling.

Why it matters

For engineering teams managing shared Kubernetes clusters, especially for AI training or high-performance computing, resource contention is a constant challenge. The introduction of NamespaceQueue empowers team leads to manage their own resource hierarchies without bottlenecking on central platform engineers. This decentralization speeds up operational workflows while maintaining guardrails through administrator-authorized parent queues. It bridges the gap between rigid cluster-level policies and the flexibility needed by agile development teams.

The fairshare plugin addresses a common pain point in multi-tenant environments: the "noisy neighbor" problem. By accounting for historical usage, the scheduler ensures that no single team can hog resources indefinitely, even if they have a high volume of short-lived jobs. This leads to more predictable access to compute resources for all teams. Meanwhile, the Unschedulable Job Cache directly impacts cluster stability and cost efficiency by reducing the control plane overhead associated with large pending job backlogs, allowing the scheduler to focus on jobs that can actually run.

What you can do

• Enable NamespaceQueue in your Helm chart by setting custom.namespace_queue_enable=true and defining allowed namespaces in your parent Queue specs.
• Configure the fairshare plugin in your scheduler configuration, placing it before the DRF plugin to influence pending-job ordering based on historical usage.
• Activate the Unschedulable Job Cache via the feature gate UnschedulableJobCache=true to reduce scheduler load in clusters with persistent pending jobs.
• Deploy the standalone HyperNode controller using custom.hypernode_controller_mode=standalone if you want to use Volcano’s topology discovery with a different scheduler.
• Update your PodGroup definitions to include topology anti-affinity rules if you need to spread inference replicas across different supernodes for higher availability.
• Review your CRDs and RBAC permissions before upgrading, as NamespaceQueue requires specific updates to support the new namespaced resources.