Building dynamic Kubernetes APIs without custom controllers

In a post on the Kubernetes Blog in November 2024, Andrei Kvapil from Ænix detailed how his team built a dynamic extension API server for Cozystack. The article explores the technical implementation of the Kubernetes API aggregation layer, contrasting it with standard Custom Resource Definitions (CRDs) and operators.

What happened

Kvapil explained that while most Kubernetes extensions rely on CRDs and controllers, this approach has limitations for specific use cases. Standard operators excel at declarative state reconciliation but struggle with imperative logic, real-time data generation, or complex validation requirements. To address these gaps, the Cozystack team implemented a custom extension API server that integrates directly into the Kubernetes API framework via the aggregation layer.

The core of this implementation involves registering an APIService object within the cluster. This registration tells the main Kubernetes API server to proxy requests for specific resource groups to the external extension server. For Cozystack, this allowed the platform to dynamically expose new resource types based on available Helm charts without requiring code changes or recompilation. The system maps user-friendly kinds, such as Postgres or Redis, directly to underlying Helm releases, abstracting complexity from end users.

This approach also solved significant RBAC challenges. Standard Kubernetes Role-Based Access Control cannot filter list operations by labels or specific spec fields, only by resource names. By generating distinct resource kinds for each service type, Cozystack could leverage native RBAC policies to restrict access precisely. Additionally, the extension server handles two-way conversion between the new custom kinds and the internal HelmRelease resources, ensuring backward compatibility with existing dashboards and tools.

How it works

The API aggregation layer acts as a proxy within the Kubernetes control plane. When a user sends a request to the Kubernetes API for a resource group served by an extension, the main API server forwards that request to the extension API server. This server operates independently of the primary control plane components and can implement its own business logic, validation, and storage mechanisms.

Unlike standard controllers that sync state to etcd, an extension API server can generate responses on the fly. This is similar to how the metrics-server works, fetching real-time data from Kubelets rather than storing it. In Cozystack’s case, the server dynamically discovers available services and registers them as API resources. It validates inputs, converts them into HelmRelease objects, and submits them to the cluster, all while maintaining a clean separation between the user-facing API and the internal implementation.

Key details

• The API aggregation layer allows extension servers to handle imperative logic and subresources like /exec or /log.
• Extension APIs are registered via an APIService object, which directs traffic for specific API groups to the external server.
• Unlike CRDs, extension servers do not need to store state in etcd, enabling real-time data generation and reduced storage overhead.
• Cozystack uses this model to dynamically map service kinds like Postgres to underlying Helm charts without recompiling code.
• The implementation supports complex server-side validation and custom table output formatting beyond CRD capabilities.
• Unstable extension servers can block namespace deletion or cause API latency, so reliability is critical.

Why it matters

For platform engineers, understanding the aggregation layer opens up design patterns that are difficult or impossible with CRDs alone. It enables the creation of APIs that behave like native Kubernetes resources but operate with imperative logic or external backends. This is particularly useful for integrating legacy systems, exposing real-time metrics, or creating simplified interfaces for complex applications.

However, this power comes with operational risks. If an extension API server becomes unavailable, it can degrade the entire cluster’s performance. Operations like namespace deletion may hang while waiting for the extension to confirm resource cleanup. Engineers must weigh the benefits of custom API behavior against the added complexity and potential stability impacts of running additional API servers.

What you can do

• Evaluate whether your use case requires imperative logic or real-time data before choosing between CRDs and an extension API.
• Use kubectl get apiservices to inspect currently registered aggregation layers in your cluster.
• Implement robust health checks and timeouts for any extension API server to prevent cluster-wide blocking.
• Leverage the aggregation layer for subresources like logs or exec proxies where standard CRUD operations do not apply.
• Consider using extension servers to expose external databases or services as native Kubernetes resources for easier management.
• Test failure scenarios thoroughly, especially namespace deletion and API discovery, to ensure cluster stability.