Building secure AI agents with kagent on Kubernetes

Developers can now run kagent, a CNCF Sandbox project, on local laptops to build AI agents that operate as Kubernetes resources. Published on October 12, 2026, by Antonio Berben and Felipe Vicens, this guide demonstrates how to create an agent builder that generates other agents safely. The process relies on strict guardrails rather than trusting the large language model implicitly.

What happened

The article details a step-by-step lab for setting up an agent builder using kagent. Instead of treating agents as standalone processes, kagent defines them as custom resources in Kubernetes. This approach allows an existing agent to generate YAML manifests for new agents, effectively creating a self-service builder. The setup uses open-source tools including Docker, kind, kubectl, helm, jq, and Ollama to keep costs minimal and avoid cloud dependencies.

A critical part of the setup involves configuring the tool server securely. By default, kagent charts might grant cluster-admin privileges, which is dangerous even for local testing. The guide instructs users to restrict permissions so the builder agent can only create specific resource types. This ensures that the generated agents operate within a defined scope, preventing unauthorized access to sensitive cluster components like secrets or deployments.

The demonstration walks through installing the necessary infrastructure, selecting a capable local model, and interacting with the builder via a user interface or API. The goal is to show that building agents is not magic but a structured engineering task. It emphasizes that while the initial setup is straightforward, the real challenge lies in managing governance and security as these agents scale beyond a single laptop environment.

How it works

The system relies on three distinct layers of security that do not depend on the prompt alone. First, an allowlist restricts which tools the model can see. Out of twenty-two available tools, the builder only accesses three. Second, a human-in-the-loop mechanism requires user approval before any tool execution that modifies state. When the agent proposes a change, the run pauses, allowing the user to review the exact manifest before it applies. Third, Role-Based Access Control (RBAC) at the Kubernetes level ensures that even if a manifest is approved, the API server rejects any action outside the service account’s permissions.

The builder agent operates by receiving a natural language request, such as creating an agent for support teams. It then generates a Kubernetes manifest for a new Agent resource. This manifest includes a system prompt, model configuration, and a restricted set of read-only tools. The builder does not execute the creation directly; instead, it outputs the YAML for review. Once approved, the kagent controller applies the manifest, spinning up a new pod that speaks the Agent-to-Agent (A2A) protocol.

This architecture treats prompts as requests rather than guarantees. The actual enforcement comes from the infrastructure. For example, if the model hallucinates a tool name or attempts to create a deployment instead of an agent, the allowlist or RBAC policies block the action. This separation of concerns ensures that errors in model output do not lead to security breaches or unintended cluster modifications.

Key details

• The recommended local model is qwen3-coder:30b-a3b, as smaller models often fail to generate valid Kubernetes manifests.
• The builder uses an allowlist of three tools out of twenty-two available on the server to limit its capabilities.
• Human approval is required for any tool call that changes state, providing a checkpoint before execution.
• RBAC policies restrict the builder’s service account to creating only Agent resources, blocking access to secrets or deployments.
• The setup uses kind for the Kubernetes cluster and Ollama for local model inference to minimize costs.
• The UI runs on localhost via kubectl port-forward, ensuring it is not exposed to the public internet during testing.

Why it matters

For software engineers and technical leads, this approach shifts the focus from prompt engineering to infrastructure design. Relying solely on prompts for safety is insufficient because models can misunderstand instructions or hallucinate tool names. By embedding security into the Kubernetes layer, teams can build agents that are robust against model errors. This method provides a clear audit trail and enforceable boundaries, which are essential for production environments.

The article also highlights the gap between local experimentation and enterprise governance. On a laptop, all actions appear to come from the same user, making it difficult to track costs or ownership. In a corporate setting, multiple agents and frameworks may coexist, each with different security postures. This complexity argues for a centralized governance layer that proxies traffic, handles authentication, and provides observability across all agent interactions.

Understanding these mechanics helps developers move beyond simple demos to building reliable systems. It demonstrates that AI agents can be integrated into existing DevOps workflows using familiar tools like YAML and kubectl. This lowers the barrier to entry for teams already invested in Kubernetes, allowing them to leverage AI without adopting entirely new operational paradigms.

What you can do

• Install Docker, kind, kubectl, helm, jq, and Ollama to prepare your local environment for the lab.
• Pull the qwen3-coder:30b-a3b model using Ollama to ensure reliable manifest generation.
• Create a kind cluster named kagent-builder and install kagent with restricted RBAC policies.
• Use kubectl port-forward to access the kagent UI at localhost:8080 and test the agent builder.
• Review the generated manifests carefully before approving them to understand what will be applied.
• Explore agentgateway as a potential governance layer for managing authentication and observability in larger setups.