Detecting container drift in Kubernetes with admission controllers

In a post on the Kubernetes Blog in December 2021, engineers at Box described a system they built to detect and manage container drift caused by interactive commands. The team developed a custom admission controller and a corresponding kubectl plugin to identify pods altered via kubectl exec or attach and enforce automatic eviction policies.

What happened

Box runs hundreds of microservices on Kubernetes to handle petabyte-scale data streaming. Their deployment workflow relies on GitOps, using kube-applier to apply declarative configurations from a Git repository after code reviews and automated checks. This process ensures that all changes to production environments are tracked, reviewed, and consistent with the desired state defined in code.

However, developers can bypass these controls by using interactive commands like kubectl exec to modify running containers directly. These ad-hoc changes create "container drift," where the live state of a container diverges from its declared configuration. Such modifications subvert change control processes and allow impacted containers to continue serving traffic in production without oversight.

To address this security and operational gap, Box created kube-exec-controller, a custom Kubernetes component, along with a kubectl plugin named kubectl pi. This system detects when developers interact with running containers, labels the affected pods, and eventually evicts them to restore the declared state. The project was open-sourced to help other teams manage similar risks in their clusters.

How it works

The solution leverages Kubernetes admission controllers, which intercept requests to the API server before objects are persisted. Specifically, the team used a ValidatingAdmissionWebhook configured to watch for CONNECT operations on pods/exec and pods/attach resources. When a developer runs an interactive command, the webhook sends the request to the kube-exec-controller service for validation.

The controller does not block the initial request, as immediate denial would hinder debugging. Instead, it permits the connection and asynchronously labels the target pod with metadata such as the interactor’s username and a timestamp. It also logs a warning event on the pod. A separate process within the controller tracks a time-to-live (TTL) timer for each labeled pod. Once the TTL expires, the controller evicts the pod. Eviction is preferred over deletion because it respects PodDisruptionBudgets, ensuring that service availability is maintained during the cleanup process.

To improve visibility and usability, Box developed the kubectl pi plugin. Since Kubernetes events are retained for only one hour by default, the plugin reads the labels and annotations attached by the controller to provide human-readable information about pod interactions. It includes a get subcommand to view interaction details and an extend subcommand. The extend command allows developers to request more time before eviction by updating pod annotations, which resets the eviction timer after passing through another validating webhook.

Key details

• The system uses a ValidatingAdmissionWebhook to intercept pods/exec and pods/attach CONNECT requests.
• Affected pods are labeled with metadata including the interactor’s username and initial interaction timestamp.
• Pods are evicted after a predefined TTL, which varies by environment (longer for dev, shorter for production).
• Eviction respects PodDisruptionBudgets to prevent service outages during forced restarts.
• The kubectl pi plugin provides get and extend subcommands for viewing status and requesting TTL extensions.
• The project, named kube-exec-controller, was open-sourced on GitHub by Box.

Why it matters

For teams building software on Kubernetes, maintaining the integrity of the deployed state is critical for security and reliability. Interactive access to containers is a common necessity for debugging, but it introduces untracked changes that can lead to configuration drift. Without mechanisms to detect and revert these changes, clusters can accumulate inconsistencies that make troubleshooting difficult and increase security risks.

This approach demonstrates how admission controllers can be used not just for policy enforcement, but for operational hygiene. By automating the detection and remediation of drift, teams can allow necessary debugging access while ensuring that production systems eventually return to their declared, reviewed state. It balances developer flexibility with platform governance.

The use of eviction rather than immediate termination highlights a mature understanding of production constraints. Respecting PodDisruptionBudgets ensures that security measures do not inadvertently cause downtime. This pattern is applicable to any organization using Kubernetes where strict change management is required alongside active development and support workflows.

What you can do

• Evaluate your current Kubernetes security policies to see if interactive commands like kubectl exec are unrestricted.
• Consider implementing a validating admission webhook to monitor and log pods/exec and pods/attach requests.
• Define clear TTL policies for drifted pods based on environment, with stricter limits for production clusters.
• Use pod eviction instead of deletion when cleaning up drifted containers to respect disruption budgets.
• Develop or adopt kubectl plugins to provide developers with visibility into pod interaction history and eviction timers.
• Review the open-source kube-exec-controller project on GitHub to understand the implementation details and adapt them to your needs.