How Kubernetes CRI handles exec, attach, and port forwarding

In a post on the Kubernetes Blog in May 2024, Sascha Grunert detailed the internal mechanics of three specific Container Runtime Interface (CRI) remote procedure calls. The article explains how Exec, Attach, and PortForward differ from standard gRPC interactions by using a distinct URL-based streaming model that has remained consistent since its design in 2016.

What happened

The Kubernetes CRI serves as the primary bridge between the kubelet and container runtimes, requiring runtimes to expose a gRPC server that adheres to a defined Protocol Buffer interface. While most CRI operations rely on simple unary calls or server-side streaming, Grunert highlighted that Exec, Attach, and PortForward operate differently. These three functions are critical for developers who need to run commands inside containers, view live output, or forward network ports for debugging.

Grunert traced the history of these features back to a 2016 design document that predated modern Kubernetes Enhancement Proposals. Before the CRI initiative, these capabilities were tightly bound to specific runtimes like Docker or rkt. The community considered implementing native RPC streaming but rejected it because it would create network bottlenecks in the kubelet and restrict runtime flexibility. Instead, they adopted a model where the runtime provides a streaming server, allowing each implementation to manage connections independently.

This architectural decision means that while the initial request goes through the standard gRPC interface, the actual data transfer happens over a separate HTTP connection. This separation allows runtimes to evolve their streaming implementations without modifying the core CRI definition. Although minor enhancements have been merged over the years, the fundamental pattern of requesting a URL and then connecting to it directly has remained unchanged.

How it works

The process begins when a client, such as kubectl or crictl, sends a gRPC request to the runtime for an Exec, Attach, or PortForward session. Unlike typical API calls that return data directly, the runtime validates the request and stores it in a connection tracking cache. It then returns a response containing only a fully qualified URL. The client must then connect to this URL, upgrading the connection to use either the SPDY protocol or, increasingly, WebSockets, to begin streaming data.

For Exec and Attach, Kubernetes defines a specific protocol with five versions, currently up to v5.channel.k8s.io. This protocol uses the first byte of every packet to identify the stream type, such as standard input, standard output, standard error, or control signals like terminal resize and close. The kubelet source code provides a reusable library that handles this protocol interpretation, requiring runtimes only to implement the logic for executing commands or attaching to processes. PortForward operates differently, as it lacks a strict protocol definition. Instead, the runtime enters the container’s network namespace and streams raw SPDY frames, relying on libraries like moby/spdystream to manage the data flow.

Key details

• The CRI Exec, Attach, and PortForward RPCs return only a URL string in their response, not the actual data stream.
• Clients must upgrade the HTTP connection to SPDY or WebSockets to establish the streaming session after receiving the URL.
• The Exec and Attach protocols use the first byte of each packet to distinguish between stdin, stdout, stderr, errors, resize events, and close signals.
• Five versions of the remote command protocol exist, with v5 adding support for a CLOSE signal for WebSockets.
• The kubelet provides a reusable library with a Runtime interface that runtimes must implement to handle the underlying command execution and network namespace entry.
• Future efforts focus on replacing SPDY with WebSockets, with tools like crictl v1.30 already supporting a --transport flag to choose between them.

Why it matters

For software engineers building or maintaining container runtimes, understanding this architecture is essential for correct implementation. The decoupling of the control plane (gRPC) from the data plane (HTTP streaming) means that runtime developers cannot simply treat these calls as standard API requests. They must manage concurrent streaming sessions, handle protocol upgrades, and ensure compatibility with multiple protocol versions. Misinterpreting the first byte of a data packet or failing to support terminal resize events can lead to broken user experiences in common development workflows.

This design also impacts how debugging tools interact with clusters. Because the data bypasses the kubelet after the initial URL retrieval, network policies and proxies must allow direct communication between the client and the node hosting the container. As the ecosystem moves toward WebSockets, engineers must ensure their tooling supports the new transport layer. The ongoing work in projects like CRI-O, which moves streaming logic to conmon-rs, demonstrates how this flexibility allows runtimes to keep sessions alive even if the main runtime process restarts, improving reliability for long-running debug sessions.

What you can do

• Verify that your container runtime supports the latest v5.channel.k8s.io protocol to ensure proper handling of stream closure signals.
• Update client tools like crictl to version 1.30 or later to test WebSocket transport support using the --transport flag.
• Review your network policies to ensure that clients can reach the node IPs and ports used for streaming connections, not just the API server.
• If you are developing a custom runtime, use the kubelet’s reusable streaming library to handle protocol parsing rather than implementing it from scratch.
• Monitor the adoption of WebSockets in your environment, as SPDY is being phased out in favor of more modern web standards.
• Check if your runtime can offload streaming sessions to external monitors like conmon-rs to improve resilience during runtime updates.