Assume vulnerability: monitoring microservice behavior for security

In a post on the Kubernetes Blog in January 2023, David Hadas from IBM Research Labs argued that developers must accept their microservices are inherently vulnerable. He proposed shifting focus from building impenetrable systems to monitoring behavioral anomalies that signal exploitation attempts.

What happened

Cybersecurity investments have grown annually, yet the number of successful cyber incidents continues to rise. Hadas noted that this trend suggests traditional strategies focused on eliminating every weakness are failing. Offensive tools are becoming more sophisticated, and financial incentives for attackers ensure they will always find an entry point if one exists. Consequently, relying on the creation of a completely non-vulnerable service is no longer a viable strategy.

The article posits that organizations should consciously admit their services contain unknown weaknesses. Instead of trying to remove all vulnerabilities, which is often impossible in practice, teams should focus on preventing those vulnerabilities from being exploited. If an attacker cannot successfully leverage a weakness, the risk remains theoretical rather than realized. This mindset shift moves the defense perimeter from code integrity to runtime behavior.

Hadas introduced the concept of "Security-Behavior Analytics" as the primary mechanism for this new approach. By analyzing how clients interact with services and how services respond, teams can detect irregularities that indicate an attack is in progress. This method does not require knowing the specific vulnerability beforehand; it only requires recognizing that the current interaction deviates from expected norms.

How it works

Security-behavior monitoring relies on the predictability of microservice interactions. In a well-designed system, clients send regular, structured requests, and services respond in consistent ways. An exploit, such as an SQL injection, forces the system to behave irregularly. For example, a malicious client might send a username containing special characters like spaces or equal signs, which benign users never do. Similarly, the service might take longer to respond or return unusually large datasets when processing such a request.

By monitoring these deviations, security tools can block attacks at multiple stages. Client-side monitoring detects irregular request patterns before they reach the application logic. Service-side monitoring identifies abnormal internal calls, response times, or data outputs. Combining both layers creates a robust defense that makes many vulnerabilities unexploitable, even if the underlying code remains flawed. The attacker must craft an exploit that mimics normal behavior perfectly, which is significantly harder than finding a code bug.

This approach is particularly effective in microservice architectures compared to monoliths. Monolithic applications intertwine various functions, making it difficult to distinguish between different types of requests and internal behaviors. Microservices, by design, have bounded contexts and clear interfaces. This modularity exposes internal traffic and defines strict expectations for each component, making it easier for observers to spot anomalies in specific services without noise from unrelated processes.

Key details

• The article identifies four stages of service life requiring different monitoring strategies: normal operation, known CVE presence, active exploit availability, and pod misuse.
• During the "Vulnerable" stage, when a CVE is published but patching takes weeks, monitoring can block requests matching the specific vulnerability pattern.
• In the "Exploitable" stage, tools can filter incoming traffic based on known exploit signatures to prevent execution.
• If an offender misuses a pod, the system can identify the compromised instance and restart it while keeping healthy pods running.
• Guard, an open-source project under the CNCF Knative project, is cited as a tool that provides this standalone security-behavior monitoring for Kubernetes HTTP workloads.
• Microservice architecture is inherently better suited for this monitoring because its modular nature exposes clear boundaries and predictable interaction patterns.

Why it matters

For software engineers and technical leads, this perspective reduces the pressure to achieve perfect code security, which is often an unrealistic goal. Instead, it emphasizes operational resilience. By accepting that vulnerabilities will exist, teams can prioritize building detection and response mechanisms into their infrastructure. This aligns with Zero Trust principles, where trust is never assumed, and verification is continuous.

Implementing behavior-based security also changes how teams view incident response. Rather than waiting for a patch to fix a known CVE, they can deploy behavioral rules to mitigate the threat immediately. This allows services to remain online and functional while security teams address the root cause. It transforms security from a gatekeeping function into a continuous operational safeguard.

Furthermore, this approach leverages the existing benefits of microservices. Teams already invest in observability and monitoring for performance reasons. Extending these tools to include security-behavior analytics adds a layer of protection without requiring a complete architectural overhaul. It turns standard telemetry data into a security asset, making the investment in observability pay double dividends.

What you can do

• Audit your current monitoring setup to see if it captures detailed request and response metadata needed for behavioral analysis.
• Define baseline behaviors for your critical microservices, including typical request structures, response times, and data volumes.
• Implement alerts for deviations from these baselines, such as unusual character sets in input fields or spikes in response payload size.
• Explore tools like Guard from the Knative project to add dedicated security-behavior monitoring to your Kubernetes clusters.
• Develop playbooks for restarting compromised pods automatically when misuse is detected, ensuring service continuity.
• Shift security reviews to include behavioral risk assessments, not just static code analysis, to identify potential exploitation paths.