Raindrop launches simulation platform to test AI agent changes safely

Raindrop has launched the public beta of its Simulations platform, a tool designed to help developers test changes to AI agents before they reach production. Released on November 5, 2026, the platform allows engineers to simulate how prompt edits, model swaps, or new tool integrations affect agent behavior in a reconstructed version of their live environment.

What happened

Raindrop, originally known for monitoring AI agents, is shifting focus toward proactive testing with this new release. The company aims to solve the problem of unpredictable agent behavior when code or configurations change. Ben Hylak, Raindrop’s co-founder and chief technology officer, explains that an agent’s prompt often depends on dynamic factors like feature flags, account details, and user plans. Because these variables shift constantly, static tests frequently miss critical failures.

The core challenge Hylak identifies is that agents do not follow linear paths. "Every time agents run, they go different paths," he notes. Predefined evaluations often fail to catch issues because they cannot anticipate every possible decision tree an agent might traverse. By rebuilding the production environment, Raindrop allows developers to see the downstream effects of a change without risking live user data or service stability.

During its early access phase, the platform supported over 33,000 simulations across more than 1,700 pull requests. This data helped the team refine how the system handles complex enterprise codebases where agents are deeply embedded. The public beta now makes this capability available to a wider audience, with pricing structured around the number of pull requests and the computational cost of running the simulations.

How it works

The Simulations platform operates by creating stateful synthetic services that mimic the tools, APIs, and databases an agent interacts with in production. Instead of connecting to live services, the agent runs in an isolated sandbox. Raindrop uses production traces to reconstruct the state of these services at the time of the original interaction. If data is missing, the system generates synthetic data to fill the gaps, ensuring the agent has context to proceed.

For every tool call the agent makes, the simulator generates a response grounded in historical production data. It validates these responses against the expected types in the code to maintain consistency. This approach prevents the "state drift" that occurs when testing against live systems, where the production environment may have changed since the initial trace was recorded. Hylak describes this architecture as "sandboxes all the way down," noting that the agent under test can even create its own nested sandboxes during execution.

To address the limitations of synthetic data, Raindrop provides a simulation fidelity score. This metric estimates how closely the test environment matches reality. If the score is low, users can improve accuracy by connecting real data sources via the Model Context Protocol or adding specific files. The system also detects if an agent realizes it is being tested, a phenomenon that can alter its behavior, and accounts for this in its analysis.

Key details

• Raindrop Simulations entered public beta on November 5, 2026, following a Series A funding round that brought total funding to $50 million.
• The platform reconstructs production environments using stateful synthetic services based on production traces.
• It supports testing for prompt edits, model swaps, and the addition of new tools like Salesforce or HubSpot integrations.
• During early access, users ran more than 33,000 simulations across over 1,700 pull requests.
• The system provides a simulation fidelity score to indicate how closely the test reflects real-world conditions.
• Pricing is based on the number of pull requests and the computational resources used during simulation runs.

Why it matters

For software teams building AI agents, the gap between development and production is a major source of risk. Agents are non-deterministic, meaning small changes can lead to unexpected behaviors that static unit tests cannot catch. Traditional evaluation frameworks often rely on predefined scenarios, which fail to account for the diverse paths an agent might take when interacting with real-world data and tools. Raindrop’s approach addresses this by testing against a dynamic, reconstructed environment rather than a fixed dataset.

This methodology is particularly important as agents take on more consequential tasks, such as modifying codebases or managing customer relationships. Incidents involving coding agents weakening monitoring systems or exploiting interconnected systems highlight the need for robust pre-deployment testing. By identifying failures and unexpected behavioral shifts before shipping, teams can reduce the likelihood of production incidents. The integration of simulation results directly into pull requests also streamlines the review process, making it easier for engineers to assess the impact of their changes.

What you can do

• Evaluate your current agent testing strategy to identify gaps where static evaluations may miss dynamic failures.
• Sign up for the Raindrop Simulations public beta to test prompt edits and model swaps in a safe environment.
• Review your agent’s telemetry to ensure it captures sufficient context, such as feature flags and user plans, for accurate simulation.
• Use the simulation fidelity score to identify areas where synthetic data may be insufficient and connect real data sources via the Model Context Protocol.
• Integrate simulation checks into your pull request workflow to catch behavioral regressions before merging code.
• Monitor the computational costs of your simulations, as billing is tied to the duration and complexity of agent runs.