Raindrop simulates your agent’s world to catch what your changes break

Raindrop, an agent monitoring startup, is now offering a Simulations platform in public beta to help developers understand the potential consequences of changes to their agents. Co-founder and CTO Ben Hylak explains that an agent's prompt can depend on various factors, including feature flags, account details, and the user's plan. To evaluate an agent thoroughly, Hylak emphasizes the need to "put the agent in motion."

Raindrop's Simulations platform creates a production-like environment, including tools, APIs, and databases, so developers can observe how a change impacts the agent's behavior before deploying it. Agent runs can take different paths, and predefined tests may miss unforeseen issues, especially as agents tackle more critical tasks.

The company reconstructs an agent's environment as a set of stateful synthetic services, using production traces to generate responses for calls to tools and APIs. Raindrop runs tests in isolated sandboxes, ensuring they don't access live services or credentials. The platform generates synthetic data when necessary, but Hylak acknowledges that it may not be as faithful as real-world data.

To increase confidence in the simulations, Raindrop provides a fidelity score and flags factors that reduce it, allowing customers to enhance the test conditions.

Raindrop also suggests pull requests to improve the agent's telemetry, helping it record context that may have been missing. The platform runs multiple simulations to account for normal variation between runs and compare findings with production history. Raindrop's approach is applicable to agents across various frameworks and codebases, with results delivered via pull requests.

During early access, Raindrop ran over 33,000 simulations across more than 1,700 pull requests, demonstrating the effectiveness of its simulations in detecting failures and unexpected changes in agent behavior.