Gremlin now uses AI to break distributed systems faster

Gremlin, a startup that specializes in chaos engineering, has introduced AI-powered Foresight AI to its services. This new add-on autonomously breaks software infrastructure to uncover hidden bugs before they cause issues in production. Chaos engineering, which involves deliberately introducing controlled failures into working systems to test their resilience, has been popularized by Netflix more than a decade ago.

Gremlin's AI, Foresight AI, automates many of the preparatory and post-testing tasks that were previously done manually, speeding up the process considerably. Unlike other AI solutions that diagnose issues after a system has crashed, Gremlin's AI purposefully induces failures and then provides expert guidance on how to prevent them from happening again.

Gremlin's Failure Atlas, a repository of millions of chaos engineering experiments, helps the AI provide more technically grounded answers about what went wrong. The AI uses a variety of open and closed large language models (LLMs) to generate solutions, keeping the system on point and minimizing hallucinations. Gremlin's existing guardrails ensure that the blast radius is kept to a minimum, and the AI determines the root cause, generates code or configuration changes to fix the issue, and prepares a report for site reliability engineers (SREs) to read.

This loop of test-and-replace continues until the failure is no longer induced. Gremlin's current user base consists of larger, compute-heavy enterprises, particularly those in financial services, retail, and enterprise SaaS. These companies use Gremlin to test disaster recovery plans, ensure correct operation under less-than-ideal conditions, and verify Kubernetes scaling.

The introduction of AI to chaos engineering brings its own set of complications, as AI works at a speed that makes it impossible for human reviewers to keep up, and AI can make mistakes. Effective implementation requires a change in mindset about risk at the highest corporate levels.