Sentinel: The Automated Adversarial Testing Harness for LLM Applications
This is a submission for the MLH x DEV Writing Challenge What I Built AI products are shipping faster than anyone is testing them. In traditional software engineering, QA teams write tests to verify that code does what it is supposed to do. But in Generative AI, the critical question is whether your product refuses to do what it is NOT supposed to do : Can your chatbot be talked out of its…
This submission for the MLH x DEV Writing Challenge introduces Sentinel, an automated adversarial testing harness for Large Language Model (LLM) applications. As AI products are released faster than they are tested, Sentinel addresses this gap by firing dynamic attack prompts against a target chatbot, evaluating responses with a bias-aware LLM judge, and producing a scored diagnostic report in under two minutes.
The dashboard displays an executive summary of the test results, including an overall pass rate, number of attacks run, categories flagged, and those requiring manual review. The OWASP Category Health provides a visual representation of the categories affected by the attacks, such as Prompt Injection, System Prompt Leakage, Hallucination, Excessive Agency, and Jailbreaks.
Demonstrating Sentinel's effectiveness, the submission includes a vulnerable demo chatbot called ShopCo. Three tests were conducted on this bot, exposing vulnerabilities such as System Prompt Leakage, Hallucination, and Excessive Agency Authorization Bypass. The terminal CLI and CI/CD pipeline also enable easy integration with GitHub Actions or local terminal workflows, providing a comprehensive solution to adversarial testing for LLM applications.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
