Why Did the Test Fail? A Triage Rulebook, Tested on Two LLMs
Why did the test fail: the product, the test, the data or the environment? An evidence-based triage rulebook with rules and 21 labelled cases.
A failing test indicates a problem, but not what the problem is. A new rulebook aims to sort test failures by determining which artifact is incorrect: the product, the test itself, the test data, or the environment. Two large language models (LLMs) applied this rulebook to 21 failures. Initially, one model assumed the API change caused a new test's expected value to be incorrect.
After applying the rulebook, that assumption was eliminated, and each failure was associated with the missing evidence needed to identify the root cause. The rulebook helps teams avoid chasing the wrong issue and clarifies the specific artifact that's responsible for the failure.
Written by urgent.news from HackerNoon's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
