False Green: My Incident Agent Said “Healthy” Without Live Evidence
This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry . The pipeline was healthy. At least, that was what my incident agent told me. There was one problem: it could not see the live Fivetran pipeline. The live check had failed, so the app fell back to cached evidence. That snapshot still said connected and on_schedule . The final verdict was still: Healthy. Nothing…
The incident report describes a bug in the Pipeline Rescue Agent, a Next.js app used for incident investigation. The issue arose when the application incorrectly reported a "Healthy" status for a pipeline, despite not having access to live Fivetran data. The agent fell back to cached evidence, which falsely appeared to be current and trustworthy.
This led to the agent making incorrect decisions about the pipeline's health. The problem was traced to the decision-making process, which did not require live evidence for a pipeline to be considered healthy. The corrected code now ensures that a Healthy verdict requires live evidence, addressing the false-green vulnerability.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
