The shell step that could not fail no matter what we did to it

Our integration suite had remained green for eleven weeks, indicating a healthy state. However, a colleague deliberately created a broken test to understand what a failure looked like. They pushed the changes, and the pipeline remained green. This single shell step, `npm run test:integration | tee integration.log || true`, contained two mistakes.

The pipe was used to upload the log as an artifact, while `set -o pipefail` was not employed, causing the pipeline exit status to always be zero. The `|| true` command, added fourteen months ago, aimed to prevent occasional cleanup commands from returning an error. This combination of errors left failures hidden. While investigating, seven more steps were found across four repositories that could not report failure.

Three had the same pipe issue, two had `|| true`, one ran the real command inside a subshell with discarded status, and one used a for loop over services that overlooked each iteration's result. The fixes were straightforward; all script steps now began with `set -euo pipefail`, enforced by a lint job that flagged any workflow YAML without `run:` blocks. `|| true` was banned outside a limited allowlist, and every allowed use required a comment explaining its necessity.

Additionally, a canary job was introduced in every pipeline, running a command that always exits with an error status. This job asserts its failure, runs on every build, and costs only two seconds. The author believes that this single change could have caught the issue much sooner, rather than uncovering it after eleven weeks.