The Green Test That Lied

The visual-prompt file for episode 09 passed all automated tests, displaying a green signal. This green status, however, was misleading. The file was fundamentally flawed. A successful test for a known defect creates an illusion of confidence in the flawed system. This narrative explains how green can mean something when it should mean something else, using an idiosyncratic codebase: a git repository functioning as a one-person film studio.

The issue is specific to the creator's domain, but the solution applies broadly. While filming, the creator caught this class of error repeatedly, necessitating a fix. When preparing to record build-along sessions for episode 09, the intention was to demonstrate the issue and resolve it simultaneously. The plan was to show the problem and the subsequent fix in the same session.

The project consists of a 10-episode film series, with the protagonist undergoing three distinct body phases throughout the series (pristine, damaged, gold-mended). Each phase corresponds to a reference image, ensuring visual consistency across scenes. The visual-prompt file describes each shot, including a block of text for the image generator and metadata specifying the reference image to attach.

A critical rule mandates that damaged-body episodes must attach the damaged reference. However, episode 09 mistakenly attached the pristine reference image, leading to an unnoticed error. Two blind spots contributed to this issue. The first validation search used the wrong keyword in the prompt text to identify scenes involving the protagonist and requiring policing.

The search never engaged because the prompt never included the keyword "robotiko." The second blind spot was that the validator did not read the metadata field where the actual error resided. The incorrect reference was not present in the prose; it was stored in a metadata field that the validator failed to parse. To address both blind spots, the creator introduced a transferable rule: check the data where the rule actually resides.

Invariants stored in structured fields (references, upload lists, config) are often invisible to prose-level or naming-level checks. Failure to validate these fields leads to silent failures rather than isolated bugs. The creator's deeper problem was that the rule governing body placement within episodes existed only in their mind.

This tribal knowledge, unique to the creator, became a piece of data: a phase_reference_map in the character-profiles JSON file. Each phase is mapped to its reference image, with explicit exceptions recorded for individual episodes. For example, one episode intentionally keeps the damaged body even after the story resolves. Another transitions mid-episode at a specific scene.

This information was documented in an Architecture Decision Record, ensuring a written source of truth. Without a documented source, any check is merely an individual's opinion hard-coded into a script, lacking an objective standard. The creator created a red check that verified the metadata fields against the phase_reference_map.

The initial run on episode 09 returned red, indicating that the file contained the forbidden reference and required correction. This red result confirmed that the check functioned correctly. Following this, the generator was fixed, and the authoring skill file was updated to prevent the error from recurring in subsequent episodes.

Implementing checks with proof of functionality is crucial, as blindly trusting a checker without validation is risky. The creator implemented meta-tests that freeze fixtures representing real broken files and their corrected versions. These tests assert both failure modes, ensuring the broken fixture fails only on the reference check and pinning down the reason for its incorrect green status.

This bidirectional fixture discipline, now mandatory across the suite, resulted in 245 meta-tests. The creator established two key rules: first, make the check fail first, as a green that has never been seen turning red is not evidence. Second, never loosen a check without a proof that it still catches real bugs and ignores the intended cases.

These rules maintain the integrity of the entire system. The sweeping application of new checks revealed reds in seven episodes, EP02 through EP08, exposing a more significant issue. Episode 06 used a different scene-header format, causing the parser to match zero scenes and pass the reference check erroneously. A green that indicated "I checked everything" and a green that indicated "I checked nothing" appeared identical.

The suite now asserts that every shipped file parses a non-zero number of scenes. An empty iteration passing all tests is as deceptive as a green that masks a lack of checks. The red indications required careful triage, classifying them into three categories: FIX, meaning real defects; WHIT, where the shipped images remained unchanged due to already-released films; and WIP, for work in progress.

For EP04 and EP05, only the text needed correction, while the shipped images were retained. The sweeping process identified reds in multiple episodes, prompting systematic fixes for real defects while preserving the integrity of released content.