The AI-generated date test passes today and fails next week
You asked the agent to cover a subscription expiry. It wrote a unit test, ran it locally, reported green, and moved on. A week later CI goes red on that exact test. Nothing in the product changed. The test reads the real clock, so it only held under the date the agent happened to run on. This is one of the gentlest failure modes an agent introduces: the test genuinely passed, the assertion looked…
A recent development in AI-generated code has raised concerns about the reliability of certain tests. When an AI agent writes a unit test, it appears to pass locally, but fails in the CI pipeline a week later. This occurs because the test reads the real clock, meaning it only held under the specific date it was run. The test essentially relies on a short-term prediction that fails on its own schedule.
The issue arises when the test computes expiry from the current time and compares it against the calendar date. While it may seem correct on the machine that wrote it, the test fails later when the actual time moves. Reviewers often overlook this issue since a passing test appears deterministic, even though it depends on the wall clock.
The source material presents several solutions to prevent this clock-dependent test behavior. One approach is to inject a clock into the unit under test, passing a fixed instant instead of using system time. Another method is to freeze time in the test and assert the offset rather than the absolute date string. This ensures both sides of the expiry boundary are tested under the same pinned time.
By advancing the fake clock during testing, you can verify that the assertions still predict the outcome, catching potential time bombs before they cause problems in CI.
In summary, it is crucial to freeze time, assert the offset, and cover both sides of the expiry boundary to ensure that the test remains reliable in the future. This approach will help prevent surprising failures and provide confidence in the test's long-term stability.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
