My AI-Written Test Suite Hit 94% Coverage. Then I Mutated the Code.
I have a small Python library that parses CSV files and extracts course schedules. It has no tests. I know, I know. So last week, I decided to see if a free model could write them for me. I used MonkeyCode's free model access and free server. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The goal was simple: generate a test suite for my library, run it, and…
I recently wrote a small Python library to parse CSV files and extract course schedules. Initially, the library had no tests, but I decided to have a free AI model generate some tests for me. Using MonkeyCode's free model access and server, I prompted the model to create pytest tests that covered normal cases, missing values, and invalid dates. The model produced 20 tests, which seemed reasonable. However, when I ran them on my library with coverage.py, the coverage report showed 94% coverage.
One mutation test caught a potential blind spot in the generated tests. I modified the code to return True instead of None when parsing an invalid time. The mutation test showed that none of the generated tests caught this change, indicating a missing test case. The model's tests focused on obvious paths but didn't consider the scenario where a row has no 'Course' key at all.
This experience highlighted the difference between coverage and behavior verification. While the model generated tests that executed most lines, it didn't generate tests that would fail if specific logic changed in a particular way. The lesson learned is that coverage is just a measure of lines executed, not a guarantee that all behaviors are verified.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
