It had 100% coverage and yet production fell.

Today, a large part of the code that goes into my projects is written by a model. It's not an experiment, it's how we work. And that brought me a question that I hardly ever asked before, because I wrote the code and more or less knew where the gaps were: if I didn't write this, how do I know it works? The obvious answer is "I measure coverage".

I measure it. And once I had everything green while the page fell in production. This post is about that contradiction, and about the decisions that make the number mean something or mean nothing.

First, what do I measure?

Nothing exotic: flutter test --coverage generates lcov.info and then I filter with lcov --remove before getting the percentage.

lcov --remove coverage/lcov.info \

'**/*.freezed.dart' \

'**/*.g.dart' \

'**/*.config.dart' \

'**/constants/*.dart' \

'**/theme/*.dart' \

'**/di/*.dart' \

'**/router/*.dart' \

-o coverage/lcov_filtered.info

What I take out of the account and why:

The generated files (freezed, json_serializable, injection configuration) I didn't write them. Testing them is testing the generator. If freezed is broken, that's not my test.

Constants and theme don't have branches. There's no input that makes them behave differently. A test there only confirms that a constant is worth what it's worth.

DI and routing are wiring. Verifying that the container resolves a dependency is testing the framework.

The rule, if I had to summarize it, is that I exclude what can't make a wrong decision. Everything else goes in.

And here it's convenient to say something uncomfortable before someone else says it: the list of exclusions is where you can cheat. I can get to the percentage I want by moving those patterns. That's why I don't touch it when the number doesn't like me. If I have to justify why I took something out, the answer can't be "because I didn't reach it".

The test that runs everything and doesn't verify anything

This is the misunderstanding that makes experienced people distrust coverage, and they have reason to distrust it.

lcov measures which lines were executed. It doesn't measure if you checked something. Those are different things and a model generating tests confuses them all the time.

test('applies discount', () {

final result = calculator.applyDiscount(100, 0.2);

expect(result, isA<double>());

});

This test runs each line of applyDiscount. Coverage: 100%.

Real checks: zero.

If the formula were inverted, price * discount instead of price * (1 - discount), it would pass just as green.

test('applies 20% discount', () {

expect(calculator.applyDiscount(100, 0.2), 80.0);

});

test('rejects discounts greater than 100%', () {

expect(() => calculator.applyDiscount(100, 1.5), throwsArgumentError);

});

Same line coverage. The difference isn't measured by any tool. It's measured by a question: does the test assert something, or just run?

Models write the first one by default. That's what they do best: they go through the code, get to green, and raise the number.

When I review a generated test, I don't look if the percentage went up, I look at what happens if I break the function on purpose. If the test still passes, the test doesn't exist.

The presentation layer doesn't go in, and it's not because of laziness

This was the decision that cost me the most and that I defend the most.

I started writing widget tests like everyone else. Two things made me stop.

The first one is silly but real: the expects couldn't find the widget. Sometimes yes, sometimes no, depending on how the tree had turned out. I ended up fighting with the finders instead of verifying behavior. Anyone who's written widget tests in Flutter knows what I'm talking about.

The second one is what matters.

In the app, there are options that appear or not depending on the user's role, and there are quite a few cases. But that decision doesn't live in the widget: it lives in the BLoC. The widget only paints what the state tells it.

So if I test from the widget that "user X sees option Y", I'm verifying the same rule that I already verified in the BLoC, but through a slower door, more fragile and that breaks when someone changes a Padding.

It's not that I don't test the UI. It's that I don't test the same thing twice, and I choose to do it where the test is cheap and stable.

(What I still lack, and I say it because an honest post also says where it didn't reach: golden tests for what is stable. It's on the list. I haven't done it yet.)

Legacy: I don't pursue the past

In projects that I start from scratch, I aim to cover all the logic. But I also maintain inherited code, without tests, written by people who are no longer there. There, I don't pursue 100% backwards. It would take months to write tests for code that I'm not going to touch, and worse, tests that freeze the current implementation instead of protecting a behavior.

What I do is simpler: the old code stays as it is, but everything new that I add to it comes in with its tests. The bar goes up every time I touch that module.

I don't stop delivery to cover the past, I stop the bleeding.

It's the difference between "this project has 100%" and "this project doesn't lose coverage again".

The second one is achievable without stopping the world.

The fall

And here's the part where the number lied to me in the face.

My suite runs on mocks. The entire data layer is mocked, so the tests don't talk to a server: they talk to a frozen version of what the server said the last time I wrote that mock.

We have a system of points for delivered orders. One day, due to a backend error, a delivered order didn't load its points and the field came null. It had never come null before.

The app couldn't handle it, the web couldn't either, and the page threw an error.

All my tests were green. All of them.

And the reason is almost humiliatingly simple: my mock kept returning the old world.

I had complete coverage of a reality that no longer existed.

The test never saw a null because I never wrote a mock that returned it, and I didn't write it because in my head that field always came.

That's the answer I give to anyone who says that coverage means nothing.

You have reason, and here's the proof from my own suite: 100% means...