I Let AI Write My Tests for 6 Months. Here Is What Actually Survived Production

Last month, a teammate shared a Playwright test on our PR channel, claiming it was AI-generated in just four seconds, questioning why we continue to write tests manually. The test passed, but it only clicked a button, waited three seconds, and confirmed the page still existed. This yielded no real value, just a green tick. Having worked in test automation for over three years with Playwright on web and Flutter on mobile, I have been using AI in that process for about six months. Some improvements have genuinely changed my workflow, while much of it is unnecessary hype.

There are three main areas where AI proves useful:

1. Converting bug reports into test cases. This is the most significant benefit and rarely discussed. Our QA team writes bug reports in plain language. By providing the bug description and our page object file to the AI, I can generate a failing test in under a minute. Although not perfect, it saves a substantial amount of typing – approximately 60% of the initial work. I still need to refine the assertions, but it does most of the groundwork.

2. Explaining flaky tests. When a test fails randomly, it can be frustrating, especially if it was written months ago by someone who left the team. You open the test, find nested waits and hardcoded timeouts, and wonder what caused the race condition. Sharing the test and trace with the AI can help identify the issue more quickly, even if it's not always correct. The AI can suggest possible reasons to test, which is faster than trying to decipher the existing code.

3. Suggesting locators for complex DOM structures. By providing the AI with a piece of HTML, it can recommend the most stable locators, often recommending using getByRole and getByLabel instead of convoluted CSS selectors. It acts like a linter with opinions, helping to improve the maintainability of tests.

However, AI has its limitations:

1. It cannot identify important edge cases. AI generates tests based on the happy path, which is already present in the code. It won't ask about scenarios like multiple payments arriving at once, a problem you typically learn from experience. Roughly 80% of the real bugs I've caught came from tests that AI wouldn't generate. Mobile web testing is particularly challenging for AI, as it is trained on desktop web test code.

It often fails to understand viewport-specific behavior, touch targets, or how a sticky header can interfere with clicks on small screens. This leads to poor results, requiring human intervention to correct.

2. The training data for Flutter is insufficient. If your app is built with Flutter, you'll need to write tests manually. The model generates plausible-looking tests using outdated APIs, and it struggles to adapt to new versions. I built our smoke and regression testing setup for Flutter almost entirely by hand, as the AI failed to understand the framework's structure.

3. Self-healing locators are mostly marketing hype. AI testing tools claim they can fix locators automatically, but in practice, a locator that silently repairs itself is one that stops telling you when the UI changes. In such cases, the UI change itself is the bug, and the test should fail loudly to alert you to the problem.

My current workflow involves writing the test plan myself in plain English, describing what should break and why it matters. The AI generates the test skeleton using my existing page objects, and I then rewrite every assertion manually. I run the tests locally 20 times before they reach CI, deleting any flaky tests and starting over rather than adding unnecessary waits. This approach ensures that the tests I write are valuable and catch actual issues.

In summary, while AI can speed up the initial test-writing process, it does not improve one's ability to identify key failure modes. Test automation professionals entering the field should focus on learning how to reason about potential issues rather than solely relying on AI-generated tests. I invite readers to share their experiences with AI-generated tests, including the most ridiculous examples, instances where self-healing locators actually worked, and insights on using AI for Flutter or React Native development.