Vibe coding testing: check your app before users do
Originally published at pantheontable.com/blog/vibe-coding-testing-check-your-app-before-users-do Andrej Karpathy vibe coded a website and felt 80% done. By his own account, it was a bit closer to 20%. If you built an app with an AI tool and it seems to work when you click through it, this is for you. Below is what Karpathy, David Heinemeier Hansson, John Carmack, Joel Spolsky, Simon Willison,…
Andrej Karpathy, a computing expert, built a website using an AI tool and felt it was 80% complete. However, he later realized it was closer to 20% done. He advises that clicking around only verifies paths you have tried, so AI testing involves giving the AI a way to run tests and using the app as a customer would. It is crucial to ask the AI to write tests for core actions, show the command and output, and have a fresh review session.
Test every change on a preview copy before deployment and collect user errors to know when something breaks. Clicking through the app is insufficient; you must test for empty forms, double clicks, weak connections, and second accounts. Karpathy warns that clicking through gives a false sense of completion, and AI tools are good at simple tasks but struggle with complex system integrity.
To check AI-written code, provide a way to know if it succeeded, like a test file, build, linter, or screenshot. The AI should explain the code and run the tests, providing exact commands and outputs. Non-developers should seek expert opinions to validate AI output.
Rewritten by urgent.news; may contain errors — check the original.