Acceptance testing a small app with fake data: 61 scenarios

Automated tests confirm that the code functions as intended, but acceptance tests ask a different question: can users perform their actual job with the app? Before real users interact with the software, an AI session takes on the role of the salon owner on a test computer, generating 61 scenarios in four rounds. The first round uncovered 19 defects, some of them serious, which were subsequently fixed.

The remaining rounds came back clean. These scenarios were specifically chosen to be challenging, focusing on tasks such as the first login with a forced password change, multiple failed attempts locking the user out, and various data formats like Italian column names and phone numbers with spaces, dashes, and country codes. The AI also tested the app with an Excel file containing accented names and other irregularities, as well as attempting to cancel an import midway through.

Additionally, the AI introduced a canceled appointment between two imports, a computer with the wrong time zone setting, multiple salons with various users, and a large dataset of 500 customers and 2000 appointments to ensure the app's performance under heavy load. Other scenarios included a double-click, slow network speed, no network connection, a session that expired mid-form, and checking the app's appearance on multiple devices and themes.

The report acknowledges the limitations of the test, noting that the real Excel export of the booking software and a real WhatsApp message were inaccessible from the test computer. For the first session with the salon owner, the AI session created a short list focusing on log in, importing two weeks of appointments, manually creating a free slot, and allowing the owner to open the link on their phone. The acceptance testing process reveals a different type of mistake compared to automated tests.