We Built a Test to Catch Contact Forms That Lie. It Lied to Us Five Times First

My team built a form backend, and we often received messages like "my form stopped working" when in reality the form never worked. To quantify this issue, we decided to test contact forms on sites built with Lovable and Bolt. We simulated form submissions with test values and monitored their responses. Our goal was to identify fake forms that appeared to send messages when they actually didn't.

We developed strict rules for what constituted a successful form submission, such as requiring the request to contain the test values we provided. However, our initial test turned out to have several flaws that led to incorrect results. After six rounds of testing, we identified five instances where our test incorrectly classified forms as functional.

The first mistake was that our test counted any request after the click as the form sending something, even if it wasn't sending a real request. We addressed this by excluding known analytics hosts and only counting requests carrying the test values. Another issue was that some forms showed success messages through alerts without making any network requests.

We modified our test to sample the page every 250 milliseconds for five seconds, which helped us capture these hidden success messages. One particularly tricky case involved forms that sent visitors to mailto: links, which never generated network requests. We had to separately detect these navigations. Additionally, some forms used honeypot fields to detect bots, causing our test to mistakenly identify them as legitimate.

We refined our test to skip hidden text fields, but this caused issues with forms using shadcn/Radix select menus, which rendered hidden native elements. We then adjusted our rules to fill these hidden selects and checkboxes. Throughout the testing process, we verified that the test could accurately classify known scenarios, including fake toasts, real API calls, native POST requests, mailto forms, and AI chat boxes.

We also ran a deliberately broken version of the classifier to ensure that the test could catch broken implementations. Out of the 105 forms that produced a clear result, 54 successfully sent messages, while 27 showed success messages but failed to send the message. Twelve forms appeared to send messages but did so silently, and seven opened the visitor's email app.

The remaining 7 forms either had no visible response or did something else. After reviewing the 27 forms classified as fake, we confirmed 21 by examining their submit handlers. We could not locate the submit handlers for the remaining 6, so we reported them as flagged by the test rather than proven to be fake. In summary, out of the 105 forms with a clear result, 54 sent messages, 27 showed success messages without sending anything, 12 gave no visible response, and 7 opened email applications. Among the 105 forms, 54 were legitimate, while 27 were classified as fake.