We lost our own benchmark the first time. Web search vs. one API call on 25 companies
If you build an agent that onboards suppliers or customers, sooner or later it has to check that a company is still registered, and that it isn't sanctioned or insolvent. We make an API that checks this against official registers, so we wanted to know if an agent needs it at all, or if a decent model with web search gets there by itself. So we ran a small benchmark, and the first run went badly…
In a benchmark test comparing web search with an API call to verify company registration status, a single API call performed better than web search for 25 companies from 10 countries. The web search method correctly identified 23 out of 25 companies, while the API method got 17 correct. The test revealed four bugs in the API method that needed to be fixed before running a second test.
After fixing the bugs, the API method improved its score to 19 correct out of 25 companies, while the web search method remained at 20 correct. The API method's accuracy in identifying sanctioned companies and insolvent companies was also higher than the web search method. However, the web search method did better on the 12 harder cases, correctly identifying 6 out of 12 companies, compared to the API method's 11 correct answers.
The web search method also performed better on the 13 more well-known companies, correctly identifying 13 out of 13 companies, while the API method only got 9 correct. Despite these differences, the overall score for the API method was higher due to the fixed bugs. The benchmark highlights the importance of addressing bugs in automated systems and the potential benefits of combining different methods for more accurate results.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.