We Wanted a Fast Offline AI. First, We Had to Fix Our Benchmark.

Our team sought to create a browser assistant capable of performing tasks like reading invoices, comparing offers, and finding page buttons, all without requiring an internet connection. After rigorous testing with various model types and configurations, we determined that speed, usefulness, and reliability were paramount rather than model size.

However, we soon realized that our initial benchmarking did not account for the specific characteristics of the machines we were testing on. The first iteration of our benchmarking took place on a powerful machine, while the actual target was an older, less capable CPU. This discrepancy meant that our initial results did not accurately reflect the performance of our models on the intended target machine.

Furthermore, we found that our evaluation process had shortcomings. For instance, our coding assistant had provided premature assessments on the thoroughness of our reviews, and our first question about a specific CPU model led us down a path that obscured the true focus of our experiment. To truly gauge the performance of our models in a browser assistant context, we needed to establish a consistent benchmarking framework that accurately reflected the conditions in which these models would be used.

This involved documenting the target machine specifications before running any tests and ensuring that all measurements were consistent across different hardware configurations. In essence, we learned that a well-defined target machine and a comprehensive evaluation process are crucial for accurately assessing the capabilities of local AI models intended for real-world applications.