I Counted Drops as Wrongs. The Chart Was Theater.

The first number on an evaluation dashboard is often inaccurate. It results from an error in the calculation process. Instead of relying on this misleading figure, it is important to accurately assess the performance of models. Scoring completions should be halted until the envelope is properly evaluated. Not distinguishing between a drop and a wrong answer leads to ranking models rather than evaluating the weather conditions affecting the process.

Free endpoints create unnecessary noise by generating idle processes, truncated streams, and empty choices that still appear in HTTP 200 responses. The true status may appear fine, but the grade may seem like a failure. To create a more reliable leaderboard, a harness is needed that fails at the transport layer before it manifests as errors at the semantics layer.

The provided code demonstrates this experiment, with the percentages representing the fixture talking rather than a vendor scoreboard. The most common evaluation method still considers a successful completion as having both a 200 HTTP status code and the expected text present. This method is flawed and can lead to inaccurate results.

To properly assess models, it is crucial to consider various failure modes, such as dropped connections, stalled responses, truncated streams, and other issues that may occur, particularly when using free inference services. These services can drop, stall, or truncate data in ways that dedicated servers would not, leading to incorrect grading of free model paths.

The author emphasizes the importance of having a hostile mailbox, or a robust evaluation environment, to ensure that models can handle failures gracefully. If a model cannot recover from planted errors on disk, it will not be able to handle real-world failures either. The author has prepared an example file (envelope_eval.py) that demonstrates a planted-envelope scorer, which does not rely on network calls.

To run the scorer, simply execute python envelope_eval.py. The scorer classifies envelopes based on their HTTP status code, body content, and expected values, returning different labels for various failure scenarios. By analyzing the yield (number of gradeable bodies) and accuracy-on-yield (accuracy of the received objects), one can obtain a more accurate representation of a model's performance.

Simply publishing the product of these numbers is insufficient, as it only provides a theatrical result.