Everyone Knows It Scores Half. Nobody Checks Which Half.

The Kaggle competition page for the Mars Environmental Dynamics Analyzer (MEDA) virtual sensor recovery project displays a small line of text under the standings, stating that the leaderboard is calculated using approximately 50% of the test data. While everyone reads and nods at this line, few question which specific half of the data is being utilized. Intrigued by this ambiguity, the reporter decided to investigate further.

Upon analyzing the competition results, it was discovered that the public score is not based on a random sample but rather the first 1,974,995 rows in spacecraft clock order, representing exactly half of the file. This revelation has significant implications for the competition, as a chronological split introduces different regimes between the public and private score measurements.

The reporter then delves into the competition's structure, where the task is to reconstruct pressure, temperature, wind, and dust measurements for sols 201 through 300, while only training on sols 1 through 100. The scores are based on mean squared error (MSE), and the competition has been approached by splitting the data into three components: a seasonal baseline, a diurnal term, and a residual component that is learned by a machine model.

After iterating through multiple versions, the final submission achieved a score of 18.99. The reporter questions whether the half of the data used for scoring is a fair representation of the entire dataset, as the seasonal baseline fluctuates by more than a hundred pascals across the test window. The reporter concludes that tuning against the public leaderboard using a time split is extrapolation across a boundary that cannot be measured, leading to a potential failure mode in the competition.

The reporter then explains how the mean squared error (MSE) can be mathematically manipulated to measure the hidden labels (labels not seen during training) without directly accessing them. By solving a quadratic equation concerning the perturbation direction, the reporter demonstrates how to obtain one exact number per submission, allowing for the measurement of the hidden labels without ever seeing a single label.

This technique is well-documented and commonly used, yet the reporter had previously overlooked its potential significance.

Throughout the investigation, the reporter found that the leaderboard probing technique is inexpensive, requiring only the integration of previously written files and no additional model re-runs or re-fitting. Two independent basis vectors, one for the seasonal coordinate and another for the diurnal component, were reconstructed from CSVs from previous weeks, revealing the competition's scoring methodology in a new light.