{
  "id": 34181,
  "title": "Everyone Knows It Scores Half. Nobody Checks Which Half.",
  "url": "https://urgent.news/2026/08/02/everyone-knows-it-scores-half-nobody-checks-which-half",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-08-02T07:06:11.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/alan_scottencinas/everyone-knows-it-scores-half-nobody-checks-which-half-953"
  },
  "original_language": "en",
  "account": "The Kaggle competition page for the Mars Environmental Dynamics Analyzer (MEDA) virtual sensor recovery project displays a small line of text under the standings, stating that the leaderboard is calculated using approximately 50% of the test data. While everyone reads and nods at this line, few question which specific half of the data is being utilized. Intrigued by this ambiguity, the reporter decided to investigate further.\n\nUpon analyzing the competition results, it was discovered that the public score is not based on a random sample but rather the first 1,974,995 rows in spacecraft clock order, representing exactly half of the file. This revelation has significant implications for the competition, as a chronological split introduces different regimes between the public and private score measurements.\n\nThe reporter then delves into the competition's structure, where the task is to reconstruct pressure, temperature, wind, and dust measurements for sols 201 through 300, while only training on sols 1 through 100. The scores are based on mean squared error (MSE), and the competition has been approached by splitting the data into three components: a seasonal baseline, a diurnal term, and a residual component that is learned by a machine model.\n\nAfter iterating through multiple versions, the final submission achieved a score of 18.99. The reporter questions whether the half of the data used for scoring is a fair representation of the entire dataset, as the seasonal baseline fluctuates by more than a hundred pascals across the test window. The reporter concludes that tuning against the public leaderboard using a time split is extrapolation across a boundary that cannot be measured, leading to a potential failure mode in the competition.\n\nThe reporter then explains how the mean squared error (MSE) can be mathematically manipulated to measure the hidden labels (labels not seen during training) without directly accessing them. By solving a quadratic equation concerning the perturbation direction, the reporter demonstrates how to obtain one exact number per submission, allowing for the measurement of the hidden labels without ever seeing a single label. This technique is well-documented and commonly used, yet the reporter had previously overlooked its potential significance.\n\nThroughout the investigation, the reporter found that the leaderboard probing technique is inexpensive, requiring only the integration of previously written files and no additional model re-runs or re-fitting. Two independent basis vectors, one for the seasonal coordinate and another for the diurnal component, were reconstructed from CSVs from previous weeks, revealing the competition's scoring methodology in a new light.",
  "summary": "The article discusses a Kaggle competition focused on reconstructing missing pressure readings from the Mars Environmental Dynamics Analyzer (MEDA) on the Perseverance rover. The competition required participants to build a model using data from sols 1 to 100 to predict sols 201 to 300, with the final score based on mean squared error. The article highlights a crucial detail about the competition's scoring methodology: the leaderboard was calculated using approximately 50% of the test data, specifically the first 1,974,995 rows in spacecraft clock order. This chronological split means the public score is not a sample of the private score but rather a measurement of a different regime, which significantly impacts the model's performance and interpretation of the results.",
  "key_points": [],
  "editors_take": null,
  "illustration": "https://urgent.news/ill/34181.png",
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}