{
  "id": 18945574,
  "title": "olmOCR-bench on Hugging Face: Three Rows Use a Different Average, and One Moves From 13th to 7th or 19th",
  "url": "https://urgent.news/2026/11/08/olmocr-bench-on-hugging-face-three-rows-use-a-different-average-and",
  "topic": "ai",
  "section": "AI",
  "published": "2026-11-08T13:47:17.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/ward_ed_6b5e6aa8ded94a987/olmocr-bench-on-hugging-face-three-rows-use-a-different-average-and-one-moves-from-13th-to-7th-or-5gih"
  },
  "original_language": "en",
  "account": "On November 8, 2026, the Hugging Face olmOCR-bench leaderboard presented a peculiar ranking of 24 repositories based on their OCR performance. Out of these, 18 rows were derived from the main branches of the repositories, while the remaining six came from open pull requests submitted by a single user account.\n\nThe overall ranking was determined by averaging eight columns, encompassing seven test categories and a baseline column, for a total of fifteen repositories. However, three rows - ranked 13, 20, and 23 - utilized a distinct average that excluded the 'headers_footers' category. Their respective notes explicitly indicated this deviation.\n\nAdditionally, six more rows remained unverifiable due to various reasons: one was gated, another only provided a mean value, and four contained fewer than eight columns. In the case of rank 13, repository 'LightOnOCR-2-1B', the overall score of 83.2 was derived from eight columns averaging 75.24. Removing the 'headers_footers' category shifted its position to the 7th place among 22 rows, whereas including it placed the row at the 19th position among 21 rows. Consequently, rank 13 presented a blend of both scenarios.\n\nThe leaderboard demonstrated a stark contrast in rankings across the board, with rank 1 remaining consistent across multiple recomputations, and rank 2 surviving all but a test-count weighting. Rank 1 excelled in three out of the eight columns, occupying the 16th position among 17 when considering the baseline. Moreover, six different repositories led the eight columns individually.\n\nIt is important to note that the dataset card accompanying the olmOCR-bench also provided its own table of results, encompassing a range of ± 0.9 to 1.2 for the Overall scores. Therefore, when assessing the top positions on the leaderboard, it is prudent to consider this range.",
  "summary": "TL;DR On November 8, 2026 the Hugging Face olmOCR-bench board lists 24 rows from 24 repos, all higher-is-better pass rates, 0 verified. 18 rows sit on a repo's main branch and 6 come from pull requests that are still open, all opened by one account. The board's overall is the plain average of eight columns (seven test categories plus a baseline column) for 15 of the 24 rows. Three rows, ranks 13,…",
  "key_points": [
    "Three rows used a different average excluding headersfooters category",
    "LightOnOCR-2-1B moved from 13th to 7th or 19th place",
    "OlmOCR-bench leaderboard ranked 24 repositories"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}