{
  "id": 15713246,
  "title": "OpenAI's COBOL Migration Score Jumped From 5.4% to 86.6%. The Tests Are Hidden",
  "url": "https://urgent.news/2026/10/20/openais-cobol-migration-score-jumped-from-5-4-to-86-6-the-tests-are",
  "topic": "ai",
  "section": "AI",
  "published": "2026-10-20T07:05:39.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/gabrielanhaia/openais-cobol-migration-score-jumped-from-54-to-866-the-tests-are-hidden-29go"
  },
  "original_language": "en",
  "account": "OpenAI's COBOL migration evaluation score has increased significantly, from 5.4% to 86.6%. This assessment is based on the hidden tests that check if the outputs of the migrated code are identical to the original code. The tests cover various languages, including COBOL, Fortran, Visual Basic, and others. GPT-6.1 Sol, at low reasoning effort, passed 86.6% of the tasks, while GPT-5.6 Sol achieved 5.4% at the same setting. When the reasoning effort is increased, GPT-6.1 Sol's passing rate rose to 97.9%. However, these migration tests are not publicly available, and only one of the two evaluations can be replicated. OpenAI introduced this update as a capability enhancement rather than a new product. Users can test the latest models through the API for coding tasks in their preferred languages. The evaluation focuses on languages used in critical systems, such as banking, card, and ATM transactions, weather forecasting, and nuclear safety analysis. The migration score is determined by comparing the outputs of the original and migrated code byte-for-byte. To build your own parity check, collect real inputs, run the original and migrated programs on each input, and compare the outputs. This method ensures that the migrated code produces the same results as the original code, providing a reliable way to validate the migration.",
  "summary": "Book: AI Agents Pocket Guide Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools Me: xgabriel.com | GitHub You hand a COBOL program to a model and ask for Java. The Java compiles. The tests you wrote pass. Then one batch run pays a…",
  "key_points": [
    "OpenAI's COBOL migration score increased from 5.4% to 86.6%",
    "Hidden tests verify identical outputs of migrated code to original code",
    "GPT-6.1 Sol achieved 97.9% passing rate with increased reasoning effort"
  ],
  "editors_take": "The massive leap in migration accuracy positions advanced reasoning models as viable tools for updating the legacy codebases that underpin critical global infrastructure like banking and nuclear safety.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}