OpenAI's COBOL Migration Score Jumped From 5.4% to 86.6%. The Tests Are Hidden

OpenAI's COBOL migration evaluation score has increased significantly, from 5.4% to 86.6%. This assessment is based on the hidden tests that check if the outputs of the migrated code are identical to the original code. The tests cover various languages, including COBOL, Fortran, Visual Basic, and others. GPT-6.1 Sol, at low reasoning effort, passed 86.6% of the tasks, while GPT-5.6 Sol achieved 5.4% at the same setting.

When the reasoning effort is increased, GPT-6.1 Sol's passing rate rose to 97.9%. However, these migration tests are not publicly available, and only one of the two evaluations can be replicated. OpenAI introduced this update as a capability enhancement rather than a new product. Users can test the latest models through the API for coding tasks in their preferred languages.

The evaluation focuses on languages used in critical systems, such as banking, card, and ATM transactions, weather forecasting, and nuclear safety analysis. The migration score is determined by comparing the outputs of the original and migrated code byte-for-byte. To build your own parity check, collect real inputs, run the original and migrated programs on each input, and compare the outputs.

This method ensures that the migrated code produces the same results as the original code, providing a reliable way to validate the migration.