Grading a Machine That Read 51,008 Pages With No Answer Key

Konstantin Tsiolkovsky, a prolific Russian scientist, devised key concepts in rocketry and space travel over a century ago. His personal papers are now housed at the Archive of the Russian Academy of Sciences in Moscow. Though the archive digitized and made these works available online, their digital presence falls short of being easily searchable or analyzable.

This prompted the author to create a catalog of all 2,019 files, a dating for 1,969 of them, classify the pages as handwriting or typescript, and transcribe each page. The project was undertaken to generate a comprehensive dataset accessible for research. However, a significant challenge arose in verifying the machine-transcribed pages, as no human has yet validated any of the content.

The result is a collection of transcribed files that serve as a finding aid, not a scholarly edition. The archive's website presents certain limitations that only become apparent once one attempts to use it as a data source. First, the identifier in a page address does not correspond to the archival file number, causing confusion when trying to locate specific files.

Second, the inventory number in the URL is decorative and does not reflect the true archival address, which is only found within the file paths of the scans. Additionally, scans larger than 800 KB are cut off at about 130 KB, requiring special requests to retrieve the full content. These technical issues highlight the discrepancy between the images on the internet and a collection that can be queried and analyzed.

Measuring the accuracy of machine reading is complicated due to the lack of ground truth, as there is no corresponding text keyed in by a person. Traditional workarounds, such as using the model's own confidence scores, do not hold up well, particularly for old and difficult handwriting. The author's approach was to compare the machine-transcribed pages against their original manuscript and typescript versions, which often share the same text.

Analyzing 1,759 pairs across 224 files, the median agreement between the two transcriptions stands at 37%, with the longest consistent run of words being only 10. This illustrates the significant difficulty in deciphering early twentieth-century manuscript handwriting. For comparison purposes, the author used the presence of printed editions as a proxy metric.

When paired readings against printed editions are available, the estimated error rate is unbiased to within a percentage point of the true error. By ranking pages by difficulty using this method, the accuracy reaches 98.1% at the character level and 81.1% at the word level. However, these metrics are only applicable to the subset of the corpus that was initially read and scored against editions.

The author cautions against making sweeping claims about the overall accuracy of the transcription, as only a small part of the collection has been validated in this manner.