I audited a 5.5 GB AI training dataset by downloading 0.8% of it
Two months ago a repository turned up in my corner of the internet: 3,358 stars, 744 forks, an MIT license, and a release advertised as 518,400 training samples — 5.5 GB, split across three zip parts. My corner is Chinese astrology software, a domain with almost no machine-readable data, so a corpus that size was either the most useful thing published in years or it was nothing at all. I wanted…
On a recent discovery in the realm of Chinese astrology software, a repository emerged with an impressive size of 5.5 GB and 518,400 training samples. Intrigued by its potential, the author decided to delve into the archive to ascertain its worth. The archive, split across three zip files, contained the complete dataset.
The initial examination revealed that the sample count was not a tally of observed instances, but rather the size of a nested loop. This realization led to further investigation into the archive's contents and the reasons behind its composition. The author focused on the method 0 .jsonl.gz shards, which were stored as standalone, unmodified text files, making them independently addressable.
The investigation uncovered several key findings:
1. The archive contained the complete library of Chinese astrology data, including a 2,124-byte stub file and 183,182 characters of TypeScript code. This library generated 1.09 × 10¹⁰ characters of prose, solely composed from 183 K characters of source code. Notably, the generator did not employ the Math.random function, indicating a deterministic output based on a five-integer key.
2. The archive included 2,520 entries corresponding to non-existent dates. Specifically, every 31st day was absent from the data, along with specific days in February across multiple years, resulting in 2,520 rows that represented dates never occurring.
These findings highlighted the effectiveness of using the archive's structure to access its contents without downloading the entire 5.5 GB release. The author demonstrated that a mere 0.81% of the release could be obtained by requesting specific entries, showcasing the efficiency of this approach.
In conclusion, the investigation revealed a comprehensive dataset of Chinese astrology, generated through a deterministic process rather than observational data collection. The archive's use of nested loops and lack of randomization made it a unique example of a memo table. Furthermore, the discovery of non-existent dates within the dataset added an unexpected layer to the analysis, underscoring the importance of thorough examination of data sources.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
