We collected 1,045 hours of sleep audio. Only 53 hours were usable for training.

We gathered 1,045 hours of sleep audio for training a model to identify sleep phases. However, only 53 hours of this data were deemed useful for training. This ratio, while seemingly insignificant, is actually the most crucial aspect of the project. Many experts debate about model architecture, but few discuss the fact that each additional hour of audio rarely yields new learnable information. Recording a night's audio is straightforward, but making the data usable is challenging.

The primary challenge lies in labeling the audio to identify sleep phases. This requires a trustworthy reference, which in this case comes from wearable devices. However, using two devices introduces complications, such as differing clocks and the potential for independent failures. Even a small discrepancy in time can render a night unusable for labeling. If a wearable device stops recording early in the night, it means that half of the night has no label, effectively removing a significant portion of data.

Out of the 111 nights recorded, only 30 nights were considered valid, meaning they had complete sleep onset, full duration, and the phone was left untouched. The rest were aborted starts, test recordings, phones that were picked up again during the night, or recordings that never reached a sleep onset. The majority of the collected data was discarded, not because of a lack of data, but due to a lack of a well-defined criterion for what constitutes a valid night.

The process of collecting and processing data was meticulous. Every recording had to have a reason for existing, with notes on whether sleep onset was present and a flag for completeness. Incomplete nights were not silently averaged into the set, but rather discarded. Alignment of audio with wearable output was done before labeling, never after, to avoid introducing errors. Every label was made explicit about its source, whether it was from the audio device, the wearable device, or both.

After these rigorous steps, the team realized that the real challenge was not the amount of data collected, but the amount of data that was discarded due to strict criteria. The ratio of 20:1 between the total collected data and the usable data highlights the importance of a strict definition of what constitutes a valid night. This bookkeeping is essential for the rest of the project to be possible.

Ultimately, the ratio matters beyond the scope of this app, as sleep apps often publish accuracy figures without disclosing the reference used for validation. Without transparency about the data and the definition of a valid night, such numbers are meaningless.