Quantified Self: How I Handled 100GB of Messy Biosensor Data with Polars and Dask

In the era of the Quantified Self, we generate massive amounts of data from various biosensors like Oura rings, Apple Watches, and continuous glucose monitors. However, this raw data is inconsistent, high-frequency, and often exceeds 100GB, making it challenging to process. This tutorial demonstrates how to build a high-performance data processing pipeline to handle such large-scale biosensor data.

The architecture involves a lazy approach, where we don't load all the data at once but build a computation graph and execute it only when necessary. The pipeline includes three main stages: loading the raw biosensor CSVs/JSONs, cleaning the data through resampling and interpolation, and storing the results in Parquet/Arrow format.

Before diving into the processing, ensure you have the required stack: Polars, Apache Arrow, Dask, and Streamlit installed. Polars, written in Rust, is a blazing-fast DataFrame library. Apache Arrow serves as the backbone for memory-efficient data interchange, while Dask is used for distributed task scheduling. Streamlit allows visualization of the results without writing any CSS.

The first step is to solve the RAM crisis using Polars Lazy API. Instead of loading the entire 100GB of data into memory, we scan the files and create a pointer to the files, inferring the schema. We then load the data into a Polars LazyFrame, which allows us to define operations without executing them immediately.

The second step addresses the nightmare of varying sampling rates among biosensors. Since heart rate sensors might tick every second while accelerometers tick at 50Hz, we need to align these time series. Using Polars, we can sort the data by time, handle duplicate timestamps, upsample to a consistent 1Hz frequency, and interpolate missing heart rate values.

The third step involves scaling the processing using Dask and Apache Arrow. Dask orchestrates the processing of hundreds of files in parallel, and Apache Arrow stores the results in Parquet format, which supports predicate pushdown for efficient querying.

Lastly, we visualize the processed data using a Streamlit dashboard. Since we saved our data as Parquet, the dashboard remains incredibly snappy. The tutorial concludes by recommending additional resources for building production-ready data lakes and scaling Python pipelines to the petabyte scale.