Urgent.News

What's breaking now, across thousands of outlets.

Tech

TS-Zarr: a cloud-optimized Zarr layout for viewing and analyzing biomedical time series

Background: Biomedical time series are recorded at high sampling rates for hours to days across tens to hundreds of channels. Each recording occupies gigabytes to terabytes of storage. FAIR data sharing requirements often place recordings in cloud storage. Visual analysis requires downsampling windows wider than a few seconds to the pixel width of the screen. Visualization tools therefore either…

Biomedical time series, captured at high sampling rates for extended periods and across numerous channels, occupy vast amounts of storage space. These recordings, which often range from gigabytes to terabytes in size, are frequently stored in cloud servers to comply with FAIR data sharing requirements. To analyze such large datasets visually, researchers must downsample data, typically on a scale wider than a few seconds, to match the pixel width of their visual screens.

Visualization tools often read a local copy of the recordings or query a separate resampling server located in front of the data.

The researchers introduce TS-Zarr, a Zarr v3 layout designed specifically for time series data. This layout stores the samples of each channel alongside a series of summary statistics stored at progressively coarser levels in the form of a pyramid. By utilizing Zarr on cloud storage, the layout allows for low-latency, high-throughput parallel reads.

This means that a reader can fetch only the specific channels and windows they need, rather than the entire recording, thereby enabling more efficient and scalable visual analysis.

In comparison to HDF5 and EDF formats, TS-Zarr demonstrates superior performance in targeted reads. The precomputed levels in this layout introduce some additional storage requirements and the need for error-bound signal transformation. However, these trade-offs enable faster and more scalable visual analysis, particularly as users expand their research and the size of the windows they analyze widens.

As the window under view grows in size, the benefits of precomputed reads in cloud storage overshadow the limitations of a resampling server that may fail under such demands.

The conclusions drawn from this study highlight the advantages of precomputed levels in visualization tasks. Researchers can now visualize a recording without the need to download a local copy of the data or query a dedicated resampling server. Importantly, the time it takes to read a particular view remains bounded, even as the window under analysis widens.

Furthermore, TS-Zarr's design allows for efficient, highly parallelizable, targeted reads of raw sample data, providing researchers with a powerful tool for advanced analysis in the realm of biomedical time series data.

Written by urgent.news from bioRxiv's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.

Read the original at biorxiv.org →

More in Tech

More from Wednesday 18 November →