Every compression library I tried ran out of RAM. So I built one that doesn't.
Compress a 1 GB file in the browser with most JavaScript libraries and you'll use 1.6 GB of RAM . Try 4 GB and the tab crashes. zstd-stream does it in 35 MB , whatever the file size, and the file never leaves your device: import { compressStream } from " zstd-stream " ; const compressed = await compressStream ( file . stream (), { level : 19 }); TL;DR Any file size, ~35 MB of memory. Verified…
The author of the story had a problem when trying to compress large files in the browser using various JavaScript libraries. Each library used significantly more RAM than the file size itself, causing the browser to crash. The author then decided to build their own compression library called zstd-stream that addressed this issue.
The zstd-stream library uses Meta's open-source Zstandard C library, compiles it to WebAssembly, and adds a streaming layer on top. This streaming layer allows for processing files in 128 KB chunks, keeping the memory usage constant regardless of the file size. The library takes a ReadableStream as input and returns a ReadableStream as output, ensuring compatibility with existing APIs such as File.stream(), fetch, and Node.js's Readable.toWeb().
Key features of the zstd-stream library include:
1. The ability to compress and decompress files of any size, using approximately 35 MB of memory.
2. Verified performance with files up to 4 GB in Chrome, Edge, Firefox, Safari, and Node.js.
3. Better compression than the browser's built-in gzip (5.5× compression at 147 MB/s vs. 3.5× at 51 MB/s).
4. A single installation package with no additional dependencies, no .wasm files, and no CDN.
5. The WebAssembly binary is embedded within the package, eliminating the need for external .wasm files or bundler plugins.
6. The library is completely self-contained, making it easy to use for both browser and Node.js environments.
7. The output is in the standard .zst format, which can be opened by the zstd command-line tool and other zstd tools.
The author explains that most JavaScript compression libraries work by compressing the entire input and output at once, which is impractical for large files due to memory constraints. The zstd-stream library solves this problem by processing files in small chunks, using backpressure to handle slower consumers, and providing a consistent API across different engines.
In summary, zstd-stream is a lightweight and efficient compression library that can handle files of any size while keeping memory usage minimal. It offers superior compression performance compared to the browser's built-in gzip and can be used in both browser and Node.js environments with a single installation package.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
