How to Let gzip Find the Signal in a Pile of Documents

When confronted with a directory filled with text documents, many of which are repetitive or low in signal content, one may consider manually reading each file or employing an embedding pipeline coupled with an LLM to rank them. Alternatively, a simpler method involves utilizing gzip. The rationale behind this approach is rooted in the principle that repetitive text compresses well, whereas varied text generally does not.

Thus, compression ratio can serve as a crude yet effective proxy for redundancy; however, it should not be mistaken for an indicator of document quality or semantic importance.

To achieve this, one can follow a straightforward procedure: for each document, measure its original size, compress it individually using gzip, and then measure the compressed size. Calculate the ratio of the compressed size to the original size; a lower ratio indicates a document that compressed well, typically suggesting more repetition.

Conversely, a higher ratio implies the document was harder to compress, which may point to more varied or information-dense content. In essence, a lower ratio signifies higher redundancy, while a higher ratio suggests less redundancy.

The Bash command to rank .txt files by compression ratio is as follows:

```bash

find ./documents -type f -name '*.txt' -print0 |

while IFS= read -r -d '' file; do

raw=$(wc -c <"$file")

compressed=$(gzip -n -c --"$file" | wc -c)

awk -v file="$file" -v raw="$raw" -v gz="$compressed" 'raw>0 {printf "%.3f\t%8d\t%8d\t%s\n", gz/raw, raw, gz, file}'

done | sort -nr

```

The output comprises four columns: the compression ratio, original bytes, compressed bytes, and the filename. Files are sorted in descending order, placing the least compressible files at the top. These would be the first documents to inspect when seeking potentially valuable information. To identify the most repetitive documents, reverse the sort order using `sort -n`.

It is crucial to note that gzip -n prevents the inclusion of the original filename and timestamp in the output, thus ensuring more comparable compressed sizes across files and runs. Moreover, small files may produce noisy ratios due to gzip's overhead, so it might be wise to disregard files below a certain size threshold.

The technique, while not measuring truth, relevance, writing quality, or semantic importance, effectively measures compressibility. This metric can serve as a first-pass ranking signal, akin to a metal detector rather than a treasure map, highlighting documents that contain less repetition. However, it is essential to consider caveats such as small files yielding noisy results, already compressed formats misguiding the measurement, and incompressible content like encrypted data or random identifiers.

Additionally, repetition alone does not always equate to fluff, as contracts, API documentation, technical specifications, and scientific papers often require precision, leading to consistent repetition.

Several factors, such as document length, whitespace, markup, tables, repeated headings, source language, character encoding, and templated metadata, can influence compression ratios. Normalizing the documents before compression, for instance, by removing excess whitespace or converting everything to lowercase, can help yield a fairer comparison.

For larger collections, filtering out tiny files and calculating the percentage of bytes saved can provide a slightly more useful output, sorting files with the lowest percentage saved first. This approach can be useful for quickly triaging scraped web pages, support tickets, meeting transcripts, research notes, log samples, generated reports, and document archives.

It is particularly valuable for rapidly triaging large sets of Markdown files without the need for a database, embedding model, or external API. Ultimately, treating compression ratio as a lightweight feature in a ranking system, alongside other signals such as document length, vocabulary diversity, duplicate paragraph counts, keyword density, entropy, embedding similarity, recency, and source reputation, can enhance document triage capabilities.

In essence, even a 40-year-old compression algorithm can offer insights into which files contain repetitive content, serving as an excellent starting point for document triage.