{
  "id": 414707,
  "title": "SQLite compressed text-history prototypes",
  "url": "https://urgent.news/2026/08/09/sqlite-compressed-text-history-prototypes",
  "topic": "culture",
  "section": "Culture",
  "published": "2026-08-09T22:05:00.000Z",
  "source": {
    "name": "Simon Willison",
    "slug": "simon-willison",
    "url": "https://simonwillison.net/2026/Aug/9/sqlite-text-history-prototype/"
  },
  "original_language": "en",
  "account": "Perpetually intrigued by methods of storing revision histories in relational databases, the author conceived a novel approach during a dog walk: compressing the full text of every prior version into a large JSON array of strings using zlib or zstd compression. The concept was discussed with the GPT‑Live voice mode in the ChatGPT iPhone app. The reporter relayed their thoughts as a stream of consciousness, outlining the prototype idea.\n\nTo implement the scheme, the reporter built systems in the past where every previous copy of the document added a significant amount of data to the database with each edit. The new idea was to compress all versions of the document, including the initial one, into a single JSON array and store it in a BLOB (binary large object) column. This would minimize redundant text and significantly reduce the database size.\n\nThe reporter proposed two columns: one for the JSON array of text and another for a JSON array of timestamps, which could be stored as integers. This approach would allow for efficient storage and retrieval of revision histories.\n\nAfter further discussion with GPT‑5.6, the reporter developed an experimental prototype using Python. The prototype simulated 1,000 revisions to a document, resulting in 20.4 MB of raw revision text that compressed to a mere 80.3 KB using Zstandard compression. To optimize the system, the reporter suggested breaking the history into multiple rows, with each row containing a maximum of either 128 revisions or 3 MB of uncompressed JSON. This would prevent the overhead of decompressing and recompressing the entire array on every edit.\n\nThe reporter concluded by acknowledging the success of the prototype, stating that it worked remarkably well. They also mentioned building the prototype using Python and building experimental prototypes around the idea. The whole process took 38 minutes, delivering the solution and the files seen in the folder.",
  "summary": "Research: SQLite compressed text-history prototypes I'm perennially interested in options for storing revision histories in relational databases. While out on a dog walk I had a new idea: how about taking the full text of every prior version in a big JSON array of strings and then applying zlib or zstd compression to the whole thing? Surely that would compress really well due to all of the…",
  "key_points": [
    "Compress full text of every prior version into JSON array using zlib or zstd compression.",
    "Store compressed JSON array in BLOB column to minimize redundant text.",
    "Prototype achieved 20.4 MB raw text compressed to 80.3 KB using Zstandard."
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}