{
  "id": 13658155,
  "title": "How shredding JSON is giving Logfire 1000x query speedups",
  "url": "https://urgent.news/2026/10/11/how-shredding-json-is-giving-logfire-1000x-query-speedups",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-10-11T06:16:26.000Z",
  "source": {
    "name": "Lobsters",
    "slug": "lobsters",
    "url": "https://pydantic.dev/articles/dynamic-shredding-2026-01-26"
  },
  "original_language": "en",
  "account": "Logfire, a real-time observability platform, has dramatically improved query speeds by implementing dynamic shredding of semi-structured JSON data. In some cases, queries that previously took 30 seconds are now completed in under a second. The company plans to roll out this feature to all customers in February 2026.\n\nOpenTelemetry data, which Logfire ingests, is semi-structured and often sent as JSON blobs over gRPC. Previously, this data was stored as JSON blobs in a single column, using Jiter to parse the data quickly. However, this approach led to suboptimal compression and high IO costs due to large JSON blobs and the time taken for downloads.\n\nTo optimize query performance, Logfire introduced dynamic shredding. This technique extracts the most frequently accessed attributes into separate, strongly typed columns during ingestion. Instead of storing all attributes in a single JSON column, attributes are extracted dynamically based on actual data, prioritizing the most frequently seen or largest attributes.\n\nThe result is a more efficient data layout that enables queries to scan only the shredded column, significantly reducing processing time. For example, a query for a specific attribute like \"my_llm_response\" no longer has to parse and scan potentially GBs of irrelevant JSON data. Instead, it can efficiently scan the dedicated \"lf_attributes_my_llm_response\" column.\n\nThis optimization allows Logfire to provide much faster query responses, even for complex JSON data, and is expected to greatly enhance the user experience for customers relying on real-time observability.",
  "summary": null,
  "key_points": [
    "Logfire implements dynamic shredding of semi-structured JSON data.",
    "Queries that previously took 30 seconds now complete in under a second.",
    "Feature to be rolled out to all customers in February 2026."
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}