How shredding JSON is giving Logfire 1000x query speedups

Logfire, a real-time observability platform, has dramatically improved query speeds by implementing dynamic shredding of semi-structured JSON data. In some cases, queries that previously took 30 seconds are now completed in under a second. The company plans to roll out this feature to all customers in February 2026.

OpenTelemetry data, which Logfire ingests, is semi-structured and often sent as JSON blobs over gRPC. Previously, this data was stored as JSON blobs in a single column, using Jiter to parse the data quickly. However, this approach led to suboptimal compression and high IO costs due to large JSON blobs and the time taken for downloads.

To optimize query performance, Logfire introduced dynamic shredding. This technique extracts the most frequently accessed attributes into separate, strongly typed columns during ingestion. Instead of storing all attributes in a single JSON column, attributes are extracted dynamically based on actual data, prioritizing the most frequently seen or largest attributes.

The result is a more efficient data layout that enables queries to scan only the shredded column, significantly reducing processing time. For example, a query for a specific attribute like "my_llm_response" no longer has to parse and scan potentially GBs of irrelevant JSON data. Instead, it can efficiently scan the dedicated "lf_attributes_my_llm_response" column.

This optimization allows Logfire to provide much faster query responses, even for complex JSON data, and is expected to greatly enhance the user experience for customers relying on real-time observability.