Logfire's JSON shredding makes queries 1,000x faster
Shredding JSON columns gave Logfire up to 1000x faster queries
Logfire, Pydantic's observability platform, stored semi-structured OpenTelemetry attributes as JSON blobs, forcing queries to scan and parse gigabytes of irrelevant data. Dynamic shredding automatically extracts the most frequently accessed attributes into strongly typed columns during ingestion, enabling column pruning, zone-map skipping, and zero JSON parsing. Some queries that timed out after 30 seconds now finish in under a second. Rollout to all customers begins February 2026.
Queries that previously timed out after 30 seconds now complete in under a second.
- sgarland
It is hilarious to me when companies act like they've made some huge discovery that's been well-known to anyone working in DBs for the past decade, if not more.
JSON is slow as hell, both for serdes, and especially querying it. Use it sparingly, if at all. Normalization is not that hard, and it's trivial now with LLMs. Tell Claude you want your schema in BCNF, or 5NF if you're feeling froggy. Denormalize if and only if you have performance issues.
- zX41ZdbW
This is quite an old technique. In ClickHouse, it appeared before it was named "shredding": https://github.com/ClickHouse/ClickHouse/issues/54864
- japgolly
> Rather than storing all attributes in a single JSON column, we automatically extract the most frequently accessed attributes into separate, strongly typed columns.
I'm glad they made the discovery but this is DB 101