Zerodha Built a Blob Store on SQLite That Handles 1.66 GB/s of JSON
Distributed, high-performance blob storage on SQLite

Zerodha's Kite trading platform stores millions of JSON blobs for user layouts and preferences, and its aging SSDB store couldn't scale. So the team built Blobasaur, an open-source Rust server that speaks the Redis protocol and shards data across multiple SQLite files. With 16 shards it hits 25,324 writes per second at single-digit millisecond latency, replacing SSDB in production.
Putting two and two together, we thought, why not build a system on top of SQLite to store millions of blobs and see what happens?
- anacrolix
I spent years trying to squeeze this use case out of sqlite. All roads lead to tears, it is not made for this use case.
I wrote https://github.com/anacrolix/possum for this exact use case. It has a few rough edges but it absolutely hauls ass for caching on disk. Incidentally it uses sqlite for transaction/TTL metadata which is more suited to sqlite but it's still the bottleneck.
sqlite transaction commit overhead is too high and uncompetitive without batching but it's a design flaw specific to being multiprocess.
- lukax
OpenStack Swift uses SQLite for storing metadata. Each account has one SQLite db. Each container also initially had one SQLite db but in 2018 they added support for sharding so if a container has more than 1 million objects, the container is split into multiple SQLite databases. The performance is not ideal. Ceph scales much better.
- gkamal
This looks like a classic case of nih/reinventing wheel.
- bob1029
> JSON blobs
An alternative to faster blob storage is to represent the blob's schema in SQL directly.
If most blob updates are incremental you can eliminate a significant amount of storage IO and garbage collection this way.
One SQLite database per blob type with a metadata SQLite db that tracks each. We could leverage powerful AI reasoning models to produce SQLite schemas and virtual table adapters for each encountered JSON schema type.
- matja
What is the consistency model? What happens if a node is down unexpectedly?