SQLite Is All You Need: Building a Social Network on a Single File
I built a social network called Chirp using only SQLite to prove its production viability. With 50,000 users and a million posts in one file, our Node API handled 315 million requests daily on a laptop. By enabling WAL mode and STRICT tables, we achieved massive throughput without a database server, showing that most teams do not need Postgres containers.
The Postgres container you spun up out of habit was never needed.
- rmunn
There's some bad advice in the article:
"Backups are a file copy. [...] you can back up a live SQLite database, under write load, without stopping anything."
This is straight out of section 1.2 of https://www.sqlite.org/howtocorrupt.html. Yes, you can do that, and sometimes you will end up with a valid, non-corrupt backup. But it's timing-dependent: lose the race and you'll end up backing up a partially written transaction, making the backup corrupt. They didn't end up losing that race when they wrote the article, but that doesn't mean it is safe 100% of the time.
The section later on about running "VACUUM INTO backup-$(date +%F).db" is 100% safe, though: SQLite guarantees that you'll get consistent state if you do that.
- petcat
> the Postgres container you spun up out of habit was never needed
These posts need to stop comparing their contrived use-cases for SQLite to Postgres. Sure, SQLite is all you need if it really is all you need. But Postgres does so much more than just act as a data dump with an SQL engine on top.
Dr. Hipp himself even said that SQLite does not, and will never, compete with the likes of Postgres. It competes with fopen.
- chistev
The SQLite documentation says that
"SQLite works great as the database engine for most low to medium traffic websites (which is to say, most websites). The amount of web traffic that SQLite can handle depends on how heavily the website uses its database. Generally speaking, any site that gets fewer than 100K hits/day should work fine with SQLite. The 100K hits/day figure is a conservative estimate, not a hard upper bound. SQLite has been demonstrated to work with 10 times that amount of traffic.
The SQLite website (https://www.sqlite.org/) uses SQLite itself, of course, and as of this writing (2015) it handles about 400K to 500K HTTP requests per day, about 15-20% of which are dynamic pages touching the database. Dynamic content uses about 200 SQL statements per webpage. This setup runs on a single VM that shares a physical server with 23 others and yet still keeps the load average below 0.1 most of the time."
- arpinum
I use a in-memory database per unit test with both rocksdb and sqlite, it is a game-changer to get better quality tests.
Overall premise is wrong though. Moving the database out of process will change performance characteristics and data architecture too much and will cause massive headaches at exactly the time when you are trying to scale with success. You should have out of process performance tests early to catch these issues, even if you do deploy a single node.
If success can be satisfied with a single node and you are satisfied with availability and recovery that gives you then great, but it isn't all I need.
- afonsosoares
I could not agree more KISS is always the way to go.
For those people that are interested you can have a sqld instance on a different VM with S3 backup.
On my case I use k8s and the backend pod use rust libsqld crate with local first sql database file with remote sync to sqld.
When pod start if db file is not available libsqld will try recover it from slqd otherwise it will just load local db file and sync.