PostHog abandons DuckDB after its 2 PB warehouse breaks in ways it couldn't fix
Moving our warehouse off DuckDB
PostHog rebuilt its data warehouse on DuckDB five months ago, giving each customer a dedicated instance. But customer workloads grew far more complex than expected, forcing oversized machines and making resource prediction nearly impossible. DuckLake, the lakehouse extension, also buckled: a single-table write could take over three minutes, and 99% of catalog statistics belonged to deleted tables. PostHog is now rebuilding on its own Trino flavor and a new open table format called HogLake.
When we moved off it, it held about 2 PB: roughly 1 trillion rows and 15.3 million snapshots. It broke in lots of ways that were hard to fix within DuckLake's scope.
- Aurornis
> Five months ago, we rebuilt our warehouse on DuckDB instead of ClickHoue
They switched from ClickHouse to DuckDB five months ago, then switched from DuckDB to their own Trino derivative?
The article then ends before saying if the new setup solved their problems.
It’s great that we can now make these migrations with little downtime (I assume?) and try different things, but I wish we could have read more about what testing they did before jumping. The fact that they jumped to DuckDB without realizing it wasn’t going to work at their scale and spent only a few months with it before jumping again is not convincing.
I hoe there’s more to this story that just didn’t make it into the blog posts. As written it’s giving me flashbacks to some teams I worked with who changed tech stacks every six months because they liked migrating more than tuning and maintaining.
- 0c3ca83
Am I missing something here, or is the only technical detail in the post "a trillion rows is a lot of data, maybe trying trinio will work this time"?
It feels like a lot of churn and not a lot of measurements or engineering.
- bitpush
> Five months ago, we rebuilt our warehouse [1] on DuckDB instead of ClickHouse.
Why did they write this post if they had not even allowed things to settle. It sounds like someone wanted to claim victory (and promo?)
- kwillets
Sometimes a company just isn't able to manage a data warehouse; I've worked for a few.
I even phone screened with this one but couldn't cover their crazy quilt of requirements; at that time they were also looking for someone to rewrite the clickhouse optimizer, and they were stuck trying to reshard their cluster, which should take an afternoon.
I still like the company because their business prop reminds me of what we were doing in 2009, a kind of federated analytics organization on a centralized data warehouse that enables a lot of tools to be developed quickly at large scale.
But goddammit they keep picking these half-working platforms and flip-flopping from one to another. We were crushing millions of queries against 2T-row tables on Vertica in 2010, and it's only gotten better for customer-facing data apps, which is what Posthog is. The company doesn't seem to have a baseline of what to expect; the market is too saturated with slow tools that lower expectations.
- hasyimibhar
I wonder if they have considered moving back to ClickHouse. ClickHouse Iceberg support has improved a lot over the past few months. You can easily run stateless ClickHouse on top of Iceberg. This solves the tenancy issue, and it can easily handle petabytes scale [1].
[1] https://clickhouse.com/blog/jump-trading-uses-clickhouse-wit...
- avree
Haven't seen Posthog's website in a while. Man, it's hideous.
- whinvik
The key insight I am getting from this is that as migrations get easier to do with agents, there will be a lot more of them.
But more importantly a lot of them will happen because we will all get more trigger happy to start one. On the other hand I also don't know if at scales like PostHog, a successful migration can be tested without doing all of it.
- yukIttEft
Not sure their singlenode customers will be happy with Trino
https://benchmark.clickhouse.com/#system=+Hoaus|ucqs|Tuts&ty...