How Snowflake Turned Postgres Replication into Clockwork
How We Pushed CDC into Postgres
Snowflake's Postgres service introduces data mirroring, a new feature that pushes changes directly from Postgres into Iceberg tables, eliminating fragile connectors and complex failure modes. By using a Postgres extension for change data capture and transactional apply on the Snowflake side, replication becomes a self-maintaining process with low lag and high performance. The article details the engineering behind this push-based approach, including timeline management, transactional boundaries, and live views.
You press a button, and you have your Postgres tables in Snowflake.
- bastawhiz
Clickhouse really nailed this with the acquisition of peerdb. I used it with many terabyte databases and I essentially never thought about it. The only thing we really had to watch for was trying to replicate too much at once (because of the physical compute/io capacity of the postgres or clickhouse clusters).
- gopalv
This was basically Vertica's party trick for quite a long time to have a WOS and ROS formats for the same row and anti-caching between those two.
You could've built a similar system with dezebium and delta lake for quite some time but it would fail compactions, if you run it fast enough. I've seen Oracle GoldenGate 12c do this trick in 2014 or so, using Mysql as the cheap replica. But they are all fragile to schema updates in some direction.
The closest batteries-included equivalent to this is the Aurora -> Redshift bridge[1].
- hasyimibhar
It's interesting to watch how different companies that offer both Postgres and warehousing solution under 1 roof approach the same problem:
- ClickHouse focuses on traditional CDC (ClickPipes) and just make it blazingly fast
- Databricks leans on their unified storage architecture (LTAP) to avoid copying data (though you can argue there is still a copy in the cache)
- Snowflake uses a data mirroring CDC as extension so it runs directly on Postgres
I'm still waiting for a Postgres provider to just let me mirror data directly to Iceberg, so I can plug in my own stateless query engine.
- whateveracct
Snowflake is a really amazing product. It's been a delight using it the last few years.
- jauntywundrkind
Although pg_lake is open source, worth noting that it heavily refers to but is missing CDC capabilities.
There's a bunch of comments/links to a closed https://github.com/snowflake-eng/sfpg-extension-pg_lake_repl...