DuckDB's DuckLake unifies data lake and catalog in SQL
DuckDB Ducklake
DuckLake is an open lakehouse format from DuckDB that stores metadata in a catalog database and data in Parquet files. The DuckDB extension lets you attach a DuckLake database and use standard SQL for tables, updates, time travel, schema evolution, and change data feeds. It supports multiple catalog backends including PostgreSQL and SQLite.
DuckLake is an open Lakehouse format that is built on SQL and Parquet.
- smithclay
Don't think it's widely known or understood, but Ducklake doesn't require duckdb: it's just a (good) data lake spec that works in duckdb.
There's a cool alternate rust/datafusion ecosystem initiative going on at https://github.com/datafusion-contrib/datafusion-ducklake, and think the Quack protocol opens up a lot of cool possibilities too.
If you need an idea for what do do with ducklake: recommend throwing all of your agent traces in it.
- engineeringwoke
It's alright, it's pretty alpha software. On v1.5.4, catalog filtered counts are broken, afaik. I went to main/v2 to fix it, and then the SQL parser in duckdb v2 is 10x slower, which was another wrench in the gears. It's been a bit of a pain tbh
- celias
Motherduck is offering a free copy of O'reilly's "DuckLake: The Definitive Guide" book on their DuckLake web page