DuckDB Brings Data Power Tools to Your Laptop, Now in Clojure
DuckDB – Data power tools for your laptop, now in Clojure (2023)
TechAscent's tech.ml.dataset (TMD) platform now integrates DuckDB, enabling Clojure developers to process massive datasets locally. A 50GB CSV with 400 million rows loads into DuckDB in under two minutes, and a 1.4 billion-row join completes in 2.5 seconds. This integration leverages DuckDB's vectorized SQL engine, batched C interface, and zero-copy query pathway, offering a functional, memory-efficient alternative to distributed systems.
DuckDB complements TMD well and greatly increases a small team's ability to efficiently manage and process large datasets without needing to resort to expensive distributed solutions.
- eterm
Duckdb CLI is a powerhouse, it can load files as diverse as gzipped json lines, so you can stuff compressed logs straight into a directory yet still easily query them with SQL when you need to.
- kianN
I’m a big fan of tmducken. We use it heavily in our prod systems. That said, we’ve recently started exploring ducktape [1] in our new projects and have been really impressed with the performance. It also support more complex types on insertions and queries which has been helpful for us. Not affiliated with the project, but just wanted to show it some love since it’s a bit newer. It was created by an active contributor to tmducken.
- didibus
Impressive, you can really do a lot on a single node when it comes to big-data queries nowadays, I agree too many jump straight to a Spark cluster or something similar when you can just write a small script on a single node.