DuckDB's Asynchronous I/O: 3x Faster Queries on S3
Asynchronous I/O in DuckDB: Work, Thread, Work
DuckDB v2.0, due this fall, introduces asynchronous I/O for Parquet and CSV files, dramatically speeding up queries on remote storage like S3. By using a dedicated thread pool and read-ahead queue, DuckDB can saturate network bandwidth, cutting TPC-H Query 6 runtime from 8.23 to 2.84 seconds (3x faster). The feature is available now in preview builds.
It doesn't matter how fast query operators are in a database system if we can't pull in the data quickly.