Spotify Solves Data Lake Latency with Random Access Parquet

Indexing the Data Lake for Online Point Queries

We address the latency bottleneck in cloud data lakes by introducing Random Access Parquet, or RAP. This approach uses an external index to map keys directly to file locations, enabling fast point queries without the overhead of distributed SQL engines. By collapsing dependent read chains into parallel operations, we allow AI agents and online services to retrieve user data from exabytes of storage at interactive speeds while maintaining a single source of truth.