Indexing the Data Lake for Online Point Queries

Spotify's Random Access Parquet approach makes a large analytical dataset usable for targeted lookups, such as retrieving one listener's history. An external index records which files and rows contain each key. Cached file metadata then identifies the relevant pages, allowing the reader to request those byte ranges directly. Spotify's engineering article explains how this avoids a chain of reads just to discover where the requested data lives.

The approach reuses the data lake's existing Parquet files, so a separate full serving copy is unnecessary. It still requires an index and its upkeep. Unmodified files may also require reading a large page for a small result; preparing the file layout can reduce that overhead.