Infino moves its search index to S3 and slashes round trips
What changes when your search index lives on S3?
Infino built a retrieval engine whose full-text index lives on object storage (S3, GCS, Azure Blob) instead of local disk. Because every random read becomes a billed HTTP GET, the team optimized for fewer round trips: each index unit is a single Parquet superfile, a manifest inlines footers and index headers so opening a file needs no GET, and a term index tells queries which superfiles contain their terms. A single-term count that once opened 141 superfiles now opens none.
The scarcest resource is not CPU, and not bytes, but the number of round trips you make to the object store, and most design decisions below follow from that.
- mlhpdx
Putting reverse indexes in S3 alongside large aggregate log objects has been an effective pattern for more than a decade. With objects in cheap access modes and indexes in modes that are cheaper to access it becomes far more cost effective than most options (cheap S3 clones aside).
The earliest code I have to do this looks to have been created in 2012.
Separation of compute and storage is such a fabulous pattern.
- MattPalmer1086
Well, it answers the technical questions of what changes when you stick your search indexes on slower and more expensive storage - but not why you would do that in the first place.
- tilt_error
Isn't this the same as Elasticsearch searchable snapshots?