PlanetScale's TIN brings full-text search to Postgres, and it's 25x faster than ParadeDB
Tin: full-text search for Postgres
PlanetScale has launched TIN, a GA full-text search extension for Postgres that supports boolean, phrase, and span queries, fuzzy matching, and BM25 ranking. Benchmarks on an 85 GB Stack Exchange corpus show TIN handles 25× the queries per second of ParadeDB with 26× lower p99 latency, while also sustaining hundreds of updates per second. TIN uses Postgres ctid values as document identifiers, enabling vectorized operations and dramatically less disk I/O.
TIN handles 25× as many queries per second as ParadeDB does, with p99 latencies 26× lower.
- andrenotgiant
I think what we're seeing with every database company providing new full-text search capabilities is an example of AI coding productivity showing up in the real world.
It started with paradeDB and pg_search https://www.paradedb.com/blog/introducing-search
Timescale has pg_textsearch https://github.com/timescale/pg_textsearch
Neon and Databricks have Lakebase Search https://docs.databricks.com/aws/en/oltp/projects/lakebase-se...
Now PlanetScale.
AFAIK all of these are implementations of the BM25 algorithm. You can just tell an agent to read about BM25 and implement it in your system of choice. Cool to see. Seems like there's still a lot of juice to be squeezed out of how it's architected and integrated into each system, but you can't help but wonder if this will lead to aggressive commodification
- Doohickey-d
There's another aspect which none of the FTS search solutions for Postgres do well in my opinion: multi-language support.
For example this one: it doesn't mention support for CJK languages (meaning tokenization for e.g. Chinese will resolve to one token per character, which will technically work and give results, but is inefficient). Also word stemming (databases -> database) is also missing as far as I can see, so the kind of queries where you'd expect related words to show up will be missing. Just doing case-folding and accent-folding is a bit of a functional but bruteforce solution.
Ideally I'd want something that supports:
- language aware tokenization, with ability to define the language per record. Including stemming, etc. And have useful predefined configuration for common languages (e.g. the Postgres built in one is missing many languages).
- CJK support, tokenizing at word boundaries.
- Optional accent- and case-folding.
Most solutions just seem to assume English content, I have not found anything that does all of this yet.
- Tiberium
If anyone's curious - https://planetscale.com/docs/postgres/search/get-started#loc...:
They're not providing a local extension with the same performance at the time - it's only offered on their cloud services.
The local version https://github.com/planetscale/lead is mainly just for testing the syntax, it doesn't have the same perf characteristics.
- groundzeros2015
Please read the Postgres manual. It has incredible built-in search capability.
- tannhaeuser
Postgres does have pg_fts (tsvector/tsquery/tsrank) which is a quite sophisticated full text search package integrated with functional indexing and query optimization. Why would I use something vibecoded that isn't part of core Postgres instead?