PCA beats Matryoshka truncation for shrinking embeddings

Honey, I shrunk the embeddings: Matryoshka vs. PCA

A comparison of Matryoshka Representation Learning (MRL) truncation and Principal Component Analysis (PCA) for reducing embedding dimensions across eight BEIR datasets reveals that PCA often matches or outperforms MRL, especially at smaller sizes. On OpenAI's text-embedding-3-small, PCA retains 65% of retrieval quality at 32 dimensions versus 46% for truncation. PCA also works well on non-MRL models, and fitting on a small or out-of-domain sample has minimal impact at moderate compression.