PCA beats Matryoshka truncation for shrinking embeddings
Honey, I shrunk the embeddings: Matryoshka vs. PCA
A comparison of Matryoshka Representation Learning (MRL) truncation and Principal Component Analysis (PCA) for reducing embedding dimensions across eight BEIR datasets reveals that PCA often matches or outperforms MRL, especially at smaller sizes. On OpenAI's text-embedding-3-small, PCA retains 65% of retrieval quality at 32 dimensions versus 46% for truncation. PCA also works well on non-MRL models, and fitting on a small or out-of-domain sample has minimal impact at moderate compression.
PCA matched or outperformed MRL truncation at nearly every dimension across both MRL-trained models.
- purplemoonx
You train the model with the loss applied at several prefix lengths at once: the first 64 dimensions, the first 128, and so on. This teaches it to pack the most important information at the start of the vector, like a set of nested matryoshka dolls.
This is fascinating if it works as well as the experiments make it seem. For example, how does it compare to classic image resize algorithms like Seam Carving or Inpainting: https://en.wikipedia.org/wiki/Seam_carving, https://en.wikipedia.org/wiki/Inpainting
(can they be compared?)
Compression/loss, and the opposite - scaling up and oversmoothing - are fascinating in that any time even the tiniest innovation happens in those areas, all this other technology improves overnight, and a bunch of new technology becomes possible.
- 5555watch
PCA is good, but you could also try playing around with Sparse (robust) PCA. The sparsification loses orthogonality, but does not necessarily lose information, it can yield a different rotation and cleaner vectors. Now whether that matters in the context of LLMs/Embeddings - I cannot tell.
- hanneshdc
Good benchmarks!
> You can push this further by combining quantization with truncation or PCA. The resulting vectors can be dramatically smaller while still preserving a surprising amount of retrieval quality.
Counterintuitively - quantisation can also be combined with a random rotation step before the quantisation. A random rotation spreads information across more dimensions, allowing more aggressive quantisation without losing accuracy. Ironically - almost the opposite of a PCA.
I do wonder if relevant here though. It relies on the embeddings having "structure", i.e. that principal components point along basis vectors, which may not be the case with text embeddings.
Source: https://research.google/blog/turboquant-redefining-ai-effici...