Why LLMs Fail at Tabular Prediction: Dimensionality Is the Killer
Why Large Language Models Fail at Tabular Prediction
Large language models excel at many tasks but struggle with tabular data. A new study tests a frontier LLM in a pure inference setting, ruling out noise, CSV formatting, tokenization, and query size as culprits. Instead, dimensionality is decisive: as input dimensions increase, LLM accuracy drops while classical baselines stay flat or improve. The LLM's predictions resemble distance-based methods in 2D but diverge in higher dimensions, suggesting a fundamental limitation.
The LLM's capability dissolves with dimension in a way no noise-corrupted classical learner mimics.
- _joel
The first thing I'd do if working with an LLM on tabular data is to ask what the best tool would be to work with that data and build up a proper harness to work with the data sensibly. Rawdogging LLM isn't the tool for forecasting like this, as they found.
- rustyconover
Look at the white text on white background in Appendix F. Pretty funny.
- tough
Google's TabFM [1] (and its previous TimesFM) seem the best approach so far on this area
1. https://research.google/blog/introducing-tabfm-a-zero-shot-f...
- ismailmaj
Unsure if it's LLMs that fail at tabular data or its just that tree boosting are spectacular at that task.
- pjot
One step further are those who want to point an llm directly at the data warehouse to get the data needed to run predictions