Dimensionality, Not Tokenization, Drives LLM Failure in Tabular Prediction

Frontier large language models (LLMs) consistently struggle with predictive analytics on tabular data, often trailing behind classical machine learning baselines. A new systematic study has identified dimensionality as the decisive factor in this failure, revealing that LLM performance dissolves as the number of features grows.

Researchers tested five hypotheses in a pure inference regime—using a single generation pass without tools or fine-tuning—across 31 benchmark datasets. The study falsified four common theories for why LLMs fail: the inability to handle noisy or non-linear data, the obscuring nature of CSV formatting, numeric tokenization issues, and the number of test points per query. Instead, dimensionality emerged as the critical failure point. Among nine methods tested, the LLM was the only one whose accuracy decreased as dimensionality increased. In low-dimensional settings of just two dimensions, the LLM's predictions aligned with local, distance-based methods with up to 91.6% grid agreement.

The Classical Gap

Tabular data prediction is one of the most ubiquitous workloads in machine learning, yet generic LLMs have historically underperformed compared to classical baselines, some of which are decades old. This persistent gap has fueled the development of specialized "tabular foundation models," but the exact mechanism behind the failure of general-purpose LLMs has remained an open question until this evaluation.

To isolate the cause, researchers compared the LLM against 252 configured classical models. They found that no classical model, even those specifically tuned with noise, could reproduce the unique failure patterns the LLM exhibited in higher dimensions. The authors of the paper noted that "the LLM's capability dissolves with dimension in a way no noise-corrupted classical learner mimics."

Implications for AI Architecture

These findings narrow the search for the mechanism of LLM failure by ruling out surface-level issues like tokenization and formatting. The results suggest that the current architecture of LLMs is qualitatively different from classical learners in how it processes tabular features. This explains why LLMs cannot simply "reason" their way through complex tables without specialized training or external tools; they face a fundamental struggle with high-dimensional data processing.

What's Next

While the study pinpointed dimensionality as the culprit, the specific internal reason why the transformer architecture fails to scale with tabular dimensions remains to be fully decoded. Future research will likely focus on whether this is a limitation of the attention mechanism or a byproduct of how these models are pre-trained on natural language rather than structured data.