When an LLM Beats a Statistical Model, and When It Doesn't

Insurance pricing, time series forecasting, and the creation of GPT-3 all serve as case studies where large language models (LLMs) outperform classical machine learning models. In insurance pricing, researchers found that converting data points into a sentence and feeding it to an LLM resulted in better claim frequency predictions than a traditional generalized linear model (GLM), especially when data was limited.

The LLM's ability to automatically capture interactions between variables proved advantageous when data was scarce. However, as more data became available, the LLM's edge faded, and the GLM performed just as well. Time series forecasting revealed that simply swapping an LLM for its constituent attention layer did not improve performance, as the model failed to learn critical sequential dependencies.

Google's TimesFM and Uber's forecasting stack, which transform numeric data into fixed vocabularies before applying transformer-based models, demonstrated the importance of proper input representation. In summary, LLMs prove effective when they can effectively capture interactions between variables or when input representation is tailored to the problem at hand. Otherwise, classical machine learning models may suffice.