NVIDIA Kumo Tabular Sets a New Accuracy-Efficiency Frontier for Tabular Prediction

NVIDIA Kumo Tabular is an open foundation model for tabular data prediction, now available on Hugging Face. This model sets a new accuracy-efficiency frontier for tabular prediction tasks. Unlike traditional gradient-boosted trees, Kumo Tabular can predict labels of new rows in a single forward pass, without any training, tuning, or feature engineering. It comes in three sizes (28M to 215M parameters) and can be used for both classification and regression tasks.

Kumo Tabular is built around the structure of a table, utilizing column, row, and in-context attention. Cell Embedding turns groups of cells into tokens, treating numerical and categorical values separately. Missing values are handled without imputation. Row Embedding compresses each row into an embedding using column attention and row attention, enabling the model to understand values within columns and interactions between features.

In-context Learning allows the final Transformer to operate on the row embeddings, with context rows attending to each other while query rows attend only to context rows. This approach ensures that predictions depend only on the context and the specific row, not on other rows being scored concurrently. Query rows utilize Test-GQA to shrink the cache and obtain point predictions and uncertainty estimates.

Finally, Length-aware Attention Temperature adjusts the softmax attention based on the number of keys, ensuring that the model maintains sharp attention as tables grow longer or wider. Kumo Tabular is pretrained on artificial tables, each generated from a Structural Causal Model, incorporating various real-world imperfections like missing values, coarsened features, and heavy-tailed targets. This extensive exposure to diverse tables enables the model to handle imperfections without additional cleanup.