A single network that ensembles with itself on tabular data

What it does

TabM is a PyTorch architecture for tabular data that mimics an ensemble of k MLPs while physically remaining one model. It trains those virtual MLPs in parallel and shares their weights, which trims memory and runtime compared to a literal bag of separate networks. The package also exposes layers for building custom TabM-like architectures.

The interesting bit

The twist is that weight sharing is not just a compression hack; the README notes it acts as an effective regularizer that actually improves task performance over a true ensemble. You also monitor the ensemble performance during training rather than individual model performance, letting you stop when the collective is optimal, not when one member is.

Key highlights

• Won multiple Kaggle competitions outright, and placed 25th out of 3300+ entries when used solo without any external ensembling.
• Outperforms prior tabular deep-learning methods on the TabReD benchmark, which uses industrial datasets with time drift and hundreds of features.
• Scales to large datasets: tested on 13 million objects in the paper, with reported successful runs on 100 million+ rows.
• Ships as a PyTorch package with building blocks for custom efficient-ensemble architectures.

Caveats

• Inference throughput is slower than plain MLPs and gradient-boosted trees, though faster than prior tabular deep-learning approaches.
• Training on 100-million-row datasets is possible but explicitly noted to take more time.
• The README warns that constructing correct custom architectures requires understanding implementation details; the simplicity can be deceptive.

Verdict

Data scientists working on tabular competitions or industrial datasets with distribution drift should try this before reaching for a heavy stack of separate models. If you need absolute inference speed above all else and gradient-boosted trees already suffice, it is likely overkill.