Seldon uses synthetic causal graphs to train tabular foundation models

Neuralk-AI has released Seldon, a new tabular foundation model hosted on Hugging Face. Published in early November 2026, this architecture targets machine learning engineers who need robust performance on structured data without relying on massive public datasets for pretraining.

What happened

The release details a model that diverges from standard pretraining practices by avoiding real-world data entirely. Instead of scraping public repositories or using proprietary corporate tables, the developers trained Seldon exclusively on synthetically generated datasets. This approach aims to solve the scarcity and privacy issues often associated with gathering large-scale tabular data for foundation models.

Seldon follows architectural patterns seen in Kumo Tabular and TabFM. It compresses entire tables into row-level representations and then applies in-context learning. The model treats labeled context rows as its training set during inference, allowing it to adapt to new tasks dynamically. This design enables the model to generalize across different tabular problems by learning the underlying structure of data relationships rather than memorizing specific dataset statistics.

How it works

The architecture begins with a specialized tokenizer. Each cell in a table is converted into a token using a Fourier feature projection for numerical values, combined with a categorical mask and a missing-value indicator. Features are grouped before this projection to maintain structural coherence. These tokens pass through interleaved column and row transformers. The column attention mechanism uses inducing points to process information across rows efficiently. The output is compressed into CLS readout tokens, resulting in one dense vector per row.

Once the rows are embedded, the model enters the in-context learning phase. A transformer stack processes these compressed row representations. Labeled context rows provide the necessary signals, and the model predicts outcomes for query rows based on this context. Finally, a multi-layer perceptron head maps each query row’s representation to class probabilities. The total parameter count is 213.46 million, split between the embedding stage and the in-context learning stack.

The pretraining strategy relies on a prior developed by Neuralk based on structural causal models. Each synthetic task originates from a randomly sampled causal graph. Nodes in these graphs are linked by randomly drawn functional relationships and noise. The model reads features and targets directly from these graphs. This method generates diverse scenarios by varying graph structures, functional dependencies, feature types, distributions, noise levels, missingness patterns, and class balances.

Key details

• Total parameters: 213.46 million, with 9.93 million in the cell/row embedding module and 203.53 million in the in-context learning stack.
• Pretraining data: Exclusively synthetic datasets generated from structural causal models; no real-world data was used.
• Embedding shape: 256 channels, 6 layer pairs, 256 inducing points, and 4 readout tokens.
• ICL stack configuration: 24 layers, 1024 channels, and 16 attention heads.
• Tokenization method: Uses Fourier feature projection for numerical values, categorical masks, and missing-value indicators.
• Inference mechanism: Uses labeled context rows as the training set to predict query rows via in-context learning.

Why it matters

For engineers building data products, the shift to synthetic pretraining offers a significant advantage in data privacy and compliance. Since the base model never sees real user data during its initial training, organizations can fine-tune or use it in-context without worrying about leaking sensitive information from public sources. This makes it easier to deploy in regulated industries where data provenance is strictly audited.

The use of structural causal models for generating synthetic data also addresses the problem of overfitting to specific dataset biases. Real-world datasets often contain spurious correlations or limited diversity in feature distributions. By exposing the model to a wider range of causal structures and noise patterns during pretraining, Seldon learns more robust inference mechanisms. This generalization capability means the model may perform better on unseen real-world tables that differ significantly from common benchmark datasets.

What you can do

• Download the Seldon model weights and tokenizer from the Hugging Face repository to test local inference.
• Generate your own synthetic datasets using structural causal models to evaluate how the model handles specific noise patterns.
• Experiment with different numbers of context rows during in-context learning to find the optimal balance for your latency requirements.
• Compare the Fourier feature projection results against standard normalization techniques to understand the impact on numerical tokenization.
• Review the inducing point mechanism in the column attention layer to optimize memory usage for large tables.
• Test the model on tabular tasks with high missingness rates to leverage its built-in missing-value indicators.