talent

Tabular Foundation Models on TALENT

This benchmark evaluates structured-data-models on the corrected 300-dataset TALENT benchmark.

Setup

Run the commands below from the repository root:

pip install . \
  "TALENT @ git+https://github.com/LAMDA-Tabular/TALENT.git@08301d6"

Download and extract the datasets from the official TALENT dataset page.

Run

• TabICLv2:

  python -m benchmark.tabular.talent.main --model tabiclv2 --dataset-path /path/to/talent/data

• KumoTabular:

  python -m benchmark.tabular.talent.main --model kumo-tabular-large --dataset-path /path/to/talent/data
  python -m benchmark.tabular.talent.main --model kumo-tabular-medium --dataset-path /path/to/talent/data
  python -m benchmark.tabular.talent.main --model kumo-tabular-small --dataset-path /path/to/talent/data

• TabFM:

  python -m benchmark.tabular.talent.main --model tabfm --dataset-path /path/to/talent/data

Note

Weights of TabFM are distributed under the TabFM Non-Commercial License v1.0. Review the license before running the TabFM benchmark, which will download its weights noninteractively.

Pass a dataset name to run only that TALENT dataset:

python -m benchmark.tabular.talent.main \
  --model kumo-tabular-large \
  --dataset-path /path/to/talent/data \
  --dataset Bank_Customer_Churn_Dataset

Fine-tuning

Add --finetune to full fine-tune every parameter of the selected --model on each dataset's training split before evaluating (see benchmark/tabular/finetune.py). Unlike TabArena/BeyondArena's dedicated -ft model choices, this is a flag applied to whichever model you picked. --model kumo-tabular-small-ft is kumo-tabular-small capped at 10 classes, for fine-tuning stability on multiclass datasets:

python -m benchmark.tabular.talent.main \
  --model kumo-tabular-small-ft \
  --dataset-path /path/to/talent/data \
  --finetune \
  --finetune-epochs 75 \
  --finetune-lr 1e-6

Fine-tuned results use <model>-finetuned as their output folder and model label, keeping them separate from baseline results and the capped kumo-tabular-small-ft variant.

Evaluate

Evaluate all available model results with:

python -m benchmark.tabular.talent.evaluate

Install matplotlib and scikit-posthocs, then add --plot-cd to generate critical-difference diagrams as PNG files. The evaluator downloads TALENT's published result tables by default; pass their local directory with --official-results to run offline.