Skip to content

About

Official code for 'Where Does Warm-Up Come From? Adaptive Scheduling for Norm-Constrained Optimizers' (NeurIPS 2026)

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Where Does Warm-Up Come From? Adaptive Scheduling for Norm-Constrained Optimizers

Official code for the NeurIPS 2026 paper by Artem Riabinin, Andrey Veprikov, Arman Bolatov, Martin Takáč and Aleksandr Beznosikov (arXiv:2602.05813).

For norm-constrained optimizers, whose step is a linear minimization oracle (LMO) over a norm ball (Muon, Lion, normalized SGD, Scion), we show that the learning rate selected by the convergence analysis first grows and then decays as a function of the suboptimality gap $\Delta = f(x) - f^\star$:

$$\eta(\Delta) = \frac{\Delta}{K_0 + K_1 \Delta + K_2 \Delta^2}.$$

The scheduler in this repository calibrates $K_0, K_1, K_2$ from the peak learning rate, the initial learning rate and a target schedule shape, so the warm-up length is set by the training trajectory instead of a grid search. Once the gap reaches the end of the warm-up phase, training continues with a standard decay (cosine, linear or WSD).

The code is built on llm-baselines (EPFL MLO), following the setup of Semenov et al. (2025). The upstream README is kept as README_llm_baselines.md.

Installation

python3.11 -m venv optim_venv
source optim_venv/bin/activate
pip install -r requirements.txt

FineWeb is tokenized with the GPT-2 tokenizer into train.bin and val.bin under <datasets_dir>/fineweb-100BT/ (src/data/fineweb.py). Pass --datasets_dir if the data lives outside ./src/data/datasets/. Swin-B experiments expect ImageNet-1K in the layout read by src/data/imagenet.py.

What is where

File Content
src/optim/lipschitz_scheduler.py The adaptive scheduler (Algorithm 1 in the paper)
src/optim/lipschitz_computeK.py Calibration of $K_0, K_1, K_2$ and $\Delta'$, closed-form $\kappa$ for each norm
src/optim/lipschitz_analyzer.py Measurement of the empirical smoothness ratio $\mathcal{K}^t$ along a trajectory
src/optim/muon.py, lion.py, normalized.py, scion.py The LMO optimizers
src/models/llama.py, src/models/swin.py LLaMA and Swin-B models

Using the scheduler

The adaptive warm-up is enabled with --use_lip_warmup. The remaining flags are the usual ones:

python ./src/main.py --model llama --dataset fineweb --optimizer d-muon \
    --lr 1e-3 --div_factor 100 --scheduler cos \
    --use_lip_warmup \
    --lipschitz_loss_star 3.2 \
    --lipschitz_sigma_F 0.001 \
    --lipschitz_rho 2.0 \
    --lipschitz_mode linear_and_cos \
    ...
  • --lr is the peak learning rate and --lr / --div_factor the initial one.
  • --lipschitz_loss_star is the estimate of $f^\star$. We use a value close to the loss the model can reach at its token budget. The ablations in the paper show that the result is not sensitive to it within a reasonable range.
  • --lipschitz_sigma_F is the width of the calibration window. All experiments in the paper use 1e-3.
  • --lipschitz_mode linear_and_cos is the calibration objective used in the paper.
  • --scheduler is the decay used after the warm-up phase ends.

No --warmup_steps is needed. The scheduler is stepped with the current training loss (src/optim/base.py).

Reproducing the experiments

Every script launches ./src/main.py from the repository root and logs to Weights & Biases (--wandb). The GPU is chosen with GPU=<id> in the newer scripts and with CUDA_VISIBLE_DEVICES inside the older ones.

Paper Scripts
LLaMA 124M, Muon / Lion / normSGD, bs 32 and 256 (Figure 2) scripts_124m/ (*_cos*.sh: fixed warm-up sweep, *_lip*.sh: adaptive)
LLaMA 210M (Figure 3) scripts_210m/
LLaMA 583M and 720M, Muon (Figure 4a, Appendix) scripts_583m_720m/
Swin-B on ImageNet-1K (Figure 4b) scripts_swin/
RAdam and Scion baselines (Section 6.1) scripts_baselines/
Ablations over $f^\star$, $\sigma_F^2$ and the schedule type (Figure 5) scripts_ablations/
Smoothness measurements, including the no-warm-up control (Figure 1, Section 3, Appendix) scripts_smooth/

The 583M and 720M scripts take their configuration from environment variables (WS, LIP, FSTAR, DIV, LR, SEED, GPU), documented at the top of each file. For example, the adaptive 720M run of the paper is LIP=1 FSTAR=3.0 DIV=10 bash scripts_583m_720m/dmuon_fineweb_720m_param.sh.

Citation

@inproceedings{riabinin2026warmup,
  title     = {Where Does Warm-Up Come From? Adaptive Scheduling for Norm-Constrained Optimizers},
  author    = {Riabinin, Artem and Veprikov, Andrey and Bolatov, Arman and Tak{\'a}{\v{c}}, Martin and Beznosikov, Aleksandr},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026}
}

License

MIT, inherited from llm-baselines. See LICENSE.

About

Official code for 'Where Does Warm-Up Come From? Adaptive Scheduling for Norm-Constrained Optimizers' (NeurIPS 2026)

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages