Official code for the NeurIPS 2026 paper by Artem Riabinin, Andrey Veprikov, Arman Bolatov, Martin Takáč and Aleksandr Beznosikov (arXiv:2602.05813).
For norm-constrained optimizers, whose step is a linear minimization oracle (LMO) over a norm ball (Muon, Lion, normalized SGD, Scion), we show that the learning rate selected by the convergence analysis first grows and then decays as a function of the suboptimality gap
The scheduler in this repository calibrates
The code is built on llm-baselines (EPFL MLO), following the setup of Semenov et al. (2025). The upstream README is kept as README_llm_baselines.md.
python3.11 -m venv optim_venv
source optim_venv/bin/activate
pip install -r requirements.txtFineWeb is tokenized with the GPT-2 tokenizer into train.bin and val.bin under <datasets_dir>/fineweb-100BT/ (src/data/fineweb.py). Pass --datasets_dir if the data lives outside ./src/data/datasets/. Swin-B experiments expect ImageNet-1K in the layout read by src/data/imagenet.py.
| File | Content |
|---|---|
src/optim/lipschitz_scheduler.py |
The adaptive scheduler (Algorithm 1 in the paper) |
src/optim/lipschitz_computeK.py |
Calibration of |
src/optim/lipschitz_analyzer.py |
Measurement of the empirical smoothness ratio |
src/optim/muon.py, lion.py, normalized.py, scion.py
|
The LMO optimizers |
src/models/llama.py, src/models/swin.py
|
LLaMA and Swin-B models |
The adaptive warm-up is enabled with --use_lip_warmup. The remaining flags are the usual ones:
python ./src/main.py --model llama --dataset fineweb --optimizer d-muon \
--lr 1e-3 --div_factor 100 --scheduler cos \
--use_lip_warmup \
--lipschitz_loss_star 3.2 \
--lipschitz_sigma_F 0.001 \
--lipschitz_rho 2.0 \
--lipschitz_mode linear_and_cos \
...-
--lris the peak learning rate and--lr / --div_factorthe initial one. -
--lipschitz_loss_staris the estimate of$f^\star$ . We use a value close to the loss the model can reach at its token budget. The ablations in the paper show that the result is not sensitive to it within a reasonable range. -
--lipschitz_sigma_Fis the width of the calibration window. All experiments in the paper use1e-3. -
--lipschitz_mode linear_and_cosis the calibration objective used in the paper. -
--scheduleris the decay used after the warm-up phase ends.
No --warmup_steps is needed. The scheduler is stepped with the current training loss (src/optim/base.py).
Every script launches ./src/main.py from the repository root and logs to Weights & Biases (--wandb). The GPU is chosen with GPU=<id> in the newer scripts and with CUDA_VISIBLE_DEVICES inside the older ones.
| Paper | Scripts |
|---|---|
| LLaMA 124M, Muon / Lion / normSGD, bs 32 and 256 (Figure 2) |
scripts_124m/ (*_cos*.sh: fixed warm-up sweep, *_lip*.sh: adaptive) |
| LLaMA 210M (Figure 3) | scripts_210m/ |
| LLaMA 583M and 720M, Muon (Figure 4a, Appendix) | scripts_583m_720m/ |
| Swin-B on ImageNet-1K (Figure 4b) | scripts_swin/ |
| RAdam and Scion baselines (Section 6.1) | scripts_baselines/ |
| Ablations over |
scripts_ablations/ |
| Smoothness measurements, including the no-warm-up control (Figure 1, Section 3, Appendix) | scripts_smooth/ |
The 583M and 720M scripts take their configuration from environment variables (WS, LIP, FSTAR, DIV, LR, SEED, GPU), documented at the top of each file. For example, the adaptive 720M run of the paper is LIP=1 FSTAR=3.0 DIV=10 bash scripts_583m_720m/dmuon_fineweb_720m_param.sh.
@inproceedings{riabinin2026warmup,
title = {Where Does Warm-Up Come From? Adaptive Scheduling for Norm-Constrained Optimizers},
author = {Riabinin, Artem and Veprikov, Andrey and Bolatov, Arman and Tak{\'a}{\v{c}}, Martin and Beznosikov, Aleksandr},
booktitle = {Advances in Neural Information Processing Systems},
year = {2026}
}MIT, inherited from llm-baselines. See LICENSE.