togethercomputer · qywu · Jun 16, 2026 · Apr 20, 2026 · Apr 20, 2026 · Apr 21, 2026
diff --git a/.codespellrc b/.codespellrc
@@ -1,3 +1,3 @@
 [codespell]
 skip = *.lock,*.json,submodules/*,.venv/*,.git,docs/node_modules/*
-ignore-words-list = dout,te,subtile,parm,mot,numer
+ignore-words-list = dout,te,subtile,parm,mot,numer,notin
diff --git a/.gitignore b/.gitignore
@@ -76,3 +76,13 @@ trace/
 docs/node_modules/
 docs/dist/
 docs/.astro/
+
+# Generated tokenized datasets and packing caches (per-benchmark)
+experiments/local_benchmark/dataset_cache/
+experiments/local_benchmark/datasets/
+
+# Kimi topology sweep scratch (tracked on branch `kimi-sweep-archive` instead)
+experiments/local_benchmark/sweeps/
+
+# Multi-adapter LoRA E2E artifacts can contain full per-rank logs and copied checkpoints.
+experiments/multi_adapter_lora/results/
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -4,15 +4,16 @@
 
 1. **Branch** off `main` with a descriptive name: `feature/my-feature`, `fix/bug-description`
 2. **Commit** early and often on your branch — commit messages don't matter much here
-3. **Open a PR** against `main` when ready for review
+3. **Open a PR** against `main` when ready for review. The PR title must follow [Conventional Commits](https://www.conventionalcommits.org/) (see below), since it becomes the squash-merge commit message and drives automated release versioning.
 4. **Squash merge** — all PRs are merged as a single squash commit; write a clean PR title and description since that becomes the commit message
 
 ## PR Guidelines
 
 - Keep PRs focused — one feature or fix per PR
 - Add tests for new behavior; existing tests must pass
 - Update relevant docs if behavior changes
-- PR title should be imperative and descriptive: `Add chunked cross-entropy loss` not `chunked ce`
+- PR title must follow Conventional Commits: `type: description` or `type(scope): description`. Allowed types: `feat`, `fix`, `perf`, `revert` (trigger releases); `chore`, `docs`, `test`, `refactor`, `ci`, `build` (no release). Append `!` for breaking changes (`feat!: ...` → major bump). An optional `[TICKET-123]` ticket prefix is allowed before the type. Examples: `feat: add chunked cross-entropy loss`, `fix(moe): correct expert routing`, `feat!: drop Python 3.9 support`.
+- A CI check enforces this on every PR and comments the detected version bump.
 
 ## Commit Message (Squash Merge)
 

diff --git a/README.md b/README.md
@@ -22,7 +22,7 @@ XoRL is a distributed training framework designed for large language models with
 
 | Repo &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; | Description |
 |---|---|
-| **[xorl](https://github.com/togethercomputer/xorl-internal)** | Distributed training framework — local SFT/pretraining and server-mode RL training |
+| **[xorl](https://github.com/togethercomputer/xorl)** | Distributed training framework — local SFT/pretraining and server-mode RL training |
 | **[xorl-client](https://github.com/togethercomputer/xorl-client)** | Lightweight Python SDK for driving the xorl training server (forward/backward, optimizer steps, checkpointing, sampling) |
 | **[xorl-sglang](https://github.com/togethercomputer/xorl-sglang)** | Fork of [SGLang](https://github.com/sgl-project/sglang) with weight-sync APIs, MoE routing export, and numerical alignment for online RL |
 
@@ -48,8 +48,8 @@ XoRL is a distributed training framework designed for large language models with
 ## 🚀 Installation
 
 ```bash
-git clone --recurse-submodules git@github.com:togethercomputer/xorl-internal.git
-cd xorl-internal
+git clone --recurse-submodules git@github.com:togethercomputer/xorl.git
+cd xorl
 ```
 
 > Already cloned without `--recurse-submodules`? Run `git submodule update --init --recursive`
@@ -102,7 +102,7 @@ pip install -e .
 
 > **Note:** The default `pyproject.toml` uses PyTorch 2.10.0. sglang requires PyTorch 2.9.1, so the two cannot coexist in the same environment unless you use `pyproject.sglang.toml`.
 
-See the [installation guide](https://togethercomputer.github.io/xorl-internal/getting-started/installation/) for full setup including optional dependencies (DeepEP, Flash Attention).
+See the [installation guide](https://togethercomputer.github.io/xorl/getting-started/installation/) for full setup including optional dependencies (DeepEP, Flash Attention).
 
 ## ⚡ Quick Start
 

diff --git a/docs/src/content/docs/config-reference/local.md b/docs/src/content/docs/config-reference/local.md
@@ -35,7 +35,7 @@ torchrun --nproc_per_node=8 -m xorl.cli.train config.yaml \
 | `ep_dispatch` | `alltoall` | Expert-parallel dispatch: `alltoall` or `deepep` (NVLink-optimized). |
 | `deepep_buffer_size_gb` | `2.0` | DeepEP NVLink buffer size per GPU in GB. Only active when `ep_dispatch: deepep`. |
 | `deepep_num_sms` | `20` | SMs assigned to DeepEP communication kernels. Must be even. Lower values leave more SMs for overlapped compute. |
-| `deepep_async_combine` | `false` | Overlap DeepEP combine with the next layer's compute (experimental). |
+| `deepep_async_combine` | `false` | Overlap DeepEP combine with the next layer's compute (experimental, unsafe). Forced to `false` in code unless `XORL_DEEPEP_UNSAFE_ASYNC_COMBINE=1` is exported; without that env var, deferring the comm-stream sync races the transformer block's read of the combined tensor on the default stream. |
 | `merge_qkv` | `true` | Keep Q/K/V projections fused as `qkv_proj`. Set `false` for tensor parallelism or per-projection LoRA. |
 | `basic_modules` | `[]` | Additional module names (beyond `_no_split_modules`) to shard as separate FSDP units. |
 | `foundation` | `{}` | Extra foundation model config (dict). |
@@ -167,7 +167,7 @@ Each entry in `datasets` (or `test_datasets`) is a dict:
 | `activation_gpu_limit` | `0.0` | GB of activations to keep on GPU when offloading. `0.0` = offload all. |
 | `enable_compile` | `false` | `torch.compile` for model forward pass. |
 | `init_device` | `cuda` | Device for weight initialization: `cpu` (rank 0 only), `cuda`, `meta` (required for FSDP2), `npu`. |
-| `load_weights_mode` | `broadcast` | `broadcast`: rank 0 reads weights, broadcasts to other ranks (reduces disk I/O). `all_ranks`: every rank reads from disk. |
+| `load_weights_mode` | `grouped` | `grouped`: one reader per node for dense/shared weights plus one reader per EP-FSDP group for expert weights, with rank-0 fallback when grouped fanout groups are unavailable. `all_ranks`: every rank reads from disk. `skip`: skip HuggingFace weight loading and materialize model weights from `load_checkpoint_path` (DCP). |
 | `enable_full_determinism` | `false` | Full determinism mode. Requires `allow_cuda_launch_blocking: true`. Degrades performance. |
 | `allow_cuda_launch_blocking` | `false` | Allow `CUDA_LAUNCH_BLOCKING=1`. Off by default to prevent accidental performance degradation. |
 | `empty_cache_steps` | `500` | Call `torch.cuda.empty_cache()` every N steps. |

diff --git a/docs/src/content/docs/config-reference/server.md b/docs/src/content/docs/config-reference/server.md
@@ -33,7 +33,7 @@ python -m xorl.server.launcher --mode auto --config config.yaml \
 | `ep_dispatch` | `alltoall` | Expert-parallel dispatch: `alltoall` or `deepep` (NVLink-optimized). |
 | `deepep_buffer_size_gb` | `2.0` | DeepEP NVLink buffer size per GPU in GB. Only active when `ep_dispatch: deepep`. |
 | `deepep_num_sms` | `20` | SMs assigned to DeepEP communication kernels. Must be even. |
-| `deepep_async_combine` | `false` | Overlap DeepEP combine with the next layer's compute (experimental). |
+| `deepep_async_combine` | `false` | Overlap DeepEP combine with the next layer's compute (experimental, unsafe). Forced to `false` in code unless `XORL_DEEPEP_UNSAFE_ASYNC_COMBINE=1` is exported; without that env var, deferring the comm-stream sync races the transformer block's read of the combined tensor on the default stream. |
 | `merge_qkv` | `true` | Keep Q/K/V projections fused. Set `false` for tensor parallelism. |
 | `basic_modules` | `[]` | Additional module names to shard as separate FSDP units. |
 | `foundation` | `{}` | Foundation model extra config (dict). |
@@ -86,7 +86,7 @@ These flags align the training model's numerics with the inference engine (SGLan
 | `enable_reentrant` | `false` | Use reentrant gradient checkpointing. |
 | `enable_forward_prefetch` | `false` | FSDP forward prefetch. |
 | `init_device` | `meta` | Model initialization device: `cpu`, `meta`, `cuda`. |
-| `load_weights_mode` | `auto` | Weight loading: `auto`, `safetensors`, `dcp`. |
+| `load_weights_mode` | `grouped` | Weight loading mode: `grouped` (default, with rank-0 fallback), `all_ranks`, or `skip`. |
 | `ce_mode` | `compiled` | Cross-entropy implementation: `compiled` (recommended, `torch.compile`) or `eager` (may OOM at 32K+ seq len). |
 
 ---
@@ -170,6 +170,7 @@ ZMQ communication between the launcher, workers, and API server.
 | `qlora_exclude_modules` | `null` | Modules to exclude from quantization (e.g., `[lm_head]`). |
 | `merge_lora_interval` | `0` | Merge LoRA into base weights every N steps. `0` = never. |
 | `reset_optimizer_on_merge` | `false` | ReLoRA optimizer reset after merge. |
+| `adapter_state_load_mode` | `all_ranks` | How to restore multi-adapter checkpoints: `all_ranks` loads on every rank; `rank0_broadcast` loads on rank 0 and broadcasts weights, metadata, and optimizer state. |
 
 ---
 

diff --git a/docs/src/content/docs/getting-started/installation.md b/docs/src/content/docs/getting-started/installation.md
@@ -13,8 +13,8 @@ title: "Installation"
 ## Clone the repo
 
 ```bash
-git clone --recurse-submodules https://github.com/togethercomputer/xorl-internal
-cd xorl-internal
+git clone --recurse-submodules https://github.com/togethercomputer/xorl
+cd xorl
 ```
 
 > Already cloned without `--recurse-submodules`? Run `git submodule update --init --recursive`

diff --git a/docs/src/content/docs/getting-started/quickstart.md b/docs/src/content/docs/getting-started/quickstart.md
@@ -109,7 +109,7 @@ future = requests.post(f"{base_url}/api/v1/optim_step", json={
 
 ### Example: SFT on No Robots
 
-[`examples/server/no_robot_sft/`](https://github.com/togethercomputer/xorl-internal/tree/main/examples/server/no_robot_sft) — Supervised fine-tuning on the [No Robots](https://huggingface.co/datasets/HuggingFaceH4/no_robots) dataset using `xorl_client`.
+[`examples/server/no_robot_sft/`](https://github.com/togethercomputer/xorl/tree/main/examples/server/no_robot_sft) — Supervised fine-tuning on the [No Robots](https://huggingface.co/datasets/HuggingFaceH4/no_robots) dataset using `xorl_client`.
 
 ```bash
 # 1. Start the training server
@@ -131,7 +131,7 @@ The script uses `xorl_client.TrainingClient` to drive a LoRA SFT loop with onlin
 
 ### Example: Password Memorization (end-to-end weight sync)
 
-[`examples/server/password_memorization/`](https://github.com/togethercomputer/xorl-internal/tree/main/examples/server/password_memorization) — End-to-end test for the training → weight sync → inference pipeline. Trains a model to memorize 3 secret codes via SFT, syncs weights to a running xorl-sglang instance, and queries inference to verify recall.
+[`examples/server/password_memorization/`](https://github.com/togethercomputer/xorl/tree/main/examples/server/password_memorization) — End-to-end test for the training → weight sync → inference pipeline. Trains a model to memorize 3 secret codes via SFT, syncs weights to a running xorl-sglang instance, and queries inference to verify recall.
 
 ```bash
 # 1. Start the training server
@@ -149,7 +149,7 @@ python examples/server/password_memorization/run_password_test.py \
     --model Qwen/Qwen3-8B --steps 16 --lr 1e-5
 ```
 
-Supports all training modes (full, LoRA, QLoRA nvfp4/block_fp8/nf4), LR schedules (constant, cosine, warmup+cosine), and FP8 weight sync re-quantization. See the [example README](https://github.com/togethercomputer/xorl-internal/tree/main/examples/server/password_memorization/README.md) for the full test matrix across Qwen3-8B, Qwen3-30B, and Qwen3-235B.
+Supports all training modes (full, LoRA, QLoRA nvfp4/block_fp8/nf4), LR schedules (constant, cosine, warmup+cosine), and FP8 weight sync re-quantization. See the [example README](https://github.com/togethercomputer/xorl/tree/main/examples/server/password_memorization/README.md) for the full test matrix across Qwen3-8B, Qwen3-30B, and Qwen3-235B.
 
 ## LoRA Fine-tuning
 

diff --git a/docs/src/content/docs/moe/deepep.mdx b/docs/src/content/docs/moe/deepep.mdx
@@ -88,7 +88,7 @@ model:
 | `ep_dispatch` | `alltoall` | Set to `deepep` to enable |
 | `deepep_buffer_size_gb` | `2.0` | Per-GPU NVLink buffer pool in GB. Larger = fewer chunked transfers. Rule of thumb: `2 × token_budget × hidden_dim × sizeof(bf16)` |
 | `deepep_num_sms` | `20` | SMs dedicated to communication kernels. Must be even. |
-| `deepep_async_combine` | `false` | Overlap combine with next layer's compute (experimental) |
+| `deepep_async_combine` | `false` | Overlap combine with next layer's compute (experimental, **currently disabled** — see [Async Combine](#async-combine-experimental)) |
 
 ---
 
@@ -223,17 +223,29 @@ If you see OOM during initialization, reduce `deepep_buffer_size_gb`. If profili
 
 ## Async Combine (Experimental)
 
-`deepep_async_combine: true` overlaps the combine communication (outputs flowing back from expert ranks) with the next layer's compute:
+:::caution[Currently disabled]
+The async combine path is **forced off in code**.  The combined
+tensor is consumed by the rest of the transformer block (residual add, norm,
+attention) on the default stream **before** the next `token_pre_dispatch()`
+runs, so deferring `event.current_stream_wait()` races those consumers.
+Setting `deepep_async_combine: true` alone is a no-op (logs a one-time warning).
+To actually run async, also export `XORL_DEEPEP_UNSAFE_ASYNC_COMBINE=1` — only
+do this if you have separately ensured the downstream consumer waits on the
+combine event.
+:::
+
+`deepep_async_combine: true` requests overlap of the combine communication
+(outputs flowing back from expert ranks) with the next layer's compute:
 
 ```
 Step N:    dispatch → compute → [combine starts]
 Step N+1:  [combine finishes] + next layer compute (overlapped)
 ```
 
-**Benefit:** Hides combine latency behind useful compute, especially when combine > dispatch (typical for large output projections).
+**Benefit (when working):** Hides combine latency behind useful compute, especially when combine > dispatch (typical for large output projections).
 
 **Limitations:**
-- Experimental — correctness verified on Qwen3 but not all architectures
+- Gated off behind `XORL_DEEPEP_UNSAFE_ASYNC_COMBINE=1`
 - Requires careful ordering of CUDA streams
 - Not compatible with pipeline parallelism (PP > 1)
 
@@ -247,7 +259,7 @@ On Qwen3-235B-A22B with EP=64 (8 nodes, 64 H100 NVLink GPUs):
 |---|---|---|
 | AllToAll (NCCL) | ~5s per forward_backward | Baseline |
 | DeepEP (num_sms=20) | ~1s per forward_backward | ~5× faster |
-| DeepEP + async_combine | ~0.6s per forward_backward | ~8× faster |
+| DeepEP + async_combine | ~0.6s per forward_backward | ~8× faster (currently gated off, see [Async Combine](#async-combine-experimental)) |
 
 Gains are most pronounced at high EP sizes (≥ 32) where AllToAll's O(EP) staging bottleneck dominates.
 
@@ -259,7 +271,7 @@ Gains are most pronounced at high EP sizes (≥ 32) where AllToAll's O(EP) stagi
 | EP ≤ 8, multi-node InfiniBand | AllToAll — IB already uses RDMA |
 | EP ≥ 16, NVLink cluster | DeepEP |
 | EP ≥ 32, NVLink cluster | DeepEP strongly recommended |
-| EP ≥ 64, NVLink cluster | DeepEP + async_combine |
+| EP ≥ 64, NVLink cluster | DeepEP (async_combine currently gated off) |
 
 ---
 

diff --git a/docs/src/content/docs/parallelism/data_parallelism.mdx b/docs/src/content/docs/parallelism/data_parallelism.mdx
@@ -205,7 +205,7 @@ train:
   enable_full_shard: true
   enable_mixed_precision: true
   init_device: meta               # required for FSDP2
-  load_weights_mode: broadcast    # rank0 loads, broadcasts to all ranks
+  load_weights_mode: grouped      # grouped fanout, with rank-0 fallback
 ```
 
 `init_device: meta` is **required** for FSDP2. Parameters are initially created on the meta device (zero-cost), then materialized by FSDP2 after `fully_shard` is applied.
@@ -267,7 +267,7 @@ train:
   enable_full_shard: true
   enable_mixed_precision: true
   init_device: meta
-  load_weights_mode: broadcast
+  load_weights_mode: grouped
 ```
 
 The constraint that must hold: `data_parallel_shard_size × data_parallel_replicate_size == data_parallel_size` (where `data_parallel_size = world_size / (TP × PP × CP)`). xorl validates this in `TrainingArguments.__post_init__`.
@@ -478,7 +478,7 @@ train:
   enable_gradient_checkpointing: true
   enable_full_shard: true
   init_device: meta
-  load_weights_mode: broadcast
+  load_weights_mode: grouped
 ```
 
 **Memory profile (Qwen3-8B, ~8B params):**
@@ -505,7 +505,7 @@ train:
   enable_gradient_checkpointing: true
   enable_full_shard: true
   init_device: meta
-  load_weights_mode: broadcast
+  load_weights_mode: grouped
 ```
 
 Here `world_size = dp_shard × ulysses = 2 × 4 = 8`. Each 2-GPU FSDP shard group uses 4-way Ulysses to handle long sequences that would not fit on 2 GPUs individually.
@@ -522,7 +522,7 @@ train:
   enable_gradient_checkpointing: true
   enable_full_shard: true
   init_device: meta
-  load_weights_mode: broadcast
+  load_weights_mode: grouped
 ```
 
 Cross-node IB traffic is limited to the all-reduce of *averaged gradients* (not full shard all-gathers), which is typically 4–8x less than pure FSDP2 would require across nodes.
@@ -578,7 +578,7 @@ train:
   enable_full_shard: true
   reshard_after_forward: false    # PP: keep params gathered between fwd micro-batches
   init_device: meta
-  load_weights_mode: broadcast
+  load_weights_mode: grouped
 ```
 
 `world_size = PP × dp_shard = 2 × 4 = 8`. Each PP stage has 4 GPUs running FSDP2 over non-expert params and EP=4 over expert params.
@@ -643,15 +643,16 @@ When EP is enabled, xorl replaces FSDP2's automatic prefetching with explicit pe
 |-------|-------------|-----------------|
 | `meta` | Parameters on meta device; materialized lazily by FSDP2 | FSDP2 only (required) |
 | `cuda` | Parameters initialized directly on GPU | DDP, or FSDP2 debugging |
-| `cpu` | Parameters on CPU (rank 0 only for broadcast) | DDP only; not supported with EP |
+| `cpu` | Parameters on CPU (rank 0 only for rank-0 fallback) | DDP only; not supported with EP |
 | `npu` | Ascend NPU device | DDP, FSDP2 |
 
 ### `load_weights_mode`
 
 | Value | Description | When to use |
 |-------|-------------|-------------|
-| `broadcast` (default) | Rank 0 reads checkpoint from disk, broadcasts shards to all ranks | Default; avoids N-way disk I/O bottleneck |
+| `grouped` (default) | Dense/shared tensors are read once per node and expert tensors once per EP-FSDP group, then replicated/scattered within those groups | Default; best for large MoE/EP loads and falls back to rank-0 loading when grouped fanout groups are unavailable |
 | `all_ranks` | Every rank reads the checkpoint independently | Fast parallel storage (e.g., Lustre, object storage), or when EP requires each rank to load its own expert shard |
+| `skip` | Skip HuggingFace checkpoint loading and load model weights later from `load_checkpoint_path` | Model-only DCP checkpoints created by `scripts/convert_checkpoint.py --save-format dcp` |
 
 ---
 
@@ -665,9 +666,9 @@ Meta-device initialization creates parameter tensors with zero CPU/GPU memory co
 init_device: meta
 ```
 
-### Use `load_weights_mode: broadcast` by default
+### Use `load_weights_mode: grouped` by default
 
-`broadcast` mode has rank 0 load the checkpoint from disk and distribute shards. This is the safest option when the storage system cannot handle parallel reads from all ranks simultaneously. Switch to `all_ranks` only on parallel filesystems with guaranteed per-rank I/O performance, or when loading EP shards that differ per rank.
+`grouped` mode uses grouped fanout for large MoE checkpoints so dense/shared weights are read once per node and expert weights are read once per EP-FSDP group. When grouped fanout groups are unavailable, it falls back to rank-0 loading. Switch to `all_ranks` only on parallel filesystems with guaranteed per-rank I/O performance.
 
 ### Set `reshard_after_forward: false` for pipeline parallelism