Skip to content

docs: prefix absolute internal links with /xorl/ base path #10

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kiddyboots216 merged 1 commit into from
Apr 25, 2026
Merged

docs: prefix absolute internal links with /xorl/ base path #10

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/src/content/docs/adapters/lora.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: "LoRA"
---

For MoE-specific LoRA (expert weight layout, EP sharding), see [MoE LoRA](/moe/lora/).
For MoE-specific LoRA (expert weight layout, EP sharding), see [MoE LoRA](/xorl/moe/lora/).

LoRA (Low-Rank Adaptation) freezes the base model weights and adds trainable low-rank matrices to selected linear layers. xorl implements FSDP2-compatible LoRA for both local and server training.

Expand Down Expand Up @@ -156,7 +156,7 @@ model = PeftModel.from_pretrained(base_model, "outputs/my_run/weights/.../step_4

For long training runs, periodically merge LoRA into the base weights and restart with fresh LoRA parameters. This avoids rank saturation and allows the effective rank to grow over training.

For QLoRA, see [QLoRA — Periodic Merge](/adapters/qlora/#periodic-merge-relora).
For QLoRA, see [QLoRA — Periodic Merge](/xorl/adapters/qlora/#periodic-merge-relora).

## LoRA in Server Training

Expand Down
2 changes: 1 addition & 1 deletion docs/src/content/docs/adapters/qlora.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ title: "QLoRA"
---


For QLoRA with MoE models, see [MoE LoRA](/moe/lora/).
For QLoRA with MoE models, see [MoE LoRA](/xorl/moe/lora/).

QLoRA stores base model weights in a quantized format and adds trainable LoRA adapters on top. This reduces the GPU memory footprint of the base weights by 2–8× while keeping LoRA parameters in full precision.

Expand Down
2 changes: 1 addition & 1 deletion docs/src/content/docs/config-reference/local.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -159,7 +159,7 @@ Each entry in `datasets` (or `test_datasets`) is a dict:
|---|---|---|
| `enable_mixed_precision` | `true` | BF16 mixed-precision training. |
| `enable_gradient_checkpointing` | `true` | Enable activation recomputation to reduce memory. |
| `gradient_checkpointing_method` | `recompute_full_layer` | What to recompute in backward. Valid values: `recompute_full_layer` (recompute entire decoder layer, most memory-efficient), `recompute_before_dispatch` (recompute attn+router, keep dispatch+expert+combine, +25-34% throughput), `no_recompute` (no recomputation, max throughput, highest memory). See [gradient checkpointing guide](/training/local_training#gradient-checkpointing). |
| `gradient_checkpointing_method` | `recompute_full_layer` | What to recompute in backward. Valid values: `recompute_full_layer` (recompute entire decoder layer, most memory-efficient), `recompute_before_dispatch` (recompute attn+router, keep dispatch+expert+combine, +25-34% throughput), `no_recompute` (no recomputation, max throughput, highest memory). See [gradient checkpointing guide](/xorl/training/local_training#gradient-checkpointing). |
| `enable_reentrant` | `false` | Use reentrant gradient checkpointing. Default (non-reentrant) is generally preferred. |
| `enable_full_shard` | `true` | FSDP2 full parameter sharding (ZeRO-3). Set `false` for ZeRO-2. |
| `enable_forward_prefetch` | `true` | Prefetch next FSDP unit's parameters during forward pass. |
Expand Down
2 changes: 1 addition & 1 deletion docs/src/content/docs/development.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ pytest tests/
pytest tests/server/api_server/test_checkpoint_paths.py -v
```

Tests must pass locally before requesting review. See the [Testing](/testing/overview) section for full details.
Tests must pass locally before requesting review. See the [Testing](/xorl/testing/overview) section for full details.

## Branch Protection

Expand Down
2 changes: 1 addition & 1 deletion docs/src/content/docs/getting-started/installation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -155,4 +155,4 @@ sudo cat /proc/driver/nvidia/params | grep -E "EnableStreamMemOPs|RegistryDwords

## Next Steps

Head to the [Quick Start](/getting-started/quickstart/) to run your first training job.
Head to the [Quick Start](/xorl/getting-started/quickstart/) to run your first training job.
2 changes: 1 addition & 1 deletion docs/src/content/docs/getting-started/quickstart.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ title: "Quick Start"
---


Before running, make sure xorl is installed — see the [Installation guide](/getting-started/installation/).
Before running, make sure xorl is installed — see the [Installation guide](/xorl/getting-started/installation/).

## Choosing a Training Mode

Expand Down
2 changes: 1 addition & 1 deletion docs/src/content/docs/loss-functions/gradient-accumulation.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -178,7 +178,7 @@ for module in model.modules():

With `divide_factor=1.0`, the reduce-scatter **sums** gradients across ranks rather than averaging them. This is correct because `GradientAccumulateLoss` already scales each rank's backward contribution by `local_valid_tokens / global_valid_tokens` — so summing the contributions across ranks gives the correct token-average.

The only exception: EP expert modules retain `set_gradient_divide_factor(ep_size)` and are marked `_is_ep_fsdp = True` so the loop above skips them. See [Expert Parallelism](/parallelism/expert_parallelism/#6-gradient-handling) for details.
The only exception: EP expert modules retain `set_gradient_divide_factor(ep_size)` and are marked `_is_ep_fsdp = True` so the loop above skips them. See [Expert Parallelism](/xorl/parallelism/expert_parallelism/#6-gradient-handling) for details.

---

Expand Down
2 changes: 1 addition & 1 deletion docs/src/content/docs/models.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,4 +52,4 @@ The following model + training-mode combinations have pre-built example configs

MoE checkpoints from HuggingFace store experts as a `ModuleList` (one module per expert). xorl uses fused grouped-kernel (GKN) tensors for efficient expert dispatch. This conversion happens **automatically during model loading** — no separate preprocessing step is needed. Simply point `model_path` at the standard HuggingFace checkpoint and xorl will fuse the expert weights on the fly.

See the [MoE section](/moe/overview) for details on MoE-specific config options including `expert_parallel_size`, `ep_dispatch`, and `moe_implementation`.
See the [MoE section](/xorl/moe/overview) for details on MoE-specific config options including `expert_parallel_size`, `ep_dispatch`, and `moe_implementation`.
4 changes: 2 additions & 2 deletions docs/src/content/docs/moe/deepep.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ DeepEP is an NVLink-optimized expert parallelism dispatch backend. It replaces N
## Requirements

- NVLink-connected GPU cluster (H100 NVLink or DGX H100)
- `deep_ep` wheel installed (see [Installation](/getting-started/installation/))
- `deep_ep` wheel installed (see [Installation](/xorl/getting-started/installation/))
- `ep_dispatch: deepep` in model config
- For internode (multi-node) EP: `nvidia_peermem` kernel module loaded and IBGDA enabled on all nodes (see below)

Expand Down Expand Up @@ -294,7 +294,7 @@ XORL_DEBUG_EP=1 torchrun ... -m xorl.cli.train config.yaml
```

**Shape mismatch errors during backward:**
With the default `gradient_checkpointing_method: recompute_full_layer`, R3 routing replay is used automatically to ensure correct expert assignments when alltoall is recomputed. If you see shape mismatches, verify routing replay is enabled. Using `recompute_before_dispatch` or `no_recompute` avoids alltoall recomputation entirely and does not require replay. See [MoE Routing Replay](/moe/overview/#routing-replay-r3).
With the default `gradient_checkpointing_method: recompute_full_layer`, R3 routing replay is used automatically to ensure correct expert assignments when alltoall is recomputed. If you see shape mismatches, verify routing replay is enabled. Using `recompute_before_dispatch` or `no_recompute` avoids alltoall recomputation entirely and does not require replay. See [MoE Routing Replay](/xorl/moe/overview/#routing-replay-r3).

**No fallback to AllToAll:**
If DeepEP fails to initialize, xorl does not fall back automatically. Set `ep_dispatch: alltoall` explicitly.
Expand Down
16 changes: 8 additions & 8 deletions docs/src/content/docs/moe/expert-parallelism.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: "Expert Parallelism"
---

Expert Parallelism (EP) shards expert weights across `expert_parallel_size` GPUs. Each GPU holds `num_experts / ep_size` experts — a contiguous slice of the G dimension in [GKN layout](/moe/overview/#weight-layout-gkn-format). Non-expert parameters (attention, norms, gate linear) are replicated across the EP group.
Expert Parallelism (EP) shards expert weights across `expert_parallel_size` GPUs. Each GPU holds `num_experts / ep_size` experts — a contiguous slice of the G dimension in [GKN layout](/xorl/moe/overview/#weight-layout-gkn-format). Non-expert parameters (attention, norms, gate linear) are replicated across the EP group.

```yaml
train:
Expand All @@ -11,7 +11,7 @@ model:
ep_dispatch: alltoall # or deepep
```

For the full technical reference — dispatch internals, device mesh construction, gradient handling, R3 routing replay, EP composition with FSDP2/PP/CP, and constraint table — see **[Expert Parallelism (Parallelism section)](/parallelism/expert_parallelism/)**.
For the full technical reference — dispatch internals, device mesh construction, gradient handling, R3 routing replay, EP composition with FSDP2/PP/CP, and constraint table — see **[Expert Parallelism (Parallelism section)](/xorl/parallelism/expert_parallelism/)**.

---

Expand Down Expand Up @@ -74,7 +74,7 @@ For the full technical reference — dispatch internals, device mesh constructio
| Backend | Set via | Best for |
|---|---|---|
| `alltoall` | `ep_dispatch: alltoall` | Any GPU topology, no extra install |
| `deepep` | `ep_dispatch: deepep` | NVLink clusters, EP ≥ 16 — see [DeepEP](/moe/deepep/) |
| `deepep` | `ep_dispatch: deepep` | NVLink clusters, EP ≥ 16 — see [DeepEP](/xorl/moe/deepep/) |

---

Expand Down Expand Up @@ -150,11 +150,11 @@ train:

| Page | What it covers |
|---|---|
| [Expert Parallelism (full reference)](/parallelism/expert_parallelism/) | AllToAll internals, DeepEP buffer lifecycle, gradient divide factor, R3 mechanics, EP+PP/CP/FSDP composition, constraints, parameter reference |
| [MoE: Expert Kernels](/moe/kernels/) | Triton/Quack/native/eager backend details, torch.compile, moe_act checkpointing |
| [MoE: Router](/moe/router/) | Routing algorithm, R3 routing replay in detail |
| [DeepEP](/moe/deepep/) | NVLink-optimized dispatch, SM tuning, buffer sizing |
| [MoE: LoRA & QLoRA](/moe/lora/) | LoRA sharding on expert weights, hybrid shared LoRA |
| [Expert Parallelism (full reference)](/xorl/parallelism/expert_parallelism/) | AllToAll internals, DeepEP buffer lifecycle, gradient divide factor, R3 mechanics, EP+PP/CP/FSDP composition, constraints, parameter reference |
| [MoE: Expert Kernels](/xorl/moe/kernels/) | Triton/Quack/native/eager backend details, torch.compile, moe_act checkpointing |
| [MoE: Router](/xorl/moe/router/) | Routing algorithm, R3 routing replay in detail |
| [DeepEP](/xorl/moe/deepep/) | NVLink-optimized dispatch, SM tuning, buffer sizing |
| [MoE: LoRA & QLoRA](/xorl/moe/lora/) | LoRA sharding on expert weights, hybrid shared LoRA |

## Source

Expand Down
10 changes: 5 additions & 5 deletions docs/src/content/docs/moe/overview.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,11 +111,11 @@ HF MoE checkpoints store experts as `nn.ModuleList` of `nn.Linear` layers. xorl

| Page | What it covers |
|---|---|
| [Router](/moe/router/) | TopKRouter algorithm, norm_topk_prob, router_fp32, freeze_router, Routing Replay (R3) |
| [Expert Kernels](/moe/kernels/) | Backend comparison table, Triton GEMM details, torch.compile guidance, moe_act checkpointing |
| [Expert Parallelism](/moe/expert-parallelism/) | EP mesh, AllToAll dispatch, EP+FSDP2, EP+Ring Attention, example configs |
| [LoRA & QLoRA](/moe/lora/) | MoEExpertsLoRA, adapter shapes, hybrid shared LoRA, QLoRA on experts |
| [DeepEP](/moe/deepep/) | NVLink-optimized dispatch with DeepEP |
| [Router](/xorl/moe/router/) | TopKRouter algorithm, norm_topk_prob, router_fp32, freeze_router, Routing Replay (R3) |
| [Expert Kernels](/xorl/moe/kernels/) | Backend comparison table, Triton GEMM details, torch.compile guidance, moe_act checkpointing |
| [Expert Parallelism](/xorl/moe/expert-parallelism/) | EP mesh, AllToAll dispatch, EP+FSDP2, EP+Ring Attention, example configs |
| [LoRA & QLoRA](/xorl/moe/lora/) | MoEExpertsLoRA, adapter shapes, hybrid shared LoRA, QLoRA on experts |
| [DeepEP](/xorl/moe/deepep/) | NVLink-optimized dispatch with DeepEP |

---

Expand Down
2 changes: 1 addition & 1 deletion docs/src/content/docs/parallelism/expert_parallelism.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ AllToAll collective (the default `alltoall` backend) or DeepEP's NVLink-optimize
dispatch/combine kernels.

:::note
This page is the technical reference for EP internals: dispatch backends, device mesh construction, gradient handling, R3, and multi-parallelism composition. For MoE-specific topics — GKN weight layout, compute backends (Triton/Quack/native), LoRA on experts, and quick-start configs — see the [MoE Expert Parallelism](/moe/expert-parallelism/) page.
This page is the technical reference for EP internals: dispatch backends, device mesh construction, gradient handling, R3, and multi-parallelism composition. For MoE-specific topics — GKN weight layout, compute backends (Triton/Quack/native), LoRA on experts, and quick-start configs — see the [MoE Expert Parallelism](/xorl/moe/expert-parallelism/) page.
:::

<svg viewBox="0 0 700 190" style="max-width:700px;width:100%;margin:1.5rem 0" font-family="sans-serif">
Expand Down
10 changes: 5 additions & 5 deletions docs/src/content/docs/parallelism/overview.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,11 @@ xorl supports five orthogonal parallelism dimensions that can be freely composed

| Dimension | Doc | Config field | When to use |
|---|---|---|---|
| Data (FSDP2) | [Data Parallelism](/parallelism/data_parallelism/) | `data_parallel_mode`, `data_parallel_shard_size` | Always; primary memory reduction tool |
| Tensor | [Tensor Parallelism](/parallelism/tensor_parallelism/) | `tensor_parallel_size` | When FSDP2 alone can't fit the model |
| Pipeline | [Pipeline Parallelism](/parallelism/pipeline_parallelism/) | `pipeline_parallel_size` | Very large models across nodes |
| Expert | [Expert Parallelism](/parallelism/expert_parallelism/) | `expert_parallel_size` | MoE models |
| Sequence | [Context Parallelism](/parallelism/context_parallelism/) | `ulysses_parallel_size`, `ringattn_parallel_size` | Long sequences (>32K) |
| Data (FSDP2) | [Data Parallelism](/xorl/parallelism/data_parallelism/) | `data_parallel_mode`, `data_parallel_shard_size` | Always; primary memory reduction tool |
| Tensor | [Tensor Parallelism](/xorl/parallelism/tensor_parallelism/) | `tensor_parallel_size` | When FSDP2 alone can't fit the model |
| Pipeline | [Pipeline Parallelism](/xorl/parallelism/pipeline_parallelism/) | `pipeline_parallel_size` | Very large models across nodes |
| Expert | [Expert Parallelism](/xorl/parallelism/expert_parallelism/) | `expert_parallel_size` | MoE models |
| Sequence | [Context Parallelism](/xorl/parallelism/context_parallelism/) | `ulysses_parallel_size`, `ringattn_parallel_size` | Long sequences (>32K) |

## When to Use Each Dimension

Expand Down
2 changes: 1 addition & 1 deletion docs/src/content/docs/server-training/client-sdk/loss-functions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,4 +146,4 @@ fwd = client.forward_backward([datum], loss_fn="policy_loss")

The current `xorl-client` SDK only exposes `routed_experts`. The server can also consume `routed_expert_logits`, but that field is not yet wired through `Datum` / `TrainingClient`.

See the [Router page](/moe/router/#routing-replay-r3) for details on how R3 works, and the [xorl-sglang page](/server-training/sglang/) for how routing data is exported from inference.
See the [Router page](/xorl/moe/router/#routing-replay-r3) for details on how R3 works, and the [xorl-sglang page](/xorl/server-training/sglang/) for how routing data is exported from inference.
2 changes: 1 addition & 1 deletion docs/src/content/docs/server-training/client-sdk/overview.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ Or from source:
pip install git+https://github.com/togethercomputer/xorl-client.git
```

xorl-client is also installed automatically with xorl — see the [installation guide](/getting-started/installation/).
xorl-client is also installed automatically with xorl — see the [installation guide](/xorl/getting-started/installation/).

---

Expand Down
24 changes: 12 additions & 12 deletions docs/src/content/docs/server-training/overview.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,9 @@ Online RL requires a **training server**, an **inference server**, and an **orch

| Component | Provided by | Description |
|---|---|---|
| Training server | **[xorl](/server-training/training-server/launching/)** | Forward/backward, optimizer, checkpointing, parallelism |
| Inference server | **[xorl-sglang](/server-training/sglang/)** | Rollout generation, per-token logprobs, weight sync |
| Client SDK | **[xorl-client](/server-training/client-sdk/overview/)** | Python SDK: `TrainingClient`, `SamplingClient`, `RestClient` |
| Training server | **[xorl](/xorl/server-training/training-server/launching/)** | Forward/backward, optimizer, checkpointing, parallelism |
| Inference server | **[xorl-sglang](/xorl/server-training/sglang/)** | Rollout generation, per-token logprobs, weight sync |
| Client SDK | **[xorl-client](/xorl/server-training/client-sdk/overview/)** | Python SDK: `TrainingClient`, `SamplingClient`, `RestClient` |
| Reward / environment | **You** | Reward model, code sandbox, math verifier, rule-based scorer |

---
Expand Down Expand Up @@ -81,7 +81,7 @@ for step in range(num_steps):
).result()
```

For multi-node setup, server configuration, and launcher CLI options, see [Launching & Configuration](/server-training/training-server/launching/).
For multi-node setup, server configuration, and launcher CLI options, see [Launching & Configuration](/xorl/server-training/training-server/launching/).

### Available configs

Expand All @@ -106,7 +106,7 @@ For multi-node setup, server configuration, and launcher CLI options, see [Launc

## xorl-client

The [xorl-client](/server-training/client-sdk/overview/) Python SDK drives the training server — see the [Client SDK page](/server-training/client-sdk/overview/) for installation, client classes, loss functions, and training loop examples.
The [xorl-client](/xorl/server-training/client-sdk/overview/) Python SDK drives the training server — see the [Client SDK page](/xorl/server-training/client-sdk/overview/) for installation, client classes, loss functions, and training loop examples.

---

Expand Down Expand Up @@ -165,10 +165,10 @@ XoRL's server training mode is designed for online RL with LLMs. The following p

| Topic | Page |
|---|---|
| Server architecture, multi-node, launcher CLI | [Launching & Configuration](/server-training/training-server/launching/) |
| REST API endpoints | [API Reference](/server-training/training-server/api-reference/) |
| xorl-sglang: weight sync, numerical alignment | [Inference: xorl-sglang](/server-training/sglang/) |
| Client SDK, loss functions, training patterns | [Client SDK (xorl-client)](/server-training/client-sdk/overview/) |
| NCCL weight sync protocol | [Weight Sync](/server-training/weight-sync/overview/) |
| SFT fine-tuning example | [SFT on No Robots](/server-training/examples/sft-no-robots/) |
| End-to-end weight sync test | [Password Memorization](/server-training/examples/password-memorization/) |
| Server architecture, multi-node, launcher CLI | [Launching & Configuration](/xorl/server-training/training-server/launching/) |
| REST API endpoints | [API Reference](/xorl/server-training/training-server/api-reference/) |
| xorl-sglang: weight sync, numerical alignment | [Inference: xorl-sglang](/xorl/server-training/sglang/) |
| Client SDK, loss functions, training patterns | [Client SDK (xorl-client)](/xorl/server-training/client-sdk/overview/) |
| NCCL weight sync protocol | [Weight Sync](/xorl/server-training/weight-sync/overview/) |
| SFT fine-tuning example | [SFT on No Robots](/xorl/server-training/examples/sft-no-robots/) |
| End-to-end weight sync test | [Password Memorization](/xorl/server-training/examples/password-memorization/) |
4 changes: 2 additions & 2 deletions docs/src/content/docs/server-training/sglang.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ Key implementation details:
- **EP scatter**: For MoE models with expert parallelism, each EP training rank sends its local experts directly into the corresponding slice of `param.data` via P2P ops — no intermediate buffers.
- **Health check bypass**: During weight updates, the `/health` endpoint returns 200 immediately instead of running a test generation, avoiding timeouts during NCCL operations.

The training server's `nccl_broadcast` backend drives these calls — see [Backend: nccl_broadcast](/server-training/weight-sync/nccl-broadcast/) for the full protocol.
The training server's `nccl_broadcast` backend drives these calls — see [Backend: nccl_broadcast](/xorl/server-training/weight-sync/nccl-broadcast/) for the full protocol.

### 2. MoE Routing Data Export (R3)

Expand Down Expand Up @@ -139,7 +139,7 @@ cp pyproject.sglang.toml pyproject.toml
uv sync # or: pip install -e .
```

See the [installation guide](/getting-started/installation/#install-submodules) for full details.
See the [installation guide](/xorl/getting-started/installation/#install-submodules) for full details.

## Launching xorl-sglang

Expand Down
2 changes: 1 addition & 1 deletion docs/src/content/docs/server-training/training-server/api-reference.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Poll `POST /api/v1/retrieve_future` with that ID to get the actual result.
The `xorl-client` SDK handles polling automatically.
:::

All endpoints are served at `http://<host>:<port>/`. Training operations use a two-phase async protocol — see [Launching & Configuration](/server-training/training-server/launching/#api-server) for details.
All endpoints are served at `http://<host>:<port>/`. Training operations use a two-phase async protocol — see [Launching & Configuration](/xorl/server-training/training-server/launching/#api-server) for details.

## Training Operations

Expand Down
Loading
Loading