chore: standardize model feature flags and adopt cargo-nextest as test runner

.config/nextest.toml

@@ -0,0 +1,57 @@
+# Test runner config. Rationale in docs/conventions/coding-style.md.
+#
+# The contract: a bare `cargo nextest run --workspace` is the sweep — lib unit
+# tests only, safe on any single-GPU box. Hardware gates (integration tests
+# needing model weights / multi-GPU) are excluded by the default filter and
+# reported as skipped; run them per package with --ignore-default-filter.
+
+[profile.default]
+# Sweep = lib tests, minus the one lib test that needs RDMA hardware:
+# gdr_copy_flag_GPU requires a GDRCopy handle (gdrdrv), absent on plain GPU
+# workstations. It is excluded here — not env-probed inside the test — so the
+# selection stays visible; run it explicitly on RDMA hosts with
+# `cargo nextest run -p openinfer-comm-cuda-lib --ignore-default-filter`.
+default-filter = "kind(lib) - (package(openinfer-comm-cuda-lib) & test(gdr_copy_flag_GPU))"
+# Flaky = bug. Exact-match greedy gates are this repo's core asset; retries
+# would let nondeterminism rot unnoticed.
+retries = 0
+# A wedged NCCL collective or CUDA deadlock must go red, not hang the session.
+slow-timeout = { period = "60s", terminate-after = 5 }
+
+# Integration gates load real model weights — legitimately slow. Mark slow at
+# 5 min, kill at 1 h (a gate that quiet for an hour is hung, not loading).
+[[profile.default.overrides]]
+filter = "kind(test)"
+slow-timeout = { period = "300s", terminate-after = 12 }
+
+# nextest is process-per-test, so parallel GPU tests each create their own
+# CUDA context on the same device. Two tiers:
+#   gpu-gate: integration gates load full engines (GBs of VRAM, NCCL ports) —
+#             strictly one at a time.
+#   gpu-lib:  CUDA-touching lib tests hold small contexts — bounded, not
+#             serial, so pure-logic lib tests in the same packages don't queue
+#             behind a single slot.
+# CPU-only suites (kv-cache, sim, vllm-support, ...) keep full parallelism.
+[test-groups]
+gpu-gate = { max-threads = 1 }
+gpu-lib = { max-threads = 4 }
+
+[[profile.default.overrides]]
+filter = "kind(test)"
+test-group = "gpu-gate"
+
+[[profile.default.overrides]]
+filter = """
+kind(lib) & (
+  package(openinfer-core)
+  | package(openinfer-kernels)
+  | package(openinfer-sample)
+  | package(openinfer-qwen3)
+  | package(openinfer-qwen35-4b)
+  | package(openinfer-glm52)
+  | package(openinfer-kimi-k2)
+  | package(openinfer-deepseek-v2-lite)
+  | package(openinfer-deepseek-v4)
+)
+"""
+test-group = "gpu-lib"

.pre-commit-config.yaml

@@ -19,15 +19,16 @@ repos:
         args: [--release, --all-targets]
 
   # The hook above lints `default-members` (openinfer-server) with default
-  # features, so the feature-gated kimi-k2 crate is never compiled and never
-  # linted. This local hook closes that hole — but only when kimi-k2 source
-  # changes (the CUDA build is expensive), and `--no-deps` keeps it scoped to
-  # the crate itself rather than its shared workspace dependencies.
+  # features, whose dep graph does not include the kimi-k2 crate, so it is
+  # never compiled and never linted there. This local hook closes that hole —
+  # but only when kimi-k2 source changes (the CUDA build is expensive), and
+  # `--no-deps` keeps it scoped to the crate itself rather than its shared
+  # workspace dependencies.
   - repo: local
     hooks:
       - id: clippy-kimi-k2
         name: clippy (kimi-k2, -D warnings)
-        entry: cargo clippy -p openinfer-kimi-k2 --no-deps --release --features kimi-k2,kernel-report --all-targets -- -D warnings
+        entry: cargo clippy -p openinfer-kimi-k2 --no-deps --release --features kernel-report --all-targets -- -D warnings
         language: system
         types: [rust]
         files: ^openinfer-kimi-k2/

CLAUDE.md

@@ -47,17 +47,20 @@ cargo run --release --features glm52 -- --model-path models/GLM5.2
 
 ## Tests
 
+The runner is cargo-nextest (`cargo binstall cargo-nextest`, >= 0.9.138; config and rationale in `.config/nextest.toml` + `docs/conventions/coding-style.md`). Bare `cargo test` still works, but nextest owns the sweep contract: hardware gates are excluded by the config's default filter and reported as skipped, and hung GPU tests are killed instead of wedging the session.
+
 ```bash
-# Unit tests (~9s)
-cargo test --release --workspace --lib
+# Sweep (~10s): lib unit tests, green on any single-GPU box
+cargo nextest run --release --workspace
 
-# Accuracy and integration tests — require GPU + model weights
-cargo test --release -p openinfer-qwen3 --test hf_golden_gate
-OPENINFER_TEST_MODEL_PATH=models/Qwen3.5-4B cargo test --release -p openinfer-qwen35-4b --features qwen35-4b --test hf_golden_gate
-OPENINFER_TEST_MODEL_PATH=models/Qwen3.5-4B cargo test --release -p openinfer-qwen35-4b --features qwen35-4b --test e2e_scheduler
+# Accuracy and integration gates — require GPU + model weights, opt in per package
+OPENINFER_TEST_MODEL_PATH=models/Qwen3-4B \
+  cargo nextest run --release -p openinfer-qwen3 --ignore-default-filter -E 'binary(hf_golden_gate)'
+OPENINFER_TEST_MODEL_PATH=models/Qwen3.5-4B \
+  cargo nextest run --release -p openinfer-qwen35-4b --features qwen35-4b --ignore-default-filter -E 'binary(e2e_scheduler)'
 
 # Single test (filter by name)
-cargo test --release --workspace --lib prefix_cache -- --nocapture
+cargo nextest run --release --workspace -E 'test(prefix_cache)'
 ```
 
 Qwen accuracy gates compare logits against stored HF golden fixtures. Qwen3.5 exact-text JSON baselines are retired; keep `e2e_scheduler` for scheduler liveness and request-flow coverage.

docs/conventions/coding-style.md

@@ -4,6 +4,24 @@
 
 Don't test for the sake of testing. Prefer integration tests over unit tests — if the E2E test catches it, a unit test is ceremony. Unit tests earn their place for silent failures: GPU kernels, tricky data-structure invariants, edge-case-rich pure logic.
 
+## Test runner
+
+`cargo nextest run --workspace` (config in `.config/nextest.toml`) is the sweep: it runs lib unit tests, excludes hardware gates via the config's `default-filter`, and *reports* everything it excluded as skipped — selection is explicit, never silent. Hardware gates run per package with `--ignore-default-filter`:
+
+```bash
+OPENINFER_TEST_MODEL_PATH=/path/to/Qwen3-4B \
+  cargo nextest run --release -p openinfer-qwen3 --ignore-default-filter -E 'binary(hf_golden_gate)'
+```
+
+Why nextest over bare `cargo test`: process-per-test isolation (thread-local cuBLAS/CUDA-context poisoning can't leak between tests — see `lessons/exact-match-gate-thread-cublas.md`), hang-kill via `slow-timeout` (a wedged NCCL collective goes red instead of hanging the session), and GPU concurrency groups (gates strictly serial, CUDA-touching lib tests bounded). It also *compiles* every test target where `cargo test --lib` compiles none of the integration tests — that alone caught a gate that had silently rotted against a renamed API.
+
+Two rules encoded in the config that must not regress:
+
+- `retries = 0`. Flaky = bug. Exact-match greedy gates are this repo's core asset; a retry culture rots them.
+- No env-probing skips inside tests. If a test can't run on a machine class, exclude it in `default-filter` with a comment (visible in config + counted as skipped) — never auto-skip from inside the test, which reports a green that guards nothing.
+
+nextest does not run doctests. The workspace has 3 runnable ones today; `cargo test --release --doc --workspace` covers them when doc examples change.
+
 ## Logging
 
 Log through `openinfer-core::logging`. The text layout already prints each record's module target, so don't prefix messages with a module or model name — no `kimi-k2:`, no `Qwen3.5 `. Error messages in `anyhow!` / `bail!` keep their prefix; they surface to callers without a target.

docs/conventions/feature-flags.md

@@ -0,0 +1,64 @@
+# Cargo feature-flag convention
+
+**TL;DR**: One feature name per model line, spelled identically in `openinfer-server` and `openinfer-kernels`. A model crate itself carries **no** self-named feature — unless its kernels need a build-time Python toolchain, in which case the *only* allowed gate is a single whole-crate `#![cfg(feature = "...")]`. Capability features (`moe`, `pplx-ep`, `kernel-report`, …) are named after the capability, never after a model.
+
+## Why features exist here at all
+
+A feature answers exactly one question: **does this binary compile this model line's kernels?** Everything else — which model actually runs, EP topology, backend selection — is runtime configuration, not a feature. Two forces shape the rules:
+
+1. `openinfer-kernels/build.rs` is expensive (nvcc, and for some lines AOT codegen through Python). Features scope what it compiles.
+2. Two model lines cannot build without external toolchains (`qwen35-4b`: Python + Triton; `deepseek-v4`: Python + TileLang + CuTe DSL). A featureless `cargo build --workspace` must stay buildable on a machine that has CUDA but no Python — so those two, and only those two, stay gated at the crate level.
+
+## The rules
+
+### 1. Feature names
+
+The feature for a model line is the crate-name suffix: `openinfer-kimi-k2` → `kimi-k2`. The same string is used in `openinfer-server` and `openinfer-kernels`. Capability features describe the capability (`moe`, `pplx-ep`, `kernel-call-trace`, `kernel-report`, `deepseek-v4-cutedsl-diagnostic`), and may imply model features but are never a synonym for one.
+
+### 2. `openinfer-kernels` — one feature per model's kernels
+
+Each model line gets a same-named feature that scopes its `csrc/<model>/` sources (and AOT codegen, if any). Shared substrate gets a capability feature that model features imply (`glm52 = ["moe"]`, `kimi-k2 = ["moe"]`). Features that need more than nvcc must say so in a comment next to the declaration.
+
+### 3. Model crates — Tier A (pure CUDA): no self feature
+
+Applies to: **qwen3, glm52, kimi-k2, deepseek-v2-lite** — anything whose kernels need only nvcc.
+
+- No self-named feature. The crate always compiles its full self.
+- `openinfer-kernels = { workspace = true, features = ["<model>"] }`, **not** optional.
+- No `#[cfg(feature = "<model>")]` anywhere in the crate, no reject-only stub functions, no `required-features` on tests/bins (except for genuine capability features like `kernel-report`).
+- Cost accepted knowingly: workspace-wide builds (`cargo test --workspace --lib`) compile these kernels once, then cache. That trade was made in #550 (glm52) and extended to kimi-k2 / deepseek-v2-lite.
+- Corollary: without `required-features`, a bare `cargo test --workspace` (no `--lib`) also builds and *runs* Tier A GPU gates, which fail loudly on hosts missing the weights or GPU count — same as the qwen3/glm52 gates today. The routine sweep is `--lib`; gates run per package.
+
+### 4. Model crates — Tier B (build-time Python toolchain): one whole-crate gate
+
+Applies to: **qwen35-4b** (Triton), **deepseek-v4** (TileLang + CuTe DSL).
+
+- Keep the self-named feature, forwarding to the kernels feature: `deepseek-v4 = ["openinfer-kernels/deepseek-v4"]`.
+- The *only* in-crate gate is `#![cfg(feature = "<model>")]` at the top of `lib.rs`. With the feature off, the crate compiles to nothing. Scattered item-level `#[cfg]` gates and reject-only stubs are forbidden — they rot into the load-only scaffold #550 had to remove.
+- Tests/bins/benches declare `required-features = ["<model>"]`.
+
+### 5. `openinfer-server` — the selection point
+
+- Tier A: `<model> = ["dep:openinfer-<model>"]`.
+- Tier B: `<model> = ["dep:openinfer-<model>", "openinfer-<model>/<model>"]`.
+- `default = ["qwen3"]` keeps the stock build pure Rust + CUDA.
+- Server code touches a model crate only under `#[cfg(feature = "<model>")]`; the "rebuild with --features" error message lives in the server dispatch (`server_engine.rs`), not in model crates.
+
+## What this means in practice
+
+| Command | Features in play |
+| --- | --- |
+| `cargo run --release -- --model-path …` | server `qwen3` (default) |
+| `cargo run --release --features kimi-k2 -- …` | server feature; the kimi crate needs no flag of its own |
+| `cargo test --release -p openinfer-kimi-k2` | none needed — Tier A crates build whole |
+| `cargo test --release -p openinfer-qwen35-4b --features qwen35-4b --test hf_golden_gate` | Tier B: the self feature is required |
+| `cargo test --release --workspace --lib` | compiles Tier A kernels (incl. `moe` substrate → needs NCCL), skips Tier B (no Python required) |
+
+Feature unification note: cargo compiles `openinfer-kernels` once per distinct feature set in a build graph. Default-member builds (server + qwen3) and `--workspace` builds are two different sets; both stay cached side by side in `target/`, so switching between them does not thrash nvcc after the first build of each.
+
+## Adding a new model line
+
+1. Add the kernels feature (`<model> = []` or `= ["moe"]`) and gate its `csrc/<model>/` sources in `build.rs`.
+2. Decide the tier by one question: *does building the kernels need anything beyond nvcc?* No → Tier A. Yes → Tier B, and document the toolchain requirement next to the feature declaration.
+3. Wire the server feature per rule 5 and add the `#[cfg]` arms in `server_engine.rs` / `main.rs`.
+4. Bring-up scaffolding (load-only binaries, reject-only coordinators) is fine on a branch but must not merge behind a crate-internal feature — see #550 for the cleanup that pattern forces later.

docs/index.md

@@ -201,4 +201,5 @@ Organized by domain (model line / subsystem / playbook / lesson) instead of by l
 | Path | TL;DR |
 | --- | --- |
 | `conventions/bench-regression.md` | Benchmark regression tracking: one snapshot per model, git-tracked history, TPOT >2% / TTFT >3% thresholds. |
-| `conventions/coding-style.md` | Testing principle: prefer integration tests, don't test what E2E catches. |
+| `conventions/coding-style.md` | Testing principle (prefer integration tests) + nextest runner contract: sweep vs per-package gates, retries=0, no env-probing skips. |
+| `conventions/feature-flags.md` | Cargo feature standard: one name per model line; pure-CUDA model crates carry no self feature (Tier A), Python-toolchain crates keep a single whole-crate gate (Tier B); selection lives in server + kernels. |

docs/models/deepseek-v2-lite/decode-attribution-gate.md

@@ -39,14 +39,14 @@ OPENINFER_TEST_MODEL_PATH=models/DeepSeek-V2-Lite \
 OPENINFER_DSV2_LITE_E2E_CASE_SET=test_data/deepseek-v2-lite-ep2-cases.json \
 OPENINFER_DSV2_LITE_E2E_JSON_OUT=target/accuracy/dsv2-lite-ep2/host-staged.json \
   cargo test --release -p openinfer-deepseek-v2-lite \
-  --features deepseek-v2-lite --test e2e_ep2 -- --nocapture
+  --test e2e_ep2 -- --nocapture
 
 OPENINFER_TEST_MODEL_PATH=models/DeepSeek-V2-Lite \
 OPENINFER_DSV2_LITE_E2E_CASE_SET=test_data/deepseek-v2-lite-ep2-cases.json \
 OPENINFER_DSV2_LITE_EP_BACKEND=nccl \
 OPENINFER_DSV2_LITE_E2E_JSON_OUT=target/accuracy/dsv2-lite-ep2/nccl.json \
   cargo test --release -p openinfer-deepseek-v2-lite \
-  --features deepseek-v2-lite --test e2e_ep2 -- --nocapture
+  --test e2e_ep2 -- --nocapture
 
 python tools/accuracy/compare_dsv2_lite_ep2_outputs.py \
   --hf target/accuracy/dsv2-lite-ep2/hf.json \
@@ -60,15 +60,13 @@ Collect batch-1 attribution:
 
 ```bash
 cargo run --release -p openinfer-deepseek-v2-lite \
-  --features deepseek-v2-lite \
   --bin dsv2_lite_ep2_decode_attribution \
   -- --model-path models/DeepSeek-V2-Lite \
   --batch-size 1 \
   --out target/accuracy/dsv2-lite-ep2/host-staged-attribution.json
 
 OPENINFER_DSV2_LITE_EP_BACKEND=nccl \
   cargo run --release -p openinfer-deepseek-v2-lite \
-  --features deepseek-v2-lite \
   --bin dsv2_lite_ep2_decode_attribution \
   -- --model-path models/DeepSeek-V2-Lite \
   --batch-size 1 \
@@ -80,7 +78,6 @@ Run the full decode graph probe:
 ```bash
 OPENINFER_DSV2_LITE_EP_BACKEND=nccl \
   cargo run --release -p openinfer-deepseek-v2-lite \
-  --features deepseek-v2-lite \
   --bin dsv2_lite_ep2_decode_attribution \
   -- --model-path models/DeepSeek-V2-Lite \
   --batch-size 1 \

docs/models/deepseek-v2-lite/device-resident-nccl-combine.md

@@ -37,10 +37,10 @@ Validated 2026-06-08 on the provided 2x RTX 5090 host with DeepSeek-V2-Lite snap
 Commands run:
 
 ```bash
-cargo test --offline --release -p openinfer-deepseek-v2-lite --features deepseek-v2-lite --test e2e_ep2 --no-run
+cargo test --offline --release -p openinfer-deepseek-v2-lite --test e2e_ep2 --no-run
 
 cargo clippy --offline --release -p openinfer-deepseek-v2-lite \
-  --features deepseek-v2-lite --bins --tests -- \
+  --bins --tests -- \
   -D warnings \
   -A clippy::option_option \
   -A clippy::manual_midpoint \
@@ -54,12 +54,12 @@ python tools/accuracy/hf_dump_dsv2_lite_ep2_greedy.py \
 
 OPENINFER_TEST_MODEL_PATH=models/DeepSeek-V2-Lite \
 OPENINFER_DSV2_LITE_E2E_JSON_OUT=target/accuracy/dsv2-lite-ep2/host-staged.json \
-  cargo test --offline --release -p openinfer-deepseek-v2-lite --features deepseek-v2-lite --test e2e_ep2 -- --nocapture
+  cargo test --offline --release -p openinfer-deepseek-v2-lite --test e2e_ep2 -- --nocapture
 
 OPENINFER_TEST_MODEL_PATH=models/DeepSeek-V2-Lite \
 OPENINFER_DSV2_LITE_EP_BACKEND=nccl \
 OPENINFER_DSV2_LITE_E2E_JSON_OUT=target/accuracy/dsv2-lite-ep2/nccl-after-decouple-cleanup.json \
-  cargo test --offline --release -p openinfer-deepseek-v2-lite --features deepseek-v2-lite --test e2e_ep2 -- --nocapture
+  cargo test --offline --release -p openinfer-deepseek-v2-lite --test e2e_ep2 -- --nocapture
 
 python tools/accuracy/compare_dsv2_lite_ep2_outputs.py \
   --hf target/accuracy/dsv2-lite-ep2/hf.json \
@@ -70,7 +70,6 @@ python tools/accuracy/compare_dsv2_lite_ep2_outputs.py \
 
 OPENINFER_DSV2_LITE_EP_BACKEND=nccl \
   cargo run --offline --release -p openinfer-deepseek-v2-lite \
-    --features deepseek-v2-lite \
     --bin dsv2_lite_ep2_decode_attribution \
     -- --model-path models/DeepSeek-V2-Lite \
     --batch-size 1 \
@@ -92,7 +91,7 @@ Follow-up review gate on 2026-06-09 after fixing those lints:
 cargo fmt --all --check
 
 cargo clippy --release -p openinfer-deepseek-v2-lite \
-  --features deepseek-v2-lite --bins --tests -- -D warnings
+  --bins --tests -- -D warnings
 ```
 
 Both commands passed on the same remote source copy after syncing the follow-up `host_ops.rs` and `logging.rs` fixes. The clippy command ran without `clippy::manual_midpoint`, `clippy::needless_range_loop`, or `clippy::option_option` allows.

docs/models/deepseek-v2-lite/hf-accuracy-gate.md

@@ -59,13 +59,13 @@ python tools/accuracy/hf_dump_dsv2_lite_ep2_greedy.py \
 OPENINFER_TEST_MODEL_PATH=models/DeepSeek-V2-Lite \
 OPENINFER_DSV2_LITE_E2E_CASE_SET=test_data/deepseek-v2-lite-ep2-cases.json \
 OPENINFER_DSV2_LITE_E2E_JSON_OUT=target/accuracy/dsv2-lite-ep2/host-staged.json \
-  cargo test --release -p openinfer-deepseek-v2-lite --features deepseek-v2-lite --test e2e_ep2 -- --nocapture
+  cargo test --release -p openinfer-deepseek-v2-lite --test e2e_ep2 -- --nocapture
 
 OPENINFER_TEST_MODEL_PATH=models/DeepSeek-V2-Lite \
 OPENINFER_DSV2_LITE_E2E_CASE_SET=test_data/deepseek-v2-lite-ep2-cases.json \
 OPENINFER_DSV2_LITE_EP_BACKEND=nccl \
 OPENINFER_DSV2_LITE_E2E_JSON_OUT=target/accuracy/dsv2-lite-ep2/nccl.json \
-  cargo test --release -p openinfer-deepseek-v2-lite --features deepseek-v2-lite --test e2e_ep2 -- --nocapture
+  cargo test --release -p openinfer-deepseek-v2-lite --test e2e_ep2 -- --nocapture
 
 python tools/accuracy/compare_dsv2_lite_ep2_outputs.py \
   --hf target/accuracy/dsv2-lite-ep2/hf.json \

docs/models/deepseek-v2-lite/source-layout.md

@@ -66,7 +66,7 @@ Remote validation ran on Ubuntu 22.04 with 2x NVIDIA GeForce RTX 5090, driver
 
 Passed gates:
 
-- `cargo check --offline --release -p openinfer-deepseek-v2-lite --features deepseek-v2-lite --lib --tests`
+- `cargo check --offline --release -p openinfer-deepseek-v2-lite --lib --tests`
 - HF oracle dump with `tools/accuracy/hf_dump_dsv2_lite_ep2_greedy.py`
 - host-staged `tests/e2e_ep2.rs`
 - NCCL `tests/e2e_ep2.rs` using `LD_LIBRARY_PATH=/root/autodl-tmp/nccl-2.27.7/nvidia/nccl/lib`

docs/models/kimi-k2/accuracy-gate.md

@@ -63,7 +63,7 @@ GPU-argmax-vs-host-log-softmax disagreement on the same logits.
 
 # Run the gate (8 GPUs; vLLM must be stopped first — both need the full node):
 OPENINFER_TEST_MODEL_PATH=/data/models/Kimi-K2.6 \
-cargo test -p openinfer-kimi-k2 --features kimi-k2 --release \
+cargo test -p openinfer-kimi-k2 --release \
   --test vllm_golden_gate -- --nocapture
 ```