Repository navigation
Commit 64aa6f1
committed
feat(vector): fuse the BBQ rerank distance into SimdRuntime's kernel table
Reranking against 1-bit BBQ candidates materialized a `Vec<f32>` per candidate
and per query; at an oversample x ef candidate count that dominated the pass and
defeated SIMD.
The kernel now reads the centred query straight from the prepared payload bytes:
zero allocation per candidate and per query, one pass. It is a `SimdRuntime`
field (`l2_bbq`), selected in `detect()` beside the f16/bf16 fused kernels, with
the per-tier kernels beside their siblings in
`distance/simd/{avx512,avx2,neon,wasm_simd128}.rs`.
Every safe entry point validates both byte slices against `dim` before any
pointer arithmetic (`bbq::assert_payload_shapes`): dispatch guarantees the host's
features, never the shape of the data. Each tier carries a `should_panic` case on
that guard, and the crate's `simd_length_safety` suite gains the same contract for
the dispatched kernel.
The x86 tier modules are `pub(crate)`. Their entry points are safe `pub fn`s that
call `#[target_feature]` implementations, so exporting them would let safe code
outside the crate execute AVX2 or AVX-512 on a host that does not have it.
`SimdRuntime::detect()` is the only supported way to reach a tier, and it selects
one under a runtime feature probe.
`fluxbench` moves to the non-wasm dev-dependency set. It pulls `tokio`, which does
not build for wasm32, so the crate's own wasm unit tests could not be built at
all — and this change moves the wasm f32 kernels from scalar to SIMD, which would
have left them unverified. With the dev-dependency gated, the wasm lib suite
builds and runs: 374 pass on wasm32-wasip1 +simd128 under Node's WASI runner,
including the simd128 tier parity test.
`distance_prepared` rejects a candidate whose header names another dimension or
another quantizer instead of scoring it, and the prepared-payload length is
computed with checked arithmetic. A candidate buffer long enough to parse is not
proof that it belongs to this codec.
The wasm simd128 arm is wired into `detect()` and not merely compiled, gated on
the same compile-time condition as the tier module, since wasm has no runtime
feature probe. The NEON arm is gated to little-endian aarch64 because it
reinterprets the little-endian payload; its loads go through `vld1q_u8` plus
`vreinterpretq_f32_u8` so they do not depend on 4-byte alignment.
Evidence on the host (avx2+fma): `cargo nextest run -p nodedb-vector
--all-features --cargo-profile ci --profile ci` passes 468 tests with no
failures; the wasm lib suite passes 374 under Node's WASI runner; `cargo check
--workspace --all-features` is clean; `cargo fmt --all --check` and `cargo clippy
-p nodedb-vector --profile ci --all-targets -- -D warnings` are clean; the crate
checks for `aarch64-unknown-linux-gnu` and for `wasm32-wasip1` with and without
`+simd128`. Both jobs in CI run on `ubuntu-24.04-arm`, so CI exercises the NEON
tier and no x86 tier; the x86 tiers are checked on an x86_64 host, and the
512-bit tier under Intel SDE. No big-endian aarch64 target ships a prebuilt std,
so that path is not compiled here.
Benches against the reconstruct-and-measure baseline, re-run on this revision:
6.4x at dim 128 and 10.2x at dim 768, the fused path reporting zero allocations
and the baseline one `Vec<f32>` per candidate. The baseline approximates the
pre-fusion pass (fixed 1/sqrt(dim) scale, fixed header offset), so the ratio is
indicative rather than exact.
The AVX10/256 arm the first revision carried is gone: `AVX512VL` implies
`AVX512F`, so no host could reach it while AVX10 detection is absent from
`std::arch`.1 parent 1ff3551 commit 64aa6f1
13 files changed
Lines changed: 884 additions & 58 deletions
File tree
- nodedb-vector
- benches
- src
- distance/simd
- hnsw/graph/index
- rerank/codecs
- tests/vector_suite/cases
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
44 | 44 | | |
45 | 45 | | |
46 | 46 | | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
| 7 | + | |
| 8 | + | |
7 | 9 | | |
8 | 10 | | |
9 | 11 | | |
| |||
114 | 116 | | |
115 | 117 | | |
116 | 118 | | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
| 7 | + | |
| 8 | + | |
7 | 9 | | |
8 | 10 | | |
9 | 11 | | |
| |||
99 | 101 | | |
100 | 102 | | |
101 | 103 | | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
0 commit comments