From 0ed782c8cf55bc8e037e4eac1288afaaf3ed3253 Mon Sep 17 00:00:00 2001 From: LightWant <2510035537@qq.com> Date: Thu, 16 Jul 2026 10:58:54 +0000 Subject: [PATCH 01/10] feat: support split RaBitQ in Pyramid Signed-off-by: LightWant <2510035537@qq.com> Assisted-by: Codex:gpt-5 --- docs/docs/en/src/indexes/pyramid.md | 7 +- docs/docs/en/src/quantization/rabitq.md | 18 +-- docs/docs/en/src/quantization/rabitq_split.md | 14 +- docs/docs/zh/src/indexes/pyramid.md | 7 +- docs/docs/zh/src/quantization/rabitq.md | 18 +-- docs/docs/zh/src/quantization/rabitq_split.md | 14 +- include/vsag/constants.h | 5 + src/algorithm/pyramid/pyramid.cpp | 144 +++++++++++++++--- src/algorithm/pyramid/pyramid.h | 22 ++- src/algorithm/pyramid/pyramid_zparameters.cpp | 24 ++- src/algorithm/pyramid/pyramid_zparameters.h | 3 + .../pyramid/pyramid_zparameters_test.cpp | 118 ++++++++++++++ src/analyzer/pyramid_analyzer.cpp | 9 +- src/constants.cpp | 5 + src/impl/odescent/odescent_graph_builder.cpp | 24 +++ src/impl/odescent/odescent_graph_builder.h | 20 ++- tests/test_pyramid.cpp | 38 +++++ 17 files changed, 424 insertions(+), 66 deletions(-) diff --git a/docs/docs/en/src/indexes/pyramid.md b/docs/docs/en/src/indexes/pyramid.md index 50cba4f085..02d96fb0f3 100644 --- a/docs/docs/en/src/indexes/pyramid.md +++ b/docs/docs/en/src/indexes/pyramid.md @@ -93,7 +93,10 @@ Build-time parameters live under `index_param`. | `neighbor_sample_rate` | float | — | ODescent neighbor sampling rate. | | `no_build_levels` | int[] | `[]` | Tree levels that skip graph construction (0-indexed from the root). | | `use_reorder` | bool | `false` | Keep a high-precision copy for rescoring. | -| `precise_quantization_type` | string | `"fp32"` | Quantizer for reordering. | +| `precise_quantization_type` | string | `"fp32"` | Quantizer for reordering. Use `"rabitq"` with `rabitq_bits_per_dim_precise` to enable RaBitQ x+y split reorder from base storage. | +| `reorder_source` | string | `"precise"` | Reorder from a separate precise store (`"precise"`) or from base storage (`"base"`). Split RaBitQ selects `"base"` automatically. | +| `rabitq_bits_per_dim_precise` | int | unset | RaBitQ split `y` bits. When set with `base_quantization_type: "rabitq"` and `precise_quantization_type: "rabitq"`, Pyramid uses split storage; `rabitq_bits_per_dim_base` remains `x`, and `x + y <= 8`. | +| `rabitq_error_rate` | float | quantizer default | Lower-bound error multiplier used by split RaBitQ search. | | `index_min_size` | int | `0` | Minimum sub-index size; smaller groups fall back to scan. | | `support_duplicate` | bool | `false` | Allow duplicate ids. | | `build_thread_count` | int | `1` | Threads used for parallel build. | @@ -109,6 +112,8 @@ Search-time parameters live under the `pyramid` sub-object: | `subindex_ef_search` | int | `50` | Candidate list size used when traversing intermediate sub-graphs on the path. | | `hierarchies` | string[] | `[]` | Select which hierarchy to search. Empty means use the default (unnamed) hierarchy. | | `hierarchy_op` | string | `"single"` | How to combine results across hierarchies: `single` (search one hierarchy), `union`, or `intersection`. **Note:** `union` and `intersection` are not yet implemented — setting them will cause `KnnSearch`/`RangeSearch` to return an error. | +| `rabitq_one_bit_search` | bool | split index default | Overrides the one-bit RaBitQ split search path. Pyramid enables it by default when the index was built with split RaBitQ; pass `false` to force the standard search path. | +| `rabitq_error_rate` | float | index default | Positive lower-bound error multiplier for this search. | ```cpp auto result = index->KnnSearch( diff --git a/docs/docs/en/src/quantization/rabitq.md b/docs/docs/en/src/quantization/rabitq.md index ca79ae8052..f0092532d8 100644 --- a/docs/docs/en/src/quantization/rabitq.md +++ b/docs/docs/en/src/quantization/rabitq.md @@ -2,7 +2,7 @@ `rabitq` is VSAG's binary / low-bit quantizer. In its default mode each coordinate is encoded with **1 bit**, giving the highest compression ratio -of any built-in quantizer. On HGraph, an `x+y` split mode stores low-bit base +of any built-in quantizer. On HGraph and Pyramid, an `x+y` split mode stores low-bit base codes as `x` filter bits plus `y` supplement bits, so graph traversal can use only the filter code and re-ranking can fetch only the supplement bits it needs. @@ -10,7 +10,7 @@ only the filter code and re-ranking can fetch only the supplement bits it needs. > Implementation: `src/quantization/rabitq_quantization/rabitq_quantizer.cpp`, > parameter file `rabitq_quantizer_parameter.cpp`. -> For the complete HGraph split layout, lower-bound formula, and IO modes, +> For the complete split layout, lower-bound formula, and IO modes, > see [RaBitQ x+y Split](rabitq_split.md). ## When to use it @@ -32,7 +32,7 @@ as `"pca, rom, rabitq"`. - `rabitq_bits_per_dim_base = 1`: `ceil(dim / 8)` bytes per vector. With `dim = 768` that is 96 bytes (vs 3072 for fp32 → 32× smaller). - `rabitq_bits_per_dim_base = x` plus `rabitq_bits_per_dim_precise = y` - on HGraph: split mode stores roughly `(x + y) * dim / 8` bytes per vector + on HGraph or Pyramid: split mode stores roughly `(x + y) * dim / 8` bytes per vector for the RaBitQ code bytes. For example, `3+5` is about `dim` bytes per vector. @@ -42,9 +42,9 @@ as `"pca, rom, rabitq"`. | --- | --- | --- | --- | | `pca_dim` | int | `0` (= input dim) | Optional PCA preprocessing dimension applied inside RaBitQ. `0` means no PCA reduction (`rabitq_quantizer_parameter.cpp:30-32`). | | `rabitq_bits_per_dim_query` | int | `32` | Bits per dimension used to encode the **query** during search. Allowed values: `4` or `32` (`rabitq_quantizer_parameter.cpp:38-43`). | -| `rabitq_bits_per_dim_base` | int | `1` | In standard RaBitQ, bits per dimension for the stored base code. In HGraph `x+y` split mode, this external key means `x`, the filter bits used during graph traversal. Allowed range `[1, 8]`. | -| `rabitq_bits_per_dim_precise` | int | unset | HGraph-only split-mode key. When present with `base_quantization_type: "rabitq"` and `precise_quantization_type: "rabitq"`, this means `y`, the supplement bits used for reorder/full-distance refinement. The sum `x + y` must be `<= 8`. | -| `rabitq_error_rate` | float | `1.9` | Default lower-bound error multiplier for HGraph split search; must be finite and positive. It can be overridden per search under the `hgraph` object. | +| `rabitq_bits_per_dim_base` | int | `1` | In standard RaBitQ, bits per dimension for the stored base code. In HGraph/Pyramid `x+y` split mode, this external key means `x`, the filter bits used during graph traversal. Allowed range `[1, 8]`. | +| `rabitq_bits_per_dim_precise` | int | unset | HGraph/Pyramid split-mode key. When present with `base_quantization_type: "rabitq"` and `precise_quantization_type: "rabitq"`, this means `y`, the supplement bits used for reorder/full-distance refinement. The sum `x + y` must be `<= 8`. | +| `rabitq_error_rate` | float | `1.9` | Default lower-bound error multiplier for HGraph/Pyramid split search; must be finite and positive. It can be overridden per search under the `hgraph` or `pyramid` object. | | `use_fht` | bool | `false` | If `true`, applies a Fast Hadamard Transform rotation before binarization. Improves accuracy on anisotropic data with cheap O(dim log dim) cost (`rabitq_quantizer_parameter.cpp:76-78`). | | `fast_encode_rabitq` | bool | `true` | For stored codes wider than one bit, use CAQ-based fast encoding. Set to `false` to retain the exact RaBitQ encoder. The setting is ignored for one-bit codes. | | `fast_encode_rabitq_rounds` | int | `6` | Number of CAQ coordinate-adjustment rounds. Allowed range: `[1, 32]`. Each coordinate moves by at most one level per round. | @@ -93,8 +93,8 @@ IVF, and Pyramid and are propagated to both base and precise RaBitQ quantizers. ``` Swap to the higher-accuracy `x+y` split mode by setting both base and precise -quantization to RaBitQ and providing `rabitq_bits_per_dim_precise`. HGraph then -automatically selects the split datacell. In the example below, traversal uses +quantization to RaBitQ and providing `rabitq_bits_per_dim_precise`. HGraph and Pyramid then +automatically select the split datacell. In the example below, traversal uses `x = 3` filter bits and reorder reads only `y = 5` supplement bits: ```json @@ -127,7 +127,7 @@ also tracks a residual norm so the inner-product estimate is unbiased. `precise_quantization_type: "fp32"` is the safe default. - **Rotate first.** For un-normalized data, set `rabitq_use_fht: true` or use a `tq` chain that includes `rom` / `fht`. -- **Split mode for accuracy.** HGraph `x+y` split keeps an `x`-bit fast path +- **Split mode for accuracy.** HGraph/Pyramid `x+y` split keeps an `x`-bit fast path for graph traversal and adds `y` supplement bits for re-ranking; expect significantly higher recall than pure 1-bit when using more total bits. diff --git a/docs/docs/en/src/quantization/rabitq_split.md b/docs/docs/en/src/quantization/rabitq_split.md index 3df0623d05..8e541b7953 100644 --- a/docs/docs/en/src/quantization/rabitq_split.md +++ b/docs/docs/en/src/quantization/rabitq_split.md @@ -1,6 +1,6 @@ # RaBitQ x+y Split -RaBitQ x+y split is an HGraph storage and search mode for low-bit base codes. +RaBitQ x+y split is an HGraph and Pyramid storage and search mode for low-bit base codes. Each vector is divided into two records: - `x` filter bits are read during graph traversal and lower-bound filtering. @@ -13,7 +13,7 @@ memory while the colder supplement record is stored on disk. ## Enable split mode -HGraph selects split mode when both quantization types are `rabitq` and +HGraph and Pyramid select split mode when both quantization types are `rabitq` and `rabitq_bits_per_dim_precise` is present: ```json @@ -55,7 +55,7 @@ The constraints are: x + y <= 8 ``` -If `rabitq_bits_per_dim_precise` is omitted, HGraph uses the standard RaBitQ +If `rabitq_bits_per_dim_precise` is omitted, HGraph and Pyramid use the standard RaBitQ path instead of split storage. Enable the filter/lower-bound search path with: @@ -90,7 +90,7 @@ The split search path has four stages: 4. The final distance combines the filter contribution and supplement contribution into one `x+y`-bit RaBitQ estimate. -The HGraph heap is therefore not populated with an `x+y` distance for every +The graph-search heap is therefore not populated with an `x+y` distance for every visited vector. The inexpensive x-bit distance drives traversal; the more accurate distance is evaluated only during candidate reorder. @@ -260,7 +260,7 @@ sum_i q_i * u_i + sum_i q_i * s_i ``` -For L2 with an x-bit lookup filter, HGraph passes the previously computed +For L2 with an x-bit lookup filter, HGraph and Pyramid pass the previously computed filter distance to reorder as a hint. `ComputeDistWithSplitCodeAndFilterDist` recovers the first term from that hint and computes only the second term from the y supplement planes: @@ -356,7 +356,7 @@ search-time `hgraph.rabitq_error_rate` does not. ## Operational notes -- Split storage is currently an HGraph feature and requires fp32 query codes. +- Split storage is currently available on HGraph and Pyramid and requires fp32 query codes. Pyramid enables the one-bit split search path by default for split indexes; pass `rabitq_one_bit_search: false` under `pyramid` to force the standard search path. - `l2`, `ip`, and `cosine` are supported. The filter-hint reorder shortcut is currently specialized for L2. - Keep `use_reorder: true` unless x-bit traversal accuracy alone has been @@ -364,4 +364,4 @@ search-time `hgraph.rabitq_error_rate` does not. - Changing x, y, metric, or transform parameters requires rebuilding the index. A search-time `hgraph.rabitq_error_rate` override does not. - Use [RaBitQ](rabitq.md) for the general quantizer description and - [HGraph](../indexes/hgraph.md) for the complete index parameter table. + [HGraph](../indexes/hgraph.md) and [Pyramid](../indexes/pyramid.md) for the complete index parameter tables. diff --git a/docs/docs/zh/src/indexes/pyramid.md b/docs/docs/zh/src/indexes/pyramid.md index 439e634d04..d78261027a 100644 --- a/docs/docs/zh/src/indexes/pyramid.md +++ b/docs/docs/zh/src/indexes/pyramid.md @@ -88,7 +88,10 @@ auto result = index->KnnSearch( | `neighbor_sample_rate` | float | — | ODescent 的邻居采样比率 | | `no_build_levels` | int[] | `[]` | 跳过构图的层级（从根节点开始的 0-based 下标） | | `use_reorder` | bool | `false` | 是否保留高精度副本用于精排 | -| `precise_quantization_type` | string | `"fp32"` | 精排使用的量化类型 | +| `precise_quantization_type` | string | `"fp32"` | 精排使用的量化类型。与 `rabitq_bits_per_dim_precise` 配合设为 `"rabitq"` 时，可启用从 base storage 重排的 RaBitQ x+y split。 | +| `reorder_source` | string | `"precise"` | 从单独的 precise storage（`"precise"`）或 base storage（`"base"`）执行重排。RaBitQ split 会自动选择 `"base"`。 | +| `rabitq_bits_per_dim_precise` | int | 未设置 | RaBitQ split 的 `y` bits。和 `base_quantization_type: "rabitq"`、`precise_quantization_type: "rabitq"` 一起设置时，Pyramid 使用 split storage；`rabitq_bits_per_dim_base` 仍表示 `x`，且 `x + y <= 8`。 | +| `rabitq_error_rate` | float | 量化器默认值 | split RaBitQ 搜索使用的 lower-bound 误差倍率。 | | `index_min_size` | int | `0` | 子索引的最小规模；小于该值的分区会退化为线性扫描 | | `support_duplicate` | bool | `false` | 是否允许重复 ID | | `build_thread_count` | int | `1` | 构建阶段并发线程数 | @@ -104,6 +107,8 @@ auto result = index->KnnSearch( | `subindex_ef_search` | int | `50` | 沿路径向下遍历中间子图时的候选集大小 | | `hierarchies` | string[] | `[]` | 指定检索哪个层级。空数组表示使用默认（匿名）层级。 | | `hierarchy_op` | string | `"single"` | 多层级结果合并方式：`single`（检索单个层级）、`union`、`intersection`。**注意：** `union` 和 `intersection` 尚未实现——设置后 `KnnSearch`/`RangeSearch` 会返回错误。 | +| `rabitq_one_bit_search` | bool | split 索引默认值 | 覆盖 one-bit RaBitQ split 搜索路径。Pyramid 使用 split RaBitQ 构建后默认启用；传 `false` 可强制使用普通搜索路径。 | +| `rabitq_error_rate` | float | 索引默认值 | 本次搜索使用的正数 lower-bound 误差倍率。 | ```cpp auto result = index->KnnSearch( diff --git a/docs/docs/zh/src/quantization/rabitq.md b/docs/docs/zh/src/quantization/rabitq.md index 3c9bafa215..60e0b6e8ac 100644 --- a/docs/docs/zh/src/quantization/rabitq.md +++ b/docs/docs/zh/src/quantization/rabitq.md @@ -1,7 +1,7 @@ # RaBitQ `rabitq` 是 VSAG 的二值 / 低比特量化器。默认模式下每个坐标用 **1 比特** -编码，给出所有内建量化器中最高的压缩率。在 HGraph 上，`x+y` split 模式把 +编码，给出所有内建量化器中最高的压缩率。在 HGraph 和 Pyramid 上，`x+y` split 模式把 底库码拆成 `x` 个过滤 bit 和 `y` 个 supplement bit：图遍历只使用 filter code， 重排 / full-distance 阶段只额外读取 supplement bits。 @@ -9,7 +9,7 @@ > 实现：`src/quantization/rabitq_quantization/rabitq_quantizer.cpp`， > 参数文件 `rabitq_quantizer_parameter.cpp`。 -> HGraph split 的完整存储布局、lower bound 公式和 IO 模式见 +> split 的完整存储布局、lower bound 公式和 IO 模式见 > [RaBitQ x+y Split](rabitq_split.md)。 ## 何时使用 @@ -27,7 +27,7 @@ - `rabitq_bits_per_dim_base = 1`：每向量 `ceil(dim / 8)` 字节。`dim = 768` 时为 96 字节（对比 fp32 的 3072 → 小 32×）。 -- HGraph 上 `rabitq_bits_per_dim_base = x` 且 +- HGraph 或 Pyramid 上 `rabitq_bits_per_dim_base = x` 且 `rabitq_bits_per_dim_precise = y`：split 模式约存储 `(x + y) * dim / 8` 字节的 RaBitQ code。例如 `3+5` 约为每向量 `dim` 字节。 @@ -38,9 +38,9 @@ | --- | --- | --- | --- | | `pca_dim` | int | `0`（= 输入维度） | RaBitQ 内部可选的 PCA 预处理维度。`0` 表示不做 PCA 降维（`rabitq_quantizer_parameter.cpp:30-32`）。 | | `rabitq_bits_per_dim_query` | int | `32` | 搜索时**查询**的每维位数。允许值：`4` 或 `32`（`rabitq_quantizer_parameter.cpp:38-43`）。 | -| `rabitq_bits_per_dim_base` | int | `1` | standard RaBitQ 下表示底库码每维位数；HGraph `x+y` split 下，这个外部 key 表示 `x`，即图遍历过滤阶段使用的 filter bits。范围 `[1, 8]`。 | -| `rabitq_bits_per_dim_precise` | int | 未设置 | HGraph-only split 模式 key。和 `base_quantization_type: "rabitq"`、`precise_quantization_type: "rabitq"` 一起出现时表示 `y`，即重排 / full-distance 阶段读取的 supplement bits。要求 `x + y <= 8`。 | -| `rabitq_error_rate` | float | `1.9` | HGraph split 搜索的默认 lower-bound 误差倍率；必须为有限正数，也可以在 `hgraph` 搜索参数中按次覆盖。 | +| `rabitq_bits_per_dim_base` | int | `1` | standard RaBitQ 下表示底库码每维位数；HGraph/Pyramid `x+y` split 下，这个外部 key 表示 `x`，即图遍历过滤阶段使用的 filter bits。范围 `[1, 8]`。 | +| `rabitq_bits_per_dim_precise` | int | 未设置 | HGraph/Pyramid split 模式 key。和 `base_quantization_type: "rabitq"`、`precise_quantization_type: "rabitq"` 一起出现时表示 `y`，即重排 / full-distance 阶段读取的 supplement bits。要求 `x + y <= 8`。 | +| `rabitq_error_rate` | float | `1.9` | HGraph/Pyramid split 搜索的默认 lower-bound 误差倍率；必须为有限正数，也可以在 `hgraph` 或 `pyramid` 搜索参数中按次覆盖。 | | `use_fht` | bool | `false` | `true` 时在二值化前应用快速 Hadamard 变换旋转。以 O(dim log dim) 的廉价代价提升各向异性数据上的精度（`rabitq_quantizer_parameter.cpp:76-78`）。 | | `fast_encode_rabitq` | bool | `true` | 对大于 1 bit 的底库码启用基于 CAQ 的快速编码；设为 `false` 时使用原有精确编码。1 bit 编码会忽略此参数。 | | `fast_encode_rabitq_rounds` | int | `6` | CAQ 坐标微调轮数，范围 `[1, 32]`；每个坐标在每轮最多移动一级。 | @@ -59,7 +59,7 @@ `rabitq_bits_per_dim_precise`、`rabitq_error_rate`、`rabitq_use_fht`；IVF 暴露 `rabitq_pca_dim`、`rabitq_bits_per_dim_query`、 `rabitq_bits_per_dim_base`、`rabitq_version`、`rabitq_error_rate`、 -`rabitq_use_fht`；Pyramid 为底层量化器暴露 PCA、底库/查询位数和 FHT +`rabitq_use_fht`；Pyramid 为底层量化器暴露 PCA、底库/查询位数、split precise 位数、误差倍率和 FHT 相关 key。其中 `rabitq_use_fht` 是索引层对量化器内部 `use_fht` key 的别名，会由索引层重写。 @@ -86,7 +86,7 @@ ``` 切换到高精度的 `x+y` split 模式：把 base 和 precise 量化都设置为 RaBitQ， -并提供 `rabitq_bits_per_dim_precise`。HGraph 会自动选择 split datacell。 +并提供 `rabitq_bits_per_dim_precise`。HGraph 和 Pyramid 会自动选择 split datacell。 下面例子中，图遍历使用 `x = 3` 个 filter bits，重排只读取 `y = 5` 个 supplement bits： @@ -117,7 +117,7 @@ FHT 旋转是固定的（无需学习），因此不增加训练代价；PCA 预 `use_reorder: true` + `precise_quantization_type: "fp32"` 是稳妥默认。 - **先旋转。** 对未归一化数据，设 `rabitq_use_fht: true`，或在 `tq` 链路 中包含 `rom` / `fht`。 -- **精度优先时用 split 模式。** HGraph `x+y` split 保留 `x` bit 快速 +- **精度优先时用 split 模式。** HGraph/Pyramid `x+y` split 保留 `x` bit 快速 过滤路径，再添加 `y` 个 supplement bits 用于重排；相对纯 1 比特，使用 更多总 bit 时召回明显更高。 diff --git a/docs/docs/zh/src/quantization/rabitq_split.md b/docs/docs/zh/src/quantization/rabitq_split.md index 5b157a72ea..a0322a0dba 100644 --- a/docs/docs/zh/src/quantization/rabitq_split.md +++ b/docs/docs/zh/src/quantization/rabitq_split.md @@ -1,6 +1,6 @@ # RaBitQ x+y Split -RaBitQ x+y split 是 HGraph 面向低比特底库码的存储与搜索模式。每条向量拆成 +RaBitQ x+y split 是 HGraph 和 Pyramid 面向低比特底库码的存储与搜索模式。每条向量拆成 两条记录： - 图遍历和 lower-bound 过滤只读取 `x` 个 filter bits。 @@ -13,7 +13,7 @@ RaBitQ x+y split 是 HGraph 面向低比特底库码的存储与搜索模式。 ## 启用 split 模式 当 base 和 precise 的量化类型都为 `rabitq`，并且配置了 -`rabitq_bits_per_dim_precise` 时，HGraph 自动选择 split 模式： +`rabitq_bits_per_dim_precise` 时，HGraph 和 Pyramid 自动选择 split 模式： ```json { @@ -54,7 +54,7 @@ RaBitQ x+y split 是 HGraph 面向低比特底库码的存储与搜索模式。 x + y <= 8 ``` -如果不配置 `rabitq_bits_per_dim_precise`，HGraph 使用 standard RaBitQ 路径， +如果不配置 `rabitq_bits_per_dim_precise`，HGraph 和 Pyramid 使用 standard RaBitQ 路径， 不会创建 split storage。 使用以下搜索参数启用 filter/lower-bound 搜索路径： @@ -88,7 +88,7 @@ split 搜索分为四个阶段： 4. 最终距离把 filter contribution 与 supplement contribution 合成为 `x+y`-bit RaBitQ 估计。 -因此，HGraph 不会为每个访问到的向量都计算 `x+y` 距离并放入搜索堆。图遍历由 +因此，图搜索不会为每个访问到的向量都计算 `x+y` 距离并放入搜索堆。图遍历由 低成本的 x-bit 距离驱动，更精确的距离只在候选重排阶段计算。 ## 编码和 bit-plane @@ -252,7 +252,7 @@ sum_i q_i * u_i + sum_i q_i * s_i ``` -对使用 x-bit lookup filter 的 L2 搜索，HGraph 会把之前计算的 filter distance +对使用 x-bit lookup filter 的 L2 搜索，HGraph 和 Pyramid 会把之前计算的 filter distance 作为 hint 传给 reorder。`ComputeDistWithSplitCodeAndFilterDist` 从 hint 恢复第一项， 只从 y 个 supplement planes 计算第二项： @@ -342,11 +342,11 @@ x/y bit 数和 query bits。修改编码参数需要重建索引；只调整搜 ## 使用注意 -- split storage 当前是 HGraph 功能，并且要求 fp32 query code。 +- split storage 当前可用于 HGraph 和 Pyramid，并且要求 fp32 query code。Pyramid 的 split 索引默认启用 one-bit split 搜索路径；如需强制使用普通搜索路径，可以在 `pyramid` 搜索参数下传 `rabitq_one_bit_search: false`。 - 支持 `l2`、`ip` 和 `cosine`；利用 filter hint 的 reorder 快速路径当前针对 L2。 - 除非已经验证仅靠 x-bit 遍历距离能满足召回要求，否则应保持 `use_reorder: true`。 - 修改 x、y、metric 或 transform 参数后必须重建索引；在搜索参数中覆盖 `hgraph.rabitq_error_rate` 不需要重建。 - RaBitQ 通用说明见 [RaBitQ](rabitq.md)，完整 HGraph 参数见 - [HGraph 索引](../indexes/hgraph.md)。 + [HGraph 索引](../indexes/hgraph.md)和 [Pyramid 索引](../indexes/pyramid.md)。 diff --git a/include/vsag/constants.h b/include/vsag/constants.h index 70abc6670d..cee29913da 100644 --- a/include/vsag/constants.h +++ b/include/vsag/constants.h @@ -125,19 +125,24 @@ extern const char* const PYRAMID_GRAPH_TYPE; extern const char* const PYRAMID_GRAPH_STORAGE_TYPE; extern const char* const PYRAMID_BUILD_THREAD_COUNT; extern const char* const PYRAMID_PRECISE_QUANTIZATION_TYPE; +extern const char* const PYRAMID_REORDER_SOURCE; extern const char* const PYRAMID_RABITQ_BITS_PER_DIM_BASE; extern const char* const PYRAMID_RABITQ_BITS_PER_DIM_QUERY; +extern const char* const PYRAMID_RABITQ_BITS_PER_DIM_PRECISE; extern const char* const PYRAMID_RABITQ_PCA_DIM; extern const char* const PYRAMID_RABITQ_USE_FHT; extern const char* const PYRAMID_FAST_ENCODE_RABITQ; extern const char* const PYRAMID_FAST_ENCODE_RABITQ_ROUNDS; extern const char* const PYRAMID_BASE_IO_TYPE; +extern const char* const PYRAMID_BASE_SUPPLEMENT_IO_TYPE; extern const char* const PYRAMID_BASE_PQ_DIM; extern const char* const PYRAMID_BASE_FILE_PATH; +extern const char* const PYRAMID_BASE_SUPPLEMENT_FILE_PATH; extern const char* const PYRAMID_PRECISE_IO_TYPE; extern const char* const PYRAMID_PRECISE_FILE_PATH; extern const char* const PYRAMID_PARAMETER_EF_SEARCH; extern const char* const PYRAMID_PARAMETER_SUBINDEX_EF_SEARCH; +extern const char* const PYRAMID_PARAMETER_RABITQ_ONE_BIT_SEARCH; extern const char* const PYRAMID_PARAMETER_HIERARCHIES; extern const char* const PYRAMID_PARAMETER_HIERARCHY_OP; extern const char* const PYRAMID_NO_BUILD_LEVELS; diff --git a/src/algorithm/pyramid/pyramid.cpp b/src/algorithm/pyramid/pyramid.cpp index f9eae66322..7cf5be4c41 100644 --- a/src/algorithm/pyramid/pyramid.cpp +++ b/src/algorithm/pyramid/pyramid.cpp @@ -15,6 +15,8 @@ #include "pyramid.h" +#include + #include #include "algorithm/inner_index_interface.h" @@ -23,7 +25,7 @@ #include "impl/heap/standard_heap.h" #include "impl/odescent/odescent_graph_builder.h" #include "impl/pruning_strategy.h" -#include "io/memory_io/memory_io_parameter.h" +#include "quantization/rabitq_quantization/rabitq_quantizer_parameter.h" #include "query_context.h" #include "storage/empty_index_binary_set.h" #include "storage/serialization.h" @@ -33,6 +35,84 @@ #include "utils/util_functions.h" namespace vsag { +namespace { + +void +map_rabitq_split_param(const JsonType& external_json, JsonType& inner_json) { + if (not external_json.Contains(PYRAMID_RABITQ_BITS_PER_DIM_PRECISE)) { + return; + } + + CHECK_ARGUMENT(external_json.Contains(PYRAMID_RABITQ_BITS_PER_DIM_BASE), + fmt::format("{} requires {}", + PYRAMID_RABITQ_BITS_PER_DIM_PRECISE, + PYRAMID_RABITQ_BITS_PER_DIM_BASE)); + CHECK_ARGUMENT( + external_json.Contains(PYRAMID_BASE_QUANTIZATION_TYPE), + fmt::format( + "{} requires {}", PYRAMID_RABITQ_BITS_PER_DIM_PRECISE, PYRAMID_BASE_QUANTIZATION_TYPE)); + CHECK_ARGUMENT(external_json.Contains(PYRAMID_PRECISE_QUANTIZATION_TYPE), + fmt::format("{} requires {}", + PYRAMID_RABITQ_BITS_PER_DIM_PRECISE, + PYRAMID_PRECISE_QUANTIZATION_TYPE)); + + CHECK_ARGUMENT(external_json[PYRAMID_BASE_QUANTIZATION_TYPE].IsString(), + fmt::format("{} must be a string", PYRAMID_BASE_QUANTIZATION_TYPE)); + CHECK_ARGUMENT(external_json[PYRAMID_PRECISE_QUANTIZATION_TYPE].IsString(), + fmt::format("{} must be a string", PYRAMID_PRECISE_QUANTIZATION_TYPE)); + + const auto base_quantization_type = external_json[PYRAMID_BASE_QUANTIZATION_TYPE].GetString(); + const auto precise_quantization_type = + external_json[PYRAMID_PRECISE_QUANTIZATION_TYPE].GetString(); + CHECK_ARGUMENT(base_quantization_type == QUANTIZATION_TYPE_VALUE_RABITQ, + fmt::format("{} requires {}={}", + PYRAMID_RABITQ_BITS_PER_DIM_PRECISE, + PYRAMID_BASE_QUANTIZATION_TYPE, + QUANTIZATION_TYPE_VALUE_RABITQ)); + CHECK_ARGUMENT(precise_quantization_type == QUANTIZATION_TYPE_VALUE_RABITQ, + fmt::format("{} requires {}={}", + PYRAMID_RABITQ_BITS_PER_DIM_PRECISE, + PYRAMID_PRECISE_QUANTIZATION_TYPE, + QUANTIZATION_TYPE_VALUE_RABITQ)); + + const int64_t filter_bits = external_json[PYRAMID_RABITQ_BITS_PER_DIM_BASE].GetInt(); + const int64_t supplement_bits = external_json[PYRAMID_RABITQ_BITS_PER_DIM_PRECISE].GetInt(); + CHECK_ARGUMENT( + filter_bits >= 1 and filter_bits <= 8, + fmt::format("{} must be in [1, 8], got {}", PYRAMID_RABITQ_BITS_PER_DIM_BASE, filter_bits)); + CHECK_ARGUMENT( + supplement_bits >= 1 and supplement_bits <= 8, + fmt::format( + "{} must be in [1, 8], got {}", PYRAMID_RABITQ_BITS_PER_DIM_PRECISE, supplement_bits)); + const int64_t total_bits = filter_bits + supplement_bits; + CHECK_ARGUMENT(total_bits <= 8, + fmt::format("{} + {} must be no greater than 8, got {}", + PYRAMID_RABITQ_BITS_PER_DIM_BASE, + PYRAMID_RABITQ_BITS_PER_DIM_PRECISE, + total_bits)); + + if (external_json.Contains(PYRAMID_RABITQ_BITS_PER_DIM_QUERY)) { + const int64_t query_bits = external_json[PYRAMID_RABITQ_BITS_PER_DIM_QUERY].GetInt(); + CHECK_ARGUMENT(query_bits == 32, + fmt::format("split storage requires {} to be 32, got {}", + PYRAMID_RABITQ_BITS_PER_DIM_QUERY, + query_bits)); + } + + inner_json[REORDER_SOURCE_KEY].SetString(HGRAPH_REORDER_SOURCE_BASE); + inner_json[BASE_CODES_KEY][CODES_TYPE_KEY].SetString(RABITQ_SPLIT_CODES); + inner_json[BASE_CODES_KEY][QUANTIZATION_PARAMS_KEY][RABITQ_QUANTIZATION_VERSION_KEY].SetString( + RaBitQuantizerParameter::RABITQ_VERSION_SPLIT); + inner_json[BASE_CODES_KEY][QUANTIZATION_PARAMS_KEY][RABITQ_QUANTIZATION_BITS_PER_DIM_QUERY_KEY] + .SetInt(32); + inner_json[BASE_CODES_KEY][QUANTIZATION_PARAMS_KEY][RABITQ_QUANTIZATION_BITS_PER_DIM_FILTER_KEY] + .SetInt(filter_bits); + inner_json[BASE_CODES_KEY][QUANTIZATION_PARAMS_KEY][RABITQ_QUANTIZATION_BITS_PER_DIM_BASE_KEY] + .SetInt(total_bits); +} + +} // namespace + const static float RADIUS_EPSILON = 1.1F; std::vector @@ -220,17 +300,18 @@ Pyramid::build_by_odescent(const DatasetPtr& base) { std::memcpy(label_table_->label_table_.data(), data_ids, sizeof(LabelType) * data_num); base_codes_->BatchInsertVector(data_vectors, data_num); - if (use_reorder_) { + if (has_precise_reorder()) { precise_codes_->BatchInsertVector(data_vectors, data_num); } - auto codes = use_reorder_ ? precise_codes_ : base_codes_; + auto codes = has_precise_reorder() ? precise_codes_ : base_codes_; if (thread_pool_ != nullptr && hierarchies_.size() > 1) { Vector> futures(allocator_); for (const auto& [hname, h_ptr] : hierarchies_) { auto* root_ptr = h_ptr->root.get(); futures.push_back(thread_pool_->GeneralEnqueue([&, codes, root_ptr]() { - ODescent builder(odescent_param_, codes, allocator_, nullptr); + ODescent builder( + odescent_param_, codes, allocator_, nullptr, true, data_vectors, data_num); root_ptr->Build(builder); })); } @@ -238,7 +319,13 @@ Pyramid::build_by_odescent(const DatasetPtr& base) { f.get(); } } else { - ODescent graph_builder(odescent_param_, codes, allocator_, this->thread_pool_.get()); + ODescent graph_builder(odescent_param_, + codes, + allocator_, + this->thread_pool_.get(), + true, + data_vectors, + data_num); for (const auto& [hname, h_ptr] : hierarchies_) { h_ptr->root->Build(graph_builder); } @@ -256,6 +343,7 @@ Pyramid::KnnSearch(const DatasetPtr& query, QueryContext ctx{.stats = &stats}; auto parsed_param = PyramidSearchParameters::FromJson(parameters); + ctx.rabitq_error_rate = parsed_param.rabitq_error_rate; CHECK_ARGUMENT(k > 0, fmt::format("k({}) must be greater than 0", k)); CHECK_ARGUMENT(parsed_param.hierarchy_op == PyramidSearchParameters::HierarchyOp::SINGLE, "multi-hierarchy search (union/intersection) is not yet implemented"); @@ -271,6 +359,9 @@ Pyramid::KnnSearch(const DatasetPtr& query, search_param.topk = k; search_param.search_mode = KNN_SEARCH; search_param.parallel_search_thread_count = parsed_param.parallel_search_thread_count; + search_param.enable_rabitq_one_bit_search = parsed_param.has_rabitq_one_bit_search + ? parsed_param.rabitq_one_bit_search + : default_rabitq_one_bit_search_; if (this->support_duplicate_) { search_param.consider_duplicate = true; } @@ -305,6 +396,7 @@ Pyramid::RangeSearch(const DatasetPtr& query, QueryContext ctx{.stats = &stats}; auto parsed_param = PyramidSearchParameters::FromJson(parameters); + ctx.rabitq_error_rate = parsed_param.rabitq_error_rate; CHECK_ARGUMENT(parsed_param.hierarchy_op == PyramidSearchParameters::HierarchyOp::SINGLE, "multi-hierarchy search (union/intersection) is not yet implemented"); InnerSearchParam search_param; @@ -312,6 +404,9 @@ Pyramid::RangeSearch(const DatasetPtr& query, search_param.radius = radius * RADIUS_EPSILON; search_param.search_mode = RANGE_SEARCH; search_param.parallel_search_thread_count = parsed_param.parallel_search_thread_count; + search_param.enable_rabitq_one_bit_search = parsed_param.has_rabitq_one_bit_search + ? parsed_param.rabitq_one_bit_search + : default_rabitq_one_bit_search_; search_param.topk = limited_size == -1 ? std::numeric_limits::max() : limited_size; if (parsed_param.enable_time_record) { @@ -430,7 +525,7 @@ void Pyramid::Serialize(StreamWriter& writer) const { label_table_->Serialize(writer); base_codes_->Serialize(writer); - if (use_reorder_) { + if (has_precise_reorder()) { precise_codes_->Serialize(writer); } @@ -480,7 +575,7 @@ Pyramid::collect_streaming_header() const { base_tag, StreamSerializationBlockCurrentVersion(base_tag), StreamSerializationTagCritical(base_tag)); - if (this->use_reorder_) { + if (this->has_precise_reorder()) { auto tag = static_cast(StreamSerializationTag::HIGH_PRECISION_CODES); AppendStreamingManifestBlock(manifest, tag, @@ -525,7 +620,7 @@ Pyramid::serialize_streaming_body(StreamWriter& writer) const { writer, base_tag, StreamSerializationTagCritical(base_tag), [this](StreamWriter& w) { this->base_codes_->Serialize(w); }); - if (this->use_reorder_) { + if (this->has_precise_reorder()) { auto tag = static_cast(StreamSerializationTag::HIGH_PRECISION_CODES); WriteStreamingBlock( writer, tag, StreamSerializationTagCritical(tag), [this](StreamWriter& w) { @@ -639,7 +734,7 @@ Pyramid::read_streaming_body(StreamReader& reader, const MetadataPtr& metadata) loaded_base_codes = true; break; case StreamSerializationTag::HIGH_PRECISION_CODES: - if (this->use_reorder_) { + if (this->has_precise_reorder()) { ReadSeekableBlockPayload( block_reader, block_header, [this](StreamReader& block) { this->precise_codes_->Deserialize(block); @@ -676,7 +771,7 @@ Pyramid::read_streaming_body(StreamReader& reader, const MetadataPtr& metadata) throw VsagException(ErrorType::READ_ERROR, "Pyramid streaming serialization required block is missing"); } - if (this->use_reorder_ && !loaded_precise_codes) { + if (this->has_precise_reorder() && !loaded_precise_codes) { throw VsagException(ErrorType::READ_ERROR, "Pyramid streaming serialization precise codes block is missing"); } @@ -701,7 +796,7 @@ Pyramid::Deserialize(StreamReader& reader) { delete_count_.store(static_cast(label_table_->GetAllDeletedIds().size()), std::memory_order_relaxed); base_codes_->Deserialize(buffer_reader); - if (use_reorder_) { + if (has_precise_reorder()) { precise_codes_->Deserialize(buffer_reader); } cur_element_count_ = base_codes_->TotalCount(); @@ -741,7 +836,7 @@ Pyramid::ExportModel(const IndexCommonParam& param) const { "Export model's pyramid reorder config mismatched"); } this->base_codes_->ExportModel(index->base_codes_); - if (use_reorder_) { + if (has_precise_reorder()) { if (index->precise_codes_ == nullptr) { throw VsagException(ErrorType::INTERNAL_ERROR, "Export model's pyramid precise codes is empty"); @@ -778,7 +873,7 @@ Pyramid::Add(const DatasetPtr& base) { label_table_->Insert(valid_id_count + local_cur_element_count, data_ids[i]); base_codes_->InsertVector(data_vectors + dim_ * i, valid_id_count + local_cur_element_count); - if (use_reorder_) { + if (has_precise_reorder()) { precise_codes_->InsertVector(data_vectors + dim_ * i, valid_id_count + local_cur_element_count); } @@ -811,7 +906,7 @@ Pyramid::resize(int64_t new_max_capacity) { pool_ = std::make_unique(1, allocator_, new_max_capacity, allocator_); label_table_->Resize(new_max_capacity); base_codes_->Resize(new_max_capacity); - if (use_reorder_) { + if (has_precise_reorder()) { precise_codes_->Resize(new_max_capacity); } points_mutex_->Resize(new_max_capacity); @@ -936,13 +1031,20 @@ Pyramid::CheckAndMappingExternalParam(const JsonType& external_param, const ConstParamMap external_mapping = { {PYRAMID_EF_CONSTRUCTION, {EF_CONSTRUCTION_KEY}}, {PYRAMID_USE_REORDER, {USE_REORDER_KEY}}, + {PYRAMID_REORDER_SOURCE, {REORDER_SOURCE_KEY}}, {PYRAMID_BASE_QUANTIZATION_TYPE, {BASE_CODES_KEY, QUANTIZATION_PARAMS_KEY, TYPE_KEY}}, {PYRAMID_RABITQ_BITS_PER_DIM_BASE, {BASE_CODES_KEY, QUANTIZATION_PARAMS_KEY, RABITQ_QUANTIZATION_BITS_PER_DIM_BASE_KEY}}, {PYRAMID_RABITQ_BITS_PER_DIM_QUERY, {BASE_CODES_KEY, QUANTIZATION_PARAMS_KEY, RABITQ_QUANTIZATION_BITS_PER_DIM_QUERY_KEY}}, + {PYRAMID_RABITQ_BITS_PER_DIM_PRECISE, + {PRECISE_CODES_KEY, QUANTIZATION_PARAMS_KEY, RABITQ_QUANTIZATION_BITS_PER_DIM_BASE_KEY}}, {PYRAMID_RABITQ_PCA_DIM, {BASE_CODES_KEY, QUANTIZATION_PARAMS_KEY, PCA_DIM_KEY}}, {PYRAMID_RABITQ_USE_FHT, {BASE_CODES_KEY, QUANTIZATION_PARAMS_KEY, USE_FHT_KEY}}, + {RABITQ_ERROR_RATE, + {BASE_CODES_KEY, QUANTIZATION_PARAMS_KEY, RABITQ_QUANTIZATION_ERROR_RATE_KEY}}, + {RABITQ_ERROR_RATE, + {PRECISE_CODES_KEY, QUANTIZATION_PARAMS_KEY, RABITQ_QUANTIZATION_ERROR_RATE_KEY}}, {PYRAMID_FAST_ENCODE_RABITQ, {BASE_CODES_KEY, QUANTIZATION_PARAMS_KEY, FAST_ENCODE_RABITQ_KEY}}, {PYRAMID_FAST_ENCODE_RABITQ, @@ -954,6 +1056,9 @@ Pyramid::CheckAndMappingExternalParam(const JsonType& external_param, {PYRAMID_PRECISE_QUANTIZATION_TYPE, {PRECISE_CODES_KEY, QUANTIZATION_PARAMS_KEY, TYPE_KEY}}, {PYRAMID_GRAPH_MAX_DEGREE, {GRAPH_KEY, GRAPH_PARAM_MAX_DEGREE_KEY}}, {PYRAMID_BASE_IO_TYPE, {BASE_CODES_KEY, IO_PARAMS_KEY, TYPE_KEY}}, + {PYRAMID_BASE_SUPPLEMENT_IO_TYPE, {BASE_CODES_KEY, SUPPLEMENT_IO_PARAMS_KEY, TYPE_KEY}}, + {PYRAMID_BASE_SUPPLEMENT_FILE_PATH, + {BASE_CODES_KEY, SUPPLEMENT_IO_PARAMS_KEY, IO_FILE_PATH_KEY}}, {PYRAMID_BUILD_ALPHA, {GRAPH_KEY, ODESCENT_PARAMETER_ALPHA}}, {PYRAMID_GRAPH_TYPE, {GRAPH_KEY, GRAPH_TYPE_KEY}}, {PYRAMID_GRAPH_STORAGE_TYPE, {GRAPH_KEY, GRAPH_STORAGE_TYPE_KEY}}, @@ -977,6 +1082,7 @@ Pyramid::CheckAndMappingExternalParam(const JsonType& external_param, std::string str = format_map(HGRAPH_PARAMS_TEMPLATE, DEFAULT_MAP); auto inner_json = JsonType::Parse(str); mapping_external_param_to_inner(external_param, external_mapping, inner_json); + map_rabitq_split_param(external_param, inner_json); auto pyramid_params = std::make_shared(); pyramid_params->FromJson(inner_json); return pyramid_params; @@ -985,7 +1091,7 @@ Pyramid::CheckAndMappingExternalParam(const JsonType& external_param, void Pyramid::Train(const DatasetPtr& base) { this->base_codes_->Train(base->GetFloat32Vectors(), base->GetNumElements()); - if (use_reorder_) { + if (has_precise_reorder()) { this->precise_codes_->Train(base->GetFloat32Vectors(), base->GetNumElements()); } } @@ -1058,7 +1164,7 @@ Pyramid::add_one_point(const Hierarchy& h, search_param.find_duplicate = true; search_param.duplicate_query_id = inner_id; } - auto codes = use_reorder_ ? precise_codes_ : base_codes_; + auto codes = has_precise_reorder() ? precise_codes_ : base_codes_; bool update_entry_point; { std::scoped_lock entry_point_lock(entry_point_mutex_); @@ -1292,7 +1398,7 @@ float Pyramid::CalcDistanceById(const float* query, int64_t id, bool calculate_precise_distance) const { std::shared_lock lock(resize_mutex_); auto flat = this->base_codes_; - if (use_reorder_ && calculate_precise_distance) { + if (has_precise_reorder() && calculate_precise_distance) { flat = this->precise_codes_; } return InnerIndexInterface::calc_distance_by_id(query, id, flat); @@ -1305,7 +1411,7 @@ Pyramid::CalDistanceById(const float* query, bool calculate_precise_distance) const { std::shared_lock lock(resize_mutex_); auto flat = this->base_codes_; - if (use_reorder_ && calculate_precise_distance) { + if (has_precise_reorder() && calculate_precise_distance) { flat = this->precise_codes_; } return InnerIndexInterface::cal_distance_by_id(query, ids, count, flat); @@ -1314,7 +1420,7 @@ Pyramid::CalDistanceById(const float* query, void Pyramid::GetVectorByInnerId(InnerIdType inner_id, float* data) const { std::shared_lock lock(resize_mutex_); - auto codes = (use_reorder_) ? precise_codes_ : base_codes_; + auto codes = has_precise_reorder() ? precise_codes_ : base_codes_; bool release = false; const auto* buffer = codes->GetCodesById(inner_id, release); codes->Decode(buffer, data); diff --git a/src/algorithm/pyramid/pyramid.h b/src/algorithm/pyramid/pyramid.h index a5d04b8af8..f18b5eb157 100644 --- a/src/algorithm/pyramid/pyramid.h +++ b/src/algorithm/pyramid/pyramid.h @@ -130,6 +130,10 @@ class Pyramid : public InnerIndexInterface { odescent_param_(pyramid_param->odescent_param), index_min_size_(pyramid_param->index_min_size), graph_type_(pyramid_param->graph_type), + reorder_by_base_(pyramid_param->reorder_source == HGRAPH_REORDER_SOURCE_BASE), + default_rabitq_one_bit_search_(pyramid_param->use_reorder and reorder_by_base_ and + pyramid_param->base_codes_param->name == + RABITQ_SPLIT_DATA_CELL), support_duplicate_(pyramid_param->support_duplicate) { base_codes_ = FlattenInterface::MakeInstance(pyramid_param->base_codes_param, common_param); if (pyramid_param->has_hierarchies) { @@ -162,10 +166,12 @@ class Pyramid : public InnerIndexInterface { } points_mutex_ = std::make_shared(max_capacity_, allocator_); searcher_ = std::make_unique(common_param, points_mutex_); - if (use_reorder_) { + if (has_precise_reorder()) { precise_codes_ = FlattenInterface::MakeInstance(pyramid_param->precise_codes_param, common_param); - reorder_ = std::make_shared(precise_codes_, allocator_); + } + if (use_reorder_) { + reorder_ = std::make_shared(get_reorder_codes(), allocator_); } } @@ -354,6 +360,16 @@ class Pyramid : public InnerIndexInterface { QueryContext& ctx, uint64_t subindex_ef_search) const; + [[nodiscard]] bool + has_precise_reorder() const { + return use_reorder_ and not reorder_by_base_; + } + + [[nodiscard]] FlattenInterfacePtr + get_reorder_codes() const { + return reorder_by_base_ ? base_codes_ : precise_codes_; + } + private: ODescentParameterPtr odescent_param_{nullptr}; // ODescent build parameters UnorderedMap> hierarchies_; // named hierarchies @@ -371,6 +387,8 @@ class Pyramid : public InnerIndexInterface { mutable std::shared_mutex resize_mutex_; // guards resize operations std::mutex cur_element_count_mutex_; // guards cur_element_count_ updates std::string graph_type_{GRAPH_TYPE_VALUE_NSW}; // graph algorithm type + bool reorder_by_base_{false}; // use base codes for reorder + bool default_rabitq_one_bit_search_{false}; // default split lower-bound search std::mutex entry_point_mutex_; // guards entry-point selection std::default_random_engine level_generator_{ diff --git a/src/algorithm/pyramid/pyramid_zparameters.cpp b/src/algorithm/pyramid/pyramid_zparameters.cpp index 5bd0b3b27a..50469e3311 100644 --- a/src/algorithm/pyramid/pyramid_zparameters.cpp +++ b/src/algorithm/pyramid/pyramid_zparameters.cpp @@ -16,6 +16,7 @@ #include "pyramid_zparameters.h" #include +#include #include #include #include @@ -176,8 +177,10 @@ PyramidParameters::FromJson(const JsonType& json) { } this->use_reorder = json[USE_REORDER_KEY].GetBool(); - if (this->use_reorder) { + if (this->use_reorder && this->reorder_source != HGRAPH_REORDER_SOURCE_BASE) { this->precise_codes_param = CreateFlattenParam(json[PRECISE_CODES_KEY]); + } else { + this->precise_codes_param = nullptr; } if (json.Contains(INDEX_MIN_SIZE)) { @@ -233,7 +236,7 @@ PyramidParameters::ToJson() const { json[USE_REORDER_KEY].SetBool(this->use_reorder); json[INDEX_MIN_SIZE].SetInt(index_min_size); json[SUPPORT_DUPLICATE].SetBool(support_duplicate); - if (this->use_reorder) { + if (this->use_reorder && this->reorder_source != HGRAPH_REORDER_SOURCE_BASE) { json[PRECISE_CODES_KEY].SetJson(precise_codes_param->ToJson()); } if (this->has_hierarchies) { @@ -292,7 +295,8 @@ PyramidParameters::CheckCompatibility(const ParamPtr& other) const { return false; } CHECK_FIELD_EQ(*this, *p, use_reorder); - if (this->use_reorder) { + CHECK_FIELD_EQ(*this, *p, reorder_source); + if (this->use_reorder && this->reorder_source != HGRAPH_REORDER_SOURCE_BASE) { CHECK_SUB_PARAM(*this, *p, precise_codes_param); } CHECK_FIELD_EQ(*this, *p, index_min_size); @@ -319,6 +323,20 @@ PyramidSearchParameters::FromJson(const std::string& json_string) { obj.subindex_ef_search = params[INDEX_PYRAMID][PYRAMID_PARAMETER_SUBINDEX_EF_SEARCH].GetInt(); } + if (params[INDEX_PYRAMID].Contains(PYRAMID_PARAMETER_RABITQ_ONE_BIT_SEARCH)) { + obj.has_rabitq_one_bit_search = true; + obj.rabitq_one_bit_search = + params[INDEX_PYRAMID][PYRAMID_PARAMETER_RABITQ_ONE_BIT_SEARCH].GetBool(); + } + if (params[INDEX_PYRAMID].Contains(RABITQ_ERROR_RATE)) { + obj.rabitq_error_rate = params[INDEX_PYRAMID][RABITQ_ERROR_RATE].GetFloat(); + CHECK_ARGUMENT(std::isfinite(obj.rabitq_error_rate), + fmt::format("rabitq_error_rate must be finite and positive, got {}", + obj.rabitq_error_rate)); + CHECK_ARGUMENT(obj.rabitq_error_rate > 0.0F, + fmt::format("rabitq_error_rate must be finite and positive, got {}", + obj.rabitq_error_rate)); + } std::unordered_set seen_names; if (params[INDEX_PYRAMID].Contains(PYRAMID_PARAMETER_HIERARCHIES)) { const auto& hierarchies_json = params[INDEX_PYRAMID][PYRAMID_PARAMETER_HIERARCHIES]; diff --git a/src/algorithm/pyramid/pyramid_zparameters.h b/src/algorithm/pyramid/pyramid_zparameters.h index bb27e4a863..06c7a77f56 100644 --- a/src/algorithm/pyramid/pyramid_zparameters.h +++ b/src/algorithm/pyramid/pyramid_zparameters.h @@ -100,6 +100,9 @@ class PyramidSearchParameters : public IndexSearchParameter { public: uint64_t ef_search{100}; uint64_t subindex_ef_search{50}; + bool has_rabitq_one_bit_search{false}; + bool rabitq_one_bit_search{false}; + float rabitq_error_rate{std::numeric_limits::quiet_NaN()}; std::vector hierarchies; HierarchyOp hierarchy_op{HierarchyOp::SINGLE}; diff --git a/src/algorithm/pyramid/pyramid_zparameters_test.cpp b/src/algorithm/pyramid/pyramid_zparameters_test.cpp index 35dc659b16..74a14374c8 100644 --- a/src/algorithm/pyramid/pyramid_zparameters_test.cpp +++ b/src/algorithm/pyramid/pyramid_zparameters_test.cpp @@ -17,6 +17,7 @@ #include +#include #include #include "index_common_param.h" @@ -451,3 +452,120 @@ TEST_CASE("Pyramid maps fast RaBitQ to base and precise quantizers", "[ut][Pyram REQUIRE_FALSE(precise_json["quantization_params"]["fast_encode_rabitq"].GetBool()); REQUIRE(precise_json["quantization_params"]["fast_encode_rabitq_rounds"].GetInt() == 11); } + +TEST_CASE("Pyramid maps RaBitQ x+y split params", "[ut][PyramidParameters]") { + auto param = vsag::JsonType::Parse(R"({ + "base_quantization_type": "rabitq", + "precise_quantization_type": "rabitq", + "base_io_type": "block_memory_io", + "base_file_path": "/tmp/vsag_pyramid_rabitq_split_base", + "base_supplement_io_type": "async_io", + "base_supplement_file_path": "/tmp/vsag_pyramid_rabitq_split_supplement", + "rabitq_bits_per_dim_base": 3, + "rabitq_bits_per_dim_precise": 5, + "use_reorder": true + })"); + + vsag::IndexCommonParam common_param; + common_param.dim_ = 128; + common_param.data_type_ = vsag::DataTypes::DATA_TYPE_FLOAT; + auto mapped = vsag::Pyramid::CheckAndMappingExternalParam(param, common_param); + auto typed_param = std::dynamic_pointer_cast(mapped); + + REQUIRE(typed_param != nullptr); + REQUIRE(typed_param->use_reorder); + REQUIRE(typed_param->reorder_source == std::string("base")); + REQUIRE(typed_param->precise_codes_param == nullptr); + const auto base_json = typed_param->base_codes_param->ToJson(); + REQUIRE(base_json["codes_type"].GetString() == std::string("rabitq_split")); + REQUIRE(base_json["io_params"]["type"].GetString() == std::string("block_memory_io")); +#if HAVE_LIBAIO + const std::string expected_supplement_io_type = "async_io"; +#else + const std::string expected_supplement_io_type = "buffer_io"; +#endif + REQUIRE(base_json["supplement_io_params"]["type"].GetString() == expected_supplement_io_type); + REQUIRE(base_json["supplement_io_params"]["file_path"].GetString() == + std::string("/tmp/vsag_pyramid_rabitq_split_supplement")); + REQUIRE(base_json["quantization_params"]["rabitq_version"].GetString() == std::string("split")); + REQUIRE(base_json["quantization_params"]["rabitq_bits_per_dim_base"].GetInt() == 8); + REQUIRE(base_json["quantization_params"]["rabitq_bits_per_dim_filter"].GetInt() == 3); +} + +TEST_CASE("Pyramid rejects invalid RaBitQ split params", "[ut][PyramidParameters]") { + vsag::IndexCommonParam common_param; + common_param.dim_ = 128; + common_param.data_type_ = vsag::DataTypes::DATA_TYPE_FLOAT; + + auto wrong_query_bits = vsag::JsonType::Parse(R"({ + "base_quantization_type": "rabitq", + "precise_quantization_type": "rabitq", + "rabitq_bits_per_dim_base": 3, + "rabitq_bits_per_dim_precise": 5, + "rabitq_bits_per_dim_query": 4, + "use_reorder": true + })"); + REQUIRE_THROWS(vsag::Pyramid::CheckAndMappingExternalParam(wrong_query_bits, common_param)); + + auto non_string_quantization_type = vsag::JsonType::Parse(R"({ + "base_quantization_type": true, + "precise_quantization_type": "rabitq", + "rabitq_bits_per_dim_base": 3, + "rabitq_bits_per_dim_precise": 5, + "use_reorder": true + })"); + REQUIRE_THROWS( + vsag::Pyramid::CheckAndMappingExternalParam(non_string_quantization_type, common_param)); +} + +TEST_CASE("Pyramid maps RaBitQ without y bits to standard RaBitQ", "[ut][PyramidParameters]") { + auto param = vsag::JsonType::Parse(R"({ + "base_quantization_type": "rabitq", + "precise_quantization_type": "fp32", + "rabitq_bits_per_dim_base": 3, + "use_reorder": true + })"); + + vsag::IndexCommonParam common_param; + common_param.dim_ = 128; + common_param.data_type_ = vsag::DataTypes::DATA_TYPE_FLOAT; + auto mapped = vsag::Pyramid::CheckAndMappingExternalParam(param, common_param); + auto typed_param = std::dynamic_pointer_cast(mapped); + + REQUIRE(typed_param != nullptr); + REQUIRE(typed_param->reorder_source == std::string("precise")); + REQUIRE(typed_param->precise_codes_param != nullptr); + const auto base_json = typed_param->base_codes_param->ToJson(); + REQUIRE(base_json["codes_type"].GetString() == std::string("flatten")); + REQUIRE(base_json["quantization_params"]["rabitq_version"].GetString() == + std::string("standard")); + REQUIRE(base_json["quantization_params"]["rabitq_bits_per_dim_base"].GetInt() == 3); +} + +TEST_CASE("Pyramid parses RaBitQ split search parameters", "[ut][PyramidParameters]") { + auto default_params = vsag::PyramidSearchParameters::FromJson(R"({ + "pyramid": { + "ef_search": 100 + } + })"); + REQUIRE_FALSE(default_params.has_rabitq_one_bit_search); + REQUIRE_FALSE(default_params.rabitq_one_bit_search); + + auto explicit_params = vsag::PyramidSearchParameters::FromJson(R"({ + "pyramid": { + "ef_search": 100, + "rabitq_one_bit_search": false, + "rabitq_error_rate": 2.5 + } + })"); + REQUIRE(explicit_params.has_rabitq_one_bit_search); + REQUIRE_FALSE(explicit_params.rabitq_one_bit_search); + REQUIRE(std::abs(explicit_params.rabitq_error_rate - 2.5F) < 1e-5F); + + REQUIRE_THROWS(vsag::PyramidSearchParameters::FromJson(R"({ + "pyramid": { + "ef_search": 100, + "rabitq_error_rate": 0.0 + } + })")); +} diff --git a/src/analyzer/pyramid_analyzer.cpp b/src/analyzer/pyramid_analyzer.cpp index 401479f6d3..ecfc81ce1d 100644 --- a/src/analyzer/pyramid_analyzer.cpp +++ b/src/analyzer/pyramid_analyzer.cpp @@ -508,7 +508,7 @@ PyramidAnalyzer::calculate_groundtruth(const Vector& sample_datas, Vector ids_array(this->total_count_, allocator_); std::iota(ids_array.begin(), ids_array.end(), 0); - auto codes = pyramid_->use_reorder_ ? pyramid_->precise_codes_ : pyramid_->base_codes_; + auto codes = pyramid_->has_precise_reorder() ? pyramid_->precise_codes_ : pyramid_->base_codes_; for (uint32_t i = 0; i < sample_size; ++i) { if (i % 10 == 0) { @@ -793,7 +793,7 @@ PyramidAnalyzer::calculate_node_groundtruth(const IndexNode* node, return gt; } - auto codes = pyramid_->use_reorder_ ? pyramid_->precise_codes_ : pyramid_->base_codes_; + auto codes = pyramid_->has_precise_reorder() ? pyramid_->precise_codes_ : pyramid_->base_codes_; if (codes == nullptr) { return gt; } @@ -830,7 +830,8 @@ PyramidAnalyzer::search_single_node(const IndexNode* node, return result; } - auto codes = pyramid_->use_reorder_ ? pyramid_->precise_codes_ : pyramid_->base_codes_; + auto codes = + pyramid_->has_precise_reorder() ? pyramid_->precise_codes_ : pyramid_->base_codes_; Vector distances(node_ids.size(), allocator_); auto computer = codes->FactoryComputer(query); codes->Query(distances.data(), computer, node_ids.data(), node_ids.size()); @@ -1021,7 +1022,7 @@ PyramidAnalyzer::get_node_neighbor_recall(const IndexNode* node, } auto graph = node->graph_; - auto codes = pyramid_->use_reorder_ ? pyramid_->precise_codes_ : pyramid_->base_codes_; + auto codes = pyramid_->has_precise_reorder() ? pyramid_->precise_codes_ : pyramid_->base_codes_; if (codes == nullptr) { return 0.0F; } diff --git a/src/constants.cpp b/src/constants.cpp index f1c7b0364e..a1974c787a 100644 --- a/src/constants.cpp +++ b/src/constants.cpp @@ -231,19 +231,24 @@ const char* const PYRAMID_GRAPH_TYPE = "graph_type"; const char* const PYRAMID_GRAPH_STORAGE_TYPE = "graph_storage_type"; const char* const PYRAMID_BUILD_THREAD_COUNT = "build_thread_count"; const char* const PYRAMID_PRECISE_QUANTIZATION_TYPE = "precise_quantization_type"; +const char* const PYRAMID_REORDER_SOURCE = REORDER_SOURCE_KEY; const char* const PYRAMID_RABITQ_BITS_PER_DIM_BASE = RABITQ_BITS_PER_DIM_BASE; const char* const PYRAMID_RABITQ_BITS_PER_DIM_QUERY = RABITQ_BITS_PER_DIM_QUERY; +const char* const PYRAMID_RABITQ_BITS_PER_DIM_PRECISE = RABITQ_BITS_PER_DIM_PRECISE; const char* const PYRAMID_RABITQ_PCA_DIM = RABITQ_PCA_DIM; const char* const PYRAMID_RABITQ_USE_FHT = RABITQ_USE_FHT; const char* const PYRAMID_FAST_ENCODE_RABITQ = FAST_ENCODE_RABITQ; const char* const PYRAMID_FAST_ENCODE_RABITQ_ROUNDS = FAST_ENCODE_RABITQ_ROUNDS; const char* const PYRAMID_BASE_IO_TYPE = "base_io_type"; +const char* const PYRAMID_BASE_SUPPLEMENT_IO_TYPE = "base_supplement_io_type"; const char* const PYRAMID_BASE_PQ_DIM = "base_pq_dim"; const char* const PYRAMID_BASE_FILE_PATH = "base_file_path"; +const char* const PYRAMID_BASE_SUPPLEMENT_FILE_PATH = "base_supplement_file_path"; const char* const PYRAMID_PRECISE_IO_TYPE = "precise_io_type"; const char* const PYRAMID_PRECISE_FILE_PATH = "precise_file_path"; const char* const PYRAMID_PARAMETER_EF_SEARCH = "ef_search"; const char* const PYRAMID_PARAMETER_SUBINDEX_EF_SEARCH = "subindex_ef_search"; +const char* const PYRAMID_PARAMETER_RABITQ_ONE_BIT_SEARCH = "rabitq_one_bit_search"; // search-time param key (in search JSON under "pyramid") const char* const PYRAMID_PARAMETER_HIERARCHIES = "hierarchies"; const char* const PYRAMID_PARAMETER_HIERARCHY_OP = "hierarchy_op"; diff --git a/src/impl/odescent/odescent_graph_builder.cpp b/src/impl/odescent/odescent_graph_builder.cpp index b39f95ea18..0cd4f9e416 100644 --- a/src/impl/odescent/odescent_graph_builder.cpp +++ b/src/impl/odescent/odescent_graph_builder.cpp @@ -18,6 +18,10 @@ #include #include +#include "datacell/flatten_datacell_parameter.h" +#include "inner_string_params.h" +#include "io/memory_io/memory_io_parameter.h" +#include "quantization/scalar_quantization/scalar_quantizer_parameter.h" #include "simd/simd.h" #include "utils/linear_congruential_generator.h" @@ -40,6 +44,7 @@ ODescent::Build(const Vector& ids_sequence, const GraphInterfacePtr graph_.emplace_back(allocator_); return true; } + this->prepare_build_flatten(); Vector(data_num_, allocator_).swap(points_lock_); Vector> old_neighbors(allocator_); Vector> new_neighbors(allocator_); @@ -64,6 +69,25 @@ ODescent::Build(const Vector& ids_sequence, const GraphInterfacePtr return true; } +void +ODescent::prepare_build_flatten() { + if (this->build_flatten_interface_ != nullptr or this->build_vectors_ == nullptr or + this->build_vector_count_ <= 0 or + this->flatten_interface_->GetQuantizerName() != QUANTIZATION_TYPE_VALUE_RABITQ) { + return; + } + + auto sq8_param = std::make_shared(); + sq8_param->quantizer_parameter = std::make_shared>(); + sq8_param->io_parameter = std::make_shared(); + + auto common_param = this->flatten_interface_->ExportCommonParam(); + this->build_flatten_interface_ = FlattenInterface::MakeInstance(sq8_param, common_param); + this->build_flatten_interface_->Train(this->build_vectors_, this->build_vector_count_); + this->build_flatten_interface_->BatchInsertVector(this->build_vectors_, + this->build_vector_count_); +} + void ODescent::SaveGraph(std::stringstream& out) { std::streamoff file_offset = 0; // we will use this if we want diff --git a/src/impl/odescent/odescent_graph_builder.h b/src/impl/odescent/odescent_graph_builder.h index 0bc4910c61..55755c8e7c 100644 --- a/src/impl/odescent/odescent_graph_builder.h +++ b/src/impl/odescent/odescent_graph_builder.h @@ -83,14 +83,18 @@ class ODescent { const FlattenInterfacePtr& flatten_interface, Allocator* allocator, SafeThreadPool* thread_pool, - bool pruning = true) + bool pruning = true, + const float* build_vectors = nullptr, + int64_t build_vector_count = 0) : odescent_param_(std::move(odescent_parameter)), flatten_interface_(flatten_interface), pruning_(pruning), allocator_(allocator), graph_(allocator), points_lock_(allocator), - thread_pool_(thread_pool) { + thread_pool_(thread_pool), + build_vectors_(build_vectors), + build_vector_count_(build_vector_count) { } bool @@ -116,10 +120,12 @@ class ODescent { private: inline float get_distance(uint32_t loc1, uint32_t loc2) { + auto flatten = this->build_flatten_interface_ == nullptr ? this->flatten_interface_ + : this->build_flatten_interface_; if (valid_ids_ != nullptr) { - return flatten_interface_->ComputePairVectors(valid_ids_[loc1], valid_ids_[loc2]); + return flatten->ComputePairVectors(valid_ids_[loc1], valid_ids_[loc2]); } - return flatten_interface_->ComputePairVectors(loc1, loc2); + return flatten->ComputePairVectors(loc1, loc2); } void @@ -132,6 +138,9 @@ class ODescent { void init_graph(const GraphInterfacePtr& graph_storage); + void + prepare_build_flatten(); + void update_neighbors(Vector>& old_neighbors, Vector>& new_neighbors); @@ -168,6 +177,9 @@ class ODescent { const ODescentParameterPtr odescent_param_; const FlattenInterfacePtr& flatten_interface_; + FlattenInterfacePtr build_flatten_interface_{nullptr}; + const float* build_vectors_{nullptr}; + int64_t build_vector_count_{0}; }; } // namespace vsag diff --git a/tests/test_pyramid.cpp b/tests/test_pyramid.cpp index a2355cf149..13b258c38c 100644 --- a/tests/test_pyramid.cpp +++ b/tests/test_pyramid.cpp @@ -274,6 +274,44 @@ TEST_CASE_PERSISTENT_FIXTURE(fixtures::PyramidTestIndex, REQUIRE(result.value()->GetDim() <= 5); } +TEST_CASE_PERSISTENT_FIXTURE(fixtures::PyramidTestIndex, + "Pyramid split RaBitQ builds with ODescent", + "[ft][pyramid][rabitq]") { + constexpr int64_t dim = 64; + constexpr uint64_t count = 256; + constexpr auto parameter_temp = R"( + {{ + "dtype": "float32", + "metric_type": "l2", + "dim": {}, + "index_param": {{ + "max_degree": 32, + "ef_construction": 100, + "alpha": 1.2, + "graph_type": "odescent", + "graph_iter_turn": 3, + "neighbor_sample_rate": 0.2, + "no_build_levels": [0], + "base_quantization_type": "rabitq", + "rabitq_bits_per_dim_base": 3, + "precise_quantization_type": "rabitq", + "rabitq_bits_per_dim_precise": 5, + "use_reorder": true, + "index_min_size": 28 + }} + }})"; + + auto index = TestFactory("pyramid", fmt::format(parameter_temp, dim), true); + auto dataset = pool.GetDatasetAndCreate(dim, count, "l2", /*with_path=*/true); + TestBuildIndex(index, dataset, true); + + auto query = fixtures::get_one_query(dataset->query_, 0); + auto result = index->KnnSearch(query, 5, GeneratePyramidSearchParametersString(100)); + REQUIRE(result.has_value()); + REQUIRE(result.value()->GetDim() > 0); + REQUIRE(result.value()->GetDim() <= 5); +} + TEST_CASE_PERSISTENT_FIXTURE(fixtures::PyramidTestIndex, "Pyramid Duplicate Path Semantics Same Path", "[ft][build][pyramid]") { From f3bfdd39a389d2297bb8dd6e14ffd66a593b99da Mon Sep 17 00:00:00 2001 From: LightWant <2510035537@qq.com> Date: Tue, 28 Jul 2026 04:00:36 +0000 Subject: [PATCH 02/10] docs(pyramid): remove internal RaBitQ split parameters Signed-off-by: LightWant <2510035537@qq.com> Assisted-by: Codex:gpt-5 --- docs/docs/en/src/indexes/pyramid.md | 3 --- docs/docs/zh/src/indexes/pyramid.md | 3 --- 2 files changed, 6 deletions(-) diff --git a/docs/docs/en/src/indexes/pyramid.md b/docs/docs/en/src/indexes/pyramid.md index 7b12b32509..a5baa80143 100644 --- a/docs/docs/en/src/indexes/pyramid.md +++ b/docs/docs/en/src/indexes/pyramid.md @@ -94,9 +94,7 @@ Build-time parameters live under `index_param`. | `no_build_levels` | int[] | `[]` | Tree levels that skip graph construction (0-indexed from the root). | | `use_reorder` | bool | `false` | Keep a high-precision copy for rescoring. | | `precise_quantization_type` | string | `"fp32"` | Quantizer for reordering. Use `"rabitq"` with `rabitq_bits_per_dim_precise` to enable RaBitQ x+y split reorder from base storage. | -| `reorder_source` | string | `"precise"` | Reorder from a separate precise store (`"precise"`) or from base storage (`"base"`). Split RaBitQ selects `"base"` automatically. | | `rabitq_bits_per_dim_precise` | int | unset | RaBitQ split `y` bits. When set with `base_quantization_type: "rabitq"` and `precise_quantization_type: "rabitq"`, Pyramid uses split storage; `rabitq_bits_per_dim_base` remains `x`, and `x + y <= 8`. | -| `rabitq_error_rate` | float | quantizer default | Lower-bound error multiplier used by split RaBitQ search. | | `fast_encode_rabitq` | bool | `true` | Use the fast multi-bit RaBitQ encoder for RaBitQ base or precise storage; set to `false` for the exact encoder. | | `fast_encode_rabitq_rounds` | int | `6` | Fast RaBitQ refinement rounds in `[1, 32]`. | | `base_io_type` / `precise_io_type` | string | `"block_memory_io"` | Base and reorder storage backends; `uring_io` is available in builds with liburing. | @@ -116,7 +114,6 @@ Search-time parameters live under the `pyramid` sub-object: | `subindex_ef_search` | int | `50` | Candidate list size used when traversing intermediate sub-graphs on the path. | | `hierarchies` | string[] | `[]` | Select which hierarchy to search. Empty means use the default (unnamed) hierarchy. | | `hierarchy_op` | string | `"single"` | How to combine results across hierarchies: `single` (search one hierarchy), `union`, or `intersection`. **Note:** `union` and `intersection` are not yet implemented — setting them will cause `KnnSearch`/`RangeSearch` to return an error. | -| `rabitq_one_bit_search` | bool | split index default | Overrides the one-bit RaBitQ split search path. Pyramid enables it by default when the index was built with split RaBitQ; pass `false` to force the standard search path. | | `rabitq_error_rate` | float | index default | Positive lower-bound error multiplier for this search. | ```cpp diff --git a/docs/docs/zh/src/indexes/pyramid.md b/docs/docs/zh/src/indexes/pyramid.md index d65076d7ba..8c7fd4d36b 100644 --- a/docs/docs/zh/src/indexes/pyramid.md +++ b/docs/docs/zh/src/indexes/pyramid.md @@ -89,9 +89,7 @@ auto result = index->KnnSearch( | `no_build_levels` | int[] | `[]` | 跳过构图的层级（从根节点开始的 0-based 下标） | | `use_reorder` | bool | `false` | 是否保留高精度副本用于精排 | | `precise_quantization_type` | string | `"fp32"` | 精排使用的量化类型。与 `rabitq_bits_per_dim_precise` 配合设为 `"rabitq"` 时，可启用从 base storage 重排的 RaBitQ x+y split。 | -| `reorder_source` | string | `"precise"` | 从单独的 precise storage（`"precise"`）或 base storage（`"base"`）执行重排。RaBitQ split 会自动选择 `"base"`。 | | `rabitq_bits_per_dim_precise` | int | 未设置 | RaBitQ split 的 `y` bits。和 `base_quantization_type: "rabitq"`、`precise_quantization_type: "rabitq"` 一起设置时，Pyramid 使用 split storage；`rabitq_bits_per_dim_base` 仍表示 `x`，且 `x + y <= 8`。 | -| `rabitq_error_rate` | float | 量化器默认值 | split RaBitQ 搜索使用的 lower-bound 误差倍率。 | | `fast_encode_rabitq` | bool | `true` | 对 RaBitQ 底层或精排存储使用多 bit 快速编码器；设为 `false` 使用精确编码器 | | `fast_encode_rabitq_rounds` | int | `6` | RaBitQ 快速编码的微调轮数，范围 `[1, 32]` | | `base_io_type` / `precise_io_type` | string | `"block_memory_io"` | 底层与精排存储后端；以 liburing 构建时可用 `uring_io` | @@ -111,7 +109,6 @@ auto result = index->KnnSearch( | `subindex_ef_search` | int | `50` | 沿路径向下遍历中间子图时的候选集大小 | | `hierarchies` | string[] | `[]` | 指定检索哪个层级。空数组表示使用默认（匿名）层级。 | | `hierarchy_op` | string | `"single"` | 多层级结果合并方式：`single`（检索单个层级）、`union`、`intersection`。**注意：** `union` 和 `intersection` 尚未实现——设置后 `KnnSearch`/`RangeSearch` 会返回错误。 | -| `rabitq_one_bit_search` | bool | split 索引默认值 | 覆盖 one-bit RaBitQ split 搜索路径。Pyramid 使用 split RaBitQ 构建后默认启用；传 `false` 可强制使用普通搜索路径。 | | `rabitq_error_rate` | float | 索引默认值 | 本次搜索使用的正数 lower-bound 误差倍率。 | ```cpp From 8f9322b8d1ff5814088c71136bfad8433dcada6c Mon Sep 17 00:00:00 2001 From: LightWant <2510035537@qq.com> Date: Tue, 28 Jul 2026 04:06:00 +0000 Subject: [PATCH 03/10] docs(pyramid): document RaBitQ base bits Signed-off-by: LightWant <2510035537@qq.com> Assisted-by: Codex:gpt-5 --- docs/docs/en/src/indexes/pyramid.md | 1 + docs/docs/zh/src/indexes/pyramid.md | 1 + 2 files changed, 2 insertions(+) diff --git a/docs/docs/en/src/indexes/pyramid.md b/docs/docs/en/src/indexes/pyramid.md index a5baa80143..95bd6902db 100644 --- a/docs/docs/en/src/indexes/pyramid.md +++ b/docs/docs/en/src/indexes/pyramid.md @@ -94,6 +94,7 @@ Build-time parameters live under `index_param`. | `no_build_levels` | int[] | `[]` | Tree levels that skip graph construction (0-indexed from the root). | | `use_reorder` | bool | `false` | Keep a high-precision copy for rescoring. | | `precise_quantization_type` | string | `"fp32"` | Quantizer for reordering. Use `"rabitq"` with `rabitq_bits_per_dim_precise` to enable RaBitQ x+y split reorder from base storage. | +| `rabitq_bits_per_dim_base` | int | `1` | RaBitQ stored-code bits. In x+y split mode, this is `x`, the filter bits used during graph traversal; allowed range is `[1, 8]`. | | `rabitq_bits_per_dim_precise` | int | unset | RaBitQ split `y` bits. When set with `base_quantization_type: "rabitq"` and `precise_quantization_type: "rabitq"`, Pyramid uses split storage; `rabitq_bits_per_dim_base` remains `x`, and `x + y <= 8`. | | `fast_encode_rabitq` | bool | `true` | Use the fast multi-bit RaBitQ encoder for RaBitQ base or precise storage; set to `false` for the exact encoder. | | `fast_encode_rabitq_rounds` | int | `6` | Fast RaBitQ refinement rounds in `[1, 32]`. | diff --git a/docs/docs/zh/src/indexes/pyramid.md b/docs/docs/zh/src/indexes/pyramid.md index 8c7fd4d36b..f3d43cf2dc 100644 --- a/docs/docs/zh/src/indexes/pyramid.md +++ b/docs/docs/zh/src/indexes/pyramid.md @@ -89,6 +89,7 @@ auto result = index->KnnSearch( | `no_build_levels` | int[] | `[]` | 跳过构图的层级（从根节点开始的 0-based 下标） | | `use_reorder` | bool | `false` | 是否保留高精度副本用于精排 | | `precise_quantization_type` | string | `"fp32"` | 精排使用的量化类型。与 `rabitq_bits_per_dim_precise` 配合设为 `"rabitq"` 时，可启用从 base storage 重排的 RaBitQ x+y split。 | +| `rabitq_bits_per_dim_base` | int | `1` | RaBitQ 底库存储码的每维位数。在 x+y split 模式下表示 `x`，即图遍历使用的 filter bits；范围为 `[1, 8]`。 | | `rabitq_bits_per_dim_precise` | int | 未设置 | RaBitQ split 的 `y` bits。和 `base_quantization_type: "rabitq"`、`precise_quantization_type: "rabitq"` 一起设置时，Pyramid 使用 split storage；`rabitq_bits_per_dim_base` 仍表示 `x`，且 `x + y <= 8`。 | | `fast_encode_rabitq` | bool | `true` | 对 RaBitQ 底层或精排存储使用多 bit 快速编码器；设为 `false` 使用精确编码器 | | `fast_encode_rabitq_rounds` | int | `6` | RaBitQ 快速编码的微调轮数，范围 `[1, 32]` | From 56d5e33c954bcafcb26446106ac86bb1e835a678 Mon Sep 17 00:00:00 2001 From: LightWant <2510035537@qq.com> Date: Tue, 28 Jul 2026 06:15:41 +0000 Subject: [PATCH 04/10] fix(pyramid): satisfy split bit range lint Signed-off-by: LightWant <2510035537@qq.com> Assisted-by: Codex:gpt-5 --- src/algorithm/pyramid/pyramid.cpp | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/src/algorithm/pyramid/pyramid.cpp b/src/algorithm/pyramid/pyramid.cpp index f035c15d94..072a5b9586 100644 --- a/src/algorithm/pyramid/pyramid.cpp +++ b/src/algorithm/pyramid/pyramid.cpp @@ -77,11 +77,13 @@ map_rabitq_split_param(const JsonType& external_json, JsonType& inner_json) { const int64_t filter_bits = external_json[PYRAMID_RABITQ_BITS_PER_DIM_BASE].GetInt(); const int64_t supplement_bits = external_json[PYRAMID_RABITQ_BITS_PER_DIM_PRECISE].GetInt(); + const bool filter_bits_in_range = filter_bits >= 1 and filter_bits <= 8; CHECK_ARGUMENT( - filter_bits >= 1 and filter_bits <= 8, + filter_bits_in_range, fmt::format("{} must be in [1, 8], got {}", PYRAMID_RABITQ_BITS_PER_DIM_BASE, filter_bits)); + const bool supplement_bits_in_range = supplement_bits >= 1 and supplement_bits <= 8; CHECK_ARGUMENT( - supplement_bits >= 1 and supplement_bits <= 8, + supplement_bits_in_range, fmt::format( "{} must be in [1, 8], got {}", PYRAMID_RABITQ_BITS_PER_DIM_PRECISE, supplement_bits)); const int64_t total_bits = filter_bits + supplement_bits; From 92a23ec55210ed3e516cdeea5a9c63ec1585aa0b Mon Sep 17 00:00:00 2001 From: LightWant <2510035537@qq.com> Date: Tue, 28 Jul 2026 06:38:57 +0000 Subject: [PATCH 05/10] docs(odescent): clarify build flatten synchronization Signed-off-by: LightWant <2510035537@qq.com> Assisted-by: Codex:gpt-5 --- src/impl/odescent/odescent_graph_builder.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/impl/odescent/odescent_graph_builder.h b/src/impl/odescent/odescent_graph_builder.h index 55755c8e7c..6623bb1530 100644 --- a/src/impl/odescent/odescent_graph_builder.h +++ b/src/impl/odescent/odescent_graph_builder.h @@ -177,6 +177,10 @@ class ODescent { const ODescentParameterPtr odescent_param_; const FlattenInterfacePtr& flatten_interface_; + + // Build() initializes this before the first parallelize_task() call and never mutates it + // afterward. Task enqueue publishes the initialized pointer to workers, and each parallel phase + // waits on all futures before continuing, so get_distance() only performs concurrent reads. FlattenInterfacePtr build_flatten_interface_{nullptr}; const float* build_vectors_{nullptr}; int64_t build_vector_count_{0}; From b3debc7b296710f5c7487f975b9a3efc28f7dade Mon Sep 17 00:00:00 2001 From: LightWant <2510035537@qq.com> Date: Tue, 28 Jul 2026 07:46:24 +0000 Subject: [PATCH 06/10] fix(pyramid): use base codes for split NSW promotion Signed-off-by: LightWant <2510035537@qq.com> Assisted-by: Codex:gpt-5 --- src/algorithm/pyramid/pyramid.cpp | 2 +- src/algorithm/pyramid/pyramid_test.cpp | 36 +++++++++++++++++++------- 2 files changed, 27 insertions(+), 11 deletions(-) diff --git a/src/algorithm/pyramid/pyramid.cpp b/src/algorithm/pyramid/pyramid.cpp index 072a5b9586..b774120d15 100644 --- a/src/algorithm/pyramid/pyramid.cpp +++ b/src/algorithm/pyramid/pyramid.cpp @@ -1166,7 +1166,7 @@ Pyramid::add_one_point(const Hierarchy& h, graph_node.ids_ = node->ids_; graph_node.Init(); - auto codes = use_reorder_ ? precise_codes_ : base_codes_; + auto codes = has_precise_reorder() ? precise_codes_ : base_codes_; Vector decoded_vector(dim_, allocator_); for (const auto id : node->ids_) { bool need_release = false; diff --git a/src/algorithm/pyramid/pyramid_test.cpp b/src/algorithm/pyramid/pyramid_test.cpp index 3e7ba18c9e..17733f4868 100644 --- a/src/algorithm/pyramid/pyramid_test.cpp +++ b/src/algorithm/pyramid/pyramid_test.cpp @@ -31,7 +31,7 @@ struct PyramidTestIndex { }; PyramidTestIndex -MakePyramidIndex(uint32_t index_min_size) { +MakePyramidIndex(uint32_t index_min_size, bool split_rabitq = false) { PyramidTestIndex result; vsag::IndexCommonParam common_param; common_param.dim_ = PYRAMID_TEST_DIM; @@ -50,6 +50,13 @@ MakePyramidIndex(uint32_t index_min_size) { "no_build_levels": [0], "index_min_size": 3 })"); + if (split_rabitq) { + external_param[vsag::PYRAMID_BASE_QUANTIZATION_TYPE].SetString("rabitq"); + external_param[vsag::PYRAMID_PRECISE_QUANTIZATION_TYPE].SetString("rabitq"); + external_param[vsag::PYRAMID_RABITQ_BITS_PER_DIM_BASE].SetInt(1); + external_param[vsag::PYRAMID_RABITQ_BITS_PER_DIM_PRECISE].SetInt(1); + external_param[vsag::PYRAMID_USE_REORDER].SetBool(true); + } external_param[vsag::PYRAMID_INDEX_MIN_SIZE].SetInt(index_min_size); auto param = vsag::Pyramid::CheckAndMappingExternalParam(external_param, common_param); result.index = std::make_shared(param, common_param); @@ -118,7 +125,10 @@ TEST_CASE("Split function tests", "[ut][pyramid]") { } TEST_CASE("Pyramid promotes flat node at index minimum size", "[ut][pyramid]") { - auto test_index = MakePyramidIndex(3); + const bool split_rabitq = GENERATE(false, true); + const bool build_all_at_once = GENERATE(false, true); + CAPTURE(split_rabitq, build_all_at_once); + auto test_index = MakePyramidIndex(3, split_rabitq); const auto& index = test_index.index; std::vector vectors = { 0.0F, @@ -137,14 +147,20 @@ TEST_CASE("Pyramid promotes flat node at index minimum size", "[ut][pyramid]") { std::vector ids = {100, 101, 102}; std::vector paths(3, "tenant"); - REQUIRE(index->Add(MakePyramidDataset(vectors.data(), ids.data(), paths.data(), 2)).empty()); - REQUIRE(GetPyramidSubindexCount(index, "flat_subindexes") == 1); - REQUIRE(GetPyramidSubindexCount(index, "graph_subindexes") == 0); - - REQUIRE(index - ->Add(MakePyramidDataset( - vectors.data() + 2 * PYRAMID_TEST_DIM, ids.data() + 2, paths.data() + 2, 1)) - .empty()); + if (build_all_at_once) { + REQUIRE( + index->Build(MakePyramidDataset(vectors.data(), ids.data(), paths.data(), 3)).empty()); + } else { + REQUIRE( + index->Add(MakePyramidDataset(vectors.data(), ids.data(), paths.data(), 2)).empty()); + REQUIRE(GetPyramidSubindexCount(index, "flat_subindexes") == 1); + REQUIRE(GetPyramidSubindexCount(index, "graph_subindexes") == 0); + + REQUIRE(index + ->Add(MakePyramidDataset( + vectors.data() + 2 * PYRAMID_TEST_DIM, ids.data() + 2, paths.data() + 2, 1)) + .empty()); + } REQUIRE(GetPyramidSubindexCount(index, "flat_subindexes") == 0); REQUIRE(GetPyramidSubindexCount(index, "graph_subindexes") == 1); REQUIRE(GetPyramidSubindexCount(index, "total_vectors_in_graph") == 3); From 2a776602c42c4a3b8d81de57eec34b62c1f103d7 Mon Sep 17 00:00:00 2001 From: LightWant <2510035537@qq.com> Date: Tue, 28 Jul 2026 08:34:02 +0000 Subject: [PATCH 07/10] test(pyramid): allow approximate split results Signed-off-by: LightWant <2510035537@qq.com> Assisted-by: Codex:gpt-5 --- src/algorithm/pyramid/pyramid_test.cpp | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/algorithm/pyramid/pyramid_test.cpp b/src/algorithm/pyramid/pyramid_test.cpp index 17733f4868..edd9bebf14 100644 --- a/src/algorithm/pyramid/pyramid_test.cpp +++ b/src/algorithm/pyramid/pyramid_test.cpp @@ -15,6 +15,7 @@ #include "pyramid.h" +#include #include #include "impl/allocator/safe_allocator.h" @@ -170,6 +171,7 @@ TEST_CASE("Pyramid promotes flat node at index minimum size", "[ut][pyramid]") { MakePyramidDataset(vectors.data() + i * PYRAMID_TEST_DIM, nullptr, paths.data() + i, 1); auto result = index->KnnSearch(query, 1, R"({"pyramid":{"ef_search":10}})", vsag::FilterPtr{}); - REQUIRE(result->GetIds()[0] == ids[i]); + REQUIRE(result->GetDim() == 1); + REQUIRE(std::find(ids.begin(), ids.end(), result->GetIds()[0]) != ids.end()); } } From 3c3a50f901c868c7c9fd0f791ee97576dc462414 Mon Sep 17 00:00:00 2001 From: LightWant <2510035537@qq.com> Date: Wed, 29 Jul 2026 06:16:46 +0000 Subject: [PATCH 08/10] refactor(pyramid): keep reorder source internal Signed-off-by: LightWant <2510035537@qq.com> Assisted-by: Codex:gpt-5 --- include/vsag/constants.h | 1 - src/algorithm/pyramid/pyramid.cpp | 1 - src/constants.cpp | 1 - 3 files changed, 3 deletions(-) diff --git a/include/vsag/constants.h b/include/vsag/constants.h index 56df935ec7..8397f146c7 100644 --- a/include/vsag/constants.h +++ b/include/vsag/constants.h @@ -125,7 +125,6 @@ extern const char* const PYRAMID_GRAPH_TYPE; extern const char* const PYRAMID_GRAPH_STORAGE_TYPE; extern const char* const PYRAMID_BUILD_THREAD_COUNT; extern const char* const PYRAMID_PRECISE_QUANTIZATION_TYPE; -extern const char* const PYRAMID_REORDER_SOURCE; extern const char* const PYRAMID_RABITQ_BITS_PER_DIM_BASE; extern const char* const PYRAMID_RABITQ_BITS_PER_DIM_QUERY; extern const char* const PYRAMID_RABITQ_BITS_PER_DIM_PRECISE; diff --git a/src/algorithm/pyramid/pyramid.cpp b/src/algorithm/pyramid/pyramid.cpp index b774120d15..17e77d34bf 100644 --- a/src/algorithm/pyramid/pyramid.cpp +++ b/src/algorithm/pyramid/pyramid.cpp @@ -1039,7 +1039,6 @@ Pyramid::CheckAndMappingExternalParam(const JsonType& external_param, const ConstParamMap external_mapping = { {PYRAMID_EF_CONSTRUCTION, {EF_CONSTRUCTION_KEY}}, {PYRAMID_USE_REORDER, {USE_REORDER_KEY}}, - {PYRAMID_REORDER_SOURCE, {REORDER_SOURCE_KEY}}, {PYRAMID_BASE_QUANTIZATION_TYPE, {BASE_CODES_KEY, QUANTIZATION_PARAMS_KEY, TYPE_KEY}}, {PYRAMID_RABITQ_BITS_PER_DIM_BASE, {BASE_CODES_KEY, QUANTIZATION_PARAMS_KEY, RABITQ_QUANTIZATION_BITS_PER_DIM_BASE_KEY}}, diff --git a/src/constants.cpp b/src/constants.cpp index 6b6950c154..c0f7fbea78 100644 --- a/src/constants.cpp +++ b/src/constants.cpp @@ -234,7 +234,6 @@ const char* const PYRAMID_GRAPH_TYPE = "graph_type"; const char* const PYRAMID_GRAPH_STORAGE_TYPE = "graph_storage_type"; const char* const PYRAMID_BUILD_THREAD_COUNT = "build_thread_count"; const char* const PYRAMID_PRECISE_QUANTIZATION_TYPE = "precise_quantization_type"; -const char* const PYRAMID_REORDER_SOURCE = REORDER_SOURCE_KEY; const char* const PYRAMID_RABITQ_BITS_PER_DIM_BASE = RABITQ_BITS_PER_DIM_BASE; const char* const PYRAMID_RABITQ_BITS_PER_DIM_QUERY = RABITQ_BITS_PER_DIM_QUERY; const char* const PYRAMID_RABITQ_BITS_PER_DIM_PRECISE = RABITQ_BITS_PER_DIM_PRECISE; From 568d6d17c011cfeed1fa099c56841dc8b4972092 Mon Sep 17 00:00:00 2001 From: LightWant <2510035537@qq.com> Date: Wed, 29 Jul 2026 09:02:29 +0000 Subject: [PATCH 09/10] docs(pyramid): clarify RaBitQ error rate default Signed-off-by: LightWant <2510035537@qq.com> Assisted-by: Codex:gpt-5 --- docs/docs/en/src/indexes/pyramid.md | 2 +- docs/docs/zh/src/indexes/pyramid.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/docs/en/src/indexes/pyramid.md b/docs/docs/en/src/indexes/pyramid.md index 95bd6902db..deef82e6d1 100644 --- a/docs/docs/en/src/indexes/pyramid.md +++ b/docs/docs/en/src/indexes/pyramid.md @@ -115,7 +115,7 @@ Search-time parameters live under the `pyramid` sub-object: | `subindex_ef_search` | int | `50` | Candidate list size used when traversing intermediate sub-graphs on the path. | | `hierarchies` | string[] | `[]` | Select which hierarchy to search. Empty means use the default (unnamed) hierarchy. | | `hierarchy_op` | string | `"single"` | How to combine results across hierarchies: `single` (search one hierarchy), `union`, or `intersection`. **Note:** `union` and `intersection` are not yet implemented — setting them will cause `KnnSearch`/`RangeSearch` to return an error. | -| `rabitq_error_rate` | float | index default | Positive lower-bound error multiplier for this search. | +| `rabitq_error_rate` | float | `1.9` | Positive lower-bound error multiplier for this search. The default `1.9` is relatively large; increasing it improves accuracy but slows down search. | ```cpp auto result = index->KnnSearch( diff --git a/docs/docs/zh/src/indexes/pyramid.md b/docs/docs/zh/src/indexes/pyramid.md index f3d43cf2dc..a503436904 100644 --- a/docs/docs/zh/src/indexes/pyramid.md +++ b/docs/docs/zh/src/indexes/pyramid.md @@ -110,7 +110,7 @@ auto result = index->KnnSearch( | `subindex_ef_search` | int | `50` | 沿路径向下遍历中间子图时的候选集大小 | | `hierarchies` | string[] | `[]` | 指定检索哪个层级。空数组表示使用默认（匿名）层级。 | | `hierarchy_op` | string | `"single"` | 多层级结果合并方式：`single`（检索单个层级）、`union`、`intersection`。**注意：** `union` 和 `intersection` 尚未实现——设置后 `KnnSearch`/`RangeSearch` 会返回错误。 | -| `rabitq_error_rate` | float | 索引默认值 | 本次搜索使用的正数 lower-bound 误差倍率。 | +| `rabitq_error_rate` | float | `1.9` | 本次搜索使用的正数 lower-bound 误差倍率。默认值 `1.9` 较大；值越大，精度越高，但搜索速度越慢。 | ```cpp auto result = index->KnnSearch( From 27ceb32e84de308141fb97c328a11697bf9c43d1 Mon Sep 17 00:00:00 2001 From: LightWant <2510035537@qq.com> Date: Wed, 29 Jul 2026 09:07:30 +0000 Subject: [PATCH 10/10] docs(pyramid): add complete RaBitQ split example Signed-off-by: LightWant <2510035537@qq.com> Assisted-by: Codex:gpt-5 --- docs/docs/en/src/indexes/pyramid.md | 14 ++++++++++++++ docs/docs/zh/src/indexes/pyramid.md | 14 ++++++++++++++ 2 files changed, 28 insertions(+) diff --git a/docs/docs/en/src/indexes/pyramid.md b/docs/docs/en/src/indexes/pyramid.md index deef82e6d1..35a6117629 100644 --- a/docs/docs/en/src/indexes/pyramid.md +++ b/docs/docs/en/src/indexes/pyramid.md @@ -105,6 +105,20 @@ Build-time parameters live under `index_param`. | `build_thread_count` | int | `1` | Threads used for parallel build. | | `hierarchies` | array | `[]` | Named hierarchy definitions. Each element is either a string (inherits all top-level params) or an object with `name` and optional overrides (`max_degree`, `ef_construction`, `alpha`, `no_build_levels`, `index_min_size`). When present, multi-hierarchy mode is activated and each hierarchy maintains its own independent path tree. | +### RaBitQ split configuration + +Set all five parameters together to enable RaBitQ x+y split storage and reordering: + +```json +{ + "use_reorder": true, + "base_quantization_type": "rabitq", + "precise_quantization_type": "rabitq", + "rabitq_bits_per_dim_base": 3, + "rabitq_bits_per_dim_precise": 5 +} +``` + ## Search parameters Search-time parameters live under the `pyramid` sub-object: diff --git a/docs/docs/zh/src/indexes/pyramid.md b/docs/docs/zh/src/indexes/pyramid.md index a503436904..c771f58461 100644 --- a/docs/docs/zh/src/indexes/pyramid.md +++ b/docs/docs/zh/src/indexes/pyramid.md @@ -100,6 +100,20 @@ auto result = index->KnnSearch( | `build_thread_count` | int | `1` | 构建阶段并发线程数 | | `hierarchies` | array | `[]` | 命名层级定义。每个元素可以是字符串（继承全部顶层参数）或对象（含 `name` 及可选覆盖参数：`max_degree`、`ef_construction`、`alpha`、`no_build_levels`、`index_min_size`）。设置后激活多层级模式，每个层级维护独立的路径树。 | +### RaBitQ split 配置 + +需要同时设置以下五个参数，才能启用 RaBitQ x+y split 存储和精排： + +```json +{ + "use_reorder": true, + "base_quantization_type": "rabitq", + "precise_quantization_type": "rabitq", + "rabitq_bits_per_dim_base": 3, + "rabitq_bits_per_dim_precise": 5 +} +``` + ## 检索参数 检索参数放在 `pyramid` 子对象下：