Skip to content

feat(sindi): implement value-aware term pruning #2542

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
Roxanne0321 wants to merge 6 commits into
base: main
Choose a base branch
from
Draft

feat(sindi): implement value-aware term pruning #2542

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
5a0d7ec
feat(sindi): implement value-aware term pruning
Roxanne0321 Jul 27, 2026
bab5797
refactor(sparse): restore transform input behavior
Roxanne0321 Jul 31, 2026
9cc1392
refactor(sindi): sort all affected postings
Roxanne0321 Jul 31, 2026
07223eb
feat(sindi): allow prune ratios below one
Roxanne0321 Jul 31, 2026
155d727
refactor(sindi): rename term retain threshold
Roxanne0321 Jul 31, 2026
2cb6367
refactor(sindi): simplify sparse fixtures and posting sorting
Roxanne0321 Jul 31, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 0 additions & 11 deletions benchs/indexes/hgraph-90.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,14 +90,3 @@ HGRAPH/COHERE/90:
topk: 10
search_mode: "knn"
delete_index_after_search: false

HGRAPH/INTERNET/90:
datapath: "/tmp/data/wholenet-sparse-1m-ip.hdf5"
type: "build,search" # build, search
index_name: "sindi"
create_params: '{"dim":512,"dtype":"sparse","metric_type":"ip","index_param":{"use_reorder": true, "doc_prune_ratio": 0.0, "window_size": 60000}}'
search_params: '{"sindi":{"query_prune_ratio": 0.25, "term_prune_ratio": 0, "n_candidate": 35}}'
index_path: "/tmp/wholenet-sparse-1m-ip/index/wholenet_index"
topk: 10
search_mode: "knn"
delete_index_after_search: false
11 changes: 0 additions & 11 deletions benchs/indexes/hgraph-95.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,14 +90,3 @@ HGRAPH/COHERE/95:
topk: 10
search_mode: "knn"
delete_index_after_search: false

HGRAPH/INTERNET/95:
datapath: "/tmp/data/wholenet-sparse-1m-ip.hdf5"
type: "build,search" # build, search
index_name: "sindi"
create_params: '{"dim":512,"dtype":"sparse","metric_type":"ip","index_param":{"use_reorder": true, "doc_prune_ratio": 0.0, "window_size": 60000}}'
search_params: '{"sindi":{"query_prune_ratio": 0.1, "term_prune_ratio": 0, "n_candidate": 30}}'
index_path: "/tmp/wholenet-sparse-1m-ip/index/wholenet_index"
topk: 10
search_mode: "knn"
delete_index_after_search: false
11 changes: 0 additions & 11 deletions benchs/indexes/hgraph-99.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,14 +90,3 @@ HGRAPH/COHERE/99:
topk: 10
search_mode: "knn"
delete_index_after_search: false

HGRAPH/INTERNET/99:
datapath: "/tmp/data/wholenet-sparse-1m-ip.hdf5"
type: "build,search" # build, search
index_name: "sindi"
create_params: '{"dim":512,"dtype":"sparse","metric_type":"ip","index_param":{"use_reorder": false, "doc_prune_ratio": 0.0, "use_quantization": false, "window_size": 60000}}'
search_params: '{"sindi":{"query_prune_ratio": 0.0, "term_prune_ratio": 0, "n_candidate": 0}}'
index_path: "/tmp/wholenet-sparse-1m-ip/index/wholenet_index"
topk: 10
search_mode: "knn"
delete_index_after_search: false
7 changes: 4 additions & 3 deletions docs/docs/en/src/indexes/sindi.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,8 +66,8 @@ auto result = index->KnnSearch(

## Build parameters

Build-time parameters live under `index_param`. `dtype` **must** be `"sparse"`
and `metric_type` **must** be `"ip"`.
Build-time parameters live under `index_param`. `dtype` **must** be `"sparse"`,
`metric_type` **must** be `"ip"`, and all indexed and query sparse vector values must be positive.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
Expand Down Expand Up @@ -141,7 +141,8 @@ Search-time parameters live under the `sindi` sub-object:
|-----------|------|---------|-------------|
| `n_candidate` | int | `0` | Candidate heap size. When `0`, defaults to `SPARSE_AMPLIFICATION_FACTOR · topk` (500×). If set, must satisfy `1 ≤ n_candidate ≤ SPARSE_AMPLIFICATION_FACTOR · topk`. |
| `query_prune_ratio` | float | `0.0` | Fraction of lowest-weight query terms skipped (0.0 – 0.9). |
| `term_prune_ratio` | float | `0.0` | Fraction of term-list entries skipped (0.0 – 0.9). |
| `term_prune_ratio` | float | `0.0` | Fraction of the lowest-value postings skipped from each term list (0.0 – 0.9). |
| `term_prune_threshold` | uint64 | `0` | Maximum postings for one term across all windows. A value of `0` disables this limit; positive values allow each window to scan at most `floor(threshold / window_count)` postings. |
Comment thread
Roxanne0321 marked this conversation as resolved.
Outdated
Comment thread
Roxanne0321 marked this conversation as resolved.
Outdated

SINDI chooses the heap-insertion strategy automatically from the build-time
`doc_prune_ratio` and search-time `query_prune_ratio`. With the current `0.1`
Expand Down
5 changes: 3 additions & 2 deletions docs/docs/zh/src/indexes/sindi.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ auto result = index->KnnSearch(
## 构建参数

构建参数放在 `index_param` 下。`dtype` **必须** 为 `"sparse"`，`metric_type` **必须**
为 `"ip"`。
为 `"ip"`，且建库与查询稀疏向量的值均必须为正数。

| 参数 | 类型 | 默认值 | 说明 |
|------|------|--------|------|
Expand Down Expand Up @@ -131,7 +131,8 @@ auto result = index->KnnSearch(
|------|------|--------|------|
| `n_candidate` | int | `0` | 候选堆大小。为 `0` 时自动取 `SPARSE_AMPLIFICATION_FACTOR · topk`（500 倍）；若显式设置，须满足 `1 ≤ n_candidate ≤ SPARSE_AMPLIFICATION_FACTOR · topk` |
| `query_prune_ratio` | float | `0.0` | 查询时丢弃权重最低查询项的比例（0.0 – 0.9） |
| `term_prune_ratio` | float | `0.0` | 查询时丢弃倒排表中低权项的比例（0.0 – 0.9） |
| `term_prune_ratio` | float | `0.0` | 每条倒排链中按 value 丢弃低权 posting 的比例（0.0 – 0.9） |
Comment thread
Roxanne0321 marked this conversation as resolved.
Outdated
| `term_prune_threshold` | uint64 | `0` | 单个 term 在所有 window 中最多扫描的 posting 总数；`0` 表示关闭此限制，正数使每个 window 最多扫描 `floor(threshold / window_count)` 个 |
Comment thread
Roxanne0321 marked this conversation as resolved.
Outdated

SINDI 会根据构建阶段的 `doc_prune_ratio` 与检索阶段的 `query_prune_ratio`
自动选择堆插入策略。按当前 `0.1` 阈值，当两个比例都 `<= 0.1` 时，SINDI 使用
Expand Down
4 changes: 4 additions & 0 deletions examples/cpp/109_index_sindi.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,12 +123,16 @@ main(int argc, char** argv) {
*
* - sindi: Parameters specific to sparse indexing search:
* - query_prune_ratio: Ratio of term pruning for the query (0 = no pruning).
* - term_prune_ratio: Fraction of the lowest-value postings to skip.
* - term_prune_threshold: Per-term posting budget across all index windows.
* - n_candidate: Number of candidates for re-ranking. Must be greater than topK.
* This parameter is ignored if use_reorder is false in the build parameters.
*/
auto sindi_search_parameters = R"({
"sindi": {
"query_prune_ratio": 0,
"term_prune_ratio": 0.1,
"term_prune_threshold": 100000,
"n_candidate": 0
}
})";
Expand Down
1 change: 0 additions & 1 deletion examples/cpp/403_persistent_streaming_load.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -186,7 +186,6 @@ get_example_config() {
"use_reorder": true,
"use_quantization": false,
"doc_prune_ratio": 0.0,
"term_prune_ratio": 0.0,
"window_size": 10000,
"term_id_limit": 30001,
"avg_doc_term_length": 100
Expand Down
3 changes: 2 additions & 1 deletion examples/python/110_index_sindi_sq8_dmq_reorder.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,8 @@ def sindi_sq8_dmq8_reorder_test():
{
"sindi": {
"query_prune_ratio": 0.2, # Prune 20% of least-relevant query terms
"term_prune_ratio": 0.1, # Prune 10% of least-relevant terms from posting lists
# Prune the lowest-value 10%, capped by the per-window share of 100,000 postings.
"term_prune_ratio": 0.1, "term_prune_threshold": 100000,
"n_candidate": 10, # Keep top 10 candidates after inverted phase for reranking
}
}
Expand Down
Loading
Loading