Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -268,7 +268,8 @@ Thrive together in VSAG community with users and developers from all around the
phase physically remove vectors and shrink storage instead of leaving tombstones. Vector
updates use the same public API in both flat and graph phases.
- **IVF (Inverted File Index)**: Optimized for large-scale search (high `k`) and batch queries.
- **SINDI (Sparse Inverted Non-redundant Distance Index)**: Optimized sparse vector index.
- **SINDI (Sparse Inverted Non-redundant Distance Index)**: Optimized sparse vector index with
date-window filtering for online queries.

- **Advanced Quantization Methods**
- **RaBitQ (BQ)**: Extreme compression for minimal memory usage.
Expand Down
2 changes: 2 additions & 0 deletions docs/docs/en/src/advanced/new_serialization.md
Original file line number Diff line number Diff line change
Expand Up @@ -193,10 +193,12 @@ SINDI writes these streaming blocks in order:
| `label_table` | external labels and label remap | yes |
| `sindi_rerank_index` | optional rerank flat index when rerank is enabled | conditional |
| `sindi_term_id_mapper` | optional term-id remapping table | conditional |
| `sindi_window_metadata` | window boundaries and date labels | no (absent in legacy streams) |

`DeserializeStreaming` restores the full in-memory SINDI index. `Index::Load` can create the SINDI
index directly from streaming metadata and currently loads all emitted SINDI blocks into memory.
Immutable SINDI runtime serialization is not supported by this streaming path.
When `sindi_window_metadata` is absent, the reader reconstructs legacy fixed-size, undated windows.

## Pyramid Blocks

Expand Down
1 change: 1 addition & 0 deletions docs/docs/en/src/api/dataset.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,7 @@ For documents that hold several dense sub-vectors each:
| `ExtraInfoSize(int64_t)` | `GetExtraInfoSize()` | `int64_t` | Bytes per extra-info blob. |
| `Paths(const std::string*)` | `GetPaths()` | `const std::string*` | Hierarchy paths (Pyramid). Default hierarchy. |
| `Paths(const std::string& hierarchy, const std::string*)` | `GetPaths(const std::string& hierarchy)` | `const std::string*` | Paths for a named hierarchy. |
| `Dates(const std::string*)` | `GetDates()` | `const std::string*` | Optional per-element SINDI date labels (`YYYY`, `YYYY/MM`, or `YYYY/MM/DD`). |
| `SourceID(const std::string*)` | `GetSourceID()` | `const std::string*` | Optional source identifier. |

See [Attribute Filter (Hybrid Search)](../advanced/attribute_filter.md) and
Expand Down
32 changes: 28 additions & 4 deletions docs/docs/en/src/indexes/sindi.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,11 @@ pairs and is the only VSAG index that accepts `dtype: "sparse"`.

## How it works

1. **Window-based inverted lists.** Documents are grouped into fixed-size windows
(`window_size`). Within each window, an inverted list per term maps a term id
to the `(doc_id, value)` pairs that mention it.
1. **Window-based inverted lists.** Documents are grouped into windows capped by
`window_size`. When date labels are supplied, each window contains one exact
date label; a label that exceeds `window_size` is split across multiple windows.
Within each window, an inverted list per term maps a term id to the
`(doc_id, value)` pairs that mention it.
2. **Optional pruning and quantization.** During construction, `doc_prune_ratio`
drops low-weight terms per document, and `use_quantization` compresses the term
values to shrink memory further.
Expand Down Expand Up @@ -51,6 +53,7 @@ auto base = vsag::Dataset::Make();
base->NumElements(n)
->SparseVectors(sparse_vectors) // vsag::SparseVector*
->Ids(ids)
->Dates(dates) // optional std::string[n]
->Owner(false);
index->Build(base);

Expand All @@ -59,7 +62,7 @@ auto query = vsag::Dataset::Make();
query->NumElements(1)->SparseVectors(&query_vec)->Owner(false);
auto result = index->KnnSearch(
query, /*topk=*/10,
R"({"sindi": {"n_candidate": 100}})").value();
R"({"sindi": {"n_candidate": 100, "date": "2026/07"}})").value();
```

## Build parameters
Expand Down Expand Up @@ -92,6 +95,7 @@ Search-time parameters live under the `sindi` sub-object:
| `n_candidate` | int | `0` | Candidate heap size. When `0`, defaults to `SPARSE_AMPLIFICATION_FACTOR · topk` (500×). If set, must satisfy `1 ≤ n_candidate ≤ SPARSE_AMPLIFICATION_FACTOR · topk`. |
| `query_prune_ratio` | float | `0.0` | Fraction of lowest-weight query terms skipped (0.0 – 0.9). |
| `term_prune_ratio` | float | `0.0` | Fraction of term-list entries skipped (0.0 – 0.9). |
| `date` | string | `""` | Optional date-window selector in `YYYY`, `YYYY/MM`, or `YYYY/MM/DD` form. |

SINDI chooses the heap-insertion strategy automatically from the build-time
`doc_prune_ratio` and search-time `query_prune_ratio`. With the current `0.1`
Expand All @@ -107,6 +111,26 @@ auto result = index->KnnSearch(
R"({"sindi": {"n_candidate": 200, "query_prune_ratio": 0.1}})").value();
```

## Date-window filtering

Attach one optional date label to each build or add document with
`Dataset::Dates(const std::string*)`. Labels must use a canonical calendar form:
`YYYY`, `YYYY/MM`, or `YYYY/MM/DD`, including two-digit months and days. Documents
with the same exact label share windows; documents without a date (no dates array,
or an empty string entry) share undated windows. Different granularities are
different labels, so a `2026/05/17` document is stored only in a `2026/05/17`
window, not also in a `2026/05` window.

Pass the online query selector in the SINDI search parameters. A year selects its
year, month, and day windows; a month selects its month and day windows; a day
selects only that exact day. For example, `"date": "2026"` matches `2026`,
`2026/05`, and `2026/05/17`. A missing or empty query date disables date
filtering. A non-empty query date excludes undated windows.

Date filtering first prunes the windows to search. Any ID filter, bitset, or
`Filter` callback is then applied inside those windows, so the two filters have
AND semantics. Both KNN and range search support the date selector.

## When to use SINDI

- Sparse retrieval with BM25, SPLADE, uniCOIL, or similar learned-sparse encoders.
Expand Down
2 changes: 2 additions & 0 deletions docs/docs/zh/src/advanced/new_serialization.md
Original file line number Diff line number Diff line change
Expand Up @@ -173,10 +173,12 @@ SINDI 按顺序写入以下 streaming blocks:
| `label_table` | 外部 label 和 label remap | 是 |
| `sindi_rerank_index` | rerank 开启时的可选 rerank flat index | 条件必需 |
| `sindi_term_id_mapper` | 可选 term-id remap 表 | 条件必需 |
| `sindi_window_metadata` | 窗口边界与日期标签 | 否(旧版 stream 中不存在) |

`DeserializeStreaming` 会恢复完整的内存 SINDI 索引。`Index::Load` 可以直接从 streaming metadata
创建 SINDI 索引对象,当前会把写出的 SINDI blocks 都加载到内存中。immutable SINDI runtime 暂不支持
该 streaming 序列化路径。
缺少 `sindi_window_metadata` 时,读取端会按旧版固定大小规则恢复为无日期窗口。

## Pyramid Blocks

Expand Down
1 change: 1 addition & 0 deletions docs/docs/zh/src/api/dataset.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,7 @@ DatasetPtr DeepCopy(Allocator* allocator = nullptr) const; // 独立副本
| `ExtraInfoSize(int64_t)` | `GetExtraInfoSize()` | `int64_t` | 每个 extra-info 数据块的字节数。 |
| `Paths(const std::string*)` | `GetPaths()` | `const std::string*` | 层级路径(Pyramid)。默认层级。 |
| `Paths(const std::string& hierarchy, const std::string*)` | `GetPaths(const std::string& hierarchy)` | `const std::string*` | 命名层级的路径。 |
| `Dates(const std::string*)` | `GetDates()` | `const std::string*` | 可选的逐元素 SINDI 日期标签(`YYYY`、`YYYY/MM` 或 `YYYY/MM/DD`)。 |
| `SourceID(const std::string*)` | `GetSourceID()` | `const std::string*` | 可选的来源标识。 |

见 [属性过滤(混合搜索)](../advanced/attribute_filter.md) 与
Expand Down
25 changes: 22 additions & 3 deletions docs/docs/zh/src/indexes/sindi.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,9 @@ SINDI(**S**parse **IN**verted **D**ense **I**ndex)是 VSAG 面向 **稀疏

## 工作原理

1. **基于窗口的倒排表。** 文档按固定窗口大小(`window_size`)分组,每个窗口独立维护一套
倒排表——即“词项 → `(doc_id, value)` 列表”的映射。
1. **基于窗口的倒排表。** 文档按不超过 `window_size` 的窗口分组。提供日期标签时,每个窗口
只包含一个完全相同的日期标签;同一标签超过 `window_size` 时拆分为多个窗口。每个窗口独立
维护一套倒排表——即“词项 → `(doc_id, value)` 列表”的映射。
2. **可选的剪枝与量化。** 构建时可通过 `doc_prune_ratio` 按文档粒度丢弃权重最低的词项;
通过 `use_quantization` 压缩词项权重以进一步节省内存。
3. **打分。** 检索时,SINDI 遍历查询向量的非零项,按窗口访问对应的倒排表,使用大小为
Expand Down Expand Up @@ -47,6 +48,7 @@ auto base = vsag::Dataset::Make();
base->NumElements(n)
->SparseVectors(sparse_vectors) // vsag::SparseVector*
->Ids(ids)
->Dates(dates) // 可选的 std::string[n]
->Owner(false);
index->Build(base);

Expand All @@ -55,7 +57,7 @@ auto query = vsag::Dataset::Make();
query->NumElements(1)->SparseVectors(&query_vec)->Owner(false);
auto result = index->KnnSearch(
query, /*topk=*/10,
R"({"sindi": {"n_candidate": 100}})").value();
R"({"sindi": {"n_candidate": 100, "date": "2026/07"}})").value();
```

## 构建参数
Expand Down Expand Up @@ -87,6 +89,7 @@ auto result = index->KnnSearch(
| `n_candidate` | int | `0` | 候选堆大小。为 `0` 时自动取 `SPARSE_AMPLIFICATION_FACTOR · topk`(500 倍);若显式设置,须满足 `1 ≤ n_candidate ≤ SPARSE_AMPLIFICATION_FACTOR · topk` |
| `query_prune_ratio` | float | `0.0` | 查询时丢弃权重最低查询项的比例(0.0 – 0.9) |
| `term_prune_ratio` | float | `0.0` | 查询时丢弃倒排表中低权项的比例(0.0 – 0.9) |
| `date` | string | `""` | 可选的日期窗口选择条件,格式为 `YYYY`、`YYYY/MM` 或 `YYYY/MM/DD` |

SINDI 会根据构建阶段的 `doc_prune_ratio` 与检索阶段的 `query_prune_ratio`
自动选择堆插入策略。按当前 `0.1` 阈值,当两个比例都 `<= 0.1` 时,SINDI 使用
Expand All @@ -99,6 +102,22 @@ auto result = index->KnnSearch(
R"({"sindi": {"n_candidate": 200, "query_prune_ratio": 0.1}})").value();
```

## 日期窗口过滤

构建或 Add 时,通过 `Dataset::Dates(const std::string*)` 为每篇文档提供可选日期标签。
标签必须使用规范日历格式:`YYYY`、`YYYY/MM` 或 `YYYY/MM/DD`,月和日固定为两位。
完全相同的标签共用窗口;未提供 dates 数组或数组元素为空字符串时,文档进入无日期窗口。
不同精度是不同标签,因此 `2026/05/17` 文档只存入 `2026/05/17` 窗口,不会再重复存入
`2026/05` 窗口。

线上查询时在 SINDI 检索参数中传入日期。年份会选中该年的年、月、日窗口;月份会选中该月的
月、日窗口;日期只选中完全相同的日窗口。例如 `"date": "2026"` 会匹配 `2026`、
`2026/05` 和 `2026/05/17`。缺少 `date` 或传入空字符串表示不启用日期过滤;非空日期不会
匹配无日期窗口。

日期条件先裁剪待检索窗口,已有的 ID filter、bitset 或 `Filter` callback 再在命中窗口内部
执行,因此两者为 AND 关系。KNN 与范围检索均支持日期条件。

## 何时选择 SINDI

- 使用 BM25、SPLADE、uniCOIL 等学习稀疏编码器的稀疏检索场景。
Expand Down
6 changes: 5 additions & 1 deletion examples/cpp/109_index_sindi.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ main(int argc, char** argv) {

std::vector<int64_t> ids(num_vectors);
std::vector<vsag::SparseVector> sparse_vectors(num_vectors);
std::vector<std::string> dates(num_vectors, "2026/07/21");

for (int64_t i = 0; i < num_vectors; ++i) {
ids[i] = i;
Expand All @@ -56,6 +57,7 @@ main(int argc, char** argv) {
base->NumElements(num_vectors)
->SparseVectors(sparse_vectors.data())
->Ids(ids.data())
->Dates(dates.data())
->Owner(false);

/******************* Create SINDI Index *****************/
Expand Down Expand Up @@ -123,11 +125,13 @@ main(int argc, char** argv) {
* - query_prune_ratio: Ratio of term pruning for the query (0 = no pruning).
* - n_candidate: Number of candidates for re-ranking. Must be greater than topK.
* This parameter is ignored if use_reorder is false in the build parameters.
* - date: Optional date-window filter (YYYY, YYYY/MM, or YYYY/MM/DD).
*/
auto sindi_search_parameters = R"({
"sindi": {
"query_prune_ratio": 0,
"n_candidate": 0
"n_candidate": 0,
"date": "2026/07"
}
})";

Expand Down
1 change: 1 addition & 0 deletions include/vsag/constants.h
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ extern const char* const SPARSE_VECTORS;
extern const char* const INT8_VECTORS;
extern const char* const ATTRIBUTE_SETS;
extern const char* const DATASET_PATHS;
extern const char* const DATASET_DATES;
extern const char* const EXTRA_INFOS;
extern const char* const EXTRA_INFO_SIZE;
extern const char* const VECTOR_COUNTS;
Expand Down
17 changes: 17 additions & 0 deletions include/vsag/dataset.h
Original file line number Diff line number Diff line change
Expand Up @@ -354,6 +354,23 @@ class Dataset : public std::enable_shared_from_this<Dataset> {
virtual const std::string*
GetPaths(const std::string& hierarchy_name) const = 0;

/**
* @brief Sets the date-label array for the dataset.
*
* @param dates Pointer to an array with one date label per element.
* @return DatasetPtr A shared pointer to the dataset with updated date labels.
*/
virtual DatasetPtr
Dates(const std::string* dates) = 0;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Do not insert new virtual slots into the public Dataset ABI

These pure virtual methods are inserted before existing methods such as ExtraInfos, shifting every subsequent vtable slot. An application compiled against the previous header but loading the new library will dispatch those calls to the wrong functions (for example, the old ExtraInfos slot now invokes Dates), while external Dataset subclasses also become source-incompatible. Preserve existing slot ordering, such as by appending an ABI-safe/defaulted extension instead.


/**
* @brief Retrieves the date-label array for the dataset.
*
* @return const std::string* Pointer to the date-label array, or nullptr when absent.
*/
virtual const std::string*
GetDates() const = 0;

/**
* @brief Sets the extra info for the dataset.
*
Expand Down
Loading
Loading