feat: support pyramid raw vector retrieval by jac0626 · Pull Request #2503 · antgroup/vsag

jac0626 · 2026-07-22T04:08:44Z

Change Type

Linked Issue

Fixes: [feat](other): support raw vector retrieval in pyramid #2502

What Changed

Add Pyramid store_raw_vector support and expose raw vectors through GetRawVectorByIds/GetVectorByInnerId.
Reuse an existing in-memory FP32 base or reorder data cell when possible; otherwise maintain separate FP32 raw-vector storage across Train, Build, Add, ODescent, and serialization.
Add live Pyramid GetMemoryUsage()/GetMemoryUsageDetail() reporting, including hierarchy graphs and separate raw-vector storage without double-counting aliases.
Persist FP32 hold_molds, validate legacy deserialization parameters before reading the body, and preserve backward compatibility for older Pyramid parameter snapshots.
Document the new parameters in English and Chinese.

Test Evidence

Test details:

/opt/homebrew/opt/llvm@15/bin/clang-format --dry-run --Werror <changed C++ files>
cmake --build build --target unittests -j6
./build/tests/unittests
  All tests passed (62,573,683 assertions in 561 test cases)

./build/tests/unittests '[ut][pyramid][raw_vector]'
  All tests passed (45 assertions in 6 test cases)

Compatibility Impact

API/ABI compatibility: additive behavior; no public signature changes
Behavior changes: Pyramid can preserve exact FP32 vectors when store_raw_vector=true; FP32 parameter snapshots now retain hold_molds

Performance and Concurrency Impact

Performance impact: separate FP32 storage is allocated only when existing in-memory FP32 codes cannot be reused
Concurrency/thread-safety impact: memory reporting and raw-vector reads use the existing Pyramid resize lock

Documentation Impact

Risk and Rollback

Risk level: medium
Rollback plan: revert this commit to restore the previous Pyramid storage and serialization behavior

Checklist

I have linked the relevant issue (required for kind/bug and kind/feature; see "Linked Issue" above)
I have added/updated tests for new behavior or bug fixes
I have considered API compatibility impact
I have updated docs if behavior/workflow changed
My commit messages follow project conventions (Conventional Commits, optional [skip ci] prefix)

gemini-code-assist · 2026-07-22T04:08:47Z

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

vsag-bot · 2026-07-22T04:08:48Z

/label status/waiting-for-review
/waiting-on reviewer
/request-review @jiaweizone
/request-review @wxyucs
/request-review @inabao

vsag-bot · 2026-07-22T04:08:59Z

Automated pull request review completed.

Review effort: high (855 changed lines across 9 files).

Submitted 3 inline comments.
Review: #2503 (review)

mergify · 2026-07-22T04:09:20Z

Merge Protections

🟢 All 3 merge protections satisfied — ready to merge.

Show 3 satisfied protections

🟢 Require kind label

label~=^kind/

🟢 Require version label

label~=^version/

🟢 Require linked issue for feature/bug PRs

body~=(?im)(?:^|[\s\-\*])(?:close[sd]?|fix(?:e[sd])?|resolve[sd]?)\s*:?\s+(?:#\d+|[\w.\-]+/[\w.\-]+#\d+|https?://github\.com/[\w.\-]+/[\w.\-]+/issues/\d+)

Copilot

Pull request overview

Adds Pyramid support for preserving and retrieving FP32 raw vectors (including through streaming/legacy serialization), along with detailed live memory reporting to enable accurate HGraph vs Pyramid memory comparisons.

Changes:

Implement Pyramid raw-vector storage/reuse logic and expose it via GetVectorByInnerId / GetRawVectorByIds, including streaming + legacy serialization support.
Add Pyramid GetMemoryUsage() / GetMemoryUsageDetail() with hierarchy graph accounting and alias-aware raw-vector reporting.
Update unit tests, docs (EN/ZH), and add a C++ example to reproduce HGraph/Pyramid memory comparisons.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/quantization/fp32_quantizer_parameter.cpp	Persist `hold_molds` in FP32 quantizer JSON.
src/quantization/fp32_quantizer_parameter_test.cpp	Extend FP32 parameter JSON roundtrip coverage for `hold_molds`.
src/algorithm/pyramid/pyramid.h	Add raw-vector members/hooks and memory usage APIs to Pyramid declarations.
src/algorithm/pyramid/pyramid.cpp	Implement raw-vector reuse/storage, serialization blocks, and memory usage reporting.
src/algorithm/pyramid/pyramid_zparameters.cpp	Backward-compat restore for legacy FP32 `hold_molds` and compatibility checks for raw-vector params.
src/algorithm/pyramid/pyramid_test.cpp	Add regression tests covering raw vectors, (de)serialization, cosine behavior, and memory reporting.
examples/cpp/README.md	Document new memory comparison example.
examples/cpp/CMakeLists.txt	Build target for the new memory comparison example.
examples/cpp/325_feature_compare_hgraph_pyramid_memory.cpp	New example program to compare RSS/allocator/index-reported memory.
docs/docs/zh/src/indexes/pyramid.md	Document new Pyramid raw-vector parameters (ZH).
docs/docs/zh/src/advanced/new_serialization.md	Document Pyramid streaming `raw_vector` block (ZH).
docs/docs/en/src/indexes/pyramid.md	Document new Pyramid raw-vector parameters (EN).
docs/docs/en/src/advanced/new_serialization.md	Document Pyramid streaming `raw_vector` block (EN).

jac0626 · 2026-07-22T06:01:36Z

/retest

gemini-code-assist · 2026-07-22T06:02:01Z

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

Copilot

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

src/algorithm/pyramid/pyramid.cpp:978

SUPPORT_GET_RAW_VECTOR_BY_IDS is enabled whenever raw_vector_ != nullptr, but for cosine metrics an FP32 flatten without HoldMolds() stores normalized vectors, so GetRawVectorByIds would not return the original/raw vectors. This should follow the same gating used by BruteForce/HGraph: only advertise raw-vector support for cosine when molds are preserved.

    if (raw_vector_ != nullptr) {
        this->index_feature_list_->SetFeature(IndexFeature::SUPPORT_GET_RAW_VECTOR_BY_IDS);
    }

vsag-bot

Automated inline review completed.

Review effort: high (531 changed lines across 10 files).
Submitted 4 inline comments.

Reviewed commit c519510.

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated no new comments.

wxyucs · 2026-07-27T12:45:14Z

/version 1.1

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

LHT129

[suggestion] ExportModel does not export raw_vector_ data. When create_new_raw_vector_ is true, the exported model will lose raw vector storage, causing GetRawVectorByIds to fail on the exported index. Consider adding:

if (create_new_raw_vector_) {
    this->raw_vector_->ExportModel(index->raw_vector_);
}

in Pyramid::ExportModel (pyramid.cpp, around line 877).

vsag-bot

Automated inline review completed.

Review effort: high (811 changed lines across 8 files).
Submitted 2 inline comments.

Reviewed commit 220ef23.

jac0626 · 2026-07-29T03:38:49Z

/retest

jac0626 · 2026-07-30T07:50:08Z

Review follow-up: I also evaluated the non-inline ExportModel suggestion and intentionally made no change. Pyramid::ExportModel exports reusable quantizer/model state, not indexed vector data. A dedicated Pyramid raw cell is constrained to FP32, which has no trained model state to transfer; the target is already constructed with an empty raw cell from the same parameters. Copying raw vectors would violate the existing empty-model contract rather than fix retrieval.

Signed-off-by: jac0626 <jac0626@users.noreply.github.com> Assisted-by: Codex:gpt-5

Signed-off-by: jc543239 <jc543239@antgroup.com> Assisted-by: Codex:gpt-5

Signed-off-by: jac0626 <jac0626@users.noreply.github.com> Assisted-by: Codex:gpt-5

jac0626 · 2026-07-30T07:54:12Z

Final rebase note: upstream/main advanced again while validation was running. The review-fix commit referenced in the thread replies was rewritten from 44608a1 to d8dc386; the fixes and tests are unchanged.

LHT129

Review of the latest push (44608a1). This push addresses several prior review comments (label publication order, null-pointer checks, factory load path). A few remaining observations below.

LHT129 · 2026-07-30T08:00:44Z

+std::unordered_map<std::string, uint64_t>
+Pyramid::GetMemoryUsageDetail() const {
+    std::lock_guard count_lock(cur_element_count_mutex_);
+    std::unique_lock resize_lock(resize_mutex_);


[suggestion] GetMemoryUsageDetail() acquires unique_lock on resize_mutex_, which blocks concurrent search, GetVectorByInnerId, and distance-by-id calls. Since this method only reads state, a shared_lock would be sufficient and would avoid serializing all read operations against memory-usage queries.

Consider changing std::unique_lock resize_lock(resize_mutex_) to std::shared_lock resize_lock(resize_mutex_).

LHT129 · 2026-07-30T08:00:44Z

+    for (const auto& [key, child] : children_) {
+        memory += key.capacity() + 1;
+        memory += child->GetMemoryUsage();
+    }


[suggestion] IndexNode::GetMemoryUsage() multiplies children_.bucket_count() by value_type size, which overestimates memory because bucket_count() is the hash-table bucket count, not the number of stored entries. Use children_.size() instead to get the actual number of child nodes.

LHT129 · 2026-07-30T08:00:44Z

+        "{RAW_VECTOR_KEY}": {
+            "{IO_PARAMS_KEY}": {
+                "{TYPE_KEY}": "{IO_TYPE_VALUE_BLOCK_MEMORY_IO}",
+                "{IO_FILE_PATH_KEY}": "{DEFAULT_FILE_PATH_VALUE}"


[suggestion] The raw_vector cell defaults to the same ./default_file_path used by base and precise codes. With a documented configuration such as SQ8 base_io_type=buffer_io and raw_vector_io_type=buffer_io, both cells would open the same file path, leading to corruption. Consider using a distinct default path (e.g. ./default_raw_vector_path) or requiring an explicit path when the raw-vector backend is file-backed.

LHT129

Additional observations on the latest push.

LHT129 · 2026-07-30T08:01:21Z


+void
+Pyramid::check_and_init_raw_vector(const FlattenInterfaceParamPtr& raw_vector_param,
+                                   const IndexCommonParam& common_param) {


[suggestion] check_and_init_raw_vector checks raw_vector_param->io_parameter->GetTypeName() to decide whether to attempt aliasing, but the aliasing decision should be based on the candidate cell's IO type (i.e. whether base_codes_ or precise_codes_ is in-memory FP32), not the requested raw-vector backend. If the user requests raw_vector_io_type=buffer_io but base_codes_ happens to be in-memory FP32, aliasing is still desirable and safe.

Consider calling find_raw_vector_source(true) unconditionally (regardless of raw_vector_param IO type) and only falling back to MakeInstance when no alias is found.

LHT129 · 2026-07-30T08:01:21Z

+    auto io_type_name = raw_vector_param->io_parameter->GetTypeName();
+    if (io_type_name == IO_TYPE_VALUE_BLOCK_MEMORY_IO || io_type_name == IO_TYPE_VALUE_MEMORY_IO) {
+        raw_vector_ = find_raw_vector_source(true);
+    }


[note] check_and_init_raw_vector sets has_raw_vector_ = raw_vector_ != nullptr at line 1546, but raw_vector_ is guaranteed non-null at this point (either from find_raw_vector_source or MakeInstance). The null check is redundant; has_raw_vector_ will always be true when this code is reached. Consider setting has_raw_vector_ = true directly.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

LHT129

Code Review for PR #2503

[critical] Missing `has_raw_vector_` declaration in `pyramid.h`

has_raw_vector_ is assigned in check_and_init_raw_vector() (pyramid.cpp line ~1558) and read in InitFeatures() (pyramid.cpp line ~999), but this member variable is never declared in pyramid.h. The header only declares create_new_raw_vector_. This will cause a compilation error.

Fix: Add bool has_raw_vector_{false}; to the member variables section of pyramid.h, alongside create_new_raw_vector_:

bool create_new_raw_vector_{false};  // whether raw_vector_ owns separate storage
bool has_raw_vector_{false};         // whether raw vector retrieval is available

[suggestion] Unnecessary `FlattenInterface` allocation in `check_and_init_raw_vector`

In check_and_init_raw_vector(), FlattenInterface::MakeInstance(raw_vector_param, common_param) is always called first (line ~1543), even when raw_vector_ will ultimately alias base_codes_ or precise_codes_ via find_raw_vector_source(). When aliasing succeeds, the newly constructed instance is leaked (overwritten by the aliased pointer). Consider checking for an aliasable source first, and only constructing a new instance when necessary.

[note] `CalDistanceById` precedence order

Both CalcDistanceById overloads override flat with raw_vector_ when raw_vector_ != nullptr && calculate_precise_distance. This means raw_vector_ takes precedence over precise_codes_ which takes precedence over base_codes_. This is correct behavior (raw vectors are always the most precise), but the precedence chain is implicit. A brief comment noting the priority would help future readers.

vsag-bot

Automated inline review completed.

Review effort: high (855 changed lines across 9 files).
Submitted 3 inline comments.

Reviewed commit d8dc386.

vsag-bot · 2026-07-30T09:04:07Z

+        "{RAW_VECTOR_KEY}": {
+            "{IO_PARAMS_KEY}": {
+                "{TYPE_KEY}": "{IO_TYPE_VALUE_BLOCK_MEMORY_IO}",
+                "{IO_FILE_PATH_KEY}": "{DEFAULT_FILE_PATH_VALUE}"


[P1] Give raw storage a cell-specific backing file

This uses the same DEFAULT_FILE_PATH_VALUE as the base and precise cells. With accepted settings such as SQ8 base_io_type=buffer_io plus raw_vector_io_type=buffer_io and omitted paths, both cells truncate and write the same file from offset zero; FP32 raw writes then overwrite the quantized search codes, silently corrupting the index. Reject shared file paths or derive a raw-specific default.

vsag-bot · 2026-07-30T09:04:07Z

+                    precise_codes_->InsertVector(data_vectors + dim_ * i, inner_id);
+                }
+                if (create_new_raw_vector_) {
+                    raw_vector_->InsertVector(data_vectors + dim_ * i, inner_id);


[P2] Roll back counts when the raw-vector write fails

FlattenDataCell::InsertVector advances its count before performing the IO write, and base/precise cells have already advanced by this line. If the separate raw backend fails here, no label, logical count, or hierarchy node is published, but GetNumElements() still observes the advanced base count; in a multi-vector Add, retrying can reuse inner ID 0 and overwrite an earlier published vector because cur_element_count_ was never updated. Make the cell writes transactional or explicitly restore every affected count/state on failure.

vsag-bot · 2026-07-30T09:04:07Z

+                if (create_new_raw_vector_) {
+                    raw_vector_->InsertVector(data_vectors + dim_ * i, inner_id);
                }
+                label_table_->Insert(inner_id, data_ids[i]);


[P2] Synchronize label publication with raw-vector lookup

This mutates LabelTable while holding only cur_element_count_mutex_, whereas the newly exposed GetRawVectorByIds path delegates to GetVectorByIds, which calls GetIdByLabel without that mutex or label_lookup_mutex_. A raw lookup concurrent with Add can therefore race the label vector/hash map and invoke undefined behavior even though the vector bytes are written first. Publish and resolve labels under the same lock.

Copilot AI review requested due to automatic review settings July 22, 2026 04:08

pull-request-size Bot added the size/XL label Jul 22, 2026

vsag-bot assigned LHT129 Jul 22, 2026

jac0626 added kind/feature Brand-new functionality or capabilities 引入全新的功能、新特性或新能力 version/1.0 labels Jul 22, 2026

vsag-bot self-requested a review July 22, 2026 04:08

Copilot started reviewing on behalf of jac0626 July 22, 2026 04:09 View session

mergify Bot added module/docs module/example labels Jul 22, 2026

Copilot AI reviewed Jul 22, 2026

View reviewed changes

Comment thread src/algorithm/pyramid/pyramid.cpp Outdated

jac0626 marked this pull request as ready for review July 22, 2026 06:01

jac0626 requested review from LHT129, inabao, jiaweizone and wxyucs as code owners July 22, 2026 06:01

Copilot AI review requested due to automatic review settings July 22, 2026 06:10

Copilot started reviewing on behalf of jac0626 July 22, 2026 06:10 View session

Copilot AI reviewed Jul 22, 2026

View reviewed changes

Comment thread src/algorithm/pyramid/pyramid.cpp Outdated

vsag-bot reviewed Jul 22, 2026

View reviewed changes

Comment thread src/quantization/fp32_quantizer_parameter.cpp Outdated

Comment thread src/algorithm/pyramid/pyramid.cpp

Comment thread src/algorithm/pyramid/pyramid.cpp Outdated

Comment thread src/algorithm/pyramid/pyramid.cpp

Copilot AI review requested due to automatic review settings July 23, 2026 03:19

vsag-bot self-requested a review July 23, 2026 03:19

Copilot started reviewing on behalf of jac0626 July 23, 2026 03:19 View session

Copilot AI reviewed Jul 23, 2026

View reviewed changes

jac0626 force-pushed the codex/pyramid-raw-vector branch from 08b4de1 to 360048a Compare July 23, 2026 03:31

LHT129 reviewed Jul 27, 2026

View reviewed changes

Comment thread src/algorithm/pyramid/pyramid.cpp

LHT129 reviewed Jul 27, 2026

View reviewed changes

Comment thread src/algorithm/pyramid/pyramid.cpp

vsag-bot added version/1.1 and removed version/1.0 labels Jul 27, 2026

jac0626 force-pushed the codex/pyramid-raw-vector branch from cff0f1c to 220ef23 Compare July 29, 2026 03:03

Copilot AI review requested due to automatic review settings July 29, 2026 03:03

vsag-bot self-requested a review July 29, 2026 03:03

Copilot started reviewing on behalf of jac0626 July 29, 2026 03:03 View session

Copilot AI reviewed Jul 29, 2026

LHT129 reviewed Jul 29, 2026

View reviewed changes

vsag-bot reviewed Jul 29, 2026

View reviewed changes

Comment thread src/algorithm/pyramid/pyramid.cpp

Comment thread src/algorithm/pyramid/pyramid.cpp Outdated

Copilot AI review requested due to automatic review settings July 30, 2026 07:49

jac0626 force-pushed the codex/pyramid-raw-vector branch from 220ef23 to 44608a1 Compare July 30, 2026 07:49

vsag-bot self-requested a review July 30, 2026 07:50

jac0626 and others added 5 commits July 30, 2026 15:51

feat: support pyramid raw vector retrieval

a3041ff

Signed-off-by: jac0626 <jac0626@users.noreply.github.com> Assisted-by: Codex:gpt-5

chore: remove pyramid memory comparison example

0629ca6

Signed-off-by: jac0626 <jac0626@users.noreply.github.com> Assisted-by: Codex:gpt-5

fix: harden pyramid raw vector storage

78aa96c

Signed-off-by: jac0626 <jac0626@users.noreply.github.com> Assisted-by: Codex:gpt-5

fix: resolve pyramid clang-tidy errors

a64fd2c

Signed-off-by: jc543239 <jc543239@antgroup.com> Assisted-by: Codex:gpt-5

fix: address pyramid raw vector review feedback

d8dc386

Signed-off-by: jac0626 <jac0626@users.noreply.github.com> Assisted-by: Codex:gpt-5

jac0626 force-pushed the codex/pyramid-raw-vector branch from 44608a1 to d8dc386 Compare July 30, 2026 07:54

LHT129 reviewed Jul 30, 2026

View reviewed changes

Copilot AI reviewed Jul 30, 2026

LHT129 reviewed Jul 30, 2026

View reviewed changes

Copilot started reviewing on behalf of jac0626 July 30, 2026 08:46 View session

vsag-bot reviewed Jul 30, 2026

View reviewed changes

Uh oh!

Conversation

jac0626 commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Change Type

Linked Issue

What Changed

Test Evidence

Compatibility Impact

Performance and Concurrency Impact

Documentation Impact

Risk and Rollback

Checklist

Uh oh!

gemini-code-assist Bot commented Jul 22, 2026

Uh oh!

vsag-bot commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vsag-bot commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mergify Bot commented Jul 22, 2026

Merge Protections

🟢 Require kind label

🟢 Require version label

🟢 Require linked issue for feature/bug PRs

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

jac0626 commented Jul 22, 2026

Uh oh!

gemini-code-assist Bot commented Jul 22, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

vsag-bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

wxyucs commented Jul 27, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

LHT129 left a comment

Choose a reason for hiding this comment

Uh oh!

vsag-bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jac0626 commented Jul 29, 2026

Uh oh!

jac0626 commented Jul 30, 2026

Uh oh!

jac0626 commented Jul 30, 2026

Uh oh!

LHT129 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jac0626 commented Jul 22, 2026 •

edited

Loading

vsag-bot commented Jul 22, 2026 •

edited

Loading

vsag-bot commented Jul 22, 2026 •

edited

Loading

[critical] Missing `has_raw_vector_` declaration in `pyramid.h`

[suggestion] Unnecessary `FlattenInterface` allocation in `check_and_init_raw_vector`

[note] `CalDistanceById` precedence order