perf(hgraph): add reusable max_degree graph reduction

[jac0626]

Change Type

• Bug fix
• New feature
• Improvement/Refactor
• Documentation
• CI/Build/Infra

Linked Issue

• Closes: [improve](hgraph): derive compact lower-degree graphs from one build #2578

What Changed

• Add an internal HGraph capability to derive a lower-max_degree graph from an already built
  higher-degree graph.
• Rank existing adjacency lists once with the existing diversity heuristic, then materialize the
  selected prefixes into ordinary compact graph storage.
• Support successive reductions, serialization, deserialization, and search through the normal
  HGraph paths.
• Restrict the capability to compatible mutable in-memory NSW configurations and reject invalid
  or unsupported reductions.
• Add focused tests for degree reduction, repeated reduction, invalid input, serialization,
  deserialization, and search.

This PR intentionally provides only the HGraph primitive. AutoTune or any other consumer can call
it later, but no AutoTune dependency or orchestration is included here.

Test Evidence

• make fmt
• make lint
• make test
• make cov, run tests, and collect coverage
• Other (describe below)

Test details:

cmake --build build-release --target algorithm_test -j$(nproc)
  passed

cmake --build build-release --target vsag -j$(nproc)
  passed

/tmp/hgraph_degree_reduction_test
  All tests passed (22 assertions in 1 test case)

clang-format-15 --dry-run --Werror <changed C++ files>
  passed

clang-tidy-15 -p build-release \
  src/algorithm/hgraph/hgraph_degree_reduction.cpp --quiet
  passed

git diff --check upstream/main...HEAD
  passed

Performance validation used the first 500,000 vectors from SIFT1M, fp32, NSW,
ef_construction=100, 48 build threads, and degrees 16/32/64. Serialization was excluded:

Independent M16/M32/M64 builds: 23.70s
M64 build + M32/M16 projection:  9.61s
Conservative construction speedup: 2.47x
Construction time reduction:       59.4%

A separate five-repeat SIFT 20K comparison found median recall deltas from -0.0002 to +0.0112
and a maximum median projected-versus-independent latency increase of 6.9%. Searching the same
M64 artifact before and after this change produced identical recall and no measurable normal-path
latency regression.

Compatibility Impact

• API/ABI compatibility: No installed public API or ABI change. The new methods are internal to
  HGraph.
• Behavior changes: None unless an internal caller explicitly invokes degree reduction.
• Serialization: No format change; reduced graphs use the existing physical graph format.

Performance and Concurrency Impact

• Performance impact: Consumers evaluating multiple max_degree values can replace repeated full
  graph construction with one construction plus inexpensive reductions. Normal build and search
  paths are unchanged.
• Concurrency/thread-safety impact: Preparation and reduction take the existing HGraph add/global
  locks.

Documentation Impact

• No docs update needed
• Updated docs

This is an internal capability with no user-facing workflow in this PR.

Risk and Rollback

• Risk level: medium
• Rollback plan: Revert commit 9df7758a; normal HGraph paths and the serialization format are
  otherwise unchanged.

The reduced graph has the requested compact degree but inherits topology and hierarchy from the
larger graph. It is not byte-for-byte or topology-equivalent to a graph independently built with
the smaller max_degree, so consumers must evaluate the reduced artifact itself.

Checklist

• I have linked the relevant issue
• I have added/updated tests for new behavior or bug fixes
• I have considered API compatibility impact
• I have updated docs if behavior/workflow changed
• My commit messages follow project conventions

[vsag-bot]

/label status/waiting-for-review
/waiting-on reviewer
/request-review @jiaweizone
/request-review @wxyucs
/request-review @inabao
/request-review @vsag-bot

[vsag-bot]

Automated pull request review failed.

Review effort: high (7201 changed lines across 45 files).

the pull request head changed after /review; run the command again for the latest commit

No GitHub review was submitted.

[mergify]

Merge Protections

🟢 All 2 merge protections satisfied — ready to merge.

Show 2 satisfied protections

🟢 Require kind label

• label~=^kind/

🟢 Require version label

• label~=^version/

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[LHT129]

[suggestion] The parallel_for helper has a subtle exception-safety issue: when GeneralEnqueue throws, the loop breaks, leaving already-submitted futures running. Those futures may also throw, and the collection loop silently discards later exceptions when first_exception is already set. More importantly, the remaining work items are never submitted, leaving the graph in a partially-processed state with no rollback.

Consider either:

1. Collecting all exceptions and reporting them together, or
2. Ensuring that partial execution is safe (e.g., by documenting that rank_graph and materialize_graph are idempotent per-node and partial execution is acceptable).

At minimum, the break on submission failure should be replaced with a strategy that either cancels already-submitted work or documents why partial execution is safe.

[LHT129]

[suggestion] The std::find on line 137 performs an O(N) linear scan over neighbors for each original neighbor, making the per-node complexity O(M*K) where M is the original degree and K is the number of selected neighbors. For graphs with large max_degree (e.g., 64 or 128), this becomes a noticeable hotspot.

Since neighbors is the result of select_edges_by_heuristic (typically small), consider using an std::unordered_set<InnerIdType> or sorting both vectors and using std::set_difference to avoid the quadratic behavior. Alternatively, since neighbors is modified in-place by select_edges_by_heuristic and then reversed, you could track which elements were kept during the heuristic selection itself rather than re-discovering them with std::find.

[LHT129]

[note] The flatten parameter in materialize_graph is only used for ExportCommonParam() to create the target graph instance. The actual neighbor copying does not use flatten at all — it only reads neighbor IDs from the source graph and writes them to the target. This is fine functionally, but the parameter name and presence may mislead readers into thinking distance computation is involved.

Consider either:

1. Passing CommonParam directly instead of the full FlattenInterfacePtr, or
2. Adding a brief comment clarifying that flatten is only used for graph instantiation, not for distance calculations.

[LHT129]

[note] degree_reduction_prepared_ is not serialized/deserialized. After deserializing a previously-reduced graph, the flag will be false, so PrepareDegreeReduction would need to be called again before further reductions — which would re-rank the already-materialized neighbors unnecessarily. This is not incorrect but is wasteful.

If the reduced graph is intended to be further reducible after deserialization, consider either:

1. Persisting the flag in the serialization format, or
2. Making PrepareDegreeReduction detect that neighbors are already ranked (e.g., by checking if the first neighbor of each node is the closest).

[LHT129]

Overall this is a well-structured PR that introduces a useful internal capability for HGraph. The design is clean: rank adjacency lists once with the diversity heuristic, then materialize prefixes into compact storage for each target max_degree. The separation of PrepareDegreeReduction (rank) from ReduceMaxDegree (materialize) is a good choice that allows multiple reductions from a single ranking pass.

Summary of findings:

1. [suggestion] parallel_for exception safety — The break on submission failure leaves already-submitted futures running with no cancellation, and remaining work items are silently dropped. This leaves the graph in a partially-processed state. Consider documenting why this is safe or adding a rollback mechanism.

2. [suggestion] std::find in rank_graph — The O(M*K) linear scan per node can become a hotspot for large max_degree values. An unordered_set or set-based approach would be more efficient.

3. [note] materialize_graph flatten parameter — The parameter is only used for ExportCommonParam() and is otherwise dead. Consider passing CommonParam directly or adding a clarifying comment.

4. [note] degree_reduction_prepared_ not serialized — After deserialization the flag resets to false, causing unnecessary re-ranking if further reductions are attempted. This is not incorrect but is wasteful.

The test coverage is solid — it covers the happy path (build → reduce → serialize → deserialize → search), repeated reductions, and invalid input rejection. The performance data in the PR description is compelling (2.47x construction speedup).

[LHT129]

Overall this is a well-structured PR. The degree reduction capability is cleanly separated into its own translation unit, the public API surface on HGraph is minimal (3 methods), and the guard conditions in can_reduce_max_degree_unlocked are thorough. The test covers the happy path including successive reductions, serialization round-trip, and invalid input rejection.

The existing inline comments cover the main areas for improvement: parallel_for exception safety, the O(N) linear scan in rank_graph, the flatten parameter naming, and the non-persisted degree_reduction_prepared_ flag. No additional blocking issues found.