[WIP]fix(eval): correct peak memory measurement

.github/workflows/pr-ci.yml

@@ -27,6 +27,7 @@ jobs:
               - 'src/**'
               - 'include/**'
               - 'tests/**'
+              - 'tools/**'
               - 'CMakeLists.txt'
               - 'Makefile'
               - 'extern/**'
@@ -165,6 +166,8 @@ jobs:
           key: build-macos-${{ hashFiles('./CMakeLists.txt') }}-${{ hashFiles('./.circleci/fresh_ci_cache.commit') }}
       - name: Make Asan
         run: export CMAKE_GENERATOR="Ninja" CMAKE_EXPORT_COMPILE_COMMANDS=ON; make asan VSAG_ENABLE_EXAMPLES=ON COMPILE_JOBS=3
+      - name: Run Eval Tests
+        run: ./build/tests/unittests "[eval]"
 
   build-python-x86:
     name: Python Build X86

docs/docs/en/src/resources/eval.md

@@ -85,7 +85,45 @@ Note: under `global.exporters`, each entry is a **named** exporter (a YAML map),
 - **Efficiency**: QPS, TPS
 - **Quality**: average recall and quantile recall (P0/P10/P50/P90...)
 - **Latency**: average, P50/P95/P99
-- **Resource**: peak memory usage
+- **Resource**: sampled process RSS peak and index-owned memory usage
+
+## Memory Metrics
+
+Memory monitoring samples the process resident set size (RSS) every 5 ms while the measured phase
+is running. The human-readable `memory_peak(build)` and `memory_peak(search)` fields report the
+largest sampled RSS increase above the phase baseline, using 1024-based units. A sampled peak can
+miss an allocation that exists for less than one sampling interval.
+
+The phase boundaries are:
+
+- `build`: from immediately before `Index::Build()` until it returns. Dataset loading and index
+  serialization are outside the interval.
+- `search`: from immediately before the first measured query pass until that pass finishes. Index
+  deserialization is outside the interval. When a latency/QPS measurement pass is present, RSS
+  sampling runs alongside that existing first pass instead of adding a separate pass that would
+  pre-warm later measurements. If there is no latency pass, a memory-only pass runs before recall
+  so recall bookkeeping is not counted. KNN result statistics are collected afterward, outside
+  all memory and performance monitor intervals.
+
+JSON output also includes exact numeric fields for each `<phase>` (`build` or `search`):
+
+| Field | Meaning |
+| --- | --- |
+| `memory_rss_baseline_bytes(<phase>)` | Process RSS at the start of the phase |
+| `memory_rss_peak_bytes(<phase>)` | Largest process RSS sampled during the phase |
+| `memory_peak_delta_bytes(<phase>)` | Peak RSS minus baseline RSS, clamped at zero |
+| `memory_peak_sample_count(<phase>)` | Number of successful RSS samples |
+| `memory_peak_failed_sample_count(<phase>)` | Number of failed RSS samples |
+| `memory_peak_available(<phase>)` | Whether a valid baseline and sampler were available |
+| `memory_peak_error(<phase>)` | Sampling error, or an empty string on success |
+
+`index_memory(B)` reports memory owned by the index according to `Index::GetMemoryUsage()`, while
+`memory_detail(B)` contains the available component breakdown. These index-level values are
+separate from process RSS, which also includes the dataset, evaluator bookkeeping, allocator
+overhead, and other process state.
+
+On an unsupported platform or when the baseline cannot be read, the human-readable value is
+`N/A`, `memory_peak_available(<phase>)` is `false`, and the error field explains the failure.
 
 ## Search Modes
 

docs/docs/zh/src/resources/eval.md

@@ -80,7 +80,42 @@ eval_case1:
 - **效率**：QPS、TPS
 - **效果**：平均召回率、分位召回率（P0/P10/P50/P90...）
 - **延迟**：平均延迟、P50/P95/P99 延迟
-- **资源**：峰值内存占用
+- **资源**：进程 RSS 采样峰值与索引自身内存占用
+
+## 内存指标
+
+内存监控在被测阶段运行期间每 5 ms 采样一次进程常驻集大小（RSS）。
+`memory_peak(build)` 和 `memory_peak(search)` 是便于阅读的兼容字段，表示相对于阶段
+基线的最大 RSS 采样增量，单位按 1024 进位。持续时间短于一个采样周期的瞬时分配可能
+不会被捕获。
+
+测量边界如下：
+
+- `build`：从调用 `Index::Build()` 前开始，到该调用返回为止；数据集加载和索引序列化
+  不在测量区间内。
+- `search`：从第一个被测查询批次开始前到该批次完成为止；索引反序列化不在测量区间内。
+  存在延迟/QPS 测量轮次时，RSS 采样与原有的第一个查询轮次同时运行，不再额外执行一个
+  会预热后续指标的内存轮次。没有延迟轮次时，内存专用轮次会先于召回率轮次执行，避免
+  把召回率统计开销计入 RSS。KNN 结果统计在此后独立采集，不属于内存或性能监控区间。
+
+JSON 输出还会为每个 `<phase>`（`build` 或 `search`）提供以下精确数值字段：
+
+| 字段 | 含义 |
+| --- | --- |
+| `memory_rss_baseline_bytes(<phase>)` | 阶段开始时的进程 RSS |
+| `memory_rss_peak_bytes(<phase>)` | 阶段内采样到的最大进程 RSS |
+| `memory_peak_delta_bytes(<phase>)` | 峰值 RSS 减去基线 RSS，最小为零 |
+| `memory_peak_sample_count(<phase>)` | 成功取得 RSS 的采样次数 |
+| `memory_peak_failed_sample_count(<phase>)` | RSS 采样失败次数 |
+| `memory_peak_available(<phase>)` | 是否取得有效基线并成功启动采样器 |
+| `memory_peak_error(<phase>)` | 采样错误；成功时为空字符串 |
+
+`index_memory(B)` 通过 `Index::GetMemoryUsage()` 报告索引自身占用，
+`memory_detail(B)` 则提供可用的组件明细。这些索引级指标与进程 RSS 分开：进程 RSS
+还会包含数据集、评估器自身开销、allocator 开销和进程中的其他状态。
+
+在不支持的平台或无法读取基线时，便于阅读的字段为 `N/A`，
+`memory_peak_available(<phase>)` 为 `false`，错误字段会说明失败原因。
 
 ## 搜索模式
 

tests/CMakeLists.txt

@@ -15,19 +15,32 @@
 
 # fixtures is already added by top-level CMakeLists.txt before src
 
+find_package (Threads REQUIRED)
 
 #  unittests
 add_executable (unittests
         test_main.cpp
+        ${PROJECT_SOURCE_DIR}/tools/eval/case/search_pass_runner.cpp
+        ${PROJECT_SOURCE_DIR}/tools/eval/case/search_pass_runner_test.cpp
+        ${PROJECT_SOURCE_DIR}/tools/eval/monitor/memory_peak_monitor_test.cpp
+        ${PROJECT_SOURCE_DIR}/tools/eval/monitor/memory_peak_monitor.cpp
+        ${PROJECT_SOURCE_DIR}/tools/eval/monitor/monitor.cpp
         $<TARGET_OBJECTS:fixtures_allocator>
         $<TARGET_OBJECTS:fixtures_core>
         $<TARGET_OBJECTS:fixtures_data>
         $<TARGET_OBJECTS:fixtures_framework>
 )
 
+target_include_directories (unittests PRIVATE
+        ${PROJECT_SOURCE_DIR}/tools/eval/case
+        ${PROJECT_SOURCE_DIR}/tools/eval/monitor
+)
+
 target_link_libraries (unittests
         PRIVATE
-        Catch2::Catch2 
+        Catch2::Catch2
+        Threads::Threads
+        nlohmann_json::nlohmann_json
         simd_test vsag_test algorithm_test factory_test attr_test
         datacell_test quantizer_test storage_test io_test utils_test impl_test
         vsag_static

tools/eval/CMakeLists.txt

@@ -16,6 +16,7 @@
 
 set (eval_srcs
     case/eval_case.cpp
+    case/search_pass_runner.cpp
     case/search_eval_case.cpp
     case/build_eval_case.cpp
     exporter/exporter.cpp

tools/eval/case/build_eval_case.cpp

@@ -16,6 +16,7 @@
 #include "./build_eval_case.h"
 
 #include <algorithm>
+#include <exception>
 #include <filesystem>
 #include <utility>
 
@@ -25,11 +26,54 @@
 
 namespace vsag::eval {
 
+namespace {
+
+class StartedMonitorGuard {
+public:
+    explicit StartedMonitorGuard(std::vector<MonitorPtr>& monitors) : monitors_(monitors) {
+    }
+
+    ~StartedMonitorGuard() {
+        while (stopped_monitor_count_ < started_monitor_count_) {
+            const auto monitor_id = started_monitor_count_ - stopped_monitor_count_ - 1;
+            ++stopped_monitor_count_;
+            try {
+                monitors_[monitor_id]->Stop();
+            } catch (...) {
+            }
+        }
+    }
+
+    void
+    StartAll() {
+        for (auto& monitor : monitors_) {
+            monitor->Start();
+            ++started_monitor_count_;
+        }
+    }
+
+    void
+    StopNext() {
+        const auto monitor_id = started_monitor_count_ - stopped_monitor_count_ - 1;
+        monitors_[monitor_id]->Stop();
+        ++stopped_monitor_count_;
+    }
+
+private:
+    std::vector<MonitorPtr>& monitors_;
+    uint64_t started_monitor_count_{0};
+    uint64_t stopped_monitor_count_{0};
+};
+
+}  // namespace
+
 BuildEvalCase::BuildEvalCase(const std::string& dataset_path,
                              const std::string& index_path,
                              vsag::IndexPtr index,
-                             EvalConfig config)
-    : EvalCase(dataset_path, index_path, index), config_(std::move(config)) {
+                             EvalConfig config,
+                             EvalDatasetPtr dataset)
+    : EvalCase(dataset_path, index_path, std::move(index), std::move(dataset)),
+      config_(std::move(config)) {
     this->init_monitors();
 }
 
@@ -68,16 +112,15 @@ BuildEvalCase::do_build() {
     } else {
         base->SparseVectors((const SparseVector*)this->dataset_ptr_->GetTrain());
     }
-    for (auto& monitor : monitors_) {
-        monitor->Start();
-    }
+    StartedMonitorGuard monitor_guard(monitors_);
+    monitor_guard.StartAll();
     auto build_index = index_->Build(base);
     if (not build_index.has_value()) {
         throw std::runtime_error(build_index.error().message);
     }
-    for (auto& monitor : monitors_) {
-        monitor->Record();
-        monitor->Stop();
+    for (auto monitor = monitors_.rbegin(); monitor != monitors_.rend(); ++monitor) {
+        (*monitor)->Record();
+        monitor_guard.StopNext();
     }
 }
 void
@@ -105,6 +148,11 @@ BuildEvalCase::process_result() {
     result["index_info"] = JsonType::parse(config_.build_param);
     result["action"] = "build";
     result["index"] = config_.index_name;
+    try {
+        result["index_memory(B)"] = this->index_->GetMemoryUsage();
+    } catch (const std::exception& e) {
+        logger_->Error(e.what());
+    }
     try {
         auto detail = this->index_->GetMemoryUsageDetail();
         for (const auto& [name, size] : detail) {

tools/eval/case/build_eval_case.h

@@ -25,7 +25,8 @@ class BuildEvalCase : public EvalCase {
     BuildEvalCase(const std::string& dataset_path,
                   const std::string& index_path,
                   vsag::IndexPtr index,
-                  EvalConfig config);
+                  EvalConfig config,
+                  EvalDatasetPtr dataset = nullptr);
 
     ~BuildEvalCase() override = default;
 

tools/eval/case/build_search_eval_case.h

@@ -25,19 +25,27 @@ class BuildSearchEvalCase : public EvalCase {
     BuildSearchEvalCase(const std::string& dataset_path,
                         const std::string& index_path,
                         vsag::IndexPtr index,
-                        EvalConfig config)
-        : EvalCase(dataset_path, index_path, index) {
-        build_ = EvalCase::MakeInstance(config, "build");
-        search_ = EvalCase::MakeInstance(config, "search");
-        config_ = std::move(config);
+                        EvalConfig config,
+                        EvalDatasetPtr dataset = nullptr)
+        : EvalCase(dataset_path, index_path, nullptr, std::move(dataset)),
+          config_(std::move(config)) {
+        build_ = std::make_shared<BuildEvalCase>(
+            dataset_path_, index_path_, std::move(index), config_, dataset_ptr_);
     }
 
     ~BuildSearchEvalCase() override = default;
 
     JsonType
     Run() override {
-        auto build_result = build_->Run();
-        auto search_result = search_->Run();
+        auto build = std::move(build_);
+        if (build == nullptr) {
+            build = EvalCase::MakeInstance(config_, "build", dataset_ptr_);
+        }
+        auto build_result = build->Run();
+        build.reset();
+
+        auto search = EvalCase::MakeInstance(config_, "search", dataset_ptr_);
+        auto search_result = search->Run();
         return merge_results(build_result, search_result);
     }
 
@@ -50,9 +58,8 @@ class BuildSearchEvalCase : public EvalCase {
     }
 
 private:
-    EvalCasePtr build_;
-    EvalCasePtr search_;
     EvalConfig config_;
+    EvalCasePtr build_;
 };
 
 }  // namespace vsag::eval

tools/eval/case/eval_case.cpp

@@ -26,15 +26,23 @@
 
 namespace vsag::eval {
 
-EvalCase::EvalCase(std::string dataset_path, std::string index_path, vsag::IndexPtr index)
-    : dataset_path_(std::move(dataset_path)), index_path_(std::move(index_path)), index_(index) {
-    this->dataset_ptr_ = EvalDataset::Load(dataset_path_);
+EvalCase::EvalCase(std::string dataset_path,
+                   std::string index_path,
+                   vsag::IndexPtr index,
+                   EvalDatasetPtr dataset)
+    : dataset_path_(std::move(dataset_path)),
+      index_path_(std::move(index_path)),
+      dataset_ptr_(std::move(dataset)),
+      index_(std::move(index)) {
+    if (this->dataset_ptr_ == nullptr) {
+        this->dataset_ptr_ = EvalDataset::Load(dataset_path_);
+    }
     this->logger_ = vsag::Options::Instance().logger();
     this->basic_info_ = this->dataset_ptr_->GetInfo();
 }
 
 EvalCasePtr
-EvalCase::MakeInstance(const EvalConfig& config, std::string type) {
+EvalCase::MakeInstance(const EvalConfig& config, std::string type, EvalDatasetPtr dataset) {
     auto dataset_path = config.dataset_path;
     auto index_path = config.index_path;
     auto index_name = config.index_name;
@@ -48,14 +56,16 @@ EvalCase::MakeInstance(const EvalConfig& config, std::string type) {
     }
 
     if (type == "build") {
-        return std::make_shared<BuildEvalCase>(dataset_path, index_path, index.value(), config);
+        return std::make_shared<BuildEvalCase>(
+            dataset_path, index_path, index.value(), config, std::move(dataset));
     }
     if (type == "search") {
-        return std::make_shared<SearchEvalCase>(dataset_path, index_path, index.value(), config);
+        return std::make_shared<SearchEvalCase>(
+            dataset_path, index_path, index.value(), config, std::move(dataset));
     }
     if (type == "build,search") {
         return std::make_shared<BuildSearchEvalCase>(
-            dataset_path, index_path, index.value(), config);
+            dataset_path, index_path, index.value(), config, std::move(dataset));
     }
     return nullptr;
 }

tools/eval/case/eval_case.h

@@ -31,7 +31,9 @@ using EvalCasePtr = std::shared_ptr<EvalCase>;
 class EvalCase {
 public:
     static EvalCasePtr
-    MakeInstance(const EvalConfig& config, std::string type = "none");
+    MakeInstance(const EvalConfig& config,
+                 std::string type = "none",
+                 EvalDatasetPtr dataset = nullptr);
 
     static void
     MergeJsonType(const JsonType& input, JsonType& output) {
@@ -86,7 +88,10 @@ class EvalCase {
     }
 
 public:
-    explicit EvalCase(std::string dataset_path, std::string index_path, vsag::IndexPtr index);
+    explicit EvalCase(std::string dataset_path,
+                      std::string index_path,
+                      vsag::IndexPtr index,
+                      EvalDatasetPtr dataset = nullptr);
 
     virtual ~EvalCase() = default;