Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion cmake/VSAGOptions.cmake
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,8 @@ option (DISABLE_SVE_FORCE "Force disable sve instructions" OFF)

set (NUM_BUILDING_JOBS "4" CACHE STRING "Default compilation parallelism for third-party builds.")
set (BUILD_INFO_DIR "${CMAKE_BINARY_DIR}/.vsag-build-info" CACHE PATH "Metadata directory for ExternalProject state.")
set (DOWNLOAD_DIR "${CMAKE_BINARY_DIR}/.vsag-downloads" CACHE PATH "Download cache directory for ExternalProject archives.")
set (DOWNLOAD_DIR "${CMAKE_BINARY_DIR}/.vsag-downloads" CACHE PATH
"Download cache directory for third-party artifacts.")
set (BUILDING_PATH "" CACHE STRING "Optional PATH prefix for third-party build tools.")

# Policy for resolving third-party dependencies from the host system instead of
Expand Down
41 changes: 28 additions & 13 deletions docs/docs/en/src/development/offline_build.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,15 @@ The resolution order is therefore:
**not** user-configurable — for a fully internal mirror, use the environment
variable.

On `main`, CRoaring is the exception to this archive-oriented flow. VSAG
downloads the three official v3.0.1 amalgamation assets directly, and its
override may name a base directory or URL containing `roaring.c`, `roaring.h`,
and `roaring.hh`. Existing overrides that name the complete v3.0.1 source
archive remain supported. After a base override, CRoaring falls back to the
versioned official GitHub release; it does not use the archive-oriented Aliyun
fallback. Set the override to an internal mirror or local directory when GitHub
is unavailable.

> **Availability:** the `VSAG_THIRDPARTY_*` override is available on `main` and on
> the `0.15`, `0.16`, `0.17`, and `0.18` release lines — see
> [Version availability](#version-availability).
Expand All @@ -50,11 +59,13 @@ The resolution order is therefore:
- **The value may be a local path or a URL.** Accepted forms include an
absolute filesystem path (`/data/deps/fmt-10.2.1.tar.gz`), a `file://` URL, or
any `http(s)://` URL — including an internal HTTP server or an OSS / S3 bucket.
- **The archive hash is still verified.** Each dependency declares a
`URL_HASH` (MD5 or SHA256). Your mirrored / local archive must be **byte
identical to the upstream archive**, otherwise CMake aborts with a hash
mismatch. The simplest safe approach is to download the exact upstream file
once and re-host it unchanged.
On `main`, CRoaring additionally accepts a local directory or base HTTP(S) URL;
the previous v3.0.1 source-archive form remains supported.
- **The artifact hash is still verified.** Each dependency declares an MD5 or
SHA-256 hash. Your mirrored / local files must be **byte identical to the
upstream artifacts**, otherwise CMake aborts with a hash mismatch. The
simplest safe approach is to download the exact upstream files once and
re-host them unchanged.
- **Overrides are read at configure time.** If you change a variable after a
previous configure, re-run CMake configure or run `make clean` first so the
new value takes effect.
Expand All @@ -71,7 +82,7 @@ The resolution order is therefore:

## Environment variables

| Environment variable | Library | Upstream archive to mirror | Pulled in when |
| Environment variable | Library | Upstream files to mirror | Pulled in when |
| --- | --- | --- | --- |
| `VSAG_THIRDPARTY_JSON` | nlohmann/json 3.11.3 | `github.com/nlohmann/json/.../v3.11.3.tar.gz` | always |
| `VSAG_THIRDPARTY_ANTLR4` | ANTLR4 runtime 4.13.2 | `github.com/antlr/antlr4/.../4.13.2.tar.gz` | always |
Expand All @@ -81,7 +92,7 @@ The resolution order is therefore:
| `VSAG_THIRDPARTY_FMT` | fmt 10.2.1 | `github.com/fmtlib/fmt/.../10.2.1.tar.gz` | always (unless system fmt) |
| `VSAG_THIRDPARTY_THREAD_POOL` | log4cplus/ThreadPool | `github.com/log4cplus/ThreadPool/archive/3507796e...tar.gz` | always |
| `VSAG_THIRDPARTY_TSL` | Tessil/robin-map 1.4.0 | `github.com/Tessil/robin-map/.../v1.4.0.tar.gz` | always |
| `VSAG_THIRDPARTY_ROARINGBITMAP` | CRoaring 3.0.1 | `github.com/RoaringBitmap/CRoaring/.../v3.0.1.tar.gz` | always |
| `VSAG_THIRDPARTY_ROARINGBITMAP` | CRoaring 3.0.1 | release `roaring.c`, `roaring.h`, and `roaring.hh` | always |
| `VSAG_THIRDPARTY_CATCH2` | Catch2 3.7.1 | `github.com/catchorg/Catch2/.../v3.7.1.tar.gz` | `ENABLE_TESTS=ON` |
| `VSAG_THIRDPARTY_HDF5` | HDF5 1.14.4 | `github.com/HDFGroup/hdf5/.../hdf5_1.14.4.tar.gz` | `ENABLE_TOOLS=ON` (+ C++11 ABI) |
| `VSAG_THIRDPARTY_ARGPARSE` | p-ranav/argparse 3.1 | `github.com/p-ranav/argparse/.../v3.1.tar.gz` | `ENABLE_TOOLS=ON` (+ C++11 ABI) |
Expand All @@ -90,7 +101,7 @@ The resolution order is therefore:
| `VSAG_THIRDPARTY_HTTPLIB` | cpp-httplib 0.35.0 | `github.com/yhirose/cpp-httplib/.../v0.35.0.tar.gz` | `ENABLE_TOOLS=ON` (+ C++11 ABI) |
| `VSAG_THIRDPARTY_PYBIND11` | pybind11 2.11.1 | `github.com/pybind/pybind11/.../v2.11.1.tar.gz` | Python bindings (`pyvsag` / `ENABLE_PYBINDS=ON`) |

> The exact upstream URL **and** the expected `URL_HASH` for each dependency are
> The exact upstream URLs **and** the expected hashes for each dependency are
> the single source of truth in the corresponding
> [`extern/<lib>/<lib>.cmake`](https://github.com/antgroup/vsag/tree/main/extern)
> file. Check that file when mirroring, especially after a version bump.
Expand Down Expand Up @@ -134,7 +145,8 @@ export VSAG_THIRDPARTY_CPUINFO=$VSAG_MIRROR/cpuinfo-ca678952.tar.gz
export VSAG_THIRDPARTY_FMT=$VSAG_MIRROR/fmt-10.2.1.tar.gz
export VSAG_THIRDPARTY_THREAD_POOL=$VSAG_MIRROR/thread_pool-3507796e.tar.gz
export VSAG_THIRDPARTY_TSL=$VSAG_MIRROR/robin-map-1.4.0.tar.gz
export VSAG_THIRDPARTY_ROARINGBITMAP=$VSAG_MIRROR/CRoaring-3.0.1.tar.gz
# This base URL must contain roaring.c, roaring.h, and roaring.hh.
export VSAG_THIRDPARTY_ROARINGBITMAP=$VSAG_MIRROR/CRoaring-3.0.1

make release
```
Expand All @@ -157,7 +169,8 @@ export VSAG_THIRDPARTY_CPUINFO=/data/vsag-deps/cpuinfo-ca678952.tar.gz
export VSAG_THIRDPARTY_FMT=/data/vsag-deps/fmt-10.2.1.tar.gz
export VSAG_THIRDPARTY_THREAD_POOL=/data/vsag-deps/thread_pool-3507796e.tar.gz
export VSAG_THIRDPARTY_TSL=/data/vsag-deps/robin-map-1.4.0.tar.gz
export VSAG_THIRDPARTY_ROARINGBITMAP=/data/vsag-deps/CRoaring-3.0.1.tar.gz
# This directory must contain roaring.c, roaring.h, and roaring.hh.
export VSAG_THIRDPARTY_ROARINGBITMAP=/data/vsag-deps/CRoaring-3.0.1

make release
```
Expand Down Expand Up @@ -201,9 +214,11 @@ for the list of dependencies that currently support system reuse.
## Version availability

The per-dependency `VSAG_THIRDPARTY_*` override is available on the `main`
development line and on the `0.15`, `0.16`, `0.17`, and `0.18` release lines, so
local-path and internal-mirror overrides behave the same way across all of them.
It was introduced on `main` by
development line and on the `0.15`, `0.16`, `0.17`, and `0.18` release lines.
On those release lines, `VSAG_THIRDPARTY_ROARINGBITMAP` must name the single
v3.0.1 source archive; the CRoaring directory/base-URL form described above is
specific to `main`. Other dependency overrides use the same archive form on all
of these lines. The override mechanism was introduced on `main` by
[#1606](https://github.com/antgroup/vsag/pull/1606) and backported to the release
lines (tracked in [#2308](https://github.com/antgroup/vsag/issues/2308)). The
built-in upstream + Aliyun OSS mirror fallback remains present on every line, and
Expand Down
31 changes: 22 additions & 9 deletions docs/docs/zh/src/development/offline_build.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,12 @@ ExternalProject_Add (antlr4
(`vsagcache.oss-rg-china-mainland.aliyuncs.com`)。该兜底地址始终存在,在中国大陆 /
弱网环境下很有帮助,但**不可由用户配置**——若要使用纯内网镜像,请使用环境变量。

在 `main` 分支上,CRoaring 是上述压缩包流程的例外。VSAG 会直接下载三个官方 v3.0.1
合并文件;其覆盖项可以指向包含 `roaring.c`、`roaring.h` 与 `roaring.hh` 的基础目录或 URL。
原有的、指向完整 v3.0.1 源码压缩包的覆盖项仍受支持。尝试基础路径覆盖项后,CRoaring 会
回退到带版本的 GitHub 官方发布地址;它不使用面向压缩包的阿里云回退地址。
GitHub 不可用时,请将覆盖项设置为内网镜像或本地目录。

> **可用版本:** `VSAG_THIRDPARTY_*` 覆盖能力在 `main` 分支以及 `0.15`、`0.16`、
> `0.17`、`0.18` 发布线上均可用——详见[版本可用性](#版本可用性)。

Expand All @@ -43,8 +49,10 @@ ExternalProject_Add (antlr4
- **取值可以是本地路径,也可以是 URL。** 支持绝对文件路径
(`/data/deps/fmt-10.2.1.tar.gz`)、`file://` URL,或任意 `http(s)://` URL——
包括内网 HTTP 服务或 OSS / S3 存储桶。
- **依然会校验压缩包哈希。** 每个依赖都声明了 `URL_HASH`(MD5 或 SHA256)。你镜像 /
本地的压缩包必须与**上游压缩包逐字节一致**,否则 CMake 会因哈希不匹配而中止。最稳妥的
在 `main` 分支上,CRoaring 还支持本地目录或 HTTP(S) 基础 URL;原有的 v3.0.1
源码压缩包形式仍然受支持。
- **依然会校验文件哈希。** 每个依赖都声明了 MD5 或 SHA-256 哈希。你镜像 /
本地的文件必须与**上游文件逐字节一致**,否则 CMake 会因哈希不匹配而中止。最稳妥的
做法是把上游原始文件下载一次,原封不动地重新托管。
- **覆盖项在配置阶段读取。** 如果你在上一次配置之后修改了变量,请重新执行 CMake 配置或
先运行 `make clean`,新值才会生效。
Expand All @@ -59,7 +67,7 @@ ExternalProject_Add (antlr4

## 环境变量

| 环境变量 | 库 | 需镜像的上游压缩包 | 何时被拉取 |
| 环境变量 | 库 | 需镜像的上游文件 | 何时被拉取 |
| --- | --- | --- | --- |
| `VSAG_THIRDPARTY_JSON` | nlohmann/json 3.11.3 | `github.com/nlohmann/json/.../v3.11.3.tar.gz` | 始终 |
| `VSAG_THIRDPARTY_ANTLR4` | ANTLR4 runtime 4.13.2 | `github.com/antlr/antlr4/.../4.13.2.tar.gz` | 始终 |
Expand All @@ -69,7 +77,7 @@ ExternalProject_Add (antlr4
| `VSAG_THIRDPARTY_FMT` | fmt 10.2.1 | `github.com/fmtlib/fmt/.../10.2.1.tar.gz` | 始终(除非使用系统 fmt) |
| `VSAG_THIRDPARTY_THREAD_POOL` | log4cplus/ThreadPool | `github.com/log4cplus/ThreadPool/archive/3507796e...tar.gz` | 始终 |
| `VSAG_THIRDPARTY_TSL` | Tessil/robin-map 1.4.0 | `github.com/Tessil/robin-map/.../v1.4.0.tar.gz` | 始终 |
| `VSAG_THIRDPARTY_ROARINGBITMAP` | CRoaring 3.0.1 | `github.com/RoaringBitmap/CRoaring/.../v3.0.1.tar.gz` | 始终 |
| `VSAG_THIRDPARTY_ROARINGBITMAP` | CRoaring 3.0.1 | release 中的 `roaring.c`、`roaring.h` 与 `roaring.hh` | 始终 |
| `VSAG_THIRDPARTY_CATCH2` | Catch2 3.7.1 | `github.com/catchorg/Catch2/.../v3.7.1.tar.gz` | `ENABLE_TESTS=ON` |
| `VSAG_THIRDPARTY_HDF5` | HDF5 1.14.4 | `github.com/HDFGroup/hdf5/.../hdf5_1.14.4.tar.gz` | `ENABLE_TOOLS=ON`(且 C++11 ABI) |
| `VSAG_THIRDPARTY_ARGPARSE` | p-ranav/argparse 3.1 | `github.com/p-ranav/argparse/.../v3.1.tar.gz` | `ENABLE_TOOLS=ON`(且 C++11 ABI) |
Expand All @@ -78,7 +86,7 @@ ExternalProject_Add (antlr4
| `VSAG_THIRDPARTY_HTTPLIB` | cpp-httplib 0.35.0 | `github.com/yhirose/cpp-httplib/.../v0.35.0.tar.gz` | `ENABLE_TOOLS=ON`(且 C++11 ABI) |
| `VSAG_THIRDPARTY_PYBIND11` | pybind11 2.11.1 | `github.com/pybind/pybind11/.../v2.11.1.tar.gz` | Python 绑定(`pyvsag` / `ENABLE_PYBINDS=ON`) |

> 每个依赖确切的上游 URL **以及**期望的 `URL_HASH`,其唯一权威来源是对应的
> 每个依赖确切的上游 URL **以及**期望的哈希,其唯一权威来源是对应的
> [`extern/<lib>/<lib>.cmake`](https://github.com/antgroup/vsag/tree/main/extern)
> 文件。镜像时(尤其是版本升级后)请以该文件为准。

Expand Down Expand Up @@ -118,7 +126,8 @@ export VSAG_THIRDPARTY_CPUINFO=$VSAG_MIRROR/cpuinfo-ca678952.tar.gz
export VSAG_THIRDPARTY_FMT=$VSAG_MIRROR/fmt-10.2.1.tar.gz
export VSAG_THIRDPARTY_THREAD_POOL=$VSAG_MIRROR/thread_pool-3507796e.tar.gz
export VSAG_THIRDPARTY_TSL=$VSAG_MIRROR/robin-map-1.4.0.tar.gz
export VSAG_THIRDPARTY_ROARINGBITMAP=$VSAG_MIRROR/CRoaring-3.0.1.tar.gz
# 此基础 URL 必须包含 roaring.c、roaring.h 与 roaring.hh。
export VSAG_THIRDPARTY_ROARINGBITMAP=$VSAG_MIRROR/CRoaring-3.0.1

make release
```
Expand All @@ -140,7 +149,8 @@ export VSAG_THIRDPARTY_CPUINFO=/data/vsag-deps/cpuinfo-ca678952.tar.gz
export VSAG_THIRDPARTY_FMT=/data/vsag-deps/fmt-10.2.1.tar.gz
export VSAG_THIRDPARTY_THREAD_POOL=/data/vsag-deps/thread_pool-3507796e.tar.gz
export VSAG_THIRDPARTY_TSL=/data/vsag-deps/robin-map-1.4.0.tar.gz
export VSAG_THIRDPARTY_ROARINGBITMAP=/data/vsag-deps/CRoaring-3.0.1.tar.gz
# 此目录必须包含 roaring.c、roaring.h 与 roaring.hh。
export VSAG_THIRDPARTY_ROARINGBITMAP=/data/vsag-deps/CRoaring-3.0.1

make release
```
Expand Down Expand Up @@ -178,8 +188,11 @@ make release

## 版本可用性

按依赖配置的 `VSAG_THIRDPARTY_*` 覆盖能力在 `main` 开发线以及 `0.15`、`0.16`、`0.17`、
`0.18` 发布线上均可用,因此本地路径与内网镜像覆盖在所有这些分支上的行为完全一致。该能力最初由
按依赖配置的 `VSAG_THIRDPARTY_*` 覆盖能力在 `main` 开发线以及 `0.15`、`0.16`、
`0.17`、`0.18` 发布线上均可用。在这些发布线上,
`VSAG_THIRDPARTY_ROARINGBITMAP` 必须指向单个 v3.0.1 源码压缩包;上文所述 CRoaring
目录 / 基础 URL 形式仅适用于 `main`。其他依赖在这些分支上均使用相同的压缩包形式。
该能力最初由
[#1606](https://github.com/antgroup/vsag/pull/1606) 在 `main` 引入,并已合入各发布线(跟踪于
[#2308](https://github.com/antgroup/vsag/issues/2308))。内置的“上游 + 阿里云 OSS 镜像”兜底在
每条线上依然保留;若你不想镜像某个依赖,也仍可使用[复用系统库](#备选方案复用系统库)。
Loading
Loading