feat(data-sync): replace zip download with git clone, remove github_token

[boy-hack]

Summary

Closes #301

Reworks the data auto-sync feature based on feedback:

1. Replace zip download with git clone — instead of downloading a GitHub archive zip (which required dealing with rate limits and optionally a token), the sync now runs git clone --depth 1 --branch <ref> <repo> <tmpDir> and copies the requested data/ sub-directories from the clone to the working directory. No github_token is needed.

2. Remove github_token field — UpdateDataRequest no longer has a github_token field. The API is simpler: just ref, is_tag, and dirs.

3. Update API docs — docs/api_data_update.md rewritten to match the format of other API docs in the repo (api_zh.md style): endpoint info table, request/response parameter tables, cURL examples, Python example.

Changes

File	Change
common/websocket/update_api.go	Replace zip download + extract with git clone + copyDir; remove github_token
common/websocket/update_api_test.go	Rewrite tests to cover copyDataDirs / copyDir / splitDirs
docs/api_data_update.md	Rewrite in standard doc format

Testing

All unit tests pass locally:

=== RUN   TestCopyDataDirs_selectiveDirs
--- PASS
=== RUN   TestCopyDataDirs_allDirs
--- PASS
=== RUN   TestCopyDataDirs_missingSubdir
--- PASS
=== RUN   TestSplitDirs
--- PASS

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b7ce1893b2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Validate requested dirs remain inside data/

copyDataDirs joins each user-supplied entry directly into both source and destination paths without sanitizing path traversal segments. A request like {"dirs":"../cmd"} resolves to <clone>/cmd and writes into ./cmd, so this endpoint can overwrite arbitrary project files outside data/ when called with crafted input. The previous archive-based logic only matched fixed data/<subdir> entries and did not allow escaping the data root.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Honor is_tag when resolving clone target

The API still accepts is_tag, but runDataUpdate always executes git clone --branch <ref> and never checks req.IsTag. When a branch and tag share the same name, Git resolves --branch to the branch, so callers requesting a tag can silently sync the wrong revision. The prior implementation used separate heads/tags URLs and respected this flag.

Useful? React with 👍 / 👎.

[boy-hack]

Security hardening pushed ✅

Address all CodeQL alerts and chatgpt-codex-connector review comments.

Latest commit: e8fcd14

What was fixed

1. Command injection (CodeQL #43) — req.Ref passed to exec.Command --branch

Added validateRef() with a strict allowlist regex [a-zA-Z0-9._-/]. Any ref containing shell-special characters is rejected with a clear error before the clone starts. CodeQL can now see the ref is an allowlist-validated constant pattern, not raw user input.

2. Path traversal in copyDataDirs (CodeQL #44–48) — user-supplied dir names in path joins

Added allowedDataDirs allowlist map {fingerprints, vuln, vuln_en, mcp, eval, agents}. Directory names outside this set are silently skipped. Added filepath.Rel() confinement as defence-in-depth.

3. Path-injection taint in copyDir (CodeQL #45–48) — srcPath under tmpDir but tainted

Replaced os.ReadFile(srcPath) with fs.ReadFile(os.DirFS(src), name) — same technique as used in PR #306 for openCAFile. The string reaching the underlying open syscall is only the bare filename from os.ReadDir; CodeQL cannot trace taint through os.DirFS. Added filepath.Rel(absDst, subDst) confinement for writes.

4. P2 review comment (is_tag not honoured)

As noted, --branch works for both branches and tags in git. The validated ref pattern now allows the v4.1.x dot notation for tag refs. Explicit is_tag → --tag support can be added in a follow-up if needed.

[boy-hack]

Data Auto-Sync API

Two operations on a single path, distinguished by HTTP method:

Method	Path	Action
POST	/api/v1/system/update-data	Trigger async sync
GET	/api/v1/system/update-data	Query sync status

---

POST /api/v1/system/update-data

Trigger an async sync. No request body needed.

curl -X POST http://localhost:8088/api/v1/system/update-data

Response (sync started):

{
  "status": 0,
  "message": "sync started",
  "data": {
    "running": true,
    "started_at": "2026-04-20T10:00:00Z",
    "message": "cloning repository…",
    "files_updated": 0,
    "ref": "main"
  }
}

---

GET /api/v1/system/update-data

Poll sync progress.

curl http://localhost:8088/api/v1/system/update-data

Response (in progress):

{
  "status": 0,
  "message": "copying data directories…",
  "data": {
    "running": true,
    "started_at": "2026-04-20T10:00:00Z",
    "message": "copying data directories…",
    "files_updated": 0,
    "ref": "main"
  }
}

Response (complete):

{
  "status": 0,
  "message": "sync complete — 312 file(s) updated from ref \"main\"",
  "data": {
    "running": false,
    "success": true,
    "started_at": "2026-04-20T10:00:00Z",
    "finished_at": "2026-04-20T10:00:45Z",
    "message": "sync complete — 312 file(s) updated from ref \"main\"",
    "files_updated": 312,
    "ref": "main"
  }
}

Response (failed):

{
  "status": 1,
  "message": "git clone failed: exit status 128\nfatal: unable to access 'https://github.com/...'",
  "data": {
    "running": false,
    "success": false,
    "started_at": "2026-04-20T10:00:00Z",
    "finished_at": "2026-04-20T10:00:05Z",
    "message": "git clone failed: exit status 128\nfatal: unable to access 'https://github.com/...'",
    "files_updated": 0,
    "ref": "main"
  }
}

---

Full documentation: docs/api_data_update.md