Skip to content

Resolve relative submodule URLs for mirrors - #4287

Open
davido wants to merge 1 commit into
buildkite:mainfrom
davido:issue-2815-resolve-relative-submodule-mirrors
Open

Resolve relative submodule URLs for mirrors#4287
davido wants to merge 1 commit into
buildkite:mainfrom
davido:issue-2815-resolve-relative-submodule-mirrors

Conversation

@davido

@davido davido commented Aug 29, 2026

Copy link
Copy Markdown

Description

When git-mirrors-path is enabled, the agent pre-populates a Git mirror for each
submodule enumerated from .gitmodules before running git submodule update.
Submodule URLs in .gitmodules may be relative to the superproject remote,
e.g. url = ../java-prettify. The mirror prefetch path passed those raw values
directly to git clone --mirror while cd'd into the mirror cache directory, so
../java-prettify was interpreted as a local filesystem path under
git-mirrors-path instead of a repository relative to the superproject remote —
and checkout failed before any job command ran.

This PR resolves relative submodule URLs against e.Repository only for mirror
selection/population
. The actual git submodule update command is left
unchanged (aside from whether it receives --reference <mirrorDir>), so Git
remains the authority on submodule checkout semantics.

Resolution is fail-open for relative submodules: if URL resolution or the
resolved mirror update fails, the agent warns and falls back to a plain
git submodule update (no --reference), letting Git apply its own
authoritative relative-URL handling. Absolute submodule mirror failures keep the
existing hard-fail behavior.

Alternatives considered: a minimal fix that simply skipped mirrors for any
relative submodule URL. It was safe but gave up the mirror optimization even when
the agent can resolve the URL correctly. The chosen approach keeps that safety net
(via the fail-open fallback) while still getting mirror performance in the common
case where the resolver agrees with Git.

Context

Example resolution: https://gerrit.googlesource.com/gerrit.git + ../java-prettify
https://gerrit.googlesource.com/java-prettify.

Changes

  • internal/job/checkout.go — in updateGitSubmodules, resolve relative
    submodule URLs before the mirror update; add fail-open fallback for relative
    submodules (resolver error or resolved-mirror failure → plain submodule update);
    preserve hard-fail for absolute submodules.
  • internal/job/git.go — add resolveGitSubmoduleURL and
    isRelativeSubmoduleURL. Resolution preserves the original URL form:
    https/ssh via url.URL, scp-like (git@host:path, including absolute
    git@host:/abs/path) via string reconstruction, and local paths via
    filepath.Join. scpLikeURLPattern now captures the leading slash after the
    colon so absolute scp remote paths survive resolution; parseGittableURL is
    updated to preserve that slash and is behavior-identical for all other inputs.
  • Tracing/logging — the submodule mirror span reports git.repo as the
    resolved URL and adds git.raw_repo (the raw .gitmodules value) only when it
    differs; a comment logs the resolution. Repository URLs are credential-redacted
    in logs and span attributes.
  • Tests — resolver unit coverage (absolute/relative × HTTPS/SSH/scp-like/scp
    absolute-path/local, cleaned paths, bare name unchanged, malformed-URL parse
    error, relative-URL detection) plus a parseGittableURL absolute-scp case; two
    integration tests (..._WithRelativeSubmoduleURL_WithGitMirrors and
    ..._WithRelativeSubmoduleURLFallsBackWhenMirrorFails_WithGitMirrors) sharing a
    setupRelativeSubmoduleFixture helper.

Testing

  • Tests have run locally (with go test ./...). Buildkite employees may check this if the pipeline has run automatically.
  • Code is formatted (with go tool gofumpt -extra -w .)

Focused runs:

go test ./internal/job -run 'TestResolveGitSubmoduleURL|TestIsRelativeSubmoduleURL|TestParseGittableURL'
go test ./internal/job/integration -run 'TestCheckingOutLocalGitProjectWith(Submodules|RelativeSubmoduleURL|RelativeSubmoduleURLFallsBackWhenMirrorFails|SubmodulesDisabled)_WithGitMirrors'
go vet ./internal/job/...
go build ./...
git diff --check

Affiliation (optional, external contributors)

External contributor, filing on behalf of the Gerrit project's Bazel downstream Buildkite CI pipeline.

Submodule URLs in .gitmodules may be relative to the superproject
remote. The mirror prefetch path previously passed those raw values
directly to git clone --mirror from the mirror cache directory. Values
such as ../java-prettify were therefore interpreted as local paths
under the mirror cache, not as repositories relative to the
superproject remote.

Resolve relative submodule URLs before updating a submodule mirror,
while leaving the actual git submodule update command to Git. If
relative URL resolution or the resolved mirror update fails, warn and
fall back to a plain submodule update so Git can apply its
authoritative relative URL semantics. Absolute submodule mirror
failures retain the existing hard-fail behavior.

Add resolver unit coverage for absolute, relative, HTTP(S), SSH,
scp-like, local path, malformed URL, and relative-detection cases. Add
integration coverage proving relative submodules are mirror-cloned via
the resolved repository URL, raw relative URLs are never passed to git
clone --mirror, and failed resolved mirrors fall back to non-referenced
submodule update.

Closes buildkite#4282.

Refs bazelbuild/continuous-integration#2815.
@swebb

swebb commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Thanks for opening this PR. I’ve created a ticket to track it internally.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Git mirror checkout fails for relative submodule URLs

2 participants