Skip to content

feat(cache): cache v2 in the agent #4237

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ss1909 wants to merge 6 commits into
base: main
Choose a base branch
from
Open

feat(cache): cache v2 in the agent #4237

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
a50ff2c
feat(cache): cache v2 in the agent
ss1909 Aug 12, 2026
5265b8c
integration
ss1909 Aug 13, 2026
d51dc42
integration
ss1909 Aug 14, 2026
359d095
integration - every step
ss1909 Aug 16, 2026
69eefd0
fix windows dir resolution
ss1909 Aug 16, 2026
cbe3873
give race its own cache
ss1909 Aug 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions .buildkite/cache.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
caches:
- name: gomodcache
cache_key:
- v2
- { agent: os }
- { agent: arch }
- { checksum: go.sum }
# One level above the checkout (not inside it -- see docker-compose.yml)
# and not a host bind mount. Resolves the same way whether cwd is /work
# in a container or a Windows build checkout directory.
target_paths:
- ../.gomodcache
Comment thread
buildsworth-bk-app[bot] marked this conversation as resolved.
Outdated
- name: gocache
cache_key:
- v2
- { agent: os }
- { agent: arch }
Comment thread
buildsworth-bk-app[bot] marked this conversation as resolved.

@buildsworth-bk-app buildsworth-bk-app Bot Aug 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: This exact address never changes, and cache save explicitly skips an address that already exists. After the first successful Linux/AMD64 save, gocache is frozen at that snapshot: new dependency or toolchain outputs are rebuilt on every job but can never refresh the shared entry. A changing exact suffix after a fallback boundary would preserve cross-branch reuse while allowing refreshed entries.

@buildsworth-bk-app buildsworth-bk-app Bot Aug 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open — gocache still has a fixed exact key, so an existing entry cannot be refreshed.

@buildsworth-bk-app buildsworth-bk-app Bot Aug 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open — the gocache key is still fixed, so once the first entry exists, later runs cannot refresh it.

@buildsworth-bk-app buildsworth-bk-app Bot Aug 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open — the new platform writers do not change the fixed key, so once each platform’s first entry exists, later runs still cannot refresh it.

@buildsworth-bk-app buildsworth-bk-app Bot Aug 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open — both gocache and the new race cache have fixed exact addresses, so successful entries cannot be refreshed.

@buildsworth-bk-app buildsworth-bk-app Bot Aug 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open — both build-cache definitions retain fixed exact addresses, so existing entries still cannot be refreshed.

target_paths:
- ../.gocache
18 changes: 10 additions & 8 deletions .buildkite/docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,29 +5,31 @@ services:
dockerfile: Dockerfile-lint
volumes:
- ../:/work:cached
- ~/gocache:/gocache
- ~/gomodcache:/gomodcache
working_dir: /work
environment:
- GOCACHE=/gocache
- GOMODCACHE=/gomodcache
# NOT inside /work (git's own checkout-wipe logic doesn't chmod
# read-only Go module cache files before removing them, unlike cache
# v2's own cleanPath) and NOT a host bind mount (that's finding #3 --
# the kernel refuses to unlink a mount point's own directory entry).
# A plain, non-mounted, non-checkout path avoids both.
- GOCACHE=/.gocache
- GOMODCACHE=/.gomodcache

agent:
build:
context: .
dockerfile: Dockerfile-compile
volumes:
- ../:/work:cached
- ~/gocache:/gocache
- ~/gomodcache:/gomodcache
working_dir: /work
environment:
- BUILDKITE_BUILD_NUMBER
- BUILDKITE_JOB_ID
- "BUILDKITE_AGENT_TAGS=queue=default"
- "BUILDKITE_BUILD_PATH=/buildkite"
- GOCACHE=/gocache
- GOMODCACHE=/gomodcache
# See the lint service's GOCACHE/GOMODCACHE comment above.
- GOCACHE=/.gocache
- GOMODCACHE=/.gomodcache
- BUILDKITE_TEST_ENGINE_API_ACCESS_TOKEN

e2e:
Expand Down
61 changes: 60 additions & 1 deletion .buildkite/pipeline.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,25 +17,49 @@ steps:
- go.{mod,sum}
- "**.go"
- .buildkite/steps/check-code-committed.sh
env:
BUILDKITE_AGENT_CACHE_STORE_URL: "s3://bk-agent-cache-v2-dogfood-445615400570-us-east-1?region=us-east-1"
plugins:
- &cache_v2_role
aws-assume-role-with-web-identity#v1.4.0:
role_arn: "arn:aws:iam::445615400570:role/pipeline-buildkite-agent"
session-tags:
- organization_slug
- organization_id
- pipeline_slug
- docker-compose#v4.14.0:
config: .buildkite/docker-compose.yml
cli-version: 2
mount-buildkite-agent: true
run: lint
env:
- BUILDKITE_AGENT_ACCESS_TOKEN
- BUILDKITE_AGENT_CACHE_STORE_URL
- AWS_ACCESS_KEY_ID
- AWS_SECRET_ACCESS_KEY
- AWS_SESSION_TOKEN

- name: ":go::robot_face: Check protobuf generation"
key: check-protobuf-genreation
command: .buildkite/steps/check-protobuf-generation.sh
if_changed:
- api/proto/**
- .buildkite/steps/check-protobuf-generation.sh
env:
BUILDKITE_AGENT_CACHE_STORE_URL: "s3://bk-agent-cache-v2-dogfood-445615400570-us-east-1?region=us-east-1"
plugins:
- *cache_v2_role
- docker-compose#v4.14.0:
config: .buildkite/docker-compose.yml
cli-version: 2
mount-buildkite-agent: true
run: lint
env:
- BUILDKITE_AGENT_ACCESS_TOKEN
- BUILDKITE_AGENT_CACHE_STORE_URL
- AWS_ACCESS_KEY_ID
- AWS_SECRET_ACCESS_KEY
- AWS_SESSION_TOKEN

# Send git commit metadata to the Test Engine plan-metadata pipeline on
# every build. Plan output is discarded -- the Tests and Coverage group
Expand Down Expand Up @@ -82,7 +106,10 @@ steps:
artifact_paths:
- junit-*.xml
- "coverage-*/**"
env:
BUILDKITE_AGENT_CACHE_STORE_URL: "s3://bk-agent-cache-v2-dogfood-445615400570-us-east-1?region=us-east-1"
plugins:
- *cache_v2_role
- tests#v1.0.0:
suite-slug: buildkite-agent
test-runner: gotest
Expand All @@ -98,7 +125,13 @@ steps:
config: .buildkite/docker-compose.yml
cli-version: 2
propagate-environment: true
mount-buildkite-agent: true
run: agent
env:
- BUILDKITE_AGENT_CACHE_STORE_URL
- AWS_ACCESS_KEY_ID
- AWS_SECRET_ACCESS_KEY
- AWS_SESSION_TOKEN

- name: ":linux: Linux ARM64 Tests"
key: test-linux-arm64
Expand All @@ -109,7 +142,10 @@ steps:
- "coverage-*/**"
agents:
queue: $AGENT_RUNNERS_LINUX_ARM64_QUEUE
env:
BUILDKITE_AGENT_CACHE_STORE_URL: "s3://bk-agent-cache-v2-dogfood-445615400570-us-east-1?region=us-east-1"
plugins:
- *cache_v2_role
- tests#v1.0.0:
suite-slug: buildkite-agent
test-runner: gotest
Expand All @@ -125,8 +161,19 @@ steps:
config: .buildkite/docker-compose.yml
cli-version: 2
propagate-environment: true
mount-buildkite-agent: true
run: agent

env:
- BUILDKITE_AGENT_CACHE_STORE_URL
- AWS_ACCESS_KEY_ID
- AWS_SECRET_ACCESS_KEY
- AWS_SESSION_TOKEN

# No docker-compose here (bare Windows host) -- buildkite-agent is
# already on PATH natively, so no mount-buildkite-agent equivalent is
# needed. aws-assume-role-with-web-identity's hooks are bash scripts,
# same as the command itself; not yet verified end-to-end on this
# queue specifically.
- name: ":windows: Windows AMD64 Tests"
key: test-windows
command: "bash .buildkite\\steps\\tests.sh"
Expand All @@ -136,7 +183,10 @@ steps:
- "coverage-*/**"
agents:
queue: $AGENT_RUNNERS_WINDOWS_QUEUE
env:
BUILDKITE_AGENT_CACHE_STORE_URL: "s3://bk-agent-cache-v2-dogfood-445615400570-us-east-1?region=us-east-1"
plugins:
- *cache_v2_role
- tests#v1.0.0:
suite-slug: buildkite-agent
test-runner: gotest
Expand All @@ -159,7 +209,10 @@ steps:
- "coverage-*/**"
agents:
queue: $AGENT_RUNNERS_LINUX_ARM64_QUEUE
env:
BUILDKITE_AGENT_CACHE_STORE_URL: "s3://bk-agent-cache-v2-dogfood-445615400570-us-east-1?region=us-east-1"
plugins:
- *cache_v2_role
- tests#v1.0.0:
suite-slug: buildkite-agent
test-runner: gotest
Expand All @@ -175,7 +228,13 @@ steps:
config: .buildkite/docker-compose.yml
cli-version: 2
propagate-environment: true
mount-buildkite-agent: true
run: agent
env:
- BUILDKITE_AGENT_CACHE_STORE_URL
- AWS_ACCESS_KEY_ID
- AWS_SECRET_ACCESS_KEY
- AWS_SESSION_TOKEN

- name: ":coverage: Test coverage report Linux ARM64"
key: test-coverage-linux-arm64
Expand Down
9 changes: 9 additions & 0 deletions .buildkite/steps/check-code-committed.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@

set -euf

echo --- :inbox_tray: cache restore
buildkite-agent cache restore --name gomodcache --name gocache

echo --- :go: Checking go mod tidyness
go mod tidy
if ! git diff --no-ext-diff --exit-code; then
Expand Down Expand Up @@ -65,3 +68,9 @@ EOF
fi

echo +++ Everything is clean and tidy! 🎉

# Sole writer: every other step that restores these caches (protobuf check,
# Tests and Coverage) only restores, never saves, so concurrent/parallel
# shards can't race each other's saves. This step runs once, unparallelized.
echo --- :outbox_tray: cache save
buildkite-agent cache save --name gomodcache --name gocache
Comment thread
buildsworth-bk-app[bot] marked this conversation as resolved.
3 changes: 3 additions & 0 deletions .buildkite/steps/check-protobuf-generation.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@

set -euf

echo --- :inbox_tray: cache restore
buildkite-agent cache restore --name gomodcache --name gocache

cd api/proto

echo --- :buf: Installing buf...
Expand Down
9 changes: 9 additions & 0 deletions .buildkite/steps/tests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,15 @@ set -euo pipefail
go version
echo arch is "$(uname -m)"

# One level above the checkout (PWD), not inside it -- see docker-compose.yml
# and cache.yml for why. Resolves the same way on every platform this script
# runs on, containerized or bare Windows host.
export GOCACHE="$(dirname "$PWD")/.gocache"
export GOMODCACHE="$(dirname "$PWD")/.gomodcache"

echo --- :inbox_tray: cache restore
buildkite-agent cache restore --name gomodcache --name gocache

RACE=''
if [[ $* == *-race* ]] ; then
RACE='-race'
Expand Down