Skip to content

Bound the BEP artifact-upload wait with a timeout - #2843

Open
bor-p-s wants to merge 1 commit into
bazelbuild:masterfrom
bor-p-s:bep-upload-timeout
Open

Bound the BEP artifact-upload wait with a timeout#2843
bor-p-s wants to merge 1 commit into
bazelbuild:masterfrom
bor-p-s:bep-upload-timeout

Conversation

@bor-p-s

@bor-p-s bor-p-s commented Sep 8, 2026

Copy link
Copy Markdown

Description

Adds a timeout parameter to execute_command(), passed through to subprocess.run. upload_test_logs_from_bep() uses it on the bazelci-agent artifact upload call with a 2 hour limit; on timeout the process is killed, the failure is logged, and the function returns without re-raising, since the build's pass/fail result is already decided by bazel test before this step runs -- a killed upload only loses debug artifacts, not the build outcome. The same timeout is applied to the java.log upload in the same function, which had the same unbounded execute_command() call.

The 2 hour value is an estimate, not a measurement. The only normal-runtime data available is from rules_scala itself (macOS test_rules_scala: 40-55 min); other projects on this shared fleet may run longer, worth confirming with maintainers before treating this as final. A real fix inside bazelci-agent itself (detecting that the writer process died) needs a new agent release; out of scope here.

Motivation

rules_scala#1906 found jobs that stall inside upload_test_logs_from_bep() until Buildkite's 8 hour step timeout kills them. This stays invisible because the build still ends up green (a fast retry on some builds, soft_fail on others), so the job burns most of 8 hours doing nothing and nobody notices.

The exact hang mechanism inside bazelci-agent remains unverified -- the leading theory (bazel exits before writing lastMessage, so the agent waits forever) failed to reproduce with the real binaries. That's why this bounds the wait from the outside instead of targeting a specific mechanism.

@google-cla

google-cla Bot commented Sep 8, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@bor-p-s
bor-p-s marked this pull request as ready for review September 8, 2026 16:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant