Bound the BEP artifact-upload wait with a timeout - #2843
Open
bor-p-s wants to merge 1 commit into
Open
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
bor-p-s
force-pushed
the
bep-upload-timeout
branch
from
September 8, 2026 09:35
51d8ebf to
1288d27
Compare
bor-p-s
marked this pull request as ready for review
September 8, 2026 16:02
bor-p-s
requested review from
Wyverald,
fweikert and
meteorcloudy
as code owners
September 8, 2026 16:02
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds a
timeoutparameter toexecute_command(), passed through tosubprocess.run.upload_test_logs_from_bep()uses it on thebazelci-agent artifact uploadcall with a 2 hour limit; on timeout the process is killed, the failure is logged, and the function returns without re-raising, since the build's pass/fail result is already decided bybazel testbefore this step runs -- a killed upload only loses debug artifacts, not the build outcome. The same timeout is applied to thejava.logupload in the same function, which had the same unboundedexecute_command()call.The 2 hour value is an estimate, not a measurement. The only normal-runtime data available is from rules_scala itself (macOS
test_rules_scala: 40-55 min); other projects on this shared fleet may run longer, worth confirming with maintainers before treating this as final. A real fix insidebazelci-agentitself (detecting that the writer process died) needs a new agent release; out of scope here.Motivation
rules_scala#1906 found jobs that stall inside
upload_test_logs_from_bep()until Buildkite's 8 hour step timeout kills them. This stays invisible because the build still ends up green (a fast retry on some builds,soft_failon others), so the job burns most of 8 hours doing nothing and nobody notices.The exact hang mechanism inside
bazelci-agentremains unverified -- the leading theory (bazel exits before writinglastMessage, so the agent waits forever) failed to reproduce with the real binaries. That's why this bounds the wait from the outside instead of targeting a specific mechanism.