Skip to content

ci(samples): retry transient LLM 503/429 errors and propagate runner exit codes - #4442

Open
DylanVolz wants to merge 2 commits into
mainfrom
fix/sample-tests-retry-and-subshell
Open

DylanVolz wants to merge 2 commits into
mainfrom
fix/sample-tests-retry-and-subshell

Conversation

@DylanVolz

Copy link
Copy Markdown
Contributor

Wrap sample test invocations in run_with_retry (up to 3 attempts with exponential backoff and DB table reset) when transient upstream LLM errors (503 UNAVAILABLE / 429 RESOURCE_EXHAUSTED) occur, and add matching retry helpers in the Go sample test harnesses. Also replace the piped find | while loop with process substitution and capture test exit codes before cleanup so subshell failures are never masked by rm -rf.

Description

Hardens sample integration tests against transient upstream Gemini API capacity errors (503 UNAVAILABLE / 429 RESOURCE_EXHAUSTED) and fixes exit-code masking in .ci/sample_tests/run_tests.sh:

  • Adds run_with_retry and is_transient_llm_error in .ci/sample_tests/run_tests.sh to retry sample test runs up to 3 times with exponential backoff and a fresh setup_db_table reset when transient 503/429 LLM errors occur.
  • Adds matching retry logic in docs/en/documentation/getting-started/quickstart/go/quickstart_test.go and docs/en/documentation/configuration/pre-post-processing/go/agent_test.go for standalone go test runs (disabled via SAMPLE_TEST_MAX_ATTEMPTS=1 when already wrapped by run_tests.sh).
  • Captures test runner exit codes before rm -rf cleanup in run_python_test, run_js_test, and run_go_test, and replaces find ... | while read with while read ... done < <(find ... | sort) so failures inside the loop are not lost in a pipeline subshell.

PR Checklist

Thank you for opening a Pull Request! Before submitting your PR, there are a
few things you can do to make sure it goes smoothly:

  • Make sure you reviewed
    CONTRIBUTING.md
  • Make sure to open an issue as a
    bug/issue
    before writing your code! That way we can discuss the change, evaluate
    designs, and agree on the general idea
  • Ensure you have manually reviewed the entire diff before requesting a
    review
  • Ensure the tests and linter pass
  • Code coverage does not decrease (if any source code was changed)
  • Appropriate docs were updated (if necessary)
  • Make sure to add ! if this involve a breaking change

🛠️ Fixes #<issue_number_goes_here>

…exit codes

Wrap sample test invocations in run_with_retry (up to 3 attempts with exponential backoff and DB table reset) when transient upstream LLM errors (503 UNAVAILABLE / 429 RESOURCE_EXHAUSTED) occur, and add matching retry helpers in the Go sample test harnesses. Also replace the piped find | while loop with process substitution and capture test exit codes before cleanup so subshell failures are never masked by rm -rf.
@DylanVolz
DylanVolz requested a review from a team as a code owner November 7, 2026 02:35

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces transient LLM error detection and retry mechanisms with exponential backoff to the CI test runner (run_tests.sh) and Go integration tests (agent_test.go and quickstart_test.go), alongside improved error handling in the shell script. The reviewer suggested expanding the list of transient error markers in both Go test files to match the comprehensive patterns checked in the shell script (such as specific status codes and JSON-formatted errors) for more robust detection.

@DylanVolz

Copy link
Copy Markdown
Contributor Author

/gcbrun

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants