Retry only failed test cases using a framework-provided filter

[eddy-seager]

Hello, I would like feedback on an mechanism for retrying only the failed test cases within a Bazel test target.

Currently, --flaky_test_attempts reruns the whole target, which can be expensive for large suites. It would be much more efficient to only rerun the failed test cases within a target.

Could the test framework supply the retry filter itself? For example:

bazel test //example:tests \
    --retry_failed_cases \
    --flaky_test_attempts=3

Bazel would provide a path through a new proposed TEST_RETRY_FILTER_OUTPUT_FILE variable. A supporting framework would write its failed-case filter there, and Bazel would pass that filter through TESTBRIDGE_TEST_ONLY on the next attempt.

Details would need discussion, including sharding, interaction with an existing filter, crashes or incomplete runs, remote execution, and reporting results across attempts.

I originally posted the idea on Bazel Slack, where another developer reported the same need.

Would this be a reasonable addition to Bazel’s test CLI? I am seeking initial feedback.

[fmeum]

I see a lot of value in faster retries.

What I would be worried about with this approach is that no run will have a fully accurate test.xml/test.log: the first run has all test cases but reports some as failed, the later ones only cover a subset of test cases. Would we need some kind of merge logic?

[eddy-seager]

I agree that a later attempt’s test.xml would describe only the cases run in that attempt.

I would keep XML merging separate from the retry-selection proposal. Bazel already preserves every attempt under test_attempts.

A framework-independent way to combine case results could be designed separately, and could also benefit existing uses of --flaky_test_attempts without this feature, for example to merge a failed -> passed -> failed test case to become a passed in the final merge.