New flag --shard-by-suite

pkgs/test/CHANGELOG.md

@@ -13,9 +13,11 @@
   all tests with OS `'windows'` would previously still run browser tests on
   windows, but will now skip all tests including browser tests.
 * Use a DevTools URL instead of a defunct observatory URL.
+* Add flag `--shard-by-suite` to control sharding strategy.
 * Disable throttling in chrome launch arguments.
 * Allow package_config `3.x.x`.
 
+
 ## 1.31.1
 
 * Ignore an error locating the SDK directory on platforms where the

pkgs/test/README.md

@@ -204,12 +204,38 @@ dart test --total-shards 3 --shard-index 0 path/to/test.dart
 dart test --total-shards 3 --shard-index 1 path/to/test.dart
 dart test --total-shards 3 --shard-index 2 path/to/test.dart
 ```
-Sharding: This refers to the process of splitting up a large test suite into
-smaller subsets (called shards) that can be run independently. Sharding is
-particularly useful for distributed testing, where multiple machines are used
-to run tests simultaneously. By dividing the test suite into smaller subsets,
-you can run tests in parallel across multiple machines, which can significantly
-reduce the overall testing time.
+
+By default, sharding is done by individual tests within each suite.
+You can shard by entire test suites (files) using the `--shard-by-suite` flag.
+
+* When sharding by test (default): Distribute individual test cases across
+  shards. This balances the test load at the test case level. Test cases from
+  each suite are sliced continuously, which minimizes how often a suite is
+  split across shards and helps maximize the re-use of suite `setUpAll` and
+  `tearDownAll` setups.
+* When sharding by suite (using `--shard-by-suite`): Distribute entire test suites
+  across shards. This can be faster for suites with many small suites as it
+  avoids loading every suite in every shard. Because test suites are not split,
+  any `setUpAll` and `tearDownAll` setups within a suite (including shared helper setups imported across suites) run only on the shard executing that suite.
+
+Sharding is particularly useful for distributed testing, where multiple
+machines are used to run tests simultaneously. By dividing the test suite into
+smaller subsets, you can run tests in parallel across multiple machines, which
+can significantly reduce the overall testing time.
+
+### Interaction with Test Filters
+
+The sharding modes interact differently with filters like `--name` or `--tags`:
+
+* When sharding by test (default), the sharding partition is calculated *after*
+  applying filters. This guarantees that the matching tests are distributed
+  as evenly as possible across all shards.
+* When sharding by suite (using `--shard-by-suite`), suite-level annotations
+  (such as `@TestOn` platform selectors and suite-level `@Tags`) are evaluated
+  before suite sharding, filtering out non-matching suites so shards do not receive
+  empty workloads. In contrast, test-case/group filters (such as `--name` or test-level tags)
+  operate on test cases within loaded suites after sharding. If a test-case filter only matches tests in a
+  few suites, some shards might run no tests because those suites were allocated to other shards.
 
 ### Test concurrency
 

pkgs/test/test/runner/runner_test.dart

@@ -84,6 +84,7 @@ $_runtimeCompilers
                                       (defaults to "$_defaultConcurrency")
     --total-shards                    The total number of invocations of the test runner being run.
     --shard-index                     The index of this test runner invocation (of --total-shards).
+    --shard-by-suite                  Distribute entire test suites (_test.dart files) across shards instead of individual tests.
     --timeout                         The default test timeout. For example: 15s, 2x, none
                                       (defaults to "30s")
     --suite-load-timeout              The timeout for loading a test suite. Loading the test suite includes compiling the test suite. For example: 15s, 2m, none