Support for running without Cargo (using bazel or buck2 as the build system)

[thinety]

Is there support for running Nextest without Cargo? That is, using only the binary that rustc --test outputs.

I have skimmed through the codebase, but I'm afraid I don't have the skills to be sure about it.

[sunshowers]

Thanks for the question!

Nextest does support running without cargo, with the build archive and reuse functionality. See https://nexte.st/book/reusing-builds.html#non-requirements.

You do need Cargo to actually do the build though. At this time we have no interest in supporting alternative build systems like Bazel.

[thinety]

I had no idea about the archive/reuse functionality. Thanks for pointing it out.

Unfortunately, I am more interested in the second part of your response. Is the metadata provided by Cargo a hard requirement, or is it technically possible for Nextest to interface directly with a rustc --test-binary? If so, would you be willing to accept a PR?

[sunshowers]

Sorry about the late response -- I had a long comment typed up but lost it because my browser crashed.

I'm honestly not sure! It really depends on the specifics: how invasive it is (moderately concerned about this) and how much work it would be to maintain and test against (very concerned).

How about if you try forking nextest under its Free Software licenses and making your changes there? We can figure out whether an integration works once you have some changes done. Worst case you can maintain your fork.

[sunshowers]

See also #1789.

[k-cross]

Hi there,

I worked on a prototype for Buck2 specifically that looks something like:

flowchart
  buck["buck2-nextest"]
  cargo["cargo-nextest"]
  session["nextest-session"]
  runner["nextest-runner"]
  
  buck --> session
  cargo --> session
  session --> runner

Loading

The idea is that all shared logic and contracts between clients happens in nextest-session and a good chunk of required changes to make multiple client integrations work all happens there. The changes to nextest-runner and any other crate/library are to be as minimal as possible only allowing for required changes to either allow integration to be possible or to decouple and abstract away cargo specific details.

Infeasible

So while this works for rust, the test_runner is a singular thing that must be overwritten in buck2 which means that the test runner that's overwritten must provide orchestration for all languages... So while I can get this to work with nextest and rust it will break all other language tests which is far from the point.

Tpx is what facebook uses and is not open source and the shipped open version in buck2 is not extensible in this way as of today.

Context

No current open test orchestration system like tpx exists in order to make this a reality either... BXL was a recommended path forward by AI and it's really not, it's a different script where the normal test functions in buck2 wouldn't even get the benefit of knowing a test ran at all. Shell scripts and hacked workarounds that way are a likely better option.

Potential Paths Forward

It would probably be working with buck2 to provide the capability to choose a test runner for a particular language as part of the open test runner that ships with the project. Waiting for an open source test orchestration platform seems like quite the ask/wait.

[k-cross]

ExternalTestRunnerInfo looks like it might be able to contain some additional info via rules. The project setup might require more work with buck2 in order to use nextest but it does seem like a real possibility. I'll play with it while I've got the time.

[k-cross]

It's not much better, not worth the changes. You're still cobbling together shell scripts and get target level granularity instead of test level granularity. This is something anyone could make work with cargo-nextest alone.

[sunshowers]

@thoughtpolice linked me to something they wrote: https://github.com/thoughtpolice/a/blob/canon/buck/shims/rust_test_internal.bzl

I'm wondering if this + the work you did could possibly be integrated. Thoughts?

[k-cross]

Seems like most the work stands. I suspected it would mostly be rewriting the client and that seems to hold (buck2 not cargo). It looks like this work is all very fresh though, so may need to work with upstream buck2 to get the whole thing across the finish line. I'll keep up the work as long as there's a meaningful path forward that is not "rewrite every test runner for every language" lol.

[k-cross]

Blocked on this issue but there's active work that I'm monitoring. I have a fix that works locally, but it looks like there's someone that's already authoring a change.

[k-cross]

Area	Standard Buck2 Rust project	With buck2-nextest integration
Runner binary	None needed (uses Buck2's default test runner)	buck2-nextest binary must be on PATH (or built as a target)
Cell configuration	Declares root, prelude, toolchains	Adds a nextest cell in .buckconfig and maps its platform
Toolchains	Only standard Rust toolchains (e.g. system_demo_toolchains())	Adds system_nextest_toolchain (or nextest_toolchain)
Target definitions	Uses rust_test from the prelude	Uses nextest_test from @nextest//:nextest_test.bzl
Configuration	Configured via Buck2 flags/prelude	Can use standard .config/nextest.toml at project root
Execution unit	The whole test binary runs as a single opaque action	Buck2 discovers and runs each #[test] function as a separate action

Currently, these are the differences between a standard buck2 configured rust project and one with the nextest integration. Just about any tooling integration at this level will require some configuration so I wouldn't call this non-standard or a hack but more like the mechanism buck2 allows for customization.

[sunshowers]

All of this seems quite reasonable to me. I think some level of configuration is expected with the way buck2 works.

[k-cross]

Great! I am waiting on some upstream changes and about to make some upstream changes to buck2, but current output in success/fail scenarios looks like this in modified branches:

 PATH="$(cd ../../target/release && pwd):$PATH" ~/src/buck2/target/release/buck2 test //example-failure/...
✗ Fail: root//example-failure:demo-failing-test - tests::always_fails (0.0s)
thread 'tests::always_fails' (12312472) panicked at example-failure/src/lib.rs:14:9:
assertion `left == right` failed: this test always fails, to exercise the failure path
  left: 1
 right: 2
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
--- stdout ---

running 1 test
test tests::always_fails ... FAILED

failures:

failures:
    tests::always_fails

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

--- stderr ---

thread 'tests::always_fails' (12312472) panicked at example-failure/src/lib.rs:14:9:
assertion `left == right` failed: this test always fails, to exercise the failure path
  left: 1
 right: 2
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Build ID: 694c57d8-4016-4c26-ae2c-976ad3b695de
Phase                 Remaining                   Details
Command test
elapsed 0.4s
Build ID: 694c57d8-4016-4c26-ae2c-976ad3b695de
Tests finished: Pass 0. Fail 1. Timeout 0. Fatal 0. Skip 0. Omit 0. Infra Failure 0. Build failure 0
1 TESTS FAILED
  ✗ root//example-failure:demo-failing-test - tests::always_fails (prelude//platforms:default#9cddb5969014779c)

 PATH="$(cd ../../target/release && pwd):$PATH" ~/src/buck2/target/release/buck2 test //example/...
↷ Skip: root//example:demo-lib-test - tests::add_saturates_at_the_maximum
Build ID: 1849f024-2dd7-4b5f-8293-1d0971449b68
Phase                 Remaining                   Details
Command test
elapsed 0.0s
Build ID: 1849f024-2dd7-4b5f-8293-1d0971449b68
Tests finished: Pass 6. Fail 0. Timeout 0. Fatal 0. Skip 1. Omit 0. Infra Failure 0. Build failure 0

[sunshowers]

Thanks. Is that buck2's output? I'm a little surprised at how unpolished it looks compared to nextest.

[k-cross]

It probably looks worse because there are not that many tests/targets running but at the same time, I also prefer nextest! Maybe they make the details opaque while running to save time on output rendering, I could see that as a thing they might be avoiding dealing with right now because they support both local and remote execution but those are guesses. I'll pivot back when there's more news, as of right now it's just kind of, waiting on buck2.

[k-cross]

Update: The core PR that makes this usable landed in Buck2 but I have another PR built on top of it that addresses performance and usability considerations waiting to land.