native modules: build rust workspace crates through the repo flake by jeff-hykin · Pull Request #3770 · dimensionalOS/dimos

jeff-hykin · 2026-08-30T05:59:32Z

Use nix build instead of raw cargo (fails on Jetson)

…missing Non-interactive shells (ssh commands, systemd units) don't source the rustup PATH entry, so `cargo build --release` build_commands died with exit 127 on robots even though cargo was installed.

…argo is missing" This reverts commit 5241122.

rustup's PATH entry is absent in non-interactive shells (ssh commands, systemd units), so the plain `cargo build --release` build_commands died with exit 127 on robots. Run them through `nix develop` instead, like the realsense module already does, and put cargo/rustc in the flake so the shell is self-contained.

main's native-module publish gate AST-scans each module's config and refuses an f-string default, so the DIMOS_PROJECT_ROOT interpolation has to go. A cwd-relative nix path keeps the symlink behavior the resolve() was there for.

codecov · 2026-08-31T06:14:06Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ All tests successful. No failed tests found.

@@           Coverage Diff           @@
##             main    #3770   +/-   ##
=======================================
  Coverage   77.59%   77.60%           
=======================================
  Files        1317     1317           
  Lines      124794   124792    -2     
  Branches    10868    10868           
=======================================
+ Hits        96840    96841    +1     
+ Misses      24818    24816    -2     
+ Partials     3136     3135    -1

Flag	Coverage Δ
OS-ubuntu-24.04-arm	`74.25% <100.00%> (-0.01%)`	⬇️
OS-ubuntu-latest	`74.80% <100.00%> (+<0.01%)`	⬆️
Py-3.10	`74.79% <100.00%> (-0.01%)`	⬇️
Py-3.11	`74.79% <100.00%> (-0.01%)`	⬇️
Py-3.12	`74.79% <100.00%> (-0.01%)`	⬇️
Py-3.13	`74.79% <100.00%> (-0.01%)`	⬇️
Py-3.14	`74.79% <100.00%> (+<0.01%)`	⬆️
Py-3.14t	`74.79% <100.00%> (+<0.01%)`	⬆️
SelfHosted-Linux	`34.75% <100.00%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
dimos/mapping/ray_tracing/module.py	`100.00% <100.00%> (ø)`
...avigation/nav_3d/mls_planner/mls_planner_native.py	`100.00% <100.00%> (ø)`

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

greptile-apps · 2026-08-31T06:17:07Z

Greptile Summary

This change routes automatic release builds for the ray-tracing and MLS planner native modules through the repository Nix development shell, and adds Cargo and rustc to that shell.

Both configured relative paths resolve from their Rust working directories to the repository root, where flake.nix is present. Equivalent release Cargo builds completed successfully and created target/release/voxel_ray_tracing and target/release/mls_planner.

T-Rex validation blocked

The exact Nix-wrapped commands could not run because the nix tool is missing from this environment. Both commands stopped with nix: not found (exit 127), so this environment could not directly confirm that nix develop supplies Cargo and rustc.

Confidence Score: 5/5

No defects were found; the configured build locations and expected release outputs behaved as intended.

The relative flake paths resolved to the repository root, and release builds for both affected native packages succeeded and produced their expected executables. No final review findings remain.

Files Needing Attention: No files need changes. The Nix-wrapped commands should be exercised in a Nix-enabled development environment when available.

T-Rex Logs

What T-Rex did

I confirmed that both dimos/mapping/ray_tracing/rust and dimos/navigation/nav_3d/mls_planner/rust resolve to the repository root and that a flake.nix exists there.
I attempted to run the Nix-based build commands from both Rust working directories, but Nix was not found in this environment.
I ran equivalent cargo builds for both crates; dimos-voxel-ray-tracing completed in 29.92 seconds and dimos-mls-planner completed in 44.66 seconds, producing the expected executables under target/release.
I re-ran the commands with PATH set to /usr/bin:/bin; both Nix commands still failed and both cargo commands failed because cargo was not available, confirming the wrapper boundary for tool provisioning.

_{Ran code and verified through T-Rex}

_{Reviews (1): Last reviewed commit: "native modules: drop the build_command c..." | Re-trigger Greptile}

aclauer · 2026-08-31T17:19:25Z

    # The crate is a workspace member, so cargo builds into the repo-root target dir.
    executable: str = str(DIMOS_PROJECT_ROOT / "target" / "release" / "voxel_ray_tracing")
-    build_command: str | None = "cargo build --release"
+    build_command: str | None = "nix develop path:../../../.. -c cargo build --release"


Suggested change

build_command: str | None = "nix develop path:../../../.. -c cargo build --release"

build_command: str | None = "nix develop ../../../.. -c cargo build --release"

So if we use path that will cause the entire workspace to get snapshotted on every build, can you double check this isn't going to spam the nix store?

Yeah... I'm afraid this is looking like a "pick your poison" kinda decision. I've got a couple ideas but I'm gonna have to do testing.

aclauer · 2026-08-31T17:19:50Z

    # The crate is a workspace member, so cargo builds into the repo-root target dir.
    executable: str = str(DIMOS_PROJECT_ROOT / "target" / "release" / "mls_planner")
-    build_command: str | None = "cargo build --release"
+    build_command: str | None = "nix develop path:../../../../.. -c cargo build --release"


Suggested change

build_command: str | None = "nix develop path:../../../../.. -c cargo build --release"

build_command: str | None = "nix develop ../../../../.. -c cargo build --release"

A flake ref copies its whole source tree into the nix store, and a fresh copy each time that tree changes. Aimed at the repo root that is the build dirs, .venv and .git, or the tracked LFS blobs -- tens of gigabytes per build. nix/rust/ is two files, so the snapshot is stable and free. Also run cargo fmt and cargo clippy inside that shell, and stop the clippy hook from re-entering the repo-root flake: the root has its own flake.nix, so the per-crate branch was matching it and shelling into the whole tree.

`nix develop ... -c cargo build` left the result in a per-machine target dir and needed a working cargo cache. A `nix build` of the whole cargo workspace is one hermetic derivation that Cachix can substitute, and its src is assembled from path literals so a doc or python edit neither rebuilds it nor invalidates the publish marker. Also give the clippy/fmt shell librealsense on Linux, so the realsense crate can join the workspace without breaking macOS.

…_cargo_path

CI builds the PR merge commit, so the dependabot cargo bump on main (de2b0fc) changed the lock the vendor derivation sees. Take the value that commit's lock actually vendors to.

The root flake hardcoded every crate directory in the workspace, so adding a module meant editing a file far from it. Each module now owns a deps.nix naming the trees it contributes, the cargo packages that are module executables, and any system libraries it links; the flake only composes them. The inputs-hash scanner follows imported .nix files for the same reason it follows path: inputs -- otherwise a crate reachable only through a deps.nix would not key the Cachix publish marker.

Adding a rust native module no longer means editing the repo-root flake, which was the last place the module list was duplicated. The search rule is deliberately the dumbest one available -- every file named deps.nix, anywhere in the tree -- because the Cachix publish gate in bin/build-native-modules has to arrive at the same set without evaluating nix, and it does that with a glob. Anything cleverer would be two implementations that drift. Searching makes the flake name ./. , which the publish gate would otherwise hash as the whole repository, rekeying the marker on every commit. readDir reads directory names rather than file contents, so the deps.nix files and the crates they name are the only content that actually reaches the derivation, and both are already collected. The exemption is therefore limited to that one expression: any other ./. in the root flake is now a hard error.

cargoHash is one fixed-output hash over every vendored crate, so any Cargo.lock change invalidates it -- including a dependabot bump landing on main, which CI surfaces through the merge commit and so fails pull requests that never touched rust. Recovering means a human reading a hash out of a vendor-staging log and pasting it back. cargoLock uses the per-crate checksums already in the lock, so there is no aggregate hash to maintain. Git dependencies still need an entry because the lock records no checksum for them, but that hash is keyed to a pinned rev rather than to the lock as a whole. An earlier revision of this branch claimed nixpkgs could not fetch crates this way because crates.io rejects the curl user agent. That was measured with a manual curl against a different URL and never tested against the actual build, which fetches without trouble.

…gate hash" This reverts commit d06824e.

The 403 does not reproduce locally -- the crate tarballs are fixed-output, so they substitute from cache.nixos.org and the fetch never runs. Only CI, on a cache miss, actually hits crates.io. Note that, and the reason the CDN cannot be substituted in, so the next attempt does not have to rediscover it from a red build.

…_cargo_path

#3741 changed Cargo.lock on main. CI builds the merge commit, so the vendor hash went stale on a branch that never touched rust -- the failure mode the comment above the hash describes.

Cargo.nix gives every crate its own derivation, so editing one module does not rebuild the others and each crate caches on its own. Measured: a cold ray_tracing build is 11m43s, and mls_planner right after is 21.8s. It also removes the aggregate cargoHash, which had to be re-pasted by hand whenever Cargo.lock moved -- including from main, which breaks branches that never touched rust, because CI builds the merge commit. The replacement failure mode is a stale generated file, which the new cargo-nix-current job prints as a diff instead of a number to re-paste. Regenerate with bin/regen-cargo-nix. The generated pair is excluded from the byte-rewriting pre-commit hooks so it stays reproducible, and Cargo.nix is exempted from the large-files check -- LFS would be wrong for a file CI has to diff as text.

Adding a module is now a deps.nix and nothing else, so the examples can be real build targets instead of a workspace member nothing referenced. They build in 42s here because every crate under them is already a store path. Doubles as the regression test for that claim: if adding a module ever starts requiring a flake.nix edit again, this file is the one that stops building.

Its keys are scoped to a git dependency's branch and carry no rev, so a committed copy silently goes stale the second that branch moves, pinning a hash that no longer matches the revision Cargo.lock resolves to. That is the exact failure this whole change set exists to remove. It is also redundant: it is only crate2nix's prefetch cache, and the hashes are already inlined in Cargo.nix. Regenerating with the file absent produces a byte-identical Cargo.nix, so nothing in the repo depends on it. Dropping it also retires two problems rather than papering over them: it no longer needs a pre-commit exclusion (pretty-format-json was rewriting it on every commit, which had already introduced a trailing newline the generator never emits), and it no longer needs an editorconfig exception for that missing final newline.

crate2nix shells out to `cargo metadata`, so the generated file depends on whichever cargo is on PATH. The generator itself was pinned by flake.lock but cargo was not: locally that was a rustup 1.98.0, while the toolchain this repo pins for its clippy and fmt hooks is 1.94.0, and CI would have used whatever the runner image ships. The two happen to agree today -- checked by regenerating under 1.94.0 and getting a byte-identical Cargo.nix -- so this fixes a latent problem rather than a live one. Worth doing anyway: the failure it prevents is a diff that only appears on the machine with the odd cargo, which is the hardest kind to read when cargo-nix-current is what reports it.

I had commented it as crate2nix's per-crate override hook. It is not: the `crateConfig` argument Cargo.nix defaults to this path is declared and then never read anywhere in the generated file. Overrides added here would have silently done nothing, which is a worse trap than having no file at all. The file still has to exist -- `bin/build-native-modules --inputs-hash` walks nix path literals textually, cannot evaluate the `builtins.pathExists` guard around this one, and exits 1 with "reference './crate-config.nix' does not exist". Verified by moving the file away. Kept an empty file rather than teaching that parser about conditional paths, since a parser that learns to skip guarded references can also skip one that matters.

native_module: put ~/.cargo/bin on the auto-build PATH when cargo is …

5241122

…missing Non-interactive shells (ssh commands, systemd units) don't source the rustup PATH entry, so `cargo build --release` build_commands died with exit 127 on robots even though cargo was installed.

jeff-hykin commented Aug 30, 2026

View reviewed changes

Comment thread dimos/core/native_module.py Outdated

jeff-hykin added 2 commits August 29, 2026 23:06

Revert "native_module: put ~/.cargo/bin on the auto-build PATH when c…

112c9d5

…argo is missing" This reverts commit 5241122.

jeff-hykin changed the title ~~native_module: find cargo in ~/.cargo/bin during auto-build~~ native modules: build rust workspace crates through the repo flake Aug 30, 2026

jeff-hykin added 2 commits August 30, 2026 23:00

Merge remote-tracking branch 'origin/main' into _flake_work

4cd942a

jeff-hykin commented Aug 31, 2026

View reviewed changes

Comment thread dimos/mapping/ray_tracing/module.py Outdated

jeff-hykin commented Aug 31, 2026

View reviewed changes

Comment thread dimos/navigation/nav_3d/mls_planner/mls_planner_native.py Outdated

native modules: drop the build_command comments

7a4f671

jeff-hykin marked this pull request as ready for review August 31, 2026 06:06

jeff-hykin requested review from Dreamsorcerer, aclauer, leshy, mustafab0, paul-nechifor and spomichter as code owners August 31, 2026 06:06

github-actions Bot added the ready-to-merge Required CI checks have passed on this PR label Aug 31, 2026

aclauer reviewed Aug 31, 2026

View reviewed changes

github-actions Bot added ready-to-merge Required CI checks have passed on this PR and removed ready-to-merge Required CI checks have passed on this PR labels Aug 31, 2026

github-actions Bot removed the ready-to-merge Required CI checks have passed on this PR label Aug 31, 2026

jeff-hykin had a problem deploying to cachix August 31, 2026 23:54 — with GitHub Actions Failure

jeff-hykin added 2 commits August 31, 2026 16:59

Merge remote-tracking branch 'origin/main' into jeff/fix/native_build…

ac844e9

…_cargo_path

Update the workspace cargoHash for main's Cargo.lock bump

6d274a2

CI builds the PR merge commit, so the dependabot cargo bump on main (de2b0fc) changed the lock the vendor derivation sees. Take the value that commit's lock actually vendors to.

jeff-hykin temporarily deployed to cachix September 1, 2026 00:06 — with GitHub Actions Inactive

jeff-hykin temporarily deployed to cachix September 1, 2026 00:42 — with GitHub Actions Inactive

github-actions Bot added the ready-to-merge Required CI checks have passed on this PR label Sep 1, 2026

jeff-hykin added 2 commits August 31, 2026 18:14

github-actions Bot removed the ready-to-merge Required CI checks have passed on this PR label Sep 1, 2026

jeff-hykin had a problem deploying to cachix September 1, 2026 04:22 — with GitHub Actions Failure

jeff-hykin added 2 commits August 31, 2026 21:34

Revert "Vendor the rust workspace from Cargo.lock instead of an aggre…

7d19b37

…gate hash" This reverts commit d06824e.

jeff-hykin had a problem deploying to cachix September 1, 2026 04:43 — with GitHub Actions Failure

jeff-hykin added 2 commits August 31, 2026 21:54

Merge remote-tracking branch 'origin/main' into jeff/fix/native_build…

7ea8f45

…_cargo_path

Re-paste cargoHash for the Cargo.lock main brought in

701453c

#3741 changed Cargo.lock on main. CI builds the merge commit, so the vendor hash went stale on a branch that never touched rust -- the failure mode the comment above the hash describes.

jeff-hykin temporarily deployed to cachix September 1, 2026 04:57 — with GitHub Actions Inactive

jeff-hykin added 5 commits August 31, 2026 22:20

jeff-hykin mentioned this pull request Sep 1, 2026

native modules: build each crate separately with crate2nix #3861

Merged

github-actions Bot added ready-to-merge Required CI checks have passed on this PR and removed ready-to-merge Required CI checks have passed on this PR labels Sep 1, 2026

jeff-hykin deployed to cachix September 1, 2026 16:31 — with GitHub Actions Active

github-actions Bot added the ready-to-merge Required CI checks have passed on this PR label Sep 2, 2026

Navigation Menu

Uh oh!

native modules: build rust workspace crates through the repo flake - #3770

native modules: build rust workspace crates through the repo flake#3770
jeff-hykin wants to merge 22 commits into
mainfrom
jeff/fix/native_build_cargo_path

jeff-hykin commented Aug 30, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Aug 31, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Aug 31, 2026

T-Rex Logs

Uh oh!

aclauer Aug 31, 2026

Uh oh!

jeff-hykin Aug 31, 2026

Uh oh!

aclauer Aug 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	build_command: str \| None = "nix develop path:../../../.. -c cargo build --release"
	build_command: str \| None = "nix develop ../../../.. -c cargo build --release"

Uh oh!

Conversation

jeff-hykin commented Aug 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

greptile-apps Bot commented Aug 31, 2026

Greptile Summary

T-Rex validation blocked

Confidence Score: 5/5

T-Rex Logs

Uh oh!

aclauer Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

jeff-hykin Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

aclauer Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jeff-hykin commented Aug 30, 2026 •

edited

Loading

codecov Bot commented Aug 31, 2026 •

edited

Loading