Add nix derivation source-origins command - #5
Open
ethancedwards8 wants to merge 1 commit into
Open
ethancedwards8 wants to merge 1 commit into
ethancedwards8 wants to merge 1 commit into
Conversation
`nix derivation source-origins [--no-recursive] INSTALLABLES...` evaluates the given installables and, for every derivation in their build closure (walking `inputDrvs` via `Derivation::inputs`), prints a JSON mapping from each `inputSrcs` store path back to the filesystem path it was copied from during that evaluation. This lets tooling (e.g. monorepo CI) determine which working-tree directories and files contributed to a build. The provenance is recorded in two reverse maps on `EvalState` (`boost::concurrent_flat_map`s, like the sibling caches, since evaluation may be concurrent): * `storeToSrc` (StorePath -> SourcePath), populated by `copyPathToStore()` and by `addPath()` (`builtins.path`, `builtins.filterSource`, hence also `lib.cleanSourceWith`) via the new `recordPathOrigin()`. * `sourceStoreToOriginalPath` (StorePath -> std::filesystem::path), populated by `mountInput()` for inputs whose accessor carries an `originalRootPath`, and by `recordPathOrigin()`, which resolves a source path to an original filesystem path by (1) mapping `/nix/store/<src>/rel` through the known source roots, (2) using the accessor's `originalRootPath`, (3) matching the accessor by identity against the mounts in `storeFS`, or (4) falling back to the physical path for plain filesystem paths. `SourceAccessor` gains an `originalRootPath` field, which the `git` input scheme sets to the repository root on workdir accessors (also on the `MountedSourceAccessor` wrapper used when submodules are present) and the `path` input scheme sets to the input path on the store accessor it now returns. Using the repo root (rather than `Input::getSourcePath()`, i.e. the flake directory) means paths in `git+file://repo?dir=sub` flakes resolve correctly. `sourcePath` is `null` for store paths not produced by the current evaluation (substituted paths, `builtins.toFile`, ...). When a source is a directory, `sourceFiles` enumerates the contents of the store object (via the store's accessor, breadth-first, treating symlinks as leaves so cycles like `link -> .` are not followed) mapped back under `sourcePath`; since the store object is the *result* of any filtering, this gives file-level precision for `cleanSourceWith` sources with a broad root. The command disables the evaluation cache, since a cached evaluation would never copy anything to the store. Includes a functional test covering `--file`, `path:` and `git+file:` evaluations, filtered sources, null origins, symlinks, `--no-recursive` and eval-cache bypass, plus a manual page and release note. Re-implementation of the exa-labs/nix-src series ee40e5e..d4ebfeb against NixOS/nix 2.36 (GIGA-3360, GIGA-2286). Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Part 2 of 3 for GIGA-3360 (feature from GIGA-2286 / nix-src PR #2). Single squashed commit re-implementing
nix derivation source-origins [--no-recursive] INSTALLABLES...against upstream 2.36: for each derivation (recursing throughinputDrvsby default) it maps everyinputSrcsstore path back to the working-tree path it was copied from during this evaluation, emitting JSON withstorePath,sourcePath(null if not produced by this eval) and asourceFilesenumeration for directory sources.How it works on 2.36:
SourceAccessor::originalRootPathset by thegit(workdir + submodule mount) andpath:input schemesEvalStategains reverse mapsstoreToSrcandsourceStoreToOriginalPath(concurrent_flat_map, like the existing caches), populated fromcopyPathToStore,addPath(builtins.path/filterSource/cleanSourceWith) andmountInputsrcToStorecache now lives insidefetchToStore2upstream, so the reverse map is maintained separatelyDeviations from the nix-src version (all deliberate): origin resolution centralised in
EvalState::recordPathOrigin(with a cheap early-return on repeat store paths since it's on the copy hot path); mount matching by accessor identity instead of an always-true path compare; recursion walksDerivation::inputs(SingleDerivedPath) sinceinputDrvs/inputSrcsno longer exist;sourceFilesuses the store's FS accessor so it works with chroot stores;MixPrintJSONhas no--jsonflag upstream (output is always JSON).This feature was silently dropped in the last consolidation, so it now has a functional test —
tests/functional/derivation-source-origins.shcovers-f,path:flake,git+file://…?dir=subflake,builtins.pathfilter, nullsourcePathfortoFile, symlink leaves,--no-recursive, multiple installables and eval-cache bypass.Test plan
clang-format,meson-format,shellcheckpre-commit hooks passmeson test derivation-source-originsOKnix-expr-tests,nix-fetchers-tests,nix-util-tests,filter-source,fetchGit,fetchGitSubmodules,restricted,pure-eval,derivation-json,fetchTree-file, fullflakessuite all OKnix3-derivation-source-originsmanpage builds with-Ddoc-gen=trueexa-build.ymlGenerated with Devin