Skip to content

queryMissing: read derivations from the eval store #16433

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Mic92 wants to merge 1 commit into
base: master
Choose a base branch
from
Open

queryMissing: read derivations from the eval store #16433

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions src/libcmd/installables.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -599,7 +599,7 @@ std::vector<std::pair<ref<Installable>, BuiltPathWithResult>> Installable::build

case Realise::Nothing:
case Realise::Derivation:
printMissing(store, pathsToBuild, lvlError);
printMissing(store, pathsToBuild, lvlError, &*evalStore);

for (auto & path : pathsToBuild) {
for (auto & aux : backmap[path]) {
Expand Down Expand Up @@ -629,7 +629,7 @@ std::vector<std::pair<ref<Installable>, BuiltPathWithResult>> Installable::build

case Realise::Outputs: {
if (settings.printMissing)
printMissing(store, pathsToBuild, lvlInfo);
printMissing(store, pathsToBuild, lvlInfo, &*evalStore);

auto buildResults = store->getBuilder(evalStore)->buildPathsWithResults(pathsToBuild, bMode);
throwBuildErrors(buildResults, *store);
Expand Down
3 changes: 2 additions & 1 deletion src/libmain/include/nix/main/shared.hh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,8 @@ void printGCWarning();
class Store;
struct MissingPaths;

void printMissing(ref<Store> store, const std::vector<DerivedPath> & paths, Verbosity lvl = lvlInfo);
void printMissing(
ref<Store> store, const std::vector<DerivedPath> & paths, Verbosity lvl = lvlInfo, Store * evalStore = nullptr);

void printMissing(ref<Store> store, const MissingPaths & missing, Verbosity lvl = lvlInfo);

Expand Down
4 changes: 2 additions & 2 deletions src/libmain/shared.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,9 +54,9 @@ void printGCWarning()
"the result might be removed by the garbage collector");
}

void printMissing(ref<Store> store, const std::vector<DerivedPath> & paths, Verbosity lvl)
void printMissing(ref<Store> store, const std::vector<DerivedPath> & paths, Verbosity lvl, Store * evalStore)
{
printMissing(store, store->queryMissing(paths), lvl);
printMissing(store, store->queryMissing(paths, evalStore), lvl);
}

void printMissing(ref<Store> store, const MissingPaths & missing, Verbosity lvl)
Expand Down
2 changes: 1 addition & 1 deletion src/libstore/include/nix/store/remote-store.hh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,7 +146,7 @@ public:

void addSignatures(const StorePath & storePath, const std::set<Signature> & sigs) override;

MissingPaths queryMissing(const std::vector<DerivedPath> & targets) override;
MissingPaths queryMissing(const std::vector<DerivedPath> & targets, Store * evalStore = nullptr) override;

void addBuildLog(const StorePath & drvPath, std::string_view log) override;

Expand Down
5 changes: 4 additions & 1 deletion src/libstore/include/nix/store/store-api.hh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -944,8 +944,11 @@ public:
* Given a set of paths that are to be built, return the set of
* derivations that will be built, and the set of output paths that
* will be substituted.
*
* @param evalStore If given, derivations not (yet) present in this
* store are read from there, like the builder does.
*/
virtual MissingPaths queryMissing(const std::vector<DerivedPath> & targets);
virtual MissingPaths queryMissing(const std::vector<DerivedPath> & targets, Store * evalStore = nullptr);

@Mic92 Mic92 Sep 6, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I kept the existing pattern of having pointers here. Also we might have better mechanisms now to solve this. But this looks like another PR.


/**
* Sort a set of paths topologically under the references
Expand Down
11 changes: 7 additions & 4 deletions src/libstore/misc.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -167,10 +167,12 @@ void Store::querySubstitutablePathInfos(const StorePathCAMap & paths, Substituta
std::rethrow_exception(ex);
}

MissingPaths Store::queryMissing(const std::vector<DerivedPath> & targets)
MissingPaths Store::queryMissing(const std::vector<DerivedPath> & targets, Store * evalStore_)
{
Activity act(*logger, lvlDebug, actUnknown, "querying info about missing paths");

auto & evalStore = evalStore_ ? *evalStore_ : *this;

MissingPaths res;

auto mustBuildDrv = [&](const StorePath & drvPath, const Derivation & drv, std::set<DerivedPath> & edges) {
Expand Down Expand Up @@ -202,7 +204,8 @@ MissingPaths Store::queryMissing(const std::vector<DerivedPath> & targets)
}
auto & drvPath = drvPathP->path;

if (!isValidPath(drvPath)) {
Comment thread
Mic92 marked this conversation as resolved.
auto * drvStore = evalStore.isValidPath(drvPath) ? &evalStore : this;
if (!drvStore->isValidPath(drvPath)) {
// FIXME: we could try to substitute the derivation.
res.unknown.insert(drvPath);
co_return;
Expand All @@ -212,7 +215,7 @@ MissingPaths Store::queryMissing(const std::vector<DerivedPath> & targets)
/* true for regular derivations, and CA derivations for which we
have a trust mapping for all wanted outputs. */
auto knownOutputPaths = true;
for (auto & [outputName, pathOpt] : queryPartialDerivationOutputMap(drvPath)) {
for (auto & [outputName, pathOpt] : queryPartialDerivationOutputMap(drvPath, drvStore)) {
if (!pathOpt) {
knownOutputPaths = false;
break;
Expand All @@ -223,7 +226,7 @@ MissingPaths Store::queryMissing(const std::vector<DerivedPath> & targets)
if (knownOutputPaths && invalid.empty())
co_return;

auto drv = make_ref<Derivation>(derivationFromPath(drvPath));
auto drv = make_ref<Derivation>(drvStore->readDerivation(drvPath));
DerivationOptions<SingleDerivedPath> drvOptions;
try {
// FIXME: this is a lot of work just to get the value
Expand Down
7 changes: 4 additions & 3 deletions src/libstore/remote-store.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -840,9 +840,9 @@ void RemoteStore::addSignatures(const StorePath & storePath, const std::set<Sign
readInt(conn->from);
}

MissingPaths RemoteStore::queryMissing(const std::vector<DerivedPath> & targets)
MissingPaths RemoteStore::queryMissing(const std::vector<DerivedPath> & targets, Store * evalStore)
{
{
if (!evalStore || evalStore == this) {
auto conn(getConnection());
if (conn->protoVersion.number < WorkerProto::Version::Number{1, 19})
// Don't hold the connection handle in the fallback case
Expand All @@ -860,7 +860,8 @@ MissingPaths RemoteStore::queryMissing(const std::vector<DerivedPath> & targets)
}

fallback:
return Store::queryMissing(targets);
// The daemon cannot see the eval store, so do the traversal here.
return Store::queryMissing(targets, evalStore);
}

void RemoteStore::addBuildLog(const StorePath & drvPath, std::string_view log)
Expand Down
6 changes: 3 additions & 3 deletions src/libstore/restricted-store.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,7 @@ struct RestrictedStore : public virtual IndirectRootStore, public virtual GcStor
unsupported("addSignatures");
}

MissingPaths queryMissing(const std::vector<DerivedPath> & targets) override;
MissingPaths queryMissing(const std::vector<DerivedPath> & targets, Store * evalStore = nullptr) override;

virtual std::optional<std::string> getBuildLogExact(const StorePath & path) override
{
Expand Down Expand Up @@ -370,7 +370,7 @@ RestrictedBuilder::buildPathsWithResults(const std::vector<DerivedPath> & paths,
return results;
}

MissingPaths RestrictedStore::queryMissing(const std::vector<DerivedPath> & targets)
MissingPaths RestrictedStore::queryMissing(const std::vector<DerivedPath> & targets, Store * evalStore)
{
/* This is slightly impure since it leaks information to the
client about what paths will be built/substituted or are
Expand All @@ -385,7 +385,7 @@ MissingPaths RestrictedStore::queryMissing(const std::vector<DerivedPath> & targ
unknown.insert(pathPartOfReq(req));
}

auto res = next->queryMissing(allowed);
auto res = next->queryMissing(allowed, evalStore);

for (auto & p : unknown)
res.unknown.insert(p);
Expand Down
2 changes: 1 addition & 1 deletion src/nix/build.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,7 @@ struct CmdBuild : InstallablesCommand, MixOutLinkByDefault, MixDryRun, MixJSON,
for (auto & b : i->toDerivedPaths())
pathsToBuild.push_back(b.path);

printMissing(store, pathsToBuild, lvlError);
printMissing(store, pathsToBuild, lvlError, &*getEvalStore());

if (json)
printJSON(derivedPathsToJSON(pathsToBuild, *store));
Expand Down
2 changes: 1 addition & 1 deletion src/nix/flake.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -814,7 +814,7 @@ struct CmdFlakeCheck : FlakeCommand, MixPrintOutPaths, MixOutLinkBase
// For now, we skip building derivations whose outputs are already available
// via substitution, as `nix flake check` only needs to verify buildability,
// not actually produce the outputs.
auto missing = store->queryMissing(drvPaths);
auto missing = store->queryMissing(drvPaths, &*getEvalStore());

std::vector<DerivedPath> toBuild;
for (auto & path : missing.willBuild) {
Expand Down
2 changes: 1 addition & 1 deletion src/nix/nix-build/nix-build.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -448,7 +448,7 @@ static void main_nix_build(int argc, char ** argv)

auto buildPaths = [&](const std::vector<DerivedPath> & paths) {
if (settings.printMissing)
printMissing(ref<Store>(store), paths);
printMissing(ref<Store>(store), paths, lvlInfo, &*evalStore);

if (!dryRun)
store->getBuilder(evalStore)->buildPaths(paths, buildMode);
Expand Down
6 changes: 6 additions & 0 deletions tests/functional/eval-store.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,12 @@ needLocalStore "“--eval-store” doesn't achieve much with the daemon"

eval_store=$TEST_ROOT/eval-store

# The build store has not seen the .drv files yet. queryMissing must read
# them from the eval store instead of reporting them as unknown.
out=$(nix build -f dependencies.nix --eval-store "$eval_store" --dry-run 2>&1)
[[ $out != *"don't know how to build"* ]]
[[ $out == *"will be built"* ]]

nix build -f dependencies.nix --eval-store "$eval_store" -o "$TEST_ROOT/result"
[[ -e $TEST_ROOT/result/foobar ]]
if [[ -z "${NIX_TESTS_CA_BY_DEFAULT:-}" ]]; then
Expand Down
Loading