A very restricted recursive nix socket in the sandbox

[roberth]

Is your feature request related to a problem? Please describe.

• For RFC 92 dynamic derivations we want to add derivations to the store from within the sandbox. While writing a derivation text to a predefined location such as $out would get the job done for a single derivation, the real power comes from adding multiple derivations. After all, if you're going to produce only one derivation, you might as well "inline" the work - not that useful.
• When running nix inside the nix sandbox, users have to perform a lot of nontrivial setup just to get instantiation to work (EDIT: this is also Make Nix configure itself better when it runs in the sandbox #8698, but I haven't removed this aspect from the issue because there's a strong interaction)

Describe the solution you'd like

• A very restricted recursive nix socket in the sandbox, supporting addToStore and little else, if anything
• An environment variable that's set in the sandbox configures nix with a default store that behaves correctly

If both solutions apply in all derivations, that would significantly improve the user experience for testing.

Describe alternatives you've considered

Without considering RFC 92, we could view as a goal: easier setup of a separate store in the sandbox. In other words, we could make this setup easier.

Additional context

• This was blocked by such a broken test setup flake: update nixpkgs: 22.11 -> 23.05 #8569 (or build with nixos-unstable channel #8590)
• Complicated test setup linked earlier
• Tracking issue for RFC 92: Dynamic derivations #6316

Priorities

Add 👍 to issues you find important.

[roberth]

Which protocol

The protocol of this recursive nix socket will be part of the derivation interface which we intend to support indefinitely. That's rather significant. We might not want to expose the actual, current daemon protocol to the builder; certainly not all of it. Some of it will be effectively excluded because we deny those operations, but we probably don't want to carry over the legacy operations either.

Returning arbitrary outputs

If we create the derivation for RFC 92 using, say, nix-instantiate, how do we put whatever path that returns as the output of the build that produces the .drv? My first thought was to support symlinks, but @Ericson2314 mentions that this is a more general opportunity that applies to all ca derivations. Interestingly, we could even support returning an output early, so that, say, a dev output unblocks dependencies even before the binary is built. This would suggest making the completion of an output part of the protocol rather than waiting for a symlink to appear (which could potentially still be modified by the build, and requires polling, inotify or equivalent which isn't great).

[Ericson2314]

#13768 take a stab at this. #15103 would have Nix be a client of varlink anyways, so it also being a server means there is some code reuse (general varlink infra) we can probably do between these two things.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/drowse-dynamic-derivations-made-easy/75540/3

[physics-enthusiast]

While writing a derivation text to a predefined location such as $out would get the job done for a single derivation, the real power comes from adding multiple derivations. After all, if you're going to produce only one derivation, you might as well "inline" the work - not that useful.

I'm probably missing something, but could you not simply generate an intermediate derivation with more outputs that fills those outputs with however many final derivations you want?

[ArchercatNEO]

While writing a derivation text to a predefined location such as $out would get the job done for a single derivation, the real power comes from adding multiple derivations. After all, if you're going to produce only one derivation, you might as well "inline" the work - not that useful.

I'm probably missing something, but could you not simply generate an intermediate derivation with more outputs that fills those outputs with however many final derivations you want?

That's more or less something I'm planning on doing for incremental compilation.

Begin with a "configure" derivation (written in nix) which has 3 outputs: $out, $stage2, $drv.
During the build stage we run the buildsystem (e.g. some nix-compatible configure) which will generate derivations (and assume it can generate .drvs without evaluating nix expressions).

To $drv we write all of our generated .drv files, one for each .c file (in a plain directory, we're going to copy from here later).
In $stage2 we're going to write a .drv with a dynamic number of outputs (one for each drv) and all it does is copy from the $drv directory to the outputs.
Then finally $out is yet another derivation, except this one has a dynamic number of inputs, one for each output of $stage2 plus whatever else is required, which links all files together and produces the binary.

I believe you can get the final binary by wrapping the $out drv in a makeScript which can helpful set extra args and env variables as well.

If I understood correctly, what this feature would allow is for us to skip the $drv and $stage2 derivations and allow for the original configure derivation to add all the inputs $out will need to the store without these intermediate steps which to be fair is very convenient.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/dynamic-derivations-next-steps/79409/1

[tomberek]

Initial implementation: #15793