Sketch at Varlink interface for #8602

[Ericson2314]

Motivation

Just a draft PR. Really, just putting this up for ease of liking for conversation at this moment.

#8602

Context

---

Add 👍 to pull requests you find important.

The Nix maintainer team uses a GitHub project board to schedule and track reviews.

[artemist]

If non-fixed-output derivations can use dynamic derivations to create fixed-output derivations then that can be used to send data to the internet: e.g. derivation A that should be fully sandboxed could create fixed-output derivation B with a curl command sending information to a remote server.

[xokdvium]

Isn't that an existing issue with recursive-nix?

[artemist]

Yes, I had just noticed the issue now though

[Ericson2314]

We could prohibit that.

[Ericson2314]

This is wrong, the file descriptor is a *socket( over which a NAR and (later) also arbitrary many file descriptors for individual files are sent.

[domenkozar]

I think this draft is the right place to define a dedicated builder-facing API. I would prefer not to stabilize the Nix daemon worker protocol as a derivation-facing ABI.

#15793 has now merged builder-rpc-v0. Its stated reason for reusing the daemon protocol was reducing implementation work, and the current manual describes it as a limited form of the Nix daemon socket. The discussion also characterized v0 as exploratory and possibly not something that will be stabilized. Could we explicitly keep it as a prototype/compatibility mechanism while developing the dedicated interface here?

The three operations in this PR look like the right capability boundary:

• AddToStore
• AddDerivation
• SubmitOutput

I would change how AddToStore transports its NAR. The current design requires SCM_RIGHTS, but file-descriptor passing is deliberately not part of Varlink because it is transport-specific. It also excludes Windows named pipes. The Varlink specification recommends upgrade for large or non-JSON payloads.

One possible API is:

method BeginAddToStore(
    name: string,
    method: string
) -> (uploadId: string)

method FinishAddToStore(
    uploadId: string
) -> (path: string)

BeginAddToStore is called with the standard Varlink upgrade: true header. The server confirms the upgrade and returns uploadId; the upgraded connection then carries exactly one NAR. The client closes the connection after writing it. FinishAddToStore runs on a fresh Varlink connection, waits for ingestion to finish, and returns the resulting store path or a typed error.

This does not introduce another RPC or binary framing format. Control messages remain Varlink, and the only post-upgrade payload is one existing NAR stream. JSON-RPC 2.0 would not solve this part: its base specification defines neither stream framing nor binary transfer, whereas Varlink already provides an IDL, typed errors, introspection, and the upgrade mechanism.

The same interface can be transported over:

• Unix: an AF_UNIX stream socket.
• Windows: a local duplex, byte-mode named pipe carrying the same NUL-delimited Varlink records.

Varlink permits any connection-oriented transport, but it does not currently define a standard Windows named-pipe address or guarantee that bindings implement one. We would therefore need a named-pipe transport implementation and address convention, without changing the Varlink call/reply semantics. For concurrent calls, the Windows listener needs multiple pipe instances or overlapped I/O. It should use PIPE_REJECT_REMOTE_CLIENTS and an explicit DACL restricted to the identity assigned to the build.

Independently of transport, the service should remain per-build and capability-limited:

• no general store/daemon operations;
• references restricted to the parent’s permitted inputs and objects created through this endpoint;
• output names restricted to those declared by the parent;
• aborted uploads cleaned up and subject to resource limits.

There is also an unresolved security rule: reference restrictions alone do not prevent a generated derivation from acquiring capabilities its parent lacked, for example by generating a fixed-output derivation that receives network access. Before stabilization, AddDerivation needs an explicit rule preventing generated derivations from escalating impurity, network, or other build capabilities.

Would maintainers be open to revising this draft along those lines, using #15810 for the bootstrap-friendly client/shim story, and keeping the daemon-protocol implementation confined to experimental builder-rpc-v0?

[Ericson2314]

Thanks @domenkozar, that is very much along the line of what I was thinking for next steps, and I am glad to hear that you are interested in this feature and figuring out its details.

Some things I would like to add are:

• systemd has lots of precedent for sockets and SCM_RIGHTS. varlink vs varlinkctl, and the latter's --push-fd= indicates how systemd is effectively making its own varlink++. On Unix, I do think that is the one to target.
• On windows we can do WSADuplicateSocket and DuplicateHandle for something a bit similar to SCM_RIGHTS, but just (alas) not tied to the socket.
• an in-band fallback when neither are available is still fine with me.

@xokdvium and I were interested in the SCM_RIGHTS approach because it has the potential to be much more efficient, thanks to stuff like copy_file_range where data doesn't need to be transferred at all, and also support a rootless Nix daemon securely. (With toot you can receive paths and do copy_file_range on freshly-opened files, but without root you wouldn't be able to open the paths received in general.)

[artemist]

Derivations with network access generated by cargo-dyndrv sound convenient, but I don't think they're needed in the majority of cases.

e.g. before a program can generate dynamic derivations for Rust crates, it needs the full dependency tree crate manifests. By far the easiest way to accomplish this is importCargoLock, which downloads the entire sources of every dependency in the graph, so by the time the generator runs there is no more network access to be done.

[artemist]

I just noticed that JSON derivations are less flexible than ATerm, since ATerm derivations are capable of representing bytes that are not valid Unicode, but JSON is not (the specification describes it as a sequence of Unicode codepoints, not a sequence of bytes. It can then be encoded in any Unicode encoding, though Nix always uses UTF-8 iirc.)

e.g. try:

drv="$(nix eval --raw --expr $(echo -ne '(builtins.derivation { name = "a"; system = "x86_64-linux"; builder = "/bin/sh"; args = [ "-c" "echo -n \xfe > $out" ]; }).drvPath'))"
# succeeds
cat $drv
# succeeds
xxd $(nix build --print-out-paths $drv^out)
# fails
nix derivation show $drv

This would not be an issue with e.g. CBOR, which has native support for binary strings.

[domenkozar]

I just noticed that JSON derivations are less flexible than ATerm, since ATerm derivations are capable of representing bytes that are not valid Unicode, but JSON is not (the specification describes it as a sequence of Unicode codepoints, not a sequence of bytes. It can then be encoded in any Unicode encoding, though Nix always uses UTF-8 iirc.)

e.g. try:

drv="$(nix eval --raw --expr $(echo -ne '(builtins.derivation { name = "a"; system = "x86_64-linux"; builder = "/bin/sh"; args = [ "-c" "echo -n \xfe > $out" ]; }).drvPath'))"
# succeeds
cat $drv
# succeeds
xxd $(nix build --print-out-paths $drv^out)
# fails
nix derivation show $drv

This would not be an issue with e.g. CBOR, which has native support for binary strings.

Thanks @artemist

To me that sounds like Varlink has hit a blocker, we should also strive for maximum performance.

I'll start on CBOR representation of Derivation and then look into using CDDL

[domenkozar]

Playing with #16429 I'm thinking we need separate protocol for generating derivations and content.