Create CBOR representation of derivations

[domenkozar]

Add CBOR version 1 import and export for existing derivations through nix derivation show --cbor and nix derivation add --cbor. Builder paths, arguments, and environment names and values can contain arbitrary bytes, including values that the current JSON format cannot represent.

JSON and CBOR share derivation field definitions and output/input conversion logic. The existing JSON wire format remains unchanged; this does not introduce non-standard JSON \x escapes.

The encoder emits deterministic CBOR with shortest integer and length encodings, definite lengths, ordered map keys, and sorted sets. The decoder accepts nonminimal encodings and unsorted maps and sets, then normalizes them on export. Different accepted encodings produce the same normalized CBOR and derivation path.

Structured attributes are carried as the original JSON bytes, preserving whitespace, escaping, key order, and number spellings. Their numbers are not converted to CBOR integers or floats. This preserves existing derivation identity: import uses the current validation, ATerm storage, and hashing rules, and CBOR bytes are not hashed directly. A new derivation addressing scheme, semantic JSON canonicalization, and placeholder redesign are outside this PR's scope.

The manual documents the schema, normalization rules, and identity guarantees. Regression tests cover arbitrary bytes, malformed input, deterministic encoding, normalization without identity changes, and verbatim structured attributes.

Validation with rebuilt store library, CLI, and unit-test binary: all 28 focused CBOR/ATerm/JSON tests pass, and both main and CA derivation-json functional tests pass. Formatting and git diff --check pass. The broader store suite passed 796 of 808 tests; the remaining 12 failed during OpenSSL setup or because /bin/sh was unavailable in the test build environment.

Related: JSON byte limitation discussion and the corresponding Rust nix-derivation PR.

Assisted-by: Codex (GPT-6)

[Ericson2314]

Is there a way we can reuse more of the definition between the JSON and CBOR formats?

My ideal would be a single "aeson instance", using an AST with CBOR's two types of strings (binary and unicode), and the former when translated to JSON uses non-standard \x escape sequences.

I'm still trying to land this monster #10760 and so having fewer format implementations to keep in sync really does help not make that implementation metastasize further.

I do need to write down all the derivation plans in my head regardless, but also it would probably good to discuss live in a Nix meeting.

[xokdvium]

I think the consensus is that CBOR is not suitable for derivations because it doesn't have a canonical representation? On those grounds of course JSON can be considered unsuitable too and should be considered a convenience for CLI/internals tests only.

[domenkozar]

I think the consensus is that CBOR is not suitable for derivations because it doesn't have a canonical representation? On those grounds of course JSON can be considered unsuitable too and should be considered a convenience for CLI/internals tests only.

You have to define the representation, which this PR does.

[xokdvium]

I'm referring more to:

The Major type 0 numbers come in five different encoding sizes, as indicated by their initial byte: immediate ("1+0") encoding (0..23), one-byte ("1+1") (0..255), two-byte ("1+2", 0..65535), four-byte, and eight-byte. The Preferred Serialization always uses the shortest of the major type 0 encodings available for an unsigned integer. The intention is that there is no semantic difference between the major type 0 encodings, and there also is no semantic difference between major type 0 and tag 2. This means that Preferred Serialization always uses major type 0 over tag 2 when possible, and the shortest encoding of these (and thus no leading zero bytes for the tagged encodings). Major type 1 and tag 3 are analogous.

Does this PR enforce a particular way to serialise integers in structured attrs? If not then we'd be kinda in the same hole of having equivalent codings of the same thing which doesn't seem good for derivations that are content addressed (well, if the intention is to have NG derivations and addressing schemes). It's also quite hard to read, so forgive me if I'm missing the particulars of how structured attrs are handled - I've only skimmed the code here.

There's of course other concerns that would have to be included in the ng format (like @Ericson2314 mentioned above). For one, that would be better handling of placeholders that doesn't necessitate resorting to string rewriting and such. Taking an opportunity to collect design considerations for such seems very prudent

[domenkozar]

Thanks for the feedback. JSON and CBOR now share derivation field definitions and input/output conversion logic. The JSON wire format remains unchanged; this doesn’t implement the suggested \x escapes.

CBOR output uses deterministic ordering and shortest integer/length encodings. Import accepts alternative encodings and normalizes them on export, with tests confirming identical normalized bytes and derivation paths.

Structured attributes retain their original JSON bytes, including number spellings. Their numbers aren’t encoded as CBOR integers or floats, and preserving those bytes preserves existing derivation identity.

This PR provides interchange for existing derivations; CBOR bytes aren’t hashed directly. A new addressing scheme and placeholder redesign would need a separate design discussion.