Preserve non-UTF-8 paths in NAR streams

[domenkozar]

Summary

• preserve raw filesystem bytes when encoding NAR filenames and symlink targets
• order directory entries by their encoded NAR name bytes
• add regression coverage for the Latin-2 NetLock certificate filename, invalid-byte ordering, and an invalid-byte symlink target
• add a reproducible Hyperfine benchmark for comparing the working tree with a baseline revision

Root cause

On POSIX, System.Directory.listDirectory decodes undecodable filename bytes using GHC's filesystem round-trip encoding, representing them as surrogate code points in FilePath. The NAR streamer then converted the path with Text.pack and encodeUtf8, which cannot preserve those surrogate escapes.

This occurs in practice with Debian's Latin-2 NetLock CA certificate filename, which is also handled specially by nixpkgs' cacert package, and can surface inside bundled sysroots during Cachix pushes.

The streamer now encodes paths with GHC's filesystem encoding and uses the resulting bytes for both serialization and directory ordering.

Impact

Store paths containing non-UTF-8 filenames or symlink targets can be streamed without an encoding failure, while valid UTF-8 paths retain their existing representation.

Validation

• confirmed the new regression test fails against the original streamer before applying the fix
• all 22 hnix-store-nar tests pass
• the regression NAR matches nix-store --dump byte-for-byte
• benchmark scripts pass Bash syntax checks and the benchmark Nix shell parses
• Hyperfine benchmark: 5,000 files, 7 dumps per process, 5 warmups, 30 measurements in each ordering (60 per variant)
  • combined median: baseline 1.3853 s, fixed 1.3842 s
  • the 0.09% median difference is below observed noise; no measurable wall-time regression

[domenkozar]

@Ericson2314 @sorki

[sorki]

Thanks!

[Ericson2314]

Thanks @domenkozar for making it, and thanks @sorki for already reviewing it :)