Draft: libgit2 and tarball unpacking performance improvements

[xokdvium]

Motivation

Use these 3 simple hacks to get 3x less CPU time for tarball unpacking (that's a fresh nixpkgs tarball import into a new cache):

Command being timed: "nix flake metadata file:///home/xokdvium/work/code/nix/unstable.tar.zst --refresh -vvvv"
        User time (seconds): 20.51
        System time (seconds): 0.66
        Percent of CPU this job got: 381%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:05.55
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 1087388

With this whole stack applied:

Command being timed: "result/bin/nix flake metadata file:///home/xokdvium/work/code/nix/unstable.tar.zst --refresh -vvvv"
        User time (seconds): 6.29
        System time (seconds): 0.43
        Percent of CPU this job got: 220%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:03.04
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 631872

I'll try to upstream all this stuff (patches 2 TBD and fix for patch 3 is up in libgit2/libgit2#7366). Draft for now because obviously we won't be patching libgit2.

Context

---

Add 👍 to pull requests you find important.

The Nix maintainer team uses a GitHub project board to schedule and track reviews.

[xokdvium]

Huh, the last patch doesn't work as expected and modifies the files on disk. One second (I guess we might as well persist it in the tarball cache maybe? But I'd prefer to set that in-memory).