git: reuse cached history for exact revision fetches

[0xdeafbeef]

Motivation

TL;DR: Nix was downloading 500 MiB from the same Git repository on every update, even though the previous revision and its history were already in Nix's Git cache.

This repository advances through merges into master, so every revision fetched during an update is a descendant of the previous one.
Git should only need to send the objects added since that revision.
Instead, it was sending the whole reachable history again on every update.

When Nix fetches an exact Git revision, it runs roughly:

git fetch <url> <commit-oid>

Git stores the received objects in Nix's local bare repository cache and writes the fetched object to FETCH_HEAD, but it does not create a persistent ref.

On the next exact-revision fetch, Git normally negotiates from local refs.
The previous revision is only referenced by FETCH_HEAD, so Git does not advertise it to the server as a have.
The server assumes the client does not have that history and sends it again.

A minimal repository with two commits reproduces the same problem.
Each commit adds three objects: a commit, a tree, and a blob.

Before this change:

first fetch:  Total 3
second fetch: Total 6

The second fetch receives both commits and their trees and blobs, even though the first three objects are already cached.

After this change:

first fetch:  Total 3
second fetch: Total 3

The fetched revision does not change.
Git just stops sending objects that are already present locally.

Context

Implementation

Before running the external git fetch, GitRepoImpl::fetch() asks libgit2 to resolve:

FETCH_HEAD^{commit}

If that succeeds, Nix passes the resulting commit OID as a Git negotiation tip.

--negotiation-tip replaces Git's normal tip set rather than extending it.
To preserve normal negotiation, Nix also passes refs/* when the cached repository contains refs.
When the repository has no refs, it passes only the previous commit OID, avoiding Git's warning about an unmatched refs/* pattern.

If FETCH_HEAD is missing or does not resolve to a commit, Nix runs the previous fetch command without extra negotiation tips.
GIT_ENOTFOUND, GIT_EINVALIDSPEC, and GIT_EPEEL are treated as an unusable hint.
Other libgit2 errors remain fatal.

Why this is safe

Negotiation tips only tell the server which objects the client already has.
They do not change the URL, refspec, or requested revision.

The additional OID comes from the local object database and is explicitly peeled to a commit.
Existing refs are still included when present.

If the previous commit is unrelated to the requested revision, the server cannot use it as common history and sends the normal pack.
If FETCH_HEAD is missing or unusable, Nix falls back to the previous behavior.

This can only reduce the transfer.
It cannot select a different revision or make Nix accept an invalid object.

Alternative considered

I first considered storing the previous revision in a persistent internal ref such as refs/nix/fetch-cache.

That would be straightforward for Git, but unsafe for Nix.
Users can request arbitrary full ref names, so an internal ref could collide with a real remote ref and affect revision resolution.

Reusing FETCH_HEAD provides the negotiation hint without adding an internal ref namespace.

Git compatibility

--negotiation-tip requires Git 2.19 or newer. It's a release from 2018 so everyone should have it it guess.
The requirement is documented in the release note.

Git 2.55 added the additive option:

--negotiation-include=<oid>

There is a TODO to replace the current negotiation-tip workaround with this option once Git 2.55 becomes the minimum supported version.

Testing

Added two functional tests:

• exact-rev-fetch-cache.sh performs two related exact-revision fetches, checks that the second fetch sends have <first-revision>, and expects both fetches to receive exactly three objects. It also checks that Git does not emit an unmatched refs/* warning.
• exact-rev-fetch-head-cache.sh leaves an annotated tag pointing to a blob in FETCH_HEAD, verifies that Nix rejects it as a revision, and then verifies that a valid exact-revision fetch from the same cache still succeeds.

Also ran:

• the libnixfetchers build;
• shell syntax checks for both functional tests;

[0xdeafbeef]

cc @xokdvium @Mic92 since you seem to be looking at the Git fetcher bits right now :)