Remote building over ssh can cause hangs on lots of output, `ssh-ng://` seems fine

[nh2]

Today I ran a

nix-store -r --builders 'root@othermachine x86_64-linux,i686-linux - 16 - big-parallel,kvm,nixos-test' --option builders-use-substitutes true --max-jobs 0 --keep-going /nix/store/rsyb453zyh05kpsnqpzasmrzzp4lxz2i-openssl-1.1.1w.drv

and it hung forever.

On the remote side, there's still a nix-store --serve --write hanging around that seems to be deadlocked.

I suspect this happened because my client (laptop) disconnected from the remote builder's SSH connection due to an Internet drop.

Cancelling that nix-store -r on the client and re-running it also causes it to be hung forever. Only manually SIGKILLing the process on the remote fixes it.

Expected behavior

Nix remote builiding should not cause such random hangs that live forever and need humand intervation.

If a remote client disconnects, eventually that build should be cleaned up (at least at the TCP timeout), which isn't happening.

Metadata

Client and server nix version: nix-env (Nix) 2.34.8

Checklist

• checked latest Nix manual (source)
• checked open bug issues and pull requests for possible duplicates
• reviewed the contributing guide

---

Add 👍 to issues you find important.

[nh2]

I did a longer session with the Claude Opus 5 LLM to investigate the issue on my live hanging system.

It also managed to write add a failing test reproducer to nix's test suite.

That is in my branch issue-16437-build-remote-log-deadlock-repro: cc06c40...nh2:nix:issue-16437-build-remote-log-deadlock-repro

Note that in contrast to most of my LLM-using work, tihs is not fully reviewed by me yet, because I only had a couple hours timebox for myself on this today and do not have more time currently.
But the reproducer is probably useful (same for the fact that it doesn't repro on ssh-ng for people looking for a workaround) and I want to share that output to save some time for those that actually do have time to fix this.

LLM output below

Summary

nix-store --serve writes every build log line to its own stderr with a blocking write(), executed from inside its single-threaded Worker event loop.
That same loop is the only thing draining the builder's output.
So if the reader of the remote's stderr stops consuming, the loop blocks, the builder's pty fills, the builder blocks, and the build can never finish or be cleaned up.

On the legacy ssh:// store the remote's stderr is a 64 KiB pipe, so any build emitting more than that is exposed.
max-silent-time and timeout cannot rescue it, because the loop that would enforce them is itself stuck.

ssh-ng:// is unaffected, since it reports logs in-band over the worker protocol.

Impact

A remote build hangs indefinitely with no output, no error, and no timeout.

In the case that prompted this report, the client machine's public IP changed overnight, so the client process was gone entirely.
That left an orphaned build on the builder wedged forever rather than failing.
sshd's default keepalive was 48 minutes away and no ClientAlive* was configured, so nothing noticed.

This is easy to hit unintentionally, because a --builders entry written as a bare user@host silently becomes ssh://, i.e. the affected legacy store.
Users need not have typed a scheme at all.

The deadlock cycle

builder            blocked in write() -- its output pty is full
  ^ waits for                           v needs draining by
nix-store --serve  blocked in write(fd 2) -- its stderr pipe is full
  ^ waits for                           v needs draining by
whoever reads the remote's stderr      -- gone, or not reading

Evidence from a live hang

Observed with Nix 2.31.5 on both sides, on a builder here called corp1, while building openssl-1.1.1w (whose make test is very chatty).

The serve process on the builder:

wchan:   pipe_write
syscall: 1 0x2 0x61740a0335a0 0xbb      # write(fd=2, buf, 187)
stack:
  [<0>] pipe_write+0x35e/0x6c0
  [<0>] vfs_write+0x3a5/0x440
  [<0>] ksys_write+0xb9/0xf0

Five samples 2 s apart were byte-for-byte identical, and its stderr pipe was pinned at buffered=64547 capacity=65536 — full, and never draining.
The 187 bytes it was trying to write were a single openssl test result line.

gdb backtrace of thread 1, showing the blocking write inside the event loop:

#0  write ()
#1  nix::writeFull(int, string_view, bool)
#2  nix::writeToStderr(string_view)
#3  nix::SimpleLogger::log(nix::Verbosity, string_view)
#4  nix::SimpleLogger::result(unsigned long, nix::ResultType, vector<Field> const&)
#5  nix::Activity::result<string>(nix::ResultType, string const&) const
#6  nix::DerivationBuildingGoal::flushLine()
#7  nix::DerivationBuildingGoal::handleChildOutput(int, string_view)
#8  nix::MuxablePipePollState::iterate(...)
#9  nix::Worker::waitForInput()
#10 nix::Worker::run(...)
#11 nix::Store::buildDerivation(...)
#12 nix_store::opServe(...)

The builder underneath, blocked on the other side of the same cycle:

perl run_tests.pl:
  syscall: 1 0x1 ...            # write(fd=1)
  stack:   n_tty_write+0x406/0x500

Its parent make and bash processes were all in do_wait.
sshd was in ppoll([{fd=7,...}]), polling only its TCP socket and not reading the stderr pipe.
TCP itself was healthy and simply idle (snd_wnd:26345472, empty queues, lastsnd/lastrcv roughly 72 minutes earlier).

Root cause in the code

1. libstore/machines.cc:20-27 prepends ssh:// to any --builders entry lacking :// and / and not equal to auto/daemon/local.
   A bare user@host therefore selects LegacySSHStore.
2. libstore/machines.cc:65-68 sets log-fd=4 only for scheme ssh.
3. libstore/ssh.cc:182 dups that fd over the ssh child's STDERR_FILENO.
4. fd 4 is the write end of the 64 KiB builderOut pipe from libstore/unix/build/hook-instance.cc:62-68, whose own comment says it exists to carry occasional SSH error messages — not an entire build log.
5. On the remote, opServe runs a normal Worker loop via store->buildDerivation(...).
6. libstore/build/worker.cc:478 calls handleChildOutput inline from waitForInput().
7. That reaches BuildLog::flushLine(), which calls act->result(resBuildLogLine, ...).
8. opServe installs no structured logger, so the global default applies: libutil/logging.cc:39, makeSimpleLogger(true) — note printBuildLogs = true.
9. SimpleLogger::result() forwards resBuildLogLine to printError → writeToStderr → the blocking writeFull(fd, s, false).

Two near-misses worth noting.

verbosity = lvlError in getBuildSettings() does not suppress these lines, since result() routes them via printError (itself lvlError) rather than a verbosity-gated path.

MonitorFdHup monitor(in.fd) watches stdin.
The process is wedged writing stderr, so it never triggers.

Also, setLogFormat is only called for the nix CLI at nix/main.cc:432, never for legacy nix-store.
So makeDefaultLogger()'s LogFormat::raw → makeSimpleLogger(false) mapping, which would have disabled this path, does not apply.

Version

Reproduced on master (cc06c406c), and originally hit on 2.31.5.

The relevant code is unchanged between them: the only diffs on this path since are cosmetic (noexcept from 137096350, std::span from 83a5d1d3e).
Adding noexcept addresses exceptions but not blocking — the body is still writeFull(fd, s, false).

Reproduction

A functional test is attached in the linked PR/branch (tests/functional/build-remote-log-deadlock.{sh,nix}).
It needs no SSH, no network, and no second machine: ssh://localhost bypasses ssh and execs the remote program as a child, as already documented at tests/functional/build-remote.sh:31.

The three necessary ingredients:

1. A derivation emitting well over 64 KiB of log output (the test uses 4000 lines, roughly 280 KiB).
2. Scheme ssh://, so log-fd=4 is set.
3. A remote whose stderr nobody drains, arranged with remote-program=<wrapper> where the wrapper is exec nix-store "$@" 2>SOME_FIFO, while the test holds the fifo's read end open without ever reading it.

On unpatched master the test hangs and is killed by timeout after 30 s.
The wedged process shows exactly the production signature: wchan=pipe_write with syscall=1 0x2 (write on fd 2).

Measured with the same derivation and the same undrained stderr:

builder	result	bytes remote wrote to its own stderr
ssh:// (legacy serve protocol)	hangs	250890
ssh-ng:// (worker protocol)	completes, exit 0	0

Diagnosing this in the wild

On the builder:

p=$(pgrep -f 'nix-store --serve')
cat /proc/$p/wchan      # pipe_write
cat /proc/$p/syscall    # 1 0x2 ...   i.e. write(fd=2, ...)

wchan == pipe_write plus a syscall beginning 1 0x2 is the signature.
The builder beneath it will simultaneously be blocked writing its own output (n_tty_write for a pty).
To clear it, kill -9 the serve process; SIGTERM is not enough, since it is blocked in write().

Note on a fix that does not work

Suppressing the build log in opServe (logger = makeSimpleLogger(/*printBuildLogs=*/false)) removes the deadlock — a 30.27 s hang becomes 0.23 s.
But it also destroys remote build logs: the client went from 4000 log lines to 0.

That blocking stderr write is the build-log transport for legacy ssh://.
So a correct fix must keep the data flowing while making it unable to block the event loop.

Possible directions

1. Never block the event loop on a log write: make it non-blocking and tolerate loss, or buffer and drain via the poll loop (POLLOUT).
   This matches the intent already stated in writeFullLogging's own comment, that logging writes are best-effort and must not wedge the process.
   Bounded buffering with drop-oldest seems a reasonable trade: build logs are advisory, a permanently wedged build is not.
2. Detect a dead or stalled peer while blocked on stderr, not just stdin, since MonitorFdHup currently only covers the latter.
3. Reconsider the 64 KiB builderOut pipe, sized for occasional SSH error messages but now carrying whole build logs.
   Note build-remote.cc:273's close(5) also removes a potential local reader.
4. Possibly reconsider defaulting a bare hostname to the legacy ssh:// store, which is what silently exposes users to this path.

Workarounds

• Use ssh-ng:// explicitly in --builders.
• Set a non-zero max-silent-time, though this is imperfect because enforcement lives in the stuck loop.
• On the builder, configure ClientAliveInterval and ClientAliveCountMax so dead peers are reaped promptly.