Don't wait forever for a control command reply - #724
Conversation
dhcpcd-10.5 made the manager reply to a control command with an int error, which dhcpcd_readerror() consumes with a blocking read(3). Managers older than 10.5 action the command but never send a reply, so a 10.5 client talking to an older manager blocks in read(3) forever. This is easy to hit whenever the dhcpcd binaries are upgraded while the manager keeps running: every control command then hangs until the manager is restarted. On Debian it makes ifupdown's networking.service ExecStop (ifdown -a, which runs dhcpcd -k) hang, adding about three minutes to the first shutdown after the upgrade. The read is restarted after SIGTERM, so systemd has to SIGKILL it. Poll the control socket with a 5 second timeout before reading, as dhcpcd_readdump() already does for lease dumps. On timeout, warn and assume the command was actioned, which is what an older manager does. Also treat EOF as success, for a manager that closes without replying.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Team Run ID: 📒 Files selected for processing (1)
Included review availability: Your plan provides up to 2 included reviews per hour; 1 remains after this review. WalkthroughThe control socket read now uses a five-second poll timeout. Interrupted polls retry. A timeout or zero-length read returns success instead of blocking indefinitely. The manual documents behavior with older managers. ChangesControl command handling
Estimated code review effort: 2 (Simple) | ~10 minutes Merge Risk: ⚪ Minimal · up to Control commands no longer hang indefinitely against older managers that execute commands without replying, while compatible-manager replies and socket errors retain their expected behavior. The documented timeout behavior is ready to merge. Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Full details: Docstring CoverageExplanation Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 2 functions across 1 files. (1 skipped: 1 unsupported.)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
ColinMcInnes
left a comment
There was a problem hiding this comment.
Ah the joys of upgrading older systems. I'm ok with with this change, and I appreciate the log message in case the user wasn't aware the manager was out of sync.
I would recommend adding that "timed out" message to the manual, so it can be looked up if a user sees it.
A user who sees "timed out waiting for a reply from dhcpcd" should be able to look it up, so note in NOTES what it means and that restarting the running manager stops it.
|
Thanks! Done in d8c5d87 — the message is now documented under NOTES in dhcpcd.8, along with the fact that restarting the running manager makes it go away. One thing worth mentioning in case it changes your view on merging: the Debian package now restarts dhcpcd on upgrade, so the hang is already worked around there. I still think it's worth having upstream — anyone upgrading the binaries without restarting the manager hits it, on any distro — but there's no longer a fire to put out, so no urgency from my side if you'd rather not carry it. |
Fixes #723.
Problem
dhcpcd-10.5 made the manager reply to a control command with an
interror, anddhcpcd_readerror()consumes it with a blockingread(3):Managers older than 10.5 have no
dhcpcd_readerror()and never send that reply — the 10.3.2 client simply sent its command and exited. So a 10.5 client talking to a pre-10.5 manager blocks inread(3)forever. The manager receives and fully actions the command; only the reply the newer client waits for is missing.This happens whenever the dhcpcd binaries are upgraded while the manager keeps running, which is the normal case for distributions that run
dhcpcd -q -bfrom a service unit and do not restart it on upgrade. Every control command then hangs until the manager is restarted or the machine reboots.On Debian it surfaces through ifupdown:
networking.service'sExecStoprunsifdown -a, which runsdhcpcd -k <iface>, which never returns. That adds ~3 minutes to the first shutdown after the upgrade — and because the blockedread(3)is restarted afterSIGTERM, systemd's 90 s stop timeout does not end it either; it takes the second 90 s timeout and aSIGKILL.Fix
Poll the control socket with a 5 second timeout before reading, which is what
dhcpcd_readdump()immediately below already does for lease dumps. On timeout, warn and assume the command was actioned — that is exactly what an older manager did.EOFis likewise treated as success, for a manager that closes without replying.When the manager does reply, behaviour is completely unchanged:
poll()returns immediately and the existingread()path runs as before.Testing
Built and tested on Debian forky/sid, amd64, privsep enabled, interface managed by ifupdown (
iface enp1s0 inet dhcp). The same 10.5.2 tree built twice, with and without this commit, run back to back against a live manager:SIGKILLedUnpatched, against a 10.3.2 manager:
(exit 137 =
SIGKILL;timeout -s TERMwas not enough to end it, matching the systemd behaviour described above.)Patched, against the same 10.3.2 manager:
Patched, against a matching 10.5.2 manager:
Error reporting is unaffected — a command refused by the control socket still fails promptly (
main: control_open: Permission denied, exit 1, 12 ms) rather than being swallowed.Note on the timeout semantics
I chose "warn and treat as actioned" on timeout because that matches what a pre-10.5 manager actually does — it carries out the command and never replies — so failing would make
ifdownreport a failure for work that was in fact done. If you would rather have a timeout be a hard error, or prefer a different duration than the 5 s borrowed fromdhcpcd_readdump(), I am happy to change it.