server/world: fix Loader deadlocking the transaction goroutine against itself - #1419
Open
schphe wants to merge 2 commits into
Open
server/world: fix Loader deadlocking the transaction goroutine against itself#1419schphe wants to merge 2 commits into
schphe wants to merge 2 commits into
Conversation
schphe
force-pushed
the
fix/loader-viewchunk-deadlock
branch
4 times, most recently
from
August 18, 2026 00:45
0defa46 to
463e167
Compare
…t itself Loader.viewChunk held the Loader's mutex across both of its call-outs, the Viewer's ViewChunk and World.addViewer. Showing an entity to a Viewer runs arbitrary code: a session encoding a player's metadata calls Player.Breathing, which reads the liquid at the player's eye, and that position need not be in the chunk being viewed. Reading a block from a chunk that has a background load in flight takes the chunkFromAsyncPool path, where chunkRequest.doImmediate runs the request's callbacks inline on the calling goroutine. The callback a Loader registers in Load is viewChunk, so viewChunk re-entered itself on the same goroutine and blocked forever on a lock it already held. That goroutine is the world's owner, so the world stops permanently while the process stays up: no player on it receives anything further, no inbound packet is handled, and goroutines elsewhere keep running and logging as though nothing is wrong. It is not limited to the player who triggered it, and the Loader that deadlocks need not be their own. Teleporting a player into a chunk that is within their render distance but not yet loaded is enough, which is what a server does on join to send a player to a spawn point or a lobby. Worlds created with Config.Synchronous are immune, because loadChunkAsync loads on the calling goroutine there and never registers a request; every test in this repository sets Synchronous, which is why none of them caught this. The bookkeeping is now done under the lock and the lock is released before the Viewer is called. A re-entrant call for the same position returns at the loaded check rather than blocking, and one for a different position takes an uncontended lock, so recursion terminates after at most one entry per chunk. This requires recording the chunk as loaded before the Viewer is called rather than after it. addViewer and removeViewer now take the Viewer explicitly instead of reading it off the Loader. That read was previously protected by the caller holding the Loader's lock. Releasing the lock also means the Loader may be closed or moved to another World while the Viewer is being called, which viewChunk now detects, undoing the addition so the Viewer is not leaked into the old world.
schphe
force-pushed
the
fix/loader-viewchunk-deadlock
branch
from
August 20, 2026 07:51
463e167 to
e1781c1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What happens
Loader.viewChunkcan re-enter itself on one goroutine and block on a lock that goroutine already holds. That goroutine is the world's owner, so the world stops permanently while the process stays up: nobody on it receives anything further, inbound packets are not handled, and goroutines elsewhere keep running and logging, which makes it read as a delivery bug rather than a stall.On how it is triggered, please read the note at the end before this lands. The cycle is demonstrated by the test in this change, which hangs without it. I have not managed to show an ordinary player triggering it.
Why
Loader.viewChunkheld the Loader's mutex across both of its call-outs — the Viewer'sViewChunkandWorld.addViewer.Showing an entity to a Viewer runs arbitrary code. A session encoding a player's metadata calls
Player.Breathing, which reads the liquid at the player's eye, and that position need not be in the chunk being viewed. Reading a block from a chunk that has a background load in flight takes thechunkFromAsyncPoolpath, wherechunkRequest.doImmediateruns the request's callbacks inline on the calling goroutine. The callback a Loader registers inLoadisviewChunk, so it re-entered itself and blocked forever on a lock it already held.Worth knowing: every existing test in this repository sets
Config.Synchronous, and a synchronous world never registers a chunk request, so it is immune. That is why nothing caught this.What changed
The bookkeeping is done under the lock and the lock is released before the Viewer is called. A re-entrant call for the same position returns at the loaded check; one for a different position takes an uncontended lock, so recursion terminates after at most one entry per chunk. This requires recording the chunk as loaded before the Viewer is called rather than after.
addViewerandremoveViewernow take the Viewer explicitly instead of reading it off the Loader — that read was previously protected by the caller holding the lock. Releasing the lock also means the Loader may be closed or moved to another World mid-call, whichviewChunknow detects, undoing the addition rather than leaking the Viewer into the old world.Verification
Regression test drives the cycle deterministically through both call-outs. Without the fix both subtests hang and fail with
viewChunk did not return: the Loader deadlocked against itself; with it they pass in 0.02s. The test needs a non-synchronous world for exactly the reason above.What has and has not been shown
The deadlock itself is not in question: the test added here drives the cycle deterministically and hangs on
master, and the stack below is from that run.What I could not show is a player reaching it. I originally believed a teleport on join was enough — a player sent to a spawn point or a lobby, landing in a chunk inside their render distance that is not yet loaded. A harness driving a real client against a real server did not reproduce it that way: the background chunk loading kept up, and the only world freeze it produced came from
Loader.ChangeWorldblocking on a full transaction queue, which is a different defect with its own fix.So the honest position is that this is a real re-entrancy defect in
viewChunkwith a demonstrated mechanism and no demonstrated player-facing trigger. It may be that the trigger needs a slower generator, a fuller queue or more players than the harness had; it may be that the ordering makes it much rarer than I first thought. I would rather say that plainly than assert a frequency I cannot back.Worth knowing either way: every existing test in this repository sets
Config.Synchronous, and a synchronous world never registers a chunk request, so none of them can reach this path.