Repository navigation
fix: close the kernel of a tab whose close beacon beats its websocket - #1247
Draft
maartenbreddels wants to merge 9 commits into
Draft
maartenbreddels wants to merge 9 commits into
maartenbreddels wants to merge 9 commits into
Conversation
A close beacon can name a page that the kernel never registered. This happens when an app runs several server processes and the beacon reaches a process with a stale copy of the kernel, or when a tab closes before its websocket connects. page_close then raised a KeyError, so the close route returned a 500 and error trackers logged an exception for something harmless. page_close now logs the unknown page and returns, as it already does for a closed kernel or a page that is already closed. The kernel and its known pages stay as they are. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Review found that the test only had a connected page, where page_close never bumps the cull, so it could not catch a bump. The test now uses a disconnected page with a scheduled cull. The log line redacts the ids like the other persistence-era log lines do. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
With the 0.2 s cull timeout, a stalled CI runner could let the cull close the kernel before the assertions ran. The test now uses the default cull timeout and closes the kernel through page_close, so no timing is involved. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The server creates a kernel first and connects the page after. When the tab closes in between, the close beacon arrives first. Ignoring that beacon let the late websocket mark the closed tab as connected, and the kernel then lived until the cull timeout. page_close now marks an unknown page as closed, so page_connect refuses it. A kernel with no live pages closes, as for a known page. A page that never connected never cancelled the cull, so its close does not bump it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Closing the kernel inside page_close (previous commit) could hit a kernel that was still initializing: a beacon during the persistence takeover closed it, and the takeover then attached a flush worker to the closed kernel, which leaked. The refused late connect also raised in the websocket handler, so every tab closed during load logged an error. page_close now only marks an unknown page closed and leaves the kernel and its cull alone. page_connect refuses that page with PageClosedError. The websocket handler catches it, which happens after initialization, and closes the kernel if no page is live. The reason is not page-close, because no page used the kernel, so its persisted state is kept. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
When two pages share a kernel and one closes before it connects, the kernel stays alive for the other page. The refused websocket stayed in the kernel's websocket set, so the next broadcast tried to send to a closed socket and logged errors. The handler now removes it before it returns. The docstring of close_if_no_live_pages now gives the real reason the persisted state is kept. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…over A new context is in `contexts` while its state takeover waits on the backend. A reconnect of the same tab can reuse it, and after the tab's close beacon that reconnect is refused and closes the kernel. The first handler then attached persistence and started a flush worker on the closed kernel, which nothing stopped, and crashed while wiring streams to the closed kernel. The takeover now attaches under the teardown lock and skips the attach once close() has begun, and initialization skips the stream wiring for a closed kernel. The refused-websocket cleanup reads the kernel session only once, because a concurrent close can clear it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ized When a kernel closes during its state takeover, initialization returns the closed kernel, and page_connect then raised an unhandled error. That logged a traceback for an expected close. In a narrow case the websocket also waited for a client message before it closed, which delayed the reconnect by a few seconds. app_loop now returns at once, so the websocket closes and the client reconnects to a new kernel. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A membership check followed by a second lookup looks like a race to a reader. It is safe under the lock, but one read makes that obvious. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This branch had an error being deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A close beacon for a page that the kernel does not know no longer fails, and a tab that closes before its websocket connects no longer keeps its kernel alive until the cull.
Problem
When a browser tab closes, Solara sends a close beacon to
/_solara/api/close/<kernel id>?session_id=<page id>.KernelContext.page_closelooked up the page withself.page_status[page_id]. When the kernel had not registered that page, the lookup raised aKeyError, and the route returned a 500. Error trackers such as Sentry then logged an exception for each such beacon.A kernel can miss the page in two ways:
connectKernelreturns before the websocket opens, so the beacon can arrive after the server created the kernel but before it calledpage_connect. Before this change, the latepage_connectthen marked the closed tab as connected. The kernel lived until the cull, which is 24 hours by default.Change
page_closemarks an unknown page as closed and returns. It does not close the kernel or bump the cull, because the kernel may still be initializing.page_connectrefuses a closed page with the newPageClosedError, a subclass ofRuntimeError.app_loopcatchesPageClosedErroronce the kernel has initialized. It removes the refused websocket from the kernel. It then calls the newclose_if_no_live_pages, which closes the kernel only when no page is connected or disconnected. The close reason isclosed-before-connect, notpage-close, so the persisted state stays until its TTL._restore_on_connectnow attaches persistence and starts the flush worker under the context's teardown lock, and skips both onceclose()has begun. Initialization then skips the stream wiring, andapp_loopdrops the websocket, so the client reconnects to a new kernel.Validation
I ran
tests/unit/lifecycle_test.pyandtests/unit/state_server_test.pyafter each commit. The last run gave 47 passed. Each new test failed on the commit before its fix:test_kernel_lifecycle_close_beacon_for_unknown_page(two cases): failed onmasterwith the productionKeyError. It checks that the page is marked closed, the cull stays the same, the kernel stays open, and a latepage_connectis refused.test_app_loop_close_beacon_before_connect: failed with an unhandledPageClosedErrorbeforeapp_loopcaught it. It checks that the kernel closes with reasonclosed-before-connect.test_app_loop_close_beacon_before_connect_with_live_page: failed because the refused websocket stayed on a kernel that another page uses.test_close_during_takeover_does_not_attach: failed with anAttributeErrorin_wire_kernel_streams. It checks that no persistence manager or flush worker attaches to a kernel that closed during its takeover.Gaps
initialize_virtual_kernelcreates the kernel still finds nothing. That kernel then lives until the cull, as onmaster.page_statuskeeps one closed entry per unknown page id until the kernel closes. Without persistence the close route needs no cookie, so a client that knows a kernel id can add entries.?kernelid=URL gives.masterhas the same windows for known pages:close_if_no_live_pageschecks for live pages, releases the lock, and then closes. A page that connects in between loses its kernel and reconnects to a new one.close()waits for the attach under the teardown lock and then stops the new worker. Reviewers checked that order by reading the code.Align results
Caution
/alignwas not run on this change: the maintainer chose the design directly when the review found the race.Crossreview results
The crossreview ran 7 rounds with astra, opus, and glm. All three approved commit 72d49f5. No CRITICAL or HIGH finding stands after the last round.
page_connectlet the late connect revive the closed tab. The defect got in because my brief marked "an unknown page changes no kernel state" as settled, although I had made that decision alone. So the reviewers did not question it. Opus saw the window in round 1, but called it "not a regression", and I accepted that.page_close. All three reviewers found that a beacon during the state takeover closed a kernel that was still initializing, and a flush worker leaked. The defect got in because I did not check that a new kernel is visible to other code before it finishes initializing.app_loopnow drops a websocket whose kernel closed while it initialized, so no traceback is logged. All three reviewers raised this as LOW. I confirmed the fix as the driver, because it does only what the finding asked and adds no name.page_closenow reads the page status once with.get(), instead of a membership check and a second lookup. The maintainer flagged that pattern. It was safe under the lock, but the single read makes that obvious. I confirmed the fix as the driver, because it changes no behavior and adds no name.Constitution change that could have prevented the first defect: a crossreview brief may mark as settled only the decisions that the user made. It must give decisions that the driver made alone as claims for the reviewers to attack.
🤖 Generated with Claude Code