Repository navigation
FIX: incremental run listing off the event loop, model uploads refused, runs in flight capped per workstation - #98
Merged
Jules-GP merged 2 commits intoOct 30, 2026
Conversation
…l weights and cap each workstation's runs in flight
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.
…for an HTML filter
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A security and performance audit of the stack, in three rounds (findings, then the fix diff twice). This is what it found in the server. Numbers are measured on a local instance with 500 run directories, about 335 events per run.
Performance
runs.active()re-read and re-verified every event of every run at each call;/status,/admin-panel.json(twice),updates.jsonand the update agent call it, the pages every 2 s/statustimed out, which the update agent'swait_idlereads as "server down" and restarts it with runs in flight/status10 ms, no timeoutrmtree/remove) on the event loop/health200 ms during a 2 GB discarddisk()walked TEMP_DIR and DATA_DIR at every panel pollstatper pollruns.finish()parsed the whole events file on the loopnvidia-smiper/statusand per panel pollThe listing is now incremental: one reader per run, kept between listings and keyed on inode, size and mtime, reads only what a run appended, still verifies the signature of every new line, and stops at the terminal event as a whole read does (a 12-seed random-append test checks it gives the same answer as a whole read). Also: the event stream sends a
: pingevery 15 s of silence (the client's read timeout is 30 s, so a quiet stretch was a reconnect and a full replay); runs a previous server process left pending or running are failed at startup with "The server restarted during this run." (paused runs untouched); container logs are bounded (json-file, 5 x 20 MB).Security
deployment.tomlopts 14 model arguments out of the name-only convention with"none", which also made them uploadable files; AREG IOS then unpickled such an upload (fixed in the tool, SADT-VISOR#99). The server now refuses an upload tomodel/*_modelin both staging paths, and hides those arguments from the published schema so a panel offers no upload row.*_referencearguments stay uploadable: none feeds a pickle-based loader.MAX_RUNS_PER_CLIENT(default 10, 0 = none), 429 before anything is staged. Pending and waiting-for-the-card runs count, paused ones do not; a run sent withoutX-Run-Idholds a slot while it runs. The count, the registration and the slot claim are one locked step in a worker thread, so two requests cannot both pass, and a cancelled request keeps no slot. A benchmark battery an operator launched is exempt through a per-battery random key that never reaches a tool's environment. The cap reads the client address uvicorn derives, so behind a proxy it depends onFORWARDED_ALLOW_IPSand the proxy overwritingX-Forwarded-For.ToolExecutionErrorwas raised inmain.pywithout being imported: a detached run whose result could not be stored failed with aNameError.Tests
New tests count (calls, stats, fetches, bytes), not time; each failed before its fix. A test that a cancelled
/runkeeps no slot fails when_off_loop's shield is removed. Server suite: 1554 passed; the knowntest_dev_mountsfailure is unchanged, and the update-agent suite passes 57/57 when rerun. bandit and zizmor clean.Deferred: AREG CBCT copies the oriented scans out of supervisor scratch instead of moving them (unmeasured).