Repository navigation
perf: parse source files in parallel; make the parse timeout configurable - #166
Closed
medagent26 wants to merge 1 commit into
Closed
medagent26 wants to merge 1 commit into
medagent26 wants to merge 1 commit into
Conversation
…able Dependency analysis parsed files one at a time on a single core with a hard-coded 30 s per-file timeout. A file that times out is dropped from the graph with only a warning, so its components never reach the docs. On a ~680-file TypeScript monorepo the five central 6-8k line files timed out on every run (the largest alone takes ~91 s). - Parse files in a spawn-based process pool, largest first, and merge per-file state in the original file order. The output is identical to the sequential sweep, including order. - CODEWIKI_PARSE_TIMEOUT sets the per-file timeout (default 300 s). - CODEWIKI_PARSE_WORKERS sets the pool size. Default: one worker per usable CPU (cgroup CPU quota respected), per 8 files and per 400 MB of available memory (cgroup memory limit respected); 1 keeps the sequential path. Both values are validated before parsing starts. - Timed-out files count as failed in the analysis summary. - Each worker buffers its log records per file and returns them with the file's result; the parent replays them through its own loggers. No pipe, lock or thread, so a timeout or a killed worker cannot leave a half-written record or block the parent. - Ctrl-C stops at once: workers ignore SIGINT (an interrupted worker can die holding the call queue's lock and deadlock the pool), the parent cancels queued files and terminates the workers. - A worker that dies or fails to start (OOM kill, caller script without a __main__ guard) aborts with a hint instead of being reported as failed files. Measured on that monorepo (678 files, 32 CPUs): sequential, timeout raised to 900 s: 606.6 s wall, 11,623 functions, 32,918 relationships parallel, defaults: 108.6 s wall, 11,623 functions, 32,918 relationships Functions, relationships and visualization data are byte-identical. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VARC7X5n9RnBw6wLGrf4TW
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Dependency analysis parses files one at a time on one core, with a hard-coded 30 s per-file timeout. A file that times out is dropped from the graph with only a warning, so its components never reach the generated docs.
On a 678-file TypeScript monorepo, the five central files (6-8k lines each: the daemon client, the agent manager and three provider integrations) timed out on every run. The largest one alone takes 91 s in
analyze_typescript_file_treesitter. These were the files the docs most needed.Change
CallGraphAnalyzerparses files in aspawnprocess pool, largest first. It then merges each worker's per-file state in the original file order, so the result is identical to the sequential sweep. Per-file analysis only adds to the accumulators in_per_file_stateand never reads another file's results, which is what makes the merge safe.spawnis used because callers may hold threads, and each worker runs tasks on its main thread, where SIGALRM works.CODEWIKI_PARSE_WORKERSsets the pool size. The default is the smallest of: usable CPUs (cgroup v1/v2 CPU quota respected), files / 8, and available memory / 400 MB (cgroup memory limit respected).1keeps the sequential code path unchanged. Under 16 files the default is sequential.CODEWIKI_PARSE_TIMEOUTsets the per-file timeout. The default goes from 30 s to 300 s. A dropped file costs documentation, while a slow file only costs time. A file that never finishes, such as a minified bundle, now takes up to 300 s before it is dropped. Set the variable to get the old behaviour.Queue.getdies holding its lock, and the pool then deadlocks, which I reproduced. The parent cancels queued files and terminates the workers.pool._processesis private, but it is present in CPython 3.12-3.14.__main__guard) aborts the analysis with a message naming those causes andCODEWIKI_PARSE_WORKERS=1. It is not reported as failed files, and teardown does not hang. The parent terminates the workers and letsshutdown(wait=True)reap them (joining them itself raced the pool's manager thread). 12 interrupted runs in one process left no worker processes behind.Dependency_Analysis_Service.md) and the CHANGELOG are updated.Measurements
678 files, 32 CPUs, Python 3.12:
CODEWIKI_PARSE_TIMEOUT=30mainfunctions,relationshipsandvisualizationoutput is byte-identical to the sequential run, including order.kill -9of one worker exits in about 1 s with the error message above.Tests
tests/test_parallel_parse.py(13 tests):/proc/self/cgroupline).extraattributes are stripped so an unpicklable one cannot sink a file's result, and records are capped per file._analyze_*_filemethod. A runtime check covers any state an analysis changes, including helper writes, for the fixture languages.The tests were mutation-checked: each of these makes a test fail.
try.Full suite: 886 passed, 9 failed. The same 9 fail on
main(8 intest_zig_analyzer.pywithKeyError: 'Point', plustest_doc_write_guard.py::test_warning_reports_rejected_tool_calls), unrelated to this change.ruff checkandruff format --checkare clean.Not covered by unit tests: worker death (OOM kill) teardown, verified by hand as described above.
Trade-offs
main.🤖 Generated with Claude Code
https://claude.ai/code/session_01VARC7X5n9RnBw6wLGrf4TW