Skip to content

perf: parse source files in parallel; make the parse timeout configurable - #166

Closed
medagent26 wants to merge 1 commit into
FSoft-AI4Code:mainfrom
medagent26:perf/parallel-parse
Closed

medagent26 wants to merge 1 commit into
FSoft-AI4Code:mainfrom
medagent26:perf/parallel-parse

Conversation

@medagent26

Copy link
Copy Markdown

Problem

Dependency analysis parses files one at a time on one core, with a hard-coded 30 s per-file timeout. A file that times out is dropped from the graph with only a warning, so its components never reach the generated docs.

On a 678-file TypeScript monorepo, the five central files (6-8k lines each: the daemon client, the agent manager and three provider integrations) timed out on every run. The largest one alone takes 91 s in analyze_typescript_file_treesitter. These were the files the docs most needed.

Change

  • Parallel parsing. CallGraphAnalyzer parses files in a spawn process pool, largest first. It then merges each worker's per-file state in the original file order, so the result is identical to the sequential sweep. Per-file analysis only adds to the accumulators in _per_file_state and never reads another file's results, which is what makes the merge safe. spawn is used because callers may hold threads, and each worker runs tasks on its main thread, where SIGALRM works.
  • CODEWIKI_PARSE_WORKERS sets the pool size. The default is the smallest of: usable CPUs (cgroup v1/v2 CPU quota respected), files / 8, and available memory / 400 MB (cgroup memory limit respected). 1 keeps the sequential code path unchanged. Under 16 files the default is sequential.
  • CODEWIKI_PARSE_TIMEOUT sets the per-file timeout. The default goes from 30 s to 300 s. A dropped file costs documentation, while a slow file only costs time. A file that never finishes, such as a minified bundle, now takes up to 300 s before it is dropped. Set the variable to get the old behaviour.
  • Both values are validated before parsing starts. An invalid value fails the run instead of being swallowed per file.
  • Timed-out files now count as failed in the summary. Before, they were reported as analyzed.
  • Logging: each worker buffers the log records of the current file in memory and returns them with the file's result. The parent replays them through its own loggers, so timeout warnings still show and the parent's levels apply. There is no pipe, lock or thread: a SIGALRM timeout or a killed worker cannot leave a half-written record behind or block the parent. Records show up when their file finishes, not live.
  • Ctrl-C stops at once. Workers ignore SIGINT: a worker interrupted inside the call queue's Queue.get dies holding its lock, and the pool then deadlocks, which I reproduced. The parent cancels queued files and terminates the workers. pool._processes is private, but it is present in CPython 3.12-3.14.
  • A worker that dies or fails to start (OOM kill, or a caller script without a __main__ guard) aborts the analysis with a message naming those causes and CODEWIKI_PARSE_WORKERS=1. It is not reported as failed files, and teardown does not hang. The parent terminates the workers and lets shutdown(wait=True) reap them (joining them itself raced the pool's manager thread). 12 interrupted runs in one process left no worker processes behind.
  • Docs (Dependency_Analysis_Service.md) and the CHANGELOG are updated.

Measurements

678 files, 32 CPUs, Python 3.12:

wall CPU functions relationships timeouts
sequential, timeout raised to 900 s 606.6 s 607 s 11,623 32,918 0
this PR, defaults 102-109 s (several runs) 606-632 s 11,623 32,918 0
this PR, CODEWIKI_PARSE_TIMEOUT=30 31.5 s 396 s 9,812 27,056 the same 5 as main
  • functions, relationships and visualization output is byte-identical to the sequential run, including order.
  • Peak RSS: parent 191 MB, largest worker 133 MB.
  • Wall time is now bounded by the slowest single file.
  • Ctrl-C during parsing exits in about 1 s with no leftover processes. Before the SIGINT fix it hung indefinitely.
  • kill -9 of one worker exits in about 1 s with the error message above.

Tests

tests/test_parallel_parse.py (13 tests):

  • Parallel output equals sequential output on a Python, Go, Dart, Lean and TypeScript fixture, including the Dart, Go and Lean resolution state.
  • An interrupt (Ctrl-C) stops the workers without waiting for the file in flight and leaves no child processes.
  • A worker timeout drops only that file, its warning reaches the parent's logging, and it counts as failed.
  • An invalid timeout fails before any parsing, on both paths.
  • Worker count from env, cgroup CPU quota and memory (fake v1/v2 cgroup trees, including a limit at an ancestor and a malformed /proc/self/cgroup line).
  • Worker log buffering: extra attributes are stripped so an unpicklable one cannot sink a file's result, and records are capped per file.
  • Accumulator guards. A static AST check covers direct writes in every _analyze_*_file method. A runtime check covers any state an analysis changes, including helper writes, for the fixture languages.

The tests were mutation-checked: each of these makes a test fail.

  • Dropping a merged accumulator.
  • Reversing the merge order.
  • Adding an unlisted accumulator, directly or in a helper.
  • Reading the timeout back inside the per-file try.

Full suite: 886 passed, 9 failed. The same 9 fail on main (8 in test_zig_analyzer.py with KeyError: 'Point', plus test_doc_write_guard.py::test_warning_reports_rejected_tool_calls), unrelated to this change. ruff check and ruff format --check are clean.

Not covered by unit tests: worker death (OOM kill) teardown, verified by hand as described above.

Trade-offs

  • Worker log records arrive when their file finishes, not live. A worker that is killed loses that file's records, but the run then fails with the error above.
  • A timed-out or failed file's partial state is merged, the same as the sequential path on main.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VARC7X5n9RnBw6wLGrf4TW

…able

Dependency analysis parsed files one at a time on a single core with a
hard-coded 30 s per-file timeout. A file that times out is dropped from
the graph with only a warning, so its components never reach the docs.
On a ~680-file TypeScript monorepo the five central 6-8k line files
timed out on every run (the largest alone takes ~91 s).

- Parse files in a spawn-based process pool, largest first, and merge
  per-file state in the original file order. The output is identical to
  the sequential sweep, including order.
- CODEWIKI_PARSE_TIMEOUT sets the per-file timeout (default 300 s).
- CODEWIKI_PARSE_WORKERS sets the pool size. Default: one worker per
  usable CPU (cgroup CPU quota respected), per 8 files and per 400 MB of
  available memory (cgroup memory limit respected); 1 keeps the
  sequential path. Both values are validated before parsing starts.
- Timed-out files count as failed in the analysis summary.
- Each worker buffers its log records per file and returns them with
  the file's result; the parent replays them through its own loggers.
  No pipe, lock or thread, so a timeout or a killed worker cannot leave
  a half-written record or block the parent.
- Ctrl-C stops at once: workers ignore SIGINT (an interrupted worker can
  die holding the call queue's lock and deadlock the pool), the parent
  cancels queued files and terminates the workers.
- A worker that dies or fails to start (OOM kill, caller script without
  a __main__ guard) aborts with a hint instead of being reported as
  failed files.

Measured on that monorepo (678 files, 32 CPUs):
  sequential, timeout raised to 900 s: 606.6 s wall, 11,623 functions, 32,918 relationships
  parallel, defaults:                  108.6 s wall, 11,623 functions, 32,918 relationships
Functions, relationships and visualization data are byte-identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VARC7X5n9RnBw6wLGrf4TW
@medagent26 medagent26 closed this Nov 16, 2026
@medagent26
medagent26 deleted the perf/parallel-parse branch November 16, 2026 20:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant