perf(scan): walk the scan root once, not once per consumer - #39
Open
SYM01 wants to merge 3 commits into
Open
Conversation
A directory scan walked the identical pruned tree 3-9 times: language detection, each present frontend's CollectTarget, the raw-file secrets pass, and the Java source index each did their own O(files) traversal. walkignore.Inventory now caches ONE pruned walk (paths, sizes, first walk error) and every consumer selects from it: Select() keeps the abort-on-error -strict/coverage contract CollectSources had, Files() keeps the caller's path spelling for secret positions, AbsFiles() feeds the Java index. Single-file targets bypass the inventory unchanged. Also dedup the per-language extension predicates: IsCppFile (untagged, shared by the llvm-tagged converter and scan), IsPythonFile, IsRubyFile, IsRustFile now live in their converters, matching IsJSFamily. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NPC3XNovQxoWoXKB1f7h1e
🛡️ Quality Gate —
|
| Area | + | − |
|---|---|---|
cmd/ |
1 | 6 |
converters/ |
338 | 185 |
internal/ |
658 | 491 |
Counts
cmd converters internal pkg proto rulepacks; excludes*_test.go,testdata/,test/, generated*.pb.go.
2 · Corpus signal/noise (TP / FP / FN)
| Metric | Base | Head | Δ |
|---|---|---|---|
| TP | 218 | 218 | 0 |
| FP | 0 | 0 | 0 |
| FN | 0 | 0 | 0 |
| Precision | 1.000 | 1.000 | +0.000 |
| Recall | 1.000 | 1.000 | +0.000 |
| F1 | 1.000 | 1.000 | +0.000 |
Scored over N=303 samples on both revisions.
3 · Rule changes
No rule additions, removals, or modifications.
4 · Performance · gated (benchstat, count=10)
Engine hot paths and per-language full-pipeline scans, all compared by
benchstat so the base→head difference is statistically reliable rather than
wall-clock noise. A language whose toolchain is absent is skipped.
goos: linux
goarch: amd64
pkg: godzilla/internal/analysis
cpu: AMD EPYC 7763 64-Core Processor
│ /tmp/godzilla-qgate.FpD5Me/bench-base.txt │ /tmp/godzilla-qgate.FpD5Me/bench-head.txt │
│ sec/op │ sec/op vs base │
Engine_RuleScaling/rules=1-4 1.831m ± 1% 1.683m ± 1% -8.06% (p=0.000 n=10)
Engine_RuleScaling/rules=10-4 12.42m ± 3% 11.65m ± 2% -6.17% (p=0.000 n=10)
Engine_RuleScaling/rules=50-4 40.24m ± 2% 37.82m ± 5% -6.02% (p=0.000 n=10)
Engine_RuleScaling/rules=200-4 136.6m ± 6% 122.0m ± 4% -10.68% (p=0.000 n=10)
Engine_InertRules/inert=0-4 63.87µ ± 1% 65.08µ ± 1% +1.89% (p=0.000 n=10)
Engine_InertRules/inert=14-4 64.44µ ± 0% 82.05µ ± 2% +27.33% (p=0.000 n=10)
Engine_InertRules/inert=100-4 67.61µ ± 0% 136.30µ ± 1% +101.62% (p=0.000 n=10)
geomean 1.660m 1.819m +9.55%
│ /tmp/godzilla-qgate.FpD5Me/bench-base.txt │ /tmp/godzilla-qgate.FpD5Me/bench-head.txt │
│ B/op │ B/op vs base │
Engine_RuleScaling/rules=1-4 1.394Mi ± 0% 1.393Mi ± 0% -0.07% (p=0.000 n=10)
Engine_RuleScaling/rules=10-4 11.23Mi ± 0% 11.10Mi ± 0% -1.16% (p=0.000 n=10)
Engine_RuleScaling/rules=50-4 54.98Mi ± 0% 54.28Mi ± 0% -1.28% (p=0.000 n=10)
Engine_RuleScaling/rules=200-4 219.0Mi ± 0% 216.2Mi ± 0% -1.30% (p=0.000 n=10)
Engine_InertRules/inert=0-4 38.05Ki ± 0% 41.21Ki ± 0% +8.32% (p=0.000 n=10)
Engine_InertRules/inert=14-4 38.39Ki ± 0% 42.98Ki ± 0% +11.95% (p=0.000 n=10)
Engine_InertRules/inert=100-4 40.67Ki ± 0% 53.99Ki ± 0% +32.75% (p=0.000 n=10)
geomean 1.398Mi 1.488Mi +6.45%
│ /tmp/godzilla-qgate.FpD5Me/bench-base.txt │ /tmp/godzilla-qgate.FpD5Me/bench-head.txt │
│ allocs/op │ allocs/op vs base │
Engine_RuleScaling/rules=1-4 3.638k ± 0% 3.650k ± 0% +0.33% (p=0.000 n=10)
Engine_RuleScaling/rules=10-4 22.32k ± 0% 22.41k ± 0% +0.38% (p=0.000 n=10)
Engine_RuleScaling/rules=50-4 105.3k ± 0% 105.7k ± 0% +0.39% (p=0.000 n=10)
Engine_RuleScaling/rules=200-4 416.3k ± 0% 417.9k ± 0% +0.38% (p=0.000 n=10)
Engine_InertRules/inert=0-4 53.00 ± 0% 57.00 ± 0% +7.55% (p=0.000 n=10)
Engine_InertRules/inert=14-4 54.00 ± 0% 86.00 ± 0% +59.26% (p=0.000 n=10)
Engine_InertRules/inert=100-4 54.00 ± 0% 258.00 ± 0% +377.78% (p=0.000 n=10)
geomean 2.463k 3.333k +35.31%
pkg: godzilla/internal/rules
│ /tmp/godzilla-qgate.FpD5Me/bench-base.txt │ /tmp/godzilla-qgate.FpD5Me/bench-head.txt │
│ sec/op │ sec/op vs base │
MatchGlob-4 496.1n ± 0% 3712.5n ± 0% +648.26% (p=0.000 n=10)
│ /tmp/godzilla-qgate.FpD5Me/bench-base.txt │ /tmp/godzilla-qgate.FpD5Me/bench-head.txt │
│ B/op │ B/op vs base │
MatchGlob-4 0.000Ki ± 0% 3.438Ki ± 0% ? (p=0.000 n=10)
│ /tmp/godzilla-qgate.FpD5Me/bench-base.txt │ /tmp/godzilla-qgate.FpD5Me/bench-head.txt │
│ allocs/op │ allocs/op vs base │
MatchGlob-4 0.00 ± 0% 84.00 ± 0% ? (p=0.000 n=10)
pkg: godzilla/internal/scan
│ /tmp/godzilla-qgate.FpD5Me/bench-base.txt │ /tmp/godzilla-qgate.FpD5Me/bench-head.txt │
│ sec/op │ sec/op vs base │
Scan_Python-4 31.06m ± 3% 32.21m ± 2% +3.67% (p=0.003 n=10)
Scan_JS-4 652.5µ ± 2% 680.3µ ± 5% +4.26% (p=0.001 n=10)
Scan_Rust-4 50.83m ± 1% 52.94m ± 1% +4.14% (p=0.000 n=10)
Scan_Java-4 532.4m ± 4% 531.8m ± 3% ~ (p=0.481 n=10)
Scan_Ruby-4 76.57m ± 2% 77.88m ± 1% +1.71% (p=0.002 n=10)
Scan_GoWithDeps-4 2.705 ± 4% 2.823 ± 3% ~ (p=0.011 n=10)
Scan_GoSimple-4 136.6m ± 1% 139.4m ± 1% +2.04% (p=0.000 n=10)
geomean 76.63m 78.82m +2.85%
│ /tmp/godzilla-qgate.FpD5Me/bench-base.txt │ /tmp/godzilla-qgate.FpD5Me/bench-head.txt │
│ B/op │ B/op vs base │
Scan_Python-4 509.6Ki ± 3% 524.7Ki ± 2% ~ (p=0.029 n=10)
Scan_JS-4 372.9Ki ± 1% 384.2Ki ± 1% +3.03% (p=0.000 n=10)
Scan_Rust-4 653.6Ki ± 0% 679.9Ki ± 1% +4.03% (p=0.000 n=10)
Scan_Java-4 1.651Mi ± 0% 1.653Mi ± 0% ~ (p=0.101 n=10)
Scan_Ruby-4 579.8Ki ± 3% 586.7Ki ± 3% ~ (p=0.075 n=10)
Scan_GoWithDeps-4 1.442Gi ± 0% 1.441Gi ± 0% -0.08% (p=0.002 n=10)
Scan_GoSimple-4 8.529Mi ± 0% 8.538Mi ± 0% ~ (p=0.063 n=10)
geomean 2.804Mi 2.849Mi +1.61%
│ /tmp/godzilla-qgate.FpD5Me/bench-base.txt │ /tmp/godzilla-qgate.FpD5Me/bench-head.txt │
│ allocs/op │ allocs/op vs base │
Scan_Python-4 3.179k ± 3% 3.353k ± 3% +5.47% (p=0.000 n=10)
Scan_JS-4 1.869k ± 0% 2.024k ± 0% +8.32% (p=0.000 n=10)
Scan_Rust-4 3.879k ± 2% 4.098k ± 1% +5.63% (p=0.000 n=10)
Scan_Java-4 15.33k ± 0% 15.43k ± 0% +0.65% (p=0.000 n=10)
Scan_Ruby-4 3.901k ± 4% 4.047k ± 4% +3.76% (p=0.001 n=10)
Scan_GoWithDeps-4 18.86M ± 0% 18.86M ± 0% -0.04% (p=0.000 n=10)
Scan_GoSimple-4 63.23k ± 0% 63.42k ± 0% +0.31% (p=0.000 n=10)
geomean 20.73k 21.43k +3.40%
Gate blocks on a regression that is significant at alpha=0.01 (benchstat marks anything weaker as
~) on: Engine_RuleScaling,Engine_InertRules,MatchGlob,Scan_GoWithDeps,Scan_GoSimple,Scan_Python,Scan_JS,Scan_Rust,Scan_Java,Scan_Ruby — timesec/op> 10%, memoryB/op/allocs/op> 10%. The strict alpha keeps subprocess/GC run-to-run noise on the heavier scans from tripping the gate.
Both revisions were built and benchmarked back-to-back on this runner; numbers are only comparable within a single run.
…es/report/CLI applied; engine and converter agents in flight) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NPC3XNovQxoWoXKB1f7h1e
…code removal Engine: memoize per-(rule,callee) classification in a goroutine-local ruleState (handleCall no longer re-runs ~150 glob matches per call site per fixpoint pass); hoist rpo/preds/entry into the fnIndex memo; scan for validator guards before computing dominators; early-return isTaintedArg on an empty state; run the sink-callee prefilter inside each rule's goroutine; collapse funcAnalysis's copied fields into idx *sharedIndex + rs *ruleState; unexport BuildCallGraph. Converters: embeddable frontend.Driver replaces the five identical ConvertFile/ConvertInventory/Skipped shells; internal/chunks folded into converters/frontend; rustCommandStep now matches exact std-anchored callees (a user MyCommand::arg no longer aliases away the return's taint — regression-tested); rust smoke and llm tools use irwalk; dead ssabuild.SetComment removed. Scan/walkignore: ScanFiles reuses the single-walk inventory; Inventory caches its joined path slices; one shared walk closure; CollectSources inlined; secrets scaffold delegates to ScanSecretsInPaths and the drifted sourceFileExts list is gone (frontend table is the sole authority); IsGoFile/IsJavaFile complete the predicate dedup. Rules/report/CLI: dead MatchAny/compileGlob/globCache subsystem deleted; one CompareFindings comparator ends the console-vs-report ordering drift (console adopts numeric, nil-last); severity/confidence display order derives from the canonical Rank lists; walk.go alias shim deleted in favor of direct irwalk use; CLAUDE.md updated for the moved scaffolding. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NPC3XNovQxoWoXKB1f7h1e
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements D1 from the codebase simplification review — the one finding deferred out of #38 because it changes the frontend↔scan API.
What
A directory scan walked the identical pruned tree 3–9 times: language detection, each present frontend's
CollectTarget, the raw-file secrets pass, and the Java source index each did their own O(files) stat/readdir traversal. Now it walks once.walkignore.Inventory(new): caches one pruned walk — walk-ordered relative paths, per-file sizes, and the first walk error.Select(match)gives frontends their filtered, sorted, absolute view and preserves the abort-on-error-strict/coverage contractCollectSourceshad;Files()keeps the caller's path spelling (so secret-finding positions don't shift);AbsFiles()feeds the Java source index.CollectSourcesis now implemented asNewInventory(root).Select(match)— one filter implementation, no drift.internal/scanbuilds the Inventory once per directory scan and threads it through: language detection reads it, each converter gains aConvertInventoryentry point (plumbed through the sharedconverters/frontendbatch driver), and the secrets pass consumes the list via a newScanSecretsInPaths—internal/analysisgains no filesystem walking.ScanFilesbypass the inventory unchanged.Also dedups the per-language extension predicates left over from #38:
cpp.IsCppFile(in an untagged file so both the llvm-tagged converter and the default-build scan table share it), plus the same unification forIsPythonFile,IsRubyFile,IsRustFile— matching the existingIsJSFamilypattern.Behavior
None changed by design: same files selected per frontend, same
TooBigcaps and exclusions, same coverage/Skippedsemantics, same secrets-file selection, zero corpus expectation changes.Validation
go test ./...fully green with all seven frontends live (JDK 25, rustc, ruby, python3)make build-llvm && make test-llvmgreen (cgo backend against LLVM 18)go vetclean (default and llvm tags),-raceon prior engine work unaffectedTestInventorypinsSelect≡ oldCollectSources, the unfilteredFiles()view,SkipDirpruning, and the missing-root error split🤖 Generated with Claude Code
https://claude.ai/code/session_01NPC3XNovQxoWoXKB1f7h1e
Generated by Claude Code