perf(scan): walk the scan root once, not once per consumer

CLAUDE.md

@@ -91,8 +91,9 @@ changing one:
   and **no gIR/engine change**. Sinks use the mirror trick (the Vue/Svelte directives above).
 - Python, JS and Ruby share their scaffolding rather than re-implementing it: `internal/walkignore`
   (target resolution, pruned walk, and the scan-root-relative module names that stop same-named functions
-  in different files from colliding), `internal/proc.WriteEmbeddedScript`, `internal/chunks.Run`
-  (concurrent per-file lowering), and `converters/ssabuild` (the real-CFG builder with on-demand PHIs).
+  in different files from colliding), `internal/proc.WriteEmbeddedScript`, and `converters/ssabuild`
+  (the real-CFG builder with on-demand PHIs) — plus `converters/frontend`, the shared batch driver
+  every collecting frontend embeds (`frontend.Driver`, concurrent per-file lowering, skip accounting).
 
 **Analysis (`internal/analysis/`).** The engine's design and its precision guards — and the failures that
 motivated them — are documented in the package itself (`internal/analysis/doc.go`, `go doc ./internal/analysis`);
@@ -116,10 +117,11 @@ this is a file map.
   `String.format`, Rust `fmt::Arguments::new`, whose packed byte-template `mir.go` `decodeFmtTemplate` decodes)
   and **`builtin.identity`** (a forwarding string conversion: `to_string`/`as_str`/`clone`/…). Both markers are
   inert to taint propagation. Emitting either from a frontend is what opts that language into the reduction.
-- `callgraph.go` — `BuildCallGraph` (CHA for dynamic dispatch); the engine consumes its reverse edges
-  (`buildCallers`) to re-enqueue a callee's callers when the callee becomes taint-returning.
-- `walk.go` — the `funcs`/`instrs` gIR iterators the once-per-scan passes share instead of each
-  re-writing the nil-guarded module→function→block→instruction nest.
+- `callgraph.go` — `buildCallGraph` (CHA for dynamic dispatch, over the shared function/method index
+  built once in `Analyze`); the engine consumes its reverse edges (`buildCallers`) to re-enqueue a
+  callee's callers when the callee becomes taint-returning. Once-per-scan passes iterate the program
+  via `internal/irwalk` (`Funcs`/`Instrs`) instead of re-writing the nil-guarded
+  module→function→block→instruction nest.
 - `secrets.go` — `ScanSecrets`: applies the `kind: secret` rules' regexps to gIR string constants and to
   config files no frontend parses (CWE-798). The patterns are data in `rulepacks/secrets.yaml`, not Go.
 - `finding.go` — the `Finding` type shared across the pipeline.
@@ -155,7 +157,9 @@ secrets passes, and `scopeFindings` live in **`internal/scan`**, not `main.go`.
 NOT a clean scan, which `-strict` turns into a non-zero exit), `internal/triage` (baseline +
 `godzilla:ignore`), `internal/config` (`.godzilla.yaml`), `internal/buildpolicy` (the `-allow-build` gate on
 running a scanned project's build tool), `internal/ruletest` (backs `rules test`), plus the shared frontend
-scaffolding: `internal/chunks`, `internal/proc`, `internal/walkignore`, `internal/memlimit`.
+scaffolding: `converters/frontend`, `internal/proc`, `internal/walkignore`, `internal/memlimit`, plus the
+cross-cutting helpers `internal/irwalk` (nil-guarded gIR iterators), `internal/srclines` (source-line
+cache shared by report/LLM/triage), and `internal/testsupport` (test-only interpreter checks).
 
 ## Conventions
 

README.md

@@ -101,6 +101,25 @@ $ godzilla scan ./test/go/sql_injection
 1 finding(s); 1 at/above "medium".
 ```
 
+### Environment variables
+
+Everything routine is a CLI flag (`godzilla scan -h`); the environment only
+carries operator concerns:
+
+| Variable | Effect |
+|---|---|
+| `GODZILLA_ALLOW_BUILD=1` | Same opt-in as `-allow-build`: lets a scan run the project's build tool (Maven/Gradle/Cargo). |
+| `GODZILLA_RUSTC`, `GODZILLA_CARGO` | Paths to the Rust toolchain binaries (default: `rustc`, `cargo` on `PATH`). |
+| `GODZILLA_CC`, `GODZILLA_CXX` | C/C++ compilers for the opt-in LLVM backend (default: `clang`, `clang++`). |
+| `GODZILLA_LLM_MODEL` | Override the `-llm-review` model (default: `claude-haiku-4-5`). |
+| `GODZILLA_LLM_PROVIDER=openai`, `GODZILLA_LLM_BASE_URL` | Select an OpenAI-compatible endpoint for `-llm-review` (e.g. a local model). |
+| `ANTHROPIC_API_KEY` / `OPENAI_API_KEY` | Credentials for `-llm-review` (Anthropic also honors an `ant auth` profile). |
+| `GOMEMLIMIT` | Respected as-is: setting it disables Godzilla's automatic soft memory limit. |
+
+Subprocess deadlines are flags, not environment: `-parse-timeout` (default
+`2m0s`, each per-file parse/dump) and `-build-timeout` (default `10m0s`, a
+whole-project build under `-allow-build`).
+
 ## Run with Docker
 
 Prebuilt images ship with the toolchains a scan needs, so you can gate a repo

cmd/godzilla/main.go

@@ -22,6 +22,7 @@ import (
 	"godzilla/internal/config"
 	"godzilla/internal/llm"
 	"godzilla/internal/memlimit"
+	"godzilla/internal/proc"
 	"godzilla/internal/report"
 	"godzilla/internal/rules"
 	"godzilla/internal/rules/loader"
@@ -58,7 +59,7 @@ process with a single merged exit code and report — a changed-files entry poin
 for pre-commit hooks / CI diffs.
 
 flags:
-  -rules <file>     additional YAML rule file to load alongside the built-in rules
+  -rules <path>     additional YAML rule file — or directory of rulepacks — to load alongside the built-in rules
   -fail-on <sev>    minimum severity that fails the gate: info|low|medium|high|critical (default medium)
   -summary          also print a gIR summary (opcode histogram, intrinsics)
   -html <file>      write an HTML report to <file>
@@ -75,6 +76,8 @@ flags:
   -quiet            suppress console output; the exit code and report files still reflect findings
   -files <file>     changed-files mode: read newline-separated paths from <file> ('-' for stdin),
                     e.g. a pre-commit hook: git diff --name-only --cached | godzilla scan -files -
+  -parse-timeout <dur>  deadline per per-file parse/dump subprocess (default 2m0s)
+  -build-timeout <dur>  deadline for a whole-project build under -allow-build (default 10m0s)
 
 A .godzilla.yaml in the scan root can set fail-on, path include/exclude globs,
 and per-rule disable / severity-overrides; CLI flags override its values.
@@ -139,7 +142,7 @@ func readFileList(src string) ([]string, error) {
 func runScan(args []string) {
 	fs := flag.NewFlagSet("scan", flag.ExitOnError)
 	fs.Usage = usage
-	rulesPath := fs.String("rules", "", "additional YAML rule file")
+	rulesPath := fs.String("rules", "", "additional YAML rule file, or a directory of rulepacks")
 	failOn := fs.String("fail-on", "medium", "minimum severity that fails the gate")
 	showSummary := fs.Bool("summary", false, "also print a gIR summary")
 	htmlPath := fs.String("html", "", "write an HTML report to this file")
@@ -153,9 +156,12 @@ func runScan(args []string) {
 	configPath := fs.String("config", "", "path to a .godzilla.yaml (default: auto-loaded from the scan root)")
 	quiet := fs.Bool("quiet", false, "suppress coverage/summary/per-finding console output; the exit code and any report files still reflect findings")
 	filesList := fs.String("files", "", "changed-files mode: read newline-separated paths to scan from this file, or '-' for stdin (for pre-commit hooks / CI diffs)")
+	parseTimeout := fs.Duration("parse-timeout", proc.ParseTimeout(), "deadline for each per-file parse/dump subprocess (python3, JavaDump, rustc, clang)")
+	buildTimeout := fs.Duration("build-timeout", proc.BuildTimeout(), "deadline for a whole-project build subprocess (only runs with -allow-build)")
 	_ = fs.Parse(args)
 
 	buildpolicy.SetAllowed(*allowBuild)
+	proc.SetTimeouts(*parseTimeout, *buildTimeout)
 	report.Version = version // stamp the tool version into SARIF/JSON reports
 
 	// Collect scan targets: a `-files` list (stdin with '-'), one or more
@@ -398,12 +404,7 @@ func writeReportRaw(path string, write func(io.Writer) error) (err error) {
 // computes the gate count but prints nothing — for CI that consumes a report
 // file and only needs the exit code.
 func printFindings(w io.Writer, findings []analysis.Finding, threshold rules.Severity, quiet bool) int {
-	slices.SortStableFunc(findings, func(a, b analysis.Finding) int {
-		if c := cmp.Compare(b.Severity.Rank(), a.Severity.Rank()); c != 0 {
-			return c // worst severity first
-		}
-		return cmp.Compare(analysis.PosString(a.SinkPos), analysis.PosString(b.SinkPos))
-	})
+	slices.SortStableFunc(findings, analysis.CompareFindings)
 
 	// Suppressed findings (judged false positives by the LLM reviewer) are
 	// retained for auditability but do not count toward the gate: partition them

cmd/godzilla/main_test.go

@@ -6,6 +6,8 @@ import (
 	"path/filepath"
 	"strings"
 	"testing"
+
+	"godzilla/internal/testsupport"
 )
 
 // runCLI builds and runs the godzilla CLI (via `go run .`) with args, returning
@@ -32,9 +34,7 @@ func runCLI(t *testing.T, args ...string) (int, string) {
 // (the gate cannot certify code it never analyzed), while the same scan without
 // -strict must not fail closed. Requires python3 (the fixture is broken Python).
 func TestStrict_FailsClosedOnCoverageFailure(t *testing.T) {
-	if _, err := exec.LookPath("python3"); err != nil {
-		t.Skip("python3 not on PATH; skipping strict-mode CLI test")
-	}
+	testsupport.RequireTool(t, "python3")
 	const dir = "../../internal/scan/testdata/broken_py"
 
 	// Without -strict: fail-open (no findings, clean exit), but coverage is shown.

cmd/godzilla/rules.go

@@ -45,7 +45,7 @@ func runRules(args []string) {
 // id, severity, CWE, languages, and source/sink counts.
 func runRulesList(args []string) {
 	fs := flag.NewFlagSet("rules list", flag.ExitOnError)
-	rulesPath := fs.String("rules", "", "additional YAML rule file to include")
+	rulesPath := fs.String("rules", "", "additional YAML rule file or rulepack directory to include")
 	_ = fs.Parse(args)
 
 	rs, err := loader.LoadDefault(*rulesPath)
@@ -101,7 +101,7 @@ func runRulesLint(args []string) {
 // the in-repo corpus test.
 func runRulesTest(args []string) {
 	fs := flag.NewFlagSet("rules test", flag.ExitOnError)
-	rulesPath := fs.String("rules", "", "additional YAML rule file to include")
+	rulesPath := fs.String("rules", "", "additional YAML rule file or rulepack directory to include")
 	_ = fs.Parse(args)
 	if fs.NArg() < 1 {
 		fmt.Fprintln(os.Stderr, "usage: godzilla rules test <dir> [-rules <file>]")

converters/cpp/converter.go

@@ -20,35 +20,35 @@ import (
 	ir "godzilla/pkg/ir/v1"
 )
 
+// Converter lowers C/C++ sources into gIR: the shared frontend.Driver surface
+// (ConvertFile/ConvertInventory/Skipped) over C/C++'s batch hooks (see batch).
+// Per-file compile failures (e.g. missing headers) are tolerated in directory
+// mode, mirroring the Python/JS frontends.
 type Converter struct {
-	skipped int // files this run could not lower; see Skipped
+	frontend.Driver[cppFileResult]
 }
 
-// Skipped reports how many source files this converter could not lower. The scan
-// layer surfaces it per language, so a run that dropped most of a project is
-// visible instead of reading as clean coverage (see scan.LangCoverage.Skipped).
-func (c *Converter) Skipped() int { return c.skipped }
-
-func NewConverter() *Converter { return &Converter{} }
+func NewConverter() *Converter {
+	c := &Converter{}
+	c.NewBatch = c.batch
+	return c
+}
 
-// ConvertFile lowers the C/C++ at path (a file or directory) to gIR via the
-// shared frontend.Batch driver: per-file compile failures (e.g. missing
-// headers) are tolerated in directory mode, mirroring the Python/JS frontends.
-func (c *Converter) ConvertFile(path string) (*ir.Program, error) {
-	b := frontend.Batch[cppFileResult]{
+// batch builds the shared frontend.Batch driver with C/C++'s hooks. The file
+// predicate is IsCppFile (lang.go, untagged) — the same one internal/scan uses
+// for language detection.
+func (c *Converter) batch() *frontend.Batch[cppFileResult] {
+	return &frontend.Batch[cppFileResult]{
 		Label: "cpp_converter",
 		Lang:  "C/C++",
 		Mode:  "llvm",
-		Match: isCppSource,
+		Match: IsCppFile,
 		Parse: frontend.PerFile(func(_, f string) cppFileResult {
 			mod, err := lowerOne(f)
 			return cppFileResult{mod: mod, err: err}
 		}),
 		Result: func(r *cppFileResult) (*ir.Module, error) { return r.mod, r.err },
 	}
-	prog, skipped, err := b.Convert(path)
-	c.skipped += skipped
-	return prog, err
 }
 
 // cppFileResult is one file's outcome within a batch conversion.
@@ -57,14 +57,6 @@ type cppFileResult struct {
 	err error
 }
 
-var cppExts = map[string]bool{".c": true, ".cc": true, ".cpp": true, ".cxx": true, ".c++": true}
-
-// isCppSource reports whether p is a C/C++ translation unit this frontend
-// compiles (not a header — clang can't compile one to a standalone module).
-// internal/scan keeps its own equivalent (isCppFile) for language detection so
-// scan does not depend on this tag-gated package's file layout.
-func isCppSource(p string) bool { return cppExts[strings.ToLower(filepath.Ext(p))] }
-
 func lowerOne(src string) (*ir.Module, error) {
 	isCpp := strings.ToLower(filepath.Ext(src)) != ".c"
 	cc := compilerFor(isCpp)

converters/cpp/converter_stub.go

@@ -7,6 +7,7 @@ package cpp_converter
 import (
 	"fmt"
 
+	"godzilla/internal/walkignore"
 	ir "godzilla/pkg/ir/v1"
 )
 
@@ -17,3 +18,10 @@ func NewConverter() *Converter { return &Converter{} }
 func (c *Converter) ConvertFile(path string) (*ir.Program, error) {
 	return nil, fmt.Errorf("C/C++ analysis requires building Godzilla with -tags llvm (libLLVM); rebuild to scan %s", path)
 }
+
+// ConvertInventory mirrors the llvm-tagged converter's inventory entry point so
+// the scan pipeline can plumb its cached walk uniformly; like ConvertFile it
+// only reports that this build lacks the C/C++ backend.
+func (c *Converter) ConvertInventory(inv *walkignore.Inventory) (*ir.Program, error) {
+	return c.ConvertFile(inv.Root())
+}

converters/cpp/lang.go

@@ -0,0 +1,16 @@
+// Deliberately NOT build-tagged: internal/scan needs the C/C++ file predicate
+// for language detection in the default (no-libLLVM) build too, so it lives
+// outside the llvm-tagged converter and is the ONE definition both the scan
+// layer and the tagged frontend share (the pattern IsJSFamily set).
+package cpp_converter
+
+import (
+	"path/filepath"
+	"strings"
+)
+
+var cppExts = map[string]bool{".c": true, ".cc": true, ".cpp": true, ".cxx": true, ".c++": true}
+
+// IsCppFile reports whether path is a C or C++ translation unit this frontend
+// compiles (not a header — clang can't compile one to a standalone module).
+func IsCppFile(path string) bool { return cppExts[strings.ToLower(filepath.Ext(path))] }