Skip to content

Commit 5e7e863

Browse files
fabceolinclaude
andcommitted
feat: Lean 4 analyzer, with header-scanner structure and exact-name resolution
The tree-sitter Lean grammar fails on many commands and notations and gives `end` as a flat node, so the analyzer uses two layers. A header scanner reads declarations, the namespace/section/mutual stack, `open` (also `open X in`), imports, spans and docstrings from the source with comments and strings masked. References come from the tree-sitter identifiers in each declaration, also inside ERROR nodes, without binders, let/have/fun/intro names and match patterns. CallGraphAnalyzer resolves a Lean reference only to an exact fully qualified name, in Lean's order: the declaration's namespaces, the root, then the open namespaces, in the caller's file and the files it imports transitively. Private names stay in their file. No match, or more than one, drops the reference, and a `lean` branch keeps this authoritative like Go and Dart. Constructor and field names link to their structure or inductive, and a dotted name falls back to its longest matching prefix (`origin.x`). Lean is registered with the file discovery, the CLI validators and the MCP tools. `.lake/` and `lake-packages/` are skipped, lakefile.lean/.toml are build artifacts, and lake-manifest.json and lean-toolchain are manifests. The Docker image downloads the `lean` grammar. Refs #154 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 474a1d8 commit 5e7e863

19 files changed

Lines changed: 1201 additions & 9 deletions

‎CHANGELOG.md‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,25 @@ uses [Semantic Versioning](https://semver.org/).
6565
`addModule`; `std`, `builtin`, and third-party packages are dropped. `test`
6666
blocks and `zig-cache/`, `.zig-cache/`, `zig-out/` are skipped, `build.zig`
6767
is also a build artifact, and `build.zig.zon` is a manifest.
68+
- **Lean 4 analyzer** ([#154](https://github.com/FSoft-AI4Code/CodeWiki/issues/154)).
69+
`def`, `theorem`/`lemma`, `abbrev`, `instance`, `structure`, `class`,
70+
`inductive`, `axiom` and `opaque` declarations become components, keyed by
71+
their fully qualified name (`Geo.Point.add`). A header scanner reads the
72+
structure from the source with comments and strings masked: declarations,
73+
the `namespace`/`section`/`mutual`/`end` stack, `open` (also `open X in`),
74+
`import`, spans and `/-- -/` docstrings. So a region that the tree-sitter
75+
grammar cannot parse does not hide a declaration. References come from the
76+
tree-sitter identifiers, without binders, `let`/`have`/`fun`/`intro` names
77+
and match patterns. A reference resolves only to an exact name in the
78+
declaration's namespaces, the root or the open namespaces, in a file that
79+
the caller imports (transitively); `private` names stay in their file, and
80+
a reference with no match or more than one is dropped. Constructor and
81+
field names (`Point.mk`, `Point.x`, `Shape.dot`) link to their structure or
82+
inductive, and `origin.x` links to `origin`. `structure` is a `struct`,
83+
`class` an `interface`, `inductive` a `class`, and the rest are functions.
84+
`.lake/` and `lake-packages/` are skipped, `lakefile.lean` and
85+
`lakefile.toml` are build artifacts, and `lake-manifest.json` and
86+
`lean-toolchain` are manifests.
6887
- **Vue single-file components.** Each `.vue` file becomes a component
6988
(`ArticleList.vue::ArticleList`), and its options-API members (`methods`,
7089
`computed`, `watch`, lifecycle hooks such as `mounted` or `setup`) become

‎README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -120,8 +120,8 @@ Full list in the [CHANGELOG](./CHANGELOG.md).
120120
## Features
121121

122122
**Languages.** Python, Java, JavaScript, TypeScript, C, C++, C#, Kotlin,
123-
PHP, Ruby, Scala, Rust, Dart (Flutter-aware), Go, Zig, Vue (single-file
124-
components).
123+
PHP, Ruby, Scala, Rust, Dart (Flutter-aware), Go, Zig, Lean 4, Vue
124+
(single-file components).
125125

126126
**Hierarchical decomposition.** The dependency graph is clustered into a
127127
module tree, recursively, so a 1.4M-line repository gets the same treatment

‎codewiki/cli/utils/repo_validator.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@
3737
".dart", # Dart
3838
".go", # Go
3939
".zig", # Zig
40+
".lean", # Lean 4
4041
".vue", # Vue single-file components
4142
}
4243

‎codewiki/cli/utils/validation.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -171,6 +171,7 @@ def detect_supported_languages(directory: Path) -> list[tuple[str, int]]:
171171
"Dart": [".dart"],
172172
"Go": [".go"],
173173
"Zig": [".zig"],
174+
"Lean": [".lean"],
174175
"Vue": [".vue"],
175176
}
176177

‎codewiki/mcp/tools/analysis.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -232,6 +232,7 @@ def _detect_via_mtime(
232232
".dart",
233233
".go",
234234
".zig",
235+
".lean",
235236
".vue",
236237
}
237238

‎codewiki/src/be/dependency_analyzer/analysis/analysis_service.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -365,6 +365,7 @@ def _filter_supported_languages(self, code_files: list[dict]) -> list[dict]:
365365
"scala",
366366
"dart",
367367
"zig",
368+
"lean",
368369
"vue",
369370
}
370371

@@ -392,6 +393,7 @@ def _get_supported_languages(self) -> list[str]:
392393
"dart",
393394
"go",
394395
"zig",
396+
"lean",
395397
"vue",
396398
]
397399

‎codewiki/src/be/dependency_analyzer/analysis/call_graph_analyzer.py‎

Lines changed: 153 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,9 @@
2828

2929
logger = logging.getLogger(__name__)
3030

31+
# `_resolve_lean_name` result: no declaration has this name (try a shorter one).
32+
_LEAN_NO_MATCH = object()
33+
3134

3235
class TimeoutError(BaseException):
3336
"""Raised when file parsing exceeds timeout.
@@ -76,6 +79,7 @@ def __init__(self):
7679
self._dart_unscoped = False
7780
self._go_visible: dict[str, set[str]] = {}
7881
self._go_index = None
82+
self._reset_lean_state()
7983
self.call_relationships: list[CallRelationship] = []
8084
self._python_project_modules: set = set()
8185
self._python_external_import_roots: set = set()
@@ -108,6 +112,7 @@ def analyze_code_files(self, code_files: list[dict], base_dir: str) -> dict:
108112
self._dart_unscoped = False
109113
self._go_visible = {}
110114
self._go_index = None
115+
self._reset_lean_state()
111116

112117
files_analyzed = 0
113118
files_failed = 0
@@ -293,6 +298,8 @@ def _analyze_code_file(self, repo_dir: str, file_info: dict):
293298
self._analyze_go_file(file_path, content, repo_dir)
294299
elif language == "zig":
295300
self._analyze_zig_file(file_path, content, repo_dir)
301+
elif language == "lean":
302+
self._analyze_lean_file(file_path, content, repo_dir)
296303
elif language == "vue":
297304
self._analyze_vue_file(file_path, content, repo_dir)
298305
# else:
@@ -668,6 +675,145 @@ def _analyze_vue_file(self, file_path: str, content: str, repo_dir: str):
668675
except Exception:
669676
logger.exception(f"Failed to analyze Vue file {file_path}")
670677

678+
def _reset_lean_state(self) -> None:
679+
# Relative path -> imported module names, per Lean file.
680+
self._lean_imports: dict[str, list[str]] = {}
681+
# Node id -> (namespace prefixes innermost first, open namespaces).
682+
self._lean_scopes: dict[str, tuple[list[str], list[str]]] = {}
683+
# Constructor/field FQN -> ids of the structures or inductives it names.
684+
self._lean_aliases: dict[str, list[str]] = defaultdict(list)
685+
self._lean_private: set[str] = set()
686+
self._lean_index: dict[str, list[str]] | None = None
687+
self._lean_modules: dict[str, set[str]] = {}
688+
self._lean_visible: dict[str, frozenset[str]] = {}
689+
690+
def _analyze_lean_file(self, file_path: str, content: str, repo_dir: str):
691+
"""
692+
Analyze a Lean 4 file.
693+
694+
Besides components and references, records the file's imports, each
695+
declaration's namespaces and open namespaces, and the constructor and
696+
field names of its structures, for Lean resolution by exact name.
697+
698+
Args:
699+
file_path: Path to the Lean file
700+
content: File content string
701+
repo_dir: Repository base directory
702+
"""
703+
from codewiki.src.be.dependency_analyzer.analyzers.lean import TreeSitterLeanAnalyzer
704+
705+
try:
706+
analyzer = TreeSitterLeanAnalyzer(str(file_path), content, repo_path=repo_dir)
707+
708+
for func in analyzer.nodes:
709+
self.functions[func.id] = func
710+
711+
self.call_relationships.extend(analyzer.call_relationships)
712+
self._lean_imports[analyzer.relative_path] = analyzer.imports
713+
self._lean_scopes.update(analyzer.scopes)
714+
for alias, node_id in analyzer.aliases.items():
715+
self._lean_aliases[alias].append(node_id)
716+
self._lean_private.update(analyzer.private_ids)
717+
except Exception:
718+
logger.exception(f"Failed to analyze Lean file {file_path}")
719+
720+
def _build_lean_index(self) -> dict[str, list[str]]:
721+
"""FQN -> Lean node ids, and module name -> files. A module name maps
722+
to every file whose path ends with it (`Geo.Point` -> `src/Geo/Point.lean`),
723+
because the Lake source folder is not known."""
724+
if self._lean_index is None:
725+
index: dict[str, list[str]] = defaultdict(list)
726+
for func_id, func in self.functions.items():
727+
if func.language == "lean" and func.qualified_name:
728+
index[func.qualified_name].append(func_id)
729+
for alias, node_ids in self._lean_aliases.items():
730+
for node_id in node_ids:
731+
if node_id not in index[alias]:
732+
index[alias].append(node_id)
733+
modules: dict[str, set[str]] = defaultdict(set)
734+
for rel in self._lean_imports:
735+
parts = rel[: -len(".lean")].split("/")
736+
for k in range(len(parts)):
737+
modules[".".join(parts[k:])].add(rel)
738+
self._lean_index = index
739+
self._lean_modules = modules
740+
return self._lean_index
741+
742+
def _lean_visible_files(self, rel: str) -> frozenset[str]:
743+
"""The file itself and every repository file it imports, transitively."""
744+
if rel not in self._lean_visible:
745+
self._build_lean_index()
746+
seen = {rel}
747+
queue = [rel]
748+
while queue:
749+
current = queue.pop()
750+
for module in self._lean_imports.get(current, []):
751+
for target in self._lean_modules.get(module, ()):
752+
if target not in seen:
753+
seen.add(target)
754+
queue.append(target)
755+
self._lean_visible[rel] = frozenset(seen)
756+
return self._lean_visible[rel]
757+
758+
def _resolve_lean_callee(self, callee: str, caller: Node) -> str | None:
759+
"""Resolve a Lean reference to one declaration by exact FQN, in Lean's
760+
order: the caller's namespaces (innermost first), the root, then the
761+
open namespaces. No match, or more than one, drops the reference.
762+
763+
When the whole dotted name matches nothing, its longest prefix that
764+
matches is the constant and the rest are projections, as in Lean
765+
(`origin.x` is `origin` then the field `x`)."""
766+
parts = callee.split(".")
767+
for end in range(len(parts), 0, -1):
768+
found = self._resolve_lean_name(".".join(parts[:end]), caller)
769+
if found is not _LEAN_NO_MATCH:
770+
return found
771+
return None
772+
773+
def _resolve_lean_name(self, callee: str, caller: Node):
774+
"""One exact name: a node id, None (ambiguous or recursion: stop), or
775+
_LEAN_NO_MATCH (try a shorter prefix)."""
776+
index = self._build_lean_index()
777+
visible = self._lean_visible_files(caller.relative_path)
778+
namespaces, opens = self._lean_scopes.get(caller.id, ([], []))
779+
780+
def qualify(prefix: str, name: str) -> str:
781+
return f"{prefix}.{name}" if prefix else name
782+
783+
def matches(fqn: str) -> list[str]:
784+
return [
785+
node_id
786+
for node_id in index.get(fqn, [])
787+
if self.functions[node_id].relative_path in visible
788+
and (
789+
node_id not in self._lean_private
790+
or self.functions[node_id].relative_path == caller.relative_path
791+
)
792+
]
793+
794+
def unique(found) -> str | None:
795+
# A reference to the caller itself is recursion, not an edge.
796+
found = list(found)
797+
return found[0] if len(found) == 1 and found[0] != caller.id else None
798+
799+
if callee.startswith("_root_."):
800+
found = matches(callee[len("_root_.") :])
801+
return unique(found) if found else _LEAN_NO_MATCH
802+
803+
prefixes = [*namespaces, ""]
804+
for prefix in prefixes:
805+
found = matches(qualify(prefix, callee))
806+
if found:
807+
return unique(found) # the innermost match shadows the outer ones
808+
809+
opened = {
810+
node_id
811+
for name in opens
812+
for prefix in prefixes
813+
for node_id in matches(qualify(qualify(prefix, name), callee))
814+
}
815+
return unique(opened) if opened else _LEAN_NO_MATCH
816+
671817
def _analyze_go_file(self, file_path: str, content: str, repo_dir: str):
672818
"""
673819
Analyze Go file using tree-sitter based analyzer.
@@ -860,11 +1006,11 @@ def _resolve_call_relationships(self):
8601006
self.call_relationships.extend(link_go_interfaces(self.functions))
8611007

8621008
def _has_authoritative_resolution(self, relationship: CallRelationship) -> bool:
863-
"""A Dart or Go caller's callee that the language's scoped resolution
1009+
"""A Dart, Go or Lean caller's callee that the language's scoped resolution
8641010
rejected is external or invisible; drop it so ast_parser's name-only
8651011
fallback cannot bind it to an unrelated component."""
8661012
caller = self.functions.get(relationship.caller)
867-
return caller is not None and caller.language in ("dart", "go")
1013+
return caller is not None and caller.language in ("dart", "go", "lean")
8681014

8691015
def _dotted_project_packages(self) -> dict[str, set]:
8701016
"""Project packages/namespaces, partitioned by language. Java and C#
@@ -1007,6 +1153,9 @@ def _resolve_callee(
10071153
if caller_language == "go" and caller is not None:
10081154
# So is Go's: a name resolves only in the package it names.
10091155
return self._resolve_go_callee(relationship.callee, caller)
1156+
if caller_language == "lean" and caller is not None:
1157+
# And Lean's: only an exact FQN in a visible file.
1158+
return self._resolve_lean_callee(relationship.callee, caller)
10101159
if lang_indexes:
10111160
match = self._resolve_callee_in(
10121161
relationship, lang_indexes["exact"], lang_indexes["simple"]
@@ -1152,6 +1301,8 @@ def _generate_visualization_data(self) -> dict:
11521301
node_classes.append("lang-go")
11531302
elif file_ext == ".zig":
11541303
node_classes.append("lang-zig")
1304+
elif file_ext == ".lean":
1305+
node_classes.append("lang-lean")
11551306
elif file_ext == ".vue":
11561307
node_classes.append("lang-vue")
11571308

‎codewiki/src/be/dependency_analyzer/analyzers/artifact.py‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -244,6 +244,8 @@ def is_artifact_file_node(node: Any) -> bool:
244244
"build.sbt",
245245
"Package.swift",
246246
"build.zig.zon",
247+
"lake-manifest.json",
248+
"lean-toolchain",
247249
"pubspec.yaml",
248250
"Pipfile",
249251
"environment.yml",
@@ -296,6 +298,8 @@ def is_artifact_file_node(node: Any) -> bool:
296298
"Gruntfile.js",
297299
"Herebyfile.mjs",
298300
"build.zig",
301+
"lakefile.lean",
302+
"lakefile.toml",
299303
# Autotools inputs; the generated configure, Makefile.in and aclocal.m4
300304
# are not build sources.
301305
"configure.ac",
@@ -451,6 +455,7 @@ def classify_artifact(
451455
".scala",
452456
".sc",
453457
".zig",
458+
".lean",
454459
".vue",
455460
}:
456461
if name in _TEST_INFRA_NAMES:

0 commit comments

Comments
 (0)