Add language extraction engine and plugin framework#54
Open
janrito wants to merge 1 commit into
Open
Conversation
Extract symbols with tree-sitter queries, delegating embedded languages through grammar injection and resolving names and scopes via pluggable extractors. A per-language version map lets the blob-dedup gate skip files whose extraction is unchanged. Plugins register through the LanguageRegistration hookspec; the chunker plugins adopt a ranges-aware signature and toml, yaml, and hcl move from chunkers to queries. Ships the sample, snapshot, and extraction-case test harness.
This was referenced Jul 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extend the extraction engine with declarative injection for embedded code, two
pluggable resolution hooks, and a version-aware dedup that copes with
multi-language files.
The engine already extracts definition-like symbols through tree-sitter
queries, with a pluggable
import_extractorand scope derived from treeancestry. Injection generalises the one thing that didn't fit that model:
embedded code — a component's
<script>/<style>, fenced blocks in Markdown.A plugin now declares an
injection_query, and the engine delegates eachcaptured range to a single query-path primitive (
extract_query) viatree-sitter
included_ranges, so embedded code extracts at its real absoluteline numbers instead of being handled ad hoc. HTML and Markdown adopt it in
this PR.
Two hooks,
name_extractorandscope_extractor, are added mirroring theexisting
import_extractor, each defaulting to the engine's capture-basedresolver so a plugin overrides only where a query can't reach.
name_extractorhandles names a query can't express — bashalias ll=…(strip the trailing
=), HTML semantic elements named byidelse tag.scope_extractorwas taken over a single-language tweak specifically becauseit has two real consumers — CSS-family nested rules (ancestry-shaped) and TOML
dotted tables (a dotted-key string with no ancestry) — so it is not the
single-use abstraction an earlier review rejected.
Injection makes multi-language files the norm, which broke the blob-dedup
gate's assumption of one language per file: a
.sveltefile's chunks aretagged
typescript/css, so the oldlanguage="svelte"dedup check nevermatched and the file re-parsed on every build. The gate now checks that the
host language is present and that every chunk's
(language, plugin_version)matches the registry, so a bump to any contributing language invalidates the
file. On the back of the two hooks, toml, yaml, and hcl move from chunkers to
queries, and the remaining chunkers gain the ranges-aware signature that lets
them serve as injection targets.
This PR also lands the test harness every later PR builds on: sample
mini-projects under
samples/<lang>/, golden chunk and import-edge snapshots,and per-construct extraction cases. Its
conftestextracts through the realproduction path (
extract_query/_extract_file) rather than a reimplementation,so a sample test cannot pass against logic production never runs. The chunker-
and config-based plugins ship their updated coverage here (HTML rewritten as a
query plugin, CSS-family nested-rule scope,
@_scopefor non-lexical scoping)because they are coupled to these engine changes; the pure code-language
plugins stay at their
mainbaseline and are expanded in the PRs above.Stacked PRs (merge bottom → top): #54 · #55 · #56 · #57 · #58 · #59 · #60