Skip to content

Add language extraction engine and plugin framework#54

Open
janrito wants to merge 1 commit into
mainfrom
extraction-engine
Open

Add language extraction engine and plugin framework#54
janrito wants to merge 1 commit into
mainfrom
extraction-engine

Conversation

@janrito

@janrito janrito commented Jul 2, 2026

Copy link
Copy Markdown
Owner

Extend the extraction engine with declarative injection for embedded code, two
pluggable resolution hooks, and a version-aware dedup that copes with
multi-language files.

The engine already extracts definition-like symbols through tree-sitter
queries, with a pluggable import_extractor and scope derived from tree
ancestry. Injection generalises the one thing that didn't fit that model:
embedded code — a component's <script>/<style>, fenced blocks in Markdown.
A plugin now declares an injection_query, and the engine delegates each
captured range to a single query-path primitive (extract_query) via
tree-sitter included_ranges, so embedded code extracts at its real absolute
line numbers instead of being handled ad hoc. HTML and Markdown adopt it in
this PR.

Two hooks, name_extractor and scope_extractor, are added mirroring the
existing import_extractor, each defaulting to the engine's capture-based
resolver so a plugin overrides only where a query can't reach.
name_extractor handles names a query can't express — bash alias ll=…
(strip the trailing =), HTML semantic elements named by id else tag.
scope_extractor was taken over a single-language tweak specifically because
it has two real consumers — CSS-family nested rules (ancestry-shaped) and TOML
dotted tables (a dotted-key string with no ancestry) — so it is not the
single-use abstraction an earlier review rejected.

Injection makes multi-language files the norm, which broke the blob-dedup
gate's assumption of one language per file: a .svelte file's chunks are
tagged typescript/css, so the old language="svelte" dedup check never
matched and the file re-parsed on every build. The gate now checks that the
host language is present and that every chunk's (language, plugin_version)
matches the registry, so a bump to any contributing language invalidates the
file. On the back of the two hooks, toml, yaml, and hcl move from chunkers to
queries, and the remaining chunkers gain the ranges-aware signature that lets
them serve as injection targets.

This PR also lands the test harness every later PR builds on: sample
mini-projects under samples/<lang>/, golden chunk and import-edge snapshots,
and per-construct extraction cases. Its conftest extracts through the real
production path (extract_query/_extract_file) rather than a reimplementation,
so a sample test cannot pass against logic production never runs. The chunker-
and config-based plugins ship their updated coverage here (HTML rewritten as a
query plugin, CSS-family nested-rule scope, @_scope for non-lexical scoping)
because they are coupled to these engine changes; the pure code-language
plugins stay at their main baseline and are expanded in the PRs above.


Stacked PRs (merge bottom → top): #54 · #55 · #56 · #57 · #58 · #59 · #60

Extract symbols with tree-sitter queries, delegating embedded
languages through grammar injection and resolving names and scopes
via pluggable extractors. A per-language version map lets the
blob-dedup gate skip files whose extraction is unchanged. Plugins
register through the LanguageRegistration hookspec; the chunker
plugins adopt a ranges-aware signature and toml, yaml, and hcl move
from chunkers to queries. Ships the sample, snapshot, and
extraction-case test harness.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant