Skip to content

Extract the remaining languages into their own packages#73

Merged
janrito merged 20 commits into
mainfrom
feat/lang-extract-remaining
Jul 12, 2026
Merged

Extract the remaining languages into their own packages#73
janrito merged 20 commits into
mainfrom
feat/lang-extract-remaining

Conversation

@janrito

@janrito janrito commented Jul 8, 2026

Copy link
Copy Markdown
Owner

This finishes the language-plugin split: every language still in core moves
out into its own rbtr-lang-* package, so core no longer bundles a grammar
it isn't asked to use.

The pilot PRs below packaged SQL and Bash and proved the mechanism; the rest
— around twenty languages — still lived in core, which hard-depended on every
tree-sitter-* grammar. So pip install rbtr pulled every grammar wheel
whether or not you indexed that language, languages could not be versioned on
their own, and their tests piled into shared files in core that only grew.
After this PR the eight default languages (bash, c, cpp, html, javascript,
markdown, python, rst) are required dependencies, so a bare install still
indexes them, and every other language is an optional extra that pays for its
grammar only when asked:

pip install rbtr            # the eight defaults
pip install rbtr[go,vue]    # add Go and Vue
pip install rbtr[all]       # every language

Each package is a workspace member that registers its languages through the
existing rbtr.languages entry point and owns its own sample-and-snapshot
tests; core registers nothing directly. Most languages move unchanged —
extraction is byte-identical to core.

Two deliberate exceptions carry real extraction changes. JSON, TOML, YAML,
and HCL now key their entries as config_key rather than doc_section,
because they are data, not documentation: a JSON object key that was a
doc_section chunk is now a config_key chunk of the same name. CSS likewise
moves its rule sets, @media, and @keyframes to class — a named
collection of declarations, like a table — and @charset to config_key.

Vue is the awkward case: its grammar ships inside the heavy
tree-sitter-language-pack, yet Svelte and Vue are the same kind of
single-file component and share one chunker. Rather than push that pack onto
everyone who only wants Svelte, SFC support is two packages —
rbtr-lang-svelte holds the shared machinery and the Svelte grammar, and a
thin rbtr-lang-vue depends on it for just the Vue grammar — so
rbtr[svelte] stays light and the pack arrives only with rbtr[vue].

The .scm query language is renamed from query to tree_sitter_query,
whose old id collided with rbtr's own search query. And now that every
language owns its tests, core's shared per-language test files are retired;
what remains is the engine's own tests, driven by synthetic registrations and
two default languages — C for exterior (leading-comment) docs, Python for
interior docstrings.

Each package proves itself by running its sample project through the real
extraction pipeline and snapshotting the result, so a plain move shows an
identical snapshot and an intended change shows a kind-only diff. The
orchestrator's multi-language dedup and per-language version-gating tests,
which had used the Svelte sample, are retargeted to a Markdown-embeds-Python
fixture so core's own tests lean only on default languages.

Two fixes ride along: the Bash plugin stops claiming Makefile and
Dockerfile — they are not shell, and claiming them suppressed the
searchable plaintext fallback — and load_query is cached so a query shared
by two registrations is read once.

This sits at the top of the stack, on the surface reshape and the extraction
fold below.

janrito added 20 commits July 8, 2026 23:51
Makefile (Make) and Dockerfile (Dockerfile) are not bash. Claiming their
filenames routed those files to the bash grammar, which extracts nothing from
them and only suppressed the plaintext fallback that makes their content
searchable. Drop them from the plugin's filenames so they fall through to
plaintext; keep the shell config files. Update the detection test (they are
now unknown) and the manager/registration docstring examples. No
language_plugin_version bump yet.
Move the CSS family into three optional packages (rbtr-lang-css,
rbtr-lang-scss, rbtr-lang-less), each with its own tests, samples, and
snapshots. scss and less reuse css's public css_nesting_scope helper, so both
depend on rbtr-lang-css. Wire the three as the css/scss/less extras; drop
their entry points and grammars from core, keeping them out of core's required
dependencies. Add all three to mypy.packages.

Give the family a real ChunkKind taxonomy instead of lumping everything under
doc_section: rule sets and @media / @Keyframes blocks -> class (a named
collection of declarations, like SQL's CREATE TABLE); @charset -> config_key
(which wires the formerly-unused CONFIG_KEY kind into the engine whitelist);
@mixin / @function -> function; $/@ vars and custom properties -> variable;
@import -> import. That taxonomy is the one intended extraction change (bumping
css/scss/less language_plugin_version and regenerating their snapshots);
extraction is otherwise byte-identical to the in-tree originals.
Move the javascript/typescript/tsx plugin (one plugin.py with three
LanguageRegistrations, four composed .scm queries, a shared import_extractor)
into a new default package. As a default plugin it joins core's required
dependencies; its three entry points and two grammars leave core. Carve the
43 js/ts extraction cases and 18 JSDoc cases from the shared core suites into
the package (with their own _xfail_nested mark), parametrise test_samples over
[javascript, typescript, tsx], and move the three sample projects + six
snapshots (extraction and edges byte-identical). The package dev-depends on
rbtr-lang-css so the javascript sample's styles.css extracts and the cross-file
edges snapshot. Adds rbtr_lang_javascript to mypy.packages. No
language_plugin_version bump (pure move).
Move the html plugin (plugin.py + html.scm + injections.scm, with an
import_extractor and an injection query for inline <script>/<style>) into a
new default package. As a default plugin it joins core's required
dependencies; its entry point and grammar leave core. Carve the html symbol
case and four import/presence edge tests from the shared core suites; the
sample — an html page linking and embedding js and css — moves with it. The
package dev-depends on rbtr-lang-javascript and rbtr-lang-css so the injected
and linked chunks extract and the cross-file edges snapshot. Adds
rbtr_lang_html to mypy.packages. Byte-identical extraction and edges; no
language_plugin_version bump (pure move).
Move the python plugin (plugin.py + python.scm, with an import_extractor and
interior @_docstring handling) into a new default package. As a default plugin
it joins core's required dependencies; its entry point and grammar leave core.
Carve the 47 extraction cases, 13 docstring cases (with their _xfail_nested
mark), and the four python-specific edge tests — the engine tests that merely
use the python grammar (anonymous-name, scope-override, unknown-capture) stay
in core. test_docstrings covers the interior-doc slice. Adds rbtr_lang_python
to mypy.packages. Byte-identical extraction and edges; no
language_plugin_version bump (pure move).
Move the c and cpp plugins (query-only — no import_extractor or injection)
into two default packages. As defaults they join core's required
dependencies; their entry points and grammars leave core. Carve their
extraction and docstring cases (c: 15 + 7, cpp: 20 + 5) from the shared core
suites; both are exterior-doc (leading comments). Add rbtr_lang_c and
rbtr_lang_cpp to mypy.packages. Byte-identical extraction and edges; no
language_plugin_version bump (pure move).
Move the markdown plugin — a chunker (heading-hierarchy sectioning) rather
than a tree-sitter query, plus link extraction and a fenced-code injection
query (three .scm) — into a new default package. As a default it joins core's
required dependencies; its entry point and grammar leave core. Carve the 3
symbol cases and 12 chunker/injection/link edge tests. The package dev-depends
on the packaged plugins its sample's fenced blocks delegate to (bash, css,
html, javascript, python, scss); svelte/toml/yaml come via core until they are
packaged (add them to the dev-deps then). Adds rbtr_lang_markdown to
mypy.packages. Byte-identical extraction and edges; no language_plugin_version
bump (pure move).
Move the rst plugin — a chunker (adornment-order heading hierarchy) plus
reference/role extraction (references.scm), no injection — into the last
default package. Its entry point and grammar leave core. Carve the 3 symbol
cases and 12 chunker/reference edge tests. The package dev-depends on
rbtr-lang-python to extract the sample's helpers.py and snapshot the rst->py
reference edges. Adds rbtr_lang_rst to mypy.packages. Byte-identical extraction
and edges; no language_plugin_version bump (pure move).
Move go, java, and rust into three optional packages (query-based,
exterior-doc). Their entry points and grammars leave core; they become the
go/java/rust extras. rust keeps its import_extractor. Carve each language's
extraction + docstring cases (with the _xfail_nested mark) and rust's one impl
edge test. Add rbtr_lang_go/java/rust to mypy.packages. Byte-identical
extraction and edges; no language_plugin_version bump (pure move).
Move the ruby plugin (query-based, import_extractor, exterior-doc) into an
optional package; its entry point and grammar leave core and it becomes the
ruby extra. Carve its 26 extraction and 9 docstring cases (with the
_xfail_nested mark). Add rbtr_lang_ruby to mypy.packages. Byte-identical
extraction and edges; no language_plugin_version bump (pure move).
Move the json plugin into an optional package and change its object keys from
DOC_SECTION to CONFIG_KEY — json is data, not documentation, and this gives
the formerly-orphan CONFIG_KEY its config-family home (wired for CSS's
@charset, now used for real keys). NAME_CAPTURE_KEY maps config_key to the
existing @_section_name capture so keys keep their names; the snapshot diff is
purely doc_section→config_key. Bumps json's language_plugin_version to 2. Its
entry point and grammar leave core; adds rbtr_lang_json to mypy.packages.
First of the config family (toml/yaml/hcl follow).
Move the yaml plugin into an optional package and change its mapping keys from
DOC_SECTION to CONFIG_KEY (data, not docs) — the snapshot diff is purely
doc_section→config_key. Bumps yaml's language_plugin_version to 3. Its entry
point and grammar leave core; adds rbtr_lang_yaml to mypy.packages. Adds
rbtr-lang-yaml to rbtr-lang-markdown's dev-deps (its sample embeds a yaml
fence), and regenerates markdown's snapshot for the injected yaml chunk's
kind flip (markdown's own extraction is unchanged, so no version bump there).
Move the toml plugin into an optional package and change its tables and
array-tables from DOC_SECTION to CONFIG_KEY (data, not docs) — the snapshot
diff is purely doc_section→config_key. Bumps toml's language_plugin_version to
3. Its entry point and grammar leave core; adds rbtr_lang_toml to
mypy.packages. Adds rbtr-lang-toml to rbtr-lang-markdown's dev-deps (its
sample embeds a toml fence) and regenerates markdown's snapshot for the
injected chunk's kind flip. Completes json/yaml/toml; hcl finishes the family.
Move the hcl plugin into an optional package and change its top-level blocks
from DOC_SECTION to CONFIG_KEY (configuration, not docs) — the snapshot diff
is purely doc_section→config_key, block names preserved. Its hcl_block_name
name_extractor now gates on the config_key capture, and language_plugin_version
bumps to 3. Its entry point and grammar leave core; adds rbtr_lang_hcl to
mypy.packages. Completes the config family (json/yaml/toml/hcl all CONFIG_KEY).
Move the tree-sitter query (.scm) plugin into an optional package and rename
its id from 'query' to 'tree_sitter_query' (both id and package) — 'query'
was ambiguous against rbtr's search query. The plugin, its query file, sample
dir, and entry point all take the new name; detection re-derives .scm →
tree_sitter_query from the registration. The sample snapshot churns only its
language field (chunk ids do not embed the language id); no content change.
Its entry point and grammar leave core; adds rbtr_lang_tree_sitter_query to
mypy.packages. Keeps DOC_SECTION (a query language, not config).
Split the single-file-component plugin into two packages so the heavy Vue
grammar stays off a plain Svelte install:

- rbtr-lang-svelte (rbtr[svelte]) — the Svelte plugin plus the shared SFC
  machinery (chunk_sfc + injections.scm); depends only on tree-sitter-svelte.
- rbtr-lang-vue (rbtr[vue]) — a thin package that registers vue, reusing
  chunk_sfc and the injection query from rbtr-lang-svelte (the scss→css
  pattern); it alone pulls tree-sitter-language-pack for the Vue grammar.

Cache load_query on (package, name) so the SFC injection query, shared by both
registrations, is read once — keeping the no-module-constant convention without
double I/O. Both grammars leave core; extras svelte/vue point at their own
packages; markdown's svelte-fence dev-dep points at rbtr-lang-svelte. Snapshots
byte-identical (pure relocation).

Retarget the orchestrator's multi-language dedup and per-language version-map
tests from the (now optional) svelte sample to an inline Markdown-embeds-Python
fixture, so core's build tests use only default plugins and stay self-contained.

Completes the language rollout: all 22 plugins are packaged.
Every language now lives in an rbtr-lang-* package with its own extraction,
docstring, and sample tests, so core's parametrised per-language harness is
vestigial. Rehome the two stragglers: case_bash_assignment → rbtr-lang-bash,
and the SQL stored-procedure unsupported-construct xfail → rbtr-lang-sql (as an
inline strict-xfail beside its PRAGMA guard). Delete the empty case data files
(cases_extraction/docstrings/samples), the per-language parametrised tests, the
per-sample test_samples, and the now-unused conftest.

Rewrite the surviving engine tests to be self-contained: extract_symbols
name/scope/capture mechanics via synthetic registrations over the Python
grammar, and doc-attachment (exterior vs interior, line_start shift) driven by
two default languages — C (leading-comment) and Python (docstring) — rather
than carved per-language cases. No more empty-parameter-set skips.
rbtr[all] pulls every optional language plugin; the defaults are already
required, so it installs every language. No web extra — html and javascript
are defaults, so it would only add css (redundant with the css extra).
Rename the query plugin to tree_sitter_query throughout (query-language list,
the .scm-indexing note, the tags.scm section). Add the @config_key capture
convention (JSON/TOML/YAML/HCL keys + CSS @charset, reusing @_section_name),
and record that CSS rule sets, @media, and @Keyframes are the canonical
non-code use of @Class — fixing the stale 'unnamed @media → anonymous section'
example. Note the no-module-query-constant rule and that load_query is cached,
so inlining a shared query (svelte/vue's injection query) is free.
Every language now ships as its own rbtr-lang-* package, so core's
[project.entry-points."rbtr.languages"] table is empty and its "languages
bundled in core" comment is stale. Remove both; LanguageManager still
discovers the external packages via the same entry-point group.
Base automatically changed from refactor/fold-extraction-into-languages to main July 12, 2026 22:12
@janrito
janrito merged commit a2dcced into main Jul 12, 2026
1 check passed
@janrito
janrito deleted the feat/lang-extract-remaining branch July 12, 2026 22:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant