Extract the remaining languages into their own packages#73
Merged
Conversation
Makefile (Make) and Dockerfile (Dockerfile) are not bash. Claiming their filenames routed those files to the bash grammar, which extracts nothing from them and only suppressed the plaintext fallback that makes their content searchable. Drop them from the plugin's filenames so they fall through to plaintext; keep the shell config files. Update the detection test (they are now unknown) and the manager/registration docstring examples. No language_plugin_version bump yet.
Move the CSS family into three optional packages (rbtr-lang-css, rbtr-lang-scss, rbtr-lang-less), each with its own tests, samples, and snapshots. scss and less reuse css's public css_nesting_scope helper, so both depend on rbtr-lang-css. Wire the three as the css/scss/less extras; drop their entry points and grammars from core, keeping them out of core's required dependencies. Add all three to mypy.packages. Give the family a real ChunkKind taxonomy instead of lumping everything under doc_section: rule sets and @media / @Keyframes blocks -> class (a named collection of declarations, like SQL's CREATE TABLE); @charset -> config_key (which wires the formerly-unused CONFIG_KEY kind into the engine whitelist); @mixin / @function -> function; $/@ vars and custom properties -> variable; @import -> import. That taxonomy is the one intended extraction change (bumping css/scss/less language_plugin_version and regenerating their snapshots); extraction is otherwise byte-identical to the in-tree originals.
Move the javascript/typescript/tsx plugin (one plugin.py with three LanguageRegistrations, four composed .scm queries, a shared import_extractor) into a new default package. As a default plugin it joins core's required dependencies; its three entry points and two grammars leave core. Carve the 43 js/ts extraction cases and 18 JSDoc cases from the shared core suites into the package (with their own _xfail_nested mark), parametrise test_samples over [javascript, typescript, tsx], and move the three sample projects + six snapshots (extraction and edges byte-identical). The package dev-depends on rbtr-lang-css so the javascript sample's styles.css extracts and the cross-file edges snapshot. Adds rbtr_lang_javascript to mypy.packages. No language_plugin_version bump (pure move).
Move the html plugin (plugin.py + html.scm + injections.scm, with an import_extractor and an injection query for inline <script>/<style>) into a new default package. As a default plugin it joins core's required dependencies; its entry point and grammar leave core. Carve the html symbol case and four import/presence edge tests from the shared core suites; the sample — an html page linking and embedding js and css — moves with it. The package dev-depends on rbtr-lang-javascript and rbtr-lang-css so the injected and linked chunks extract and the cross-file edges snapshot. Adds rbtr_lang_html to mypy.packages. Byte-identical extraction and edges; no language_plugin_version bump (pure move).
Move the python plugin (plugin.py + python.scm, with an import_extractor and interior @_docstring handling) into a new default package. As a default plugin it joins core's required dependencies; its entry point and grammar leave core. Carve the 47 extraction cases, 13 docstring cases (with their _xfail_nested mark), and the four python-specific edge tests — the engine tests that merely use the python grammar (anonymous-name, scope-override, unknown-capture) stay in core. test_docstrings covers the interior-doc slice. Adds rbtr_lang_python to mypy.packages. Byte-identical extraction and edges; no language_plugin_version bump (pure move).
Move the c and cpp plugins (query-only — no import_extractor or injection) into two default packages. As defaults they join core's required dependencies; their entry points and grammars leave core. Carve their extraction and docstring cases (c: 15 + 7, cpp: 20 + 5) from the shared core suites; both are exterior-doc (leading comments). Add rbtr_lang_c and rbtr_lang_cpp to mypy.packages. Byte-identical extraction and edges; no language_plugin_version bump (pure move).
Move the markdown plugin — a chunker (heading-hierarchy sectioning) rather than a tree-sitter query, plus link extraction and a fenced-code injection query (three .scm) — into a new default package. As a default it joins core's required dependencies; its entry point and grammar leave core. Carve the 3 symbol cases and 12 chunker/injection/link edge tests. The package dev-depends on the packaged plugins its sample's fenced blocks delegate to (bash, css, html, javascript, python, scss); svelte/toml/yaml come via core until they are packaged (add them to the dev-deps then). Adds rbtr_lang_markdown to mypy.packages. Byte-identical extraction and edges; no language_plugin_version bump (pure move).
Move the rst plugin — a chunker (adornment-order heading hierarchy) plus reference/role extraction (references.scm), no injection — into the last default package. Its entry point and grammar leave core. Carve the 3 symbol cases and 12 chunker/reference edge tests. The package dev-depends on rbtr-lang-python to extract the sample's helpers.py and snapshot the rst->py reference edges. Adds rbtr_lang_rst to mypy.packages. Byte-identical extraction and edges; no language_plugin_version bump (pure move).
Move go, java, and rust into three optional packages (query-based, exterior-doc). Their entry points and grammars leave core; they become the go/java/rust extras. rust keeps its import_extractor. Carve each language's extraction + docstring cases (with the _xfail_nested mark) and rust's one impl edge test. Add rbtr_lang_go/java/rust to mypy.packages. Byte-identical extraction and edges; no language_plugin_version bump (pure move).
Move the ruby plugin (query-based, import_extractor, exterior-doc) into an optional package; its entry point and grammar leave core and it becomes the ruby extra. Carve its 26 extraction and 9 docstring cases (with the _xfail_nested mark). Add rbtr_lang_ruby to mypy.packages. Byte-identical extraction and edges; no language_plugin_version bump (pure move).
Move the json plugin into an optional package and change its object keys from DOC_SECTION to CONFIG_KEY — json is data, not documentation, and this gives the formerly-orphan CONFIG_KEY its config-family home (wired for CSS's @charset, now used for real keys). NAME_CAPTURE_KEY maps config_key to the existing @_section_name capture so keys keep their names; the snapshot diff is purely doc_section→config_key. Bumps json's language_plugin_version to 2. Its entry point and grammar leave core; adds rbtr_lang_json to mypy.packages. First of the config family (toml/yaml/hcl follow).
Move the yaml plugin into an optional package and change its mapping keys from DOC_SECTION to CONFIG_KEY (data, not docs) — the snapshot diff is purely doc_section→config_key. Bumps yaml's language_plugin_version to 3. Its entry point and grammar leave core; adds rbtr_lang_yaml to mypy.packages. Adds rbtr-lang-yaml to rbtr-lang-markdown's dev-deps (its sample embeds a yaml fence), and regenerates markdown's snapshot for the injected yaml chunk's kind flip (markdown's own extraction is unchanged, so no version bump there).
Move the toml plugin into an optional package and change its tables and array-tables from DOC_SECTION to CONFIG_KEY (data, not docs) — the snapshot diff is purely doc_section→config_key. Bumps toml's language_plugin_version to 3. Its entry point and grammar leave core; adds rbtr_lang_toml to mypy.packages. Adds rbtr-lang-toml to rbtr-lang-markdown's dev-deps (its sample embeds a toml fence) and regenerates markdown's snapshot for the injected chunk's kind flip. Completes json/yaml/toml; hcl finishes the family.
Move the hcl plugin into an optional package and change its top-level blocks from DOC_SECTION to CONFIG_KEY (configuration, not docs) — the snapshot diff is purely doc_section→config_key, block names preserved. Its hcl_block_name name_extractor now gates on the config_key capture, and language_plugin_version bumps to 3. Its entry point and grammar leave core; adds rbtr_lang_hcl to mypy.packages. Completes the config family (json/yaml/toml/hcl all CONFIG_KEY).
Move the tree-sitter query (.scm) plugin into an optional package and rename its id from 'query' to 'tree_sitter_query' (both id and package) — 'query' was ambiguous against rbtr's search query. The plugin, its query file, sample dir, and entry point all take the new name; detection re-derives .scm → tree_sitter_query from the registration. The sample snapshot churns only its language field (chunk ids do not embed the language id); no content change. Its entry point and grammar leave core; adds rbtr_lang_tree_sitter_query to mypy.packages. Keeps DOC_SECTION (a query language, not config).
Split the single-file-component plugin into two packages so the heavy Vue grammar stays off a plain Svelte install: - rbtr-lang-svelte (rbtr[svelte]) — the Svelte plugin plus the shared SFC machinery (chunk_sfc + injections.scm); depends only on tree-sitter-svelte. - rbtr-lang-vue (rbtr[vue]) — a thin package that registers vue, reusing chunk_sfc and the injection query from rbtr-lang-svelte (the scss→css pattern); it alone pulls tree-sitter-language-pack for the Vue grammar. Cache load_query on (package, name) so the SFC injection query, shared by both registrations, is read once — keeping the no-module-constant convention without double I/O. Both grammars leave core; extras svelte/vue point at their own packages; markdown's svelte-fence dev-dep points at rbtr-lang-svelte. Snapshots byte-identical (pure relocation). Retarget the orchestrator's multi-language dedup and per-language version-map tests from the (now optional) svelte sample to an inline Markdown-embeds-Python fixture, so core's build tests use only default plugins and stay self-contained. Completes the language rollout: all 22 plugins are packaged.
Every language now lives in an rbtr-lang-* package with its own extraction, docstring, and sample tests, so core's parametrised per-language harness is vestigial. Rehome the two stragglers: case_bash_assignment → rbtr-lang-bash, and the SQL stored-procedure unsupported-construct xfail → rbtr-lang-sql (as an inline strict-xfail beside its PRAGMA guard). Delete the empty case data files (cases_extraction/docstrings/samples), the per-language parametrised tests, the per-sample test_samples, and the now-unused conftest. Rewrite the surviving engine tests to be self-contained: extract_symbols name/scope/capture mechanics via synthetic registrations over the Python grammar, and doc-attachment (exterior vs interior, line_start shift) driven by two default languages — C (leading-comment) and Python (docstring) — rather than carved per-language cases. No more empty-parameter-set skips.
rbtr[all] pulls every optional language plugin; the defaults are already required, so it installs every language. No web extra — html and javascript are defaults, so it would only add css (redundant with the css extra).
Rename the query plugin to tree_sitter_query throughout (query-language list, the .scm-indexing note, the tags.scm section). Add the @config_key capture convention (JSON/TOML/YAML/HCL keys + CSS @charset, reusing @_section_name), and record that CSS rule sets, @media, and @Keyframes are the canonical non-code use of @Class — fixing the stale 'unnamed @media → anonymous section' example. Note the no-module-query-constant rule and that load_query is cached, so inlining a shared query (svelte/vue's injection query) is free.
Every language now ships as its own rbtr-lang-* package, so core's [project.entry-points."rbtr.languages"] table is empty and its "languages bundled in core" comment is stale. Remove both; LanguageManager still discovers the external packages via the same entry-point group.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This finishes the language-plugin split: every language still in core moves
out into its own
rbtr-lang-*package, so core no longer bundles a grammarit isn't asked to use.
The pilot PRs below packaged SQL and Bash and proved the mechanism; the rest
— around twenty languages — still lived in core, which hard-depended on every
tree-sitter-*grammar. Sopip install rbtrpulled every grammar wheelwhether or not you indexed that language, languages could not be versioned on
their own, and their tests piled into shared files in core that only grew.
After this PR the eight default languages (bash, c, cpp, html, javascript,
markdown, python, rst) are required dependencies, so a bare install still
indexes them, and every other language is an optional extra that pays for its
grammar only when asked:
Each package is a workspace member that registers its languages through the
existing
rbtr.languagesentry point and owns its own sample-and-snapshottests; core registers nothing directly. Most languages move unchanged —
extraction is byte-identical to core.
Two deliberate exceptions carry real extraction changes. JSON, TOML, YAML,
and HCL now key their entries as
config_keyrather thandoc_section,because they are data, not documentation: a JSON object key that was a
doc_sectionchunk is now aconfig_keychunk of the same name. CSS likewisemoves its rule sets,
@media, and@keyframestoclass— a namedcollection of declarations, like a table — and
@charsettoconfig_key.Vue is the awkward case: its grammar ships inside the heavy
tree-sitter-language-pack, yet Svelte and Vue are the same kind ofsingle-file component and share one chunker. Rather than push that pack onto
everyone who only wants Svelte, SFC support is two packages —
rbtr-lang-svelteholds the shared machinery and the Svelte grammar, and athin
rbtr-lang-vuedepends on it for just the Vue grammar — sorbtr[svelte]stays light and the pack arrives only withrbtr[vue].The
.scmquery language is renamed fromquerytotree_sitter_query,whose old id collided with rbtr's own search query. And now that every
language owns its tests, core's shared per-language test files are retired;
what remains is the engine's own tests, driven by synthetic registrations and
two default languages — C for exterior (leading-comment) docs, Python for
interior docstrings.
Each package proves itself by running its sample project through the real
extraction pipeline and snapshotting the result, so a plain move shows an
identical snapshot and an intended change shows a kind-only diff. The
orchestrator's multi-language dedup and per-language version-gating tests,
which had used the Svelte sample, are retargeted to a Markdown-embeds-Python
fixture so core's own tests lean only on default languages.
Two fixes ride along: the Bash plugin stops claiming
MakefileandDockerfile— they are not shell, and claiming them suppressed thesearchable plaintext fallback — and
load_queryis cached so a query sharedby two registrations is read once.
This sits at the top of the stack, on the surface reshape and the extraction
fold below.