Instantiate generative processes by capability, not by namespace

[ealt]

Summary

Teaches run management to instantiate a generative process that lives in the consumer's project
rather than in simplexity.generative_processes, and marks that package pending deprecation. This
is the precondition for moving generative processes out to generators.

Nothing existing changes behaviour: every in-repo process, config and test works unmodified, and the
package emits only a PendingDeprecationWarning.

Motivation

generators distributes process code by copying — "this repo is not a library, do not add it to
your dependencies" — so a conforming process shares no namespace and no base class with simplexity.
But simplexity decided whether a config section was a generative process by testing whether its
_target_ starts with "simplexity.generative_processes.". A namespace test can never pass for
vendored code, because refusing to be a shared namespace is the whole point of that distribution
model.

The conflict is narrower than "config-driven instantiation versus copy-don't-import". Hydra already
instantiates code from outside simplexity on every run — torch.optim.Adam, HookedTransformer.
The problem was specifically that generative_process is the one component discovered by a
namespace prefix and instantiated behind a nominal isinstance check.

So this replaces a nominal test (where does this code live?) with a structural one (what can
this object do?) — which is what generators' own spec licenses:

§2 Operations — "The spec defines what results are required, not how they are organized or computed."

This is also not a new pattern here: is_predictive_model_target already accepts torch.nn,
equinox.nn and penzai.models targets, and _instantiate_predictive_model does a bare
hydra.utils.instantiate with no type check at all. The change brings the most rigid component
filter into line with how the repo already treats components implemented upstream.

What was measured, not assumed

A spike vendored generators' ghmm module into a throwaway package and ran it through the real code
paths before any design was chosen:

Question	Answer
Does a vendored process work with simplexity's generation path today?	Yes — generate_data_batch runs on it unmodified, correct shapes, normalized obs_dist, correct seq_prob. No runtime change was needed; only discovery and validation.
How much of the interface does it miss?	2 of 8 members, both log-space, and neither on the training path.
What did the user see when discovery failed?	One INFO line, no generative process configs found. No warning, no error.
Does a structural protocol still accept existing processes?	Yes — it accepts build_hidden_markov_model's output, so the relaxation is back-compatible by construction.

What changed

Area	Change
run_management/protocols.py (new)	GenerativeProcess as a runtime_checkable Protocol over the operations the runner and training path exercise, plus LogSpaceGenerativeProcess for the two log-space operations only belief-state analysis calls. missing_generative_process_members reports which members are absent.
Discovery	A config section may set component: generative_process to claim a process implemented outside simplexity. The namespace prefix stays as a fast path, so existing configs and in-flight branches need no changes.
utils/config_utils.py	filter_instance_keys_by takes a config-aware predicate and is now the single implementation; filter_instance_keys is a thin adapter over it, unchanged for its callers.
Silent-failure fix	A section that declares a process and then fails validation now raises. Previously it was dropped, leaving components.generative_processes as None — indistinguishable from a config with no process. A rejected instantiation names the missing operations, and the not-found log names what it considered.
generator.py, torch_generator.py	Accept any conforming process.
generative_processes/__init__.py (new)	PendingDeprecationWarning. Ignored by default in CPython, so it adds no noise to normal runs while still surfacing in pytest.
Docs	docs/design/generators_instantiation.md (design, tradeoffs, open questions) and docs/generators_migration.md (consumer guide).

Tiered contract

Today's ABC requires all eight members of every process. Measured usage is asymmetric: the training
and generation path uses seven and never touches log space, while log_probability /
log_observation_probability_distribution are called only by mixed_state_presentation.py and by
composite processes forwarding to sub-processes. Log-space is therefore a separate protocol —
generators does not provide those operations, and requiring them would block vendored processes for
a capability their runs never exercise.

Testing

• tests/end_to_end/test_vendored_process_training.py is the acceptance test: generators'
  modules copied verbatim into tests/vendored_process/generators_copy/ at commit b1242ea,
  plus a ~50-line adapter, training end-to-end under @managed_run(). Neither the copies nor the
  adapter reference simplexity.generative_processes. It also asserts that vocabulary resolution
  reaches the model's d_vocab, which was the second thing the namespace filter broke — the same
  filter routes vocab_size to the predictive-model config.
• tests/run_management/test_protocols.py — contract pinned; parametrized tests that omitting any
  core member breaks conformance and is named in the error; log-space extension boundaries.
• tests/run_management/test_generative_process_discovery.py — declared foreign process
  instantiates and resolves vocabulary; simplexity processes need no declaration; undeclared foreign
  targets are not claimed; a config with no process is still not an error; declared-but-invalid
  raises; non-process targets fail naming the absent operations.
• Additions to test_generative_process_config.py and test_config_utils.py for the declaration
  predicates and the reshaped filter.

1033 passed, 2 skipped (978 before). Diff coverage 100%. Total coverage 94.81%, up from
94.65%. ruff format, ruff check and pyright clean of anything this branch introduces.

Narrowing tokens at the generate boundary also cleared pre-existing pyright looseness that
eqx.filter_vmap had been masking by erasing the return type to Any.

Breaking changes

None. The existing ABC satisfies both new protocols, the namespace fast path is retained, and no
in-repo process, config or test required modification. Existing tests passing unchanged is the
back-compat evidence.

simplexity/generative_processes/__init__.py is new — the package was previously an implicit
namespace package.

Suppressions

None. No # type: ignore or # noqa added.

Two configuration additions, both deliberate:

• pyproject.toml ignores D (docstring) rules for tests/vendored_process/generators_copy/*.
  Vendored code arrives with upstream's conventions — generators' ZEN.md says "Eschew excess
  documentation", so its modules have no docstrings. Editing them to comply would defeat the ability
  to diff against upstream and re-copy, which is how the vendoring model picks up improvements. Every
  consumer vendoring generators will need the same exclusion; it is called out in the migration guide.
• [tool.coverage.report] exclude_lines adds ^\s*\.\.\.$ for Protocol method bodies — the
  structural counterpart to the @abstractmethod entry already there. It excludes 16 statements
  repo-wide, all of which were previously uncovered stub bodies, so nothing executable is hidden.

Out of scope, filed separately

Everything below is deliberately not in this PR. Each is a real decision or defect surfaced by
this work, not a nice-to-have.

Gating the deprecation timeline:

• Astera-org/simplexity#194 — Decide the home
  of mixed-state presentation. Deprecation is gated on generators reaching parity, but belief-tree
  enumeration is analysis over a process rather than an operation of one, and generators' spec
  covers belief updates, not tree enumeration. If it is not going upstream, parity is unreachable by
  definition and the gate never opens. Two eden-experiments consumers import it directly.
• ealt/generators#9 — Capability gaps gating parity
  (log-space operations, framing-token augmentation, process-zoo breadth), with a measured inventory
  and a note on which of them are scope questions rather than work.
• ealt/generators#8 — Conformance vectors do not
  exist. SPEC.md §7 specifies the encoding; no fixtures ship. This is the weakest link in the whole
  chain: a structural protocol check verifies that operations exist, never that they compute the
  right thing, so the vectors are the only mechanism that would catch a vendored copy whose
  modifications broke the mathematics.

Decisions adjacent to this change:

• Astera-org/simplexity#195 — Is
  transition_matrices.py deprecated too? 16 process families vs generators' ~5, and it is data
  rather than behaviour. Changes what "parity" means.
• Astera-org/simplexity#197 — Adopt the
  declaration in the 11 in-repo process configs. Mechanical; deferred purely to avoid churn against
  in-flight branches.
• ealt/generators#10 — Should generators move to
  Astera-org? It is currently ealt/generators; Astera-org/generators does not exist. This PR
  references the URL that resolves, including inside a runtime warning string, so a later move means a
  release to change it.

Pre-existing defects:

• Astera-org/simplexity#199 — pylint silently
  skipped generative_processes entirely. The package had no __init__.py, so pylint simplexity
  never walked into it: 21 modules, ~3,750 lines, outside linter scope while the score read a clean
  10.00/10. Adding __init__.py here (to carry the deprecation warning) brought them into scope and
  surfaced 11 pre-existing messages, which this PR suppresses via per-file-ignores rather than
  refactoring a package that is being deprecated. The scoping gap — coverage shrinking invisibly, with
  nothing to surface it — is the more general bug and is what the issue tracks.
• Astera-org/simplexity#196 — The documented
  local commands do not match CI's. CLAUDE.md/CONTRIBUTING.md document
  --extra dev --extra pytorch, while CI uses --extra aws --extra dev --extra penzai --extra pytorch.
  With the documented commands, pyright reports 20 unresolved imports and a full pytest run aborts
  with 3 collection errors, none of which happen in CI. Plus three ruff findings in
  test_data_prefetcher.py that CI tolerates because its ruff check step is || true. Untouched
  here to keep this diff focused.

Open questions for review

Full list in docs/design/generators_instantiation.md §9. The two that most affect this PR's shape:

• Protocol naming. Shipped as GenerativeProcess in simplexity/run_management/protocols.py,
  reusing the name so the call site reads correctly once the ABC retires — at the cost of two
  same-named classes during the transition. The runner references only the protocol, so the
  ambiguity is confined to readers. GenerativeProcessProtocol is a cheap rename if preferred.
• The declaration field is named component, matching the runner's existing vocabulary
  (Components, component_name). component_type was unavailable — factored-process specs already
  use it for hmm/ghmm. Rejected kind (less specific) and external: true (names the exception
  rather than the role, so it ages badly once declaration becomes the norm).
• Should the 11 in-repo configs adopt the declaration now? Left alone to avoid churn against the
  60+ open branches; eventually every process config declares itself and the prefix path retires.

🤖 Generated with Claude Code

https://claude.ai/code/session_01XrVfBeUFhMToafrbgdDGUo

[cursor]

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.