Skip to content

Fix Tracr demo output_label: compiled labels are auto-numbered#1525

Merged
jlarson4 merged 1 commit into
TransformerLensOrg:devfrom
dewstend:fix-tracr-demo-output-label
Jul 21, 2026
Merged

Fix Tracr demo output_label: compiled labels are auto-numbered#1525
jlarson4 merged 1 commit into
TransformerLensOrg:devfrom
dewstend:fix-tracr-demo-output-label

Conversation

@dewstend

Copy link
Copy Markdown

Description

The Tracr demo as shipped fails at its state-dict cell. Cell 11 passes output_label="reverse", but Tracr builds labels as f"{name}_{unique_id}", so the compiled expression's residual labels are reverse_1:* and make_tracr_transformer_bridge_state_dict raises:

ValueError: Could not find output basis label 'reverse:1' in model.residual_labels. Pass the final RASP expression label as output_label.

This is a follow-up to #1460 (the TransformerBridge port addressing #481): the conversion helpers there are correct, but the demo call site — and the unit-test fixture — assume the un-numbered label. The fixture in tests/unit/test_tracr_conversion.py hand-writes residual_labels containing "reverse:1" etc., a shape real Tracr never emits, which is why the tests pass while the demo doesn't run.

Three small changes, one concept (Tracr labels are auto-numbered; ask the expression for its label instead of hardcoding its name):

  1. Demo cell 11: output_label="reverse"output_label=reverse.label, plus a sentence in the preceding markdown cell explaining the auto-numbering (and that omitting output_label also works via infer_tracr_output_label).
  2. Test fixture: "reverse:*""reverse_1:*" in _fake_tracr_model() and the matching output_label call sites, so the mock matches what the compiler actually produces. The ambiguity test still has two candidate prefixes (aggregate_1, reverse_1) and all positional assertions are unchanged.
  3. Docstring: the infer_tracr_output_label example "reverse:3""reverse_1:3".

Fixes #481.

Type of change

  • Bug fix (non-breaking change which fixes an issue)

Validation

Run locally against this branch (Python 3.12, torch CPU, real tracr from google-deepmind/tracr — note the Tracr stack needs jax<0.11, since dm-haiku still references the removed jax.core.DropVar):

  • pytest tests/unit/test_tracr_conversion.py — 5 passed.
  • The demo's code path executed end-to-end as a script (compile the demo's RASP program → TransformerBridge.boot_native → load state dict → run): with output_label="reverse" it raises as above; with output_label=reverse.label and with output_label omitted, the Bridge decoding matches Tracr (['BOS', 3, 2, 1]) and every layer's attn/MLP hook_out matches Tracr's layer_outputs (np.isclose(...).all() per layer).

Prepared with AI assistance and human-reviewed; all validation runs above are real runs on the contributor's machine.

The Tracr demo passes output_label="reverse", but Tracr builds labels as
f"{name}_{unique_id}", so the compiled expression's residual labels are
reverse_1:* and the demo raises ValueError at the state-dict cell. Pass
reverse.label instead of hardcoding the name, and correct the same wrong
label shape in the unit-test fixture (real Tracr never emits "reverse:1")
and in the infer_tracr_output_label docstring example.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@jlarson4
jlarson4 merged commit 17aca31 into TransformerLensOrg:dev Jul 21, 2026
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants