Skip to content

Clarify epistemic-transfer study conclusion - #40

Open
optaktlux wants to merge 2 commits into
masterfrom
docs/clarify-transfer-conclusion
Open

Clarify epistemic-transfer study conclusion#40
optaktlux wants to merge 2 commits into
masterfrom
docs/clarify-transfer-conclusion

Conversation

@optaktlux

Copy link
Copy Markdown
Collaborator

What changed

  • distinguish failure to beat response texture from failure of the identification method
  • record above-chance non-texture trajectory and operator results
  • state that deliberate texture removal remains untested
  • preserve the stopping decision and open-set failure

Verification

  • git diff --check
  • go test ./...

Fixes #39

@optaktlux

Copy link
Copy Markdown
Collaborator Author

Hostile review (own PR; GitHub forbids request-changes)

This PR is mostly a legitimate narrowing. It is also doing one thing it should not do.

Blocker: do not rewrite the independent review

topology-study/REVIEW.md is the record of an Opus review that already happened. Changing

  • "whether the negative conclusion was stronger than the evidence"
  • "this does not rescue the uniqueness claim"
  • "negative verdict"

into a "comparative" verdict, and appending "The review did not test deliberate response-texture removal," puts words in a reviewer who is not on this PR. Residual caution 1 is especially bad: the original said different channels do not rescue uniqueness. The new text says they support describing Lumen as a different mechanism. That is a different claim. If the interpretation of the study changed, add a dated note. Do not edit the review until it says the new thing.

Goalposts

RESEARCH.md originally: "Abandon the model-identification claim if any of these occur." After the study missed superiority and open-set, those criteria become "uniquely stronger or open-set-capable." That is closer to what the numbers support, but it is still a post-hoc rewrite of the decision rule. Keep the original criteria as historical, then state what failed. Don't silently retitle the rule the study already ran under.

The README change from "Falsification" to "Comparative decision criteria" has the same smell. The listed bullets are unchanged; only the heading and preamble moved.

The "different method" claim is true and weaker than it sounds

Above-chance closed-set attribution from non-texture features is real (64.84 / 58.59 / 28.91 vs 12.5% chance). Operator summaries at 28.91% are 2.3× chance on eight classes. That is a weak standalone fingerprint. Lead with probability/graph trajectories, not operators, if the point is "signal remains after dropping texture."

Texture-removal robustness is untested. Saying so is correct. Using that untested counterfactual as the reason the method still matters is special pleading. The honest remainder is: closed-set epistemic-transition signal exists; it did not beat hashed texture; open-set failed; style neutralization was not run.

What survives

  • Recording the non-texture accuracies in the verdict was the actual correction.
  • Stopping / no-tune-against-this-set is preserved.
  • "Failure to demonstrate superiority, not proof of equivalence" was already in RESULTS.md and should stay.

Fix REVIEW.md (revert the reviewer's sentences; add an addendum if needed) and I would approve.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Clarify epistemic-transfer study conclusion

1 participant