Skip to content

v4.1.0 — reliability repair after the 4.0.0 external review - #35

Merged
AlterLab-IEU merged 13 commits into
mainfrom
claude/ecstatic-albattani-6yk715
Nov 5, 2026
Merged

AlterLab-IEU merged 13 commits into
mainfrom
claude/ecstatic-albattani-6yk715

Conversation

@AlterLab-IEU

Copy link
Copy Markdown
Owner

Summary

An external review of v4.0.0 (6fa2616) reported 20 findings. All 20 were checked on that commit. 19 reproduced as defects or evaluation gaps and are fixed, each with a regression test that failed before the fix. The 20th (live validation of the seven workflows) is only partly done: 2 of 7 were run live, and the other 5 have mock-runtime evidence only. No skills were added or removed. The full mapping of findings to fixes, evidence and remaining limits is in docs/reliability-review-4.0.0.md, and the user-facing changes are in CHANGELOG.md under 4.1.0.

Citation verifier (alterlab-citation-verifier, report schema 2)

  • Existence, identifier, metadata and retraction are now reported separately. A DOI that resolves no longer clears an invented co-author: the entry becomes PAC and names the unsupported author.
  • Surname matching is token-wise, so Li no longer matches Williams.
  • A reference missing from the indexes is reported as NOT_FOUND (run verdict REVIEW_REQUIRED), not as fabrication. An unregistered DOI is ID_INVALID. The script no longer emits TF.
  • If one source is unreachable, the remaining sources are still checked.
  • Retraction status has three values: flagged, not flagged in the checked sources, or unknown. The citation-audit workflow follows the same rules.

Research lookup (alterlab-research-lookup)

  • Queries in any script are kept intact (Turkish, Greek, CJK).
  • Boolean queries are translated correctly for each index, or labelled as approximate when an index can't express them.
  • Records are no longer merged on an empty or truncated key.
  • The default order is relevance.
  • New --since / --until options filter by publication date.

Slide validator (alterlab-scientific-slides)

  • Each run now reports one of four statuses: valid, invalid, blocked or partial. A missing parser or an unsupported format can no longer produce a pass.
  • Font sizes inherited from layouts and masters are checked.
  • Grouped shapes and table cells are checked.

Eval tooling

  • Benchmark: two conflicting answers to one item can no longer score as correct.
  • Benchmark evidence is kept and checked separately from the label.
  • --regrade exit codes now reflect every requested file.
  • --judge (literal or substance) applies to --behavioral runs and is recorded with each result.
  • Substance, routing and skill-name vocabulary are scored as separate dimensions.
  • Output files are checked before the workspace is deleted.
  • New scripts: aggregate_results.py, activation_eval.py, check_workflow_run.py, plus fixtures for all seven workflows.
  • The workflow harness found that rebuttal and systematic-review-screening could leave Status: COMPLETE in a report after the run turned partial. This is fixed.

Compatibility (why this is a minor release with schema changes):

  • TF no longer appears in verifier or citation-audit output. Use NOT_FOUND and ID_INVALID instead.
  • retracted can now be null, meaning unknown.
  • Research lookup defaults to relevance order, and --rank auto is removed.
  • The slide validator's exit codes and the meaning of valid have changed.
  • run_evals.py now writes schema_version 2.
  • The default rubric judge for --behavioral is substance. New runs are not directly comparable with the 4.0.0 results, which used the literal judge.

Live validation (2026-11-05, Claude Code CLI login, no API key)

  • Citation verifier against the real APIs: the review's exact invented-co-author case now returns PAC.
  • Research lookup smoke queries returned correct results.
  • Activation set: 40 cases, one vote each, 39/40 correct.
  • Judge calibration: substance agreed with 13/13 hand labels, literal with 12/13.
  • citation-audit workflow: passed the output checker on a synthetic fixture.
  • claim-stress-test workflow: checked by hand.

Raw records are in docs/examples/live-2026-11-05/ and benchmarks/activation/results/.

Not established

  • No behavioural sweep or benchmark was re-run.
  • The 4.0.0 "90% vs 52%" figures can't be re-derived because their raw records aren't in the repo. The README now says so; no figure was changed.
  • Five workflows have mock-runtime evidence only.
  • Hand labels added in this PR need maintainer review: the relevance fixture, the 40 activation cases and the 5 calibration pairs.

Type of change

  • New skill (feat: add <skill-name>)
  • Skill improvement (improve: <skill-name> — <what changed>)
  • Bug fix (fix: <skill-name> — <what was wrong>)
  • Docs / chore

Authoring checklist

  • Evals shipped — uv run python scripts/run_evals.py --strict: 241 clean, 1,598 cases. No cases or assertions were removed. Rubric text that required index absence to be called TF now requires NOT_FOUND, with the same strictness. Artifact checks were added to 4 cases.
  • Body ≤500 lines
  • Description ≤1024 chars
  • Frontmatter valid
  • No fabrication — live results are labelled live; mock-runtime and unit-test evidence is labelled as such.
  • Disambiguated — not applicable (no new skills).
  • uv run python scripts/audit_skills.py (0 errors, 0 warnings) and uv run pytest tests/ (5,820 passed, 23 skipped; the baseline was 5,428 / 23) pass cleanly.
  • Marketplace regenerated — every generator --check passes; the release build verified 260 archives; claude plugin validate . passes.
  • Version bumped — 4.1.0, plus metadata.version for every skill whose shipped content changed.

Generated by Claude Code

claude added 13 commits November 5, 2026 09:52
…bsence is not fabrication

- A1: report existence, identifier, metadata (title/authors/year) and retraction
  separately (report schema_version 2, tool 2.0.0). A resolving DOI no longer clears
  an invented co-author: unsupported cited authors make the entry PAC
  (flag UNSUPPORTED_AUTHOR). "et al." / BibTeX "and others" are honoured; missing
  record metadata is unknown / not_checked, never agreement.
- A2: replace substring surname matching ("Li" matched "Williams") with token-wise
  comparison over explicit variants (accents, LaTeX, German umlauts, particles,
  compound surnames, organisations by name or acronym).
- A3: index absence is NOT_FOUND (REVIEW_REQUIRED at run level), an unregistered
  DOI / missing arXiv ID is ID_INVALID; TF is no longer emitted by the script.
  Malformed DOIs are reported.
- A4: one unreachable host no longer stops the source loop; it is recorded as not
  checked and paused for 60 s.
- A5: retraction status is flagged / not_flagged_in_checked_sources / unknown with
  sources and time; `retracted` is null when unknown.
- citation-audit workflow: ID_INVALID verdict, tri-state retraction, NOT_FOUND no
  longer relabelled TF, retraction_unknown counted.

Tests that asserted TF for index absence now assert NOT_FOUND under the same
evidence bar.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QiZcawE5UVAWiwbgqCjgbn
…es inherited font sizes

- C1: every result has status valid / invalid / blocked / partial and a checks list
  (name, required, state, detail); `valid` is true only for status valid. A missing
  required parser, an unsupported format (.ppt, unknown), a missing file or a missing
  pdflatex is blocked, never a pass. Stdlib structural checks (PDF signature, OOXML
  package) make a garbage file invalid even without parsers. Exit codes 0/1/2/3;
  --json output.
- C2: PPTX font sizes are resolved through run, paragraph, shape list style, layout
  and master placeholders, master text styles and the presentation default, scaled by
  autofit; unresolved runs make the check incomplete. Group shapes and table cells are
  traversed; the footer band uses absolute positions.
- Dev group gains python-pptx, pypdf and pymupdf (pinned <2: 2.1 pulls onnxruntime)
  so these tests run instead of skipping. SECURITY_SCAN.md regenerated for the OOXML
  namespace domain.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QiZcawE5UVAWiwbgqCjgbn
…; add activation set

- D1: run_benchmark.py no longer keeps the first of two answers for one item: a
  conflicting duplicate is unresolved_conflict (never correct, case incomplete),
  identical duplicates collapse with a count, unknown and missing ids are reported.
  Replaces the test that enshrined first-answer-wins.
- D2: evidence text is kept per item; label accuracy is reported separately from two
  deterministic evidence checks (quote_in_fixture, quote_in_item_region); anything else
  is needs_review, with --export-evidence-review for a human pass. Scoring schema 2;
  docs mark the published numbers as schema 1 (not directly comparable).
- E1: scripts/activation_eval.py with 40 hand-written cases (benchmarks/activation):
  clear, ambiguous, distractor and unavailable-skill cases in English and Turkish, over
  install profiles computed from the plugin manifests, recording the candidates shown.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QiZcawE5UVAWiwbgqCjgbn
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QiZcawE5UVAWiwbgqCjgbn
…e dedup, relevance default, date filters

- B1: query terms keep every script (Turkish, Greek, CJK); nothing is transliterated.
- B2: explicit syntax (--terms, quotes, AND/OR/NOT, parentheses) is parsed and
  translated per index: exact for Europe PMC and arXiv, labelled approximate for
  Crossref / Semantic Scholar, skipped with a reason when an exclusion cannot be
  expressed. Every result reports query_translation; malformed --terms is an error.
- B3: records merge on DOI, then arXiv/PMID/PMCID, then full Unicode title with
  compatible year and first author; never on an empty key. Preprint and published
  versions with different DOIs stay separate (related_versions); merged_from provenance.
- B4: default ranking is relevance; --rank cited orders by one declared index
  (--cite-index) instead of mixing counts across indexes.
- B5: --since / --until translated per index and re-checked client-side; undated
  records are kept and marked; "recent" wording is disclosed, never a hidden filter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QiZcawE5UVAWiwbgqCjgbn
NOT_FOUND / ID_INVALID / REVIEW_REQUIRED and tri-state retraction status across the
verifier SKILL.md, commands, examples, integrity schemas (new optional fields; v1
reports still validate), handoff schemas, pipeline and research agents, the
citation-audit reference and docs/integrity.md. Eval rubric text that required
index absence to be called TF now requires NOT_FOUND with the same strictness (no
case or assertion removed). Fixture annotations relabelled; catalogs regenerated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QiZcawE5UVAWiwbgqCjgbn
…dimensions, artifact checks

- D3: --regrade aggregates every requested file; malformed / wrong-kind files are
  reported, an unavailable judge is blocked, arms without saved responses are partial.
  Exit codes match --behavioral (0 all PASS, 1 FAIL, 2 blocked, 3 partial);
  --calibrate-judge follows the same scheme. Regrading a collect-only file now judges
  its trigger assertions.
- D4: --judge applies to --behavioral (default substance) and is recorded with its
  prompt hash. Both arms are graded on identical substantive criteria; routing and
  skill-vocabulary assertions are separate dimensions (vocabulary never decides a
  case). --audit-rubrics lists rubric text that requires internal names. Five
  hand-labelled calibration pairs added, pending maintainer review.
- D5: scripts/artifact_checks.py checks declared output files (exists, parses,
  structure, author-supplied numeric values, render-level) before the workspace is
  deleted, and stores copies with sha256. Added to four eval cases (matplotlib, eda,
  citation-mgmt). Results schema_version 2 (judge, runner, hashes, dimensions,
  artifacts); v1 files remain readable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QiZcawE5UVAWiwbgqCjgbn
…regate_results.py)

Validates v1/v2 behavioral and regrade records, applies explicit case-selection
rules, reports excluded / blocked / ungraded cases with denominators, keeps v1 and v2
methodologies apart and routing apart from substance, and renders deterministic
Markdown/JSON. Synthetic example records under docs/examples/eval-records.
docs/behavioral-results.md notes that its raw records are not in the repository, so
the published totals cannot be reconstructed from this checkout (no figure changed).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QiZcawE5UVAWiwbgqCjgbn
…ke evidence

- docs/reliability-review-4.0.0.md maps the 20 review findings to classification,
  fixes, evidence (fail-before tests, live runs) and remaining limits.
- CHANGELOG [Unreleased]: behaviour and compatibility changes; no performance claims.
- docs/evals.md: live judge calibration 2026-11-05 (substance 13/13, literal 12/13).
- docs/examples/live-2026-11-05: live verifier input/report and the live
  claim-stress-test workflow payload and report.
- README (EN/TR): what the citation verifier now distinguishes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QiZcawE5UVAWiwbgqCjgbn
…d; add live-validation harness

- E2 harness: minimal synthetic fixtures for all seven workflows (args, expect,
  mock scenario) and scripts/check_workflow_run.py, which checks a saved payload and
  the files on disk: schema, status vocabulary, coverage arithmetic, per-workflow
  invariants, that incomplete work is visible, that outputs exist and parse, and that
  the report's own status line matches the payload. It refuses to label dry-run output
  as live.
- Found by the harness: in rebuttal and systematic-review-screening a companion file
  that failed its read-back made the payload partial while the report on disk still
  said COMPLETE. Both workflows now amend the status line and confirm it with a second
  read-back; if that fails, status_reasons say the file is stale and the payload is
  authoritative. Regression tests failed before the change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QiZcawE5UVAWiwbgqCjgbn
- citation-audit run live on the synthetic fixture (checker pass, all answer-key
  findings met; payload, report and verdict saved); claim-stress-test live run
  recorded as hand-checked. Five workflows remain dry-run only.
- README (EN/TR): workflow and activation status rows now state what was run live;
  the 4.0.0 skilled-vs-bare figures are marked as literal-judge results whose raw
  records are not in the repository. No figure changed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QiZcawE5UVAWiwbgqCjgbn
…review

Version bumped in pyproject.toml and propagated by gen_marketplace / gen_catalog;
CITATION.cff and the README citation blocks updated; CHANGELOG section [4.1.0].

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QiZcawE5UVAWiwbgqCjgbn
…in 4.1.0

scientific-slides 1.2.0 (validator behaviour); deep-research 2.4.4, paper-writer
2.6.4, research-pipeline 2.7.3, literature-review 1.1.3 (agent/reference docs
aligned with verifier report schema 2). Catalogs regenerated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QiZcawE5UVAWiwbgqCjgbn
@AlterLab-IEU
AlterLab-IEU marked this pull request as ready for review November 5, 2026 16:16
@AlterLab-IEU
AlterLab-IEU merged commit 3ab6d03 into main Nov 5, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants