Repository navigation
v4.1.0 — reliability repair after the 4.0.0 external review - #35
Merged
Merged
Conversation
…bsence is not fabrication
- A1: report existence, identifier, metadata (title/authors/year) and retraction
separately (report schema_version 2, tool 2.0.0). A resolving DOI no longer clears
an invented co-author: unsupported cited authors make the entry PAC
(flag UNSUPPORTED_AUTHOR). "et al." / BibTeX "and others" are honoured; missing
record metadata is unknown / not_checked, never agreement.
- A2: replace substring surname matching ("Li" matched "Williams") with token-wise
comparison over explicit variants (accents, LaTeX, German umlauts, particles,
compound surnames, organisations by name or acronym).
- A3: index absence is NOT_FOUND (REVIEW_REQUIRED at run level), an unregistered
DOI / missing arXiv ID is ID_INVALID; TF is no longer emitted by the script.
Malformed DOIs are reported.
- A4: one unreachable host no longer stops the source loop; it is recorded as not
checked and paused for 60 s.
- A5: retraction status is flagged / not_flagged_in_checked_sources / unknown with
sources and time; `retracted` is null when unknown.
- citation-audit workflow: ID_INVALID verdict, tri-state retraction, NOT_FOUND no
longer relabelled TF, retraction_unknown counted.
Tests that asserted TF for index absence now assert NOT_FOUND under the same
evidence bar.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QiZcawE5UVAWiwbgqCjgbn
…es inherited font sizes - C1: every result has status valid / invalid / blocked / partial and a checks list (name, required, state, detail); `valid` is true only for status valid. A missing required parser, an unsupported format (.ppt, unknown), a missing file or a missing pdflatex is blocked, never a pass. Stdlib structural checks (PDF signature, OOXML package) make a garbage file invalid even without parsers. Exit codes 0/1/2/3; --json output. - C2: PPTX font sizes are resolved through run, paragraph, shape list style, layout and master placeholders, master text styles and the presentation default, scaled by autofit; unresolved runs make the check incomplete. Group shapes and table cells are traversed; the footer band uses absolute positions. - Dev group gains python-pptx, pypdf and pymupdf (pinned <2: 2.1 pulls onnxruntime) so these tests run instead of skipping. SECURITY_SCAN.md regenerated for the OOXML namespace domain. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QiZcawE5UVAWiwbgqCjgbn
…; add activation set - D1: run_benchmark.py no longer keeps the first of two answers for one item: a conflicting duplicate is unresolved_conflict (never correct, case incomplete), identical duplicates collapse with a count, unknown and missing ids are reported. Replaces the test that enshrined first-answer-wins. - D2: evidence text is kept per item; label accuracy is reported separately from two deterministic evidence checks (quote_in_fixture, quote_in_item_region); anything else is needs_review, with --export-evidence-review for a human pass. Scoring schema 2; docs mark the published numbers as schema 1 (not directly comparable). - E1: scripts/activation_eval.py with 40 hand-written cases (benchmarks/activation): clear, ambiguous, distractor and unavailable-skill cases in English and Turkish, over install profiles computed from the plugin manifests, recording the candidates shown. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QiZcawE5UVAWiwbgqCjgbn
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QiZcawE5UVAWiwbgqCjgbn
…e dedup, relevance default, date filters - B1: query terms keep every script (Turkish, Greek, CJK); nothing is transliterated. - B2: explicit syntax (--terms, quotes, AND/OR/NOT, parentheses) is parsed and translated per index: exact for Europe PMC and arXiv, labelled approximate for Crossref / Semantic Scholar, skipped with a reason when an exclusion cannot be expressed. Every result reports query_translation; malformed --terms is an error. - B3: records merge on DOI, then arXiv/PMID/PMCID, then full Unicode title with compatible year and first author; never on an empty key. Preprint and published versions with different DOIs stay separate (related_versions); merged_from provenance. - B4: default ranking is relevance; --rank cited orders by one declared index (--cite-index) instead of mixing counts across indexes. - B5: --since / --until translated per index and re-checked client-side; undated records are kept and marked; "recent" wording is disclosed, never a hidden filter. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QiZcawE5UVAWiwbgqCjgbn
NOT_FOUND / ID_INVALID / REVIEW_REQUIRED and tri-state retraction status across the verifier SKILL.md, commands, examples, integrity schemas (new optional fields; v1 reports still validate), handoff schemas, pipeline and research agents, the citation-audit reference and docs/integrity.md. Eval rubric text that required index absence to be called TF now requires NOT_FOUND with the same strictness (no case or assertion removed). Fixture annotations relabelled; catalogs regenerated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QiZcawE5UVAWiwbgqCjgbn
…dimensions, artifact checks - D3: --regrade aggregates every requested file; malformed / wrong-kind files are reported, an unavailable judge is blocked, arms without saved responses are partial. Exit codes match --behavioral (0 all PASS, 1 FAIL, 2 blocked, 3 partial); --calibrate-judge follows the same scheme. Regrading a collect-only file now judges its trigger assertions. - D4: --judge applies to --behavioral (default substance) and is recorded with its prompt hash. Both arms are graded on identical substantive criteria; routing and skill-vocabulary assertions are separate dimensions (vocabulary never decides a case). --audit-rubrics lists rubric text that requires internal names. Five hand-labelled calibration pairs added, pending maintainer review. - D5: scripts/artifact_checks.py checks declared output files (exists, parses, structure, author-supplied numeric values, render-level) before the workspace is deleted, and stores copies with sha256. Added to four eval cases (matplotlib, eda, citation-mgmt). Results schema_version 2 (judge, runner, hashes, dimensions, artifacts); v1 files remain readable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QiZcawE5UVAWiwbgqCjgbn
…regate_results.py) Validates v1/v2 behavioral and regrade records, applies explicit case-selection rules, reports excluded / blocked / ungraded cases with denominators, keeps v1 and v2 methodologies apart and routing apart from substance, and renders deterministic Markdown/JSON. Synthetic example records under docs/examples/eval-records. docs/behavioral-results.md notes that its raw records are not in the repository, so the published totals cannot be reconstructed from this checkout (no figure changed). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QiZcawE5UVAWiwbgqCjgbn
…ke evidence - docs/reliability-review-4.0.0.md maps the 20 review findings to classification, fixes, evidence (fail-before tests, live runs) and remaining limits. - CHANGELOG [Unreleased]: behaviour and compatibility changes; no performance claims. - docs/evals.md: live judge calibration 2026-11-05 (substance 13/13, literal 12/13). - docs/examples/live-2026-11-05: live verifier input/report and the live claim-stress-test workflow payload and report. - README (EN/TR): what the citation verifier now distinguishes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QiZcawE5UVAWiwbgqCjgbn
…d; add live-validation harness - E2 harness: minimal synthetic fixtures for all seven workflows (args, expect, mock scenario) and scripts/check_workflow_run.py, which checks a saved payload and the files on disk: schema, status vocabulary, coverage arithmetic, per-workflow invariants, that incomplete work is visible, that outputs exist and parse, and that the report's own status line matches the payload. It refuses to label dry-run output as live. - Found by the harness: in rebuttal and systematic-review-screening a companion file that failed its read-back made the payload partial while the report on disk still said COMPLETE. Both workflows now amend the status line and confirm it with a second read-back; if that fails, status_reasons say the file is stale and the payload is authoritative. Regression tests failed before the change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QiZcawE5UVAWiwbgqCjgbn
- citation-audit run live on the synthetic fixture (checker pass, all answer-key findings met; payload, report and verdict saved); claim-stress-test live run recorded as hand-checked. Five workflows remain dry-run only. - README (EN/TR): workflow and activation status rows now state what was run live; the 4.0.0 skilled-vs-bare figures are marked as literal-judge results whose raw records are not in the repository. No figure changed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QiZcawE5UVAWiwbgqCjgbn
…review Version bumped in pyproject.toml and propagated by gen_marketplace / gen_catalog; CITATION.cff and the README citation blocks updated; CHANGELOG section [4.1.0]. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QiZcawE5UVAWiwbgqCjgbn
…in 4.1.0 scientific-slides 1.2.0 (validator behaviour); deep-research 2.4.4, paper-writer 2.6.4, research-pipeline 2.7.3, literature-review 1.1.3 (agent/reference docs aligned with verifier report schema 2). Catalogs regenerated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QiZcawE5UVAWiwbgqCjgbn
AlterLab-IEU
marked this pull request as ready for review
November 5, 2026 16:16
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
An external review of v4.0.0 (
6fa2616) reported 20 findings. All 20 were checked on that commit. 19 reproduced as defects or evaluation gaps and are fixed, each with a regression test that failed before the fix. The 20th (live validation of the seven workflows) is only partly done: 2 of 7 were run live, and the other 5 have mock-runtime evidence only. No skills were added or removed. The full mapping of findings to fixes, evidence and remaining limits is indocs/reliability-review-4.0.0.md, and the user-facing changes are inCHANGELOG.mdunder 4.1.0.Citation verifier (
alterlab-citation-verifier, report schema 2)PACand names the unsupported author.Lino longer matchesWilliams.NOT_FOUND(run verdictREVIEW_REQUIRED), not as fabrication. An unregistered DOI isID_INVALID. The script no longer emitsTF.Research lookup (
alterlab-research-lookup)--since/--untiloptions filter by publication date.Slide validator (
alterlab-scientific-slides)Eval tooling
--regradeexit codes now reflect every requested file.--judge(literal or substance) applies to--behavioralruns and is recorded with each result.aggregate_results.py,activation_eval.py,check_workflow_run.py, plus fixtures for all seven workflows.rebuttalandsystematic-review-screeningcould leaveStatus: COMPLETEin a report after the run turned partial. This is fixed.Compatibility (why this is a minor release with schema changes):
TFno longer appears in verifier or citation-audit output. UseNOT_FOUNDandID_INVALIDinstead.retractedcan now benull, meaning unknown.--rank autois removed.validhave changed.run_evals.pynow writesschema_version2.--behavioralissubstance. New runs are not directly comparable with the 4.0.0 results, which used the literal judge.Live validation (2026-11-05, Claude Code CLI login, no API key)
PAC.citation-auditworkflow: passed the output checker on a synthetic fixture.claim-stress-testworkflow: checked by hand.Raw records are in
docs/examples/live-2026-11-05/andbenchmarks/activation/results/.Not established
Type of change
feat: add <skill-name>)improve: <skill-name> — <what changed>)fix: <skill-name> — <what was wrong>)Authoring checklist
uv run python scripts/run_evals.py --strict: 241 clean, 1,598 cases. No cases or assertions were removed. Rubric text that required index absence to be calledTFnow requiresNOT_FOUND, with the same strictness. Artifact checks were added to 4 cases.uv run python scripts/audit_skills.py(0 errors, 0 warnings) anduv run pytest tests/(5,820 passed, 23 skipped; the baseline was 5,428 / 23) pass cleanly.--checkpasses; the release build verified 260 archives;claude plugin validate .passes.metadata.versionfor every skill whose shipped content changed.Generated by Claude Code