Skip to content

4.0.0 — reliability repair: truthful evals, keyless-first skills, every skill evaluated live - #32

Merged
AlterLab-IEU merged 276 commits into
mainfrom
claude/intelligent-galileo-wqlz7w
Nov 4, 2026
Merged

AlterLab-IEU merged 276 commits into
mainfrom
claude/intelligent-galileo-wqlz7w

Conversation

@AlterLab-IEU

@AlterLab-IEU AlterLab-IEU commented Oct 23, 2026 •

Copy link
Copy Markdown
Owner

Summary

This branch prepares 4.0.0. It repairs reliability across the suite, removes every dependency on logging in to a third-party service, and evaluates every skill live with and without the skill loaded. No skills are added. alterlab-openalex is removed, so the suite has 241 skills. Version 3.0.1 → 4.0.0; the CHANGELOG [4.0.0] section says why this is a major version. No tag or release has been made.

Live behavioral evaluation (results page: docs/behavioral-results.md)

Each eval case was answered once with the skill and once without it, through the Claude Code CLI login with no API key, from 2026-10-15 to 2026-11-04.

Skills evaluated 241 / 241
Cases 1,598 (1,532 graded)
Not runnable here 50 (need a key, connector, host or program the runner lacks)
Infrastructure errors or refusals 16 (heavy GPU/CPU jobs past the runner's limits)
Passed with the skill 1,380 / 1,532 (90.1%)
Passed without it, same cases 804 / 1,532 (52.5%)
Skill helped / hurt 571 / 2
  • Which run counts: every counted case's final run was made at the skill's current content (its files outside evals/ match this release). Cases of the removed skill and of renamed evals are not counted.
  • Isolation: all counted runs recorded namespace confinement and a clean checkout.
  • First pass over hurt cases: all 32 cases the skill made worse after the last full round were read by hand. The fixes:
    • 11 routing gaps;
    • 2 timeouts that were skill defects (chembl paging, biorxiv author search);
    • the alpha-vantage contact prompt.
  • Second pass (2026-11-04): the 27 cases still hurt after the v4.0.0 push were re-read, and the cause fixed where it lay in the skill. All 24 changed skills were re-run in full, and 25 of the 27 now pass. The fixes:
    • hand-off scope in 7 skills;
    • answers that say how the result was obtained (call, endpoint, files, date) in 11;
    • script and instruction defects: datacommons' coordinate resolve, the scientific-slides overflow validator, matchms matching, a pdb similarity command, scvi-tools' Python pin;
    • one pubmed eval that used an unregistered DOI.
  • Still hurt: 2 cases, an omero near-miss and a measurement-gate grader disagreement.
  • Limits:
    • grading reads only the final message;
    • each case ran once;
    • a Claude model is the judge;
    • the skilled answer faces more checks than the bare one;
    • the activation harness and the dynamic workflows were not run live;
    • claude.ai upload is untested;
    • 7 GPL/LGPL skills still need a licensing decision.

No login to any third-party service

  • OpenAlex removed: it now meters keyless use at $0.10/day per IP, so real use needs an account. alterlab-openalex and the bundled openalex and zotero MCP servers are removed, along with their config prompts.
  • Replacements:
    • citation-verifier: Crossref / Semantic Scholar / Europe PMC / arXiv, plus Open Library for books. A reference with no DOI is called fabricated only when Crossref plus another index actually answered. Headings and prose are skipped, never called fabricated.
    • citation-graph: OpenCitations + Semantic Scholar.
    • research-lookup: Crossref / Europe PMC / arXiv, with Semantic Scholar best-effort.
  • Login audit: every host on a default path was probed, and four more problems were fixed:
    • Semantic Scholar search is now best-effort;
    • geomaster reads ERA5 from Google's public ARCO-ERA5 store, read anonymously;
    • pyhealth defaults to the open MIMIC-IV / eICU demos;
    • alpha-vantage fetches SEC fundamentals without a contact e-mail.

Behavioral evals

  • Exit codes: run_evals.py --behavioral exits 0 only when every selected case passed. Infrastructure errors, refusals and inconclusive verdicts are reported as blocked or partial.
  • Refusals: acceptable-use refusals are recorded as REFUSED.
  • Saved answers: e-mail addresses are redacted.
  • Isolation: answer arms run in a private mount namespace, with a tripwire on repo changes and leftover-job cleanup.
  • New eval options: requires (env:/cmd:/net:/mcp:), alternatives in output_contains, directory fixtures, --collect-only, --wait-on-limit, --regrade, a substance judge, --case.
  • New tools: scripts/eval_sweep.sh, scripts/defect_scan.py.

Keyless-first, enforced

  • Keyless: every skill does its core job without an external API key, account or licence. Credentials only add something and are declared in metadata.credentials.
  • Enforced: scripts/check_keyless.py and tests/test_keyless.py fail the suite on a regression.

Other changes

  • No maintainer e-mail: removed from code, docs and request headers. A contact address is opt-in via ALTERLAB_CONTACT_EMAIL; older main history still contains the address.
  • Hand-off rule: every skill names the better sibling and stops when a request belongs elsewhere.
  • Live-sweep fixes: each fix is in its commit message and in CHANGELOG [4.0.0].
  • README: the English and Turkish "What has been checked" tables reflect the live evaluation.

Note: the GitHub repository's About text still says "242 evaluated academic Claude/agent skills". It is a repository setting, not a file, so it needs updating there.

Type of change

  • New skill (feat: add <skill-name>)
  • Skill improvement (improve: <skill-name> — <what changed>)
  • Bug fix (fix: <skill-name> — <what was wrong>)
  • Docs / chore

Authoring checklist

  • Evals shipped: all 241 skills pass run_evals.py --strict with 0 issues (1,598 cases). Each has at least 3 triggering cases that need no key.
  • Body ≤500 lines: audit_skills.py reports 0 errors and 0 warnings across 241 skills.
  • Description ≤1024 chars: check_spec.py reports 0 violations, including compatibility ≤ 500.
  • Frontmatter valid: the audit is clean, and metadata.credentials is registered.
  • No fabrication: endpoints and identifiers were checked live, and each fix commit records how it was verified.
  • Disambiguated: routing rows are in place, plus the hand-off rule.
  • ruff, compileall and pytest tests/ pass (5428 passed, 23 skipped). Every generator --check, check_keyless, check_claudeai_compat, claude plugin validate . and build_bundles.py release --verify (260 archives at 4.0.0) also pass.
  • Marketplace regenerated: gen_marketplace.py --check is clean.
  • Version bumped for modified skills: done for every skill changed in the later rounds, but not for the hand-off-rule-only edits.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs

claude added 30 commits October 15, 2026 07:40
Eight hand-labelled rubric/response pairs (correct-without-jargon, jargon-reciting wrong
answer, request for missing input, confident fabrication, missing/given sibling routing,
wrong/right number). Live on 2026-10-15: literal judge 7/8, substance judge 8/8.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
…cument

58 skills told the model to run `uv run python …` (or python3 / python) while their own
allowed-tools did not permit it, so in Claude Code every documented script run needed a
manual approval and failed outright in print mode. Each now also allows Bash(uv run:*)
(or the python form it uses). Install and download commands (uv pip, conda, docker, wget)
stay unapproved on purpose.

Checked live: a skill declaring `allowed-tools: Read Bash(uv run:*)` ran
`uv run --no-project python -c …` without a prompt; the same skill without the rule was
denied. run_evals.py and gen_catalog.py split allowed-tools on whitespace, which broke such
rules in two; both now keep a parenthesised rule whole.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
A bilingual suite cannot pin a legal citation to one language: a correct Turkish answer
writes "KVKK m. 28" and failed an assertion requiring "Art. 28". A list now passes when
any one entry occurs. Schema, static validator, live grading and --regrade all accept it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
…g table

In the baseline sweep the skills knew their routes (every one has a Route-to table) but
out-of-scope answers often never named the sibling skill: they asked for missing input
first, or answered from general knowledge. Each routing table now ends with a rule: name
the target skill by its exact id in the opening line, stop applying this skill's procedure,
do only the in-scope part of a mixed request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
generate_pdf.py, search_databases.py and verify_citations.py took sys.argv[1] as a file,
so `--help` failed with "file not found: --help". They now print usage and exit 0.
tests/test_script_help.py runs --help on every stdlib-only skill CLI (81 scripts) and
fails on a non-zero exit or missing usage text; it fails on the unfixed scripts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
… always passed

kvkk-dmp and aperta pinned citations to one language ("Art. 28", "İlke-6"); a correct
Turkish answer writes "m. 28" / "İlke 6". results-transparency required the skill's own
phrase "every analysis"; equivalent phrasings now count. meta-analysis/heterogeneity
asserted the substring "I", which every response contains; it now needs I² (or I^2,
I2, I-squared).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
… supplied

"Here is my draft consent form", "here are five journals", "my öz and abstract", "my 1001
draft", "two Q3 articles and a few citations": the prompts named input that was not
there, so a correct answer had to ask for it and was scored FAIL. Each now ships the input
as a fixture:

- tr-research-ethics/consent-form-lint: an interview-study onam draft; the skill's linter
  finds exactly four gaps (date/version/page, 24-hour contact, withdrawal, no-coercion).
- trdizin/audit-venue-list-before-dossier: five real journals with ISSNs (four indexed and
  one, Toplum ve Bilim, out of TR Dizin since 2015, checked live 2026-10-15).
- akademik-tesvik/net30: an A1-field record whose score is 22.20 by tesvik_score.py, below
  the net-30 gate; the answer must state 22.2.
- tr-academic-style/trdizin-bilingual-abstract and tubitak-proposal/draft-yaygin-etki.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
…s text

An eval with two behavior assertions had both re-judged against the first spec. Labels
now consume the eval's behavior specs in order; a test spies on the judge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
… did not include

"Pool my 18 studies", "two RAs coded 30 interviews", "my 3-factor questionnaire", "my
dichotomous items", "my collaboration network", "22 cases": with no data a correct answer
can only ask for it. Each case now ships seeded synthetic data (generator kept out of the
repo; values below recomputed independently) and, where the answer is determined, a check:

- meta-analysis/pool-effect-sizes: 18 two-arm studies; DL random-effects g = 0.378
  [0.282, 0.474], I² = 26.7%.
- qualitative-analysis/two-coder-icr: 120 segments; the skill's icr.py gives
  Krippendorff alpha 0.764 (tentative band); the answer must contain .76.
- sem-psychometrics/cfa-fit (+ codebook): semopy fit CFI .978, TLI .968, RMSEA .042.
- sem-psychometrics/irt-2pl: 600 x 10 items; item7 discriminates worst (item-rest r .22).
- sna/brokerage-centrality: Selin has the highest betweenness (0.527) while three people
  tie on degree; the answer must name Selin.
- qca/fsqca-sufficiency: 22 cases of raw conditions (needs R's QCA, absent here).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
…ith, not pip install

The skills listed their Python dependencies but never said how to run without installing
them; in the sweep the model tried `uv pip install semopy` (needs approval, denied in print
mode) and gave up. Each analysis skill now gives a one-line recipe in a throwaway
environment, e.g. `uv run --python ">=3.11" --with "semopy>=2.3" --with pandas python cfa.py`,
which Bash(uv run:*) permits. Every recipe was resolved and imported live on 2026-10-15;
the pins came from that check: uv's default interpreter here was 3.9, Mesa 3.4+ needs
Python 3.12, svy needs 3.11, and `import mesa` 3.5.1 needs networkx (mesa[network]).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
…o work on

full-sem now ships a seeded 400-respondent survey (semopy: trust -> satisfaction .64,
satisfaction -> loyalty .52, CFI .993, RMSEA .027) and the answer must report those
paths. run-methods-pipeline ships a draft protocol whose causal claim rests on a one-wave
convenience survey; the orchestrator must stop it at a fail-closed gate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
… the 500-line body limit

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
During the priority sweep the bare arm of alterlab-link-health/residual-followup, holding
that skill's Write Edit Bash grants, found this repository on disk and edited 29 files in it
(left uncommitted; nothing pushed). A temporary working directory was the only isolation.

Answer arms are now confined in three layers:
- a private mount namespace per CLI call (Linux) that covers this checkout and all of its
  worktrees with an empty read-only tmpfs — probed at start, --isolation {auto,namespace,none};
- bare Write/Edit/MultiEdit/NotebookEdit grants rewritten to absolute rules for the arm's
  workspace, and environment variables pointing into the checkout dropped;
- a tripwire: the checkout is fingerprinted before the run and after every case; a change
  blocks the run (exit 2, 'isolation breach'), lists the paths, and reverts nothing.

Checked live: a real arm granted bare Write and Bash and told to write into the checkout was
denied by the Write tool and got 'Read-only file system' from the shell; no file appeared.
Tests: scoping, environment, a vandal fake that trips the breaker, and the namespace view.
The fake CLI's interpreter now lives outside the checkout and no real claude stays on PATH:
with the venv's python hidden by the namespace, sh had fallen through to the real CLI.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
Lets a case ship a small repository checkout (e.g. a docs repo with a link-checker config) so
an arm works on local input instead of going to look for a repository elsewhere.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
link-health (3): the prompts named a repository or CI log that was not there. That is what
sent the bare arm of residual-followup searching the disk for a repository to repair. Each
case now ships local input: a small our-docs checkout with one planted failure per tier and
its lychee log, a residual-error log whose regressions come from the first pass's own bulk
substitution, and a markdown-link-check run over a docs folder. New checks require the
per-tier classification and the regression to be caught.
paper-writer (3): reviewer comments (with a reviewer-vs-reviewer conflict), an APA manuscript
with four real references (IEEE numbering is checked), a completed paper for the abstract.
workflow-orchestration: the four chapter bibliographies the prompt mentions.
.lychee.toml skips evals/fixtures/: those files plant broken links and fabricated DOIs on
purpose.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
…this skill does not ship

`python scripts/thumbnail.py`, `inventory.py`, `rearrange.py` and `replace.py` belong to
Anthropic's separately installed pptx skill (removed from this suite in 1.1.0), but the text
read as if they were bundled here. SKILL.md and the two guides now name the dependency and
give the bundled fallback: export to PDF and review with scripts/pdf_to_images.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
In the after-fix sweep a deep-research arm left a process writing into its workspace after
the CLI exited; the workspace cleanup then raised 'Directory not empty' and the whole run
died without writing its results. Each CLI call now runs in its own session and its process
group is killed when the call returns or times out, and workspace cleanup ignores errors.
The same gap had left an orphaned arm running for 14 minutes after a sweep was stopped.
Test: a fake arm that starts a lingering writer; the run completes and the writer is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
…ing table

Same rule the 59 priority skills got after the sweep showed out-of-scope answers often never
named the sibling skill: name the target skill by its exact id in the opening line, stop
applying this skill's procedure, do only the in-scope part of a mixed request. Every skill in
the suite now carries it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
…he skill

They live in the HypoGeniC library's repository. Found by a scan for skill files that cite
bundled paths that do not exist; the other hits in the 183 untested skills point at named
sibling skills or upstream repositories and are correct as written.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
…ndpoint on record

- --collect-only: answer arms run and are saved in full, no judge is called; exit 0 when
  every arm answered. For defect sweeps (missing input, denied tools, crashed scripts), which
  need no judge, and for running on another model with judging done later via --regrade.
- 'requires' (env:NAME, cmd:NAME, net:HOST, mcp:NAME): a case that cannot be judged on this
  runner is NOT_RUN 'not runnable here', never FAIL; counted as not_runnable_here.
  Validated by the schema and the built-in validator.
- Run records carry 'endpoint' (claude-cli-login, or the ANTHROPIC_BASE_URL host) and
  'collect_only'; a run against another endpoint says so on screen, so a local model's
  results cannot pass as Claude's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
…ing it

A run of five skills hit the subscription session limit ('You've hit your session limit ·
resets 11:30am (UTC)'); 34 calls became infrastructure errors and the breaker marked 30
cases NOT_RUN. The harness reported that truthfully, but a long sweep should not die on it.
With --wait-on-limit a limit message that names its reset time is slept out (at most 6 h)
and the call retried; it applies to answer, judge, regrade and calibration calls alike.
Without the flag nothing changes: the call is an infrastructure error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
…canner

Run records now carry denied_tools ('Bash: uv pip install semopy', ...) next to the count;
after the priority fixes 143 arms still had denials, and a count alone could not say which
grant was missing. scripts/defect_scan.py reads saved results and flags, with no model or
judge, the failures a fixture or environment fix removes: input_missing, tool_denied,
install_blocked, script_error, network_blocked, no_answer. On the priority baseline it finds
every missing-input case triaged by hand; blocked installs fell from 27 arms to 3 after the
uv run recipes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
…ct sweeps

scripts/eval_sweep.sh runs --behavioral over a list of skills, N at a time, one results file
and log per skill, with --wait-on-limit by default. docs/local-eval-runbook.md describes
running it against a local model (llama-swap on a workstation GPU, WSL2 as the runner, a
disposable checkout) in --collect-only mode and reading the output with defect_scan.py. It
states that a local run finds eval and environment defects and says nothing about whether a
skill helps Claude, and that local results must never be mixed into Claude numbers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
…st or R

yok-tez's three live searches need the yoktez MCP connector (requires mcp:yoktez), which
answer arms never have; yok-akademik's three profile lookups need akademik.yok.gov.tr
(requires net:...), which resets connections from cloud runners; qca's two R cases need
Rscript. Where a requirement is unmet the case is now 'not runnable here' instead of FAIL;
where it is met (R is installed on this runner) the case runs as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
…d skills

Arms that could not find their input searched the whole disk. One bare arm reported 'a
sibling folder (../skilled/skill/) containing a literature-review skill' — the skill it was
the baseline for — and others listed other sessions' scratch files and installed plugins.
The namespace hid only the checkout.

Each arm now also gets a private temp root holding only its own workspace (bind-mounted back
in after a fresh tmpfs covers the temp root), fresh /var/tmp and /dev/shm, and empty
~/.claude/{plugins,skills,projects,todos} (every home) and /mnt/skills. The startup probe
checks the private view. Checked live: a bare arm granted Bash found no sibling skill copy,
no other workspace and empty skill folders; the CLI still authenticated and answered. The
only SKILL.md files left visible are agent docs bundled inside third-party packages in uv's
cache, the same for both arms.

Results from runs before this commit (the priority sweep and its re-runs) had the exposure in
both the baseline and the after-fix runs; skilled before/after comparisons are like for like,
skilled-vs-bare deltas may understate a skill's effect where a bare arm read its files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
…ning

R 4.3.3 is installed on the runner, but install.packages('QCA') fails there: the system
admisc 0.34 is found first and QCA needs > 0.39. The skill also lacked Write, so it could not
save its own R template. SKILL.md now checks R with 'Rscript --version' and installs admisc
and QCA together into ~/R/qca-lib (checked live: QCA 3.25, admisc 0.41, minimize() runs);
allowed-tools gains Write. Corrects an earlier note of mine that R was absent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
…d not include

Their 'please send it' answers passed in one run and failed in the next, so the verdicts were
noise. Each case now ships its input, and a check pins what the answer must find:
- citation-mgmt/validate-bib-file: a .bib with a DOI that does not resolve (doi.org handle
  API 404; the correct one 200), an article without a journal, and a duplicate DOI.
- citation-mgmt/format-deduplicate-bibtex: a mixed-style .bib with a duplicate, hyphen page
  ranges and no order.
- scientific-slides/validate-slide-overflow: a 4-slide PDF with overflow, a 7-pt table and an
  overlapping label (generated; slide 2's overflow checked by eye).
- syllabus-ai-policy (2): an assessment plan where two items state an AI rule and four do not
  (default-deny applies), and the university code to bind the clause to.
scientific-slides now runs pdf_to_images.py as 'uv run --with pymupdf' instead of telling the
user to pip install it (checked: all four slides converted).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
…Md.'

- alterlab-teaching-design: its routing table had no row for supervising a thesis, so 'help
  me structure my PhD student's methodology chapter and prepare the proposal defense' was
  handed to paper-writer. It now routes to alterlab-thesis-supervisor.
- kvkk-dmp evals: a correct Turkish answer cites 'Md. 28/1-b'; the alternatives list missed
  that abbreviation (it had 'm. 28' and 'madde 28'). Same for Art. 16.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
…erage'

A correct answer explaining that 95% refers to the long-run behaviour of the procedure was
failed for not using the word; equivalent phrasings now count.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
Each skill is claimed with an atomic lock directory, so a second sweep (e.g. with more
workers) skips skills another sweep has run or is running. EVAL_ROOT selects which checkout's
skills and harness to use.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
claude added 27 commits November 4, 2026 07:15
… user ideas

Case alterlab-hypothesis-gen:near-miss-open-ended-brainstorming (hurt:
the skilled arm failed its behavior assertion; bare and skilled both
passed the rubric). The answer named alterlab-scientific-brainstorm
correctly but then only listed details the user should give that skill,
with no ideas at all. The behavior assertion and the rubric want
open-ended, divergent research directions ("Just throw out lots of
creative ideas"); the shared hand-off paragraph's "a brief general
pointer is fine; the named skill does the work" led the model to leave
the user with nothing.

SKILL.md, after the shared hand-off paragraph (left unchanged): one
sentence saying that a brainstorming hand-off still gives a short, varied
list of loose research directions, and none of this skill's own outputs
(hypotheses, predictions, experiment designs).

Version 1.0.1 -> 1.0.2, last_updated 2026-11-04.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
…ol is easy to write

Case alterlab-omero:near-miss-opentrons (hurt: skilled arm failed
output_contains "alterlab-opentrons"; both arms passed the rubric). In
both recorded runs (1.0.1 and 1.1.0) the answer wrote a full Opentrons
Flex protocol and never named alterlab-opentrons, although the routing
row and the shared hand-off paragraph were present. The model read "stop
applying this skill's procedure" as satisfied (writing a protocol is not
OMERO work) and skipped naming the sibling. The row also offered two
targets joined by "or", so it did not say which one fits an Opentrons
request.

SKILL.md:
- the single two-target row is split into one row per sibling:
  Opentrons OT-2/Flex protocols, including the Flex Absorbance Plate
  Reader Module -> alterlab-opentrons; Hamilton/Tecan, plate readers or
  mixed vendors from one Python script -> alterlab-pylabrobot (matching
  each sibling's own description);
- after the shared hand-off paragraph (left unchanged), one sentence: a
  robot or plate-reader protocol has no OMERO step, so the hand-off holds
  even when the protocol could be written directly, and the first line
  names the sibling.

Version 1.1.0 -> 1.1.1, last_updated 2026-11-04.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
…t is not in scope

Two hurt cases, one cause:
- near-miss-open-science-registration-mechanics (bare PASS, skilled
  rubric FAIL): the answer named alterlab-open-science and gave registry
  pointers, then added "One thing from this skill still matters" and this
  skill's freeze checklist. The rubric says the request does not invoke
  this skill at all.
- near-miss-statistical-analysis-test-choice (skilled should_not_trigger
  FAIL): the answer named alterlab-statistical-analysis, then wrote "The
  part of your question I can cover is pre-registration" with a frozen
  decision-rule template and a data-seen check. The trigger judge scored
  that as running this skill.

Both requests only mention a plan ("my analysis plan is settled", "for
my pre-registered plan"). The shared hand-off paragraph's partial-scope
clause let that context count as an in-scope part.

SKILL.md, after the shared hand-off paragraph (left unchanged): one
sentence defining what is in scope for this skill (freezing or checking a
plan before data; judging an analysis without a plan, a peek, or a change
after results), and saying that a plan mentioned only as context adds
none of this skill's checks to a hand-off. The should-trigger cases
(no plan before analysis, outcome switch, optional stopping, locking down
a new study's plan) all stay inside that definition.

Version 1.0.1 -> 1.0.2, last_updated 2026-11-04.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
…tion-verifier even with no file

Case alterlab-research-workflows:near-miss-single-bibliography (hurt:
skilled arm failed output_contains "alterlab-citation-verifier"; both
arms passed the rubric). In two of the three recorded runs (1.1.0 and
1.1.2) the .bib file was not attached, and the answer asked for it and
described doing the DOI and title check itself, without naming
alterlab-citation-verifier. The routing row ("Checking a handful of
references, or one bibliography file") and the shared hand-off paragraph
were both present; the model treated the check as something it could
simply do and the missing file as the only thing to say.

SKILL.md, after the shared hand-off paragraph (left unchanged): one
sentence saying that a one-file or few-reference check is a hand-off even
when the check could be run directly or the file is not attached yet; the
first line names alterlab-citation-verifier (whose verify_citations.py
checks that one file), then the answer asks for the file.

Version 1.1.2 -> 1.1.3, last_updated 2026-11-04.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
… not gate work

Case alterlab-ssci-design-gate:near-miss-test-selection-guard (hurt:
skilled should_not_trigger FAIL; both arms passed the rubric). The
answer named alterlab-test-selection-guard, then wrote "Your design is
already settled, so here is what the design gate adds" and produced the
gate's outputs: design type (two-arm RCT), identifying assumption
(ignorability) and claim type (causal). The trigger judge scored that as
running the gate. The user had only stated the design as fixed context;
the shared hand-off paragraph's partial-scope clause let the model treat
it as an in-scope part.

SKILL.md, after the shared hand-off paragraph (left unchanged): one
sentence defining this gate's scope (choosing, defending or questioning a
design or the causal claim it licenses) and saying that a design stated
as already fixed adds no design type, identifying assumption or claim
type to a hand-off. The should-trigger cases (observational causal
overreach, DiD, experiment vs quasi-experiment, RDD) are all inside it.

Version 1.0.0 -> 1.0.1; last_updated 2026-11-04 added.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
…a and a false contrast

Case alterlab-tubitak-bideb:near-miss-ardeb-3501 (hurt: bare PASS,
skilled rubric FAIL). The answer named alterlab-tubitak-proposal
correctly, but its general notes listed four panel criteria (özgün
değer, yöntem, proje yönetimi, yaygın etki, which are 1001's) and said
3501's career section "does not resemble" the Kariyer Geliştirme
Potansiyeli heading of the 2218/2219 forms. Both are false: 3501 is
scored on five criteria, one of them Kariyer Geliştirme Potansiyeli. The
skill's routing row named 3501 but gave no facts about it, while its
Framework 4 table shows Kariyer Geliştirme Potansiyeli as a 2218/2219
heading, so the model filled the gap with a guessed contrast.

SKILL.md:
- the ARDEB routing row now states 3501's five criteria (Özgün Değer,
  Yöntem, Proje Yönetimi, Kariyer Geliştirme Potansiyeli, Yaygın Etki).
  Verified live on 2026-11-04 from the 3501 değerlendirme formu
  (https://tubitak.gov.tr/sites/default/files/2025-05/3501_degerlendirme_formu_x.doc):
  "Bu form, proje önerilerinin; Özgün Değer, Yöntem, Proje Yönetimi,
  Kariyer Geliştirme Potansiyeli, Yaygın Etki boyutlarında bilimsel
  değerlendirilmesi amacıyla hazırlanmıştır", section 4 "KARİYER
  GELİŞTİRME POTANSİYELİ". Same list as alterlab-tubitak-proposal.
- after the shared hand-off paragraph (left unchanged), one sentence:
  this skill holds no ARDEB caps, word ranges or form text, so an ARDEB
  hand-off quotes none and does not compare ARDEB sections with BİDEB
  headings; past the 3501 facts in its tables, details are left to
  alterlab-tubitak-proposal.

Version 1.0.0 -> 1.0.1, last_updated 2026-11-04.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
…e reply; look up before writing

Case alterlab-scientific-brainstorm:identify-research-gaps (hurt: bare
PASS, skilled behavior and rubric FAIL). The run took two turns. The gap
list was written in an earlier message, the model then made a tool call,
and the final message (the only one graded, and the only one a headless
or SDK caller receives) opened "My last reply got cut off partway
through... I won't repeat it" and held only context questions. The
earlier triage called this an environment effect. Two lines in the skill
make it likely:
- Phase 2 tells the model to "Consult references/brainstorming_methods.md"
  as one of its in-reply interaction steps, which invites a Read in the
  middle of an answer (no tool call was denied, so a Read of that file
  or a gap-check search is the likely second turn);
- the Notes ("a conversation, not a lecture... at least 50% of the
  talking") gave the model a reason to swap the gaps for questions.

SKILL.md:
- Phase 2: do any lookup (the methods file, or a search for whether a
  gap is already covered) before starting the reply, not partway through
  it, so the whole answer arrives in one message;
- Notes: when the user asks to spot research gaps or opportunities, the
  reply carries the candidate gaps, the two or three most promising and a
  concrete next step for each; context questions come before or after
  that content, not instead of it. Scoped to gap requests so the
  questions-first open-ideation flow (open-ended-ideation) is unchanged.

Version 1.0.1 -> 1.0.2, last_updated 2026-11-04.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
… not only a description

Case alterlab-plotly:plotly-interactive-html-scatter failed output_contains
[write_html, to_html]: the chart was built from the fixture and exported correctly (behavior
and rubric PASS), but the code lived only in make_yield_chart.py and the reply described the
result in prose ("change include_plotlyjs=True to 'cdn'") without naming the export call.
The skill had no reporting guidance at all.

Adds "5. Reporting the Result": quote the px/go call with its key arguments and the export
call (write_html / to_html / write_image) with its arguments, and give every file path. A
reader rebuilding or adjusting the chart needs that; it is not the test string.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
… code it wrote

Case alterlab-matplotlib:mpl-custom-rcparams-style failed its behavior assertion ("sets sizes
through plt.rcParams ... then draws a bar chart via fig, ax = plt.subplots()"). The work was
done in figstyle.py and sample_bar.py, but the reply only described them (a table of font
sizes, "four made-up conditions"), so nothing in it showed the rcParams code or the
object-oriented call. For a reusable-style request the code is the deliverable.

Adds a "Reporting the result" note under Saving Figures: show the style/rcParams settings,
the fig, ax = plt.subplots(...) call, the plotting call and savefig arguments, plus every
file path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
… and Hungarian differ

Case alterlab-matchms:cosine-similarity-two-spectra failed behavior and rubric: the answer ran
the right call (CosineGreedy(tolerance=0.1).pair(s1, s2) -> 0.9978, 4 matches) but never said
why greedy rather than Hungarian, which both checks require. The skill's main body only said
the two use "different matching algorithms" and showed calculate_scores for collections, so
it gave no basis for choosing or for reporting the choice.

Adds to Calculating Spectral Similarities: the .pair() idiom for one pair; what greedy and
Hungarian matching do and when they disagree (checked against the matchms source:
CosineGreedy sorts candidate pairs by product then score_best_matches; CosineHungarian uses
scipy.optimize.linear_sum_assignment; verified live on matchms 0.33.1 — fixture 0.9978/4 both
ways, a competing-peaks pair 0.767/1 greedy vs 0.844/2 Hungarian); and a line to report a
score with its function, parameters, filters, matched-peak count and matching choice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
…ession call and its settings

Case alterlab-scvi-tools:probabilistic-de failed output_contains [differential_expression,
mode="change", mode='change']: the script was right (change mode, delta, fdr_target,
batch_correction; rubric PASS) but it lived only in scvi_de_A_vs_B.py, and the reply
paraphrased it ("uses scvi-tools' `change` mode with delta=0.5") without the call.

Adds one instruction under Differential Expression: quote model.differential_expression(...)
with every argument set, the proba_de cutoff and the output paths. Argument names checked
against scvi-tools 1.5.2 (current on PyPI) src/scvi/model/base/_rnamixin.py: groupby, group1,
group2, mode ("vanilla"|"change"), delta, batch_correction, fdr_target.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
…le was extracted

Case alterlab-pdf-extract:explicit-question-columns: all four assertions passed (right rows,
312 enrolled, OR 1.62 ...) but the rubric failed, which expects the --column extraction
questions and MarkItDown conversion. The reply said only "I read all 8 converted papers", so
the method behind the table — which question drove each column, how the PDFs were converted,
that cells were checked against the text — was invisible.

Adds Default Workflow step 5: under the table, state the MarkItDown conversion, the
--column "label:question" specs run, that each cell was checked against the converted paper,
and the paths of the table and converted/ folder. Reporting the data-collection process is a
PRISMA 2020 requirement (item 9), and the column questions are what a user needs to rerun or
extend the sheet. Workflow re-run live on the 8 fixture PDFs (markitdown[all], heuristic
backend, --markdown-dir): exit 0, all 8 converted, Dlamini draft cell "we enrolled 312".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
…rieval date

Case alterlab-protocolsio:read-public-protocol-steps-keyless failed output_contains ['.pdf']:
the keyless route worked (real steps and materials, DOI; behavior and rubric PASS), but the
reply said only "downloading its public PDF" — no URL and no retrieval date — so the user
could not tell which file, or which day's version, the steps came from. The skill asked for
DOI + retrieval date only inside the MCP section.

Adds to Keyless Path step 4: give the exact URL the text came from (the full
/view/<slug>.pdf link for step text) and the retrieval date. Route re-verified live from this
machine, no token, 2026-11-04: GET
https://www.protocols.io/view/tree-mapping-for-leaf-collection-megantic-only-baaciaaw.pdf ->
HTTP 200 application/pdf, title/authors/DOI 10.17504/protocols.io.baaciaaw on page 1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
…ch item by key and DOI

Case alterlab-pyzotero:csljson-recent-articles-bibtex failed output_contains
'10.1136/bmj.n71'. No record was dropped or mangled: the written recent.bib holds exactly the
five expected articles with all five DOIs (behavior and rubric PASS; re-run here, identical
file). The reply listed the entries by cite key, year and journal only, although the skill
already said "Report results by citation key (and DOI)". `export -o` printed just
"exported 5 of 11 items", so the model had to rebuild the list from the file and left the
DOIs out.

- scripts/zotero_export.py: after `export -o FILE`, print one line per written entry to
  stdout (key as written to the file, year, doi:... or NO DOI, title). Output without -o is
  unchanged (still a clean file on stdout). Run live on my_library.json (5 lines, keys match
  the .bib) and my_library.bib --to ris.
- SKILL.md line 114: report by citation key and DOI (title when none), one line per item;
  `export -o` prints that list.
- tests/test_zotero_export.py (new): the listing matches the file's keys and the five DOIs,
  excludes the conference paper; stdout export stays a clean BibTeX file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
… text edges and font sizes

Case alterlab-scientific-slides:validate-slide-overflow failed its second behavior assertion:
slide 2 was flagged for the right-edge overflow but not for "a bullet cut off at the bottom".
The fixture really has one: an eighth bullet, "External validation on a second university",
is drawn at y = 546-580 pt on a 540-pt-high slide (content stream: text origin y = -32.4).
It is wholly off the page, so no rendered image shows it, and PyMuPDF's default extraction
and pdftotext both clip it away. The skill's visual-review route cannot see it, and on a PDF
validate_presentation.py only counted pages — the reply itself noted the script "only checks
font size and bullet count on PowerPoint files". PDF decks (Beamer, PowerPoint exports) also
got no font-size check at all.

- scripts/validate_presentation.py: on a PDF, read text spans with PyMuPDF using an unbounded
  clip; report every span that runs past a slide edge or lies wholly off it (issues, exit 1)
  and text under 18 pt body / 12 pt footer band (warnings), with sizes scaled to a 7.5-inch
  slide height so 11 pt Beamer body text (~22 pt equivalent) is not flagged. Logic is in a
  pure pdf_text_findings() function; without pymupdf the check is skipped with a warning.
- Live on the fixture: slide 2 three lines past the right edge + the bullet wholly below the
  bottom; slide 3 smallest 7 pt; slide 4 callout past the right edge, 10 pt labels; slide 1
  clean. A clean Beamer-size (362.8 x 272.1 pt) PDF reports no issues.
- SKILL.md step 4 and references (scripts_and_prompts.md, core_capabilities.md): the full
  uv run command with --with pymupdf, what the PDF checks find, and to log each finding.
- tests/test_validate_presentation.py (new): edge/size logic, Beamer scaling, blank spans,
  and the fixture end to end (skipped when pymupdf is absent; passes with it).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
… commands

The Installation section said `uv pip install scvi-tools`, which fails outside a virtual
environment, and the compatibility line said to run under a bare `uv run python`. scvi-tools
1.5 requires Python >= 3.12 (PyPI requires_python for 1.5.2, checked 2026-11-04). On an
older interpreter uv silently resolves an old release: `uv pip compile --python-version 3.9`
picks scvi-tools 1.1.6.post2, while 3.12 picks 1.5.2 (torch 2.15.0, lightning 2.6.6), so code
written for 1.5's API would break. The commands are now `uv run --python ">=3.12" --with
scvi-tools ...`, with the extras, and `uv pip install` is kept only for an existing venv.
Flagged by the hurt-case triage (probabilistic-de).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
…w each answer was retrieved

Addresses three hurt cases.

resolve-coordinates (behavior FAIL): resolve_coords() sent the point to
v2 resolve only and, on an empty result, went straight to the Census
geocoder; SKILL.md called the empty result a keyless limit, so the answer
said "Data Commons' own resolve endpoint gives no results for coordinates
without an API key" and never used a Data Commons coordinate method.
The script now asks v2 resolve and the keyless v1 coordinate resolver,
merges both, and falls back to Census/Nominatim only when both are empty;
its "via"/"no_answer" fields name the route. SKILL.md "Coordinates"
bullet, the endpoint table and references/resolve.md say what is true.
The bare arm's "round the coordinates" idea is not used: rounding moves
the point (37.7#-122.4 resolves to San Mateo County, geoId/06081).

county-hierarchy-income (output_contains 'containedInPlace' FAIL) and
population-timeseries (behavior "date='all'" FAIL): the data were right
but the answers never said which call produced them. New "Say how it
was retrieved" paragraph: end with the command/endpoint and parameters,
how each DCID was resolved, date=all for a full series, the relation a
hierarchy query stands for, provenanceUrl and retrieval date.

Live evidence, keyless, 2026-11-04:
- POST api.datacommons.org/v2/resolve {"nodes":["37.7749#-122.4194"],
  "property":"<-geoCoordinate->dcid"} -> {"entities":[{"node":...}]} (no
  candidates); "37.77#-122.4194" -> geoId/06075, geoId/06, country/USA;
  the docs' own example "37.42#-122.08" -> no candidates.
- POST api.datacommons.org/v1/recon/resolve/coordinate
  {"coordinates":[{"latitude":37.7749,"longitude":-122.4194}]} -> zip/94103,
  geoId/0667000 (City), geoId/06075 (County), geoId/06, country/USA, tract,
  block group, school and congressional districts; 41.0082,28.9784 ->
  nuts/TR100, nuts/TR10, nuts/TR1, country/TUR.
- query_datacommons.py resolve --coords 37.7749,-122.4194 -> via v1, 11
  places with names; with both Data Commons calls stubbed out it falls back
  to the Census geocoder (geoId/06, geoId/06075, geoId/0667000).
- observe Count_Person geoId/06 geoId/48 --date all -> 2020 CA 39527808
  (USCensusPEP); within Median_Income_Household geoId/06 County --date 2020
  -> 58 counties, geoId/06001 104888 (CensusACS5YearSurvey).

Version 1.2.1 -> 1.2.2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
…duced them

Addresses hurt case samples-in-study (output_contains FAIL on "Portal
API" / "result=sample" / "filereport"). The skilled answer had the right
916 samples of PRJEB1234 with accessions and FASTQ paths (rubric PASS)
but never said how it fetched them, so a reader could not re-run it.
The skill had no reporting instruction; the "Data Citation" best practice
now asks every answer to end with a "How this was retrieved" line: the
Portal API endpoint (/search or /filereport) with result, query or
accession, fields and limit, the record count checked against /count,
and the retrieval date.

The "all samples in a study" example now passes limit=0 and notes that
scripts/query_ena.py --limit defaults to 100, which silently truncates a
whole-study listing.

Live evidence, keyless, 2026-11-04:
- GET ebi.ac.uk/ena/portal/api/search?result=sample&query=study_accession=PRJEB1234
  &format=tsv&limit=0 -> 916 rows + header (SAMEA1905322 ... Setaria italica)
- GET .../portal/api/count?result=sample&query=study_accession=PRJEB1234 -> 916
- GET .../portal/api/filereport?accession=PRJEB1234&result=read_run
  &fields=run_accession,sample_accession,fastq_ftp&format=tsv&limit=0 -> 916 runs
- query_ena.py search --result sample --query study_accession=PRJEB1234
  --format tsv -> 100 rows; with --limit 0 -> 916 rows

Version 1.0.2 -> 1.0.3.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
…arameters, report the query

Addresses hurt case structure-similarity-search (rubric FAIL; bare PASS
with the same 1,406 hits). The rubric asks for StructSimilarityQuery
with entry_id='4HHB' and an assembly_id/chain_id. The helper had no
structure-similarity command, so the skilled run hand-wrote ss.py in its
scratchpad (two of its tool calls were denied on the way) and gave that
temporary path as the way to reproduce the result. It never gave the
query. It also described the method as BioZernike strict_shape_match, as
references/api_reference.md still did, though rcsb-api 1.6.0 (2026-03)
moved StructSimilarityQuery to the embeddings-based search and deprecated
`operator`.

- scripts/query_pdb.py: new `similar` subcommand (stdlib) that sends the
  same terminal node as rcsb-api's StructSimilarityQuery (--assembly,
  default 1, or --chain, --similarity-type, --ptmscore-cutoff) and prints
  the request and the rcsbapi equivalent with the hits. An empty HTTP 204
  search body now returns total_count 0 instead of a JSON error.
- SKILL.md: lists the command and adds "Say how it was retrieved": end
  with the re-runnable query (rcsbapi call, Search API JSON or helper
  command), key parameters, total_count and retrieval date, not a
  temporary script path. The structure-similarity comment names the
  current parameters.
- references/api_reference.md: "using BioZernike" replaced with the
  current embeddings-based parameters.

Live evidence, keyless, 2026-11-04:
- search.rcsb.org/rcsbsearch/v2/query, service "structure", value
  {"entry_id":"4HHB","assembly_id":"1"}: total_count 1408 with no operator,
  with "strict_shape_match", and with "relaxed_shape_match";
  similarity_type "global" -> 479; {"asym_id":"A"} -> 1450.
- rcsb-api 1.7.3 (uv run --with "rcsb-api>=1.7"):
  StructSimilarityQuery(entry_id="4HHB", assembly_id="1") -> 1408;
  chain_id="A" -> 1450; similarity_type="global" -> 479; its to_dict()
  is the node the helper sends (target_search_space, similarity_type
  local, ptmscore_cutoff 0.8, number_of_candidates 10000).
- query_pdb.py similar 4HHB -> 1408 (4HHB, 1COH, 2HHB first);
  --chain A -> 1450; --similarity-type global -> 479; similar ZZZZ ->
  "HTTP error 404", exit 1.
- rcsb-api CHANGELOG v1.6.0 (2026-03-10): "embeddings-based structure
  similarity search ... `operator` parameter is deprecated".

Version 1.0.2 -> 1.0.3.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
…nregistered DOI

Addresses hurt case citation-match-by-identifier (behavior FAIL:
"Resolves the DOI to a PMID and retrieves the citation record"). The
prompt's DOI 10.1056/NEJMoa123456 is a placeholder that is not registered
anywhere, so no answer can resolve it to a PMID. Both arms correctly
reported "no PMID" (both rubric PASS), and the behavior check failed on
the skilled arm only because it is the only arm graded on assertions. The
same placeholder appears in this skill's own DOI examples (SKILL.md section 5,
references/common_queries.md), which is where the skilled answer said it
came from.

Eval wrong against primary sources: the prompt now uses a registered NEJM
DOI and expected_output names the record it must find. The assertions are
unchanged. The DOI examples use the same real DOI, so they return a record.

Evidence, keyless, no e-mail, 2026-11-04:
- doi.org/api/handles/10.1056/NEJMoa123456 -> HTTP 404,
  {"responseCode":100} (handle not found)
- esearch.fcgi?db=pubmed&term=10.1056/NEJMoa123456[doi] -> count 0,
  phrasesnotfound ["NEJMoa123456"]
- doi.org/api/handles/10.1056/NEJMoa2034577 -> responseCode 1,
  URL http://www.nejm.org/doi/10.1056/NEJMoa2034577
- esearch ...term=10.1056/NEJMoa2034577[doi] -> count 1, idlist [33301246]
  (same with [aid])
- efetch.fcgi?db=pubmed&id=33301246&rettype=medline&retmode=text ->
  PMID- 33301246, TI "Safety and Efficacy of the BNT162b2 mRNA Covid-19
  Vaccine.", AU Polack FP, TA N Engl J Med, VI 383, IP 27, PG 2603-2615,
  LID 10.1056/NEJMoa2034577 [doi]

Version 1.0.3 -> 1.0.4.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
…endpoint a keyless route replaces

Addresses hurt case keyless-state-unemployment-map (behavior FAIL:
"... maps them, naming GeoFRED only as the optional-key alternative").
The data, the 51 values and the map matched the bare arm (rubric and
output_contains PASS). The answer never mentioned GeoFRED and gave only
"FRED's public CSV download" as the route, so a reader could not tell
what the key would have added or re-run the pull. The earlier triage
called it a model slip. The skill does carry the keyless/keyed pairing,
but only as a table row ("State map | {ST}UR ... | GeoFRED series/data +
shapes/file") with no instruction to report it, and its only reporting
line was a paper-citation reminder at the very end of the file.

That closing line is now a "Say how it was retrieved" paragraph in
"Access: keyless first". It asks the answer to end with the keyless URL
or fred_keyless.py command with series IDs and dates, the source agency
and the retrieval date. When a keyless route stands in for a keyed
endpoint, the answer names that endpoint once as the optional-key
alternative and says what it would add. The body stays at 499 lines.

Live evidence, keyless, 2026-11-04:
- fred.stlouisfed.org/graph/fredgraph.csv?id=CAUR&cosd=2026-08-01&coed=2026-08-01
  -> "2026-08-01,5.1"
- fred_keyless.py states UR --date 2026-08-01 -> 51 rows (DC 5.7, CA 5.1, SD 2.0)
- api.stlouisfed.org/geofred/series/data?series_id=WIPCPI&file_type=json
  -> 400 "Variable api_key is not set"
- api.stlouisfed.org/geofred/shapes/file?shape=state -> 400
  "Variable api_key is not set"

Version 1.2.0 -> 1.2.1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
…set and access terms

Addresses hurt case fpf-leverage-timeseries (behavior FAIL: "... notes
Form PF is the source and no API key is needed"). The skilled answer had
the right mnemonics (the three FPF-ALLQHF_GAVN*_LEVERAGERATIO_AVERAGE
cohorts, no FPF-ALLQHF_LEVERAGERATIO_GAVWMEAN), /series/multifull with
start_date=2015-01-01, and rubric PASS. It never told the researcher, who
asked "how do I get it", that the API needs no key or registration. The
word "key" does not occur in it, while the bare arm says so. The earlier
triage called it a model slip. The skill states "None required" under
Authentication but never asks the answer to pass that on, and its only
reporting line was a citation reminder at the end of the file.

That line is now a "Say how it was retrieved" paragraph under
Authentication. It asks the answer to end with the endpoint URL and
parameters, the dataset and its cadence (e.g. SEC Form PF, quarterly),
that no API key or registration is needed, and the retrieval date. The
paper-citation sentence is kept inside the same paragraph.

Live evidence, keyless (no headers, no registration), 2026-11-04:
- data.financialresearch.gov/hf/v1/series/multifull?mnemonics=
  FPF-ALLQHF_GAVN10_LEVERAGERATIO_AVERAGE,FPF-ALLQHF_GAV_SUM&start_date=2015-01-01
  -> 46 quarterly points each, 2015-03-31 (15.39; 5.307e12) to 2026-06-30
  (19.25; 1.3863e13), metadata observation_frequency "Quarterly"
- .../metadata/search?query=FPF-ALLQHF*LEVERAGERATIO* -> exactly
  GAVN10, GAVN11TO50 and GAVN51 *_LEVERAGERATIO_AVERAGE
- .../series/timeseries?mnemonic=FPF-ALLQHF_GAVN10_LEVERAGERATIO_AVERAGE
  &start_date=2015-01-01 -> [["2015-03-31", 15.39], ...]

Version 1.1.1 -> 1.1.2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
…lls back to Census only when both are empty

Offline tests (stubbed _request) for the resolve_coords change: v1 answers when v2 is empty,
v2 and v1 results merge without duplicates, and the Census geocoder runs only when neither
Data Commons route answers. All 3 fail on the previous script (192e010) and pass now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
…ssion from 28ce4e4)

The live re-run of 28ce4e4 (final_k) made things worse: near-miss-akademik-tesvik,
near-miss-academic-cv and near-miss-docentlik, which had passed, now failed, and
near-miss-ardeb-3501 still failed. Each answer named the right sibling, then added BİDEB
material anyway ("I can help with one part here: describing the 2218 fellowship…", "Burada
güvenle söyleyebileceklerim…" with 3501 eligibility and its five criteria). Putting 3501's
criteria in the routing row gave the model facts to recite instead of handing off.

The routing row is back to naming the programmes only. The skill-specific sentence after the
shared hand-off paragraph (unchanged) now defines scope: only choosing, checking eligibility
for or preparing a BİDEB application is in scope; a past BİDEB award behind a CV, incentive
score or doçentlik file is context; a hand-off names the sibling and what to bring, with no
programme facts, eligibility opinions or criteria, ARDEB's included. The five-criteria fact
itself was correct (3501 değerlendirme formu); it belongs to alterlab-tubitak-proposal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
…2 left of 27

Every case where the skill made the answer worse (27 after the v4.0.0 push) was re-read and
fixed where the cause lay in the skill; the 24 changed skills were re-run in full live
(final_k, final_l, final_m). 25 of the 27 now pass with the skill. Suite-wide: helped 556 ->
571, hurt 27 -> 2; skilled PASS 1,349 -> 1,380 of 1,532 graded (88.1% -> 90.1%); bare on the
same cases 813 -> 804 (53.1% -> 52.5%). Of the 168 re-run cases, 35 moved to PASS and 4 away
from it (all four also fail without the skill).

docs/behavioral-results.md, README.md, README.tr-TR.md and the CHANGELOG 4.0.0 subsection are
regenerated, with a second-pass summary, the run range to 2026-11-04 and CLI 2.1.318.
skills.json picks up tubitak-bideb 1.0.2.

Gates: ruff, compileall, pytest (5428 passed, 23 skipped), run_evals --strict, audit (241),
every generator --check, check_claudeai_compat, check_keyless, check_spec, sync-shared --check,
claude plugin validate, release_notes 4.0.0, link check, build_bundles release --verify (260).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
The release is cut on 2026-11-04, after the second pass over the hurt cases; the changelog
heading and CITATION.cff date-released said 2026-11-02. pytest 5428 passed; the release
workflow's packaging tests (test_bundles, test_install, test_claudeai_compat: 521 passed) and
release_notes.py 4.0.0 pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVTRUtoK8ypZyzCDPhMNfs
@AlterLab-IEU
AlterLab-IEU marked this pull request as ready for review November 4, 2026 08:48
@AlterLab-IEU
AlterLab-IEU merged commit c50cf64 into main Nov 4, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants