diff --git a/.claude/skills/update-biobase.md b/.claude/skills/update-biobase.md index f43c022..03a9322 100644 --- a/.claude/skills/update-biobase.md +++ b/.claude/skills/update-biobase.md @@ -1,135 +1,3 @@ # Update biobase manifest -Check all container images in the biobase manifest for newer tags and create a PR with updates. - -## Process - -### 1. Find the current manifest - -Look for the latest versioned biobase manifest in `bulker/`: - -```bash -ls bulker/biobase_*.yaml | sort -V | tail -1 -``` - -Read this file to get the current commands and image tags. - -### 2. Check each image for updates - -For each command entry, query the container registry API to find the latest available tag. - -**Skip these entries** (pinned or custom registries): -- `nsheff/pigz` — custom image, no registry API -- `databio/refgenie` — custom image, uses `latest` tag -- Any image using the `latest` tag - -`quay.io/xujishu/cellranger` is NOT skipped: it is a normal quay.io repo and -answers the same tag API as the biocontainers images below. Tags older than -6.0.0 are stored as Docker v1 manifests, which apptainer cannot convert, so -never move this entry backwards below 6.0.0. - -**For quay.io/biocontainers images:** - -```bash -curl -s "https://quay.io/api/v1/repository/biocontainers//tag/?limit=100&onlyActiveTags=true" -``` - -Parse the JSON response. Tags follow the pattern `--_`. To find the latest: -1. Filter out tags named `latest` -2. Pick the highest version prefix (the part before `--`) -3. Among tags at that version, pick the highest `_`. A different hash at - a higher build is a rebuild, not a variant -- take it. The hash changes - *because* the package was rebuilt. -4. Never move to a lower version or build than the current pin - -Some tools publish parallel `pyXXX` builds at the same version and build number. -Those are ties; take the highest `pyXXX`. - -**For `quay.io/xujishu/cellranger`:** - -```bash -curl -s "https://quay.io/api/v1/repository/xujishu/cellranger/tag/?limit=100&onlyActiveTags=true" -``` - -Tags here are plain semantic versions (`3.1.0`, `6.0.0`, `6.0.1`) — NOT the -biocontainers `--_` pattern. Sort by semantic version and -pick the highest. Ignore any tag below 6.0.0: those are Docker v1 manifests -(`application/vnd.docker.distribution.manifest.v1+prettyjws`) that apptainer -cannot convert, so selecting one silently breaks every apptainer-based consumer. - -**For Docker Hub images** (broadinstitute/*, bioconductor/*): - -```bash -curl -s "https://hub.docker.com/v2/repositories///tags/?page_size=100&ordering=last_updated" -``` - -For `broadinstitute/gatk`: pick the latest tag matching `...` pattern (semantic version sort). -For `broadinstitute/picard`: pick the latest tag matching `..` pattern. -For `bioconductor/bioconductor_docker`: pick the latest `RELEASE__` tag (highest major, then highest minor). - -**For UCSC tools** (quay.io/biocontainers/ucsc-*): - -These all use version numbers like `482--h0b57e2e_0`. Check for updates using the same quay.io API as other biocontainers. - -### 3. Compare and decide - -For each image, compare the current tag with the latest available: -- If the latest tag is different and represents a newer version, mark it for update -- Log each comparison result (tool name, current tag, latest tag, update needed) -- Present a summary table before making changes - -### 4. Create the updated manifest - -If any updates are found: - -1. **Determine the new version.** Bump the patch version of the current manifest (e.g., `0.1.0` -> `0.1.1`). Use minor bump only if tools are added or removed. - -2. **Create the new manifest file.** Copy the current manifest to `bulker/biobase_.yaml`. Update the `version:` field in the YAML. Replace updated image tags. - -3. **Update the symlink** (if one exists): `ln -sf biobase_.yaml bulker/biobase.yaml` - -4. **Do NOT delete the old versioned manifest.** Keep it for reference. - -### 5. Commit and open a PR - -```bash -git checkout -b biobase-update- -git add bulker/biobase_.yaml bulker/biobase.yaml -git commit -m "Update biobase to - -Updated images: -- : -> -- : -> -..." -git push -u origin biobase-update- -``` - -Open a PR with: -- Title: `Update biobase to ` -- Body: use this template. All three sections are required, even if empty. - -```markdown -## Updated - -| Tool | Old tag | New tag | -|---|---|---| - -## Checked, no update needed - -Every other image, with the tag you confirmed is current. - -## Skipped - -Each skipped image and the rule that skipped it. -``` - -### 6. If no updates found - -If all images are already at their latest versions, do not create any files or PRs. Just report that everything is up to date. - -## Important rules - -- **Same providers, just new tags.** Never switch an image from one registry to another. Only update the tag. -- **Preserve all other fields.** Keep `docker_command`, `docker_args`, `description`, and any other fields exactly as they are. -- **Preserve YAML formatting.** Match the indentation and style of the existing manifest. -- **Be conservative.** If you can't determine which tag is newer, skip that entry. +Follow `automation/update-biobase.md` completely. diff --git a/.github/workflows/scheduled-biobase-update-jules.yml b/.github/workflows/scheduled-biobase-update-jules.yml deleted file mode 100644 index 4a80cae..0000000 --- a/.github/workflows/scheduled-biobase-update-jules.yml +++ /dev/null @@ -1,24 +0,0 @@ -name: Scheduled Biobase Update - Jules - -on: - schedule: - - cron: '0 0 17 * *' - workflow_dispatch: - -jobs: - biobase-update: - runs-on: ubuntu-latest - permissions: - contents: write - pull-requests: write - - steps: - - name: Checkout Repository - uses: actions/checkout@v6 - - - name: Run Biobase Update with Jules - uses: google-labs-code/jules-invoke@v1.0.0 - with: - jules_api_key: ${{ secrets.JULES_API_KEY }} - starting_branch: master - prompt: "Execute the update task documented in .claude/skills/update-biobase.md" \ No newline at end of file diff --git a/.github/workflows/scheduled-biobase-update.yml b/.github/workflows/scheduled-biobase-update.yml index 28144cd..3029e1b 100644 --- a/.github/workflows/scheduled-biobase-update.yml +++ b/.github/workflows/scheduled-biobase-update.yml @@ -1,7 +1,8 @@ -name: Scheduled biobase update +name: Manual biobase update with Claude + +# Manual fallback. Recurring biobase updates are scheduled in Jules. +# The complete procedure lives in automation/update-biobase.md. on: - schedule: - - cron: "0 0 3 * *" # Once a month workflow_dispatch: permissions: @@ -16,13 +17,13 @@ jobs: steps: - uses: actions/checkout@v6 with: - fetch-depth: 1 + fetch-depth: 0 - name: Run Claude Code to update biobase uses: anthropics/claude-code-action@v1 with: prompt: | - Use the update-biobase skill to check all biobase container images for newer versions and create a PR if any updates are found. + Follow automation/update-biobase.md completely. claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }} claude_args: '--allowedTools "Bash(*),Read(*),Write(*),Edit(*),Glob(*),Grep(*)"' show_full_output: true diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..c320818 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,6 @@ +# Agent instructions + +## Biobase update + +For any task that checks or updates biobase container tags or manifests, read +and follow `automation/update-biobase.md` completely. diff --git a/CLAUDE.md b/CLAUDE.md index b199eca..7d8668a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -25,13 +25,14 @@ Run `python validate_manifests.py --check-tags` to validate manifests locally. T ## Automated biobase updates -A scheduled GitHub Actions workflow (`.github/workflows/scheduled-biobase-update.yml`) runs every Thursday at midnight UTC. It uses Claude Code with the `.claude/skills/update-biobase.md` skill to: +The complete biobase update procedure lives in +`automation/update-biobase.md`. Recurring runs are handled by a native Jules +Scheduled Task. `AGENTS.md` and `.claude/skills/update-biobase.md` are thin +pointers to that canonical procedure. -1. Check all biobase container images for newer tags via registry APIs (Quay.io, Docker Hub) -2. Create a new versioned manifest with updated tags (patch version bump) -3. Open a PR for human review - -The workflow can also be triggered manually via `workflow_dispatch`. Some images are intentionally skipped (cellranger, pigz, refgenie) because they use custom registries or pinned versions. +`.github/workflows/scheduled-biobase-update.yml` is retained as a manual +`workflow_dispatch` fallback using Claude Code. There should be only one +recurring scheduler for this task. ## Automated refgenie crate updates @@ -45,10 +46,10 @@ build host. Its pins are **derived from `bulker/biobase`** rather than discovered independently — see `update_refgenie_crate.py`, the reviewable source map `refgenie_crate_sources.yaml`, and `.claude/skills/update-refgenie-crate.md`. -`.github/workflows/scheduled-refgenie-update.yml` runs it **quarterly** (not -weekly like biobase) and always opens a PR: refgenie names each asset after the -tool version that built it, so a pin bump renames published assets, forces a -rebuild and orphans S3 objects. +`.github/workflows/scheduled-refgenie-update.yml` runs it **quarterly** and +always opens a PR: refgenie names each asset after the tool version that built +it, so a pin bump renames published assets, forces a rebuild and orphans S3 +objects. The interesting part is the **sibling map**. biobase pins `hisat2` but not `hisat2-build`, `bowtie2` but not `bowtie2-build`, `tabix` but not `bgzip`, diff --git a/automation/update-biobase.md b/automation/update-biobase.md new file mode 100644 index 0000000..041993e --- /dev/null +++ b/automation/update-biobase.md @@ -0,0 +1,231 @@ +# Update the biobase manifest + +This is the canonical procedure for recurring biobase container updates. Any +agent or automation that performs this task must follow this file completely. + +## Goal + +Check all container images in the latest versioned biobase manifest for newer +tags. If verified updates exist, create the next versioned manifest, update the +`bulker/biobase.yaml` symlink, and prepare a pull request for human review. + +If no verified updates exist, make no repository changes and create no pull +request. + +## Before starting + +Check whether there is already an open pull request for a biobase update whose +head branch starts with `biobase-update-`. + +- If one exists and the execution environment can safely continue that branch, + start from that branch and preserve all valid unmerged changes already there. +- If one exists but the execution environment cannot safely continue it, make no + changes and stop rather than creating a competing update PR. +- Otherwise start from the current `master` branch. + +Never discard valid unmerged biobase changes from an existing update PR. + +## 1. Find the current manifest + +Find the latest versioned biobase manifest in `bulker/`: + +```bash +ls bulker/biobase_*.yaml | sort -V | tail -1 +``` + +Read this file to get the current commands and image tags. + +## 2. Check each image for updates + +For each command entry, query the container registry API to find the latest +available tag. + +### Skip rules + +Skip these entries: + +- `nsheff/pigz` — custom image, no registry API +- `databio/refgenie` — custom image, uses `latest` tag +- Any image using the `latest` tag + +`quay.io/xujishu/cellranger` is **not** skipped. It is a normal quay.io repo and +uses the tag API described below. Tags older than `6.0.0` are stored as Docker +v1 manifests that Apptainer cannot convert, so never move this entry backwards +below `6.0.0`. + +### quay.io/biocontainers images + +Query: + +```bash +curl -s "https://quay.io/api/v1/repository/biocontainers//tag/?limit=100&onlyActiveTags=true" +``` + +Tags follow the pattern `--_`. + +To select the latest acceptable tag: + +1. Filter out tags named `latest`. +2. Pick the highest version prefix, meaning the part before `--`. +3. Among tags at that version, pick the highest `_`. +4. A different hash at a higher build is a rebuild, not a variant; take it. +5. Never move to a lower version or build than the current pin. + +Some tools publish parallel `pyXXX` builds at the same version and build +number. Those are ties; take the highest `pyXXX`. + +### `quay.io/xujishu/cellranger` + +Query: + +```bash +curl -s "https://quay.io/api/v1/repository/xujishu/cellranger/tag/?limit=100&onlyActiveTags=true" +``` + +Tags are plain semantic versions such as `3.1.0`, `6.0.0`, and `6.0.1`; they do +not use the biocontainers `--_` pattern. + +Sort by semantic version and select the highest tag, but ignore every tag below +`6.0.0`. Those older tags are Docker v1 manifests +(`application/vnd.docker.distribution.manifest.v1+prettyjws`) and selecting one +breaks Apptainer-based consumers. + +### Docker Hub images + +For `broadinstitute/*` and `bioconductor/*`, query: + +```bash +curl -s "https://hub.docker.com/v2/repositories///tags/?page_size=100&ordering=last_updated" +``` + +Selection rules: + +- `broadinstitute/gatk`: select the latest tag matching + `...` using semantic-version ordering. +- `broadinstitute/picard`: select the latest tag matching + `..` using semantic-version ordering. +- `bioconductor/bioconductor_docker`: select the latest + `RELEASE__` tag by highest major, then highest minor. + +### UCSC tools + +For `quay.io/biocontainers/ucsc-*`, use the same quay.io API and selection rules +as other biocontainers images. These commonly use versions such as +`482--h0b57e2e_0`. + +## 3. Compare and decide + +For every image: + +- compare the current tag with the latest acceptable tag; +- update only when the latest tag is demonstrably newer; +- never downgrade; +- log the tool name, current tag, latest tag, and whether an update is needed. + +Present a summary table before making changes. + +If it is unclear which tag is newer or whether a candidate is safe, leave that +entry unchanged. + +## 4. Create the updated manifest + +If at least one verified update exists: + +1. Determine the new manifest version. Bump the patch version of the current + manifest, for example `0.1.0` -> `0.1.1`. Use a minor bump only when tools are + added or removed. +2. Copy the current manifest to `bulker/biobase_.yaml`. +3. Update the `version:` field in the new YAML file. +4. Replace only the verified image tags that need updating. +5. Update the symlink, if present: + + ```bash + ln -sf biobase_.yaml bulker/biobase.yaml + ``` + +6. Do **not** delete the previous versioned manifest. + +## 5. Validate the result + +Run the repository's manifest validation before publication: + +```bash +python validate_manifests.py --check-tags +``` + +Do not publish a pull request if validation fails. Fix only problems caused by +this update; do not broaden the task into unrelated repository cleanup. + +## 6. Prepare the review change + +Use branch name: + +```text +biobase-update- +``` + +Commit message: + +```text +Update biobase to + +Updated images: +- : -> +- : -> +... +``` + +The pull-request title must be: + +```text +Update biobase to +``` + +The pull-request body must contain all three sections, even when one is empty: + +```markdown +## Updated + +| Tool | Old tag | New tag | +|---|---|---| + +## Checked, no update needed + +Every other image, with the tag you confirmed is current. + +## Skipped + +Each skipped image and the rule that skipped it. +``` + +Use the execution environment's native branch and pull-request publication +mechanism when it provides one. Do not duplicate that mechanism with manual +`git push` or `gh pr create` calls when the environment will publish the branch +or PR itself. + +If the environment does not publish changes automatically but authenticated git +and GitHub tooling are available, create the branch, commit the changes, push the +branch, and open the PR using the specification above. + +Never merge the pull request automatically. + +## 7. If no updates are found + +If every applicable image is already at its latest acceptable version, do not +create a new manifest, branch, commit, or pull request. Report that everything +is up to date. + +## Invariants + +- **Same providers, only newer tags.** Never switch an image from one registry + or repository to another. +- **Never downgrade.** A candidate must be demonstrably newer than the current + pin. +- **Preserve all unrelated fields.** Keep `docker_command`, `docker_args`, + `description`, and every other unrelated field exactly as it is. +- **Preserve YAML formatting.** Match the indentation and style of the current + manifest. +- **Be conservative.** If a candidate cannot be verified confidently, skip it. +- **Keep old versioned manifests.** Never replace or delete historical versions. +- **No duplicate update PRs.** Continue an existing biobase update when safe; + otherwise stop rather than competing with it.