Ship reusable analyze-action-pr skill under .claude/skills/#807
Ship reusable analyze-action-pr skill under .claude/skills/#807potiuk wants to merge 3 commits intoapache:mainfrom
Conversation
Add a repo-scoped Claude Code skill at .claude/skills/analyze-action-pr/ that triages allowlist PRs end-to-end: extract action refs via verify-action-build --from-pr, classify each finding (pipe-to-shell, unverified-download, nested-action-issue, verify-script-bug, all-clean), look up what verification material upstream already publishes, and draft the right next moves — recommend approval, open an upstream issue + ping the PR author, or fix verify-action-build itself with a regression test. Includes an "Improve this skill" step so future runs that hit a new shape (false positive, novel verification mechanism, asset-naming convention) leave the skill better than they found it: the SKILL.md grows by a row in the case table or a line in the precedents table, not by re-deriving the analysis. The skill encodes shapes recently triaged: PR apache#795 / apache#798 (TS-generic regex hole), PR apache#802 (carabiner-dev nested install actions), PR apache#803 / apache#804 (multi-action extraction), PR apache#806 (jbangdev pipe-to-shell). Replace the broad .claude/ rule in .gitignore with specific entries for personal state (settings*.json, worktrees/, debug/) so .claude/ itself can host shared assets without leaking per-user files.
Pre-commit on PR apache#807 failed because .claude/skills/analyze-action-pr/ SKILL.md was missing the Apache license header. The insert-license hook auto-fixes locally but aborts CI because there's no follow-up commit to land the change. Add the header so CI passes. Document the local pre-commit workflow in AGENTS.md so future agent runs install prek and run ``prek run --all-files`` before pushing, catching the same class of bounce locally instead of in CI.
|
cc: @dfoulks1 @raboof @dave2wave @ppkarwasz - > this skill is somethign that any of you could be using to do what I have been doing for a while (I did it mostly to gather material). I.e.: a) open your agent All the rest should happen automatically. The agent should propose you what to do - whether to propose an issue in the upstream repo or what to advice to the user who is submitting the action, possibly even open a PR fixing the issue in the upstream action. This is ultimately the way how we can make what I did here highly scalable and done by any volunteer who is able to simply review what action proposes and to make decisions what to do next. This SKILL will never post anything on it's own, it will always ask the one who runs it to confirm before anything is done. If you are afraid about security and credentials -> https://cwiki.apache.org/confluence/display/ASFP/RFC-AI-0002+Secure+Agents+setup is the RFC that describes the setup and https://github.com/apache/airflow-steward/tree/main/.claude/skills/setup-isolated-setup-update performs the secure setup for you - so you have to confirm all actions that require your token or credentials to get out of your sandbox - the |
|
@dfoulks1 @raboof @dave2wave @ppkarwasz -> I hope that will help us to distribute the load and review / approve / reject / fix workflow here. Literally anyone can do what I've been doing for the last few weeks - you do not need special knowledge on what to ask - just "analyze #NNN` when you see a failing PR - in vast majority of cases the agent should make a good proposal what to do - all what you need to do is to review the proposal and confirm. |
Case E previously read as "the script has a bug" — too narrow. Real gaps over the life of verify-action-build include whole-action-type support (Deno, Dart), new build flows (npm run start, multi-step Docker), new verification heuristics (sibling sha256sum, JSON data fetches, *Json helpers), false positives on legitimate inputs (multi-stage FROM, vendored node_modules), and extraction-shape gaps (hashes under an existing actions.yml key). Reword the case E row to cover all of these, and add a sub-table under step 5/E that lists 15 historical extensions with the PR or commit that closed each. The table is the template a future agent reaches for when "the verdict is wrong but not for a security reason" — instead of waving the finding off, they can match it to the closest precedent and propose a similar extension. Also tighten the steps: name each module under utils/verify_action_build/ and what it owns (pr_extraction, security, verification, docker_build, action_ref/release_lookup/github_client), so picking the right place for the fix isn't a treasure hunt. Add an explicit "run prek before push" step pointing at AGENTS.md.
|
All the actions that I've done now in #805 was result of running : |
1 participant
Summary
.claude/skills/analyze-action-pr/SKILL.mdthat triages action-allowlist PRs end-to-end: read the PR diff, runverify-action-build --from-pr <N>, classify each finding into one of five cases (pipe-to-shell / unverified-download / nested-action-issue / metadata-only / verify-script gap), look up what verification material upstream releases ship, and draft the right next moves — recommend approval, open an upstream issue + ping the PR author, or extendverify-action-builditself with a regression test.verify-action-build— Deno/Dart action types (verify-action-build: support Deno-based actions (deno task bundle) #749/verify-action-build: support Dart-based actions (setup-dart) #741), new build flows (Fix verify-action-build to try 'npm run start' for actions that use it #664/Fix Docker build for actions with multi-step builds #685), new verification heuristics (verify-action-build: skip binary-download check for pure data fetches #775/verify-action-build: recognize sibling verification steps #800/verify-action-build: match TS generics on @actions/http-client *Json calls #798), false positives on legitimate inputs (verify-action-build: skip pin warning for multi-stage FROM references #733), extraction shapes (verify-action-build: extract all action refs from multi-action PRs #804), per-ecosystem exemptions (verify-action-build: require a lock file for every dependency manifest #770), etc. — so a future agent matches "the verdict is wrong but not for a security reason" against the closest precedent rather than waving the finding off.SKILL.mdrather than re-deriving the analysis next time.jbangdev/setup-jbangaction with tag v0.1.1 #806 (jbangdev pipe-to-shell)..gitignore: replace the broad.claude/rule with specific entries for per-user state (settings*.json,worktrees/,debug/) so.claude/itself can host shared assets (skills) without leaking personal files.AGENTS.md: installprek(a Rust drop-in forpre-commit), runprek installonce per clone, and runprek run --all-filesbefore every push so license headers,actions.ymlsorting, etc. don't bounce CI.How to use
In any Claude Code session opened in this repo, ask: "analyze PR 802" (or any PR number). The skill is auto-loaded by Claude based on its description; the trigger phrasing is also documented in the skill body.
Test plan
git check-ignoreconfirms.claude/settings.local.jsonis still ignored under the new rules.git statusfrom the worktree root shows only the SKILL.md addition and the.gitignore/AGENTS.mddiffs (no personal settings file leaks).prek run --all-filesis clean (license headers, sort, etc.).Pre-commit Checksjob passes on the latest commit.Generated-by Claude Code.