Fail smoke discovery on GitHub API and pipeline errors

[myevolve]

Latest Smoke diagnosis (2026-10-09)

At unchanged discovery head 29e40163970538243917eaef8f742ebe220bf086, Smoke run 37867769858 executed 97 E2E jobs: 95 passed their test and Diff steps; 2 failed. The failures in this run are NuGet and pip-compile, not Gradle. Required-check success must not be described as an entirely green Smoke run; earlier head-specific results below remain historical.

• NuGet aborts during clone/fetch, before registry replay (0/0 cached calls). Its unchanged ShellGitCommandHandler discards non-authentication Git failures. A host .NET 8 runtime probe linked to the production process/handler files, using real Git and a deliberately non-listening loopback proxy, observed clone exit 128, successful handler return, then fetch raising Win32Exception(2) because the checkout directory did not exist. This verifies the misleading-error mechanism, not the original CI clone error, whose stderr was discarded.
• [INFERENCE] The CI startup log order is consistent with the existing proxy-readiness race: clone/fetch errors appear before the proxy's Listening (:1080) message. CLI #658 already addresses that startup-ordering gap and remains open. It is independent of discovery Fail smoke discovery on GitHub API and pipeline errors #16498 and replay-clock Go modules (vgo) support #668; no duplicate fix, retry, weakened assertion, or new NuGet change was added to this chain.
• pip-compile produces the same five expected dependency updates; its diff is the five pycparser==3.0 → 3.1 lockfile changes already recorded in smoke-tests #595, with 96/130 calls cached.

The six reviewed PR heads, review feedback, release gates and upstream cache publication are unchanged. Core/CLI latest releases remain v0.399.0/v1.94.0; matching upstream artifacts are still absent. Maintainer approval, release and cache-publication steps remain required. The previously denied upstream workflow dispatch was not retried, and no branch or fixture was changed by this investigation.

Problem

Smoke discovery for #16492 selected npm changes, then logged jq: Cannot index string with string "name" while consuming the anonymously fetched test-directory response. Its implicit bash -e shell let the final pipeline stage succeed, publishing an empty matrix. The workflow reported success with no E2E jobs. The log does not reveal the HTTP status/body, so no specific API error is assumed.

The same failure recurred on #16496 at 7fda22682e38f814677005cb5661adff0b7aa9f4: discovery logged the jq error, reported success, and created no E2E jobs. By contrast, #16492’s refreshed run actually executed and passed 13 E2E jobs. An intermittently successful anonymous fetch does not remove the failure-propagation bug.

Change

• Use the existing read-only GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} with gh api, matching this workflow's existing fixture-fetch convention. API failures now stop discovery instead of leaving an error response for suite generation.
• Set explicit shell: bash. GitHub's documented shell invocation includes -eo pipefail, so a failed jq stage cannot be masked by the final collector.
• Pass repository/ref values as quoted environment variables, not GitHub template expansions inside shell code.
• Document fail-closed discovery in the existing ecosystem guide.

No workflow-token permission increase, matrix-policy change, npm runtime change or fixture repinning is included.

Runtime verification

The actual YAML run block was executed with GitHub's explicit-Bash flags, real gh/jq, and authenticated public API reads:

Scenario	Observed result
npm changed	Exit 0; 13 real npm suites, with actual blob SHAs, including npm/rules/semver
No ecosystem changed	Exit 0; legitimate suites=[] preserved
Missing test-directory ref	Real HTTP 404; exit 1; no suites output
Suite pipeline receives that real API error body	jq exit 5; no suites output
Repository name containing shell command-substitution syntax	Literal API lookup returns HTTP 404; no command executes or suites output
Ref containing shell command-substitution syntax	Literal API lookup returns HTTP 404; no command executes or suites output

uvx yamllint -c .yamllint.yaml .github/workflows/smoke.yml passed. This change is independent of the npm runtime PRs and should land before their Smoke status is treated as E2E evidence. The previous false-green result is not reclassified as passing coverage.

Review follow-up

Current head: f4f87c4f4645b14edfbe7aee80c63ab38243126e, incorporating main 0286aa0868b1f0de1b908ce3358bbf994e5272c1. This merge adds upstream Julia/shared-notice changes; the workflow and ecosystem guide are unchanged from the previously verified 801b025fdc5dc83d6beac795ee2501f106d5dd01 head. The two findings on the new API command were addressed in 8b31273bca49c8b13fc48517196f2da0418ad886; all six runtime cases above and YAML lint passed. Local zizmor v1.30.1 no longer flags the API command. The whole-file scan still exits 12 for three findings in untouched expressions (one informational changes-output expression, and informational/low findings in the existing test-download command); no scanner rule was suppressed and no clean whole-file scan is claimed.

Previous-head CI: 9860a9de3c1db8b139effe1bc5de165e9b1f4501

21/21 required checks passed. Final rollup: 152 successful, 8 intentionally skipped, no failed or pending checks. The Smoke workflow completed successfully with 97/97 actual E2E jobs passing, including all 13 npm jobs. Discovery execution confirms authenticated gh api, quoted repo/ref variables and bash --noprofile --norc -e -o pipefail.

This validates the workflow fix against main's runtime/fixtures. It does not validate the separate unreleased npm-policy PRs or the recordings in smoke-tests #593, and it does not turn #16492's earlier empty discovery into E2E coverage. Maintainer review/merge is still required; no approval, merge or release was attempted.

Previous-head CI: 801b025fdc5dc83d6beac795ee2501f106d5dd01

21/21 required checks passed; 152 successful and 4 skipped check records, with no failed or pending checks. The completed Smoke run is bound to this exact head and passed 97/97 actual E2E jobs, including all 13 npm suites. All 97 test and diff steps executed successfully. Discovery logs confirm the quoted gh api command and explicit Bash -e -o pipefail.

This is workflow verification against main’s runtime and fixtures, not evidence that the separate unreleased npm-policy changes or draft fixture recordings have landed. Maintainer review remains required.

Previous-head CI: f4f87c4f4645b14edfbe7aee80c63ab38243126e

Core #16498 at f4f87c4f4645b14edfbe7aee80c63ab38243126e: 21/21 required checks passed; 150 checks succeeded, 2 failed, 4 skipped. Smoke run discovered and executed all 97 E2E jobs: 95 passed both the actual ecosystem test and Diff steps; 2 failed. All 13 npm E2E jobs passed. The failures are Gradle (missing expected PR events after Maven metadata HTTP 404s; 95/125 calls cached) and pip-compile (pycparser 3.0 → 3.1; 96/130 cached). This is main runtime/main fixtures, not the unreleased npm policy or draft recordings; the overall Smoke run is not green. No fixture assertion or unrelated ecosystem code was changed, and no failed job was rerun.

Discovery ran quoted authenticated gh api inputs under explicit Bash -e -o pipefail. NuGet completed successfully after a 35m34s job; the observer timeout did not cancel or rerun any workflow. Final-head full-repository Sorbet passed. The six prior local runtime cases and YAML-lint result refer to unchanged workflow source.

Gradle/pip-compile follow-up (2026-10-08)

The two residual suites are now diagnosed and their clean recordings are published separately in smoke-tests#595, head ddcf3e24cea11fcd30a7e8de4b2f62d5278b6421. Gradle's old Maven cache keys omit :443; pip's old PyPI keys include a Cache-Control header the current client no longer sends. These misses reached live registry data.

Actual clean-cache regeneration changes only five uncapped pycparser==3.1 lines and adds real Hibernate release notes. All job inputs, source commits, ignore conditions, selected primary versions and expected PR sets are preserved. Final released-CLI v1.94.0 replays are full-YAML byte-identical: Gradle 119/119 cached; pip-compile 130/130 cached; both PASS, with no metadata/job errors. No Core/proxy workaround or weakened check was needed.

Released CI for #595: pip-compile passes; Gradle still fails. Gradle now creates all four PRs; its sole diff is the 29-line Hibernate release-note block absent from the old cached releases response (75/116 cached). pip-compile is green at 96/130 cached. Suite discovery and yamllint pass. This confirms why matching Cache One artifacts are still required; expectations must not be weakened to accommodate stale metadata.

Hosted proof added (23:02 UTC): the unchanged Cache One workflow passes in the writable fork at the same #595 head: Gradle and pip-compile. Both downloadable artifacts were replayed without editing responses: full-YAML byte-identical at 119/119 and 130/130 cached. #595 contains artifact links, upload digests and download commands. This is fork-hosted evidence only; the upstream consumer still requires maintainer-published artifacts in dependabot/smoke-tests. No code or constraints changed; fork workflows are inactive again.

Upstream gate remains: a maintainer must run Cache One at refs/pull/595/head for gradle and python-pip-compile, publish both matching caches, and rerun/review the fixture PR before merging. Then rerun Core #16498 Smoke against the new main fixture blobs. #595 is now ready for review, with dependabot/maintainers already requested; matching upstream caches and passing Smoke checks remain required before merge. Fork/local proof does not turn the historical 95/97 CI result green. The independent npm #593, Core review and CLI clock-release gates remain unchanged.

Independent AI review follow-up

Current head: 29e40163970538243917eaef8f742ebe220bf086. The follow-up adds ruby script/test-smoke-discovery and docs; the production workflow remains unchanged from f4f87c4. All four behavioral cases and RuboCop pass; actual current API discovery yields 97 suites. Independent AI review, including retracted pre-existing coverage concerns. This is not maintainer approval or upstream cache publication. Earlier head-specific CI counts remain historical.

[myevolve]

Independent AI-assisted technical review

The separate workflow/fixture reviewer found no introduced blocking correctness defect in this failure-propagation change. Follow-up 29e40163970538243917eaef8f742ebe220bf086 adds a runnable regression harness and its documentation; the production workflow is unchanged from the previously reviewed f4f87c4.

Executed ruby script/test-smoke-discovery: 4/4 cases passed using the actual workflow step—complete success, legitimate empty selection, API failure with a valid body, and suite-generation failure. RuboCop passed. Live gh api discovery against the current 101-entry tests directory produced 97 suites.

Two preliminary concerns were checked and retracted as findings against this PR: the Contents API's 1,000-entry limit is pre-existing and not reached, and six matrix ecosystems already lack fixtures. Adding an unconditional completeness guard would break existing all-ecosystem discovery, so no such scope-expanding guard or matrix exceptions were introduced.

This is an explicitly AI-authored technical COMMENT, not maintainer approval. Cache refresh/release gates remain separate; prior end-to-end failures are not relabeled green.

Canonical CI follow-up

At 29e40163970538243917eaef8f742ebe220bf086, all 21 required checks passed, including Specs, Sorbet, lint, and ARM64 build. Required checks do not substitute for the separate smoke/cache-publication gates.