Skip to content

Regenerate npm fixtures with recorded clocks and unchanged policies - #593

Draft
myevolve wants to merge 2 commits into
dependabot:mainfrom
myevolve:fix/npm-clocked-policy-fixtures
Draft

myevolve wants to merge 2 commits into
dependabot:mainfrom
myevolve:fix/npm-clocked-policy-fixtures

Conversation

@myevolve

@myevolve myevolve commented Oct 7, 2026 •

Copy link
Copy Markdown

Dependent fixture migration

Regenerate the three npm fixtures discussed in #592 without changing their job policy or source commits. Draft until the CLI/Core prerequisites are released and the official proxy caches are refreshed.

  • Merge current main 153834ba37793c707347fcc723ce656a34045625 after regen tests that have updated packages #594; resolve the overlap without restoring out-of-policy dedicated transitive PRs.
  • Preserve every original input field, including credentials, ignore conditions, grouping, experiments and source commits. The only addition remains recorded-at; all three previously recorded clocks are unchanged.
  • Keep npm/rules recordings byte-identical to the previous PR head. Semver changes only two generated PR bodies and their commit messages; dependency versions and complete generated lockfiles are unchanged.
  • Document the distinction between replay clocks, HTTP snapshots, and metadata errors in the existing README.

Runtime verification

CLI: dependabot/cli#668 at 49b4c7532aa9fbeb59b300432713bd1b5ed859ea. Integrated Core runtime: dependabot/dependabot-core#16492 at 97c3cac44b62df8ecf93b08f9041849e85ffa20f plus dependabot/dependabot-core#16496 at 260f77e6e25e0b6cbcf00882ada974398de69a82, both incorporating main 0286aa0868b1f0de1b908ce3358bbf994e5272c1. The shared notice-rendering change was exercised directly and all three complete fixture replays were repeated successfully; no npm policy guard or fixture contract changed.

Updater image: ghcr.io/dependabot/dependabot-updater-npm@sha256:4beb224504e150d978da4212e06eb390ba4abb630eeedcfc7d44b265aa1e0d1e, mounting the integrated npm, common and updater libraries.

Suite Recorded UTC clock Replay HTTP cache Complete output
npm 2026-10-07T16:48:15.673Z 202/202 (100%) byte-identical
npm-group-rules 2026-10-07T16:48:15.648Z 200/200 (100%) byte-identical
npm-group-semver 2026-10-07T16:48:15.66Z 249/249 (100%) byte-identical

All three actual CLI tests passed, including a fresh byte-identical replay after merging v1.94.0’s container-isolation changes into CLI #668. CLI CI at this head also passed 104/104 checks, including 2/2 required checks (evidence). Final logs contain no updater ERROR lines, metadata rate-limit responses, unreadable-cache warnings, or job-error events. Fixture syntax is exercised by these actual CLI replays; the repository's yamllint configuration excludes tests/, so its green check is not claimed as fixture lint coverage.

Full-repository Sorbet passed on all three refreshed Core branches and again after the policy spec-isolation correction. Direct shared-notice rendering confirms INFO/fallback → NOTE, WARN → WARNING, ERROR → IMPORTANT. Three complete CLI replays at cooldown 97c3cac44b62df8ecf93b08f9041849e85ffa20f + policy 260f77e6e25e0b6cbcf00882ada974398de69a82 are byte-identical, with 202/202, 200/200 and 249/249 cached calls, unchanged clocks and no metadata/job errors. Policy’s subsequent de4a7cb8bf466b2667fdb4b8bc5948a1061d86c2 commit changes only spec isolation and its README; all mounted runtime library trees are unchanged. CLI source is unchanged; no new Go-suite run is claimed.

The subsequent policy spec-isolation fix is independently verified by the existing two-example failing-before/passing-after reproducer, the complete previously failing worker (564 examples, 0 failures; seed 16904), and final-head full npm CI: 2231 examples, 0 failures, 4 existing pending. All three Core PRs passed 21/21 required checks. Optional Smoke remains non-green: cooldown scheduled no E2E after the swallowed discovery error; policy ran 13 E2E with 10 passes and 3 npm/rules/semver fixture failures; discovery ran all 97 E2E with 95 passes and Gradle/pip-compile failures. These use main fixtures/official caches, not these draft recordings. No failing assertion was weakened.

Correcting the metadata evidence

Earlier semver captures included GitHub API 403 rate-limit fallbacks despite successful jobs and byte-identical cached replay. That omitted PR metadata and truncated commit messages. Those captures are superseded, not treated as a valid baseline.

The corrected recording uses fresh HTTP captures and real public metadata responses. Missing metadata was fetched with trusted, authenticated gh api GETs and cached; healthy responses from the other two fresh captures were reused. The write-enabled host credential was never supplied to the updater or proxy. The updater generated all corrected text; no expected text, versions, clocks or policies were hand-pinned. Final matching caches are retained locally, not represented as refreshed upstream artifacts.

The npm/rules jobs still ignore follow-redirects >1.15.11 and form-data >4.0.5 and reject those unsafe dedicated updates. Semver has no such caps and still records the allowed 1.16.1 / 4.0.6 updates. Old official caches mix stale full registry metadata with newer compact npm metadata, so they must be replaced rather than reused.

Landing gates

No workflow bypasses, raised ignore caps, changed source commits, or hand-edited expected versions are included. The currently released toolchain is not claimed compatible with these recordings.

Independent AI review follow-up

The fixture head remains aeed1cf2ffe310b228b69cc493de09182455372f. After review fixes at Core cooldown 50d4e42 and policy 683673c, all three complete generated fixture YAMLs again match byte-for-byte using CLI 49b4c75; inputs and expected outputs were not weakened. Independent AI review and output hashes. This supersedes the earlier runtime-source pins, not the existing release/upstream-cache gates. The PR remains draft; AI review is not maintainer approval.

@myevolve

myevolve commented Oct 7, 2026 •

Copy link
Copy Markdown
Author

Current head aeed1cf2ffe310b228b69cc493de09182455372f has no main-branch conflicts, but remains a draft: two CI checks passed and all three released-toolchain Smoke jobs failed.

  • The jobs installed CLI v1.93.0 and updater digest sha256:a032e80f8aabe1cb74d46b58d53edfc8944cd3c74f8b3724add15302c440a610. All generated files drop input.recorded-at.
  • npm and rules still generate out-of-policy follow-redirects 1.16.1 and form-data 4.0.6 PRs. Cache coverage: 150/224 (66%) and 149/222 (67%).
  • semver still selects stale lodash 4.17.23 / axios 1.13.5, rather than the recorded 4.18.1 / 1.20.0, with corresponding lockfile and PR-metadata differences. Cache coverage: 174/248 (70%).
  • The green yamllint job excludes tests/; it is not evidence of generated-fixture lint coverage.

The corrected local integrated replay is independently verified: 3/3 successful, byte-identical, 100% cached (202/202, 200/200, 249/249), with no metadata errors or job errors. The prior semver rate-limit fallbacks were removed through real authenticated public metadata reads, not hand-edited expectations; the updater/proxy never received the write-enabled host credential. Matching local caches are archived, not represented as refreshed official artifacts.

Latest runtime heads are Core #16492 97c3cac44b62df8ecf93b08f9041849e85ffa20f and Core #16496 de4a7cb8bf466b2667fdb4b8bc5948a1061d86c2, both incorporating main 0286aa0868b1f0de1b908ce3358bbf994e5272c1. The merge changes shared notice rendering and Julia stdlib notices, without weakening npm policy guards or fixture contracts.

Full-repository Sorbet passed on all three refreshed Core branches and again after the policy spec-isolation correction. Direct shared-notice rendering confirms INFO/fallback → NOTE, WARN → WARNING, ERROR → IMPORTANT. Three complete CLI replays at cooldown 97c3cac44b62df8ecf93b08f9041849e85ffa20f + policy 260f77e6e25e0b6cbcf00882ada974398de69a82 are byte-identical, with 202/202, 200/200 and 249/249 cached calls, unchanged clocks and no metadata/job errors. Policy’s subsequent de4a7cb8bf466b2667fdb4b8bc5948a1061d86c2 commit changes only spec isolation and its README; all mounted runtime library trees are unchanged. CLI source is unchanged; no new Go-suite run is claimed.

Core #16492 CI: 21/21 required checks passed; 55 successful / 4 skipped check records. Full npm/yarn specs: 2231 examples, 0 failures, 4 existing pending. No E2E jobs ran: discovery logged jq: Cannot index string with string "name" but reported success; the overall Smoke workflow concluded failure. Green check records are not E2E coverage.

Core #16496 CI: 21/21 required checks passed; 65 successful / 3 failed / 4 skipped check records, none pending. Full npm/yarn CI at de4a7cb8bf466b2667fdb4b8bc5948a1061d86c2: 2231 examples, 0 failures, 4 existing pending, seed 28164. The previous 24 required-spec failures are resolved.

Smoke actually ran 13 E2E jobs: 10 passed both the test and Diff steps, 3 failed. npm and rules omit the out-of-policy follow-redirects 1.16.1 / form-data 4.0.6 PRs still expected by main fixtures; cache coverage is 124/197 and 122/196. semver omits those dependencies from the expected group and has corresponding lockfile/metadata differences, with 148/223 cached calls. All three used released CLI v1.94.0 and main fixtures/official caches, not draft #593. These failures remain visible; no baseline-identical or released-toolchain-pass claim is made.

Core #16498 at f4f87c4f4645b14edfbe7aee80c63ab38243126e: 21/21 required checks passed; 150 checks succeeded, 2 failed, 4 skipped. Smoke run discovered and executed all 97 E2E jobs: 95 passed both the actual ecosystem test and Diff steps; 2 failed. All 13 npm E2E jobs passed. The failures are Gradle (missing expected PR events after Maven metadata HTTP 404s; 95/125 calls cached) and pip-compile (pycparser 3.0 → 3.1; 96/130 cached). This is main runtime/main fixtures, not the unreleased npm policy or draft recordings; the overall Smoke run is not green. No fixture assertion or unrelated ecosystem code was changed, and no failed job was rerun.

Maintainer actions: (1) review/merge the independent discovery workflow fix and obtain genuine runtime-branch E2E evidence; (2) review/merge CLI #668 and Core #16492/#16496, then release the CLI clock support and an updater containing both runtime fixes; (3) run Cache One for npm, npm-group-rules, and npm-group-semver at fixture head aeed1cf2ffe310b228b69cc493de09182455372f, with read-only recording credentials and checked metadata-fetch logs; (4) run released-toolchain Smoke and obtain fixture review. CLI v1.94.0 still lacks clock replay. No upstream approval, upstream merge, release, official cache refresh, raised cap, changed source commit, or workflow/protection bypass was performed.

Release follow-up: CLI v1.94.0 is now published, but it does not include dependabot/cli#668. The downloaded darwin-arm64 release binary was exercised with the same npm input, integrated Core runtime and preserved cache. Its event comparison exits 0, yet its generated YAML drops input.recorded-at and its updater uses the host clock. With v1.94.0’s isolation changes merged into #668 at 49b4c7532aa9fbeb59b300432713bd1b5ed859ea, all three actual replays remain byte-identical (202/202, 200/200 and 249/249 cached HTTP calls). The fixture content is unchanged; the earlier linked CI logs remain historical v1.93.0 evidence.

@myevolve

myevolve commented Oct 7, 2026 •

Copy link
Copy Markdown
Author

Gradle/pip-compile follow-up (2026-10-08)

The two residual suites are now diagnosed and their clean recordings are published separately in smoke-tests#595, head ddcf3e24cea11fcd30a7e8de4b2f62d5278b6421. Gradle's old Maven cache keys omit :443; pip's old PyPI keys include a Cache-Control header the current client no longer sends. These misses reached live registry data.

Actual clean-cache regeneration changes only five uncapped pycparser==3.1 lines and adds real Hibernate release notes. All job inputs, source commits, ignore conditions, selected primary versions and expected PR sets are preserved. Final released-CLI v1.94.0 replays are full-YAML byte-identical: Gradle 119/119 cached; pip-compile 130/130 cached; both PASS, with no metadata/job errors. No Core/proxy workaround or weakened check was needed.

Released CI for #595: pip-compile passes; Gradle still fails. Gradle now creates all four PRs; its sole diff is the 29-line Hibernate release-note block absent from the old cached releases response (75/116 cached). pip-compile is green at 96/130 cached. Suite discovery and yamllint pass. This confirms why matching Cache One artifacts are still required; expectations must not be weakened to accommodate stale metadata.

Hosted proof added (23:02 UTC): the unchanged Cache One workflow passes in the writable fork at the same #595 head: Gradle and pip-compile. Both downloadable artifacts were replayed without editing responses: full-YAML byte-identical at 119/119 and 130/130 cached. #595 contains artifact links, upload digests and download commands. This is fork-hosted evidence only; the upstream consumer still requires maintainer-published artifacts in dependabot/smoke-tests. No code or constraints changed; fork workflows are inactive again.

Upstream gate remains: a maintainer must run Cache One at refs/pull/595/head for gradle and python-pip-compile, publish both matching caches, and rerun/review the fixture PR before merging. Then rerun Core #16498 Smoke against the new main fixture blobs. #595 is now ready for review, with dependabot/maintainers already requested; matching upstream caches and passing Smoke checks remain required before merge. Fork/local proof does not turn the historical 95/97 CI result green. The independent npm #593, Core review and CLI clock-release gates remain unchanged.

@brettfo — the overlap with #594 remains resolved at aeed1cf2ffe310b228b69cc493de09182455372f, without raising npm ignore caps or restoring unsafe dedicated updates. This fixture PR remains a draft.

The prerequisite team reviews are already requested. Could you help complete review/release of dependabot/cli#668 and dependabot/dependabot-core#16492 / dependabot/dependabot-core#16496, plus review/merge of workflow-only dependabot/dependabot-core#16498? The latter needs no package release.

Latest runtime heads are Core #16492 97c3cac44b62df8ecf93b08f9041849e85ffa20f and Core #16496 de4a7cb8bf466b2667fdb4b8bc5948a1061d86c2, both incorporating main 0286aa0868b1f0de1b908ce3358bbf994e5272c1. The merge changes shared notice rendering and Julia stdlib notices, without weakening npm policy guards or fixture contracts.

Full-repository Sorbet passed on all three refreshed Core branches and again after the policy spec-isolation correction. Direct shared-notice rendering confirms INFO/fallback → NOTE, WARN → WARNING, ERROR → IMPORTANT. Three complete CLI replays at cooldown 97c3cac44b62df8ecf93b08f9041849e85ffa20f + policy 260f77e6e25e0b6cbcf00882ada974398de69a82 are byte-identical, with 202/202, 200/200 and 249/249 cached calls, unchanged clocks and no metadata/job errors. Policy’s subsequent de4a7cb8bf466b2667fdb4b8bc5948a1061d86c2 commit changes only spec isolation and its README; all mounted runtime library trees are unchanged. CLI source is unchanged; no new Go-suite run is claimed.

Test-isolation follow-up at de4a7cb8bf466b2667fdb4b8bc5948a1061d86c2: package-manager specs left npm selectors active across examples, producing 24 failures in the previous full npm run. Shared cleanup now resets both active and per-directory selectors after each example; redundant local cleanup hooks were removed. The existing two-example reproducer fails before / passes after, the complete previously failing worker passes 564 examples, 0 failures at seed 16904, all five changed Ruby files pass RuboCop, and full-repository Sorbet reports no errors. No production runtime code, cooldown/ignore guard or fixture contract changed.

Core #16492 CI: 21/21 required checks passed; 55 successful / 4 skipped check records. Full npm/yarn specs: 2231 examples, 0 failures, 4 existing pending. No E2E jobs ran: discovery logged jq: Cannot index string with string "name" but reported success; the overall Smoke workflow concluded failure. Green check records are not E2E coverage.

Core #16496 CI: 21/21 required checks passed; 65 successful / 3 failed / 4 skipped check records, none pending. Full npm/yarn CI at de4a7cb8bf466b2667fdb4b8bc5948a1061d86c2: 2231 examples, 0 failures, 4 existing pending, seed 28164. The previous 24 required-spec failures are resolved.

Smoke actually ran 13 E2E jobs: 10 passed both the test and Diff steps, 3 failed. npm and rules omit the out-of-policy follow-redirects 1.16.1 / form-data 4.0.6 PRs still expected by main fixtures; cache coverage is 124/197 and 122/196. semver omits those dependencies from the expected group and has corresponding lockfile/metadata differences, with 148/223 cached calls. All three used released CLI v1.94.0 and main fixtures/official caches, not draft #593. These failures remain visible; no baseline-identical or released-toolchain-pass claim is made.

Core #16498 at f4f87c4f4645b14edfbe7aee80c63ab38243126e: 21/21 required checks passed; 150 checks succeeded, 2 failed, 4 skipped. Smoke run discovered and executed all 97 E2E jobs: 95 passed both the actual ecosystem test and Diff steps; 2 failed. All 13 npm E2E jobs passed. The failures are Gradle (missing expected PR events after Maven metadata HTTP 404s; 95/125 calls cached) and pip-compile (pycparser 3.0 → 3.1; 96/130 cached). This is main runtime/main fixtures, not the unreleased npm policy or draft recordings; the overall Smoke run is not green. No fixture assertion or unrelated ecosystem code was changed, and no failed job was rerun.

CLI #668 remains at 49b4c7532aa9fbeb59b300432713bd1b5ed859ea: 104/104 checks, including 2/2 required, passed. Its full local Go race suite passed previously at the same head; it was not rerun during this Core-only update. The actual released v1.94.0 binary was already shown to drop input.recorded-at despite a successful event comparison; that failed release behavior was not rerun merely to confirm it.

Maintainer actions: (1) review/merge the independent discovery workflow fix and obtain genuine runtime-branch E2E evidence; (2) review/merge CLI #668 and Core #16492/#16496, then release the CLI clock support and an updater containing both runtime fixes; (3) run Cache One for npm, npm-group-rules, and npm-group-semver at fixture head aeed1cf2ffe310b228b69cc493de09182455372f, with read-only recording credentials and checked metadata-fetch logs; (4) run released-toolchain Smoke and obtain fixture review. CLI v1.94.0 still lacks clock replay. No upstream approval, upstream merge, release, official cache refresh, raised cap, changed source commit, or workflow/protection bypass was performed.

Review-routing correction: REST GET /repos/{owner}/{repo}/pulls/{number}/requested_reviewers confirms that dependabot/maintainers is already requested on Core #16492, #16496, #16498 and CLI #668. CLI #668 also requests dependabot/azure-dev-ops. These requests predate this continuation: Core requests are dated October 7/8 and both CLI requests October 7. gh pr view displayed empty reviewer lists, while GraphQL reports the existing requests with requestedReviewer: null; the empty display did not mean reviews were unassigned. Core REST evidence; CLI REST evidence.

The later individual requests for JamieMagee and jakecoffman returned HTTP 404 and added no individual reviewers. Existing team requests need review responses, not duplicate assignment. All four prerequisite PRs still report REVIEW_REQUIRED; the account remains read-only upstream. No new review request, approval, upstream merge, release, official cache refresh or protection bypass was performed.

The matching local fixture caches are retained; official Cache One artifacts have not been replaced. Corrected recording and release evidence.

@myevolve myevolve left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent AI-assisted technical review

Reviewed aeed1cf2ffe310b228b69cc493de09182455372f with the independent fixture/workflow reviewer. No actionable introduced fixture defect was identified. The original job inputs, source commits, ignore caps, groups, experiments, and existing recorded clocks were preserved.

After fixing the Core review findings, the parent replayed all three fixtures using CLI 49b4c7532aa9fbeb59b300432713bd1b5ed859ea and combined Core sources including cooldown 50d4e42 and policy 683673c. All complete output YAMLs were byte-identical to this unchanged fixture head:

Suite SHA-256
npm 5f9849ddaa628895b868e0a91972b4ddf650adcea30d39f5ab8eb4201d68435f
group-rules 0ca635d2ec8d6360d5b90f205ebf3f11e305d6beb8abf0a9f02fc6a2054dd357
group-semver 91b4c9b7df3a649884628a7e7222b1eced3f4b6a9213ca3796d1be7b64b102d5

No expectations were weakened. npm/rules retain their follow-redirects/form-data caps; the semver fixture's uncapped updates remain intentional.

This is an explicitly AI-authored technical COMMENT, not maintainer approval. The PR remains draft pending the Core/CLI release and matching upstream-cache gates; local replay is not proof that those gates are complete.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant