Skip to content

ci: guard against semantic merge races breaking main - #230

Merged
kevincodex1 merged 3 commits into
mainfrom
ci/merge-race-guards
Jul 22, 2026
Merged

kevincodex1 merged 3 commits into
mainfrom
ci/merge-race-guards

Conversation

@kevincodex1

@kevincodex1 kevincodex1 commented Jul 22, 2026 •

Copy link
Copy Markdown
Member

Summary

Guards against semantic merge races: adds a merge-queue trigger to PR Checks and a workflow that opens an issue whenever CI goes red on main.

Motivation & context

#113 and #145 were each green but merged 39s apart on stale bases; combined they broke test-target compilation on main (E0063, fixed in #229), which silently redded CI and blocked release PR #182. Nothing re-tests a PR against the merged result under current settings.

Kind of change

  • Tests / CI

What changed

  • .github/workflows/pr-checks.yml: add merge_group: trigger so the suite can gate a GitHub merge queue.
  • .github/workflows/main-red-alert.yml (new): on a failed push-to-main PR Checks run, opens/updates a ci-main-red issue.
  • Follow-up (settings, not in this PR): enable merge queue or "require branches to be up to date", and mark the four core checks as required.

How a reviewer can verify

# YAML parses; alert job only fires on workflow_run: PR Checks / push / failure
ruby -ryaml -e "YAML.load_file('.github/workflows/main-red-alert.yml')"

<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit

* **New Features**
  * Added an automated “main red alert” that monitors CI results on the main branch and creates or updates a labeled issue with relevant failure details.
  * Added merge-queue validation to re-check pull requests against the merged result, reducing merge-order semantic race issues.
* **Chores**
  * Updated the PR triage workflow to use a newer pinned version of the GitHub scripting action.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

@coderabbitai

coderabbitai Bot commented Jul 22, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: ec1ed7ed-09e2-4f20-94b0-72cabf14ad52

📥 Commits

Reviewing files that changed from the base of the PR and between 8a24071 and 4465beb.

📒 Files selected for processing (3)
  • .github/workflows/main-red-alert.yml
  • .github/workflows/pr-checks.yml
  • .github/workflows/pr-triage.yml
🚧 Files skipped from review as they are similar to previous changes (1)
  • .github/workflows/pr-checks.yml

📝 Walkthrough

Walkthrough

PR Checks now runs for merge queue events. A new workflow monitors current push-based PR Checks runs on main, reconciles labeled alert issues on failures or recovery, and updates the pinned PR triage action.

Changes

Main CI alerting

Layer / File(s) Summary
Workflow entrypoint updates
.github/workflows/pr-checks.yml, .github/workflows/pr-triage.yml
Adds the merge_group trigger to PR Checks and updates the pinned actions/github-script reference to v8.0.0.
Main failure detection
.github/workflows/main-red-alert.yml
Watches completed push-based PR Checks runs on main, serializes reconciliation, and skips runs whose commit is no longer the main tip.
Alert issue reconciliation
.github/workflows/main-red-alert.yml
Closes matching labeled issues after recovery and comments on or creates a ci-main-red issue for current failures.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant PRChecks
  participant MainRedAlert
  participant GitHubIssues
  PRChecks->>MainRedAlert: completed push run on main
  MainRedAlert->>MainRedAlert: compare run head SHA with main tip
  MainRedAlert->>GitHubIssues: find open ci-main-red issues
  MainRedAlert->>GitHubIssues: comment and close recovered alerts
  MainRedAlert->>GitHubIssues: comment on or create failure alert
Loading

Possibly related PRs

  • Gitlawb/node#57: Restructures the same PR Checks workflow with concurrency and additional CI jobs.
  • Gitlawb/node#58: Adds PR triage automation in the same workflow updated here.

Suggested reviewers: beardthelion

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the main change: adding CI safeguards for semantic merge races affecting main.
Description check ✅ Passed The description includes summary, motivation, kind of change, concrete changes, and verification steps, so it mostly matches the template.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch ci/merge-race-guards

Comment @coderabbitai help to get the list of available commands.

@beardthelion beardthelion added the kind:ci CI, release, or packaging pipeline label Jul 22, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/main-red-alert.yml:
- Around line 44-69: Serialize the alert workflow by adding a concurrency
configuration for this workflow/job, using a stable group key for the repository
and workflow and setting the queue behavior to retain pending runs (for example,
queue: max). Apply it to the alert execution containing the getLabel/createLabel
and listForRepo/create operations so simultaneous red-main runs cannot race or
create duplicate issues.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: aab31f50-f3be-45f6-8ab3-3d7deddea64a

📥 Commits

Reviewing files that changed from the base of the PR and between d5ec469 and 0dd8d30.

📒 Files selected for processing (2)
  • .github/workflows/main-red-alert.yml
  • .github/workflows/pr-checks.yml

Comment thread .github/workflows/main-red-alert.yml Resolved

@beardthelion beardthelion left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The merge-race prevention here is sound: merge_group: re-validation is not vacuous (no job in pr-checks.yml carries an if:, so all six run in the queue), concurrency never cancels queue or main runs, permissions are least-privilege, and there's no ${{ }} injection surface in the github-script step. The alerter is the weak half: it reflects CI events, not main's current state, and that produces a false stop-ship. Findings below, highest first. The P2s are worth landing before this is a backstop you can trust; none touches the merge-gate itself.

Findings

  • [P2] Gate the alert on the failed run's commit still being main's tip
    .github/workflows/main-red-alert.yml:30
    The job fires on conclusion == 'failure' && event == 'push' and never checks run.head_sha against main's current head. A re-run of an older failed run keeps event == 'push' and head_branch == main, so re-running a red run after a fixing commit has already gone green (or a late-completing concurrent run) opens a fresh "CI is red on main / stop merging anything" issue while main's tip is green. That false stop-ship is the same release-blocker class this PR exists to surface. Compare run.head_sha to github.rest.repos.getBranch({ branch: 'main' }).commit.sha and no-op when they differ.

  • [P2] Broaden the trigger beyond conclusion == 'failure'
    .github/workflows/main-red-alert.yml:31
    A push-to-main run can also conclude timed_out (the test job runs 45 min against a Postgres service) or startup_failure (malformed workflow). Main is red in both and no issue opens, which is the exact silence this workflow exists to end. Suggest conclusion == 'failure' || conclusion == 'timed_out' || conclusion == 'startup_failure', leaving cancelled out since that's usually a human abort.

  • [P2] Add queue: max so a pending alert isn't displaced under burst merges
    .github/workflows/main-red-alert.yml:23
    Every push-to-main completion (green and red) enters group: main-red-alert; the if: skip happens at job level, after the run has already taken a slot. Actions keeps at most one pending run per group and a newer arrival cancels the older pending one (cancel-in-progress: false guards only the in-progress run). So when several PRs land in quick succession, a red alert sitting pending can be cancelled by a later, possibly skipped-green, completion and that red main gets no issue. queue: max is allowed alongside cancel-in-progress: false and closes it; it's the sibling of the duplicate-issue race the current concurrency block already fixed.

  • [P3] Close or mark-recovered the issue when a push-to-main run goes green
    .github/workflows/main-red-alert.yml:28
    The job only runs on failure, so nothing clears ci-main-red after main is fixed; the "stop merging" issue lingers as a stale event log. A green push-to-main workflow_run should resolve open ci-main-red issues (or comment "recovered at head_sha" and close). This plus the freshness gate above are what turn the alert from an event stream into a current-state signal.

  • [P3] Exclude PRs from the open-issue lookup
    .github/workflows/main-red-alert.yml:64
    issues.listForRepo returns pull requests too. If any open PR ever carries the ci-main-red label, open.data[0] is that PR and the alert comments on it instead of creating the tracking issue. Filter entries with a pull_request key, or query is:issue.

  • [P3] github-script is pinned to v8 here while pr-triage.yml pins v7. The issues.* surface is identical so behavior is unaffected, but worth converging and confirming the SHA is the real v8.0.0 tag.

One thing I couldn't check from the diff: merge_group: only gates if these check names are marked required for the merge queue in branch protection. Runs either way, but confirm that setting or the re-validation won't block.

Two individually-green PRs can break main in combination when the second
merges on checks that ran against a stale base (#113 + #145 compiled fine
apart, but #145 added a field to ReceivedRefUpdate that #113's new test
initializer never set — E0063 on merged main, fixed by #229).

- pr-checks: add merge_group trigger so the suite can gate a GitHub merge
  queue, which re-validates every PR against the true merged result.
- main-red-alert: new workflow that opens/updates a labeled issue whenever
  a push-to-main PR Checks run fails, so a red main is loud instead of
  silent (release-please PRs inherit main breakage, as #182 showed).
Two near-simultaneous red-main events could both pass the open-issue
lookup and each create a ci-main-red issue (or 422 racing createLabel).
A workflow concurrency group with cancel-in-progress: false serializes
them; the default single-slot pending queue is enough since a superseded
alert's state is re-read by the newest run anyway.
Address review: the alerter reflected CI events rather than main's
current state, which could both cry wolf and stay silent.

- Freshness gate: a run only opens or closes anything when its commit is
  still main's tip — re-running an old red run after a fix has landed no
  longer opens a false stop-ship issue (and a stale green can't close a
  current one).
- Trigger covers timed_out and startup_failure — both are red mains that
  previously alerted nothing. cancelled stays excluded as a human abort.
- Recovery: a green run on the tip comments and closes open ci-main-red
  issues, turning the alert into a current-state signal.
- queue: max — the single-slot pending queue let a later completion
  displace a pending red alert under burst merges.
- Issue lookup filters out pull requests carrying the label.
- github-script converged on v8.0.0 (SHA verified against the tag) in
  pr-triage too.
@kevincodex1
kevincodex1 force-pushed the ci/merge-race-guards branch from 8a24071 to 4465beb Compare July 22, 2026 08:20
@kevincodex1
kevincodex1 merged commit e14c2d3 into main Jul 22, 2026
15 checks passed
@beardthelion
beardthelion deleted the ci/merge-race-guards branch November 4, 2026 22:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind:ci CI, release, or packaging pipeline

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants