Skip to content

[quality] ci.yml classify job runs node with no Node setup and fails on self-hosted ubuntu-24.04 runners #2096

New issue
New issue
Closed
#2102
Closed
[quality] ci.yml classify job runs node with no Node setup and fails on self-hosted ubuntu-24.04 runners#2096
#2102
Labels
agent/qualityApproved by a Hive merger/owner for auto-merge on green CItestingApproved by a Hive merger/owner for auto-merge on green CI

Description

@hivecommons-hive
hivecommons-hive
bot
opened on Nov 2, 2026

Finding

The classify job in .github/workflows/ci.yml ("Classify changed paths") runs

node --input-type=module -e "import { runClassifier } from './scripts/lib/content-only-ci.mjs'; runClassifier()"

with no Node setup step. It is the only job in ci.yml that does not use
./.github/actions/setup-node-deps; it depends on node being preinstalled on
whatever runner claims runs-on: ubuntu-24.04.

That assumption no longer holds. CI run
41475921935 (PR #2095,
head b97cef8) routed every job to the CNCF self-hosted gpus runner group —
runner cncf-c-ubuntu-24.04-16-64-x86-2xrtxpro6000-bdf7b-runner-g7wwt — which
advertises the ubuntu-24.04 label but ships no node on PATH:

/home/runner/_work/_temp/1082dec7-....sh: line 1: node: command not found
##[error]Process completed with exit code 127.

Why this is not the PR's fault

Same run, same runner group, from GET /repos/cncf/endusers/actions/runs/41475921935/jobs:

Job Node setup Runner group Result
Classify changed paths none gpus failure (exit 127)
Validate repository setup-node-deps gpus success
Lint repository setup-node-deps gpus success
End-to-end tests setup-node-deps gpus success
End-to-end coverage setup-node-deps gpus success

Run 41458212840 at
14:02Z passed the same job on a GitHub-hosted runner (GitHub Actions 1085783063).
The variable is runner routing, not any diff.

classify gates the rest of the workflow (needs: classify), so whenever a PR
lands on a self-hosted ubuntu-24.04 runner the job fails, the PR shows a red
required check, and the content-only fast path is lost.

Recommendation

Make the job carry its own Node rather than inherit one. content-only-ci.mjs
imports only node: builtins (node:child_process, node:fs, node:path,
node:url), so npm ci is not needed and setup-node-deps would add an install
to a job whose whole purpose is to be fast — a bare pinned actions/setup-node
is enough.

In .github/workflows/ci.yml, in the classify job, insert one step between
"Check out repository" and "Classify changed documents":

      - name: Set up Node.js
        # The classifier gates every other job and imports only `node:`
        # builtins, so it needs a runtime but no dependencies. Without this it
        # inherits whatever `node` the runner happens to ship, and a
        # self-hosted runner claiming the `ubuntu-24.04` label ships none.
        uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
        with:
          node-version: '22'
      - name: Classify changed documents

The SHA is the pin already used by .github/actions/setup-node-deps. The literal
'22' matches the devcontainer major and the engines.node field, so the
devcontainer Node major matches the Node major every workflow pins parity test
in tests/dev-environment.test.mjs stays green — that test reads
step.with['node-version'] literals out of .github/workflows/**, so the new
literal is covered by it rather than being a second unguarded place to bump.

No cache: npm is set deliberately: there is no npm ci in this job, so the
cache would only be written, never read.

Follow-up worth a maintainer's attention (separate from the fix above)

The deeper cause is that an org-level self-hosted runner group (gpus) answers
the plain ubuntu-24.04 label, so this repository's jobs can silently land on a
GPU runner image with a different toolchain. The step above makes ci.yml robust
either way, but whether CNCF wants this repository's CI on those runners at all
is an org infrastructure question, not a repository one.

Why no PR accompanies this issue

The entire fix lives in .github/workflows/ci.yml. This agent's token is minted
at a tier without the GitHub Workflows permission, so any push touching
.github/workflows/** is rejected server-side. This needs a human or an agent
with workflow-write to land — it is not a judgement call about whether the
change is ready. The replacement YAML above is complete and mechanical to apply.

Priority

  • Impact: high (gating required check, fails non-deterministically by runner routing)
  • Effort: low (one step, four lines)

Filed by quality agent (hold-gated mode)

🐝 Hive Agent: quality | Instance: hosted-available-lke648397-260827-5n31 | SHA: unknown

— hive: agent=quality backend=copilot model=claude-opus-5 copilot=1.0.88

Activity

  1. hivecommons-hive
    added
    testingApproved by a Hive merger/owner for auto-merge on green CI
    agent/qualityApproved by a Hive merger/owner for auto-merge on green CI
    stage/010-actionableTriaged; eligible for the hive to pick up
    stage/040-covered-by-prAn open PR covers this issue
    and removed
    stage/010-actionableTriaged; eligible for the hive to pick up
    on Nov 2, 2026

  2. hivecommons-hive commented on Nov 2, 2026

    @hivecommons-hive
    hivecommons-hiveBot
    on Nov 2, 2026 – with hivecommons-hive
    ContributorAuthor

    One addition to the fix above, so it lands in a single change:

    #2097 adds tests/ci-node-runtime-setup.test.mjs, a contract that every workflow step invoking node/npm/npx installs Node first. Because the classify step cannot be fixed from that PR, it is carried in that file's NODE_SETUP_EXEMPT_STEPS allowlist. The allowlist has a staleness guard, so once the setup step is added here the suite fails until the entry is removed.

    If #2097 has merged by the time this is applied, delete this entry from tests/ci-node-runtime-setup.test.mjs in the same commit:

    const NODE_SETUP_EXEMPT_STEPS = new Map([
  [
    'ci.yml#classify#Classify changed documents',
    'Known defect: this gating step runs `node` with no setup step and dies ' +
      'with exit 127 whenever the job is routed to a runner image without ' +
      'Node. Delete this entry in the change that adds the setup step.',
  ],
]);

    leaving const NODE_SETUP_EXEMPT_STEPS = new Map([]);. If #2097 has not merged, nothing extra is needed. Verified locally: with the setup step added and the entry removed, node --test tests/ci-node-runtime-setup.test.mjs is 7/7 green.

    🐝 Hive Agent: quality | Instance: hosted-available-lke648397-260827-5n31 | SHA: unknown

    — hive: agent=quality backend=copilot model=claude-opus-5 copilot=1.0.88

  3. hivecommons-hive commented on Nov 2, 2026

    @hivecommons-hive
    hivecommons-hiveBot
    on Nov 2, 2026 – with hivecommons-hive
    ContributorAuthor

    PR #2097 covers this issue: #2097

    The PR body says this is partial/non-closing work (Refs/Part of or leave-open wording), so Hive will not add a closing keyword. GitHub will not show its issue-list PR icon without a closing keyword; the stage comment is the link.

  4. hivecommons-hive commented on Nov 2, 2026

    @hivecommons-hive
    hivecommons-hiveBot
    on Nov 2, 2026 – with hivecommons-hive
    ContributorAuthor

    Scanner triage, 2026-11-02: confirming this is now a shared incident affecting multiple PRs, and reusing this issue as the single tracker (a separate [shared-ci] issue would duplicate the same file and root cause).

    Evidence added since filing:

    • Second affected PR: test: require a Node setup step before a job runs node #2097 (held, hive-authored, test-only diff) failed the identical way — run 41479256008, Classify changed paths exit 127 on gpus runner cncf-c-ubuntu-24.04-16-64-x86-2xrtxpro6000-bdf7b-runner-jrrws, with all four Node-setup jobs green on the same runner group.
    • Default-branch baseline is stale: the most recent ci.yml run on main is 38418930699 (2026-10-13, green) — three weeks old, predating the runner-group routing. A fresh main run could equally land on a gpus runner and fail the same way; this is not PR-local.
    • The passing comparison run 41458212840 was a different PR (architectures/china-merchants-bank-logo) on a GitHub-hosted runner, confirming runner routing as the variable.

    (Note: hive-baseline-check.sh could not complete its sibling-PR lookup in this session — the enumeration call is blocked for agents — so this verdict is from manual diagnosis of the three runs above, not from the helper.)

    Both affected PRs have been commented once with the incident reference and stamped <!-- hive-shared-ci-2096 -->. Both are deferred until the one-step ci.yml fix in the issue body lands (needs a maintainer or a workflow-write agent).

    🐝 Hive Agent: scanner | Instance: hosted-available-lke648397-260827-5n31 | SHA: unknown

    — hive: agent=scanner backend=copilot model=kimi-k3 copilot=1.0.88

  5. hivecommons-hive commented on Nov 2, 2026

    @hivecommons-hive
    hivecommons-hiveBot
    on Nov 2, 2026 – with hivecommons-hive
    ContributorAuthor

    Escalation: this is no longer a single unlucky run

    Re-checked against GET /repos/cncf/endusers/actions/workflows/ci.yml/runs and the
    per-run jobs API. The gpus routing started at 2026-11-02T15:52Z and has hit
    every ci.yml run since:

    Run Started Classify changed paths Runner
    41458212840 14:02Z success GitHub Actions 1085783063 (hosted)
    41475921935 15:52Z failure cncf-c-ubuntu-24.04-16-64-x86-2xrtxpro6000-bdf7b-runner-g7wwt (group gpus)
    41479256008 16:12Z failure cncf-c-ubuntu-24.04-16-64-x86-2xrtxpro6000-bdf7b-runner-jrrws (group gpus)

    Two different gpus machines fail identically, so this is the runner-group image,
    not one bad host. Every ci.yml run at or before 14:02Z was GitHub-hosted and green;
    the five earlier failures in the window (41446815943, 41442475278, 41344303455,
    41333332343, 41292442693) were all on hosted runners and failed other jobs for
    PR-local reasons — none of them is this bug.

    Because classify gates the rest of the workflow (needs: classify), both affected
    PRs (#2095, #2097) now show a red required check that tracks runner routing rather
    than their diffs.

    The one-step fix in this issue is the whole fix

    Scanned every job in .github/workflows/** on main (001d74d) for a run: step
    invoking node, npm or npx without a preceding actions/setup-node or
    ./.github/actions/setup-node-deps step in the same job. Exactly one hit:

    ci.yml :: classify :: step[1] "Classify changed documents" runs-on="ubuntu-24.04" setupIdx=-1

    So no second site needs the same treatment, and the replacement YAML in the issue
    body above is still complete and mechanical to apply.

    Still blocked on a human

    Unchanged: the fix lives entirely in .github/workflows/ci.yml, and this agent's
    token tier carries no GitHub Workflows permission, so any push touching
    .github/workflows/** is rejected server-side. This needs a maintainer or an
    agent with workflow-write to land.

    Also unchanged, and worth more than the patch: an org-level self-hosted runner group
    answers this repository's plain ubuntu-24.04 label. The patch makes ci.yml robust
    either way, but whether cncf/endusers CI should run on GPU runners at all is a CNCF
    infrastructure decision.

    🐝 Hive Agent: quality | Instance: hosted-available-lke648397-260827-5n31 | SHA: 001d74d

    — hive: agent=quality backend=copilot model=claude-opus-5 copilot=1.0.88

  6. mrbobbytables
    closed this as completedin #2102on Nov 3, 2026
  7. hivecommons-hive
    removed
    stage/040-covered-by-prAn open PR covers this issue
    on Nov 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent/qualityApproved by a Hive merger/owner for auto-merge on green CItestingApproved by a Hive merger/owner for auto-merge on green CI

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions