feat(scheduler): capacitycost node scorer + capacityUpgrade rescheduling strategy

[pfernandes21]

What type of PR is this?

/kind feature

What this PR does / why we need it:

Two pieces that make the scheduler prefer cheaper capacity tiers, sharing one static rank read from a node label (karpenter.sh/capacity-type, order reserved,spot,on-demand by default; unlabeled nodes rank 0).

capacitycost plugin (pkg/scheduler/plugins/capacitycost) — a NodeOrderFn for GPU tasks:

score(node) = weight × 100 × (maxRank − rank(node)) / maxRank

At the default capacitycost.weight: 20 a rank-0 node gets +2000 over a rank-2 node, above binpack's reach, so a Pending GPU task lands on (and preempt looks first at) the cheapest tier that passes predicates. Args: capacitycost.weight, .nodeLabelKey, .order, .unlabeledRank, .resource ("" scores every task).

capacityUpgrade rescheduling strategy (pkg/scheduler/plugins/rescheduling/capacity_upgrade.go) — runs under the existing rescheduling plugin / shuffle action next to gpuFragmentation. Per pass:

1. Candidate PodGroups: every GPU member Running, controller-owned, age ≥ minPodAgeSeconds, priority ≤ maxVictimPriority, no <optOutLabel>=false, at least one member above rank 0, cooldown (exa.ai/capacity-upgrade-last) expired. Ordered highest priority → oldest.
2. For each, try target ranks cheapest first and simulate first-fit-decreasing over a pass-wide ledger: a node's claimable room = idle + resources of strictly-lower-priority running pods (what preempt would evict for this group), with the group's own members counted as freeable. Gangs must fit inside one zoneLabel value. Predicates run through PrePredicateFn/PredicateFn, with pod (anti-)affinity relaxed for gang probes since the planner enforces the zone itself.
3. Single-member PodGroups → stamped (capacity-upgrade-last, repack-evictions=1) and returned as victims; the controller recreates the pod and the scorer places it. Multi-member gangs are not evicted: the PodGroup gets a JSON proposal on exa.ai/capacity-upgrade ({target, from, zone, nodes, members, gpus, preempting, at}) for the gang's lifecycle owner to act on with its own gates (checkpoint age, user opt-out, …).

dryRun: true by default, logs the would-be moves and counts them. Budgets: maxVictims (8), maxGangProposals (2). Kill switch EXA_CAPACITY_UPGRADE_DISABLED=true. Metrics: volcano_capacity_upgrade_{passes,moves,pods,gpus}_total{target,kind=pod|gang,mode=dry_run|proposed|live}, ..._stamp_failures_total{kind}.

Example scheduler config:

tiers:
- plugins:
  - name: capacitycost
    arguments:
      capacitycost.weight: 20
  - name: rescheduling
    enabledVictim: true
    arguments:
      interval: "5m"
      strategies:
      - name: capacityUpgrade
        params:
          dryRun: true

Which issue(s) this PR fixes:

N/A

Special notes for your reviewer:

• The ledger assumes preempt will evict lower-priority pods on the target node for the recreated task. That holds when the preemptor and victims are both scheduler-managed (the fork's podgroup controller wraps ReplicaSets, so Deployment pods qualify); dry-run output should be compared with actual preemptions before flipping live.
• Gang proposals are written once per cooldown window and never cleared by the scheduler; the consumer owns clearing/acknowledging the annotation.
• Tests: go test ./pkg/scheduler/plugins/capacitycost/ ./pkg/scheduler/plugins/rescheduling/ (13 new cases: single pod idle fit, filler-as-room, priority ceiling, single-zone gang, predicate veto, cooldown/age/opt-out/do-not-disrupt gates, ledger double-booking, ordering, budgets, mixed gang, config parsing). go build ./pkg/scheduler/..., go vet, gofmt clean.

Does this PR introduce a user-facing change?

Add the capacitycost node-order plugin (static capacity-type rank) and the capacityUpgrade rescheduling strategy (dry-run by default) that moves single-pod GPU work to cheaper capacity and proposes gang moves via the exa.ai/capacity-upgrade PodGroup annotation.

Link to Devin session: https://app.devin.ai/sessions/10aaaafe5fa84b90af9ad613fb6a3b6f
Open in Devin Desktop: https://app.devin.ai/desktop/session/10aaaafe5fa84b90af9ad613fb6a3b6f?variant=devin
Requested by: @pfernandes21

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring