fix(mariadb): galera OrderedReady stop ordering — preserve a safe bootstrap point across Stop/Start

[weicao]

Fixes #3104（根因分析、实测证据 sha、方案 A/B 划分见 issue 及其把关评论）

Scope（单变量）

仅两处行为相关变更：

1. addons/mariadb/templates/cmpd-galera.yaml：galera 组件 podManagementPolicy: Parallel → OrderedReady（replication 拓扑本就是 OrderedReady）。
2. addons/mariadb/Chart.yaml：1.2.0-alpha.26 → 1.2.0-alpha.29（CMPD 名含 chart version，spec 字段变更需新 CMPD；跳过 27/28 因验证环境的测试分支谱系已占用这两个 CMPD 名，避免 immutability 冲突）。

随附：4 个版本 pin 契约测试同步到 alpha.29（纯测试维护）。明确不含：preStop 重写、TGP 调整、B 方案（恢复侧 seqno 选举）——均为 #3104 follow-up，避免混变量导致 runtime 结果无法归因。已知遗留：galera-roleprobe 脚本内一处注释仍引用旧 Parallel 语境，行为无关，v2 一并清理。

控制器合同依据（影响面声明）

KubeBlocks Stop 复用 scale-in 的实例对齐删除循环（pkg/controller/instanceset/reconciler_instance_alignment.go:185-225）：Spec.Stop=true → 期望实例集为空 → 全部现存实例进入删除循环。排序 instance_util.go:83-126 按序号降序：

• OrderedReady：每轮 reconcile 只删一个实例，且等前一个 pod 完全终止消失后才删下一个（terminating pod 仍占据 deleteNameSet，循环 break）；pod-0 确定性最后关闭 → 作为 Primary Component 最后成员获得 safe_to_bootstrap=1。
• Parallel（改前）：默认 100% 并发一次删光 → 集群从未收缩到最后成员 → 无人获 safe=1 → fail-closed 拒选（fix(mariadb): fail closed on unsafe Galera bootstrap #2997 的正确行为）→ Stop/Start 永久卡死需人工恢复。
• create/scale-out 影响：OrderedReady 使创建/扩容也串行。Galera joiner SST 从 donor 拉取本就接近串行，预期影响小；由验收矩阵第 2 项（fresh 3-node create）实测兜底。
• Stop 路径不触发任何 lifecycle action（memberLeave/switchover/preTerminate 均只在 scale-in/删除时执行）——本修复不依赖任何 action 钩子。

实测证据

• rerun3 T6 first-red（Parallel）：Stop/Start 后三 PV 全部 seqno=44 safe_to_bootstrap=0，wsrep_cluster_size 180s 不恢复，pod-0 拒绝 crash-recovery bootstrap。packet sha 80f3ab5d...937，pod-0 readOnly 补证 sha 9ef59701...af0。
• v2c 诊断样本（Parallel，N=1）：同路径 Stop 留下 exactly one safe（pod-1 seqno=8 safe=1）且 Start 恢复 —— 证明 Parallel 下结果时序随机（至少一次全灭），进一步支持改为确定性排序。packet sha 9fd0f1b2...584。

验证状态

• 已做：helm lint 通过；helm template 渲染确认 galera CMPD podManagementPolicy: OrderedReady + CMPD 名 mariadb-galera-1.2.0-alpha.29；mariadb 全量 ShellSpec 663 examples / 0 failures / 9 pending（已知）。
• 待做（PR ready 前，由 mariadb team 按 exact 本 chart/head 执行）：①fresh 3-node Galera create（验证 OrderedReady 未拖慢/破坏正常启动与加入）；②focused T6：clean Stop 后三 PV exactly one safe_to_bootstrap=1 + Start 后 3-member Primary 自动恢复；附 cleanup proof。

Draft 保持到上述两项 runtime 验证通过。

[codecov-commenter]

Codecov Report

❌ Patch coverage is 0% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 0.00%. Comparing base (2e74b1d) to head (7a5913c).

Files with missing lines	Patch %	Lines
...adb/scripts-ut-spec/replication_switchover_spec.sh	0.00%	8 Missing ⚠️
...ripts-ut-spec/replication_user_convergence_spec.sh	0.00%	2 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff          @@
##            main   #3105   +/-   ##
=====================================
  Coverage   0.00%   0.00%           
=====================================
  Files        144     144           
  Lines      23013   23013           
=====================================
  Misses     23013   23013           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[weicao]

Runtime validation 完成 — draft 转 ready

PR-v1 focused validation 已由 mariadb team（Jack 执行 / Helen 验收）按 exact head 7a5913cfb3359dfe26bbc127345ba7b90d9da116 / chart 1.2.0-alpha.29 完成，两项全过：

Phase A — fresh 3-node Galera create: PASS

• cluster mdb-galera-prv1a-0327 Running ~85s（与 Parallel 基线同量级——「串行创建拖慢启动」实测不成立），三 pod wsrep_cluster_size=3；cleanup residue 0。

Phase B — focused T6 Stop/Start: PASS

• Stop 后三块 PV grastate（readOnly helper 采样）：pod-0 seqno=5, safe_to_bootstrap=1；pod-1 seqno=4, safe=0；pod-2 seqno=3, safe=0（uuid 一致）。
• exactly one safe=1，且 seqno 阶梯 5>4>3 正是 OrderedReady 串行关机 2→1→0 的指纹（对照：旧 Parallel 的 v2c 样本 seqno 乱序 6/8/7 + 随机 safe 位置；rerun3 首红样本全灭 44/44/44 全 0）。
• Start 后三 pod 全部 Primary / Synced / wsrep_ready=ON / wsrep_connected=ON，自动恢复无人工干预。
• cleanup residue 0。

Evidence archive sha256 b86d56948c4b90edd87bc9d7499835dc049055736c319308ddd4fbc15903d40b（含 identity pin、render 证据、OpsRequest 时间线、grastate 采样、live SQL、双 cleanup proof；归档在测试环境 work/ 目录，índex 见验收 thread）。

结论边界（验收口径原文）：PR #3105 head 7a5913cf PR-v1 focused validation PASS for fresh create + focused T6。不含 full-suite PASS / release-ready；preStop/TGP/恢复侧选举（B 方案）均未在本轮验证范围，见 #3104 follow-up。

CI 全绿（shellspec-test SUCCESS）。head 无新增代码改动，按验收指示转 ready-for-review（nopick 保持，不自行合并）。

[weicao]

Known limitation（Galera remainder run C2 实测，2026-07-08）

范围声明不变：本 PR 修复并验证的是正常 Stop/Start 保留安全引导点（focused T6 + full-run T6 双 PASS）。full force-delete 全崩恢复（C2 场景）不属于本 PR 已解决范围，且本轮实测暴露了 OrderedReady 在该场景下的一个重要 caveat：

实测事实（run galera-remainder-prv1-20260708T0341Z，本 PR head/chart）：

• C2 = kubectl delete pod --force 同时删除三 pod（绕过 KB OrderedReady 删除合同）。
• 事后三块 PV grastate：pod-0 seqno=146 safe=0、pod-1 seqno=147 safe=0、pod-2 seqno=148 safe_to_bootstrap=1（uuid 一致）。
• safe node 存在但不可达：OrderedReady 创建语义要求 pod-0 Ready 后才创建 pod-1/2；pod-0 本地 safe=0 → fail-closed 拒绝 bootstrap（正确）→ CrashLoop → pod-1 Pending、pod-2 永不创建 → 位于 ordinal-2 PV 上的 safe 节点无法上线 → 集群无法自动恢复，需人工。

诚实的 trade-off 记录：Parallel（改前）下同场景三 pod 并发重建，pod-2 会读到本地 safe=1 直接 bootstrap，大概率自动恢复。即本 PR 用「常规 Stop/Start 的确定性安全」换掉了「全崩恢复时 safe 节点恰在高 ordinal 的自愈能力」（后者在改前也无保证——safe 位置随机）。C2 为首次覆盖，无历史基线。

解决方向（follow-up，见 #3104）：

1. 恢复侧 bootstrap 选举（B 方向）：全员 PV/seqno 在场时发现最高 seqno/safe 节点并引导——本事实使 B 从「锦上添花」升级为「补齐 OrderedReady 全崩恢复」的必要项；
2. 或 PR-v2 备选形态：回到 Parallel + gate 化 preStop 有序关闭（保住 Stop/Start 修复的同时保留并发重建的全崩自愈）——两条路线的取舍待 T9-T12 覆盖完成后由 addon owner 决策。

Classification（addon owner 验收口径）：Layer 3 known bootstrap boundary with PR #3105 caveat；非 T6 修复回退，PR-v1 focused validation 结论不变。

[leon-ape]

Finding: The OrderedReady change introduces a user-visible recovery regression for full all-pod force-delete/crash recovery. The PR run galera-remainder-prv1-20260708T0341Z shows safe_to_bootstrap=1 on pod-2, but OrderedReady keeps pod-2 from being recreated while pod-0 CrashLoops with safe_to_bootstrap=0, so the cluster cannot auto recover in that scenario.

[weicao]

Thanks, this finding is correct. The C2/full all-pod force-delete behavior is the documented caveat for this PR, not a hidden pass condition.

Current owner boundary:

• This PR is scoped to normal Galera Stop/Start safety: fresh create + focused T6 passed on exact head 7a5913cfb3359dfe26bbc127345ba7b90d9da116; T9-T11 also have scoped pass evidence.
• The C2 run proved the regression/trade-off you described: the safe PV can be on a higher ordinal (pod-2), while OrderedReady recreates from pod-0; pod-0 correctly fail-closes with safe_to_bootstrap=0, and the higher-ordinal safe node never gets scheduled.
• That full-crash recovery gap is tracked as follow-up mariadb galera: recovery-side quorum-complete seqno election (direction B) — eliminate the OrderedReady full-crash caveat (#3104 follow-up) #3106: recovery-side quorum-complete seqno/safe election. The implementation requirement from this evidence is that a deferred bootstrap candidate must stay alive and publish evidence rather than CrashLoop, otherwise OrderedReady blocks the higher ordinals forever.

So I agree with the finding. The proposed handling is: keep this PR limited to the Stop/Start determinism fix, carry the C2 limitation explicitly, and close the full-crash recovery gap in #3106 instead of claiming this PR solves it.

[weicao]

Owner direction changed after review discussion: do not merge the current OrderedReady-only version as the final Galera fix.

New direction is to keep Galera startup as Parallel, so a safe higher-ordinal node can still be recreated during full all-pod force-delete/crash recovery, and solve normal Stop/Start determinism with a separate ordered-shutdown protection design.

The existing evidence on this PR remains useful as analysis of the OrderedReady trade-off, but it is no longer the route selected for merge. Follow-up code should preserve podManagementPolicy: Parallel and add the shutdown-side protection with fresh validation.

[leon-ape]

Finding: Current head still changes Galera podManagementPolicy to OrderedReady, but comment 4911126473 says the OrderedReady-only version is no longer the selected merge route and should not be merged as the final Galera fix.

[weicao]

Acknowledged. Current head is intentionally no longer considered mergeable because it still contains the OrderedReady-only change.

I am marking the PR back to draft to reflect that state and to prevent accidental merge. We are waiting for the follow-up code direction: keep Galera podManagementPolicy: Parallel and add shutdown-side ordered protection, then rerun validation on the new head.

[weicao]

Superseded by #3108.

Owner direction changed to keep Galera startup as Parallel and solve Stop/Start determinism with bounded shutdown-side protection. This PR remains useful as the OrderedReady trade-off/evidence record, but it is no longer the selected merge route and should not be merged.

[weicao]

Closing as superseded by #3108, which is the selected Parallel startup + bounded shutdown route.