fix(mariadb): fail closed after uncertain DCS switchover to avoid split-brain

[weicao]

Summary (H2)

In addons/mariadb/scripts/replication-switchover.sh, the Stage-3 (DCS switchover) failure path calls rollback_current_primary_switchover_guard, which sets the current primary back to read_only=0 and restores its remote-root write grants. This can produce a split-brain with two writable primaries.

Root cause

syncerctl_switchover is invoked with --force and wrapped by timeout(1):

output=$(timeout "${wall}" "${SYNCERCTL_BIN}" ... switchover --force --primary ... --candidate ...)
rc=$?
...
case "${rc}" in 124|125|137) ... return 1 ;; esac   # timeout -> fail-closed sentinel

A non-zero / timeout return does not prove the DCS switchover record was never persisted. Syncer can accept and persist the request, begin asynchronously promoting the candidate, and then have its client (syncerctl) killed by the wall-clock budget.

On that return, run_switchover does:

if ! syncerctl_switchover ...; then
  if switchover_final_state_already_reached ...; then return 0; fi
  rollback_current_primary_switchover_guard || true   # <-- sets old primary read_only=0
  return 1
fi

switchover_final_state_already_reached only detects the fully-completed end state (candidate already primary + current following it). During the in-progress window (syncer still promoting), it returns false, so rollback runs and un-fences the old primary. Under the alpha.79 minimalist model the old primary is still fully writable at this point (prepare_current_primary_for_switchover is a no-op; the Stage-4 fence has not run yet), so rollback actively guarantees it stays writable. When syncer then finishes promoting the candidate (also writable), both nodes accept writes.

Rollback-to-writable is only safe before syncerctl is ever invoked — i.e. the Stage-2 pre-DCS failure path (wait_candidate_sql_reachable_before_dcs), where no DCS record can exist. That call site is unchanged.

Fix

First principles: once the DCS write cannot be proven to have not landed, fail closed. On the Stage-3 failure path, fence the current primary (read_only=1) instead of rolling it back to writable, via a new fence_current_primary_after_uncertain_dcs helper. If the DCS record truly did not persist, KB roleProbe + reconcile converge the still-read_only old primary (and the failed OpsRequest may retry the whole switchover) — a bounded, recoverable availability blip, always preferable to unbounded split-brain writes.

Tests

addons/mariadb/scripts-ut-spec/replication_switchover_spec.sh updated: the two Stage-3-failure cases now assert the current primary is fenced (set_read_only=ON), never unfenced (set_read_only=OFF), and rollback is not called; plus a new case asserting a manual-verification error is surfaced when the fail-closed fence itself cannot be applied. Full mariadb suite: 664 examples, 0 failures.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 0% with 29 lines in your changes missing coverage. Please review.
✅ Project coverage is 0.00%. Comparing base (554fd84) to head (baf11e6).

Files with missing lines	Patch %	Lines
...adb/scripts-ut-spec/replication_switchover_spec.sh	0.00%	29 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff          @@
##            main   #3066   +/-   ##
=====================================
  Coverage   0.00%   0.00%           
=====================================
  Files        144     144           
  Lines      23053   23077   +24     
=====================================
- Misses     23053   23077   +24     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[weicao]

rebase 到 main 554fd84 + re-author 为 Magnus magnus@apecloud.com + 按 org 规范移除 AI 署名 trailer。内容零变化。