Skip to content

fix(mysql): fail close ORC prechecks within action budgets#3158

Closed
weicao wants to merge 3 commits into
henry/task82-orc-roleprobe-budgetfrom
henry/task200-orc-precheck-memberleave-hardening
Closed

fix(mysql): fail close ORC prechecks within action budgets#3158
weicao wants to merge 3 commits into
henry/task82-orc-roleprobe-budgetfrom
henry/task200-orc-precheck-memberleave-hardening

Conversation

@weicao

@weicao weicao commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Problem

Two ORC lifecycle paths can report success without trustworthy evidence or exceed the kbagent action window:

  1. The switchover master precheck merges orchestrator-client retry stderr into stdout. A transient curl/jq error can become the parsed master token and make the script exit 0 as "not the master" without attempting switchover.
  2. memberLeave runs three unbounded orchestrator-client calls. A hung client can be killed by the 60s clamp mid-action, and the old instance ... || true verification cannot distinguish a genuinely absent instance from a failed query.

Change

M1: fail closed on noisy master prechecks

  • preserve stdout/stderr separately in the timeout fallback
  • extract only the last non-empty host:port token from stdout
  • use the same resolver in the initial master guard and post-switchover candidate discovery
  • retain raw stdout/stderr/rc diagnostics when resolution fails

M2: bound memberLeave and prove absence

  • bound each orchestrator-client call to 10s with a 1s kill grace
  • set memberLeave action timeout to 50s; worst case is 3 x 11s + 3s settle = 36s, leaving 14s for diagnostics
  • fail rc=1 on forget, reachability, or verification timeout/error
  • replace ambiguous instance ... || true with a successful bounded all-instances read and exact hostname absence check
  • emit per-action phase and retry-safe diagnostics to action stderr and /tmp/member-leave.log

Tests

  • M1: retry stderr + final successful stdout; stderr-only noise; last valid stdout token
  • M2: positive absent closure; forget timeout; reachability timeout; list verification timeout; instance still present; missing leaving-member identity

Local gates:

  • MySQL ShellSpec: 46 examples, 0 failures (bash 3.2.57)
  • focused M1+M2 ShellSpec: 19 examples, 0 failures
  • bash-n: PASS
  • shellcheck --severity=warning: PASS
  • helm dependency build addons/mysql: PASS
  • helm lint addons/mysql: PASS
  • render: memberLeave timeoutSeconds=50, client timeout 10, settle 3, stderr mirrored
  • git diff --check: PASS
  • merge-tree: PASS, tree 6d86e28f3cb01fd7ca23f7f13251d6ae59aa6afd

Boundary

This PR is a draft child of #3117 and changes only M1/M2. It does not include PITR cleanup, ORC client/init/preTerminate, M3, or M4 work owned elsewhere. Evidence is code/static only; runtime N=0 and no acceptance or release-readiness claim is made.

@codecov-commenter

codecov-commenter commented Jul 11, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 100 lines in your changes missing coverage. Please review.
✅ Project coverage is 0.00%. Comparing base (2957aa3) to head (0bee076).

Files with missing lines Patch % Lines
...ons/mysql/scripts-ut-spec/orc_member_leave_spec.sh 0.00% 70 Missing ⚠️
...ddons/mysql/scripts-ut-spec/orc_switchover_spec.sh 0.00% 30 Missing ⚠️
Additional details and impacted files
@@                        Coverage Diff                         @@
##           henry/task82-orc-roleprobe-budget   #3158    +/-   ##
==================================================================
  Coverage                               0.00%   0.00%            
==================================================================
  Files                                    147     148     +1     
  Lines                                  23327   23427   +100     
==================================================================
- Misses                                 23327   23427   +100     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@weicao

weicao commented Jul 11, 2026

Copy link
Copy Markdown
Contributor Author

Review-fix update, exact head 0bee07600bf02bf4a4c661216289a147ce117016:

  • tightened master token from broad ^[^[:space:]]+:[0-9]+$ to the current Kubernetes-hostname contract ^[^[:space:]:]+:[0-9]+$
  • jq:error:42 alone now fails resolution with an empty master
  • jq:error:42, valid mysql-current:3306, and trailing curl:error:7 resolves only the valid hostname token

Re-run gates:

  • focused M1+M2 ShellSpec: 20 examples, 0 failures
  • full MySQL ShellSpec: 47 examples, 0 failures
  • bash-n / shellcheck warning gate / helm lint / render / diff-check: PASS
  • merge-tree: PASS, tree e692180f286a143ddcb2d23085f69c2cc3c34d48

PR remains draft; runtime N=0.

@weicao weicao marked this pull request as ready for review July 11, 2026 17:11
@weicao weicao requested review from a team, leon-ape and xuriwuyun as code owners July 11, 2026 17:11
@weicao

weicao commented Jul 13, 2026

Copy link
Copy Markdown
Contributor Author

Closing this PR because its base is henry/task82-orc-roleprobe-budget, not main or a release branch. Per repository PR policy, stacked/integration branches are for local composition only and must not be used as formal PR bases. The commits remain preserved on the head branch; any still-needed change will be re-created from an allowed main/release baseline with the correct scope and evidence.

@weicao weicao closed this Jul 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants