Bug#121063 Members of one group assign different GNOs to the same transaction

[matias-sanchez]

Bug#121063: https://bugs.mysql.com/bug.php?id=121063

What happens

Two members of the same group assign different GNOs to the same transaction, in multi-primary mode, when a member leaves and rejoins during a rolling restart under write load. The group splits into internally consistent sets that disagree on the GTID of the same transaction. It can surface on the group_replication_applier channel as Error_code 1032 or 1062, or stay silent with every member ONLINE and the disagreement present only in the binary logs.

Reproduced on 8.4.8, 8.4.11 and 9.7.2. A self contained reproducer is attached to the bug.

Where it starts

A reserved synode is only ours while we still hold the node index it was reserved under.

local_synode_allocator stamps the member's current index into the synode, synode.node = my_nodeno. Still inside reserve_synode_number, the task yields in the while (too_far(*msgno)) loop at TIMED_TASK_WAIT. During that yield, site_install_action reassigns site->nodeno. The reservation still carries the old index, so proposer_task brands and proposes into a slot that now belongs to another node.

The header comment of xcom_base.cc states the rule at line 107: only node N may propose a value for synode {X N}. With two proposers on the same slot at cnt=0, acceptor.promise is never raised, since it is assigned in exactly one place, inside handle_simple_prepare, which is the phase 1 decision. Both proposals are accepted, both are learned, and handle_learn keeps whichever LEARN arrived first because of the /* Avoid re-learn */ guard. Members that heard different values first deliver different payloads.

A bpftrace trace of one such slot followed from ACCEPT through LEARN to DELIVER is in the bug, posted 28 Aug 2026.

The change

Before proposing, verify the reservation still carries the member's own node index. If it does not, drop it through retry_new and take a new one.

This is the same check incr_msgno (xcom_base.cc:668) already makes whenever it advances, with the comment In case site and node number has changed. The client transaction is not lost, it goes out in a slot that does belong to the member.

Testing

In the lab the anomaly goes to zero: 0 stale proposals out of 15,523,048, against 993 predicted from the unpatched rate, and 0 divergences out of 5,667,504 transactions, against 17.1 expected. No run carrying the patch has reproduced the divergence, on two hosts.

Performance: 12 runs per arm across two hosts, load only, no restarts, no emulated latency, comparing the patched plugin against the same source built without the patch and against the stock image. Ratio of median successful operations per 10 s window, bootstrap over runs: patched over unpatched [0.9998, 1.0004], patched over stock [0.9994, 1.0003]. A positive control with a known throttle was detected at [0.9944, 0.9953], so the method resolves differences of about 0.5%. The workload is rate limited, so this measures behaviour under a production-like load and not peak capacity.

I do not have an MTR test for this. The reproduction needs seven members, rolling restarts and sustained write load, which does not fit the standard framework. The reproducer attached to the bug builds that environment and decides the verdict by decoding every member's binary log and comparing all 21 pairs.

[jujose-1]

Hi Matias,
Thank you for the contribution and for working on this fix. I’ve left a few review comments for your consideration. Please take a look when you have a chance.

Regards,
Justin Jose

[matias-sanchez]

Hi @jujose-1, thanks for taking the time to review this and for the comments. Added the requested changes.

[jujose-1]

Hi @matias-sanchez ,

Thank you for updating the patch and addressing the earlier feedback. The production change looks good to me. I’ve left one additional comment regarding the unit-test setup.

I’ll also wait for @tiagoportelajorge 's review and any additional feedback he may have.

Thank you again for working on this.

Regards,
Justin Jose

[kamil-holubicki]

My proposal is to add two debug asserts:

1. Proposer side (offending node) in prepare_push_2p() after the ballot setup: assert(p->proposer.bal.node == p->synode.node). The node can propose only for its synode. Detect the case when the current node index mismatches the index at the synode reservation time.

2. Acceptor side in handle_simple_accept(): assert(m->proposal.cnt != 0 || m->proposal.node == m->synode.node);. In fast path (cnt==0) the node can propose only for its synode.

[matias-sanchez]

Thanks @kamil-holubicki . However, I'm not sure on how to safely add these asserts, because the issue covered is on multi primary and I'm not sure how to add that assert so that it does not fire when paxos_single_leader is ON, as in that mode the synode carries the leader's node index, not the proposer's. @jujose-1 would adding site->max_active_leaders != active_leaders_all be the proper way to validate this? Also, do you recommend adding this on this PR or would you prefer a follow up one?

[jujose-1]

Thanks, @matias-sanchez , for updating the unit test. The revised test now uses valid memberships and verifies the same reservation across the view change. This looks good to me.

Thanks also, @kamil-holubicki, for the suggestions.

@jujose-1 would adding site->max_active_leaders != active_leaders_all be the proper way to validate this? Also, do you recommend adding this on this PR or would you prefer a follow up one?

My understanding is that the proposed assertions would cover the scenario reported in this bug, but they may not hold for remotely allocated synodes, where the synode can legitimately carry the allocating leader’s node index rather than the proposer’s.

Given that the reported issue involves a locally allocated reservation, and the current fix scopes the ownership validation to that path, I suggest keeping this PR focused on that change. The acceptor side does not currently have the allocation provenance needed to apply the same invariant safely, so I would prefer not to add the proposed assertions here.