PS-11530 [8.4] Members of one Group Replication group assign different GNOs to the same transaction

[kamil-holubicki]

https://perconadev.atlassian.net/browse/PS-11530

Upstream bug:
Bug#121063: https://bugs.mysql.com/bug.php?id=121063

Problem:
Two members of the same group assign different GNOs to the same transaction, in multi-primary mode, when a member leaves and rejoins during a rolling restart under write load. The group splits into internally consistent sets that disagree on the GTID of the same transaction. It can surface on the group_replication_applier channel as Error_code 1032 or 1062, or stay silent with every member ONLINE and the disagreement present only in the binary logs.

Cause:
A reserved synode is only ours while we still hold the node index it was reserved under.

local_synode_allocator stamps the member's current index into the synode, synode.node = my_nodeno. Still inside reserve_synode_number, the task yields in the while (too_far(*msgno)) loop at TIMED_TASK_WAIT. During that yield, site_install_action reassigns site->nodeno. The reservation still carries the old index, so proposer_task brands and proposes into a slot that now belongs to another node.

The header comment of xcom_base.cc states the rule: only node N may propose a value for synode {X N}. With two proposers on the same slot at cnt=0, acceptor.promise is never raised, since it is assigned in exactly one place, inside handle_simple_prepare, which is the phase 1 decision. Both proposals are accepted, both are learned, and handle_learn keeps whichever LEARN arrived first because of the /* Avoid re-learn */ guard. Members that heard different values first deliver different payloads.

Solution:
Before proposing, verify the reservation still carries the member's own node index. If it does not, drop it through retry_new and take a new one.

This is the same check incr_msgno already makes whenever it advances, with the comment In case site and node number has changed. The client transaction is not lost, it goes out in a slot that does belong to the member.

Ported from upstream MySQL PR
mysql/mysql-server#736
by Matias Sanchez.