diff --git a/design/server/100-async-rollback-during-binlog-recovery.md b/design/server/100-async-rollback-during-binlog-recovery.md new file mode 100644 index 000000000000..528a613e11e3 --- /dev/null +++ b/design/server/100-async-rollback-during-binlog-recovery.md @@ -0,0 +1,245 @@ +# Roll Back Prepared Transactions Asynchronously During Binlog Recovery + +Primary GitHub Issue: 100 +Design and implementation PR: 711 + +## Description + +Binlog recovery resolves internal transactions that were prepared in +a storage engine before the server stopped. If the binlog does not contain +the transaction's commit decision (`Xid_log_event`), the transaction must be +rolled back. Until now, the server has invoked `rollback_by_xid()` in the startup +thread and waited for the rollback to complete. Undoing a large transaction +row by row can therefore keep the server unavailable for hours. + +This change adds an optional recovery-specific storage-engine callback to +perform a fast rollback handoff. InnoDB uses it to durably change the state of +a prepared DML transaction back to active. The existing InnoDB recovery +rollback thread then does the expensive row-by-row undo work in the background +while server startup continues. The final transaction outcome does not change. + +## Functional Requirements + +FR1. The time binlog recovery spends rolling back a recovered prepared DML +transaction SHOULD be independent of the transaction's size. Rolling back a +large transaction SHOULD delay startup by approximately the same amount as +rolling back a small transaction. + +## Non-Functional Requirements + +NFR1. Binlog recovery MUST preserve the transaction decision: an internal +prepared transaction absent from the recovered binlog commit set must +eventually be rolled back. + +## High Level Architecture + +### Asynchronous rollback model + +During binlog recovery, an internal transaction that is prepared in InnoDB but +has no commit decision in the binlog must be rolled back. A prepared +transaction cannot be handled by the existing InnoDB recovery rollback thread, +so synchronous recovery performs the complete rollback in the startup thread. + +The key idea is to change the recovered transaction from the prepared state +back to the active state after binlog recovery makes the rollback decision. +The existing InnoDB recovery rollback thread already rolls back recovered +active transactions. It can therefore perform the expensive row-by-row undo in +the background, allowing startup to continue after the fast state transition. + +```text +PREPARED + | + | binlog recovery decides ROLLBACK + v +ACTIVE (persisted) + | + | InnoDB recovery rollback thread + v +ROLLED BACK +``` + +This changes where and when the undo work runs, but does not change the +transaction coordinator's rollback decision or the final transaction outcome. + +### Persisting the active state + +Making this transition only in memory is not crash-safe. If the server stops +again after binlog recovery advances, the next startup could find the +transaction prepared in InnoDB, but the binlog information needed to resolve +it would no longer be available. + +The prepared-to-active transition must therefore be persisted before binlog +recovery considers the rollback handoff complete. InnoDB records the active +undo-log state in redo and flushes that redo before changing the in-memory +transaction state. If the server crashes before the state has been persisted, +binlog recovery can retry the rollback decision. If the server crashes after +the state has been persisted, the transaction is recovered as active and +the background rollback thread continues the rollback. + +### Storage-engine integration + +The server introduces an optional recovery-specific rollback callback to the +storage-engine interface. Transaction-coordinator recovery invokes it after +deciding that an internal prepared transaction must be rolled back. InnoDB +implements the callback by performing the crash-safe prepared-to-active +transition described above and returning without doing row-by-row undo. + +An engine without this callback continues to use the existing synchronous +`rollback_by_xid()` path. InnoDB also keeps DDL transaction rollback synchronous +because later DDL recovery depends on its dictionary and physical-file effects +being settled before startup continues. + +## Low Level Design + +### Storage-engine interface + +`handlerton` gains the following optional callback type and member: + +```cpp +typedef xa_status_code (*recover_rollback_by_xid_t)(handlerton *hton, + XID *xid); + +recover_rollback_by_xid_t recover_rollback_by_xid; +``` +The callback has the same parameters and return values as the +`rollback_by_xid` callback. + +The server calls this callback only during transaction-coordinator recovery, +after deciding that an internal prepared XID must be rolled back. A storage +engine may complete the rollback in the callback or make the rollback durable +and delegate its execution to engine recovery. If the callback is null, +the server falls back to rollback_by_xid(). + +The callback returns the existing `xa_status_code`. InnoDB returns `XAER_NOTA` +when it cannot find the XID and `XA_OK` after a successful handoff. Errors are +handled by the existing XA recovery error path. + +### InnoDB callback + +`innobase_recover_rollback_by_xid()` performs these steps: + +1. Transactions that performed DDL continue to be rolled back synchronously. +2. For DML transactions, it changes each existing undo log from prepared to + active. +3. It flushes the redo log records for these changes to disk. +4. It adds every table recorded in the transaction's `mod_tables` set to the + recovery rollback thread's MDL acquisition list. The rollback thread + acquires shared metadata locks on these tables before server startup + continues, preventing concurrent DDL while undo is running. +5. Under `trx_sys_mutex`, it changes the in-memory transaction state from + `TRX_STATE_PREPARED` to `TRX_STATE_ACTIVE` and decrements + `trx_sys->n_prepared_trx`. + +The existing recovery rollback thread finds the active recovered transaction +and performs normal undo. No new rollback algorithm or worker pool is added. + +### Recovery-thread startup ordering + +The recovery rollback thread is created by +`srv_start_threads_after_ddl_recovery()`, which is called from InnoDB's +`post_recover` callback. The server invokes storage-engine `post_recover` +callbacks only after `tc_log->open()` has completed transaction-coordinator and +binlog recovery. Therefore every prepared DML transaction selected for +asynchronous rollback has already been changed to active before the recovery +rollback thread starts its first scan. + +### Forced-recovery behavior + +With `innodb_force_recovery` at +`SRV_FORCE_NO_BACKGROUND` (level 2) or higher, the callback still durably +changes an internal prepared DML transaction to active during TC recovery. +However, `innobase_post_recover()` returns without creating the recovery +rollback thread, so the transaction is not rolled back during that server +run. A later restart with `innodb_force_recovery` below level 2 recovers the +active transaction and completes its rollback in the background. This matches +the existing behavior of transactions that were already active at the time of +the original crash. + +The adjacent `srv_read_only_mode` early-return branch does not create another +supported deferred-rollback case: InnoDB rejects read-only startup when crash +recovery is required. + +### Persisted data and compatibility + +The design reuses existing undo-log states and redo operations. It adds no data +dictionary objects, file-format fields, binlog events, or redo record types. +Existing databases need no upgrade step, and older servers do not encounter a +new on-disk representation. + +There is no change to external XA recovery: the new callback is selected for +internal transactions by the existing binlog recovery decision path. There is +also no change to security checks, SQL interfaces, replication protocols, or +user-visible configuration. The interaction with the existing +`innodb_force_recovery` setting is described above. + +### Performance and resource use + +The startup thread still pays for the undo-state mini-transaction +and a synchronous redo flush for each handed-off DML transaction. It no longer +waits for row-by-row undo. The single recovery rollback thread is unchanged, +so rollback execution and resource use keep their current characteristics. + + +### Observability and diagnostics + +No new status variable or Performance Schema instrument is introduced. While +undo is in progress, the recovered transaction remains visible in +`INFORMATION_SCHEMA.INNODB_TRX` with the state `ROLLING BACK`. Existing +InnoDB rollback progress messages and XA recovery error reporting remain in +use. + +### Affected source areas + +- `sql/handler.h`: optional `handlerton` callback. +- `sql/xa/recovery.cc`: callback selection for rollback decisions. +- `storage/innobase/handler/ha_innodb.cc`: InnoDB registration and durable + handoff implementation. +- `storage/innobase/trx/trx0roll.cc`: recovered-transaction handling and test + synchronization. +- `storage/innobase/trx/trx0trx.cc`: scheduling tables for metadata-lock + acquisition before background rollback. + +## Alternatives Considered +None + +## Testing + +The debug-only MTR test +`binlog.binlog_recover_async_rollback_trx` exercises the following scenarios: + +1. **Prepared DML rollback and DDL exclusion (FR1, NFR1):** crash after InnoDB + flushes the prepared record but before the XID reaches the binlog. Pause the + background rollback after restart and verify that a concurrent `DROP TABLE` + waits for metadata lock. Crash again before the DDL can execute, restart + normally, and verify that the uncommitted row disappears. Reusing the + affected key verifies that no transaction or lock remains. +2. **Crash after a durable handoff (NFR1):** create a prepared DML transaction + whose XID is absent from the binlog, then restart and pause its background + rollback after binlog recovery has persisted the prepared-to-active + transition. Verify through `INNODB_TRX` that it is `ROLLING BACK`, crash + again, and verify that a normal restart recovers it as active and completes + the rollback. +3. **Crash before the handoff begins (NFR1):** use the + `crash_before_recover_rollback_undo_state_change` injection point to crash + before either undo log is changed. The next restart must recover the + transaction as prepared, retry the rollback decision, roll back the update, + and release the transaction's row lock. +4. **Synchronous DDL fallback:** crash after an atomic `CREATE TABLE` is + prepared in InnoDB but before its XID reaches the binlog. Pause background + recovery rollback before it applies undo and verify that binlog recovery + has already rolled back the DDL synchronously, leaving no recovered + transaction or table. Then verify that the same table name can be created + again. + +Existing XA and binlog recovery tests continue to cover the null-callback +fallback and synchronous recovery behavior, and existing InnoDB DDL recovery +tests still validate that DDL is settled during startup. +The test uses debug synchronization points and is excluded from Valgrind and +crash-reporter runs. + +## References + +- Primary GitHub Issue: #100 +- Design and implementation PR: #711 +- [MySQL Bug #114053](https://bugs.mysql.com/bug.php?id=114053) +- [MariaDB: Rollback Prepared Transactions Asynchronously During Binlog Crash Recovery](https://mariadb.com/resources/blog/rollback-prepared-transactions-asynchronously-during-binlog-crash-recovery/) diff --git a/mysql-test/suite/binlog/r/binlog_recover_async_rollback_trx.result b/mysql-test/suite/binlog/r/binlog_recover_async_rollback_trx.result new file mode 100644 index 000000000000..f78b12544e8e --- /dev/null +++ b/mysql-test/suite/binlog/r/binlog_recover_async_rollback_trx.result @@ -0,0 +1,91 @@ +CREATE TABLE t1 (id INT PRIMARY KEY, data INT) ENGINE = InnoDB; +INSERT INTO t1 VALUES (0, 1); +# +# 1. Prepared DML transactions without binlog XIDs are rolled back. +# +SET debug_sync = "after_flush_engine_log SIGNAL prepared1 WAIT_FOR continue"; +INSERT INTO t1 VALUES (1, 1); +SET debug_sync = "now WAIT_FOR prepared1"; +# Kill the server +# restart: --debug=d,wait_in_recv_rollback +# Expect (0, 1), (1, 1) +SET SESSION TRANSACTION ISOLATION LEVEL READ UNCOMMITTED; +SELECT * FROM t1 ORDER BY id; +id data +0 1 +1 1 +DROP TABLE t1; +# Kill the server +# restart +# Expect (0, 1) +SET SESSION TRANSACTION ISOLATION LEVEL READ UNCOMMITTED; +SELECT * FROM t1 ORDER BY id; +id data +0 1 +INSERT INTO t1 VALUES (1, 1); +UPDATE t1 SET data = data + 1 WHERE id = 0; +# Expect (0, 2), (1, 1) +SELECT * FROM t1 ORDER BY id; +id data +0 2 +1 1 +# +# 2. Recovered prepared transactions survive a crash after handoff. +# +INSERT INTO t1 +WITH RECURSIVE seq(n) AS ( +SELECT 1 UNION ALL SELECT n + 1 FROM seq WHERE n < 256 +) +SELECT n + 1, 1 FROM seq; +SET debug_sync = "after_flush_engine_log SIGNAL prepared2 WAIT_FOR continue"; +UPDATE t1 SET data = 10; +SET debug_sync = "now WAIT_FOR prepared2"; +# Kill the server +# restart: --debug=d,wait_in_recv_rollback +SELECT COUNT(*) FROM information_schema.innodb_trx +WHERE trx_state = "ROLLING BACK"; +COUNT(*) +1 +# Kill the server +# restart +# +# 3. Crash before the async rollback undo-state change. +# +SET debug_sync = "after_flush_engine_log SIGNAL prepared3 WAIT_FOR continue"; +UPDATE t1 SET data = data + 1 WHERE id = 0; +SET debug_sync = "now WAIT_FOR prepared3"; +# Kill the server +# restart_abort: --debug=d,crash_before_recover_rollback_undo_state_change +# restart +SELECT data FROM t1 WHERE id = 0; +data +2 +UPDATE t1 SET data = data + 1 WHERE id = 0; +SELECT data FROM t1 WHERE id = 0; +data +3 +# +# 4. Recovered prepared DDL transactions are rolled back synchronously. +# +START TRANSACTION; +UPDATE t1 SET data = data + 1 WHERE id = 0; +SET debug_sync = "after_flush_engine_log SIGNAL prepared4 WAIT_FOR continue"; +CREATE TABLE t_ddl (id INT PRIMARY KEY) ENGINE = InnoDB; +SET debug_sync = "now WAIT_FOR prepared4"; +# Kill the server +# restart: --debug=d,wait_in_recv_rollback +SELECT COUNT(*) FROM information_schema.innodb_trx +WHERE trx_state = "ROLLING BACK"; +COUNT(*) +1 +SELECT COUNT(*) FROM information_schema.innodb_trx; +COUNT(*) +1 +SET GLOBAL debug = "-d,wait_in_recv_rollback"; +SELECT COUNT(*) FROM information_schema.tables +WHERE table_schema = DATABASE() AND table_name = 't_ddl'; +COUNT(*) +0 +CREATE TABLE t_ddl (id INT PRIMARY KEY) ENGINE = InnoDB; +DROP TABLE t_ddl; +DROP TABLE t1; diff --git a/mysql-test/suite/binlog/t/binlog_recover_async_rollback_trx-master.opt b/mysql-test/suite/binlog/t/binlog_recover_async_rollback_trx-master.opt new file mode 100644 index 000000000000..cef79bc8585a --- /dev/null +++ b/mysql-test/suite/binlog/t/binlog_recover_async_rollback_trx-master.opt @@ -0,0 +1 @@ +--force-restart diff --git a/mysql-test/suite/binlog/t/binlog_recover_async_rollback_trx.test b/mysql-test/suite/binlog/t/binlog_recover_async_rollback_trx.test new file mode 100644 index 000000000000..bf3c3c0da359 --- /dev/null +++ b/mysql-test/suite/binlog/t/binlog_recover_async_rollback_trx.test @@ -0,0 +1,221 @@ +################################################################################ +# Test asynchronous rollback of recovered prepared transactions during binlog +# crash recovery. +# +# Scenario 1 creates an internal XA prepared DML transaction, flushes its +# prepared record to the engine log, and crashes before its XID reaches the +# binlog. Recovery should mark it for rollback, return startup quickly, and +# hold shared metadata locks until background rollback finishes. Concurrent DDL +# must wait, and the table must be left with only committed rows. +# +# Scenario 2 creates another internal XA prepared DML transaction. After binlog +# recovery durably hands it off to the background rollback thread, the server +# crashes again. The next restart should recover the transaction as active and +# finish rolling it back. +# +# Scenario 3 crashes during binlog recovery before the prepared transaction's +# async rollback undo-state change begins. The next restart should still +# recover cleanly and roll back the transaction. +# +# Scenario 4 creates a prepared atomic DDL transaction whose XID is absent from +# the binlog, plus an active DML transaction as a positive control. Recovery +# should roll back the DDL synchronously while the DML proves that asynchronous +# rollback is paused. +################################################################################ + +--source include/have_debug.inc +--source include/have_debug_sync.inc +--source include/have_binlog_format_row.inc +--source include/not_valgrind.inc +--source include/not_crashrep.inc + +CREATE TABLE t1 (id INT PRIMARY KEY, data INT) ENGINE = InnoDB; +INSERT INTO t1 VALUES (0, 1); + +--echo # +--echo # 1. Prepared DML transactions without binlog XIDs are rolled back. +--echo # + +# Stop a transaction after the prepared record is flushed to the engine log and +# before the XID is written to the binlog. Binlog recovery must roll it back. +--connect(con1, localhost, root,,) +SET debug_sync = "after_flush_engine_log SIGNAL prepared1 WAIT_FOR continue"; +--send INSERT INTO t1 VALUES (1, 1) + +--connection default +SET debug_sync = "now WAIT_FOR prepared1"; +--source include/kill_mysqld.inc + +# The pending client commands never complete because the server is killed. +--disconnect con1 + +# Pause before the recovery rollback thread applies undo. This keeps the +# transaction and its shared MDL alive while the test verifies that DDL waits. +--let $restart_parameters= restart: --debug=d,wait_in_recv_rollback +--source include/start_mysqld.inc + +# Startup has completed while the recovered transaction is still rolling back. +# READ UNCOMMITTED should expose its inserted row until undo resumes. +--echo # Expect (0, 1), (1, 1) +SET SESSION TRANSACTION ISOLATION LEVEL READ UNCOMMITTED; +SELECT * FROM t1 ORDER BY id; + +# The recovery rollback thread must acquire shared MDL before startup +# continues. DROP TABLE must wait while the rollback is paused. +--connect(con_ddl, localhost, root,,) +--send DROP TABLE t1 + +--connection default +--let $wait_condition= SELECT COUNT(*) = 1 FROM INFORMATION_SCHEMA.PROCESSLIST WHERE STATE = "Waiting for table metadata lock" AND INFO = "DROP TABLE t1" +--source include/wait_condition.inc + +# Stop the server before the pending DROP can run, then complete recovery +# normally. The pending client command never completes because of the stop. +--source include/kill_mysqld.inc +--disconnect con_ddl + +--let $restart_parameters= restart +--source include/start_mysqld.inc + +# Startup should complete while InnoDB finishes the rollback in the background. +--let $wait_condition= SELECT COUNT(*) = 0 FROM information_schema.innodb_trx +--source include/wait_condition.inc + +# The prepared transaction should be gone: the insert of id=1 is absent. +--echo # Expect (0, 1) +SET SESSION TRANSACTION ISOLATION LEVEL READ UNCOMMITTED; +SELECT * FROM t1 ORDER BY id; + +# Reusing the same key and row verifies that no lingering prepared transaction +# still owns locks or data changes from the crashed transactions. +INSERT INTO t1 VALUES (1, 1); +UPDATE t1 SET data = data + 1 WHERE id = 0; + +--echo # Expect (0, 2), (1, 1) +SELECT * FROM t1 ORDER BY id; + +--echo # +--echo # 2. Recovered prepared transactions survive a crash after handoff. +--echo # + +# Add enough rows for the recovery rollback thread to be observable while the +# wait_in_recv_rollback debug hook pauses before applying undo. +INSERT INTO t1 +WITH RECURSIVE seq(n) AS ( + SELECT 1 UNION ALL SELECT n + 1 FROM seq WHERE n < 256 +) +SELECT n + 1, 1 FROM seq; + +# Stop a large transaction after its prepared record reaches the engine log but +# before its XID reaches the binlog. +--connect(con2, localhost, root,,) +SET debug_sync = "after_flush_engine_log SIGNAL prepared2 WAIT_FOR continue"; +--send UPDATE t1 SET data = 10 + +--connection default +SET debug_sync = "now WAIT_FOR prepared2"; +--source include/kill_mysqld.inc +--disconnect con2 + +# The server should become available even though the recovered transaction is +# still rolling back in the background. +--let $restart_parameters= restart: --debug=d,wait_in_recv_rollback +--source include/start_mysqld.inc +--let $wait_condition= SELECT COUNT(*) = 1 FROM information_schema.innodb_trx WHERE trx_state = "ROLLING BACK" +--source include/wait_condition.inc +SELECT COUNT(*) FROM information_schema.innodb_trx +WHERE trx_state = "ROLLING BACK"; + +# Binlog recovery has persisted the prepared-to-active handoff. Kill while the +# recovery rollback thread is paused; the next restart must recover the +# transaction as active and complete its rollback. +--source include/kill_mysqld.inc + +--let $restart_parameters= restart +--source include/start_mysqld.inc + +--let $wait_condition= SELECT COUNT(*) = 0 FROM information_schema.innodb_trx +--source include/wait_condition.inc + +--echo # +--echo # 3. Crash before the async rollback undo-state change. +--echo # + +# Prepare another DML transaction whose XID is missing from the binlog. The +# recovery callback will be invoked on restart to change it back to ACTIVE. +--connect(con3, localhost, root,,) +SET debug_sync = "after_flush_engine_log SIGNAL prepared3 WAIT_FOR continue"; +--send UPDATE t1 SET data = data + 1 WHERE id = 0 + +--connection default +SET debug_sync = "now WAIT_FOR prepared3"; +--source include/kill_mysqld.inc +--disconnect con3 + +# Crash before modifying either undo log. The next startup must find the +# transaction still prepared and retry the rollback decision safely. +--let $restart_parameters= --debug=d,crash_before_recover_rollback_undo_state_change +--source include/start_mysqld_expecting_crash.inc + +--let $restart_parameters= restart +--source include/start_mysqld.inc + +# The crashed prepared update should eventually be rolled back. +--let $wait_condition= SELECT COUNT(*) = 0 FROM information_schema.innodb_trx +--source include/wait_condition.inc + +# id=0 should still have the value committed before scenario 3. Updating it +# again verifies that no recovered transaction still blocks the row. +SELECT data FROM t1 WHERE id = 0; +UPDATE t1 SET data = data + 1 WHERE id = 0; +SELECT data FROM t1 WHERE id = 0; + +--echo # +--echo # 4. Recovered prepared DDL transactions are rolled back synchronously. +--echo # + +# Stop an atomic DDL transaction after its prepared record reaches the engine +# log but before its XID reaches the binlog. +# First leave a DML transaction active with a known undo record. Crash recovery +# must roll it back in the background, making it a positive control for the +# wait_in_recv_rollback pause. +--connect(con4_dml, localhost, root,,) +START TRANSACTION; +UPDATE t1 SET data = data + 1 WHERE id = 0; + +--connect(con4, localhost, root,,) +SET debug_sync = "after_flush_engine_log SIGNAL prepared4 WAIT_FOR continue"; +--send CREATE TABLE t_ddl (id INT PRIMARY KEY) ENGINE = InnoDB + +--connection default +SET debug_sync = "now WAIT_FOR prepared4"; +--source include/kill_mysqld.inc +--disconnect con4 +--disconnect con4_dml + +# Pause background recovery rollback before it applies undo. The DML transaction +# above confirms that the pause is active; if the DDL is incorrectly handed off +# asynchronously, a second recovered transaction will be visible even if its +# undo log is empty. +--let $restart_parameters= restart: --debug=d,wait_in_recv_rollback +--source include/start_mysqld.inc + +# Exactly one rolling-back transaction proves that the DML positive control is +# paused and that the DDL transaction was already rolled back synchronously. +--let $wait_condition= SELECT COUNT(*) = 1 FROM information_schema.innodb_trx WHERE trx_state = "ROLLING BACK" +--source include/wait_condition.inc +SELECT COUNT(*) FROM information_schema.innodb_trx +WHERE trx_state = "ROLLING BACK"; +SELECT COUNT(*) FROM information_schema.innodb_trx; + +# The DDL fallback leaves no table behind. Reusing its name also checks DD +# cleanup after the synchronous rollback. +SET GLOBAL debug = "-d,wait_in_recv_rollback"; +--let $wait_condition= SELECT COUNT(*) = 0 FROM information_schema.innodb_trx +--source include/wait_condition.inc +SELECT COUNT(*) FROM information_schema.tables +WHERE table_schema = DATABASE() AND table_name = 't_ddl'; +CREATE TABLE t_ddl (id INT PRIMARY KEY) ENGINE = InnoDB; +DROP TABLE t_ddl; + +DROP TABLE t1; diff --git a/sql/binlog/binlog_tc_log.cc b/sql/binlog/binlog_tc_log.cc index 3b0990d16e2e..c191b4fbd64b 100644 --- a/sql/binlog/binlog_tc_log.cc +++ b/sql/binlog/binlog_tc_log.cc @@ -100,6 +100,7 @@ THD *Binlog_tc_log::fetch_and_process_flush_stage_queue( flushing them to binary log. */ ha_flush_logs(true); + DEBUG_SYNC(first_seen, "after_flush_engine_log"); } /* diff --git a/sql/handler.h b/sql/handler.h index 76cb423d3844..bb7f58e27b7e 100644 --- a/sql/handler.h +++ b/sql/handler.h @@ -1491,6 +1491,8 @@ typedef xa_status_code (*commit_by_xid_t)(handlerton *hton, XID *xid); typedef xa_status_code (*rollback_by_xid_t)(handlerton *hton, XID *xid); +typedef xa_status_code (*recover_rollback_by_xid_t)(handlerton *hton, XID *xid); + /** Instructs the storage engine to mark the externally coordinated transactions identified by the XID parameters as prepared in the server @@ -2898,6 +2900,17 @@ struct handlerton { recover_prepared_in_tc_t recover_prepared_in_tc; commit_by_xid_t commit_by_xid; rollback_by_xid_t rollback_by_xid; + /* + recover_rollback_by_xid is optional. If set, it will be called instead of + rollback_by_xid when transactions should be rolled back at server startup. + + This function should just change the transaction's state from prepared to + active before returning. The actual rollback should then happen + asynchronously, for example in a background thread. This way, rollbacks + that take a long time to complete will not block server startup, and the + database becomes available sooner to serve user queries. + */ + recover_rollback_by_xid_t recover_rollback_by_xid; set_prepared_in_tc_t set_prepared_in_tc; set_prepared_in_tc_by_xid_t set_prepared_in_tc_by_xid; create_t create; diff --git a/sql/xa/recovery.cc b/sql/xa/recovery.cc index 2dbca9a0963f..114ec696068a 100644 --- a/sql/xa/recovery.cc +++ b/sql/xa/recovery.cc @@ -261,6 +261,9 @@ void recover_one_internal_trx(xarecover_st const &info, handlerton &ht, enum xa_status_code exec_status; if (DBUG_EVALUATE_IF("xa_recovery_error_reporting", true, false)) exec_status = ::generate_xa_recovery_error(); + else if (ht.recover_rollback_by_xid != nullptr) + exec_status = + ht.recover_rollback_by_xid(&ht, const_cast(&xa_trx.id)); else exec_status = ht.rollback_by_xid(&ht, const_cast(&xa_trx.id)); diff --git a/storage/innobase/handler/ha_innodb.cc b/storage/innobase/handler/ha_innodb.cc index 920a4442c814..2e42d4d7d53c 100644 --- a/storage/innobase/handler/ha_innodb.cc +++ b/storage/innobase/handler/ha_innodb.cc @@ -199,6 +199,7 @@ this program; if not, write to the Free Software Foundation, Inc., #include "trx0rseg.h" #include "trx0sys.h" #include "trx0trx.h" +#include "trx0undo.h" #include "trx0xa.h" #include "ut0mem.h" #include "ut0test.h" @@ -1567,6 +1568,13 @@ static xa_status_code innobase_rollback_by_xid( handlerton *hton, /*!< in: InnoDB handlerton */ XID *xid); /*!< in: X/Open XA transaction identification */ +/** In binlog recovery, persistently mark that a transaction will be rolled +back. +@param[in] hton InnoDB handlerton +@param[in] xid Internal MySQL XID identifier + @return 0 or error number */ +static xa_status_code innobase_recover_rollback_by_xid(handlerton *hton, + XID *xid); /** This function is used to write mark an X/Open XA distributed transaction as been prepared in the server transaction coordinator @param[in] hton InnoDB handlerton @@ -5467,6 +5475,7 @@ static int innodb_init(void *p) { innobase_hton->recover_prepared_in_tc = innobase_xa_recover_prepared_in_tc; innobase_hton->commit_by_xid = innobase_commit_by_xid; innobase_hton->rollback_by_xid = innobase_rollback_by_xid; + innobase_hton->recover_rollback_by_xid = innobase_recover_rollback_by_xid; innobase_hton->set_prepared_in_tc = innobase_set_prepared_in_tc; innobase_hton->set_prepared_in_tc_by_xid = innobase_set_prepared_in_tc_by_xid; innobase_hton->create = innobase_create_handler; @@ -20458,6 +20467,24 @@ static xa_status_code innobase_commit_by_xid( } } +/** Roll back and release a prepared transaction. +@param[in,out] trx transaction selected for rollback +@return XA_OK or XAER_RMERR */ +static xa_status_code innobase_rollback_prepared_trx(trx_t *trx) { + int ret; + { + TrxInInnoDB trx_in_innodb(trx); + + ret = innobase_rollback_trx(trx); + } + + trx_deregister_from_2pc(trx); + ut_ad(!trx->will_lock); + trx_free_for_background(trx); + + return (ret != 0 ? XAER_RMERR : XA_OK); +} + /** This function is used to rollback one X/Open XA distributed transaction which is in the prepared state @return 0 or error number */ @@ -20470,22 +20497,97 @@ static xa_status_code innobase_rollback_by_xid( trx_t *trx = trx_get_trx_by_xid(xid); - if (trx != nullptr) { - int ret; - { - TrxInInnoDB trx_in_innodb(trx); + return trx == nullptr ? XAER_NOTA : innobase_rollback_prepared_trx(trx); +} - ret = innobase_rollback_trx(trx); - } +static xa_status_code innobase_recover_rollback_by_xid( + handlerton *hton, /*!< in: InnoDB handlerton */ + XID *xid) /*!< in: X/Open XA transaction identification */ +{ + assert(hton == innodb_hton_ptr); - trx_deregister_from_2pc(trx); - ut_ad(!trx->will_lock); - trx_free_for_background(trx); + /* + trx_get_trx_by_xid() sets trx's xid to null. Thus only one call for any + given XID can find the transaction. Subsequent calls by other threads + would return nullptr. That is what guarantees that no other thread can be + modifying the state of the transaction at this point. + */ + trx_t *trx = trx_get_trx_by_xid(xid); - return (ret != 0 ? XAER_RMERR : XA_OK); - } else { - return (XAER_NOTA); + if (trx == nullptr) { + return XAER_NOTA; } + + if (trx->ddl_operation) { + return innobase_rollback_prepared_trx(trx); + } + + /* A recovered prepared transaction necessarily has redo undo logs. */ + ut_ad(trx->rsegs.m_redo.rseg != nullptr && trx_is_redo_rseg_updated(trx)); + + DBUG_EXECUTE_IF("crash_before_recover_rollback_undo_state_change", + DBUG_SUICIDE();); + + trx_undo_ptr_t *undo_ptr = &trx->rsegs.m_redo; + mtr_t mtr; + + mtr.start(); + + trx->rsegs.m_redo.rseg->latch(); + + if (undo_ptr->insert_undo != nullptr) { + trx_undo_set_state_at_prepare(trx, undo_ptr->insert_undo, true, &mtr); + } + + if (undo_ptr->update_undo != nullptr) { + /* A binlog-internal prepared transaction never carries a GTID in undo: + TRX_UNDO_FLAG_GTID is written only at commit + (trx_write_serialisation_history()), and check_gtid_prepare() returns + false for trx_is_mysql_xa(), so trx_prepare_low() does not set + TRX_UNDO_FLAG_XA_PREPARE_GTID either. */ + ut_ad((undo_ptr->update_undo->flag & + (TRX_UNDO_FLAG_GTID | TRX_UNDO_FLAG_XA_PREPARE_GTID)) == 0); + trx_undo_set_state_at_prepare(trx, undo_ptr->update_undo, true, &mtr); + } + + trx->rsegs.m_redo.rseg->unlatch(); + + mtr.commit(); + + const lsn_t commit_lsn = mtr.commit_lsn(); + ut_ad(commit_lsn > 0 || !mtr_t::s_logging.is_enabled()); + if (commit_lsn > 0) { + /* This flush must happen before MYSQL_BIN_LOG::open_binlog() clears + LOG_EVENT_BINLOG_IN_USE_F. Otherwise, after a crash, the next startup + could reject recovery because no usable binlog recovery information + remains for this prepared transaction. */ + log_write_up_to(*log_sys, commit_lsn, true); + } + + ut_ad(!trx->mod_tables.empty()); + + /* + trx_resurrect_locks() does not add the modified tables of a recovered + prepared transaction to to_rollback_trx_tables. This transaction is about + to become ACTIVE and be rolled back in the background, so add its modified + tables now for the recovery rollback thread to acquire MDL. + */ + for (const auto table : trx->mod_tables) { + to_rollback_trx_tables.emplace_back(trx->id, table->id); + } + + /* + The above undo state changes are durable before the transaction state is + changed from PREPARED to ACTIVE. The recovery rollback thread will then + roll back this transaction. + */ + trx_sys_mutex_enter(); + ut_a(trx_sys->n_prepared_trx > 0); + trx->state.store(TRX_STATE_ACTIVE, std::memory_order_relaxed); + --trx_sys->n_prepared_trx; + trx_sys_mutex_exit(); + + return XA_OK; } static int innobase_set_prepared_in_tc(handlerton *hton, THD *thd) { diff --git a/storage/innobase/include/trx0trx.h b/storage/innobase/include/trx0trx.h index 60642bd28df4..f893758cf835 100644 --- a/storage/innobase/include/trx0trx.h +++ b/storage/innobase/include/trx0trx.h @@ -754,6 +754,9 @@ struct trx_t { Recovered XA: * NOT_STARTED -> PREPARED -> COMMITTED -> (freed) + Recovered internal transaction followed by recover_rollback_by_xid: + * NOT_STARTED -> PREPARED -> ACTIVE -> COMMITTED -> (freed) + XA (2PC) (shutdown or disconnect before ROLLBACK or COMMIT): * NOT_STARTED -> PREPARED -> (freed) @@ -764,8 +767,11 @@ struct trx_t { XA (2PC) transactions are always treated as non-autocommit. - Transitions to ACTIVE or NOT_STARTED occur when - !in_rw_trx_list (no trx_sys->mutex needed). + Transitions to ACTIVE or NOT_STARTED normally occur when + !in_rw_trx_list (no trx_sys->mutex needed). During recovery, + recover_rollback_by_xid may transition a recovered prepared transaction to + ACTIVE while it remains in rw_trx_list; this transition is protected by + trx_sys->mutex. Autocommit non-locking read-only transactions move between states without holding any mutex. They are !in_rw_trx_list. diff --git a/storage/innobase/srv/srv0start.cc b/storage/innobase/srv/srv0start.cc index 99baae6eb901..3015c48954ab 100644 --- a/storage/innobase/srv/srv0start.cc +++ b/storage/innobase/srv/srv0start.cc @@ -2287,6 +2287,10 @@ void srv_start_threads_after_ddl_recovery() { if (srv_force_recovery < SRV_FORCE_NO_TRX_UNDO && trx_sys_need_rollback()) { /* Rollback all recovered transactions that are not in committed nor in XA PREPARE state. */ + /* The rollback thread must be created only after transaction-coordinator + recovery completes. It does not poll for later handoffs or wait on a + recovery-completion signal. If its creation is moved before TC recovery, + explicit synchronization must be introduced. */ srv_threads.m_trx_recovery_rollback = os_thread_create( trx_recovery_rollback_thread_key, 0, trx_recovery_rollback_thread); diff --git a/storage/innobase/trx/trx0roll.cc b/storage/innobase/trx/trx0roll.cc index bbcd6cee626f..f9b0c4c05b67 100644 --- a/storage/innobase/trx/trx0roll.cc +++ b/storage/innobase/trx/trx0roll.cc @@ -621,6 +621,14 @@ static void trx_rollback_active(trx_t *trx) /*!< in/out: transaction */ que_run_threads(thr); ut_a(roll_node->undo_thr != nullptr); + while (DBUG_EVALUATE_IF("wait_in_recv_rollback", + trx == trx_roll_crash_recv_trx, false)) { + if (srv_shutdown_state.load() >= SRV_SHUTDOWN_RECOVERY_ROLLBACK) { + break; + } + std::this_thread::sleep_for(std::chrono::milliseconds(1)); + } + que_run_threads(roll_node->undo_thr); trx_rollback_finish(thr_get_trx(roll_node->undo_thr)); @@ -722,10 +730,12 @@ void trx_rollback_or_clean_recovered( " of uncommitted transactions"; } - /* Note: For XA recovered transactions, we rely on MySQL to - do rollback. They will be in TRX_STATE_PREPARED state. If the server - is shutdown and they are still lingering in trx_sys_t::trx_list - then the shutdown will hang. */ + /* Recovered external XA transactions remain in TRX_STATE_PREPARED until + the server decides their outcome. Internal prepared transactions selected + for rollback by binlog recovery are changed to TRX_STATE_ACTIVE through + recover_rollback_by_xid and are rolled back by this thread. If prepared + transactions still linger in trx_sys_t::trx_list at shutdown, shutdown will + hang. */ /* Loop over the transaction list as long as there are recovered transactions to clean up or recover. */ diff --git a/storage/innobase/trx/trx0trx.cc b/storage/innobase/trx/trx0trx.cc index a41a789fbf33..8fde95fac3b8 100644 --- a/storage/innobase/trx/trx0trx.cc +++ b/storage/innobase/trx/trx0trx.cc @@ -853,7 +853,9 @@ void trx_resurrect_locks(bool all) { } DICT_TF2_FLAG_SET(table, DICT_TF2_RESURRECT_PREPARED); - /* We don't rollback DDL or XA prepared transaction in background */ + /* DDL transactions are not rolled back in background. Prepared + transactions are scheduled only after TC recovery decides to roll them + back. */ if (trx->ddl_operation || is_prepared) { lock_table_ix_resurrect(table, trx); ib::info(ER_IB_RESURRECT_ACQUIRE_TABLE_LOCK, ulong(table->id),