InnoDB: Parallel DDL threads race in debug_sync on the shared THD - #713
Open
inikep wants to merge 1 commit into
Open
InnoDB: Parallel DDL threads race in debug_sync on the shared THD#713inikep wants to merge 1 commit into
inikep wants to merge 1 commit into
Conversation
|
Thank you for your pull request and welcome to our community! To contribute, please sign the Oracle Contributor Agreement (OCA).
To sign the OCA, please create an Oracle account and sign the OCA in Oracle's Contributor Agreement Application. When signing the OCA, please provide your GitHub username. After signing the OCA and getting an OCA approval from Oracle, this PR will be automatically updated. If you are an Oracle employee, please make sure that you are a member of the main Oracle GitHub organization, and your membership in this organization is public. |
Author
|
I confirm the code being submitted is offered under the terms of the OCA, and that I am authorized to contribute it |
A debug server can be killed, or an ALTER left hanging forever, by the sync points in storage/innobase/ddl/ddl0builder.cc: several DDL threads execute them concurrently on one THD, and debug_sync is not prepared for that. Reproducer ========== The attached test innodb.ddl_debug_sync_race, part 1, arms "row_log_apply_before" with two activations and runs an online ALTER adding four indexes with innodb_ddl_threads = 4. On 9.7.2 (debug) it kills the server: mysqld: sql/debug_sync.cc:946: void debug_sync_remove_action( st_debug_sync_control*, st_debug_sync_action*): Assertion `dsp_idx < ds_control->ds_active' failed. mysql#10 debug_sync_remove_action sql/debug_sync.cc:946 mysql#11 debug_sync sql/debug_sync.cc:1951 mysql#12 ddl::Builder::finalize storage/innobase/ddl/ddl0builder.cc:1994 mysql#13 ddl::Builder::finish storage/innobase/ddl/ddl0builder.cc:2086 mysql#14 ddl::Loader::Task::operator() ddl/ddl0builder.cc:2180 mysql#15 ddl::Loader::Task_queue::mt_execute ddl/ddl0loader.cc:178 ddl0builder.cc:1994 is DEBUG_SYNC(m_ctx.thd(), "row_log_apply_before"). The same test also hangs instead of crashing, depending on which of the racing threads gets ahead: the ALTER never returns and the waits run into their debug sync timeout. Why several threads share one THD ================================= The tasks driving the builder state machine are pulled off a shared queue by whichever DDL thread is free - Task_queue::mt_execute() pops under m_mutex but runs the task outside it - so an ALTER building several indexes executes setup_sort()/btree_build()/finish() for different builders at the same time on different threads, and the sort tasks of one builder are parallel by design. Loader::load() runs the queue on the connection's own thread too and clears current_thd in the workers it spawns. All of those threads hand the same THD, the owning connection's m_ctx.thd(), to DEBUG_SYNC. debug_sync() does no locking; the facility assumes a sync point is only ever hit by the thread owning the THD. Every step of it races when that assumption is broken, and each step has its own visible symptom: - debug_sync() tests action->activation_count and only then calls debug_sync_execute(), which asserts the same condition before decrementing it. A single thread can never trip that assertion - the test immediately precedes it - two can: both pass the test, then the first consumes the last activation while the second is still entering debug_sync_execute(): sql/debug_sync.cc: Assertion `action->activation_count' failed - Both threads then find activation_count == 0 on the way out and both call debug_sync_remove_action() on the same action. The second one finds ds_active already decremented, which is the assertion in the stack above; without the assertion it would decrement ds_active below zero and shift the action array by a huge count. - A WAIT_FOR clears the signal it woke up on unless NO_CLEAR_EVENT is given. With two threads waiting on the same action, the first one out clears the event and the second one goes back to sleep waiting for a signal that no longer exists - the hang. A DBUG trace (--debug=d,debug_sync,debug_sync_exec,debug_sync_point:i:o) of the reproducer shows two DDL threads inside the one sync point on the one THD, and the second one never resuming: T@113: debug_sync_point: hit: 'row_log_apply_before' T@113: sync_point: 'row_log_apply_before' activation_count: 2 ... T@113: debug_sync_exec: signal 'ddl_parked' at: 'row_log_apply_before' T@113: debug_sync_exec: wait for 'ddl_resume' at: 'row_log_apply_before' T@114: debug_sync_point: hit: 'row_log_apply_before' T@114: sync_point: 'row_log_apply_before' activation_count: 1 ... T@114: debug_sync_exec: signal 'ddl_parked' at: 'row_log_apply_before' T@114: debug_sync_exec: wait for 'ddl_resume' at: 'row_log_apply_before' T@10 : debug_sync_exec: signal 'ddl_resume' at: 'now' T@114: awoke from ddl_resume error: 0 T@114: debug_sync_exec: resume from 'ddl_resume' at: 'row_log_apply_before' T@113: awoke from ddl_resume error: 0 <-- signal already cleared Fix === Serialize the builder's sync points with a mutex, so the first thread executes the action and the ones behind it find it already removed and return without touching the control block. The mutex is per Context, i.e. per DDL statement and hence per THD, and must stay that way. A first attempt with one file-static mutex hangs innodb.innodb-index-online-purge with "debug sync point wait timed out": that test parks one DDL inside row_log_apply_before and only releases it after a second, independent DDL has reached the same sync point, which a lock shared between statements prevents. Threads sharing a THD must be serialized; statements that do not share one must stay independent. All four sync points in the file are covered, not just the one in the stack trace: ddl_btree_build_interrupt, ddl_merge_sort_interrupt and row_log_apply_before/after are the same construct in the same task-driven paths, and the sort tasks are the most parallel of them. The lock is only taken when debug sync is armed, mirroring the DEBUG_SYNC macro's own opt_debug_sync_timeout fast path, and everything compiles away in builds without DEBUG_SYNC, so release builds are unaffected. Test ==== innodb.ddl_debug_sync_race has three parts. Part 1 is the reproducer above. Part 2 is a stress: ten rounds of the same ALTER with a single activation armed, the shape that hits the `action->activation_count' assertion in the field. Part 3 checks that the serialization does not span connections: while one connection is parked in the sync point, another connection's ALTER must still reach it. Verified on a 9.7.2 debug build: unpatched, the test crashed the server with the assertion above 5 times in a --repeat=24 --parallel=8 run (and hung in others); patched, the same run passes 24/24. Every existing test using these sync points passes with the patch: innodb-alter-varchar-debug, innodb-index-online, innodb-index-online-purge, virtual_debug, bulk_create_index_online, ddl_kill, index-create-dml-rollback, gcol.gcol_rollback, plus innodb-alter-debug and innodb-alter-nullable.
inikep
force-pushed
the
ddl-debug-sync-race-oracle-submit
branch
from
August 6, 2026 16:04
3efb3fc to
090ddeb
Compare
gopshank
requested review from
mayprasa and
tjeldvoll
and removed request for
gopshank and
seemasundara
August 6, 2026 17:59
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this change do?
Serializes the four
DEBUG_SYNCcalls instorage/innobase/ddl/ddl0builder.ccwith a per-
ddl::Contextmutex, so the parallel DDL threads that share theconnection's
THDcannot execute one debug sync action at the same time.Why is it needed?
The tasks driving
ddl::Builderare pulled off a shared queue by whichever DDLthread is free (
ddl::Loader::Task_queue::mt_execute()), so an online ALTERbuilding several indexes runs
setup_sort()/btree_build()/finalize()concurrently, and every one of those threads passes the same THD -
ddl::Context::thd()- toDEBUG_SYNC.debug_sync()does no locking; itassumes a sync point is only ever hit by the thread owning the THD.
On a 9.7.2 debug build an ALTER that adds four indexes with
innodb_ddl_threads = 4, withrow_log_apply_beforearmed, kills the server:Every step of
debug_sync()races when two threads share the THD, and each hasits own symptom:
action->activation_counttest, then the first consumesthe last activation while the second is entering
debug_sync_execute()-Assertion 'action->activation_count' failed;activation_count == 0on the way out and both calldebug_sync_remove_action()on the same action - the assertion in the stackabove;
WAIT_FORclears the signal it woke up on unlessNO_CLEAR_EVENTis given,so with two threads waiting on one action the first one out clears the event
and the second waits for a signal that no longer exists - the ALTER hangs.
The commit message contains a DBUG trace
(
--debug=d,debug_sync,debug_sync_exec,debug_sync_point:i:o) showing two DDLthreads inside the one sync point on the one THD, with the second never
resuming.
The mutex is deliberately per context, i.e. per DDL statement and hence per THD.
A single file-static mutex instead hangs
innodb.innodb-index-online-purge,which parks one DDL inside
row_log_apply_beforeand only releases it after asecond, independent DDL has reached the same sync point. It is taken only when
debug sync is armed, mirroring the
DEBUG_SYNCmacro's ownopt_debug_sync_timeoutfast path, and the whole construct compiles awaywithout
ENABLED_DEBUG_SYNC, so release builds are unaffected.How was it tested?
mysql-test/scripts/ci/mtr.shpasses locallysync points -
innodb.innodb-alter-varchar-debug,innodb.innodb-index-online,innodb.innodb-index-online-purge,innodb.virtual_debug,innodb.bulk_create_index_online,innodb.ddl_kill,innodb.index-create-dml-rollback,gcol.gcol_rollback, plusinnodb.innodb-alter-debugandinnodb.innodb-alter-nullableNew test
innodb.ddl_debug_sync_race, in three parts: a reproducer that armsthe sync point with two activations so two builders of one ALTER meet in it, a
stress with a single activation (the shape that hits the
action->activation_countassertion in the field), and a check that theserialization does not span connections.
Measured on a 9.7.2 debug build (gcc 15,
CMAKE_BUILD_TYPE=Debug): unpatched,the new test crashed the server with the assertion above 5 times in a
--repeat=24 --parallel=8run and hung in others; patched, the same run passes24/24.
Re-verified on
trunk(06a5c1c) after the rebase, same build configuration:the same
--repeat=24 --parallel=8run passes 24/24 with the patch, and allten existing tests listed above pass. The four sync points and the surrounding
builder code are identical to 9.7.2 there, so the unpatched failure modes apply
unchanged; the crash counts quoted above were measured on the 9.7.2 build.
Contributor checklist
scripts/ci/format.sh)AI assistance
Claude Code was used to reproduce the failure, analyse the debug_sync control
block races, write the MTR test and draft this description. The diagnosis was
confirmed against a DBUG trace of the running server, and every claim about
pass/fail behaviour above comes from actual patched and unpatched builds of
9.7.2 rather than from inspection.
Areas touched
innodb (DDL), mysql-test