Skip to content

InnoDB: Parallel DDL threads race in debug_sync on the shared THD - #713

Open
inikep wants to merge 1 commit into
mysql:trunkfrom
inikep:ddl-debug-sync-race-oracle-submit
Open

InnoDB: Parallel DDL threads race in debug_sync on the shared THD#713
inikep wants to merge 1 commit into
mysql:trunkfrom
inikep:ddl-debug-sync-race-oracle-submit

Conversation

@inikep

@inikep inikep commented Aug 6, 2026

Copy link
Copy Markdown

What does this change do?

Serializes the four DEBUG_SYNC calls in storage/innobase/ddl/ddl0builder.cc
with a per-ddl::Context mutex, so the parallel DDL threads that share the
connection's THD cannot execute one debug sync action at the same time.

Why is it needed?

The tasks driving ddl::Builder are pulled off a shared queue by whichever DDL
thread is free (ddl::Loader::Task_queue::mt_execute()), so an online ALTER
building several indexes runs setup_sort() / btree_build() / finalize()
concurrently, and every one of those threads passes the same THD -
ddl::Context::thd() - to DEBUG_SYNC. debug_sync() does no locking; it
assumes a sync point is only ever hit by the thread owning the THD.

On a 9.7.2 debug build an ALTER that adds four indexes with
innodb_ddl_threads = 4, with row_log_apply_before armed, kills the server:

mysqld: sql/debug_sync.cc:946: void debug_sync_remove_action(
st_debug_sync_control*, st_debug_sync_action*):
Assertion `dsp_idx < ds_control->ds_active' failed.
 #10 debug_sync_remove_action        sql/debug_sync.cc:946
 #11 debug_sync                      sql/debug_sync.cc:1951
 #12 ddl::Builder::finalize          storage/innobase/ddl/ddl0builder.cc:1994
 #13 ddl::Builder::finish            storage/innobase/ddl/ddl0builder.cc:2086
 #14 ddl::Loader::Task::operator()   storage/innobase/ddl/ddl0builder.cc:2180
 #15 ddl::Loader::Task_queue::mt_execute  storage/innobase/ddl/ddl0loader.cc:178

Every step of debug_sync() races when two threads share the THD, and each has
its own symptom:

  • both threads pass the action->activation_count test, then the first consumes
    the last activation while the second is entering debug_sync_execute() -
    Assertion 'action->activation_count' failed;
  • both then find activation_count == 0 on the way out and both call
    debug_sync_remove_action() on the same action - the assertion in the stack
    above;
  • WAIT_FOR clears the signal it woke up on unless NO_CLEAR_EVENT is given,
    so with two threads waiting on one action the first one out clears the event
    and the second waits for a signal that no longer exists - the ALTER hangs.

The commit message contains a DBUG trace
(--debug=d,debug_sync,debug_sync_exec,debug_sync_point:i:o) showing two DDL
threads inside the one sync point on the one THD, with the second never
resuming.

The mutex is deliberately per context, i.e. per DDL statement and hence per THD.
A single file-static mutex instead hangs innodb.innodb-index-online-purge,
which parks one DDL inside row_log_apply_before and only releases it after a
second, independent DDL has reached the same sync point. It is taken only when
debug sync is armed, mirroring the DEBUG_SYNC macro's own
opt_debug_sync_timeout fast path, and the whole construct compiles away
without ENABLED_DEBUG_SYNC, so release builds are unaffected.

How was it tested?

  • Added/updated MTR tests under mysql-test/
  • scripts/ci/mtr.sh passes locally
  • Ran the relevant full suite (name it): every existing test using these
    sync points - innodb.innodb-alter-varchar-debug,
    innodb.innodb-index-online, innodb.innodb-index-online-purge,
    innodb.virtual_debug, innodb.bulk_create_index_online,
    innodb.ddl_kill, innodb.index-create-dml-rollback,
    gcol.gcol_rollback, plus innodb.innodb-alter-debug and
    innodb.innodb-alter-nullable

New test innodb.ddl_debug_sync_race, in three parts: a reproducer that arms
the sync point with two activations so two builders of one ALTER meet in it, a
stress with a single activation (the shape that hits the
action->activation_count assertion in the field), and a check that the
serialization does not span connections.

Measured on a 9.7.2 debug build (gcc 15, CMAKE_BUILD_TYPE=Debug): unpatched,
the new test crashed the server with the assertion above 5 times in a
--repeat=24 --parallel=8 run and hung in others; patched, the same run passes
24/24.

Re-verified on trunk (06a5c1c) after the rebase, same build configuration:
the same --repeat=24 --parallel=8 run passes 24/24 with the patch, and all
ten existing tests listed above pass. The four sync points and the surrounding
builder code are identical to 9.7.2 there, so the unpatched failure modes apply
unchanged; the crash counts quoted above were measured on the 9.7.2 build.

Contributor checklist

  • I have signed the OCA with the email on these commits
  • Code is formatted (scripts/ci/format.sh)
  • Commits are focused with descriptive messages

AI assistance

  • I did not use AI assistance for this contribution
  • I used AI assistance for this contribution

Claude Code was used to reproduce the failure, analyse the debug_sync control
block races, write the MTR test and draft this description. The diagnosis was
confirmed against a DBUG trace of the running server, and every claim about
pass/fail behaviour above comes from actual patched and unpatched builds of
9.7.2 rather than from inspection.

Areas touched

innodb (DDL), mysql-test

@inikep
inikep requested a review from a team August 6, 2026 15:39
@oracle-contributor-agreement

Copy link
Copy Markdown

Thank you for your pull request and welcome to our community! To contribute, please sign the Oracle Contributor Agreement (OCA).
The following contributors of this PR have not signed the OCA:

  • PR author: inikep

To sign the OCA, please create an Oracle account and sign the OCA in Oracle's Contributor Agreement Application.

When signing the OCA, please provide your GitHub username. After signing the OCA and getting an OCA approval from Oracle, this PR will be automatically updated.

If you are an Oracle employee, please make sure that you are a member of the main Oracle GitHub organization, and your membership in this organization is public.

@inikep

inikep commented Aug 6, 2026

Copy link
Copy Markdown
Author

I confirm the code being submitted is offered under the terms of the OCA, and that I am authorized to contribute it

@oracle-contributor-agreement oracle-contributor-agreement Bot added the OCA Required At least one contributor does not have an approved Oracle Contributor Agreement. label Aug 6, 2026
@inikep
inikep changed the base branch from 9.7 to trunk August 6, 2026 16:02
A debug server can be killed, or an ALTER left hanging forever, by the
sync points in storage/innobase/ddl/ddl0builder.cc: several DDL threads
execute them concurrently on one THD, and debug_sync is not prepared for
that.

Reproducer
==========
The attached test innodb.ddl_debug_sync_race, part 1, arms
"row_log_apply_before" with two activations and runs an online ALTER
adding four indexes with innodb_ddl_threads = 4. On 9.7.2 (debug) it
kills the server:

  mysqld: sql/debug_sync.cc:946: void debug_sync_remove_action(
  st_debug_sync_control*, st_debug_sync_action*):
  Assertion `dsp_idx < ds_control->ds_active' failed.
   mysql#10 debug_sync_remove_action    sql/debug_sync.cc:946
   mysql#11 debug_sync                  sql/debug_sync.cc:1951
   mysql#12 ddl::Builder::finalize      storage/innobase/ddl/ddl0builder.cc:1994
   mysql#13 ddl::Builder::finish        storage/innobase/ddl/ddl0builder.cc:2086
   mysql#14 ddl::Loader::Task::operator()  ddl/ddl0builder.cc:2180
   mysql#15 ddl::Loader::Task_queue::mt_execute  ddl/ddl0loader.cc:178

ddl0builder.cc:1994 is DEBUG_SYNC(m_ctx.thd(), "row_log_apply_before").

The same test also hangs instead of crashing, depending on which of the
racing threads gets ahead: the ALTER never returns and the waits run into
their debug sync timeout.

Why several threads share one THD
=================================
The tasks driving the builder state machine are pulled off a shared queue
by whichever DDL thread is free - Task_queue::mt_execute() pops under
m_mutex but runs the task outside it - so an ALTER building several
indexes executes setup_sort()/btree_build()/finish() for different
builders at the same time on different threads, and the sort tasks of one
builder are parallel by design. Loader::load() runs the queue on the
connection's own thread too and clears current_thd in the workers it
spawns. All of those threads hand the same THD, the owning connection's
m_ctx.thd(), to DEBUG_SYNC.

debug_sync() does no locking; the facility assumes a sync point is only
ever hit by the thread owning the THD. Every step of it races when that
assumption is broken, and each step has its own visible symptom:

 - debug_sync() tests action->activation_count and only then calls
   debug_sync_execute(), which asserts the same condition before
   decrementing it. A single thread can never trip that assertion - the
   test immediately precedes it - two can: both pass the test, then the
   first consumes the last activation while the second is still entering
   debug_sync_execute():
     sql/debug_sync.cc: Assertion `action->activation_count' failed

 - Both threads then find activation_count == 0 on the way out and both
   call debug_sync_remove_action() on the same action. The second one
   finds ds_active already decremented, which is the assertion in the
   stack above; without the assertion it would decrement ds_active below
   zero and shift the action array by a huge count.

 - A WAIT_FOR clears the signal it woke up on unless NO_CLEAR_EVENT is
   given. With two threads waiting on the same action, the first one out
   clears the event and the second one goes back to sleep waiting for a
   signal that no longer exists - the hang.

A DBUG trace (--debug=d,debug_sync,debug_sync_exec,debug_sync_point:i:o)
of the reproducer shows two DDL threads inside the one sync point on the
one THD, and the second one never resuming:

  T@113: debug_sync_point: hit: 'row_log_apply_before'
  T@113: sync_point: 'row_log_apply_before'  activation_count: 2 ...
  T@113: debug_sync_exec: signal 'ddl_parked'  at: 'row_log_apply_before'
  T@113: debug_sync_exec: wait for 'ddl_resume' at: 'row_log_apply_before'
  T@114: debug_sync_point: hit: 'row_log_apply_before'
  T@114: sync_point: 'row_log_apply_before'  activation_count: 1 ...
  T@114: debug_sync_exec: signal 'ddl_parked'  at: 'row_log_apply_before'
  T@114: debug_sync_exec: wait for 'ddl_resume' at: 'row_log_apply_before'
  T@10 : debug_sync_exec: signal 'ddl_resume'  at: 'now'
  T@114: awoke from ddl_resume error: 0
  T@114: debug_sync_exec: resume from 'ddl_resume' at: 'row_log_apply_before'
  T@113: awoke from ddl_resume error: 0            <-- signal already cleared

Fix
===
Serialize the builder's sync points with a mutex, so the first thread
executes the action and the ones behind it find it already removed and
return without touching the control block.

The mutex is per Context, i.e. per DDL statement and hence per THD, and
must stay that way. A first attempt with one file-static mutex hangs
innodb.innodb-index-online-purge with "debug sync point wait timed out":
that test parks one DDL inside row_log_apply_before and only releases it
after a second, independent DDL has reached the same sync point, which a
lock shared between statements prevents. Threads sharing a THD must be
serialized; statements that do not share one must stay independent.

All four sync points in the file are covered, not just the one in the
stack trace: ddl_btree_build_interrupt, ddl_merge_sort_interrupt and
row_log_apply_before/after are the same construct in the same task-driven
paths, and the sort tasks are the most parallel of them.

The lock is only taken when debug sync is armed, mirroring the DEBUG_SYNC
macro's own opt_debug_sync_timeout fast path, and everything compiles away
in builds without DEBUG_SYNC, so release builds are unaffected.

Test
====
innodb.ddl_debug_sync_race has three parts. Part 1 is the reproducer
above. Part 2 is a stress: ten rounds of the same ALTER with a single
activation armed, the shape that hits the `action->activation_count'
assertion in the field. Part 3 checks that the serialization does not
span connections: while one connection is parked in the sync point,
another connection's ALTER must still reach it.

Verified on a 9.7.2 debug build: unpatched, the test crashed the server
with the assertion above 5 times in a --repeat=24 --parallel=8 run (and
hung in others); patched, the same run passes 24/24. Every existing test
using these sync points passes with the patch: innodb-alter-varchar-debug,
innodb-index-online, innodb-index-online-purge, virtual_debug,
bulk_create_index_online, ddl_kill, index-create-dml-rollback,
gcol.gcol_rollback, plus innodb-alter-debug and innodb-alter-nullable.
@inikep
inikep force-pushed the ddl-debug-sync-race-oracle-submit branch from 3efb3fc to 090ddeb Compare August 6, 2026 16:04
@github-actions github-actions Bot added InnoDB Changes touching InnoDB storage engine code Tests Changes touching test code or test data MTR Failed MTR suite failed labels Aug 6, 2026
@gopshank
gopshank requested review from mayprasa and tjeldvoll and removed request for gopshank and seemasundara August 6, 2026 17:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

InnoDB Changes touching InnoDB storage engine code MTR Failed MTR suite failed OCA Required At least one contributor does not have an approved Oracle Contributor Agreement. Tests Changes touching test code or test data

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant