Optimize binlog handling of large transactions (BOLT)

[wxueting-aws]

Problem:
Binlog does not handle large transactions well; both commit and recovery time are proportional to transaction size. Committing a large transaction copies its entire binlog cache into the active binary log, so commit latency grows with the transaction. Because the copy happens while holding LOCK_log, concurrent commits stall behind it. Recovery is likewise expensive; it must scan the last active binlog file, and when that file contains a large transaction it reads and deserializes every byte, prolonging unavailability.

How BOLT solves it:
Instead of writing the spilled cache to a temporary file and then copying that file into the active set of binlog files, we promote the temporary file to become the last active binlog file. At commit we only rename the file and sync it, so the work is no longer proportional to transaction size. Commit latency stays minimal, and smaller transactions can commit in parallel alongside a large one.

The challenge is that a binlog file begins with header events, including Format_description_event and Previous_gtids_log_event. Format_description_event is static, but Previous_gtids_log_event is dynamic, so at spill time we cannot know the offset of the transaction's first event. To solve this we reserve additional space at the front of the header, captured by a new event we introduce called Large_transaction_header. Besides the reserved bytes, this event records the coordinates of the transaction's terminating event (XID, Query, or XA_PREPARE). Recovery uses this event to seek directly to the terminating event instead of scanning the file, skipping an expensive read of the transaction body.

Binlog crash recovery first validates the promoted file's terminating-event metadata before trusting it, and falls back to the standard sequential scan when source_verify_checksum is ON (OFF by default).

System variables:

• binlog_large_transaction_optimization_enabled (default ON) enables or disables the optimization.
• binlog_large_transaction_optimization_threshold sets the spilled cache size above which a transaction becomes eligible.

Status variables:

• binlog_large_transaction_optimization_count reports the number of transactions that successfully committed through the optimization's code path since server startup.
• binlog_large_transaction_optimization_missed_count reports the number of transactions that exceeded binlog_large_transaction_optimization_threshold but could not use the optimized code path since server startup.

Both are exposed through SHOW GLOBAL STATUS and
performance_schema.global_status.

When disabled and fallback:
A transaction falls back to the standard commit path, emits a diagnostic, and increments binlog_large_transaction_optimization_missed_count when any of the following hold:

• binlog_format is not ROW,
• the reserved header space is insufficient for the required header events,
• binlog encryption is enabled,
• binlog transaction compression is enabled,
• binlog_checksum changed while the transaction was in progress, or
• a single statement updated both transactional and non-transactional tables.

MTR tests:

• Validate sys_var behavior at runtime and startup.
• Validate commit and rollback code paths for XA and non-XA transactions.
• Validate the binlog 2PC commit protocol with detailed debug points.
• Validate the optimized recovery behavior.
• Validate savepoint correctness behavior.
• Validate rotate, purge, and other operations that alter the index file.
• Validate conditions where the optimized commit path falls back to the standard commit path.
• Validate creation of GTID events at the beginning of the promoted file, including tagged GTIDs.
• Validate that MySQL maintains the #binlog_temp_files directory at startup time.
• Validate that a replica can receive a promoted file and handles the added event correctly.
• Validate that the dependency-tracking metadata generated in the promoted binlog file is correct.

This contribution is under the OCA signed by Amazon and covering submissions to the MySQL project.

Copyright (c) 2026, Oracle and/or its affiliates.

What does this change do?

#683

Why is it needed?

#683

How was it tested?

• Added/updated MTR tests under mysql-test/
• scripts/ci/mtr.sh passes locally
• Ran the relevant full suite (name it): binlog, binlog_gtid, binlog_nogtid, rpl, rpl_gtid, rpl_nogtid

Contributor checklist

• I have signed the OCA with the email on these commits
• Code is formatted (scripts/ci/format.sh)
• Commits are focused with descriptive messages

AI assistance

• I did not use AI assistance for this contribution
• I used AI assistance for this contribution

If AI assistance was used, describe the tool(s) and extent of use:

• Claude, used for writing code. The final code is reviewed by AWS engineers.

Areas touched

Binlog, binlog replica

[ofarhat-aws]

Please note that this code change has a significant number of tests. These tests are based on years of operating this feature in Aurora MySQL for the last 6 years and catching many corner cases.

Furthermore, we have run benchmarks against this change, you can find the results here: #683 (comment).

[nacarvalho]

Thank you @wxueting-aws for the PR, we will review it.

[tiagoportelajorge]

I guess I need the Low-Level design to understand these nuaces better

[tiagoportelajorge]

Same on this file. I need to read the Low-Level design to understand the purpose of this file.

As you know, we currently cache in IO_CACHE, which inevitably spills to disk, and it is blind to what is inside. But this seems to be yet another cache

[tiagoportelajorge]

Is this a runtime production requirement?

[tiagoportelajorge]

Please add doxygen documentation to all the files

[tiagoportelajorge]

We try to avoid TODOs in the code. Is something stopping you from doing this?

[tiagoportelajorge]

In the global context, "this feature" does not have context. Please make the comment generic.

[tiagoportelajorge]

Maybe we can modernize this a bit, since we have C++20. Codex hints this:

Suggested change

	bool is_bolt_temp_file(const char *name) {
	#include <algorithm>
	#include <string_view>
	bool is_bolt_temp_file(const char *name) {
	const std::string_view file_name{name};
	constexpr std::string_view prefix{kBinlogTempFilePrefix};
	return file_name.starts_with(prefix) &&
	file_name.size() > prefix.size() &&
	std::ranges::all_of(file_name.substr(prefix.size()), [](char c) {
	return (c >= 'a' && c <= 'z') ||
	(c >= '0' && c <= '9') || c == '_';

[tiagoportelajorge]

Is this different from a regular binlog?

[tiagoportelajorge]

Consider std::format_to_n instead of snprintf

[tiagoportelajorge]

I have a fundamental question and maybe it will be answered in the LLD. Will this happen for all IO_CACHE? All instances of IO_CACHE, when they spill, will become a BOLT file?

[RidhaOracle]

@wxueting-aws Can you please rebase your branch on latest trunk to pick the fixes of the CI/CD pipelines ?

[SongLibing]

Does it mean that checksum will be calculated twice for normal small transactions?

1. when writing the events into binlog cache.
2. when writting the events into binlog file.

Did you have a benchmark for small transactions with this patch? I am not sure if twice checksum will impact the performance of small transactions, but IMHO, the first checksum should be avoid for small transactions. At lease the events fit into the memory cache should not calculate the checksum.

[karolina-szczepankiewicz]

Following up on the checksum concern raised by Libing Song, I think this may need an explicit cache representation design decision.

As I understand the current code, transaction cache events start carrying checksums from the first cached event. For transactions that later use the standard binlog path, that checksum cannot be reused as is, because the standard writer rewrites event_len and log_pos when copying the cache into the active binlog, and computes the final checksum again.

I think there is also a second concern here: the transaction cache is part of the before_commit observer API, so changing its physical representation can affect hook consumers. At least Group Replication appears to consume this cache as a logical transaction event stream. If BOLT stores events in a promotable physical representation with checksum footers, that appears to break the API contract for consumers such as Group Replication.

At the same time, BOLT cannot simply defer checksum materialization until commit and promote the existing spill file unchanged, because that file would not be in the final binlog wire format. Making it promotable would require rewriting or converting the spilled transaction body, which reduces the benefit of promotion.

Would it make sense to use a hybrid architecture here? For example, keep the default THD cache as a logical, checksumless representation for small transactions and before_commit observers, and switch or convert to a promotable, checksummed representation only after the transaction has spilled, crossed the BOLT threshold, and is still locally eligible for promotion. The cache should still expose a logical checksumless view for before_commit observers and fallback paths.

This might address both the small transaction double checksum concern and the before_commit cache contract issue, while preserving the BOLT fast path for transactions that actually become promotion candidates. The promoted path would pay an additional but bounded conversion cost for the cache contents already written before the switch, and subsequent events could be written directly in the promotable representation.

If this direction is chosen, I think it should be documented as an explicit design decision, and the final code should make the two cache representations and their conversions clear. That would make the performance tradeoff and the API contract easier to review and maintain.

[karolina-szczepankiewicz]

Thanks again for the contribution. I left a few focused inline comments for the cases I have reviewed so far. Some of them are backed by targeted local MTR reproducers.

I may add more comments if full CI/CD testing exposes additional failures. I have seen some on partial suites testing, but full including failure categorization would be desired.

[karolina-szczepankiewicz]

I may be missing an existing invariant here, but I think there could be a crash-safety corner case around the BOLT threshold boundary.

As I read the code, Binlog_tc_log::prepare() decides whether to use HA_REGULAR_DURABILITY before the terminal XID/COMMIT/XA event is appended to the transaction cache. Later, binlog_cache_data::finalize() appends the terminal event, and only after that get_cache_to_promote() decides whether the transaction should use the BOLT promotion path.

Could this allow a transaction to be below or equal to binlog_large_transaction_optimization_threshold at prepare time, then cross the threshold after finalization and still be promoted by BOLT?

My concern is that, if such a transaction prepared with the normal non-BOLT durability policy, BOLT would then bypass the ordered-commit flush stage that normally makes prepared engine state durable before binlog publication.

I have a possible MTR/debug-sync reproducer and a possible fix sketch if that would be useful, but I would first like to check whether you agree this boundary case is possible, or whether there is another invariant I missed.

[karolina-szczepankiewicz]

This BOLT commit path skips the binlog storage after_sync observer.

In the standard ordered_commit() path, after the binlog sync and before engine commit, the server calls call_after_sync_hook(). Semisync AFTER_SYNC relies on that observer path.

In commit_large_transaction(), I see the after_flush observer being called here, but I do not see a corresponding after_sync observer call before the storage-engine commit.

For semisync configurations, I think this path needs to preserve the same after-sync contract as ordered_commit(), and the PR should include a semisync MTR test for that.

I think the natural place to add it would be immediately after this after_flush observer call, once the final binlog position is known and before finish_commit(thd) / engine commit.

[karolina-szczepankiewicz]

This BOLT path appears to need an explicit replica preserve-commit-order check.

In the normal ordered_commit() path, Stage #0 calls Commit_order_manager::wait(thd) before binlog flush and before storage-engine commit. That contract matters for applier workers when log_replica_updates=ON, replica_parallel_workers>1, and replica_preserve_commit_order=ON.

Here, a promotable large transaction takes the commit_large_transaction() path instead of ordered_commit(). I do not see an equivalent commit-order wait before the promoted binlog is published and the storage engines are committed.

For applier re-logging under replica_preserve_commit_order, I think this path should preserve the same ordering contract, or fall back to ordered_commit() for that configuration.

This also needs a dedicated RPCO MTR test covering BOLT promotion on an applier worker.

[karolina-szczepankiewicz]

Minor: Remember to adjust the numbering before the final merge

[karolina-szczepankiewicz]

[security]
I wanted to check the intended design contract for BOLT and dynamic binlog_encryption changes.

My understanding of the existing binlog path is that changing binlog_encryption takes effect by rotating the binary log. After that rotation, newly opened binlog files use the current encryption state. A normal commit does not publish the binlog-cache temp file directly; it copies the cache into the current binlog stream. So if a transaction spilled while binlog_encryption=OFF, but commits after binlog_encryption=ON, the standard path should still serialize it into the encrypted binlog file.

BOLT is different because it promotes the already-existing spill file itself. The design says transactions that would otherwise qualify for BOLT should fall back when binlog encryption is enabled, but this eligibility check appears to consult only the spill file's current encryption state, not the live encryption policy at commit/promotion time.

If a transaction spills while binlog_encryption=OFF, then binlog_encryption is enabled before that transaction commits, the already-spilled non-empty cache remains plaintext, so trx_cache->get_cache()->is_encrypted() would still be false. That plaintext file can then be promoted as a binlog even though the current binlog encryption policy is ON.

Since the design says BOLT falls back when binlog encryption is enabled, I think this path should fall back to ordered_commit() whenever the current encryption policy is ON at commit/promotion time. This should also have an MTR test that changes binlog_encryption after the large transaction has spilled but before commit, then verifies BOLT falls back to the standard path.

[karolina-szczepankiewicz]

Small NFR6 question about the OFF case.

When binlog_large_transaction_optimization_enabled=OFF, FR5.1 says every transaction must commit through the standard code path. NFR6 also says that path should remain unchanged.

As I read the patch, binlog cache storage is now unified. Even with BOLT OFF, a transaction that merely exceeds binlog_cache_size and spills appears to use #binlog_temp_files, and binlog_cache_data::open() still passes BOLT reserved header space.

Should the OFF case preserve the old spill behavior: legacy temp-file placement, anonymous spill file, and zero reserved bytes? If the new storage path is intentional even with BOLT OFF, I think the design should say so explicitly and reconcile it with NFR6.

[karolina-szczepankiewicz]

This line does not compile for me with -Werror=format-truncation. Could you please check?