Optimize binlog handling of large transactions (BOLT)

[wxueting-aws]

Problem:
Binlog does not handle large transactions well; both commit and recovery time are proportional to transaction size. Committing a large transaction copies its entire binlog cache into the active binary log, so commit latency grows with the transaction. Because the copy happens while holding LOCK_log, concurrent commits stall behind it. Recovery is likewise expensive; it must scan the last active binlog file, and when that file contains a large transaction it reads and deserializes every byte, prolonging unavailability.

How BOLT solves it:
Instead of writing the spilled cache to a temporary file and then copying that file into the active set of binlog files, we promote the temporary file to become the last active binlog file. At commit we only rename the file and sync it, so the work is no longer proportional to transaction size. Commit latency stays minimal, and smaller transactions can commit in parallel alongside a large one.

The challenge is that a binlog file begins with header events, including Format_description_event and Previous_gtids_log_event. Format_description_event is static, but Previous_gtids_log_event is dynamic, so at spill time we cannot know the offset of the transaction's first event. To solve this we reserve additional space at the front of the header, captured by a new event we introduce called Large_transaction_header. Besides the reserved bytes, this event records the coordinates of the transaction's terminating event (XID, Query, or XA_PREPARE). Recovery uses this event to seek directly to the terminating event instead of scanning the file, skipping an expensive read of the transaction body.

Binlog crash recovery first validates the promoted file's terminating-event metadata before trusting it, and falls back to the standard sequential scan when source_verify_checksum is ON (OFF by default).

System variables:

• binlog_large_transaction_optimization_enabled (default ON) enables or disables the optimization.
• binlog_large_transaction_optimization_threshold sets the spilled cache size above which a transaction becomes eligible.

Status variables:

• binlog_large_transaction_optimization_count reports the number of transactions that successfully committed through the optimization's code path since server startup.
• binlog_large_transaction_optimization_missed_count reports the number of transactions that exceeded binlog_large_transaction_optimization_threshold but could not use the optimized code path since server startup.

Both are exposed through SHOW GLOBAL STATUS and
performance_schema.global_status.

When disabled and fallback:
A transaction falls back to the standard commit path, emits a diagnostic, and increments binlog_large_transaction_optimization_missed_count when any of the following hold:

• binlog_format is not ROW,
• the reserved header space is insufficient for the required header events,
• binlog encryption is enabled,
• binlog transaction compression is enabled,
• binlog_checksum changed while the transaction was in progress, or
• a single statement updated both transactional and non-transactional tables.

MTR tests:

• Validate sys_var behavior at runtime and startup.
• Validate commit and rollback code paths for XA and non-XA transactions.
• Validate the binlog 2PC commit protocol with detailed debug points.
• Validate the optimized recovery behavior.
• Validate savepoint correctness behavior.
• Validate rotate, purge, and other operations that alter the index file.
• Validate conditions where the optimized commit path falls back to the standard commit path.
• Validate creation of GTID events at the beginning of the promoted file, including tagged GTIDs.
• Validate that MySQL maintains the #binlog_temp_files directory at startup time.
• Validate that a replica can receive a promoted file and handles the added event correctly.
• Validate that the dependency-tracking metadata generated in the promoted binlog file is correct.

This contribution is under the OCA signed by Amazon and covering submissions to the MySQL project.

Copyright (c) 2026, Oracle and/or its affiliates.

What does this change do?

#683

Why is it needed?

#683

How was it tested?

• Added/updated MTR tests under mysql-test/
• scripts/ci/mtr.sh passes locally
• Ran the relevant full suite (name it): binlog, binlog_gtid, binlog_nogtid, rpl, rpl_gtid, rpl_nogtid

Contributor checklist

• I have signed the OCA with the email on these commits
• Code is formatted (scripts/ci/format.sh)
• Commits are focused with descriptive messages

AI assistance

• I did not use AI assistance for this contribution
• I used AI assistance for this contribution

If AI assistance was used, describe the tool(s) and extent of use:

• Claude, used for writing code. The final code is reviewed by AWS engineers.

Areas touched

Binlog, binlog replica

[ofarhat-aws]

Please note that this code change has a significant number of tests. These tests are based on years of operating this feature in Aurora MySQL for the last 6 years and catching many corner cases.

Furthermore, we have run benchmarks against this change, you can find the results here: #683 (comment).