MDEV-38305: Expose adaptive hash index statistics in ANALYZE FORMAT=JSON #5069
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,45 @@ | ||
| # Include file to execute a query with optional repetition and display | ||
| # values of r_ahi_stats on ANALYZE FORMAT=JSON | ||
| # | ||
| # Parameters: | ||
| # $query - The query to execute | ||
| # $repeat - Number of times to repeat the query before analyzing | ||
| # | ||
| # This include file will: | ||
| # 1. Execute the query $repeat times to warm up AHI (with logging disabled) | ||
| # 2. Execute ANALYZE FORMAT=JSON on the query and display the values | ||
|
|
||
| --disable_query_log | ||
| --disable_result_log | ||
|
|
||
| # Repeat the query to warm up AHI | ||
| let $i = $repeat; | ||
| while ($i > 0) | ||
| { | ||
| eval $query; | ||
| dec $i; | ||
| } | ||
|
|
||
| --enable_result_log | ||
| --enable_query_log | ||
|
|
||
| # Execute ANALYZE FORMAT=JSON once and capture output | ||
| --echo ANALYZE FORMAT=JSON $query | ||
| let $out=`ANALYZE FORMAT=JSON $query`; | ||
|
|
||
| # Parse JSON and extract AHI variables | ||
| --disable_query_log | ||
| evalp set @js='$out'; | ||
| set @ahi_searches = COALESCE(json_extract(@js,'$**.r_engine_stats.r_ahi_stats.ahi_searches'), 0); | ||
| set @ahi_searches_btree = COALESCE(json_extract(@js,'$**.r_engine_stats.r_ahi_stats.ahi_searches_btree'), 0); | ||
| set @ahi_rows_added = COALESCE(json_extract(@js,'$**.r_engine_stats.r_ahi_stats.ahi_rows_added'), 0); | ||
| set @ahi_pages_added = COALESCE(json_extract(@js,'$**.r_engine_stats.r_ahi_stats.ahi_pages_added'), 0); | ||
| set @r_rows = json_extract(@js,'$**.table.r_rows'); | ||
|
|
||
| # Display AHI variables | ||
| select @ahi_searches as ahi_searches, | ||
| @ahi_searches_btree as ahi_searches_btree, | ||
| @ahi_rows_added as ahi_rows_added, | ||
| @ahi_pages_added as ahi_pages_added, | ||
| @r_rows as r_rows; | ||
| --enable_query_log |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| # | ||
| # MDEV 38305 : Expose adaptive hash index statistics in ANALYZE FORMAT=JSON | ||
| # | ||
| SET @start_global_value= @@global.innodb_adaptive_hash_index; | ||
| SET GLOBAL innodb_adaptive_hash_index= ON; | ||
| CREATE TABLE t1 ( | ||
| id INT PRIMARY KEY, | ||
| col1 INT, | ||
| col2 INT, | ||
| col3 INT, | ||
| INDEX idx_1 (col1), | ||
| INDEX idx_2 (col2), | ||
| INDEX idx_3 (col3) | ||
| ) ENGINE=InnoDB; | ||
| INSERT INTO t1 SELECT seq, seq % 20, seq % 5, seq % 10 FROM seq_0_to_999; | ||
| ANALYZE FORMAT=JSON SELECT * FROM t1 FORCE INDEX(idx_1) WHERE col1 = 5 | ||
| ahi_searches ahi_searches_btree ahi_rows_added ahi_pages_added r_rows | ||
| 0 [51] 0 0 [50] | ||
| ANALYZE FORMAT=JSON SELECT * FROM t1 FORCE INDEX(idx_2) WHERE col2 = 3 | ||
| ahi_searches ahi_searches_btree ahi_rows_added ahi_pages_added r_rows | ||
| [129] [72] [797] [2] [200] | ||
| ANALYZE FORMAT=JSON SELECT * FROM t1 FORCE INDEX(idx_3) WHERE col3 = 5 | ||
| ahi_searches ahi_searches_btree ahi_rows_added ahi_pages_added r_rows | ||
| [101] 0 0 0 [100] | ||
| SET GLOBAL innodb_adaptive_hash_index = OFF; | ||
| ANALYZE FORMAT=JSON SELECT * FROM t1 FORCE INDEX(idx_2) WHERE col2 = 2 | ||
| ahi_searches ahi_searches_btree ahi_rows_added ahi_pages_added r_rows | ||
| 0 0 0 0 [200] | ||
| DROP TABLE t1; | ||
| SET GLOBAL innodb_adaptive_hash_index= @start_global_value; | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,78 @@ | ||
| --source include/have_innodb.inc | ||
| --source include/have_innodb_16k.inc | ||
| --source include/have_sequence.inc | ||
|
|
||
| --echo # | ||
| --echo # MDEV 38305 : Expose adaptive hash index statistics in ANALYZE FORMAT=JSON | ||
| --echo # | ||
|
|
||
| # Test Plan: | ||
| # When queries use secondary indexes, InnoDB must lookup clustered index records | ||
| # using the primary key reference obtained from the secondary index. These | ||
| # clustered index lookups accumulate and trigger AHI building when a threshold | ||
| # is reached. This test verifies that AHI statistics are correctly reported in | ||
| # ANALYZE FORMAT=JSON output across different scenarios: | ||
| # | ||
| # Test 1: Access 50 rows (col1=5, 50 matches) - insufficient to build AHI | ||
| # Test 2: Access 200 rows (col2=3, 200 matches) - combined with Test 1 (250 total), | ||
| # this triggers AHI construction on the clustered index | ||
| # Test 3: Access with heavy warmup (100 repetitions) - AHI fully utilized | ||
| # Test 4: AHI disabled - verify statistics are zero regardless of access patterns | ||
|
|
||
| SET @start_global_value= @@global.innodb_adaptive_hash_index; | ||
| SET GLOBAL innodb_adaptive_hash_index= ON; | ||
|
|
||
| CREATE TABLE t1 ( | ||
| id INT PRIMARY KEY, | ||
| col1 INT, | ||
| col2 INT, | ||
| col3 INT, | ||
| INDEX idx_1 (col1), | ||
| INDEX idx_2 (col2), | ||
| INDEX idx_3 (col3) | ||
| ) ENGINE=InnoDB; | ||
|
|
||
| INSERT INTO t1 SELECT seq, seq % 20, seq % 5, seq % 10 FROM seq_0_to_999; | ||
|
|
||
| # Test 1: Insufficient accesses to build AHI | ||
| # Query accesses 50 rows (col1=5 matches 50 out of 1000 rows: 5,25,45,...,985). | ||
| # When using idx_1, InnoDB performs 50 clustered index lookups to retrieve the | ||
| # full rows. This is insufficient to trigger AHI construction, so AHI statistics | ||
| # should show minimal or no AHI activity. | ||
| let $query = SELECT * FROM t1 FORCE INDEX(idx_1) WHERE col1 = 5; | ||
| let $repeat = 0; | ||
| --source suite/innodb/include/check_ahi_status.inc | ||
|
|
||
| # Test 2: Accumulated accesses trigger AHI construction | ||
| # Query accesses 200 rows (col2=3 matches 200 out of 1000 rows: 3,8,13,18,...,998). | ||
| # Combined with Test 1's 50 clustered index lookups, the accumulated total of 250 | ||
| # clustered index searches crosses the threshold to build AHI. This test verifies | ||
| # that AHI statistics reflect the construction and initial usage of the hash index. | ||
| let $query = SELECT * FROM t1 FORCE INDEX(idx_2) WHERE col2 = 3; | ||
| let $repeat = 0; | ||
| --source suite/innodb/include/check_ahi_status.inc | ||
|
|
||
| # Test 3: Verify AHI statistics with heavy warmup (AHI fully utilized) | ||
| # Query accesses 100 rows (col3=5 matches 100 out of 1000 rows: 5,15,25,...,995). | ||
| # With repeat=200, this executes the query 200 times before the ANALYZE, resulting | ||
| # in 20,000 clustered index lookups. This heavy access pattern ensures the AHI is | ||
| # fully built and actively serving lookups, allowing verification that AHI hit | ||
| # statistics are properly reported when the hash index is effectively utilized. | ||
| let $query = SELECT * FROM t1 FORCE INDEX(idx_3) WHERE col3 = 5; | ||
| let $repeat = 200; | ||
| --source suite/innodb/include/check_ahi_status.inc | ||
|
|
||
| # Test 4: Verify AHI statistics when AHI is disabled | ||
| # Query accesses 200 rows (col2=2 matches 200 out of 1000 rows: 2,7,12,...,997). | ||
| # With repeat=100, this would normally trigger heavy AHI usage (20,000 clustered | ||
| # index lookups). However, since innodb_adaptive_hash_index is turned OFF, the AHI | ||
| # is not used and all AHI statistics should be zero regardless of query repetition. | ||
| SET GLOBAL innodb_adaptive_hash_index = OFF; | ||
|
|
||
| let $query = SELECT * FROM t1 FORCE INDEX(idx_2) WHERE col2 = 2; | ||
| let $repeat = 100; | ||
| --source suite/innodb/include/check_ahi_status.inc | ||
|
|
||
| # Cleanup | ||
| DROP TABLE t1; | ||
| SET GLOBAL innodb_adaptive_hash_index= @start_global_value; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -1952,6 +1952,20 @@ static void trace_engine_stats(handler *file, Json_writer *writer) | |
| writer->add_member("pages_prefetch_read_count").add_ull(hs->pages_prefetched); | ||
| if (hs->undo_records_read) | ||
| writer->add_member("old_rows_read").add_ull(hs->undo_records_read); | ||
| if (hs->ahi_searches || hs->ahi_searches_btree || | ||
| hs->ahi_rows_added || hs->ahi_pages_added) | ||
| { | ||
| writer->add_member("r_ahi_stats").start_object(); | ||
| if (hs->ahi_searches) | ||
| writer->add_member("ahi_searches").add_ull(hs->ahi_searches); | ||
| if (hs->ahi_searches_btree) | ||
| writer->add_member("ahi_searches_btree").add_ull(hs->ahi_searches_btree); | ||
| if (hs->ahi_rows_added) | ||
| writer->add_member("ahi_rows_added").add_ull(hs->ahi_rows_added); | ||
| if (hs->ahi_pages_added) | ||
| writer->add_member("ahi_pages_added").add_ull(hs->ahi_pages_added); | ||
| writer->end_object(); | ||
| } | ||
| writer->end_object(); | ||
| } | ||
| } | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -1146,11 +1146,17 @@ dberr_t btr_cur_t::search_leaf(const dtuple_t *tuple, page_cur_mode_t mode, | |
| ut_ad(up_match != uint16_t(~0U) || mode != PAGE_CUR_LE); | ||
| ut_ad(low_match != uint16_t(~0U) || mode != PAGE_CUR_LE); | ||
| ++btr_cur_n_sea; | ||
| if (mtr->trx) | ||
| btr_ahi_inc_searches(mtr->trx); | ||
|
Comment on lines
+1149
to
+1150
Contributor
There was a problem hiding this comment. When would we have |
||
|
|
||
| return DB_SUCCESS; | ||
| } | ||
| else | ||
| { | ||
| ++btr_cur_n_non_sea; | ||
| if (mtr->trx) | ||
| btr_ahi_inc_searches_btree(mtr->trx); | ||
|
Comment on lines
+1157
to
+1158
Contributor
There was a problem hiding this comment. Can you post some stack traces where
Contributor
Author
There was a problem hiding this comment. |
||
| } | ||
| # endif | ||
| #endif | ||
|
|
||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -82,6 +82,36 @@ inline ahi_node **btr_sea::hash_chain::search(UnaryPred u) noexcept | |
| return prev; | ||
| } | ||
|
|
||
| void btr_ahi_inc_searches(trx_t *trx) noexcept | ||
|
Contributor
There was a problem hiding this comment. small nitpickery (also applicable to |
||
| { | ||
| if (ha_handler_stats *stats= trx->active_handler_stats) | ||
| stats->ahi_searches++; | ||
| } | ||
|
|
||
| void btr_ahi_inc_searches_btree(trx_t *trx) noexcept | ||
| { | ||
| if (ha_handler_stats *stats= trx->active_handler_stats) | ||
| stats->ahi_searches_btree++; | ||
| } | ||
|
Comment on lines
+85
to
+95
Contributor
There was a problem hiding this comment. Could these simple functions be defined |
||
|
|
||
| static void btr_ahi_inc_rows_added(ulonglong count= 1) noexcept | ||
| { | ||
| MONITOR_INC_VALUE(MONITOR_ADAPTIVE_HASH_ROW_ADDED, count); | ||
| if (THD *thd= current_thd) | ||
| if (trx_t *trx= thd_to_trx(thd)) | ||
| if (ha_handler_stats *stats= trx->active_handler_stats) | ||
| stats->ahi_rows_added+= count; | ||
| } | ||
|
|
||
| static void btr_ahi_inc_pages_added() noexcept | ||
| { | ||
| MONITOR_INC(MONITOR_ADAPTIVE_HASH_PAGE_ADDED); | ||
| if (THD *thd= current_thd) | ||
| if (trx_t *trx= thd_to_trx(thd)) | ||
| if (ha_handler_stats *stats= trx->active_handler_stats) | ||
| stats->ahi_pages_added++; | ||
| } | ||
|
|
||
| inline void btr_sea::partition::init() noexcept | ||
| { | ||
| latch.SRW_LOCK_INIT(btr_search_latch_key); | ||
|
|
@@ -625,7 +655,7 @@ static void btr_search_update_hash_ref(const btr_cur_t &cursor, | |
| } | ||
|
|
||
| part.insert(fold, rec, block); | ||
| btr_ahi_inc_rows_added(); | ||
|
Comment on lines
-628
to
+658
Contributor
There was a problem hiding this comment. Can we pass |
||
| } | ||
| else | ||
| { | ||
|
|
@@ -1676,7 +1706,7 @@ static void btr_search_build_page_hash_index(dict_index_t *index, | |
| part.latch.wr_rd_downgrade(SRW_LOCK_CALL); | ||
| # endif | ||
|
|
||
| btr_ahi_inc_rows_added(n_cached); | ||
|
|
||
| for (size_t i= 0; i < n_cached; i++) | ||
| { | ||
|
|
@@ -1711,7 +1741,7 @@ static void btr_search_build_page_hash_index(dict_index_t *index, | |
| goto next_redundant; | ||
| } | ||
|
|
||
| btr_ahi_inc_pages_added(); | ||
| assert_block_ahi_valid(block); | ||
| part.latch.rd_unlock(); | ||
| } | ||
|
|
@@ -1964,7 +1994,7 @@ void btr_search_update_hash_on_insert(btr_cur_t *cursor, bool reorg) noexcept | |
| goto unlock_exit; | ||
| } | ||
| part.insert(ins_fold, ins_rec, block); | ||
| btr_ahi_inc_rows_added(); | ||
| } | ||
| } | ||
| else if (fold != ins_fold) | ||
|
|
@@ -1979,7 +2009,7 @@ void btr_search_update_hash_on_insert(btr_cur_t *cursor, bool reorg) noexcept | |
| if (left_bytes_fields & buf_block_t::LEFT_SIDE) | ||
| fold= ins_fold, rec= ins_rec; | ||
| part.insert(fold, rec, block); | ||
| btr_ahi_inc_rows_added(); | ||
| } | ||
|
|
||
| if (next_is_supremum) | ||
|
|
@@ -1994,7 +2024,7 @@ void btr_search_update_hash_on_insert(btr_cur_t *cursor, bool reorg) noexcept | |
| goto rollback; | ||
| } | ||
| part.insert(ins_fold, ins_rec, block); | ||
| btr_ahi_inc_rows_added(); | ||
| } | ||
| } | ||
| else if (ins_fold != next_fold) | ||
|
|
@@ -2009,7 +2039,7 @@ void btr_search_update_hash_on_insert(btr_cur_t *cursor, bool reorg) noexcept | |
| if (!(left_bytes_fields & ~buf_block_t::LEFT_SIDE)) | ||
| next_fold= ins_fold, next_rec= ins_rec; | ||
| part.insert(next_fold, next_rec, block); | ||
| btr_ahi_inc_rows_added(); | ||
| } | ||
|
|
||
| ut_ad(!locked || index == block->index); | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -91,6 +91,14 @@ void btr_search_update_hash_on_insert(btr_cur_t *cursor, bool reorg) noexcept; | |
| @param cursor cursor positioned on the to-be-deleted record */ | ||
| void btr_search_update_hash_on_delete(btr_cur_t *cursor) noexcept; | ||
|
|
||
| #ifdef BTR_CUR_HASH_ADAPT | ||
|
Contributor
There was a problem hiding this comment. Minor: this can be removed as well, should be already checked on top of the file. Can you please try building with |
||
| /** Increment successful adaptive hash index lookups */ | ||
| void btr_ahi_inc_searches(trx_t *trx) noexcept; | ||
|
|
||
| /** Increment adaptive hash index misses (B-tree fallback) */ | ||
| void btr_ahi_inc_searches_btree(trx_t *trx) noexcept; | ||
| #endif /* BTR_CUR_HASH_ADAPT */ | ||
|
|
||
| /** Validates the search system. | ||
| @param thd connection, for checking if CHECK TABLE has been killed | ||
| @return true if ok */ | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -42,6 +42,8 @@ void mtr_t::commit() {} | |
| void mtr_t::rollback_to_savepoint(ulint, ulint) {} | ||
| void small_vector_base::grow_by_1(void *, size_t) noexcept {} | ||
| void buf_inc_get(trx_t*) noexcept {} | ||
| THD *_current_thd() { return nullptr; } | ||
| trx_t *thd_to_trx(const THD*) noexcept { return nullptr; } | ||
|
|
||
| void sql_print_error(const char*, ...) {} | ||
| ulint ut_find_prime(ulint n) { return n; } | ||
|
|
||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.