Skip to content

Remove stale tableId mapping in RDS source (MySQL) when a table id is reused - #7162

Merged
graytaylor0 merged 1 commit into
opensearch-project:mainfrom
gg-mskm:fix/rds-source-stale-table-id-mapping
Sep 30, 2026
Merged

graytaylor0 merged 1 commit into
opensearch-project:mainfrom
gg-mskm:fix/rds-source-stale-table-id-mapping

Conversation

@gg-mskm

@gg-mskm gg-mskm commented Sep 25, 2026

Copy link
Copy Markdown
Contributor

Description

In the RDS source's CDC/binlog stream mode, BinlogEventListener keeps an in-memory tableId -> TableMetadata map (tableMetadataMap) that is populated from binlog TABLE_MAP events. handleTableMapEvent returned early for any table that is not of interest without touching the map, so an entry recorded for a table of interest survived indefinitely and was never invalidated.

The binlog table_id is not stable: the server assigns it from its table definition cache and can reassign it to a different table across a server restart (and when a table is evicted from and reloaded into the cache). When a table_id previously bound to a synced table is reassigned to a non-synced table, isValidTableId still accepts the row events because the stale entry is present and points at a synced table. Those rows are then decoded with the wrong column definitions, which silently writes corrupted documents to the sink (for example a byte-array name rendered as [B@30fb30c]) and raises type-conversion errors for rows whose types do not line up. This reproduces on routine operational events such as an Aurora reboot, maintenance, or failover.

This change removes the mapping held for a table_id whenever a TABLE_MAP event shows that the id now refers to a table that is not of interest, so subsequent row events for that id are skipped as intended. Mappings for tables of interest are unaffected; they are already overwritten on every TABLE_MAP event.

Verification:

  • Added 3 unit tests to BinlogEventListenerTest. Without the fix the stale-mapping test fails (isValidTableId returns true for a reassigned id); with the fix all tests pass.
  • End-to-end on Aurora MySQL 8.0 (aurora-mysql 3.10.4): after forcing a table_id collision across a reboot, the unfixed build reproduces the Row data length (25) does not match column names size (21) WARN and writes corrupted documents, while the fixed build removes the stale entry, skips the non-synced rows, and writes no corrupted documents. Synced-table replication is unchanged in both cases.

Issues Resolved

Resolves #7125

Check List

  • New functionality includes testing.
  • New functionality has a documentation issue. Please link to it in this PR.
    • New functionality has javadoc added
  • Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

For more information on following Developer Certificate of Origin and signing off your commits, please check here.

The MySQL RDS source keeps an in-memory `tableId -> TableMetadata` map that is
built from binlog TABLE_MAP events. `handleTableMapEvent` returned early for any
table that is not of interest without touching the map, so an entry recorded for
a table of interest survived indefinitely.

The binlog `table_id` is assigned by the server from its table definition cache
and is not stable: it can be reassigned to a different table across a server
restart, and also when a table is evicted from and reloaded into the cache. When
a table id previously bound to a table of interest was reassigned to a table
that is not of interest, `isValidTableId` still accepted the row events for that
table because the stale entry was present and pointed at a table of interest.
Those rows were then decoded using the wrong column definitions, which silently
wrote corrupted documents to the sink and raised type-conversion errors for the
rows whose types did not line up.

Remove the mapping held for a table id whenever a TABLE_MAP event shows the id
now refers to a table that is not of interest, so subsequent row events for that
id are skipped as intended.

Resolves opensearch-project#7125

Signed-off-by: Masaki Mori <masakigg@amazon.co.jp>
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🧪 PR contains tests
🔒 No security concerns identified
✅ No TODO sections
🔀 No multiple PR themes
⚡ No major issues detected

@graytaylor0 graytaylor0 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing this!

@graytaylor0
graytaylor0 merged commit 0b09adb into opensearch-project:main Sep 30, 2026
8 checks passed
graytaylor0 pushed a commit to graytaylor0/data-prepper that referenced this pull request Oct 1, 2026
…pensearch-project#7162)

The MySQL RDS source keeps an in-memory `tableId -> TableMetadata` map that is
built from binlog TABLE_MAP events. `handleTableMapEvent` returned early for any
table that is not of interest without touching the map, so an entry recorded for
a table of interest survived indefinitely.

The binlog `table_id` is assigned by the server from its table definition cache
and is not stable: it can be reassigned to a different table across a server
restart, and also when a table is evicted from and reloaded into the cache. When
a table id previously bound to a table of interest was reassigned to a table
that is not of interest, `isValidTableId` still accepted the row events for that
table because the stale entry was present and pointed at a table of interest.
Those rows were then decoded using the wrong column definitions, which silently
wrote corrupted documents to the sink and raised type-conversion errors for the
rows whose types did not line up.

Remove the mapping held for a table id whenever a TABLE_MAP event shows the id
now refers to a table that is not of interest, so subsequent row events for that
id are skipped as intended.

Resolves opensearch-project#7125

Signed-off-by: Masaki Mori <masakigg@amazon.co.jp>
Co-authored-by: Masaki Mori <masakigg@amazon.co.jp>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

3 participants