Repository navigation
SNOW-4554440: Preserve binary values in DataFrameReader.jdbc - #4534
Open
sfc-gh-yuwang wants to merge 2 commits into
Open
sfc-gh-yuwang wants to merge 2 commits into
sfc-gh-yuwang wants to merge 2 commits into
Conversation
Read binary JDBC columns as bytes with Java-11-compatible hex transport and explicit HEX decoding to avoid driver text corruption and session-dependent conversion. Generated with [Snowflake CoCo](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code) Co-authored-by: Snowflake CoCo <noreply@snowflake.com>
sfc-gh-yuwang
requested review from
sfc-gh-bkogan,
sfc-gh-jzeng and
sfc-gh-yixie
and removed request for
a team
November 18, 2026 23:56
Verify partial and reordered overrides retain source reader positions and produce the intended binary or canonical hex values. Generated with [Snowflake CoCo](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code) Co-authored-by: Snowflake CoCo <noreply@snowflake.com>
2 tasks done
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## yuwang/SNOW-4537767-jdbc-type-mapping #4534 +/- ##
======================================================================
Coverage 95.32% 95.33%
======================================================================
Files 179 179
Lines 47139 47156 +17
Branches 8164 8168 +4
======================================================================
+ Hits 44937 44956 +19
+ Misses 1361 1360 -1
+ Partials 841 840 -1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
DataFrameReader.jdbcfetched binary columns withResultSet.getString(), corrupting arbitrary MySQL bytes and returning incompatible text representations across drivers. JDBCBINARY,VARBINARY,LONGVARBINARY, andBLOBnow produceBinaryTypevalues with the original bytes.getBytes(), encode uppercase hex with Java-11-compatible code, and applyTO_BINARY(..., 'HEX')rather than depend onBINARY_INPUT_FORMAT. Preserve NULL, empty values, high-bit bytes, and embedded zeros. MySQL BIT/TINYINT readers remain unchanged.custom_schema: string overrides return canonical uppercase hex; binary overrides on non-binary sources retain ordinary Snowflake cast semantics. Update API documentation, changelog, and recorded binary expectations.getBytes(), not streamed; the check occurs after the driver allocates the byte array.Dependency: stacked on open #4530 (
yuwang/SNOW-4537767-jdbc-type-mapping), exact base3e33f580a213954204018a3ce08aeb4cccb9ab63. Review against that branch; retarget after the dependency merges.Test plan
javac --release 11and execution for NULL/empty/all 256 byte values/embedded zero/high-bit/8 MiB boundary/oversize rejection.BINARY_INPUT_FORMAT, inferred/custom binary/custom string schemas, non-binary-to-binary override behavior, and 8 MiB/8 MiB + 1 byte boundary cases. Assertions execute and collect results; expected oversize errors are asserted, not swallowed.git diff --checkpassed.Initial new live tests exposed an invalid MySQL test fixture (
BINARY(256)exceeds MySQL's 255-byte maximum). Corrected the fixture and reran the entire live suite successfully. One existing unknowndatasourcepytest-marker warning remains. The full repository unit suite was not rerun (known unrelated failures); validation focused on changed and adjacent behavior.Compatibility / limitations
This intentionally changes Private Preview binary outputs from strings to bytes. Applications requesting strings receive uppercase hex rather than historical driver text. Oracle empty RAW remains NULL according to source semantics; empty BLOB remains empty bytes. Values above 8 MiB and streaming large LOBs remain unsupported.
Final-five audit (2026-11-19)
Added two mixed-column custom_schema regressions: partial binary-to-string overrides and complete overrides supplied in reverse source order. They execute the projection and verify source getter positions, column order, bytes and canonical hex. Updated datasource suite: 102 passed, including Java-11 hex boundary tests; commit hooks passed. Existing 64 live results above are prior validation, not rerun for this test-only addition. No new binary conversion defect was found in this review. The 8 MiB materialization limitation remains.
Generated with Snowflake CoCo