Skip to content

SNOW-4554440: Preserve binary values in DataFrameReader.jdbc - #4534

Open
sfc-gh-yuwang wants to merge 2 commits into
yuwang/SNOW-4537767-jdbc-type-mappingfrom
yuwang/SNOW-4554440-jdbc-binary
Open

sfc-gh-yuwang wants to merge 2 commits into
yuwang/SNOW-4537767-jdbc-type-mappingfrom
yuwang/SNOW-4554440-jdbc-binary

Conversation

@sfc-gh-yuwang

@sfc-gh-yuwang sfc-gh-yuwang commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

DataFrameReader.jdbc fetched binary columns with ResultSet.getString(), corrupting arbitrary MySQL bytes and returning incompatible text representations across drivers. JDBC BINARY, VARBINARY, LONGVARBINARY, and BLOB now produce BinaryType values with the original bytes.

  • Use the per-column reader mechanism to call getBytes(), encode uppercase hex with Java-11-compatible code, and apply TO_BINARY(..., 'HEX') rather than depend on BINARY_INPUT_FORMAT. Preserve NULL, empty values, high-bit bytes, and embedded zeros. MySQL BIT/TINYINT readers remain unchanged.
  • Keep source-driven readers with custom_schema: string overrides return canonical uppercase hex; binary overrides on non-binary sources retain ordinary Snowflake cast semantics. Update API documentation, changelog, and recorded binary expectations.
  • Reject binary values above 8 MiB explicitly: hex doubles the payload in the read UDTF's default 16 MiB VARCHAR output. LOBs are materialized with getBytes(), not streamed; the check occurs after the driver allocates the byte array.

Dependency: stacked on open #4530 (yuwang/SNOW-4537767-jdbc-type-mapping), exact base 3e33f580a213954204018a3ce08aeb4cccb9ab63. Review against that branch; retarget after the dependency merges.

Test plan

  • Baseline: 84 data-source unit tests and all 48 existing live JDBC tests passed.
  • Updated data-source unit suite: 100 passed, including actual generated Java hex helper compilation with javac --release 11 and execution for NULL/empty/all 256 byte values/embedded zero/high-bit/8 MiB boundary/oversize rejection.
  • Data-source plus adjacent schema-parsing unit suites: 234 passed.
  • Full updated live JDBC suite: 64 passed in 476.67s, using existing integration setup. Oracle RAW/BLOB, PostgreSQL bytea, MySQL binary/varbinary/blob families, SQL Server binary/varbinary(max)/image and existing rowversion fixtures; SQL Server runs Java 11.
  • Live NULL/empty/high-bit/embedded-zero/all-byte cases, UTF-8 BINARY_INPUT_FORMAT, inferred/custom binary/custom string schemas, non-binary-to-binary override behavior, and 8 MiB/8 MiB + 1 byte boundary cases. Assertions execute and collect results; expected oversize errors are asserted, not swallowed.
  • New mutable source fixtures use connection-local temporary tables only. Existing persistent source tables are read-only; Oracle table BLOB values are checked against source-side hex.
  • All applicable pre-commit hooks and git diff --check passed.

Initial new live tests exposed an invalid MySQL test fixture (BINARY(256) exceeds MySQL's 255-byte maximum). Corrected the fixture and reran the entire live suite successfully. One existing unknown datasource pytest-marker warning remains. The full repository unit suite was not rerun (known unrelated failures); validation focused on changed and adjacent behavior.

Compatibility / limitations

This intentionally changes Private Preview binary outputs from strings to bytes. Applications requesting strings receive uppercase hex rather than historical driver text. Oracle empty RAW remains NULL according to source semantics; empty BLOB remains empty bytes. Values above 8 MiB and streaming large LOBs remain unsupported.

Final-five audit (2026-11-19)

Added two mixed-column custom_schema regressions: partial binary-to-string overrides and complete overrides supplied in reverse source order. They execute the projection and verify source getter positions, column order, bytes and canonical hex. Updated datasource suite: 102 passed, including Java-11 hex boundary tests; commit hooks passed. Existing 64 live results above are prior validation, not rerun for this test-only addition. No new binary conversion defect was found in this review. The 8 MiB materialization limitation remains.

Generated with Snowflake CoCo

Read binary JDBC columns as bytes with Java-11-compatible hex transport and explicit HEX decoding to avoid driver text corruption and session-dependent conversion.

Generated with [Snowflake CoCo](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code)

Co-authored-by: Snowflake CoCo <noreply@snowflake.com>
@sfc-gh-yuwang
sfc-gh-yuwang requested a review from a team as a code owner November 18, 2026 23:56
@sfc-gh-yuwang
sfc-gh-yuwang requested review from sfc-gh-bkogan, sfc-gh-jzeng and sfc-gh-yixie and removed request for a team November 18, 2026 23:56
Verify partial and reordered overrides retain source reader positions and produce the intended binary or canonical hex values.

Generated with [Snowflake CoCo](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code)

Co-authored-by: Snowflake CoCo <noreply@snowflake.com>
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.33%. Comparing base (3e33f58) to head (fe825df).
⚠️ Report is 1 commits behind head on yuwang/SNOW-4537767-jdbc-type-mapping.

Additional details and impacted files
@@                          Coverage Diff                           @@
##           yuwang/SNOW-4537767-jdbc-type-mapping    #4534   +/-   ##
======================================================================
  Coverage                                  95.32%   95.33%           
======================================================================
  Files                                        179      179           
  Lines                                      47139    47156   +17     
  Branches                                    8164     8168    +4     
======================================================================
+ Hits                                       44937    44956   +19     
+ Misses                                      1361     1360    -1     
+ Partials                                     841      840    -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants