Skip to content

SNOW-4537767: Map booleans and time-zone types in DataFrameReader.jdbc - #4530

Open
sfc-gh-yuwang wants to merge 2 commits into
yuwang/SNOW-4537773-jdbc-temp-functionsfrom
yuwang/SNOW-4537767-jdbc-type-mapping
Open

sfc-gh-yuwang wants to merge 2 commits into
yuwang/SNOW-4537773-jdbc-temp-functionsfrom
yuwang/SNOW-4537767-jdbc-type-mapping

Conversation

@sfc-gh-yuwang

@sfc-gh-yuwang sfc-gh-yuwang commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Stacked on #4527 (part of the read.jdbc() GA epic SNOW-4522813). Review the ticket implementation and its additive audit commits.

Summary

Schema inference now also returns the vendor type name, which tells booleans
apart from bit strings (JDBC reports both as BIT). Postgres bool and SQL Server
bit become BooleanType; MySQL BIT/BOOLEAN/TINYINT(1) stay strings because the
driver reports TINYINT(1) like BIT(1) although it can hold any small integer.

Time-zone aware columns are read with an explicit Java type instead of
getString(), whose output ("-8:00", region names, JVM-local wall time) Snowflake
cannot cast: OffsetDateTime for TIMESTAMP_TZ (Oracle TIMESTAMP WITH TIME ZONE,
SQL Server datetimeoffset, Postgres timestamptz) and an instant for Oracle
TIMESTAMP WITH LOCAL TIME ZONE. Unmapped types fall back to StringType like
dbapi instead of VariantType.

Behavior change (Private Preview API): Postgres bool/SQL Server bit columns become booleans, and Oracle/SQL Server time-zone types become timestamps instead of VARIANT strings. Existing expectations in test_jdbc_data.py were updated accordingly.

Measured on live sources (JDBC type, precision, getColumnTypeName):

Source column Reported as Before After
PG bool BIT, 1, bool 't'/'f' True/False
PG bit(1), bit(4) BIT, 1/4, bit '1', '1010' unchanged
MySQL BIT(1), BOOLEAN, TINYINT(1) BIT, 1, BIT (all three) '1', '1', '5' unchanged
SQL Server bit BIT, 1, bit '1' True
Oracle TS WITH TZ -, oracle.sql.TIMESTAMPTZ VARIANT '"... -8:00"' TIMESTAMP_TZ
Oracle TS WITH LOCAL TZ -, oracle.sql.TIMESTAMPLTZ VARIANT (JVM-local text) TIMESTAMP_LTZ
SQL Server datetimeoffset -, microsoft.sql.DateTimeOffset VARIANT TIMESTAMP_TZ
PG timestamptz TIMESTAMP, timestamptz NTZ, shifted to JVM time zone TIMESTAMP_TZ
Oracle INTERVAL, PG xml / timetz - VARIANT / failed cast STRING

Test plan

  • tests/unit/test_data_source.py: 84 passed; 9 of the 10 new cases fail without the fix (the 10th is the no-type-name fallback)
  • tests/integ/datasource/test_jdbc.py against live Oracle, PostgreSQL, MySQL and SQL Server: 48 passed; 5 of the 6 new cases fail without the fix (MySQL is unchanged by design)

Resumed audit (2026-11-19)

  • Verified explicit NTZ Java fallback, TZ/LTZ readers, boolean/vendor discrimination, and unknown String fallback. Approved conservative MySQL BIT/TINYINT(1) mapping remains unchanged.
  • Binary transport remains in sibling SNOW-4554440: Preserve binary values in DataFrameReader.jdbc #4534 (SNOW-4554440); it was not integrated here.
  • Validation at 113aae0: 85 unit tests and 6 focused live Oracle/PostgreSQL/MySQL/SQL Server tests passed. No additional code change needed.

Each PR was tested in an isolated worktree at its own remote head plus the additive changes noted above. No restack, sibling integration, merge, or Jira writes were performed; this is not validation of a combined stack.

Generated with Snowflake CoCo

Schema inference now also returns the vendor type name, which tells booleans
apart from bit strings (JDBC reports both as BIT). Postgres bool and SQL Server
bit become BooleanType; MySQL BIT/BOOLEAN/TINYINT(1) stay strings because the
driver reports TINYINT(1) like BIT(1) although it can hold any small integer.

Time-zone aware columns are read with an explicit Java type instead of
getString(), whose output ("-8:00", region names, JVM-local wall time) Snowflake
cannot cast: OffsetDateTime for TIMESTAMP_TZ (Oracle TIMESTAMP WITH TIME ZONE,
SQL Server datetimeoffset, Postgres timestamptz) and an instant for Oracle
TIMESTAMP WITH LOCAL TIME ZONE. Unmapped types fall back to StringType like
dbapi instead of VariantType.

Generated with [Snowflake CoCo](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code)

Co-authored-by: Snowflake CoCo <noreply@snowflake.com>
@sfc-gh-yuwang
sfc-gh-yuwang requested a review from a team as a code owner November 18, 2026 20:17
@sfc-gh-yuwang
sfc-gh-yuwang requested review from sfc-gh-aling, sfc-gh-bkogan and sfc-gh-jzeng and removed request for a team November 18, 2026 20:17
infer_schema_udtf_registration = f"""
CREATE OR REPLACE {self._temp_function_type} FUNCTION {infer_schema_udtf_name}(query VARCHAR)
RETURNS TABLE (field_name VARCHAR, jdbc_type VARCHAR, java_type VARCHAR, precision INTEGER, scale INTEGER, nullable BOOLEAN)
RETURNS TABLE (field_name VARCHAR, jdbc_type VARCHAR, java_type VARCHAR, precision INTEGER, scale INTEGER, nullable BOOLEAN, type_name VARCHAR)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review provided by Security Bot (#ask-pr-scanner)


SQL injection

🟠 MEDIUM · imports_sql and packages_sql interpolate import/package strings into UDTF DDL without escaping single quotes · CWE-89

Fix: Escape each imports and packages entry before embedding it in SQL, mirroring the existing str_to_sql/single-quote-doubling approach used elsewhere in this module.

Full description

In JDBC.__init__, self.imports_sql and self.packages_sql build SQL fragments by wrapping each entry of imports and packages in single quotes via f-strings (e.g. f"'{imp}'") without escaping embedded single quotes. These fragments are injected verbatim into the CREATE OR REPLACE ... FUNCTION DDL in schema and read. Unlike url, properties and session_init_statement (which are escaped with to_java_string_literal) and the inference query (escaped with str_to_sql), the imports/packages list items reach the SQL sink unescaped, so a value containing a single quote or ') can alter the generated DDL.

Impact

A caller who controls udtf_configs["imports"] or udtf_configs["packages"] (for example a stage path or package coordinate assembled from another untrusted source) can inject arbitrary SQL into the UDTF registration statement executed by self.session.sql(...).collect(), running in the Snowpark session's authority. The demonstrated effect is malformed/attacker-influenced DDL execution; it does not by itself cross a tenant boundary because the statement runs with the caller's own session privileges.

Recommendation steps

  • Escape each imports and packages entry before embedding it in SQL, mirroring the existing str_to_sql/single-quote-doubling approach used elsewhere in this module.
  • Validate imports entries against an allowed stage-path grammar and packages entries against the expected group:artifact[:version] coordinate shape before building imports_sql/packages_sql.
  • Document the input contract for udtf_configs["imports"]/["packages"] so callers do not forward untrusted values.

@codecov-commenter

codecov-commenter commented Nov 19, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.33%. Comparing base (d9eaf5e) to head (113aae0).
⚠️ Report is 1 commits behind head on yuwang/SNOW-4537773-jdbc-temp-functions.

Additional details and impacted files
@@                           Coverage Diff                            @@
##           yuwang/SNOW-4537773-jdbc-temp-functions    #4530   +/-   ##
========================================================================
  Coverage                                    95.32%   95.33%           
========================================================================
  Files                                          179      179           
  Lines                                        47106    47137   +31     
  Branches                                      8157     8163    +6     
========================================================================
+ Hits                                         44905    44939   +34     
+ Misses                                        1360     1359    -1     
+ Partials                                       841      839    -2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Replace the unreachable None enum comparison so unknown JDBC codes backed by java.sql.Timestamp do not inherit session timestamp semantics.

Generated with [Snowflake CoCo](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code)

Co-authored-by: Snowflake CoCo <noreply@snowflake.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants