Skip to content

SNOW-4554443: Stream PostgreSQL JDBC reads and tag datasource actions - #4533

Open
sfc-gh-yuwang wants to merge 3 commits into
yuwang/SNOW-4537767-jdbc-type-mappingfrom
yuwang/SNOW-4554443-jdbc-parity
Open

sfc-gh-yuwang wants to merge 3 commits into
yuwang/SNOW-4537767-jdbc-type-mappingfrom
yuwang/SNOW-4554443-jdbc-parity

Conversation

@sfc-gh-yuwang

@sfc-gh-yuwang sfc-gh-yuwang commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

JDBC reads currently use the driver's default fetch size, and pgjdbc buffers the entire result even when callers supply a positive fetch size because autocommit remains enabled. Actions on JDBC DataFrames also miss the datasource statement parameter that DBAPI actions receive.

  • Default JDBC fetch_size to 10000, retaining explicit 0 as the driver-default escape hatch. Reject invalid non-integer/negative values before registering UDTFs.
  • For PostgreSQL positive-size reads, preserve session initialization under its original autocommit behavior, then disable autocommit before executing the source SELECT. Each stream owns its connection, so completing one cursor cannot invalidate another process call's cursor. Close result/statement, roll back the read transaction, and close the connection on exhaustion, exceptions, stream close, or endPartition. Query execution remains lazy. Other drivers and fetch_size=0 retain connection reuse.
  • Tag JDBC actions through the existing datasource helper with SNOWPARK_PYTHON_DATASOURCE_JDBC=1; retain caller statement parameters and DBAPI behavior. Use the existing JDBC SQL comment in generated DDL and read SQL.

Depends on #4530; based on exact commit 3e33f580a213954204018a3ce08aeb4cccb9ab63.

Correctness and tradeoffs

pgjdbc cursor requirements require positive fetch size, autocommit disabled, a forward-only result set (the JDBC default here), and a single statement. The public fetch-size docs now describe transaction ownership and the simple-query-mode limitation. Fetch size remains a driver hint, not a cross-driver memory guarantee; user-selected PostgreSQL simple query mode still buffers results.

Initialization is deliberately not moved into the cursor transaction: otherwise arbitrary initialization DML, ON COMMIT tables, and transaction-control statements would change behavior. Reads themselves are rolled back rather than implicitly committing source-side effects. Dedicated PostgreSQL connections cost additional connection setup per partition and do not preserve cross-partition session state; initialization runs independently for each read. Concurrent partitions do not provide a common source snapshot.

Validation

  • Baseline datasource unit suite: 84 passed.
  • Changed datasource suite plus dataframe, telemetry and query-tag suites: 165 passed. Generated Java compiled and executed against JDBC proxies covering lazy execution, overlapping streams, early close, endPartition cleanup, setup/query/iteration/conversion failures, rollback failure, and close failure.
  • 15 distinct live regression cases passed on SFCTEST0: initial selection 14 passed, then expanded cursor tests 3 passed (2 repeats + new default-size case). Includes PostgreSQL init/partition reads at 0/7/10000, actual driver buffering, collect/count/save_as_table/copy_into_location tagging, caller QUERY_TAG preservation, partial/full custom schemas, and existing PostgreSQL/MySQL/SQL Server boolean and Oracle/PostgreSQL/SQL Server timezone tests.
  • Actual streaming observed inside pgjdbc 42.7.7, not inferred from setFetchSize: exact baseline at size 7 retained 31 rows / no cursor; new handler retained 7,7,7,7,3 row batches / cursor present with identical 31-row results. New default checked over 20003 rows: retained batches 10000,10000,3. Reflection is confined to the pinned-driver test, not production code.
  • Scoped black/flake8 and all applicable commit hooks passed. Clean worktree after commit. Ignored credentials/proto prerequisites were not committed.

The Python connector's named snowpark OAuth token was rejected, so live runs used the existing repository fixture credentials, verified against the same SFCTEST0 account. Source mutations were limited to connection-local temporary tables; throughput probes used generated SELECT rows. No production source writes.

Bounded fetch-size measurements

Two runs per case, 20000 rows x 9 columns (id + eight 64-character strings), one partition, query timeout 60 seconds. Times are action/collect seconds, including Snowflake execution and client transfer but excluding inference/registration. All rows, ID sums, and payload widths were checked. Exact baseline JDBC implementation loaded from the base commit; pgjdbc 42.7.7 and ojdbc17 23.9.0.25.07. No unbounded load or external table population.

Driver / implementation Fetch size Run 1 (s) Run 2 (s)
PostgreSQL baseline 0 3.343 2.407
PostgreSQL baseline (still buffered) 10000 3.433 2.400
PostgreSQL new 10000 3.138 2.941
PostgreSQL new 100000 2.892 2.285
Oracle baseline 0 5.894 4.484
Oracle new 10000 3.445 2.826
Oracle new 100000 3.187 3.446

10000 is a conservative compromise: much larger than a small driver default, but a 10x smaller requested row buffer than DBAPI's 100000. Oracle improved in this bounded sample; PostgreSQL trades extra cursor round trips for bounded result buffering. These small, variable measurements are not a universal performance claim, and row count is not a byte limit. The ticket's larger Oracle measurement is supporting context, not a measurement reproduced by this PR.

Remaining scope / explicit blocker

No custom_schema inference bypass or fallback is implemented. The existing contract allows partial, name-based overrides in arbitrary order. Without metadata we cannot establish completeness, validate names/source order, or choose the source-specific readers: a Snowflake STRING override can still require OffsetDateTime/instant conversion for a source TZ/LTZ value. A fallback after inference failure would also conceal missing source columns, permissions, or broken init SQL. Binary conversion is being addressed separately and must keep its metadata-dependent reader decisions.

A safe follow-up requires an explicit complete-schema/positional contract (including duplicate/case-sensitive names and drift validation), plus runtime source-metadata reader selection or a caller-provided source-reader contract. Destination types alone are insufficient. This PR instead locks in existing partial/full override behavior and failure propagation with regressions.

The audit also removes max_workers guidance and the inaccurate inference-bypass/unlimited-timeout claims, with a documentation regression compatible with the fuller sibling docs. Binary fixes, comprehensive examples, CI enablement, and GA release remain separate. No ticket is closed by this PR.

Final-five audit (2026-11-19)

121 datasource unit tests passed on the updated branch, including executable generated-Java lifecycle tests. No live tests were rerun for these documentation-only corrections. The custom_schema bypass remains unimplemented and requires an explicit API decision; this is not excluded rollout work. Sibling #4532 still describes fetch_size=0 and no PostgreSQL cursor setup and must be reconciled when integrating the branches. The inherited inference handler lacks early-stream-close cleanup; that original-stack lifecycle concern is not changed here.

Generated with Snowflake CoCo

Co-authored-by: Snowflake CoCo noreply@snowflake.com

Use a 10000-row fetch hint and isolate cursor transactions after session initialization so reads release resources without changing initialization autocommit. Preserve metadata-based custom schema behavior until a complete-schema contract is defined.

Generated with [Snowflake CoCo](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code)

Co-authored-by: Snowflake CoCo <noreply@snowflake.com>
@sfc-gh-yuwang
sfc-gh-yuwang requested a review from a team as a code owner November 18, 2026 23:56
@sfc-gh-yuwang
sfc-gh-yuwang requested review from sfc-gh-aling, sfc-gh-mayliu and sfc-gh-yixie and removed request for a team November 18, 2026 23:56
sfc-gh-yuwang and others added 2 commits November 18, 2026 16:46
Remove unsupported worker guidance and make inference and timeout limitations explicit without changing the custom-schema API.

Generated with [Snowflake CoCo](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code)

Co-authored-by: Snowflake CoCo <noreply@snowflake.com>
Permit the sibling documentation's explicit unsupported-parameter explanation without weakening the inference requirement.

Generated with [Snowflake CoCo](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code)

Co-authored-by: Snowflake CoCo <noreply@snowflake.com>
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 80.00000% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.82%. Comparing base (3e33f58) to head (79b1217).
⚠️ Report is 1 commits behind head on yuwang/SNOW-4537767-jdbc-type-mapping.

Files with missing lines Patch % Lines
...c/snowflake/snowpark/_internal/data_source/jdbc.py 50.00% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@                            Coverage Diff                             @@
##           yuwang/SNOW-4537767-jdbc-type-mapping    #4533       +/-   ##
==========================================================================
- Coverage                                  95.32%   78.82%   -16.51%     
==========================================================================
  Files                                        179      178        -1     
  Lines                                      47139    47051       -88     
  Branches                                    8164     8166        +2     
==========================================================================
- Hits                                       44937    37089     -7848     
- Misses                                      1361     8391     +7030     
- Partials                                     841     1571      +730     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants