Repository navigation
SNOW-4554443: Stream PostgreSQL JDBC reads and tag datasource actions - #4533
Open
sfc-gh-yuwang wants to merge 3 commits into
Open
sfc-gh-yuwang wants to merge 3 commits into
sfc-gh-yuwang wants to merge 3 commits into
Conversation
Use a 10000-row fetch hint and isolate cursor transactions after session initialization so reads release resources without changing initialization autocommit. Preserve metadata-based custom schema behavior until a complete-schema contract is defined. Generated with [Snowflake CoCo](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code) Co-authored-by: Snowflake CoCo <noreply@snowflake.com>
sfc-gh-yuwang
requested review from
sfc-gh-aling,
sfc-gh-mayliu and
sfc-gh-yixie
and removed request for
a team
November 18, 2026 23:56
Remove unsupported worker guidance and make inference and timeout limitations explicit without changing the custom-schema API. Generated with [Snowflake CoCo](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code) Co-authored-by: Snowflake CoCo <noreply@snowflake.com>
Permit the sibling documentation's explicit unsupported-parameter explanation without weakening the inference requirement. Generated with [Snowflake CoCo](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code) Co-authored-by: Snowflake CoCo <noreply@snowflake.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## yuwang/SNOW-4537767-jdbc-type-mapping #4533 +/- ##
==========================================================================
- Coverage 95.32% 78.82% -16.51%
==========================================================================
Files 179 178 -1
Lines 47139 47051 -88
Branches 8164 8166 +2
==========================================================================
- Hits 44937 37089 -7848
- Misses 1361 8391 +7030
- Partials 841 1571 +730 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
JDBC reads currently use the driver's default fetch size, and pgjdbc buffers the entire result even when callers supply a positive fetch size because autocommit remains enabled. Actions on JDBC DataFrames also miss the datasource statement parameter that DBAPI actions receive.
fetch_sizeto 10000, retaining explicit0as the driver-default escape hatch. Reject invalid non-integer/negative values before registering UDTFs.SNOWPARK_PYTHON_DATASOURCE_JDBC=1; retain caller statement parameters and DBAPI behavior. Use the existing JDBC SQL comment in generated DDL and read SQL.Depends on #4530; based on exact commit
3e33f580a213954204018a3ce08aeb4cccb9ab63.Correctness and tradeoffs
pgjdbc cursor requirements require positive fetch size, autocommit disabled, a forward-only result set (the JDBC default here), and a single statement. The public fetch-size docs now describe transaction ownership and the simple-query-mode limitation. Fetch size remains a driver hint, not a cross-driver memory guarantee; user-selected PostgreSQL simple query mode still buffers results.
Initialization is deliberately not moved into the cursor transaction: otherwise arbitrary initialization DML, ON COMMIT tables, and transaction-control statements would change behavior. Reads themselves are rolled back rather than implicitly committing source-side effects. Dedicated PostgreSQL connections cost additional connection setup per partition and do not preserve cross-partition session state; initialization runs independently for each read. Concurrent partitions do not provide a common source snapshot.
Validation
The Python connector's named
snowparkOAuth token was rejected, so live runs used the existing repository fixture credentials, verified against the same SFCTEST0 account. Source mutations were limited to connection-local temporary tables; throughput probes used generated SELECT rows. No production source writes.Bounded fetch-size measurements
Two runs per case, 20000 rows x 9 columns (id + eight 64-character strings), one partition, query timeout 60 seconds. Times are action/collect seconds, including Snowflake execution and client transfer but excluding inference/registration. All rows, ID sums, and payload widths were checked. Exact baseline JDBC implementation loaded from the base commit; pgjdbc 42.7.7 and ojdbc17 23.9.0.25.07. No unbounded load or external table population.
10000 is a conservative compromise: much larger than a small driver default, but a 10x smaller requested row buffer than DBAPI's 100000. Oracle improved in this bounded sample; PostgreSQL trades extra cursor round trips for bounded result buffering. These small, variable measurements are not a universal performance claim, and row count is not a byte limit. The ticket's larger Oracle measurement is supporting context, not a measurement reproduced by this PR.
Remaining scope / explicit blocker
No custom_schema inference bypass or fallback is implemented. The existing contract allows partial, name-based overrides in arbitrary order. Without metadata we cannot establish completeness, validate names/source order, or choose the source-specific readers: a Snowflake STRING override can still require OffsetDateTime/instant conversion for a source TZ/LTZ value. A fallback after inference failure would also conceal missing source columns, permissions, or broken init SQL. Binary conversion is being addressed separately and must keep its metadata-dependent reader decisions.
A safe follow-up requires an explicit complete-schema/positional contract (including duplicate/case-sensitive names and drift validation), plus runtime source-metadata reader selection or a caller-provided source-reader contract. Destination types alone are insufficient. This PR instead locks in existing partial/full override behavior and failure propagation with regressions.
The audit also removes max_workers guidance and the inaccurate inference-bypass/unlimited-timeout claims, with a documentation regression compatible with the fuller sibling docs. Binary fixes, comprehensive examples, CI enablement, and GA release remain separate. No ticket is closed by this PR.
Final-five audit (2026-11-19)
121 datasource unit tests passed on the updated branch, including executable generated-Java lifecycle tests. No live tests were rerun for these documentation-only corrections. The custom_schema bypass remains unimplemented and requires an explicit API decision; this is not excluded rollout work. Sibling #4532 still describes fetch_size=0 and no PostgreSQL cursor setup and must be reconciled when integrating the branches. The inherited inference handler lacks early-stream-close cleanup; that original-stack lifecycle concern is not changed here.
Generated with Snowflake CoCo
Co-authored-by: Snowflake CoCo noreply@snowflake.com