DuckSync v0.3.0 — Quack server, two-stage invalidation, transparent cached-table routing#16
Merged
Conversation
Updates CI workflow inputs to DuckDB/extension-ci-tools v1.5.4 and raises documented minimum DuckDB version in the README to match. Adds three new spike test scripts under `test/` to evaluate Snowflake pushdown behavior, Quack-backed control-plane backend feasibility, and warehouse-free `SHOW TABLES` metadata heuristics for invalidation. Also ignores `plans/` in `.gitignore`.
Introduces `ducksync_serve` and `ducksync_stop` to manage Quack server lifecycle from DuckSync, including named parameters for serve options and initialization checks. The query execution path was refactored to reuse shared streaming/global-state helpers and shared SQL string escaping, and `ducksync_query` now uses the same escaping helper for passthrough Snowflake calls. Adds `StorageManager::IsPostgresBacked()` to warn when serving without a Postgres-backed control plane, updates function registration/count assertions in existing SQL tests, and adds a dedicated `test_quack_server.test` to cover serve/start-stop behavior with `quack_server_list()`.
Introduces a configurable `invalidation_mode` for cache definitions (`last_altered`, `two_stage`, `ttl_only`, `manual`) and an optional `metadata_secret` parameter for `ducksync_create_cache`. The `two_stage` mode uses a warehouse-free Stage 1 check via `SHOW TABLES` rows/bytes snapshots before falling back to the more expensive `last_altered` metadata query, reducing unnecessary warehouse spin-ups. Snapshots are persisted in a new `ducksync.table_snapshots` table. Also refactors `GetSourceTableMetadata` to query each monitored table individually (by database), fixing cross-database monitoring edge cases.
Register a DuckDB replacement scan that maps monitored source table names to DuckSync cache tables, handling case-insensitive and schema-qualified lookups. If a table is monitored but has not been refreshed yet, return a clear error directing users to refresh or query through DuckSync. Adds SQL coverage for cache hits across different table name forms and the uncached-table error path.
Updates docs and test guidance for DuckSync’s v1.5.4 surface: explicit `ducksync_serve`/`ducksync_stop`, transparent cached-table routing via ReplacementScan, and two-stage invalidation with `metadata_secret`. It also records the Snowflake Arrow scanner constraint, adjusts manual tests to avoid multi-column `snowflake_query()` crashes, and adds `SNOWFLAKE_SCHEMA_ROLE` to test env docs. Build config now enforces C++17 on MSVC to keep Windows compilation aligned.
Adds `install-adbc-driver` to `.PHONY` and introduces a new Make target that locates `libadbc_driver_snowflake.so` under `~/.duckdb/extensions`. The target first tries a direct copy to `/usr/local/lib`, then falls back to a sudo symlink if needed, with clear error messages and guidance when the driver is missing or installation fails.
Applies style-only formatting updates across `ducksync_extension.cpp`, `metadata_manager.cpp`, and `query_router.cpp` by wrapping long function signatures, string literals, and chained expressions for readability. No functional behavior was changed.
DuckDB API changed between v1.5.4 and main: BaseTableRef field names changed from catalog_name/schema_name to catalog/schema, and PreparedStatement::GetNames() return type changed from vector<string> to vector<Identifier>. This commit adds CMake compile-time checks to detect the API shape and conditionally compiles code to support both versions. Compile definitions DUCKDB_BASETABLEREF_HAS_CATALOG and DUCKDB_GETNAMES_RETURNS_IDENTIFIER are set based on detection results.
Update CMake feature detection and extension code to support both DuckDB v1.5.4 and newer mainline API shapes. The changes switch BaseTableRef handling to detect legacy `catalog_name/schema_name/table_name` vs newer qualified `table_name.{catalog,schema,name}`, and update PreparedStatement `GetNames()` handling for `vector<string>` vs `vector<Identifier>` return types. This keeps table rewrite/replacement scan and bind result naming logic working across DuckDB versions.
Owner
Author
|
Replace scattered `#ifdef` version checks with a shared `duckdb_compat.hpp` shim that abstracts `BaseTableRef` field access and prepared statement name conversion across DuckDB API variants. This removes CMake compile-time feature probes and simplifies call sites in `ducksync_extension.cpp` and `query_router.cpp` by routing all compatibility logic through reusable helpers.
Adjust `duckdb_compat.hpp` to support both DuckDB v1.5.x and main+ table/name APIs without referencing new types in old headers. The helpers now use `GetQualifiedName`/`SetQualifiedName` for `BaseTableRef`, detect `GetIdentifierName()` on name elements, and keep legacy field-based behavior for older versions via template-dependent branching.
danjsiegel
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Quack server lifecycle
ducksync_serve(listen_uri [, token, allow_other_hostname, disable_ssl]) and ducksync_stop(listen_uri) let users start and stop a DuckDB Quack endpoint explicitly. DuckSync never auto-starts a listener.
Two-stage invalidation
ducksync_create_cache accepts two new named parameters: invalidation_mode (last_altered | two_stage | ttl_only | manual) and metadata_secret. The two_stage mode runs SHOW TABLES with a no-warehouse credential and only wakes compute when row or byte counts change. Snapshots are persisted in ducksync.table_snapshots. The default remains last_altered — no behavior change for existing caches.
Transparent cached-table routing
After a cache is refreshed, naked and qualified source table names resolve directly to the local DuckLake cache via a ReplacementScan hook. All three name forms work: orders, main.orders, TEST_DATA.ORDERS. Unrefreshed monitored tables raise a specific error. Unknown tables fall through to the normal DuckDB catalog error.
Infrastructure
MSVC C++17 enforcement in extension_config.cmake fixes Windows CI. make install-adbc-driver installs the Snowflake ADBC driver to /usr/local/lib/ (version-agnostic, cp/sudo ln -sf fallback). DuckDB bumped from v1.5.0 → v1.5.4.
Testing
Breaking changes
None. All existing call signatures are backward compatible.
Users upgrading DuckDB itself: run make install-adbc-driver to move the Snowflake ADBC driver to the version-agnostic path.