Skip to content

feat: Add Spark SQL as a query engine - #2241

Open
ferenc-csaky wants to merge 1 commit into
mainfrom
feat/sparksql-dialect
Open

feat: Add Spark SQL as a query engine#2241
ferenc-csaky wants to merge 1 commit into
mainfrom
feat/sparksql-dialect

Conversation

@ferenc-csaky

Copy link
Copy Markdown
Collaborator

Key Changes

  • Add the sparksql query engine with Spark SQL statement generation backed by Iceberg deployment artifacts.
  • Add Spark SQL Calcite conversion and dialect support, including backtick identifiers and Spark-native scalar, temporal, array, map, and struct type rendering.
  • Refactor JDBC DDL generation to quote table, view, index, and column identifiers through the active SQL dialect.
  • Make query-engine server configuration optional so engines without runtime server configuration, such as Spark SQL, are supported.
  • Add a Spark SQL compile use case and snapshot covering primitive, temporal, binary, collection, row, and aggregate result types.

@ferenc-csaky
ferenc-csaky force-pushed the feat/sparksql-dialect branch from 2b05e18 to 4e4dd6e Compare July 29, 2026 18:25
@codecov

codecov Bot commented Jul 29, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 20.37037% with 86 lines in your changes missing coverage. Please review.
✅ Project coverage is 17.61%. Comparing base (8b8622f) to head (4e4dd6e).
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
...asqrl/calcite/dialect/ExtendedSparkSqlDialect.java 18.18% 27 Missing ⚠️
...engine/database/relational/ddl/CreateIndexDDL.java 0.00% 13 Missing ⚠️
...atasqrl/config/QueryEngineConfigConverterImpl.java 0.00% 7 Missing ⚠️
...qrl/engine/database/relational/SparkSqlEngine.java 0.00% 7 Missing ⚠️
.../database/relational/SparkSqlStatementFactory.java 0.00% 5 Missing ⚠️
.../database/relational/PostgresStatementFactory.java 0.00% 4 Missing ⚠️
...e/database/relational/ddl/DdlIdentifierQuoter.java 33.33% 3 Missing and 1 partial ⚠️
...se/relational/ddl/GenericCreateViewDdlFactory.java 0.00% 4 Missing ⚠️
...ine/database/relational/SparkSqlEngineFactory.java 0.00% 3 Missing ⚠️
...e/relational/ddl/GenericCreateTableDdlFactory.java 70.00% 3 Missing ⚠️
... and 8 more
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #2241      +/-   ##
============================================
- Coverage     17.63%   17.61%   -0.02%     
- Complexity     1077     1082       +5     
============================================
  Files           610      617       +7     
  Lines         17762    17835      +73     
  Branches       2167     2168       +1     
============================================
+ Hits           3132     3142      +10     
- Misses        14312    14374      +62     
- Partials        318      319       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

@ferenc-csaky
ferenc-csaky force-pushed the feat/sparksql-dialect branch from 4e4dd6e to 99114b2 Compare July 30, 2026 17:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant