Repository navigation
feat: support fabric data warehouse engine - #104
Merged
Merged
Conversation
Ports the Fabric Data Warehouse engine into LakeBench OSS, covering all four benchmarks (TPC-DS, TPC-H, ClickBench, ELTBench). Engine-specific query normalization is registered per engine via ENGINE_QUERY_NORMALIZERS rather than gated on the target dialect, so the rules apply to this engine alone instead of every T-SQL-family renderer: - TPC-DS: sample stddev (q17, q29, q35, q39a, q39b) and the q22 inventory average now register against FabricDataWarehouse. - ClickBench: positional GROUP BY resolution, GROUP BY alias expansion and integer aggregate widening move off the _is_tsql() check; q29 splits into a shared backreference rule and an engine-only host-extraction rule. Rules that describe a renderer family rather than an engine workaround stay global (TPC-DS q9/q97 widening, ClickBench length and minute truncation, and the q29 backreference rule shared with Spark/Databricks/MySQL). Loading supports both Parquet (COPY INTO / OPENROWSET, with an explicit projection for column renames) and the native pipe-delimited generator output. The native path folds the trailing delimiter into the row terminator (0x7C0A) and refuses column_name_mapping, since COPY INTO maps fields positionally and a legacy-named DDL would silently load columns out of order. The ClickBench DDL declares EventTime, ClientEventTime and LocalEventTime as DATETIME2(6). The private implementation used TIME(0), which discarded the date component produced by DATEADD(SECOND, EventTime, '01-01-1970') and changed q43 results. FabricWarehouse remains available as an alias of FabricDataWarehouse, as does the fabric_warehouse extra. Normalizer versions bump to TPC-DS 13 and ClickBench 2. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Cost was derived from the capacity SKU and the Azure retail rate for Data Warehouse Capacity Usage. Capacity cost does not reliably attribute to an individual query or load, so the figure was misleading rather than merely imprecise. Removes the cost_per_hour parameter, the retail pricing lookup and the cost_per_hour result metadata. estimated_retail_job_cost is now left unset, which is the inherited behaviour when an engine publishes no rate. SKU lookup stays, since get_compute_size() reports it; _get_cost_per_hour is now _get_fabric_sku and no longer needs the capacity unit maths or the pricing request. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds Fabric Data Warehouse engine support across all benchmarks.