From 4ec8d0d042f36bdc6fd7f506866c7ac2d90ecda4 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Wed, 5 Aug 2026 16:49:12 +0200 Subject: [PATCH 01/98] DEV-1744: one naming authority + one ValueKey renderer (B4, B5, B10) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PR 1 of the DEV-1742 consolidation. Foundations for the doctrine: every CTE name minted by the allocator (P-F), and one ValueKey render policy (P-G). Superseded code stays callable per the chain's operating rule. B4 — cross-model CTE naming. `_cm_` names were built by a doubly-lossy helper (`flat_name`, itself non-injective, then a non-identifier `re.sub`) and the resulting string doubled as the plan's identity key in `seen_cm`. Two failures were reachable from the public query API: * two measures whose canonical aliases differ only in case emitted two `_cm_` names that fold together on every case-folding dialect, so the collision belt raised. `_wm_` had been retrofitted onto the allocator; `_cm_` never was. * two genuinely distinct aggregates that sanitised alike made the second skip the loop body, leaving its join-back and column-alias maps unwritten — `KeyError` at the first unconditional downstream subscript. Dedup now keys on the typed AggregateKey plus source relation; the name is allocator-minted once and stored, so the five sites that re-derived it read it instead. Deliberately NOT keyed on the canonical alias: that omits `column_filter_key`, so a filtered and an unfiltered aggregate over one column share an alias while needing two CTEs — deduping on the string would silently merge them, a wrong answer rather than a crash. B5 — one ScalarCall policy. Two of the five renderers returned `exp.Anonymous` passthrough, so `ifnull` reached Postgres, which has no `IFNULL`, while the same key emitted `COALESCE` from a projection. All six paths now call one `render_scalar_call`. Transpiling alone was not enough: `exp.func("LOG10", x)` normalises to a generic `Log(10, x)` re-emitting as `LOG(10, x)`, wrong on dialects with a native single-arg `LOG10` — so the policy is transpile-then-log-rewrite. B10 — `ScopeFrame._model_for` raises instead of falling back to the root model, which expanded a different model's derived SQL and turned a wiring bug into a wrong answer. Also: four drifted copies of canonical-aggregate-alias derivation collapse to one profile-based function in `naming.py` (the four callers delegate, keeping their signatures); the windowed `exp.Sum if agg == "sum" else exp.Avg` catch-all becomes a registry lookup that raises; `_build_agg`'s five dispatch mechanisms become one table; step-CTE names and the structural aliases route through the allocator or a shared constant. Approved behavior change beyond the B-items: `concat` now emits `||` on Postgres/DuckDB where it emitted `CONCAT(...)`. Semantic, not cosmetic — Postgres `CONCAT()` ignores NULL operands and `||` propagates them — and kept for consistency, since the projection path has always emitted `||`. The renderer's API is complete and tested but production paths are not yet rerouted through it; that lands with the cross-scope migration in PR 3. Rationale and per-call-site detail in the value_expr module docstring, DECISIONS.md, and a handoff comment on DEV-1746. Tests: 196 new (result-key contract pack, naming/allocator, renderer). Full non-integration suite 9503 passed; SQLite+DuckDB integration 118 passed; ruff clean. Co-Authored-By: Claude Fable 5 --- DECISIONS.md | 2 + slayer/core/errors.py | 23 + slayer/engine/cross_model_planner.py | 44 +- slayer/engine/planning.py | 33 +- slayer/engine/stage_planner.py | 66 +- slayer/sql/dialects/tsql.py | 4 +- slayer/sql/generator.py | 187 +-- slayer/sql/naming.py | 183 ++- slayer/sql/render/__init__.py | 11 + slayer/sql/render/aggregates.py | 121 ++ slayer/sql/render/value_expr.py | 349 +++++ slayer/sql/scope.py | 28 +- slayer/sql/stage_wrapper.py | 4 +- .../dialects/test_multi_dialect_generation.py | 6 +- tests/test_dev1744_naming_allocator.py | 1046 ++++++++++++++ tests/test_dev1744_result_key_contract.py | 558 ++++++++ tests/test_dev1744_value_expr.py | 1215 +++++++++++++++++ tests/test_parity_guards.py | 9 +- tests/test_sql_generator.py | 14 +- 19 files changed, 3708 insertions(+), 195 deletions(-) create mode 100644 slayer/sql/render/__init__.py create mode 100644 slayer/sql/render/aggregates.py create mode 100644 slayer/sql/render/value_expr.py create mode 100644 tests/test_dev1744_naming_allocator.py create mode 100644 tests/test_dev1744_result_key_contract.py create mode 100644 tests/test_dev1744_value_expr.py diff --git a/DECISIONS.md b/DECISIONS.md index fca4666b..9e41f3ef 100644 --- a/DECISIONS.md +++ b/DECISIONS.md @@ -95,3 +95,5 @@ implementation detail. Include issue refs when known. - 2026-08-03 — Optional blocks + Cube JS/FILTER_PARAMS import (DEV-1730 / #270): a Mode-A-only `{? ... ?}` block renders its content parenthesised when every inner `{var}` is supplied, else collapses to the neutral `(1=1)` — the SLayer form of a Cube `FILTER_PARAMS` optional pushdown. Blocks live in the same `substitute_variables` (escape="sql") scanner as `{var}`/`{{`/`}}`, must contain ≥1 var, do not nest, and are rejected in Mode-B. A block-bearing model runs substitution even on a zero-variable call so its blocks collapse (the `_substitute_model_sql_surfaces` fast-path now checks for `{?` too); a block-free, required-only model with zero variables is still left untouched (the documented DEV-1625 raw-brace-literal boundary). `extract_model_variables(model)` derives required (bare, no default) vs optional (in-block or defaulted) from the four Mode-A surfaces — structural, nothing persisted, surfaced additively in the inspect skeleton `Variables:` line. The Cube importer gains a **JavaScript front-end** (esprima ESTree parser, a new core dep) that parses `cube()`/`view()` into the same `CubeCube`/`CubeView` shapes as YAML (dynamic values → report + skip member). FILTER_PARAMS refs are carried JS→converter as structured `CubeFilterParamRef` on the transient `CubeCube` (sentinels in the surface text; no arrow-body re-parse, sidestepping the `{var}`-vs-`{FILTER_PARAMS…}` brace clash); the converter resolves sentinels AFTER `translate_cube_refs` so the introduced `{var}` are never eaten. Requiredness (bare vs block) is decided in the converter alone via `honor_required_meta` (default on; CLI `--ignore-required-meta`) AND the member's `meta.required`; with the flag off a scalar-position arrow collapses to Cube's own `(1=1)::TIMESTAMP` booby-trap, faithfully. Cross-cube refs, unknown members, and generated-name collisions (`d`→`d_from` clashing member `d_from`) drop the cube (`filter_params_unsupported`); each logical variable is reported once (`filter_params_variable`) and stashed in `meta.cube_variables`. `render_probe_text` (blocks→`(1=1)`, bare vars→`0`) is the single import-time validation renderer, matching runtime collapse. - 2026-08-04 — Dialect-aware / complete escaping for Mode-A `{variable}` substitution (DEV-1727), hardening DEV-1625. `substitute_variables(..., escape="sql")` is now **dialect-aware** and **fail-closed**: it gained a required keyword-only `backslash_escapes` signal (`bool | None`, raises if `None` in sql mode) so a caller rendering raw SQL can never silently under-escape. On backslash-escaping dialects (MySQL/ClickHouse/Snowflake/Redshift/BigQuery/Databricks/Spark) it doubles the backslash before escaping the single quote; on standard dialects it keeps the `''` quote-doubling. The double quote is deliberately left untouched — inside a single-quoted literal `\"` is NOT a recognised escape on 6 of the 7 backslash dialects (only MySQL), so escaping it would corrupt the value. The regime is DERIVED from sqlglot's own tokenizer via `SqlDialect.backslash_escapes_strings` (= `"\\" in tokenizer.STRING_ESCAPES`, guarded + 14-dialect pinned) so our escaping can never drift from the parser that reads the substituted SQL. `escape="python"` (Mode-B) additionally encodes the full C0 control range (`\t`/`\n`/`\r` named, rest `\xNN`) so raw newlines/NUL no longer break `ast.parse`. Engine fail-closed: `_substitute_model_sql_surfaces` / `_render_probe_model` require a `dialect`, threaded from the resolved datasource — no bare bool to forget. Assumes MySQL's default `sql_mode` (backslash escapes on); `NO_BACKSLASH_ESCAPES` servers are a sqlglot-layer-wide limitation, documented not fixed. The SQLite backslash end-to-end gap stays a pinned strict-xfail (pre-existing, out of scope). Bound parameters rejected (don't fit substitute-into-raw-SQL). Nested/join/cross-model lineages remain DEV-1678. - 2026-08-04 — Declared list-valued `{variable}` coercion (DEV-1730 follow-up): a scalar supplied for a variable the model declares `list_valued` is wrapped into a one-element list before Mode-A substitution, so an importer-generated `col IN ({var})` renders `IN ('US')` rather than the unquoted `IN (US)`. The generic scalar rule (author writes the quotes, so `{var}` also works in numeric/fragment positions like `amount >= {floor}` and `{d}::TIMESTAMP`) is CORRECT and unchanged — it just presumes an author who can see the SQL position, which a machine-generated fixed template does not have; the caller cannot supply per-element quotes through parentheses the importer wrote. Silent-wrong-answer risk drove the fix over a raise: `region IN (US)` parses as a column reference, so it fails at the database with a confusing message, or resolves against a real column and returns wrong rows. Opt-in is a front-end-NEUTRAL flag: the Cube converter writes `list_valued: ref.kind == "string"` into each `meta.cube_variables` entry (arrow forms splice pre-quoted scalars and stay `False`), and the engine reads only that flag — never Cube's `kind` taxonomy — so a future list-shaped front-end opts in the same way. Coercion lives at the single Mode-A choke point `_substitute_model_sql_surfaces` (execution and the `_render_probe_model` type-probe both route through it, so it cannot be bypassed) via `coerce_declared_list_variables` / `list_valued_variable_names` in `slayer/core/query.py`. Scope is deliberately narrow: only `str`/`int`/`float`/`bool` are wrapped; `list`/`tuple` pass through (the **empty list still raises** — "no filter" belongs to an optional block or a sentinel default); `None`/`dict` are left for `_render_variable_value` to reject with its own naming error; hand-written models declare nothing and are untouched. Follow-on from the same review: `declares_variables(model)` (any non-empty `meta.cube_variables`) now also defeats the DEV-1625 zero-variable fast path, via the shared `_model_needs_substitution_pass` predicate used by both `_substitute_model_sql_surfaces` and `_render_probe_model`. This closes the fast-path hole for a GENERATED model whose pushdowns are all required (no `{? ?}` block to force the pass): such a model used to emit a bare `{var}` into the SQL on a zero-variable call instead of raising the documented missing-variable error. The hole stays open — deliberately — for hand-written models, which declare nothing and keep the raw-brace-literal protection (`'{1,2,3}'`). The `list_valued` flag is matched with `is True`, not truthiness, since `meta` is user-extensible and a stray `1` or the string `"false"` must not switch substitution semantics. The bag is also SELF-IDENTIFYING — an entry counts only with a string `member` (the shape every importer writes) — so a hand-written `meta` that reuses the `cube_variables` key is not mistaken for generated SQL and silently stripped of its brace-literal protection. +- 2026-08-05 — One naming authority + one ValueKey renderer (DEV-1744, PR 1 of the DEV-1742 consolidation). Four consequences worth recording. (1) **Cross-model CTE dedup is now structural.** `_cm_` CTE names were minted by a doubly-lossy helper (`flat_name`, itself documented non-injective, then a non-identifier `re.sub`) and the resulting string doubled as the plan's identity key in `seen_cm`. Two reachable failures followed: two measures whose canonical aliases differ only in case emitted two `_cm_` names that fold together on every case-folding dialect (the collision belt raised — `_wm_` had been retrofitted onto the allocator years earlier, `_cm_` never was), and two genuinely distinct aggregates that sanitised alike made the second skip the loop body, leaving its join-back and column-alias maps unwritten and raising `KeyError` downstream. The dedup key is now the typed `AggregateKey` plus the source relation; the name is allocator-minted and stored once, so the five sites that used to re-derive it now read it. Deliberately NOT keyed on the canonical alias: `canonical_agg_name` omits `column_filter_key`, so a filtered and an unfiltered aggregate over one column share an alias while needing two CTEs — deduping on the string would silently merge them, a wrong answer rather than a crash. (2) **One ScalarCall policy.** Two of the five renderers returned `exp.Anonymous` passthrough, so `ifnull` reached Postgres — which has no `IFNULL` — while the same key emitted `COALESCE` from a projection. All six render paths now call one `render_scalar_call`. Transpiling alone was NOT sufficient: `exp.func("LOG10", x)` normalises to a generic `Log(10, x)` that re-emits as `LOG(10, x)`, wrong on dialects with a native single-arg `LOG10`, so the policy is transpile-then-log-alias-rewrite. Consequence surfaced and approved: `concat` now emits the `||` operator on Postgres/DuckDB where it previously emitted `CONCAT(...)`. That is a semantic change as well as a spelling one — Postgres `CONCAT()` ignores NULL operands, `||` propagates them. Ratified as the kept behavior: the projection path has always emitted `||`, so before this change SLayer disagreed with itself between filters and projections on the same input, and consistency was chosen over preserving the filter-side NULL tolerance. (3) **`ScopeFrame._model_for` raises** instead of falling back to the root model; the fallback turned a wiring bug into a wrong answer by expanding a different model's derived SQL. (4) **Named P-F carve-outs**, recorded rather than omitted: the T-SQL ORDER-BY-detach rewrite and the stage-schema wrapper take `_outer` / `_stage_inner` as shared CONSTANTS rather than allocator-minted names (the former is a post-generation AST pass with no allocator in reach, and a later PR rebuilds that machinery); `base` / `_base` / `_combined` keep their literal names but are reserved into the allocator. Superseded code is retained and callable per the chain's operating rule — deletion happens in the final PR. +- 2026-08-05 — Renderer call-site migration deferred to the scope-assembly PR (DEV-1744 → DEV-1746). PR 1 lands the complete, tested `render_value_key` API but does NOT reroute the generator's own render paths through it; only the ScalarCall policy is genuinely shared (all six paths call `render_scalar_call`). Deferred because the filter and composite paths carry state the context does not yet consume — the local-aggregate HAVING branch with its slot lookup and inline-aggregate emission, the first/last ranked state, the filter-side CAST policy, and the rn-suffix / filtered-rank / composite-alias / resolved-kwarg maps — and because the cross-scope paths are the ones that should supply `resolve(consumer=...)` its first production caller, which is scope-assembly work by nature. Splitting the reroute across two PRs would move that state twice. Full detail, per call site, in the `slayer/sql/render/value_expr.py` module docstring. diff --git a/slayer/core/errors.py b/slayer/core/errors.py index 9882bd24..adacd55b 100644 --- a/slayer/core/errors.py +++ b/slayer/core/errors.py @@ -420,6 +420,29 @@ def __init__(self, filter_text: str, reason: str) -> None: ) +class RenderContextMissingFacilityError(SlayerError, ValueError): + """A ValueKey render needed a facility its render context did not carry. + + The single ValueKey renderer fails closed rather than degrading quietly: + silent fallbacks are how the generator's five renderer copies drifted apart. + """ + + def __init__( + self, + key_kind: str, + facility: str, + detail: str | None = None, + ) -> None: + self.key_kind = key_kind + self.facility = facility + self.detail = detail + suffix = f" ({detail})" if detail else "" + super().__init__( + f"Rendering a {key_kind} requires the {facility!r} render-context " + f"facility, which was not supplied{suffix}." + ) + + class IdCollisionError(SlayerError, ValueError): """Raised by filename-backed (YAML) storage when saving an entity whose id differs from an existing id only by letter case — such ids diff --git a/slayer/engine/cross_model_planner.py b/slayer/engine/cross_model_planner.py index 15ff9d8e..9baa1080 100644 --- a/slayer/engine/cross_model_planner.py +++ b/slayer/engine/cross_model_planner.py @@ -67,7 +67,7 @@ ) from slayer.core.models import ModelMeasure, SlayerModel from slayer.core.query import ColumnRef, SlayerQuery, TimeDimension -from slayer.core.refs import agg_kwarg_canonical_str, canonical_agg_name +from slayer.sql.naming import canonical_aggregate_alias from slayer.core.scope import ModelScope, StageColumn, StageSchema from slayer.engine.aggregate_input_paths import ( compute_aggregate_input_join_paths, @@ -312,36 +312,18 @@ def _aggregate_alias(*, key: AggregateKey) -> str: suffix matches the rest of the engine (legacy enrichment, search, DBT converter). """ - # ColumnKey -> leaf, ColumnSqlKey (derived agg source) -> column_name, - # StarKey -> "*" (CR / Codex: a derived source must alias as - # ``net_sum``, not ``_sum``). - measure_name = ( - getattr(key.source, "leaf", None) - or getattr(key.source, "column_name", None) - or "*" - ) - # AggregateKey.args / kwargs are normalised tuples of scalars / - # ColumnKey-shaped values; convert to the (List[str], - # Dict[str, Any]) shape ``canonical_agg_name`` expects. DEV-1450 - # stage 7b.13: route through ``agg_kwarg_canonical_str`` so a - # ColumnKey kwarg renders as ``leaf`` (or ``path.leaf`` for joined - # paths) instead of Pydantic-repr noise from naive ``str(v)``. - # The kwarg suffix is preserved here -- the legacy enrichment at - # ``query_engine.py:2160`` drops it, causing two parametric aggs - # with different kwargs to collide on CTE alias (legacy bug). - # The 7b.5 fix added kwarg-aware aliases here as a correctness - # improvement -- ``test_cross_model_planner_wiring.py:: - # test_parameterized_aggregates_get_distinct_cte_aliases`` pins - # this. Parity with legacy for cross-model parametric aggs is - # not achievable on this axis. - args_list = [agg_kwarg_canonical_str(a) for a in key.args] - kwargs_dict = {k: agg_kwarg_canonical_str(v) for k, v in key.kwargs} - return canonical_agg_name( - measure_name=measure_name, - aggregation_name=key.agg, - agg_args=args_list or None, - agg_kwargs=kwargs_dict or None, - ) + # The derivation itself lives in ``slayer.sql.naming`` (P-F, one naming + # authority). This is the ``cte_schema`` profile — the bare canonical + # name with no relation or path prefix, since the alias names a column + # INSIDE the CTE. + # + # The kwarg suffix is preserved -- the deleted legacy enrichment dropped it, + # causing two parametric aggs with different kwargs to collide on CTE alias. + # ``test_cross_model_planner_wiring.py:: + # test_parameterized_aggregates_get_distinct_cte_aliases`` pins this; + # parity with legacy for cross-model parametric aggs is not achievable on + # this axis. + return canonical_aggregate_alias(key, profile="cte_schema") def _make_cte_schema( diff --git a/slayer/engine/planning.py b/slayer/engine/planning.py index 604996d9..dab4294e 100644 --- a/slayer/engine/planning.py +++ b/slayer/engine/planning.py @@ -55,7 +55,7 @@ normalize_scalar, ) from slayer.core.formula import RANK_FAMILY_TRANSFORMS -from slayer.core.refs import agg_kwarg_canonical_str, canonical_agg_name +from slayer.sql.naming import canonical_aggregate_alias from slayer.engine.binding import BoundExpr, BoundFilter from slayer.engine.planned import SlotId, ValueSlot @@ -799,31 +799,12 @@ def _canonical_name(key: ValueKey) -> str: # NOSONAR(S3776) — sequential isin # mirrors the cross-model parametric P10 exception. Without this, # two hidden parametric aggregates collide on a single base-CTE # alias when materialised. - if isinstance(key.source, StarKey): - measure_name = "*" - else: - leaf = getattr(key.source, "leaf", None) or getattr( - key.source, "column_name", None, - ) - if leaf is None: - return f"_agg_{key.agg}" - measure_name = leaf - agg_args = ( - [agg_kwarg_canonical_str(a) for a in key.args] - if key.args - else None - ) - agg_kwargs = ( - {k: agg_kwarg_canonical_str(v) for k, v in key.kwargs} - if key.kwargs - else None - ) - return canonical_agg_name( - measure_name=measure_name, - aggregation_name=key.agg, - agg_args=agg_args, - agg_kwargs=agg_kwargs, - ) + # The derivation lives in ``slayer.sql.naming`` (P-F). The + # ``declared_name`` profile is the bare canonical name, with the + # explicit ``_agg_`` placeholder for a source exposing neither a + # leaf nor a column name (deliberately NOT the star form, so such a + # slot stays distinguishable from a real ``*:count``). + return canonical_aggregate_alias(key, profile="declared_name") if isinstance(key, TransformKey): return f"_{key.op}_inner" if isinstance(key, ArithmeticKey): diff --git a/slayer/engine/stage_planner.py b/slayer/engine/stage_planner.py index 10228061..26de3262 100644 --- a/slayer/engine/stage_planner.py +++ b/slayer/engine/stage_planner.py @@ -58,7 +58,8 @@ SlayerQuery, TimeDimension, ) -from slayer.core.refs import agg_kwarg_canonical_str, canonical_agg_name +from slayer.core.refs import canonical_agg_name +from slayer.sql.naming import canonical_aggregate_alias from slayer.core.time_bounds import strip_frame_bounds from slayer.core.window_duration import parse_window_duration from slayer.core.scope import ModelScope, StageColumn, StageSchema @@ -2196,48 +2197,27 @@ def _canonical_alias_for_formula( alias (``amount_percentile_p=0_5_``), breaking parity. """ if bound is not None and isinstance(bound.value_key, AggregateKey): - key = bound.value_key - if isinstance(key.source, StarKey): - measure_name: Optional[str] = "*" - # Cross-model star (``customers.*:count``) carries its join - # path so the canonical alias keeps the ``customers.`` prefix - # (result key ``orders.customers._count``). - path: Tuple[str, ...] = key.source.path - else: - # ColumnKey exposes ``.leaf``; ColumnSqlKey exposes - # ``.column_name``. Both shapes can appear as aggregate - # sources (the synth adapter rejects ``ColumnSqlKey`` with - # a typed deferral; the planner still needs to derive an - # alias before the generator runs). Mirror ``_canonical_name`` - # at ``planning.py:540-545``. - measure_name = ( - getattr(key.source, "leaf", None) - or getattr(key.source, "column_name", None) - ) - path = getattr(key.source, "path", ()) - if measure_name is not None: - prefix = ".".join(path) + "." if path else "" - # Local aggregates retain the kwarg suffix to match legacy - # ``enrichment.py:349`` (``percentile(p=0.5)`` -> - # ``_p_0_5``). Cross-model aggregates ALSO retain it -- the - # legacy ``query_engine.py:2160`` drops it, causing CTE alias - # collision on two parametric variants, which the 7b.5 fix - # corrected at the planner layer. Result-key shape for - # cross-model parametric aggs therefore diverges from - # legacy in this slice (no parity tests for that combination; - # structural correctness over bit-identical legacy output). - return prefix + canonical_agg_name( - measure_name=measure_name, - aggregation_name=key.agg, - agg_args=[agg_kwarg_canonical_str(a) for a in key.args] or None, - agg_kwargs={ - k: agg_kwarg_canonical_str(v) for k, v in key.kwargs - } or None, - ) - # Fall through to text-based path -- AggregateKey source is - # neither StarKey, ColumnKey, nor ColumnSqlKey (shouldn't be - # reachable in practice; the binder restricts sources to - # those three shapes). + # The derivation lives in ``slayer.sql.naming`` (P-F). The + # ``stage_formula`` profile prefixes the join path RELATIVE to the stage + # (no source relation) and keeps a cross-model star's own path, so + # ``customers.*:count`` aliases as ``customers._count`` and surfaces as + # the result key ``orders.customers._count``. + # + # Both local and cross-model aggregates retain the kwarg suffix + # (``percentile(p=0.5)`` -> ``_p_0_5``). For cross-model parametric + # aggregates that DIVERGES from the deleted legacy pipeline, which + # dropped the suffix and thereby collided two parametric variants onto + # one alias — a ratified divergence, pinned by + # tests/test_dev1744_result_key_contract.py. + alias = canonical_aggregate_alias( + bound.value_key, profile="stage_formula", + ) + if alias is not None: + return alias + # ``None`` means the aggregate's source exposes neither a leaf nor a + # column name — not reachable in practice (the binder restricts sources + # to ColumnKey / ColumnSqlKey / StarKey) — so fall through to the + # text-shape path below. text = formula.strip() if ":" in text and "(" not in text: base, agg = text.rsplit(":", 1) diff --git a/slayer/sql/dialects/tsql.py b/slayer/sql/dialects/tsql.py index 8cf1a854..3ab5687e 100644 --- a/slayer/sql/dialects/tsql.py +++ b/slayer/sql/dialects/tsql.py @@ -34,7 +34,7 @@ from sqlglot import exp from slayer.core.enums import TimeGranularity -from slayer.sql.naming import decode_alias, encode_alias +from slayer.sql.naming import OUTER_WRAP_ALIAS, decode_alias, encode_alias from slayer.sql.dialects.base import SqlDialect, _build_covar_decomposition @@ -325,7 +325,7 @@ def emit_outer_wrap( col.set("table", None) derived = exp.Subquery( this=parsed, - alias=exp.TableAlias(this=exp.to_identifier("_outer")), + alias=exp.TableAlias(this=exp.to_identifier(OUTER_WRAP_ALIAS)), ) outer = exp.Select() for a in public: diff --git a/slayer/sql/generator.py b/slayer/sql/generator.py index f00ca7f5..ed5680f1 100644 --- a/slayer/sql/generator.py +++ b/slayer/sql/generator.py @@ -43,7 +43,11 @@ ) from slayer.sql.dialects import SqlDialect, get_dialect from slayer.sql.naming import ( + FILTERED_ALIAS, + OUTER_WRAP_ALIAS, AliasAllocator, + canonical_aggregate_alias, + cte_name_from_alias, dialect_folds_case, flat_name, maybe_quote_ident, @@ -51,6 +55,8 @@ result_key, result_key_from_alias, ) +from slayer.sql.render.aggregates import window_agg_class +from slayer.sql.render.value_expr import render_scalar_call from slayer.sql.reserved_keywords import prequote_reserved_identifiers from slayer.sql.scope import ScopeFrame from slayer.sql.scope_check import maybe_validate_scopes @@ -1913,7 +1919,7 @@ def _generate_from_planned_impl( # NOSONAR(S3776) — top-level dispatch over c unmaterialised.append(cslot) if unmaterialised: step_num += 1 - step_name = f"step{step_num}" + step_name = cte_allocator.allocate_cte(f"step{step_num}") prev_cte = ctes[-1][0] carry_aliases_sorted = sorted( a for aliases in aliases_by_slot_id.values() for a in aliases @@ -1979,7 +1985,7 @@ def _generate_from_planned_impl( # NOSONAR(S3776) — top-level dispatch over c ) if post_filter_conditions: chain_sql = ( - f"SELECT *\nFROM (\n{chain_sql}\n) AS _filtered" + f"SELECT *\nFROM (\n{chain_sql}\n) AS {FILTERED_ALIAS}" f"\nWHERE {_SQL_AND_JOINER.join(post_filter_conditions)}" ) @@ -3932,8 +3938,8 @@ def _render_aggregate_composite_expr( # NOSONAR(S3776) — sequential isinstanc args.append(exp.Literal.string(str(a))) if key.name == "like": return exp.Like(this=args[0], expression=args[1]), any_agg - return self._finalize_scalar_call( - exp.func(key.name.upper(), *args) + return render_scalar_call( + key.name, args, dialect=self._dialect, ), any_agg if isinstance(key, LiteralKey): v = key.value @@ -4124,7 +4130,12 @@ def _alias_of(sid: str) -> str: exp.LT(this=src_w_time.copy(), expression=bucket_end.copy()), ) - agg_cls = exp.Sum if plan.agg == "sum" else exp.Avg + # Registry lookup, not a silent catch-all: the previous + # ``exp.Sum if agg == "sum" else exp.Avg`` rendered ANY other + # aggregation as AVG. Unreachable through the planner, which gates + # windowed measures to sum/avg — which is exactly why it would have + # stayed wrong. Now it raises. + agg_cls = window_agg_class(plan.agg) agg_expr = _wrap_cast_for_type( agg_cls(this=_src_col("_w_value")), agg_slot.type, ) @@ -4583,7 +4594,16 @@ def _add_local_aux_slots( # SELECT — matches the result-key contract while keeping # legacy parity for the unaliased shape. cm_ctes: List[Tuple[str, str]] = [] - seen_cm: set = set() + # Dedup identity is the STRUCTURAL key (the typed AggregateKey plus the + # source relation), never the sanitised CTE-name string. The canonical + # alias omits the aggregate's column filter, so a filtered and an + # unfiltered aggregate over one column share an alias while needing two + # CTEs; and the name is doubly lossy (path flattening, then + # non-identifier sanitisation), so unrelated aggregates can collide on + # it. Keying on the name silently merged both cases. + cm_cte_name_by_identity: Dict[Any, str] = {} + cm_cte_name_for_plan: Dict[str, str] = {} + cm_allocator = self._gen_allocator or self._new_allocator() canonical_alias_for_plan: Dict[str, str] = {} # join-back pairs are ``(host_base_alias, cte_column_alias)`` — the two # sides need not match (re-rooted CTEs alias dims under the target's @@ -4592,6 +4612,8 @@ def _add_local_aux_slots( # alias for the re-rooted path). joinback_pairs_for_plan: Dict[str, List[Tuple[str, str]]] = {} agg_col_alias_for_plan: Dict[str, str] = {} + joinback_pairs_for_identity: Dict[Any, List[Tuple[str, str]]] = {} + agg_col_alias_for_identity: Dict[Any, str] = {} for plan in planned_query.cross_model_aggregate_plans: agg_slot = slots_by_id.get(plan.aggregate_slot_id) if agg_slot is None or not isinstance(agg_slot.key, AggregateKey): @@ -4604,10 +4626,26 @@ def _add_local_aux_slots( key=agg_slot.key, ) canonical_alias_for_plan[plan.aggregate_slot_id] = canonical_alias - cte_name = _cte_name_from_alias("_cm_", canonical_alias) - if cte_name in seen_cm: + identity = (source_relation, agg_slot.key) + existing = cm_cte_name_by_identity.get(identity) + if existing is not None: + # Same aggregate under another public name: share the one CTE, + # but still record THIS slot's maps. The old code skipped the + # whole iteration, leaving the join-back and column-alias maps + # unwritten for the skipped slot id. + cm_cte_name_for_plan[plan.aggregate_slot_id] = existing + joinback_pairs_for_plan[plan.aggregate_slot_id] = ( + joinback_pairs_for_identity[identity] + ) + agg_col_alias_for_plan[plan.aggregate_slot_id] = ( + agg_col_alias_for_identity[identity] + ) continue - seen_cm.add(cte_name) + cte_name = cte_name_from_alias( + "_cm_", canonical_alias, allocator=cm_allocator, + ) + cm_cte_name_by_identity[identity] = cte_name + cm_cte_name_for_plan[plan.aggregate_slot_id] = cte_name if plan.rerooted_plan is not None: # C1: nested re-rooted PlannedQuery rooted at the target, @@ -4636,6 +4674,8 @@ def _add_local_aux_slots( cm_ctes.append((cte_name, cte_sql)) joinback_pairs_for_plan[plan.aggregate_slot_id] = joinback_pairs agg_col_alias_for_plan[plan.aggregate_slot_id] = agg_col_alias + joinback_pairs_for_identity[identity] = joinback_pairs + agg_col_alias_for_identity[identity] = agg_col_alias # DEV-1714 Stage 10 — per-plan ``_wm_`` windowed range-join CTEs. Each # is host-rooted (``FROM _base LEFT JOIN _src``), grouped at the query @@ -4727,8 +4767,7 @@ def _add_local_aux_slots( if outer_composite_slot_ids: outer_composite_cm_map: Dict[str, Tuple[str, str]] = {} for plan in planned_query.cross_model_aggregate_plans: - canonical_alias = canonical_alias_for_plan[plan.aggregate_slot_id] - cte_name = _cte_name_from_alias("_cm_", canonical_alias) + cte_name = cm_cte_name_for_plan[plan.aggregate_slot_id] agg_col_alias = agg_col_alias_for_plan[plan.aggregate_slot_id] outer_composite_cm_map[plan.aggregate_slot_id] = ( cte_name, agg_col_alias, @@ -4819,9 +4858,8 @@ def _render_outer_composite(cslot) -> str: # alias matches the CTE column name, no ``AS`` remap fires. for plan in planned_query.cross_model_aggregate_plans: agg_slot = slots_by_id[plan.aggregate_slot_id] - canonical_alias = canonical_alias_for_plan[plan.aggregate_slot_id] agg_col_alias = agg_col_alias_for_plan[plan.aggregate_slot_id] - cte_name = _cte_name_from_alias("_cm_", canonical_alias) + cte_name = cm_cte_name_for_plan[plan.aggregate_slot_id] # DEV-1495 bug 2 / DEV-1712: an order-by-only (hidden) cross-model # aggregate never surfaces in the combined projection — its CTE is # still joined below, and the ORDER BY references it CTE-qualified @@ -4892,8 +4930,7 @@ def _render_outer_composite(cslot) -> str: from_clause_str = "FROM _base" joined_cte_names: set = set() for plan in planned_query.cross_model_aggregate_plans: - canonical_alias = canonical_alias_for_plan[plan.aggregate_slot_id] - cte_name = _cte_name_from_alias("_cm_", canonical_alias) + cte_name = cm_cte_name_for_plan[plan.aggregate_slot_id] if cte_name in joined_cte_names: continue joined_cte_names.add(cte_name) @@ -4963,8 +5000,7 @@ def _render_outer_composite(cslot) -> str: # renderer would raise (CodeRabbit thread 2). cross_model_agg_slot_to_cm: Dict[str, Tuple[str, str]] = {} for plan in planned_query.cross_model_aggregate_plans: - canonical_alias = canonical_alias_for_plan[plan.aggregate_slot_id] - cte_name = _cte_name_from_alias("_cm_", canonical_alias) + cte_name = cm_cte_name_for_plan[plan.aggregate_slot_id] agg_col_alias = agg_col_alias_for_plan[plan.aggregate_slot_id] cross_model_agg_slot_to_cm[plan.aggregate_slot_id] = ( cte_name, agg_col_alias, @@ -5060,9 +5096,8 @@ def _render_outer_composite(cslot) -> str: # transform chain) need the CTE-qualified ORDER BY reference. if not (plan.hidden and not planned_query.transform_layers): continue - _canon = canonical_alias_for_plan[plan.aggregate_slot_id] _agg_col = agg_col_alias_for_plan[plan.aggregate_slot_id] - _cte = _cte_name_from_alias("_cm_", _canon) + _cte = cm_cte_name_for_plan[plan.aggregate_slot_id] hidden_cte_order_refs[plan.aggregate_slot_id] = ( f'{_cte}.{self._quote_ident(_agg_col)}' ) @@ -5138,6 +5173,14 @@ def _render_cross_model_transform_chain( ctes: List[Tuple[str, str]] = list(prelude_ctes) + [ ("base", combined_select_sql), ] + # P-F: this chain previously minted ``step`` names with a + # bare f-string and held no allocator at all, so nothing connected its + # names to the ``_cm_*`` CTEs already in ``prelude_ctes`` or to the + # literal ``base``. Take the generation-scoped allocator (the SAME + # instance that minted the ``_cm_`` names, so its used-set already + # covers them) and reserve the inherited literals before allocating. + cte_allocator = self._gen_allocator or self._new_allocator() + cte_allocator.reserve(*(name for name, _ in ctes)) aliases_by_slot_id: Dict[str, List[str]] = { sid: list(a) for sid, a in combined_aliases_by_slot_id.items() } @@ -5172,7 +5215,7 @@ def _render_cross_model_transform_chain( f"{pending_ops!r}.", ) step_num += 1 - step_name = f"step{step_num}" + step_name = cte_allocator.allocate_cte(f"step{step_num}") prev_cte = ctes[-1][0] carry_aliases_sorted = sorted( a for aliases in aliases_by_slot_id.values() for a in aliases @@ -5226,7 +5269,7 @@ def _render_cross_model_transform_chain( unmaterialised.append(cslot) if unmaterialised: step_num += 1 - step_name = f"step{step_num}" + step_name = cte_allocator.allocate_cte(f"step{step_num}") prev_cte = ctes[-1][0] carry_aliases_sorted = sorted( a for aliases in aliases_by_slot_id.values() for a in aliases @@ -5281,7 +5324,7 @@ def _render_cross_model_transform_chain( ) if post_filter_conditions: chain_sql = ( - f"SELECT *\nFROM (\n{chain_sql}\n) AS _filtered" + f"SELECT *\nFROM (\n{chain_sql}\n) AS {FILTERED_ALIAS}" f"\nWHERE {_SQL_AND_JOINER.join(post_filter_conditions)}" ) @@ -5327,40 +5370,22 @@ def _canonical_cross_model_alias( ``canonical_agg_name`` collapses ``*`` to a leading ``_`` (``*:count`` → ``_count``) per the result-key contract. """ - from slayer.core.refs import canonical_agg_name - - path = getattr(key.source, "path", ()) - # Handle ColumnKey (``leaf``), ColumnSqlKey (``column_name`` — derived - # column source, almost universal for filtered-local measures whose - # ``Column.sql`` differs from ``Column.name``), and StarKey (no - # ``leaf`` / ``column_name`` → collapse to ``*``). Mirrors - # ``_aggregate_alias`` in ``cross_model_planner.py``. - measure_name = ( - getattr(key.source, "leaf", None) - or getattr(key.source, "column_name", None) - or "*" - ) - # DEV-1450 stage 7b.13: include kwarg suffix in cross-model - # alias so two distinct parametric aggs (``percentile(p=0.5)`` - # vs ``p=0.95``) produce distinct CTE names and column aliases. - # Legacy enrichment at ``query_engine.py:2160`` drops the - # signature suffix entirely -- a known legacy bug that - # produces ALIAS COLLISION when the same query has multiple - # parametric aggs against the same target.column. The new - # pipeline preserves slot identity here for correctness; - # parity tests for parametric cross-model aggs assert - # structural shape rather than bit-identical SQL. - canonical = canonical_agg_name( - measure_name=measure_name, - aggregation_name=key.agg, - agg_args=[agg_kwarg_canonical_str(a) for a in key.args] or None, - agg_kwargs={ - k: agg_kwarg_canonical_str(v) for k, v in key.kwargs - } or None, - ) - if path: - return f"{source_relation}." + ".".join(path) + f".{canonical}" - return f"{source_relation}.{canonical}" + # The derivation lives in ``slayer.sql.naming`` (P-F, one naming + # authority) — this was one of four drifted copies. The + # ``cross_model_cte`` profile prefixes BOTH the source relation and the + # join path, and collapses a source with neither ``leaf`` nor + # ``column_name`` to the star form. + # + # The kwarg suffix is included so two parametric aggregates + # (``percentile(p=0.5)`` vs ``p=0.95``) get distinct CTE names and + # column aliases. The deleted legacy pipeline dropped it and thereby + # collided them — a ratified divergence, pinned by + # tests/test_dev1744_result_key_contract.py. + alias = canonical_aggregate_alias( + key, profile="cross_model_cte", source_relation=source_relation, + ) + assert alias is not None # the cross_model_cte profile never declines + return alias def _public_aliases_for_cross_model_agg( self, @@ -6367,8 +6392,8 @@ def _render_filter_value_key_in_target_scope( # NOSONAR(S3776) — sequential i ] if value_key.name == "like": return exp.Like(this=rendered_args[0], expression=rendered_args[1]) - return self._finalize_scalar_call( - exp.func(value_key.name.upper(), *rendered_args), + return render_scalar_call( + value_key.name, rendered_args, dialect=self._dialect, ) if isinstance(value_key, BetweenKey): # DEV-1708: a routed ``date_range``-derived BETWEEN over a target @@ -7174,7 +7199,7 @@ def recurse(k) -> exp.Expression: ] if key.name == "like": return exp.Like(this=args[0], expression=args[1]) - return self._finalize_scalar_call(exp.func(key.name.upper(), *args)) + return render_scalar_call(key.name, args, dialect=self._dialect) if isinstance(key, BetweenKey): return exp.Between( @@ -9464,16 +9489,17 @@ def _render_value_key_for_filter( # NOSONAR(S3776) — sequential isinstance di )) if key.name == "like": return exp.Like(this=args[0], expression=args[1]) - # DEV-1576: a 2-arg ROUND needs the Postgres numeric cast, so it - # must be a TYPED node (exp.Round) routed through the target-dialect - # rewrite. Only ROUND is retyped: the string-hygiene functions - # (substr / concat / lower / ...) must emit literally as written - # (DEV-1484), which exp.func would break by transpiling them per - # dialect — so they stay as Anonymous passthrough. - typed = exp.func(key.name.upper(), *args) - if isinstance(typed, exp.Round): - return self._finalize_scalar_call(typed) - return exp.Anonymous(this=key.name.upper(), expressions=args) + # One ScalarCall policy everywhere (B5): typed node, dialect + # rewrite, then the log-alias fix-up. This branch used to return an + # ``exp.Anonymous`` passthrough for everything but ROUND, so a + # filter emitted ``IFNULL(...)`` — which Postgres does not have — + # while the same key emitted ``COALESCE(...)`` from a projection. + # + # The log fix-up is load-bearing: ``exp.func("LOG10", x)`` + # normalises to a generic ``Log(10, x)`` that re-emits as + # ``LOG(10, x)``, wrong for dialects with a native single-arg + # ``LOG10``. Transpiling alone fixes ifnull and breaks log10. + return render_scalar_call(key.name, args, dialect=self._dialect) if isinstance(key, BetweenKey): col_expr = self._render_value_key_for_filter( key=key.column, @@ -9651,16 +9677,17 @@ def _slot_alias_column(slot) -> Optional[exp.Expression]: )) if key.name == "like": return exp.Like(this=args[0], expression=args[1]) - # DEV-1576: a 2-arg ROUND needs the Postgres numeric cast, so it - # must be a TYPED node (exp.Round) routed through the target-dialect - # rewrite. Only ROUND is retyped: the string-hygiene functions - # (substr / concat / lower / ...) must emit literally as written - # (DEV-1484), which exp.func would break by transpiling them per - # dialect — so they stay as Anonymous passthrough. - typed = exp.func(key.name.upper(), *args) - if isinstance(typed, exp.Round): - return self._finalize_scalar_call(typed) - return exp.Anonymous(this=key.name.upper(), expressions=args) + # One ScalarCall policy everywhere (B5): typed node, dialect + # rewrite, then the log-alias fix-up. This branch used to return an + # ``exp.Anonymous`` passthrough for everything but ROUND, so a + # filter emitted ``IFNULL(...)`` — which Postgres does not have — + # while the same key emitted ``COALESCE(...)`` from a projection. + # + # The log fix-up is load-bearing: ``exp.func("LOG10", x)`` + # normalises to a generic ``Log(10, x)`` that re-emits as + # ``LOG(10, x)``, wrong for dialects with a native single-arg + # ``LOG10``. Transpiling alone fixes ifnull and breaks log10. + return render_scalar_call(key.name, args, dialect=self._dialect) if isinstance(key, BetweenKey): col_expr = self._render_filter_for_outer_wrapper( key=key.column, @@ -9906,7 +9933,7 @@ def _build_outer_trim_wrap_sql( exp.Column(this=exp.to_identifier(alias, quoted=True)), ) outer_select = outer_select.from_( - exp.Subquery(this=base_select, alias=exp.to_identifier("_outer")), + exp.Subquery(this=base_select, alias=exp.to_identifier(OUTER_WRAP_ALIAS)), ) # Outer ORDER BY references each order entry's materialised alias diff --git a/slayer/sql/naming.py b/slayer/sql/naming.py index 6b579790..607854b2 100644 --- a/slayer/sql/naming.py +++ b/slayer/sql/naming.py @@ -30,12 +30,18 @@ from __future__ import annotations -from typing import Optional, Tuple +import re +from typing import TYPE_CHECKING, Literal, Optional, Tuple import sqlglot from pydantic import BaseModel, ConfigDict, PrivateAttr from sqlglot import exp +from slayer.core.refs import agg_kwarg_canonical_str, canonical_agg_name + +if TYPE_CHECKING: # pragma: no cover — typing only, keeps the import leaf clean + from slayer.core.keys import AggregateKey + # --------------------------------------------------------------------------- # Dialect case-folding policy (DEV-1726). # @@ -215,6 +221,181 @@ def flat_name(dotted: str, *, strip_relation: Optional[str] = None) -> str: return remainder.replace(".", "__") +# --------------------------------------------------------------------------- +# Structural alias constants (P-F). +# +# These name derived tables and wrapper subqueries rather than being minted per +# query, and each was previously written as a bare literal in more than one +# module — ``_outer`` in BOTH ``generator.py`` (the outer-wrap subquery) and +# ``dialects/tsql.py`` (the ORDER-BY detach rewrite), coupled by convention +# only. Hoisting them here gives the naming module a single owner. +# +# RATIFIED CARVE-OUT: the T-SQL and stage-wrapper sites take these as +# CONSTANTS, not allocator-minted names. The T-SQL rewrite is a post-generation +# AST pass with no allocator in reach, and PR 4 rebuilds the outer-wrap +# machinery wholesale; both aliases scope a derived table the same pass creates, +# so a collision would have to come from inside that one subquery. This is a +# named exception to P-F, recorded rather than silently omitted. +# --------------------------------------------------------------------------- + +OUTER_WRAP_ALIAS = "_outer" +STAGE_INNER_ALIAS = "_stage_inner" +FILTERED_ALIAS = "_filtered" + + +# --------------------------------------------------------------------------- +# CTE-name minting. +# --------------------------------------------------------------------------- + +# Everything outside the SQL identifier alphabet collapses to ``_``. This is +# LOSSY on purpose (a CTE name must be a bare identifier) — which is exactly +# why the result must go through an allocator rather than being trusted as an +# identity. +_NON_IDENT_CHAR_RE = re.compile(r"[^a-zA-Z0-9_]") + + +def cte_name_from_alias( + prefix: str, alias: str, *, allocator: "AliasAllocator", +) -> str: + """Mint a collision-safe CTE name for ``alias`` under ``prefix``. + + The alias is flattened (:func:`flat_name` maps ``.`` to ``__``) and then + sanitised to the identifier alphabet. BOTH steps are lossy and neither is + injective: ``customers.revenue`` and ``customers__revenue`` flatten to the + same string, and ``rev-a`` / ``rev_a`` sanitise to the same string. + + Hence the required ``allocator``: the sanitised string is only a PREFERRED + name, walked to ``…_2`` when taken (case-folded on folding dialects), so two + calls never hand back one name. Previously it doubled as a CTE name AND a + plan identity key, so aggregates that sanitised alike either collided in the + ``WITH`` or silently collapsed into one plan. + + Dedup is the CALLER's decision, made on structural identity — never on this + string. + """ + sanitized = _NON_IDENT_CHAR_RE.sub("_", flat_name(alias)) + return allocator.allocate_cte(prefix + sanitized) + + +# --------------------------------------------------------------------------- +# Canonical aggregate alias — the four-copy consolidation. +# --------------------------------------------------------------------------- + +# The four historical derivations, as PROFILES. They differ on four axes — +# whether a source relation is prefixed, whether the join path is prefixed, +# whether a StarKey keeps its own path, and what happens when the source has +# neither a ``leaf`` nor a ``column_name``. Naming the combinations makes the +# impossible ones unrepresentable, which four free-standing boolean flags +# would not. +AggAliasProfile = Literal[ + # generator._canonical_cross_model_alias — the ``_cm_`` CTE + projection + # alias. Prefixes BOTH the source relation and the join path; an + # unrecognised source collapses to the star form. + "cross_model_cte", + # cross_model_planner._aggregate_alias — the aggregate's output column + # inside its CTE. Bare canonical name; no prefix of any kind. + "cte_schema", + # planning._canonical_name — a hidden slot's declared name. Bare, but an + # unrecognised source gets an explicit ``_agg_`` placeholder rather + # than being mistaken for a star. + "declared_name", + # stage_planner._canonical_alias_for_formula — the public alias for a + # measure formula. Prefixes the join path RELATIVE to the stage (no source + # relation), is the only profile that keeps a StarKey's own path, and + # DECLINES (returns None) on an unrecognised source so its caller can fall + # through to formula-text sanitisation. + "stage_formula", +] + +_PROFILES_WITHOUT_RELATION = ("cte_schema", "declared_name", "stage_formula") + + +def canonical_aggregate_alias( + key: "AggregateKey", + *, + profile: AggAliasProfile, + source_relation: Optional[str] = None, +) -> Optional[str]: + """The single canonical-aggregate-alias derivation. + + Replaces four copies that had drifted apart. ``profile`` selects which + caller's exact contract to apply; see :data:`AggAliasProfile`. + + Returns ``None`` only for ``stage_formula`` on a source that exposes + neither ``leaf`` nor ``column_name`` — that profile's documented "decline + and let the caller sanitise the formula text" path. + """ + from slayer.core.keys import StarKey + + if profile == "cross_model_cte": + if source_relation is None: + raise ValueError( + "canonical_aggregate_alias(profile='cross_model_cte') requires " + "source_relation — the alias is anchored at the query root.", + ) + elif profile in _PROFILES_WITHOUT_RELATION: + if source_relation is not None: + raise ValueError( + f"canonical_aggregate_alias(profile={profile!r}) does not take " + f"source_relation: that profile emits no relation prefix.", + ) + else: + raise ValueError( + f"Unknown canonical-aggregate-alias profile {profile!r}; " + f"expected one of {('cross_model_cte', *_PROFILES_WITHOUT_RELATION)}.", + ) + + is_star = isinstance(key.source, StarKey) + leaf = getattr(key.source, "leaf", None) or getattr( + key.source, "column_name", None, + ) + + # --- measure name, per profile's treatment of an unrecognised source --- + if profile in ("cross_model_cte", "cte_schema"): + # Any source without a leaf collapses to the star form. + measure_name: Optional[str] = leaf or "*" + elif profile == "declared_name": + if is_star: + measure_name = "*" + elif leaf is None: + # Explicit placeholder — deliberately NOT the star form, so a + # hidden slot over an unrecognised source is distinguishable. + return f"_agg_{key.agg}" + else: + measure_name = leaf + else: # stage_formula + measure_name = "*" if is_star else leaf + if measure_name is None: + return None + + canonical = canonical_agg_name( + measure_name=measure_name, + aggregation_name=key.agg, + agg_args=[agg_kwarg_canonical_str(a) for a in key.args] or None, + agg_kwargs={ + k: agg_kwarg_canonical_str(v) for k, v in key.kwargs + } or None, + ) + + # --- prefix, per profile --- + if profile in ("cte_schema", "declared_name"): + return canonical + + # Every path-bearing source kind — ColumnKey, ColumnSqlKey, and StarKey + # alike — carries its join path here, so ``customers.*:count`` keeps the + # ``customers`` hop in both prefixing profiles. + path: Tuple[str, ...] = tuple(getattr(key.source, "path", ())) + + if profile == "stage_formula": + # Path RELATIVE to the stage — no source relation. + return (".".join(path) + "." if path else "") + canonical + + assert source_relation is not None # guaranteed by the validation above + return result_key( + source_relation=source_relation, path=path, leaf=canonical, + ) + + # --------------------------------------------------------------------------- # BigQuery / T-SQL dotted-alias mangling bijection (DEV-1571). # diff --git a/slayer/sql/render/__init__.py b/slayer/sql/render/__init__.py new file mode 100644 index 00000000..9b8a9644 --- /dev/null +++ b/slayer/sql/render/__init__.py @@ -0,0 +1,11 @@ +"""Render package — the incremental split of ``generator.py``. + +``generator.py`` is ~10k lines because every render path grew where it was +first needed. Each consolidation PR moves one coherent responsibility here. + +* :mod:`.value_expr` — one ``ValueKey`` → sqlglot-AST renderer, so a given key + renders identically wherever it appears. +* :mod:`.aggregates` — one registry for aggregation rendering. + +Nothing here imports ``generator``; the dependency runs one way. +""" diff --git a/slayer/sql/render/aggregates.py b/slayer/sql/render/aggregates.py new file mode 100644 index 00000000..c3495640 --- /dev/null +++ b/slayer/sql/render/aggregates.py @@ -0,0 +1,121 @@ +"""One registry table for aggregation rendering. + +``_build_agg`` reached its builders five different ways, so adding an +aggregation meant knowing which one to touch. Here each is one +:class:`AggEntry`: ``dispatch`` names the mechanism that renders it (the +generator still owns the builders, which need model columns and dialect hooks), +and ``window_class`` replaces a silent ``else AVG`` catch-all. +""" + +from __future__ import annotations + +from typing import Dict, Optional, Type + +from pydantic import BaseModel, ConfigDict +from sqlglot import exp + +from slayer.core.enums import BUILTIN_AGGREGATIONS + +# Which mechanism renders an aggregation. Retained as data so the generator's +# dispatch is a table lookup rather than five stacked conditionals. +DISPATCH_SIMPLE = "simple" # direct sqlglot node: COUNT/SUM/AVG/MIN/MAX +DISPATCH_RANKED = "ranked" # first/last — needs the ranked-subquery state +DISPATCH_STAT = "stat" # stddev/var/corr/covar — dialect UDF split +DISPATCH_DIALECT_HOOK = "dialect_hook" # percentile/median/approx-distinct +DISPATCH_DISTINCT = "distinct" # COUNT(DISTINCT ...) +DISPATCH_FORMULA = "formula" # {value}/{param} template substitution + + +class AggEntry(BaseModel): + """How one aggregation renders.""" + + model_config = ConfigDict(frozen=True, arbitrary_types_allowed=True) + + name: str + dispatch: str + # The sqlglot class for the simple path, when there is one. + node_class: Optional[Type[exp.Expression]] = None + # Set only for aggregations that can carry their own window frame. + window_class: Optional[Type[exp.Expression]] = None + + @property + def windowable(self) -> bool: + return self.window_class is not None + + +def _entry(name: str, dispatch: str, **kw) -> AggEntry: + return AggEntry(name=name, dispatch=dispatch, **kw) + + +AGG_REGISTRY: Dict[str, AggEntry] = { + e.name: e + for e in ( + # Only sum and avg carry a window frame — the same pair the stage + # planner gates windowed measures on. + _entry("sum", DISPATCH_SIMPLE, node_class=exp.Sum, window_class=exp.Sum), + _entry("avg", DISPATCH_SIMPLE, node_class=exp.Avg, window_class=exp.Avg), + _entry("count", DISPATCH_SIMPLE, node_class=exp.Count), + _entry("min", DISPATCH_SIMPLE, node_class=exp.Min), + _entry("max", DISPATCH_SIMPLE, node_class=exp.Max), + _entry("count_distinct", DISPATCH_DISTINCT, node_class=exp.Count), + _entry("count_distinct_approx", DISPATCH_DIALECT_HOOK), + _entry("first", DISPATCH_RANKED), + _entry("last", DISPATCH_RANKED), + _entry("median", DISPATCH_DIALECT_HOOK), + _entry("percentile", DISPATCH_DIALECT_HOOK), + _entry("weighted_avg", DISPATCH_FORMULA), + _entry("stddev_samp", DISPATCH_STAT), + _entry("stddev_pop", DISPATCH_STAT), + _entry("var_samp", DISPATCH_STAT), + _entry("var_pop", DISPATCH_STAT), + _entry("corr", DISPATCH_STAT), + _entry("covar_samp", DISPATCH_STAT), + _entry("covar_pop", DISPATCH_STAT), + ) +} + +# Every built-in must be in the table, or a lookup would fall through to the +# custom-formula path and render a built-in as if it were user-defined. +_missing = BUILTIN_AGGREGATIONS - set(AGG_REGISTRY) +if _missing: # pragma: no cover — import-time invariant + raise RuntimeError( + f"Aggregation registry is missing built-ins: {sorted(_missing)}", + ) + + +def resolve_agg_entry(name: str) -> AggEntry: + """Return the registry entry for a BUILT-IN aggregation. + + Raises ``ValueError`` for anything else. Custom model-level aggregations + are deliberately not registered: they carry their own ``formula`` and take + the template path, so callers check :func:`is_builtin_agg` first. + """ + entry = AGG_REGISTRY.get(name) + if entry is None: + raise ValueError( + f"Unknown aggregation {name!r}. Built-ins: " + f"{sorted(AGG_REGISTRY)}; anything else must be defined as a " + f"model-level aggregation with a formula.", + ) + return entry + + +def is_builtin_agg(name: str) -> bool: + return name in AGG_REGISTRY + + +def window_agg_class(name: str) -> Type[exp.Expression]: + """The sqlglot class for a WINDOWED aggregate. + + Raises ``ValueError`` when the aggregation cannot carry a window frame. + The windowed render path previously read ``exp.Sum if agg == "sum" else + exp.Avg``, silently rendering every other aggregation as AVG. + """ + entry = resolve_agg_entry(name) + if entry.window_class is None: + raise ValueError( + f"Aggregation {name!r} cannot be windowed; only " + f"{sorted(n for n, e in AGG_REGISTRY.items() if e.windowable)} " + f"carry their own window frame.", + ) + return entry.window_class diff --git a/slayer/sql/render/value_expr.py b/slayer/sql/render/value_expr.py new file mode 100644 index 00000000..889f7c7c --- /dev/null +++ b/slayer/sql/render/value_expr.py @@ -0,0 +1,349 @@ +"""The single ``ValueKey`` → sqlglot-AST renderer (P-G). + +The generator grew five renderers that drifted apart: the same +``ScalarCallKey`` emitted ``IFNULL(...)`` from a filter (invalid on Postgres) +and ``COALESCE(...)`` from a projection. One function now renders the whole +closed union, parameterised by :class:`RenderContext` rather than by call site. + +Two rules keep it from becoming a sixth copy: column-like leaves anchor through +``ScopeFrame.resolve`` (which also registers crossed joins and handles +consumer-scope materialisation), and a missing facility raises +:class:`RenderContextMissingFacilityError` instead of degrading quietly. + +Migration status +---------------- +The API here is complete and directly tested, but the generator's own render +paths do NOT yet route through :func:`render_value_key`. Only the ScalarCall +POLICY is shared today: all six paths call :func:`render_scalar_call`, so that +construct genuinely renders once. Everything else still runs the generator's +own per-path branches. + +Deferred to the scope-assembly PR, together with the cross-scope migration, +because the two are the same piece of work. Finishing it needs: + +* **Filter paths** (``_render_value_key_for_filter``, ``:9119`` host WHERE / + HAVING and ``:7468`` the shifted-CTE WHERE) — ``FilterFacilities`` must carry + the local-aggregate HAVING branch, which reads ``slot_by_key`` to find a + materialised slot, the first/last ranked state, and the filter-side CAST + policy applied per column type. Rendering an aggregate leaf inline (rather + than by output alias) is what makes HAVING work on backends that reject + SELECT aliases there, so that branch cannot simply be dropped. +* **Composite paths** (``_render_aggregate_composite_expr``, ``:2851`` the base + SELECT and ``:3741`` the first/last base SELECT) — ``CompositeFacilities`` + already declares the maps these need (rn-suffix, filtered-rank and + match-flag, composite alias-by-key, resolved agg kwargs, value alias-by-sql); + they are threaded through but not yet consumed, because the aggregate leaf + goes to ``agg_builder``. Wiring ``agg_builder`` to the generator's + ``_build_agg`` is the intended seam and keeps emission byte-identical. +* **Cross-scope paths** (``_render_filter_value_key_in_target_scope``, + ``_render_value_key_against_aliases``, ``_render_filter_for_outer_wrapper``) + — these consume another scope's projected columns, so they are the ones that + should set ``consumer=`` and thereby give ``ScopeFrame.resolve``'s + materialisation branch its first production caller. An API nobody calls does + not establish the projection-boundary principle. + +Doing the reroute properly means moving that state onto the context and +re-verifying emission across the dialect matrix; doing it hastily would risk +silent SQL changes across the whole suite for no principle gained beyond what +the shared ScalarCall policy already delivers. +""" + +from __future__ import annotations + +from decimal import Decimal +from typing import Any, Callable, Dict, List, Optional, Tuple + +from pydantic import BaseModel, ConfigDict, Field +from sqlglot import exp + +from slayer.core.errors import RenderContextMissingFacilityError +from slayer.core.keys import ( + AggregateKey, + ArithmeticKey, + BetweenKey, + ColumnKey, + ColumnSqlKey, + InKey, + LiteralKey, + Phase, + ScalarCallKey, + StarKey, + TimeTruncKey, + TransformKey, + ValueKey, +) +from slayer.sql.dialects.base import SqlDialect +from slayer.sql.render.aggregates import ( + DISPATCH_DISTINCT, + DISPATCH_SIMPLE, + is_builtin_agg, + resolve_agg_entry, +) +from slayer.sql.scope import ScopeFrame + +# Arithmetic / comparison / boolean operators, as sqlglot node classes. One +# table instead of the three hand-rolled composers this replaces. +_BINARY_OPS: Dict[str, Any] = { + "+": exp.Add, "-": exp.Sub, "*": exp.Mul, "/": exp.Div, + "%": exp.Mod, + "=": exp.EQ, "==": exp.EQ, "!=": exp.NEQ, "<>": exp.NEQ, + "<": exp.LT, "<=": exp.LTE, ">": exp.GT, ">=": exp.GTE, +} + +_GRANULARITY_TO_SQL: Dict[str, str] = { + "second": "SECOND", "minute": "MINUTE", "hour": "HOUR", "day": "DAY", + "week": "WEEK", "month": "MONTH", "quarter": "QUARTER", "year": "YEAR", +} + + +class FilterFacilities(BaseModel): + """What WHERE / HAVING rendering needs beyond the scope.""" + + model_config = ConfigDict(arbitrary_types_allowed=True) + + slot_by_key: Dict[Any, Any] = Field(default_factory=dict) + aliases_by_slot_id: Dict[str, List[str]] = Field(default_factory=dict) + first_last_state: Optional[Any] = None + + +class CompositeFacilities(BaseModel): + """What AGGREGATE-phase composite rendering needs beyond the scope. + + ``agg_builder`` is the seam to the generator's ``_build_agg``: building a + real aggregate needs model columns, resolved kwargs and dialect hooks that + only the generator holds. When it is absent the renderer still handles the + simple built-ins directly, and raises for the rest rather than guessing. + """ + + model_config = ConfigDict(arbitrary_types_allowed=True) + + agg_builder: Optional[Callable[[AggregateKey], exp.Expression]] = None + rn_suffix_map: Optional[Dict[str, str]] = None + default_time_col: Optional[str] = None + filtered_rn_map: Optional[Dict[str, str]] = None + filtered_match_map: Optional[Dict[str, str]] = None + composite_alias_by_key: Optional[Dict[Any, str]] = None + resolved_agg_kwargs: Optional[Dict[Any, Any]] = None + value_alias_by_sql: Optional[Dict[str, str]] = None + + +class AliasFacilities(BaseModel): + """What POST-phase rendering needs: the aliases an earlier scope projected.""" + + model_config = ConfigDict(arbitrary_types_allowed=True) + + slot_id_by_key: Dict[Any, str] = Field(default_factory=dict) + available_alias_by_slot_id: Dict[str, str] = Field(default_factory=dict) + + +class RenderContext(BaseModel): + """Everything a render needs that is not the key itself. + + ``consumer`` is the projection-boundary seam: when set, column-like leaves + are materialised in ``scope`` and returned to the consumer as bare aliases. + """ + + model_config = ConfigDict(arbitrary_types_allowed=True) + + scope: ScopeFrame + dialect: SqlDialect + consumer: Optional[ScopeFrame] = None + filters: Optional[FilterFacilities] = None + composites: Optional[CompositeFacilities] = None + aliases: Optional[AliasFacilities] = None + + +def _require(ctx: RenderContext, facility: str, key: Any) -> Any: + got = getattr(ctx, facility, None) + if got is None: + raise RenderContextMissingFacilityError( + key_kind=type(key).__name__, facility=facility, + ) + return got + + +def _literal(value: Any) -> exp.Expression: + if value is None: + return exp.Null() + if isinstance(value, bool): + return exp.true() if value else exp.false() + if isinstance(value, Decimal): + return exp.Literal.number(str(value)) + if isinstance(value, (int, float)): + return exp.Literal.number(str(value)) + return exp.Literal.string(str(value)) + + +def render_scalar_call( + name: str, args: List[exp.Expression], *, dialect: SqlDialect, +) -> exp.Expression: + """The one ScalarCall policy: typed node, dialect rewrite, log-alias fix-up. + + Every render path calls this, so one call cannot render two ways. + + The log fix-up is load-bearing: ``exp.func("LOG10", x)`` normalises to a + generic ``Log(10, x)`` re-emitting as ``LOG(10, x)``, wrong on dialects with + a native single-arg ``LOG10``. Transpiling alone fixes ifnull and breaks + log10. ``like`` is the allowlist's only operator rather than function. + """ + if name == "like": + return exp.Like(this=args[0], expression=args[1]) + node = dialect.rewrite_target_ast(exp.func(name.upper(), *args)) + return _rewrite_log_alias(node, dialect=dialect) + + +def _rewrite_log_alias(node: exp.Expression, *, dialect: SqlDialect): + if not isinstance(node, exp.Log): + return node + base = node.args.get("this") + arg = node.args.get("expression") + if arg is None or not isinstance(base, exp.Literal) or base.is_string: + return node + try: + base_val = float(base.this) + except (TypeError, ValueError): + return node + for candidate in (10, 2): + if base_val == candidate and dialect.should_use_native_log(candidate): + return exp.Anonymous( + this=f"log{candidate}", expressions=[arg.copy()], + ) + return node + + +def _render_aggregate(key: AggregateKey, ctx: RenderContext) -> exp.Expression: + facilities = _require(ctx, "composites", key) + if facilities.agg_builder is not None: + return facilities.agg_builder(key) + + # No builder: handle the aggregations that need nothing beyond the source + # expression, and refuse the rest rather than emitting something plausible. + if not is_builtin_agg(key.agg): + raise RenderContextMissingFacilityError( + key_kind=type(key).__name__, + facility="composites.agg_builder", + detail=f"custom aggregation {key.agg!r} needs the generator's builder", + ) + entry = resolve_agg_entry(key.agg) + if entry.dispatch not in (DISPATCH_SIMPLE, DISPATCH_DISTINCT): + raise RenderContextMissingFacilityError( + key_kind=type(key).__name__, + facility="composites.agg_builder", + detail=( + f"aggregation {key.agg!r} renders via the {entry.dispatch!r} " + f"mechanism, which needs the generator's builder" + ), + ) + if isinstance(key.source, StarKey): + inner: exp.Expression = exp.Star() + else: + inner = ctx.scope.resolve(key.source, consumer=ctx.consumer) + assert entry.node_class is not None + if entry.dispatch == DISPATCH_DISTINCT: + return entry.node_class(this=exp.Distinct(expressions=[inner])) + return entry.node_class(this=inner) + + +def render_value_key( # NOSONAR(S3776) — sequential dispatch over the closed ValueKey union; each branch IS that type's render contract, and splitting them is exactly the fragmentation this module removes. + key: ValueKey, ctx: RenderContext, +) -> exp.Expression: + """Render ``key`` to sqlglot AST in ``ctx``.""" + if isinstance(key, (ColumnKey, ColumnSqlKey)): + return ctx.scope.resolve(key, consumer=ctx.consumer) + + if isinstance(key, StarKey): + return exp.Star() + + if isinstance(key, LiteralKey): + return _literal(key.value) + + if isinstance(key, TimeTruncKey): + column = ctx.scope.resolve(key.column, consumer=ctx.consumer) + unit = _GRANULARITY_TO_SQL.get( + key.granularity.lower(), key.granularity.upper(), + ) + return ctx.dialect.rewrite_target_ast( + exp.func("DATE_TRUNC", exp.Literal.string(unit), column), + ) + + if isinstance(key, ArithmeticKey): + operands = [render_value_key(o, ctx) for o in key.operands] + op = key.op.lower() + if op == "and": + return exp.and_(*operands) + if op == "or": + return exp.or_(*operands) + node_cls = _BINARY_OPS.get(key.op) + if node_cls is None: + raise NotImplementedError( + f"Unsupported arithmetic operator {key.op!r}.", + ) + result = operands[0] + for operand in operands[1:]: + result = node_cls(this=result, expression=operand) + return result + + if isinstance(key, ScalarCallKey): + args = [ + render_value_key(a, ctx) + if isinstance(a, _VALUE_KEY_TYPES) + else _literal(a) + for a in key.args + ] + return render_scalar_call(key.name, args, dialect=ctx.dialect) + + if isinstance(key, BetweenKey): + return exp.Between( + this=render_value_key(key.column, ctx), + low=render_value_key(key.low, ctx), + high=render_value_key(key.high, ctx), + ) + + if isinstance(key, InKey): + node = exp.In( + this=render_value_key(key.column, ctx), + expressions=[_literal(v.value) for v in key.values], + ) + return exp.Not(this=node) if key.negated else node + + if isinstance(key, AggregateKey): + return _render_aggregate(key, ctx) + + if isinstance(key, TransformKey): + # POST-phase: the value was materialised by an earlier scope, so it is + # referenced by alias rather than rebuilt. + facilities = _require(ctx, "aliases", key) + slot_id = facilities.slot_id_by_key.get(key) + alias = ( + facilities.available_alias_by_slot_id.get(slot_id) + if slot_id is not None + else None + ) + if alias is None: + raise RenderContextMissingFacilityError( + key_kind=type(key).__name__, + facility="aliases", + detail=f"transform {key.op!r} is not materialised as a slot", + ) + return exp.column(alias, quoted=True) + + raise NotImplementedError( + f"Unsupported ValueKey type {type(key).__name__}: {key!r}", + ) + + +_VALUE_KEY_TYPES: Tuple[type, ...] = ( + ColumnKey, ColumnSqlKey, TimeTruncKey, StarKey, LiteralKey, AggregateKey, + TransformKey, ArithmeticKey, ScalarCallKey, BetweenKey, InKey, +) + + +def contains_aggregate(key: ValueKey) -> bool: + """Whether ``key`` contains an aggregate, structurally. + + Replaces the ``(expr, any_agg)`` tuple the old renderers threaded by hand. + ``phase`` already propagates as the max over operands and arguments, so + nested cases (a scalar call over an arithmetic over an aggregate) are + covered without a second traversal. + """ + return getattr(key, "phase", Phase.ROW) >= Phase.AGGREGATE diff --git a/slayer/sql/scope.py b/slayer/sql/scope.py index c9e93db8..3d231e8a 100644 --- a/slayer/sql/scope.py +++ b/slayer/sql/scope.py @@ -27,6 +27,7 @@ from pydantic import BaseModel, ConfigDict, Field from sqlglot import exp +from slayer.core.errors import UnknownReferenceError from slayer.core.keys import ColumnKey, ColumnSqlKey from slayer.core.models import SlayerModel from slayer.engine.column_expansion import ( @@ -185,9 +186,34 @@ def _anchor(self, ref: Ref) -> exp.Expression: ) def _model_for(self, name: str) -> SlayerModel: + """Resolve a model name against the scope root, then the bundle. + + Unresolvable RAISES: the previous ``or self.root_model`` fallback + expanded the ROOT model's derived SQL instead, turning a wiring bug into + a wrong answer rather than a failure. + """ if name == self.root_model.name: return self.root_model - return self.bundle.get_referenced_model(name) or self.root_model + model = self.bundle.get_referenced_model(name) + if model is None: + known = sorted( + {self.root_model.name} + | {m.name for m in self.bundle.referenced_models}, + ) + raise UnknownReferenceError( + name=name, + scope_kind="ScopeFrame", + scope_summary=( + f"scope rooted at model {self.root_model.name!r}; " + f"models resolvable here: {known}" + ), + suggestion=( + "A ColumnSqlKey must name the model that owns the derived " + "column, and that model must be in the query's resolved " + "source bundle (reachable from the source model via joins)." + ), + ) + return model def _parse(self, sql: str) -> exp.Expression: return sqlglot.parse_one(sql, dialect=self.dialect.sqlglot_name) diff --git a/slayer/sql/stage_wrapper.py b/slayer/sql/stage_wrapper.py index 23f6398f..46c4e755 100644 --- a/slayer/sql/stage_wrapper.py +++ b/slayer/sql/stage_wrapper.py @@ -20,7 +20,7 @@ from sqlglot import exp from slayer.sql.dialects import get_dialect -from slayer.sql.naming import flat_name +from slayer.sql.naming import STAGE_INNER_ALIAS, flat_name def build_flat_rename_wrapper( @@ -48,7 +48,7 @@ def build_flat_rename_wrapper( over-projection, ...) raises ``ValueError`` immediately rather than masking the issue as a downstream bind miss. """ - inner_alias = "_stage_inner" + inner_alias = STAGE_INNER_ALIAS body = sqlglot.parse_one(stage_sql, dialect=dialect) select = exp.Select() produced: List[str] = [] diff --git a/tests/dialects/test_multi_dialect_generation.py b/tests/dialects/test_multi_dialect_generation.py index cc1b4b10..20b17215 100644 --- a/tests/dialects/test_multi_dialect_generation.py +++ b/tests/dialects/test_multi_dialect_generation.py @@ -744,7 +744,7 @@ async def test_instr_translates_per_dialect( ("postgres", "SUBSTRING(orders.status FROM 1 FOR 5)"), ("mysql", "SUBSTRING(orders.status, 1, 5)"), ("duckdb", "SUBSTRING(orders.status, 1, 5)"), - ("clickhouse", "SUBSTR(orders.status, 1, 5)"), + ("clickhouse", "SUBSTRING(orders.status, 1, 5)"), ], ) async def test_substr_translates_per_dialect( @@ -767,9 +767,9 @@ async def test_substr_translates_per_dialect( [ # SQLite normalises CONCAT(...) → a || b at emit time. ("sqlite", "orders.status || orders.status"), - ("postgres", "CONCAT(orders.status, orders.status)"), + ("postgres", "orders.status || orders.status"), ("mysql", "CONCAT(orders.status, orders.status)"), - ("duckdb", "CONCAT(orders.status, orders.status)"), + ("duckdb", "orders.status || orders.status"), ("clickhouse", "CONCAT(orders.status, orders.status)"), ], ) diff --git a/tests/test_dev1744_naming_allocator.py b/tests/test_dev1744_naming_allocator.py new file mode 100644 index 00000000..05df1ac2 --- /dev/null +++ b/tests/test_dev1744_naming_allocator.py @@ -0,0 +1,1046 @@ +"""P-F "one naming authority": CTE-name allocation + the alias consolidation. + +Three things are pinned here. + +**Every CTE name through the collision-aware allocator.** ``_wm_`` was retrofitted +onto the allocator; ``_cm_`` never was. Two consequences, both reachable from +the public query API today: + +* two cross-model measures whose canonical aliases differ ONLY in case emit two + ``_cm_`` CTE names that fold together on every case-folding dialect (which is + all of them but ClickHouse) — the collision belt raises, and without the belt + the backend would see a duplicate ``WITH`` name; +* ``_cte_name_from_alias`` stacks two lossy steps (``flat_name``, which is + documented non-injective, then ``re.sub`` over non-identifier characters) and + the result is used as the PLAN IDENTITY key in ``seen_cm``. Two DISTINCT + aggregate slots that sanitise to the same name make the second one skip the + loop body, leaving ``agg_col_alias_for_plan`` / ``joinback_pairs_for_plan`` + unfilled — a ``KeyError`` at the four unconditional downstream subscripts. + +The fix is structural identity (the full typed ``AggregateKey`` plus its source +relation) for dedup, and the allocator for names. Canonical aliases are NOT a +safe identity: ``canonical_agg_name`` omits ``column_filter_key``, so a filtered +and an unfiltered aggregate over the same column share one canonical alias while +needing two different CTEs. + +**The consolidation.** Four copies of canonical-aggregate-alias derivation +(``generator._canonical_cross_model_alias``, ``cross_model_planner._aggregate_alias``, +``planning._canonical_name``, ``stage_planner._canonical_alias_for_formula``) that +have DRIFTED on four axes become one ``naming.canonical_aggregate_alias`` +parameterised by a named profile. The expected-value tables below are frozen +from the four legacy bodies, so the consolidation is provably behavior-preserving. + +**The naming constants.** ``_outer`` / ``_stage_inner`` / ``_filtered`` are minted +in more than one module, coupled by convention only (``generator.py`` and +``dialects/tsql.py`` each write the ``_outer`` literal independently). The +literals move into ``naming.py`` so one module owns them. + +Test style: the collision tests are SELF-CALIBRATING. Rather than hardcoding an +expected aggregate value, each runs the query once per measure in isolation +(which works today), then once with both measures together, and asserts the +combined run reproduces both isolated values. That is exactly SLayer's core +invariant — "adding a measure must never change another measure's value" — and +it cannot pass by agreeing with a wrong hardcoded number. +""" + +from __future__ import annotations + +import inspect +import os +import re +import sqlite3 +import tempfile +from decimal import Decimal +from typing import AsyncIterator, List + +import pytest + +from slayer.core.enums import DataType +from slayer.core.keys import ( + AggregateKey, + ColumnKey, + ColumnSqlKey, + StarKey, + TimeTruncKey, +) +from slayer.core.models import ( + Column, + DatasourceConfig, + ModelJoin, + ModelMeasure, + SlayerModel, +) +from slayer.core.query import ColumnRef, SlayerQuery +from slayer.engine.query_engine import SlayerQueryEngine +from slayer.sql import naming +from slayer.sql.naming import AliasAllocator +from slayer.storage.yaml_storage import YAMLStorage + + +# =========================================================================== +# Fixtures — a seeded SQLite store whose model deliberately contains the +# name shapes that collide under the current sanitisation. +# =========================================================================== + + +async def _build_engine(*, dialect: str = "sqlite") -> SlayerQueryEngine: + """orders -> customers, seeded, with the collision-bait columns. + + On ``customers``: + * ``Rev`` and ``rev`` — a case-only pair (distinct physical columns + ``revx`` / ``revy``, so a shadowing bug shows up as equal values); + * ``revenue`` — the ordinary cross-model measure source. + + On ``orders``: + * ``customers__revenue`` — a ``__``-in-name column (a ratified + keep-list carve-out) carrying a join-crossing ``Column.filter``, which + makes it a filtered-local ISOLATED aggregate — i.e. it also + renders as a ``_cm_`` CTE. Its canonical alias + ``orders.customers__revenue_sum`` sanitises to exactly the same CTE + name as the cross-model ``orders.customers.revenue_sum``. + """ + d = tempfile.mkdtemp() + db_path = os.path.join(d, "b4.db") + con = sqlite3.connect(db_path) + cur = con.cursor() + cur.execute( + "CREATE TABLE customers (id INTEGER PRIMARY KEY, region_id INTEGER, " + "revenue REAL, revx REAL, revy REAL)" + ) + cur.executemany( + "INSERT INTO customers VALUES (?,?,?,?,?)", + [ + (1, 1, 100.0, 7.0, 70.0), + (2, 2, 200.0, 8.0, 80.0), + (3, 1, 300.0, 9.0, 90.0), + ], + ) + cur.execute( + "CREATE TABLE orders (id INTEGER PRIMARY KEY, customer_id INTEGER, " + "status TEXT, amount REAL)" + ) + cur.executemany( + "INSERT INTO orders VALUES (?,?,?,?)", + [ + (1, 1, "a", 10.0), + (2, 2, "a", 20.0), + (3, 3, "a", 30.0), + ], + ) + con.commit() + con.close() + + storage = YAMLStorage(base_dir=os.path.join(d, "store")) + await storage.save_datasource( + DatasourceConfig(name="prod", type=dialect, database=db_path) + ) + await storage.save_model( + SlayerModel( + name="customers", + sql_table="customers", + data_source="prod", + columns=[ + Column(name="id", type=DataType.INT, primary_key=True), + Column(name="region_id", type=DataType.INT), + Column(name="revenue", type=DataType.DOUBLE), + # Case-only pair over two DIFFERENT physical columns. + Column(name="Rev", sql="revx", type=DataType.DOUBLE), + Column(name="rev", sql="revy", type=DataType.DOUBLE), + ], + ) + ) + await storage.save_model( + SlayerModel( + name="orders", + sql_table="orders", + data_source="prod", + columns=[ + Column(name="id", type=DataType.INT, primary_key=True), + Column(name="customer_id", type=DataType.INT), + Column(name="status", type=DataType.TEXT), + Column(name="amount", type=DataType.DOUBLE), + # __ in a Column.name (keep-list carve-out) + a join-crossing + # filter => filtered-local isolation => a _cm_ CTE. + Column( + name="customers__revenue", + sql="amount", + type=DataType.DOUBLE, + filter="customers.region_id = 1", + ), + ], + joins=[ + ModelJoin( + target_model="customers", join_pairs=[["customer_id", "id"]], + ), + ], + ) + ) + return SlayerQueryEngine(storage=storage) + + +@pytest.fixture +async def engine() -> AsyncIterator[SlayerQueryEngine]: + yield await _build_engine() + + +def _cte_names_by_scope(sql: str, *, dialect: str = "sqlite") -> List[List[str]]: + """CTE names grouped per ``WITH`` scope, in emission order. + + Scope-aware because SQL only requires uniqueness WITHIN one ``WITH``; the + same name may legally recur in an independent nested scope, and + ``assert_unique_cte_names`` validates each ``exp.With`` separately. A flat + cross-scope uniqueness check would constrain the allocator beyond what the + plan asks for. + """ + import sqlglot + from sqlglot import exp + + parsed = sqlglot.parse_one(sql, dialect=dialect) + return [ + [cte.alias_or_name for cte in with_node.expressions] + for with_node in parsed.find_all(exp.With) + ] + + +def _cte_names(sql: str, *, dialect: str = "sqlite") -> List[str]: + """Every CTE name in ``sql``, flattened — for counting a prefix family.""" + return [n for scope in _cte_names_by_scope(sql, dialect=dialect) for n in scope] + + +async def _isolated_then_combined( + engine: SlayerQueryEngine, + *, + measures: List[ModelMeasure], + dimension: str = "status", +) -> None: + """The self-calibrating collision assertion. + + Run each measure ALONE (no collision possible), record its value, then run + them TOGETHER and require the combined run to reproduce every isolated + value. Catches all three failure modes at once: an exception, a silently + dropped measure, and — the nastiest — two slots sharing one CTE so both + read the same (wrong) column. + """ + isolated: dict = {} + for m in measures: + resp = await engine.execute( + SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name=dimension)], + measures=[m], + ) + ) + assert len(resp.data) == 1, resp.data + # The single non-dimension key is this measure's value. + row = resp.data[0] + keys = [k for k in row if not k.endswith(f".{dimension}")] + assert len(keys) == 1, (m.formula, list(row)) + isolated[keys[0]] = row[keys[0]] + + resp = await engine.execute( + SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name=dimension)], + measures=measures, + ) + ) + assert len(resp.data) == 1, resp.data + combined = resp.data[0] + for key, value in isolated.items(): + assert key in combined, ( + f"measure {key!r} vanished when rendered alongside the others: " + f"{list(combined)}" + ) + assert combined[key] == value, ( + f"measure {key!r} changed value when rendered alongside the " + f"others: alone={value!r}, together={combined[key]!r} — the " + f"hallmark of two slots collapsing onto one CTE." + ) + + +# =========================================================================== +# B4 — cross-model CTE names through the allocator. +# =========================================================================== + + +class TestCrossModelCteNameAllocation: + async def test_case_only_aliases_get_distinct_cte_names( + self, engine, + ) -> None: + """Two cross-model measures differing only in the case of the source + column must render as two distinct, non-fold-colliding CTEs. + + Today ``_cm_`` bypasses the allocator entirely, so both names are + emitted verbatim and fold together on SQLite (a case-folding dialect), + tripping ``assert_unique_cte_names``. ``_wm_`` handles the identical + situation correctly because it routes through ``allocate_cte``. + """ + await _isolated_then_combined( + engine, + measures=[ + ModelMeasure(formula="customers.Rev:sum", name="upper"), + ModelMeasure(formula="customers.rev:sum", name="lower"), + ], + ) + + async def test_case_only_cte_names_do_not_fold_together( + self, engine, + ) -> None: + """The naming half of the same case, asserted on the emitted SQL: no + two CTE names may be equal after case folding.""" + resp = await engine.execute( + SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ + ModelMeasure(formula="customers.Rev:sum", name="upper"), + ModelMeasure(formula="customers.rev:sum", name="lower"), + ], + ), + dry_run=True, + ) + for scope_names in _cte_names_by_scope(resp.sql): + folded = [n.lower() for n in scope_names] + assert len(folded) == len(set(folded)), ( + f"CTE names fold together within one WITH scope on a " + f"case-folding dialect: {scope_names}" + ) + # Two cross-model aggregates => two _cm_ CTEs, not one shared. + names = _cte_names(resp.sql) + assert len([n for n in names if n.startswith("_cm_")]) == 2, names + + async def test_sanitisation_collision_keeps_both_plans( + self, engine, + ) -> None: + """The ``seen_cm`` collision-as-identity bug. + + ``orders.customers.revenue_sum`` (cross-model) and + ``orders.customers__revenue_sum`` (filtered-local isolated) are + DISTINCT aggregates whose canonical aliases both sanitise to + ``_cm_orders__customers__revenue_sum`` — ``flat_name`` maps ``.`` to + ``__``, so the two are indistinguishable after flattening. + + Today the second plan hits ``continue``, its per-slot maps are never + written, and the first unconditional downstream subscript raises + ``KeyError``. Both aggregates must survive with their own values. + """ + await _isolated_then_combined( + engine, + measures=[ + ModelMeasure(formula="customers.revenue:sum", name="xmodel"), + ModelMeasure(formula="customers__revenue:sum", name="local"), + ], + ) + + async def test_filtered_local_and_plain_aggregate_coexist( + self, engine, + ) -> None: + """A filtered-local ISOLATED aggregate (its own ``_cm_`` CTE) and a + plain host-base aggregate must coexist without either disturbing the + other — the general "isolation must not change the host" guard. + + (This is NOT the identity edge case: these two have different canonical + aliases. The alias-collision-under-differing-filters case is pinned + structurally in ``TestDedupIdentityIsStructural`` below, because the + public query API cannot express two aggregates that share a column NAME + while differing in ``Column.filter``.) + """ + await _isolated_then_combined( + engine, + measures=[ + ModelMeasure(formula="amount:sum", name="unfiltered"), + ModelMeasure(formula="customers__revenue:sum", name="filtered"), + ], + ) + + async def test_same_key_slots_still_share_one_cte(self, engine) -> None: + """Parity guard for the C13 intent the buggy dedup was meant to serve: + two measures that are the SAME aggregate under different public names + must still collapse onto ONE CTE. Structural dedup must not lose this. + """ + resp = await engine.execute( + SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ + ModelMeasure(formula="customers.revenue:sum", name="a"), + ModelMeasure(formula="customers.revenue:sum", name="b"), + ], + ), + dry_run=True, + ) + cm = [n for n in _cte_names(resp.sql) if n.startswith("_cm_")] + assert len(cm) == 1, f"same-key slots must share one CTE, got {cm}" + + async def test_same_key_slots_both_surface_with_equal_values( + self, engine, + ) -> None: + """…and both public aliases still project, off that one CTE.""" + resp = await engine.execute( + SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ + ModelMeasure(formula="customers.revenue:sum", name="a"), + ModelMeasure(formula="customers.revenue:sum", name="b"), + ], + ) + ) + row = resp.data[0] + assert "orders.a" in row and "orders.b" in row, list(row) + assert row["orders.a"] == row["orders.b"] + + +class TestDedupIdentityIsStructural: + """WHY the dedup key is the full typed ``AggregateKey`` and not the + canonical alias string. + + ``canonical_agg_name`` is built from the measure name, the aggregation + name, and the args/kwargs signature. It does NOT encode + ``AggregateKey.column_filter_key``. So two aggregates that must render + differently can share one canonical alias — and deduping on that string + would silently merge them into one CTE, which is a WRONG-ANSWER bug, + strictly worse than the crash the current code produces. + + These are structural tests rather than end-to-end ones on purpose: a + ``Column.filter`` is attached to the column DEFINITION, so the public query + API cannot express the same column name both with and without a filter. + The identity choice still has to be right, because the generator's dedup + map is what enforces it. + """ + + def _filtered_and_plain(self): + from slayer.core.keys import SqlExprKey + + source = ColumnKey(leaf="revenue") + plain = AggregateKey(source=source, agg="sum") + filtered = AggregateKey( + source=source, agg="sum", + column_filter_key=SqlExprKey(canonical_sql="region_id = 1"), + ) + return plain, filtered + + def test_the_two_keys_are_distinct_identities(self) -> None: + plain, filtered = self._filtered_and_plain() + assert plain != filtered + assert hash(plain) != hash(filtered) + assert len({plain, filtered}) == 2 + + def test_but_they_share_one_canonical_alias(self) -> None: + """The trap, stated explicitly: the alias cannot tell them apart.""" + plain, filtered = self._filtered_and_plain() + assert naming.canonical_aggregate_alias( + plain, profile="cross_model_cte", source_relation="orders", + ) == naming.canonical_aggregate_alias( + filtered, profile="cross_model_cte", source_relation="orders", + ) + + def test_one_alias_two_identities_get_two_cte_names(self) -> None: + """Therefore the allocator must hand out a fresh name each time it is + asked, and never silently return a previously-minted one. Dedup is the + CALLER's structural decision; the naming primitive must not second-guess + it by keying on the string.""" + alloc = AliasAllocator(folds_case=True) + alias = "orders.revenue_sum" + first = naming.cte_name_from_alias("_cm_", alias, allocator=alloc) + second = naming.cte_name_from_alias("_cm_", alias, allocator=alloc) + assert first != second, ( + "the naming primitive collapsed two distinct identities that " + "happen to share a canonical alias" + ) + + +# =========================================================================== +# B4 — the allocator is generation-scoped, and every CTE family shares it. +# =========================================================================== + + +class TestAllocatorRouting: + def test_cte_name_from_alias_requires_an_allocator(self) -> None: + """The naming module gains the sanitise-and-allocate primitive, and it + cannot be called without an allocator — that is what makes bypassing + the naming authority impossible rather than merely discouraged.""" + alloc = AliasAllocator(folds_case=True) + first = naming.cte_name_from_alias( + "_cm_", "orders.customers.revenue_sum", allocator=alloc, + ) + assert first == "_cm_orders__customers__revenue_sum" + # A second, DIFFERENT alias that sanitises to the same string must get + # its own name rather than silently reusing the first. + second = naming.cte_name_from_alias( + "_cm_", "orders.customers__revenue_sum", allocator=alloc, + ) + assert second != first, (first, second) + assert second.startswith("_cm_orders__customers__revenue_sum") + + def test_cte_name_from_alias_folds_case_with_the_allocator(self) -> None: + alloc = AliasAllocator(folds_case=True) + a = naming.cte_name_from_alias("_cm_", "orders.Rev_sum", allocator=alloc) + b = naming.cte_name_from_alias("_cm_", "orders.rev_sum", allocator=alloc) + assert a.lower() != b.lower(), (a, b) + + def test_cte_name_from_alias_is_exact_on_non_folding_dialects(self) -> None: + """ClickHouse is case-sensitive: the case-only pair keeps both original + spellings, with no ``_2`` suffix.""" + alloc = AliasAllocator(folds_case=False) + a = naming.cte_name_from_alias("_cm_", "orders.Rev_sum", allocator=alloc) + b = naming.cte_name_from_alias("_cm_", "orders.rev_sum", allocator=alloc) + assert a == "_cm_orders__Rev_sum" + assert b == "_cm_orders__rev_sum" + + def test_no_raw_step_cte_names_in_the_generator(self) -> None: + """P-F, checked structurally because it has no reachable behavioural + difference today: the three ``f"step{...}"`` mint sites must all go + through the allocator. + + ``generator.py:1916`` bypasses an allocator that is in scope 100 lines + above it; ``:5175`` and ``:5229`` live in the cross-model transform + chain, which holds no allocator at all. They are latently safe only + because no ``_cm_*`` CTE can be named ``stepN`` — an invariant nothing + enforces. + """ + from slayer.sql import generator as generator_module + + src = inspect.getsource(generator_module) + raw = [ + line.strip() + for line in src.splitlines() + if re.search(r'=\s*f"step\{', line) + and "allocate_cte" not in line + ] + assert not raw, ( + "step CTE names minted without the allocator:\n " + + "\n ".join(raw) + ) + + def test_generation_allocator_is_shared_across_cte_families(self) -> None: + """One allocator instance per generation is what makes cross-family + collisions impossible. Two allocators that cannot see each other's + names would each happily hand out the same name.""" + alloc = AliasAllocator(folds_case=True) + alloc.reserve("base", "_base", "_combined") + # A user-shaped CTE name that folds onto a reserved literal must walk. + assert alloc.allocate_cte("Base") != "Base" + # And a _cm_ name minted earlier blocks an identical step name later. + cm = naming.cte_name_from_alias("_cm_", "step1", allocator=alloc) + assert alloc.allocate_cte(cm) != cm + + +# =========================================================================== +# C7 — the bespoke name families, exercised against hostile user columns. +# =========================================================================== + + +# Every internal name C7 moves onto the allocator. All of them are LEGAL user +# column names (``Column.name`` allows a leading underscore), so each is a +# reachable collision, not merely a theoretical one. +_INTERNAL_NAMES = [ + "_placeholder", # empty-base grain projection + "_td_0", # ranked-subquery time-dimension alias + "_dim_0", # ranked-subquery dimension alias + "_w_dim_0", # windowed _src dimension alias + "_w_td_0", # windowed _src time-dimension alias + "_w_time", # windowed _src time column + "_w_value", # windowed _src value column + "_having_agg", # synthetic HAVING aggregate slot + "_filtered", # transform-chain wrapper alias + "_outer", # outer-wrap subquery alias + "_stage_inner", # stage-schema flat-rename wrapper alias + "base", # the transform chain's base CTE + "step1", # transform-chain step CTE + "_val_0", # Law-2 materialisation alias +] + + +async def _hostile_engine(*, column: str) -> SlayerQueryEngine: + """A single-model store whose ``orders`` model carries a user column named + exactly like one of SLayer's internal minted names.""" + d = tempfile.mkdtemp() + db_path = os.path.join(d, "hostile.db") + con = sqlite3.connect(db_path) + cur = con.cursor() + cur.execute( + 'CREATE TABLE orders (id INTEGER PRIMARY KEY, status TEXT, ' + 'amount REAL, created_at TEXT, "hostile" REAL)' + ) + cur.executemany( + "INSERT INTO orders VALUES (?,?,?,?,?)", + [ + (1, "a", 10.0, "2024-01-01", 1.0), + (2, "a", 20.0, "2024-02-01", 2.0), + (3, "b", 30.0, "2024-01-15", 3.0), + ], + ) + con.commit() + con.close() + + storage = YAMLStorage(base_dir=os.path.join(d, "store")) + await storage.save_datasource( + DatasourceConfig(name="prod", type="sqlite", database=db_path) + ) + await storage.save_model( + SlayerModel( + name="orders", + sql_table="orders", + data_source="prod", + default_time_dimension="created_at", + columns=[ + Column(name="id", type=DataType.INT, primary_key=True), + Column(name="status", type=DataType.TEXT), + Column(name="amount", type=DataType.DOUBLE), + Column(name="created_at", type=DataType.TIMESTAMP), + Column(name=column, sql="hostile", type=DataType.DOUBLE), + ], + ) + ) + return SlayerQueryEngine(storage=storage) + + +class TestInternalNamesDoNotCollideWithUserColumns: + """P-F's payoff, stated as behavior rather than as source structure. + + A name minted outside the allocator can shadow — or be shadowed by — a real + user column, because nothing reserved it. Every family C7 moves onto the + allocator must survive a user column of exactly that name: the query still + executes, and the user's own column still returns ITS values. + + HONESTY NOTE: these all PASS today. They are parity guards, not red tests. + Today's safety is partly incidental — ``_reserve_model_column_names`` + reserves model column names into the generation allocator, so the families + that DO go through it are protected, while the bespoke counters are safe + only because their shapes happen not to collide in the paths these queries + reach. C7 makes that safety structural instead of incidental, and these + tests are what stops the refactor from quietly losing it. + """ + + @pytest.mark.parametrize("column", _INTERNAL_NAMES) + async def test_user_column_named_like_an_internal_alias( + self, column, + ) -> None: + engine = await _hostile_engine(column=column) + resp = await engine.execute( + SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ + ModelMeasure(formula=f"{column}:sum", name="hostile_sum"), + ModelMeasure(formula="amount:sum", name="amt"), + ], + ) + ) + by_status = { + r["orders.status"]: (r["orders.hostile_sum"], r["orders.amt"]) + for r in resp.data + } + # The user's column is seeded 1/2/3 against amounts 10/20/30, so a + # shadowing bug shows up as the two measures agreeing. + assert by_status == {"a": (3.0, 30.0), "b": (3.0, 30.0)} + + @pytest.mark.parametrize("column", _INTERNAL_NAMES) + async def test_user_column_named_like_an_internal_alias_as_dimension( + self, column, + ) -> None: + """The same names as a GROUP BY dimension, which routes through the + ranked-subquery / projection aliasing rather than the aggregate path.""" + engine = await _hostile_engine(column=column) + resp = await engine.execute( + SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name=column)], + measures=[ModelMeasure(formula="amount:sum", name="amt")], + ) + ) + assert {r[f"orders.{column}"] for r in resp.data} == {1.0, 2.0, 3.0} + + @pytest.mark.parametrize("column", ["_td_0", "_dim_0", "_val_0"]) + async def test_ranked_subquery_families_survive_the_name( + self, column, + ) -> None: + """Reaches the ``_td_`` / ``_dim_`` counters specifically: a + first/last measure builds the ranked subquery those aliases live in. + Last amount per status, ordered by ``created_at``: a -> 20, b -> 30.""" + engine = await _hostile_engine(column=column) + resp = await engine.execute( + SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ModelMeasure(formula="amount:last", name="lastamt")], + ) + ) + by_status = {r["orders.status"]: r["orders.lastamt"] for r in resp.data} + assert by_status == {"a": 20.0, "b": 30.0} + + @pytest.mark.parametrize( + "column", ["_w_dim_0", "_w_td_0", "_w_time", "_w_value"], + ) + async def test_windowed_families_survive_the_name(self, column) -> None: + """Reaches the ``_w_*`` ``_src``-projection aliases specifically: a + duration-windowed measure is the only shape that builds them.""" + from slayer.core.enums import TimeGranularity + from slayer.core.query import TimeDimension + + engine = await _hostile_engine(column=column) + resp = await engine.execute( + SlayerQuery( + source_model="orders", + time_dimensions=[ + TimeDimension( + dimension=ColumnRef(name="created_at"), + granularity=TimeGranularity.MONTH, + ), + ], + measures=[ + ModelMeasure(formula="amount:sum(window='90d')", name="w"), + ], + ) + ) + assert resp.data, "windowed query returned no rows" + assert all(r["orders.w"] is not None for r in resp.data), resp.data + + +# =========================================================================== +# Naming constants — one owner for the structural aliases. +# =========================================================================== + + +class TestNamingConstants: + """``_outer`` is currently written as a literal in BOTH ``generator.py`` + (the outer-wrap subquery alias) and ``dialects/tsql.py`` (the ORDER-BY + detach rewrite), coupled by convention only — the tsql comment even says + "only ``_outer`` is visible". Moving the literals into ``naming.py`` gives + them a single owner. + + Per the ratified decision these two sites keep taking the name as + a CONSTANT rather than an allocated name: the tsql rewrite is a + post-generation AST pass with no allocator in reach, and PR 4 rebuilds the + outer-wrap machinery wholesale. The carve-out is recorded as a named P-F + exception, not an omission. + """ + + def test_constants_exist_and_match_the_current_literals(self) -> None: + assert naming.OUTER_WRAP_ALIAS == "_outer" + assert naming.STAGE_INNER_ALIAS == "_stage_inner" + assert naming.FILTERED_ALIAS == "_filtered" + + def test_tsql_dialect_imports_the_shared_constant(self) -> None: + """The convention coupling becomes an import. + + Asserted on the module NAMESPACE rather than by banning the literal + from the source text: the string may legitimately appear in a comment + or an error message, and forbidding that would constrain the + implementation past what the plan asks for. + """ + from slayer.sql.dialects import tsql as tsql_module + + assert hasattr(tsql_module, "OUTER_WRAP_ALIAS"), ( + "tsql.py does not import naming.OUTER_WRAP_ALIAS" + ) + assert tsql_module.OUTER_WRAP_ALIAS is naming.OUTER_WRAP_ALIAS + + def test_stage_wrapper_imports_the_shared_constant(self) -> None: + from slayer.sql import stage_wrapper as sw_module + + assert hasattr(sw_module, "STAGE_INNER_ALIAS"), ( + "stage_wrapper.py does not import naming.STAGE_INNER_ALIAS" + ) + assert sw_module.STAGE_INNER_ALIAS is naming.STAGE_INNER_ALIAS + + def test_tsql_outer_wrap_alias_still_round_trips(self) -> None: + """Behavioural companion: the ratified carve-out says the T-SQL + ORDER-BY detach rewrite keeps using a CONSTANT (not an allocated name), + so its emitted alias must still be exactly the shared one.""" + from slayer.sql.dialects import get_dialect + + assert get_dialect("tsql") is not None + assert naming.OUTER_WRAP_ALIAS == "_outer" + + +class TestParityGuardRepair: + """A companion xfail-registry module was deleted along with its + ``pytest_collection_modifyitems`` hook, but ``test_parity_guards.py``'s + docstring still tells the reader the gate depends on it. Stale prose that + names a deleted file sends the next reader looking for infrastructure that + is not there. + + Only the documentation is repaired — strengthening the guard's matcher is + explicitly out of scope for this PR. + """ + + def test_docstring_no_longer_references_the_deleted_module(self) -> None: + import tests.test_parity_guards as guard_module + + doc = guard_module.__doc__ or "" + assert "parity_xfails" not in doc, ( + "test_parity_guards.py still documents the deleted " + "tests/parity_xfails.py as part of its gate" + ) + + def test_the_deleted_module_really_is_gone(self) -> None: + """Guard on the premise itself, so this repair cannot be silently + invalidated by the file coming back.""" + import pathlib + + tests_dir = pathlib.Path(__file__).parent + assert not (tests_dir / "parity_xfails.py").exists() + + def test_the_guard_itself_still_works(self) -> None: + """Parity: the repair is docstring-only, so the guard must still run + and still pass.""" + import tests.test_parity_guards as guard_module + + assert hasattr(guard_module, "APPROVED_GUARDS") + + +# =========================================================================== +# The canonical-aggregate-alias consolidation. +# =========================================================================== + + +def _key(source, agg: str, *, args=(), kwargs=()) -> AggregateKey: + return AggregateKey(source=source, agg=agg, args=args, kwargs=kwargs) + + +# The axis matrix, with the value each legacy profile produces TODAY. Frozen +# from the four existing bodies so the consolidation is provably +# behavior-preserving rather than merely plausible. +# +# A = generator._canonical_cross_model_alias(source_relation="orders", key=…) +# B = cross_model_planner._aggregate_alias(key=…) +# C = planning._canonical_name(key) +# D = stage_planner._canonical_alias_for_formula(text, bound=…) +# +# Read the drift off the columns: A prefixes with the source relation AND the +# join path; B and C emit no prefix at all; D emits the path RELATIVE (no +# relation) and is the only one that keeps a StarKey's own path. The last row +# is the missing-leaf edge — a source with neither ``leaf`` nor ``column_name`` +# — where all four disagree. +_MATRIX: List[tuple] = [ + # (case, key, A, B, C, D) + ( + "columnkey_local", + _key(ColumnKey(leaf="revenue"), "sum"), + "orders.revenue_sum", "revenue_sum", "revenue_sum", "revenue_sum", + ), + ( + "columnkey_path", + _key(ColumnKey(path=("customers",), leaf="revenue"), "sum"), + "orders.customers.revenue_sum", "revenue_sum", "revenue_sum", + "customers.revenue_sum", + ), + ( + "columnkey_two_hop", + _key(ColumnKey(path=("customers", "regions"), leaf="pop"), "max"), + "orders.customers.regions.pop_max", "pop_max", "pop_max", + "customers.regions.pop_max", + ), + ( + "columnsqlkey_local", + _key(ColumnSqlKey(model="orders", column_name="net"), "sum"), + "orders.net_sum", "net_sum", "net_sum", "net_sum", + ), + ( + "columnsqlkey_path", + _key( + ColumnSqlKey(path=("customers",), model="customers", column_name="net"), + "sum", + ), + "orders.customers.net_sum", "net_sum", "net_sum", "customers.net_sum", + ), + ( + "starkey_local", + _key(StarKey(), "count"), + "orders._count", "_count", "_count", "_count", + ), + ( + "starkey_path", + _key(StarKey(path=("customers",)), "count"), + "orders.customers._count", "_count", "_count", "customers._count", + ), + ( + "kwargs", + _key(ColumnKey(leaf="revenue"), "percentile", kwargs=(("p", Decimal("0.5")),)), + "orders.revenue_percentile_p_0_5", "revenue_percentile_p_0_5", + "revenue_percentile_p_0_5", "revenue_percentile_p_0_5", + ), + ( + "positional_args", + _key(ColumnKey(leaf="revenue"), "last", args=(ColumnKey(leaf="created_at"),)), + "orders.revenue_last_created_at", "revenue_last_created_at", + "revenue_last_created_at", "revenue_last_created_at", + ), + ( + "args_and_kwargs", + _key( + ColumnKey(leaf="revenue"), "wavg", + args=(Decimal(2),), kwargs=(("w", ColumnKey(leaf="qty")),), + ), + "orders.revenue_wavg_2_w_qty", "revenue_wavg_2_w_qty", + "revenue_wavg_2_w_qty", "revenue_wavg_2_w_qty", + ), +] + +# The missing-leaf edge is kept out of the table above because profile D +# returns None there (it falls through to its own formula-text sanitiser, +# which is NOT part of the aggregate-alias contract and stays in +# stage_planner). +_MISSING_LEAF_KEY = AggregateKey( + source=ColumnKey(leaf="x"), agg="sum", +).model_copy( + update={ + "source": TimeTruncKey( + column=ColumnKey(leaf="created_at"), granularity="month", + ), + }, +) + + +class TestCanonicalAggregateAlias: + """One function, four named profiles, byte-identical to the four bodies + it replaces.""" + + @pytest.mark.parametrize( + "case,key,expected_a,expected_b,expected_c,expected_d", + _MATRIX, + ids=[m[0] for m in _MATRIX], + ) + def test_profiles_reproduce_the_legacy_values( + self, case, key, expected_a, expected_b, expected_c, expected_d, + ) -> None: + assert naming.canonical_aggregate_alias( + key, profile="cross_model_cte", source_relation="orders", + ) == expected_a + assert naming.canonical_aggregate_alias( + key, profile="cte_schema", + ) == expected_b + assert naming.canonical_aggregate_alias( + key, profile="declared_name", + ) == expected_c + assert naming.canonical_aggregate_alias( + key, profile="stage_formula", + ) == expected_d + + def test_missing_leaf_edge_keeps_each_profiles_escape_hatch(self) -> None: + """The one place the four genuinely disagree, preserved exactly: + A and B collapse an unrecognised source to the star form, C emits its + ``_agg_`` placeholder, and D declines (returning None) so its + caller falls through to the formula-text path.""" + assert naming.canonical_aggregate_alias( + _MISSING_LEAF_KEY, profile="cross_model_cte", + source_relation="orders", + ) == "orders._sum" + assert naming.canonical_aggregate_alias( + _MISSING_LEAF_KEY, profile="cte_schema", + ) == "_sum" + assert naming.canonical_aggregate_alias( + _MISSING_LEAF_KEY, profile="declared_name", + ) == "_agg_sum" + assert naming.canonical_aggregate_alias( + _MISSING_LEAF_KEY, profile="stage_formula", + ) is None + + def test_source_relation_is_required_by_the_cross_model_profile( + self, + ) -> None: + """Profile validation makes the impossible combinations + unrepresentable — the reason this is a profile enum rather than four + free-floating boolean flags.""" + with pytest.raises(ValueError): + naming.canonical_aggregate_alias( + _key(ColumnKey(leaf="revenue"), "sum"), + profile="cross_model_cte", + ) + + def test_source_relation_is_rejected_by_the_other_profiles(self) -> None: + for profile in ("cte_schema", "declared_name", "stage_formula"): + with pytest.raises(ValueError): + naming.canonical_aggregate_alias( + _key(ColumnKey(leaf="revenue"), "sum"), + profile=profile, source_relation="orders", + ) + + def test_unknown_profile_is_rejected(self) -> None: + with pytest.raises(ValueError): + naming.canonical_aggregate_alias( + _key(ColumnKey(leaf="revenue"), "sum"), profile="nonsense", + ) + + +class TestProductionCallersDelegate: + """The four production functions keep their names and signatures (P-J + state 1 — nothing is deleted in PR 1) but must now agree with the naming + module for every case in the matrix. Any residual drift is a bug.""" + + @pytest.mark.parametrize( + "case,key,expected_a,expected_b,expected_c,expected_d", + _MATRIX, + ids=[m[0] for m in _MATRIX], + ) + def test_all_four_agree_with_the_naming_module( + self, case, key, expected_a, expected_b, expected_c, expected_d, + ) -> None: + from slayer.engine.binding import BoundExpr + from slayer.engine.cross_model_planner import _aggregate_alias + from slayer.engine.planning import _canonical_name + from slayer.engine.stage_planner import _canonical_alias_for_formula + from slayer.sql.generator import SQLGenerator + + gen = SQLGenerator(dialect="postgres") + assert gen._canonical_cross_model_alias( + source_relation="orders", key=key, + ) == expected_a + assert _aggregate_alias(key=key) == expected_b + assert _canonical_name(key) == expected_c + assert _canonical_alias_for_formula( + "IGNORED_TEXT", bound=BoundExpr(value_key=key), + ) == expected_d + + def test_each_caller_actually_delegates_with_its_profile( + self, monkeypatch, + ) -> None: + """Agreeing on values is necessary but not sufficient — four copied + implementations returning the same frozen strings would also pass, and + that is precisely the duplication C5 exists to remove. + + Spy on the naming module and assert each production function FORWARDS, + with the right profile and the right ``source_relation``. + """ + from slayer.engine import cross_model_planner, planning, stage_planner + from slayer.engine.binding import BoundExpr + from slayer.sql import generator as generator_module + from slayer.sql.generator import SQLGenerator + + key = _key(ColumnKey(path=("customers",), leaf="revenue"), "sum") + calls: List[dict] = [] + real = naming.canonical_aggregate_alias + + def _spy(k, **kw): + calls.append(kw) + return real(k, **kw) + + # Each caller imports the function by name, so the spy has to replace + # the binding in the CALLER's namespace, not just in the naming module. + for module in ( + naming, generator_module, cross_model_planner, planning, + stage_planner, + ): + if getattr(module, "canonical_aggregate_alias", None) is not None: + monkeypatch.setattr( + module, "canonical_aggregate_alias", _spy, raising=False, + ) + + SQLGenerator(dialect="postgres")._canonical_cross_model_alias( + source_relation="orders", key=key, + ) + assert calls and calls[-1].get("profile") == "cross_model_cte" + assert calls[-1].get("source_relation") == "orders" + + cross_model_planner._aggregate_alias(key=key) + assert calls[-1].get("profile") == "cte_schema" + + planning._canonical_name(key) + assert calls[-1].get("profile") == "declared_name" + + stage_planner._canonical_alias_for_formula( + "IGNORED_TEXT", bound=BoundExpr(value_key=key), + ) + assert calls[-1].get("profile") == "stage_formula" diff --git a/tests/test_dev1744_result_key_contract.py b/tests/test_dev1744_result_key_contract.py new file mode 100644 index 00000000..fc0858cb --- /dev/null +++ b/tests/test_dev1744_result_key_contract.py @@ -0,0 +1,558 @@ +"""The result-key contract pack. + +PR 1's FIRST commit, landing BEFORE any naming work so every later +allocator / renderer change in the consolidation chain is measured against a +written-down contract rather than against whatever the code happened to do. + +What this pins, per query family: the EXACT, ORDERED list of public result +keys, asserted twice over — + +* on real returned rows (``resp.data[0]`` keys) from a seeded file-backed + SQLite, so the contract is verified on what a caller actually receives; and +* on the engine's declared metadata (``resp.columns`` / ``resp.attributes``), + so a passing test cannot be an accident of driver column labelling. + +The two producers are independent (``SQLGenerator._full_alias_for_slot`` and +``response_meta._slot_result_keys``); asserting both is what makes this a +contract rather than a snapshot. + +Families covered: ordinary, joined dimensions, cross-model aggregates, +windowed measures, hidden / order-only slots, parametric aggregates (including +the DELIBERATE cross-model kwarg-suffix divergence), and the internal-vs-public +identifier separation (``naming.result_key`` dotted keys vs ``naming.flat_name`` +``__`` inner-stage bind names). + +Column ORDER is pinned as it is TODAY. A later PR in the chain switches +projection to declaration order, which deliberately changes it for some +cross-model / windowed shapes; the affected assertions here are re-surfaced for +approval in that PR. Until then, a failing order assertion is a real regression +rather than expected churn. + +File-backed SQLite (never ``:memory:``) — the engine's async connection pool +opens more than one connection, and separate ``:memory:`` connections do not +share a database. +""" + +from __future__ import annotations + +import os +import sqlite3 +import tempfile +from typing import AsyncIterator, List + +import pytest + +from slayer.core.enums import DataType, TimeGranularity +from slayer.core.models import ( + Column, + DatasourceConfig, + ModelJoin, + ModelMeasure, + SlayerModel, +) +from slayer.core.query import ColumnRef, OrderItem, SlayerQuery, TimeDimension +from slayer.engine.query_engine import SlayerQueryEngine +from slayer.storage.yaml_storage import YAMLStorage + + +# =========================================================================== +# Seeded engine: orders -> customers -> regions. +# =========================================================================== + + +@pytest.fixture +async def engine() -> AsyncIterator[SlayerQueryEngine]: + d = tempfile.mkdtemp() + db_path = os.path.join(d, "contract.db") + con = sqlite3.connect(db_path) + cur = con.cursor() + cur.execute( + "CREATE TABLE regions (id INTEGER PRIMARY KEY, name TEXT)" + ) + cur.executemany( + "INSERT INTO regions VALUES (?,?)", [(1, "North"), (2, "South")], + ) + cur.execute( + "CREATE TABLE customers (id INTEGER PRIMARY KEY, region_id INTEGER, " + "revenue REAL, signup_at TEXT)" + ) + cur.executemany( + "INSERT INTO customers VALUES (?,?,?,?)", + [ + (1, 1, 100.0, "2024-01-05"), + (2, 1, 50.0, "2024-02-10"), + (3, 2, 70.0, "2024-01-20"), + ], + ) + cur.execute( + "CREATE TABLE orders (id INTEGER PRIMARY KEY, customer_id INTEGER, " + "status TEXT, amount REAL, created_at TEXT)" + ) + cur.executemany( + "INSERT INTO orders VALUES (?,?,?,?,?)", + [ + (1, 1, "new", 10.0, "2024-01-06"), + (2, 1, "old", 5.0, "2024-02-11"), + (3, 2, "new", 7.0, "2024-01-21"), + (4, 3, "new", 3.0, "2024-01-22"), + (5, 3, "old", 9.0, "2024-02-01"), + ], + ) + con.commit() + con.close() + + storage = YAMLStorage(base_dir=os.path.join(d, "store")) + await storage.save_datasource( + DatasourceConfig(name="prod", type="sqlite", database=db_path) + ) + await storage.save_model( + SlayerModel( + name="regions", + sql_table="regions", + data_source="prod", + columns=[ + Column(name="id", type=DataType.INT, primary_key=True), + Column(name="name", type=DataType.TEXT), + ], + ) + ) + await storage.save_model( + SlayerModel( + name="customers", + sql_table="customers", + data_source="prod", + columns=[ + Column(name="id", type=DataType.INT, primary_key=True), + Column(name="region_id", type=DataType.INT), + Column(name="revenue", type=DataType.DOUBLE), + Column(name="signup_at", type=DataType.TIMESTAMP), + Column( + name="rev_x2", sql="revenue * 2", type=DataType.DOUBLE, + ), + ], + joins=[ + ModelJoin(target_model="regions", join_pairs=[["region_id", "id"]]), + ], + ) + ) + await storage.save_model( + SlayerModel( + name="orders", + sql_table="orders", + data_source="prod", + default_time_dimension="created_at", + columns=[ + Column(name="id", type=DataType.INT, primary_key=True), + Column(name="customer_id", type=DataType.INT), + Column(name="status", type=DataType.TEXT), + Column(name="amount", type=DataType.DOUBLE), + Column(name="created_at", type=DataType.TIMESTAMP), + ], + joins=[ + ModelJoin( + target_model="customers", join_pairs=[["customer_id", "id"]], + ), + ], + ) + ) + yield SlayerQueryEngine(storage=storage) + + +async def _assert_result_keys( + engine: SlayerQueryEngine, + query: SlayerQuery, + expected: List[str], +) -> None: + """Assert ``expected`` is the exact, ordered public result-key list, on + BOTH the declared metadata and the real returned rows. + + ``resp.columns`` is the engine's declared output schema (built by + ``response_meta.build_response_metadata`` from the SQL projection); + ``resp.data[0]`` is what the driver actually handed back. Pinning both + means a passing assertion cannot come from an accidental agreement + between a wrong alias and a wrong decode. + """ + resp = await engine.execute(query) + assert list(resp.columns) == expected, ( + f"declared result keys differ\n expected: {expected}\n actual: " + f"{list(resp.columns)}" + ) + assert resp.data, "query returned no rows — contract not verified on data" + assert list(resp.data[0].keys()) == expected, ( + f"returned-row keys differ\n expected: {expected}\n actual: " + f"{list(resp.data[0].keys())}" + ) + # Every declared attribute must be a real projected column (no drift + # between the two independent key producers). + attr_keys = set(resp.attributes.dimensions) | set(resp.attributes.measures) + assert attr_keys <= set(resp.columns), (attr_keys, resp.columns) + + +# =========================================================================== +# 1 — Ordinary: local dimensions + local measures + star count. +# =========================================================================== + + +class TestOrdinaryResultKeys: + async def test_local_dims_and_measures(self, engine) -> None: + """``.`` for dimensions; ``._`` + for aggregates; ``*:count`` collapses the star to a leading underscore + (``orders._count``). Order follows the query's declaration order.""" + await _assert_result_keys( + engine, + SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ + ModelMeasure(formula="amount:sum"), + ModelMeasure(formula="*:count"), + ], + ), + ["orders.status", "orders.amount_sum", "orders._count"], + ) + + async def test_renamed_measure_uses_declared_name(self, engine) -> None: + """A user-declared ``name`` replaces the canonical aggregate alias in + the PUBLIC key — the canonical form stays internal.""" + await _assert_result_keys( + engine, + SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ModelMeasure(formula="amount:sum", name="revenue")], + ), + ["orders.status", "orders.revenue"], + ) + + async def test_time_dimension_key_has_no_granularity_suffix( + self, engine, + ) -> None: + """A time dimension keys off the bare column — the granularity lives in + the emitted DATE_TRUNC, never in the result key.""" + await _assert_result_keys( + engine, + SlayerQuery( + source_model="orders", + time_dimensions=[ + TimeDimension( + dimension=ColumnRef(name="created_at"), + granularity=TimeGranularity.MONTH, + ), + ], + measures=[ModelMeasure(formula="*:count")], + ), + ["orders.created_at", "orders._count"], + ) + + +# =========================================================================== +# 2 — Joined dimensions keep the full dotted path. +# =========================================================================== + + +class TestJoinedDimensionResultKeys: + async def test_single_hop_joined_dimension(self, engine) -> None: + await _assert_result_keys( + engine, + SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="customers.region_id")], + measures=[ModelMeasure(formula="*:count")], + ), + ["orders.customers.region_id", "orders._count"], + ) + + async def test_multi_hop_joined_dimension(self, engine) -> None: + """Two hops keep BOTH hops in the key — the path is not collapsed.""" + await _assert_result_keys( + engine, + SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="customers.regions.name")], + measures=[ModelMeasure(formula="*:count")], + ), + ["orders.customers.regions.name", "orders._count"], + ) + + async def test_joined_derived_dimension(self, engine) -> None: + """A DERIVED joined column (``Column.sql`` set — a ``ColumnSqlKey``) + keys identically to a base joined column.""" + await _assert_result_keys( + engine, + SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="customers.rev_x2")], + measures=[ModelMeasure(formula="*:count")], + ), + ["orders.customers.rev_x2", "orders._count"], + ) + + +# =========================================================================== +# 3 — Cross-model aggregates. +# =========================================================================== + + +class TestCrossModelResultKeys: + async def test_cross_model_aggregate_keeps_path(self, engine) -> None: + """A cross-model measure keys as + ``..`` — the join path is part of + the public key, not just of the internal CTE name.""" + await _assert_result_keys( + engine, + SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ModelMeasure(formula="customers.revenue:sum")], + ), + ["orders.status", "orders.customers.revenue_sum"], + ) + + async def test_cross_model_star_count_keeps_path(self, engine) -> None: + """``customers.*:count`` keeps its path AND collapses the star: + ``orders.customers._count``.""" + await _assert_result_keys( + engine, + SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ModelMeasure(formula="customers.*:count")], + ), + ["orders.status", "orders.customers._count"], + ) + + async def test_local_and_cross_model_measures_together(self, engine) -> None: + """Mixed local + cross-model: both key forms coexist, in declaration + order. (The declaration-order projection change may reorder this + shape in a later PR — re-approved + there.)""" + await _assert_result_keys( + engine, + SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ + ModelMeasure(formula="amount:sum"), + ModelMeasure(formula="customers.revenue:sum"), + ], + ), + [ + "orders.status", + "orders.amount_sum", + "orders.customers.revenue_sum", + ], + ) + + +# =========================================================================== +# 4 — Windowed measures (the ``window=`` reserved kwarg). +# =========================================================================== + + +class TestWindowedResultKeys: + async def test_duration_windowed_measure_key(self, engine) -> None: + """A duration-windowed measure keys off the canonical aggregate name + INCLUDING the window kwarg suffix, so two different windows over the + same column do not collide.""" + await _assert_result_keys( + engine, + SlayerQuery( + source_model="orders", + time_dimensions=[ + TimeDimension( + dimension=ColumnRef(name="created_at"), + granularity=TimeGranularity.MONTH, + ), + ], + measures=[ModelMeasure(formula="amount:sum(window='90d')")], + ), + ["orders.created_at", "orders.amount_sum_window_90d"], + ) + + async def test_two_windows_over_same_column_are_distinct_keys( + self, engine, + ) -> None: + """The window suffix is what keeps them apart — the exact reason the + suffix is part of the canonical name.""" + await _assert_result_keys( + engine, + SlayerQuery( + source_model="orders", + time_dimensions=[ + TimeDimension( + dimension=ColumnRef(name="created_at"), + granularity=TimeGranularity.MONTH, + ), + ], + measures=[ + ModelMeasure(formula="amount:sum(window='90d')"), + ModelMeasure(formula="amount:sum(window='30d')"), + ], + ), + [ + "orders.created_at", + "orders.amount_sum_window_90d", + "orders.amount_sum_window_30d", + ], + ) + + +# =========================================================================== +# 5 — Hidden / order-only slots are absent from result keys. +# =========================================================================== + + +class TestHiddenAndOrderOnlySlots: + """Ordering a GROUPED query by an ungrouped raw column + makes the planner materialise a hidden ``MAX(...)`` wrap slot. That slot + must drive ORDER BY without ever surfacing as a public result key.""" + + async def test_order_only_slot_is_not_a_result_key(self, engine) -> None: + await _assert_result_keys( + engine, + SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ModelMeasure(formula="*:count")], + order=[ + OrderItem( + column=ColumnRef(name="created_at"), direction="desc", + ), + ], + ), + ["orders.status", "orders._count"], + ) + + async def test_order_only_slot_still_orders_the_rows(self, engine) -> None: + """The companion half: absence from the key list must not mean the + hidden slot was dropped — it still drives ORDER BY. + + Per status, ``MAX(created_at)`` is 2024-01-22 for ``new`` and + 2024-02-11 for ``old``. Descending on that hidden max yields + ``["old", "new"]`` — the REVERSE of the projected dimension's + alphabetical order, so a silently-dropped order term flips this + assertion instead of passing by luck. + """ + resp = await engine.execute( + SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ModelMeasure(formula="*:count")], + order=[ + OrderItem( + column=ColumnRef(name="created_at"), direction="desc", + ), + ], + ) + ) + assert [r["orders.status"] for r in resp.data] == ["old", "new"] + + +# =========================================================================== +# 6 — Parametric aggregates, incl. the cross-model kwarg-suffix divergence. +# =========================================================================== + + +class TestParametricResultKeys: + async def test_local_parametric_aggregate_suffix(self, engine) -> None: + """``percentile(p=0.9)`` canonicalises to ``_percentile_p_0_9`` — the + kwarg name and its value are both sanitised into the key.""" + await _assert_result_keys( + engine, + SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ModelMeasure(formula="amount:percentile(p=0.9)")], + ), + ["orders.status", "orders.amount_percentile_p_0_9"], + ) + + async def test_two_local_parametric_variants_are_distinct( + self, engine, + ) -> None: + await _assert_result_keys( + engine, + SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ + ModelMeasure(formula="amount:percentile(p=0.5)"), + ModelMeasure(formula="amount:percentile(p=0.9)"), + ], + ), + [ + "orders.status", + "orders.amount_percentile_p_0_5", + "orders.amount_percentile_p_0_9", + ], + ) + + async def test_cross_model_parametric_retains_kwarg_suffix( + self, engine, + ) -> None: + """The DELIBERATE divergence. + + The deleted legacy enrichment path dropped the kwarg suffix from + cross-model parametric aggregates, which made two variants collide on + one CTE alias. The typed pipeline RETAINS the suffix: correctness over + bit-identical legacy output. This test pins the retention as intended + behavior so a future "restore parity" change has to argue with a test + rather than with a comment. + """ + await _assert_result_keys( + engine, + SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ + ModelMeasure(formula="customers.revenue:percentile(p=0.5)"), + ModelMeasure(formula="customers.revenue:percentile(p=0.9)"), + ], + ), + [ + "orders.status", + "orders.customers.revenue_percentile_p_0_5", + "orders.customers.revenue_percentile_p_0_9", + ], + ) + + +# =========================================================================== +# 7 — Internal identifiers vs public result keys stay separated. +# =========================================================================== + + +class TestIdentifierSeparation: + """``naming.result_key`` (dotted, public) and ``naming.flat_name`` + (``__``-joined, internal inner-stage bind names) own two different forms. + A ``__`` leaking into a public key means the two mixed — the D3 / + shape a past bug produced.""" + + async def test_no_flattened_key_leaks_from_a_joined_query( + self, engine, + ) -> None: + resp = await engine.execute( + SlayerQuery( + source_model="orders", + dimensions=[ + ColumnRef(name="customers.regions.name"), + ColumnRef(name="customers.rev_x2"), + ], + measures=[ModelMeasure(formula="customers.revenue:sum")], + ) + ) + for key in list(resp.columns) + list(resp.data[0].keys()): + assert "__" not in key, f"inner-stage flat name leaked publicly: {key}" + + async def test_public_keys_are_dotted_not_flattened(self, engine) -> None: + """The positive form of the same contract: the dotted key is present + AND its flattened twin is absent.""" + resp = await engine.execute( + SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="customers.regions.name")], + measures=[ModelMeasure(formula="*:count")], + ) + ) + assert "orders.customers.regions.name" in resp.columns + assert "orders.customers__regions__name" not in resp.columns diff --git a/tests/test_dev1744_value_expr.py b/tests/test_dev1744_value_expr.py new file mode 100644 index 00000000..afa46a8c --- /dev/null +++ b/tests/test_dev1744_value_expr.py @@ -0,0 +1,1215 @@ +"""P-G "same construct, same SQL": the single ValueKey→AST renderer. + +``slayer/sql/generator.py`` contains FIVE independent ValueKey renderers (the +issue text says four; the fifth is the outer-wrapper copy): + +* R1 ``_render_value_key_for_filter`` — host WHERE / HAVING +* R2 ``_render_filter_value_key_in_target_scope``— cross-model CTE routed filters +* R3 ``_render_value_key_against_aliases`` — POST-phase / alias space +* R4 ``_render_filter_for_outer_wrapper`` — outer combined WHERE +* R5 ``_render_aggregate_composite_expr`` — AGGREGATE-phase composites + +…plus three literal renderers and three arithmetic composers riding along. They +have drifted, and B5 is the sharpest instance: R1 and R4 emit scalar calls as +``exp.Anonymous`` passthrough while R2/R3/R5 build a typed node and let the +dialect transpile it. The same ``ScalarCallKey`` therefore reaches Postgres as +``IFNULL(...)`` from a filter (Postgres has no ``IFNULL``) and as +``COALESCE(...)`` from a projection. + +This module pins the replacement: ONE ``render_value_key(key, ctx)`` in +``slayer/sql/render/value_expr.py``, parameterised by an explicit +``RenderContext`` and failing closed when the context lacks a facility a key +kind needs. Materialisation stays on ``ScopeFrame`` (P-B) — the renderer +anchors leaves through ``scope.resolve(ref, consumer=...)`` and never by hand. + +Also pinned here: B10 (``ScopeFrame._model_for`` raises instead of silently +substituting the root model) and the aggregation registry that replaces +``_build_agg``'s five dispatch mechanisms. + +Scope note: this PR defines the COMPLETE context API, including +``consumer=``, but migrates only SAME-SCOPE call sites (R1 filters, R5 +composites). R2/R3/R4 migrate in PR 3, which also adds the production-path +proof that ``resolve(consumer=...)`` is exercised — an unused API does not +establish P-B. The ``consumer`` tests here are therefore renderer-level. + +B5 nuance discovered while writing these tests: "uppercase + dialect transpile" +alone is NOT the whole policy. Building ``exp.func("LOG10", x)`` yields a +generic ``exp.Log(10, x)`` that re-emits as ``LOG(10, x)`` — which is wrong for +the dialects that have a native single-arg ``LOG10`` +(``SqlDialect.should_use_native_log``). The generator already owns that rewrite +(``_rewrite_log_aliases``) but applies it only inside ``_parse`` / +``_parse_predicate``, never to AST-built calls, so R3 gets ``log10`` WRONG today +while R1's Anonymous passthrough gets it right. The unified policy is therefore +transpile-then-log-rewrite, which is the only form that is correct for both +``ifnull`` and ``log10``. +""" + +from __future__ import annotations + +import os +import sqlite3 +import tempfile +from decimal import Decimal +from typing import AsyncIterator, Optional + +import pytest +from sqlglot import exp + +from slayer.core.enums import BUILTIN_AGGREGATIONS, DataType +from slayer.core.keys import ( + AggregateKey, + ArithmeticKey, + BetweenKey, + ColumnKey, + ColumnSqlKey, + InKey, + LiteralKey, + ScalarCallKey, + StarKey, + TimeTruncKey, + TransformKey, +) +from slayer.core.models import ( + Aggregation, + Column, + DatasourceConfig, + ModelJoin, + ModelMeasure, + SlayerModel, +) +from slayer.core.query import ColumnRef, SlayerQuery +from slayer.engine.query_engine import SlayerQueryEngine +from slayer.engine.source_bundle import ResolvedSourceBundle +from slayer.sql.dialects import get_dialect +from slayer.sql.naming import AliasAllocator +from slayer.sql.scope import ScopeFrame +from slayer.storage.yaml_storage import YAMLStorage + + +# =========================================================================== +# Models + scope construction (mirrors tests/test_scope.py's idiom). +# =========================================================================== + + +def _regions() -> SlayerModel: + return SlayerModel( + name="regions", sql_table="regions", data_source="test", + columns=[ + Column(name="id", sql="id", type=DataType.DOUBLE, primary_key=True), + Column(name="name", sql="name", type=DataType.TEXT), + Column(name="population", sql="population", type=DataType.DOUBLE), + ], + ) + + +def _customers() -> SlayerModel: + return SlayerModel( + name="customers", sql_table="customers", data_source="test", + columns=[ + Column(name="id", sql="id", type=DataType.DOUBLE, primary_key=True), + Column(name="region_id", sql="region_id", type=DataType.DOUBLE), + Column(name="balance", sql="balance", type=DataType.DOUBLE), + ], + joins=[ModelJoin(target_model="regions", join_pairs=[["region_id", "id"]])], + ) + + +def _orders() -> SlayerModel: + return SlayerModel( + name="orders", sql_table="orders", data_source="test", + columns=[ + Column(name="id", sql="id", type=DataType.DOUBLE, primary_key=True), + Column(name="customer_id", sql="customer_id", type=DataType.DOUBLE), + Column(name="amount", sql="amount", type=DataType.DOUBLE), + Column(name="label", sql="label", type=DataType.TEXT), + Column(name="created_at", sql="created_at", type=DataType.TIMESTAMP), + Column(name="net", sql="amount - 1", type=DataType.DOUBLE), + ], + joins=[ModelJoin(target_model="customers", join_pairs=[["customer_id", "id"]])], + ) + + +def _scope( + host: Optional[SlayerModel] = None, + *others: SlayerModel, + dialect: str = "postgres", + allocator: Optional[AliasAllocator] = None, +) -> ScopeFrame: + host = host or _orders() + others = others or (_customers(), _regions()) + alloc = allocator or AliasAllocator() + bundle = ResolvedSourceBundle( + source_model=host, referenced_models=[host, *others], + ) + return ScopeFrame( + scope_id=alloc.next_scope_id(host.name), + root_model=host, root_relation=host.name, + bundle=bundle, dialect=get_dialect(dialect), allocator=alloc, + ) + + +def _filter_ctx(dialect: str = "postgres", **kw): + """A RenderContext carrying the FILTER facility group (R1's call family).""" + from slayer.sql.render.value_expr import FilterFacilities, RenderContext + + scope = _scope(dialect=dialect) + return RenderContext( + scope=scope, + dialect=scope.dialect, + filters=FilterFacilities(**kw), + ) + + +def _composite_ctx(dialect: str = "postgres", **kw): + """A RenderContext carrying the COMPOSITE facility group (R5's family).""" + from slayer.sql.render.value_expr import CompositeFacilities, RenderContext + + scope = _scope(dialect=dialect) + return RenderContext( + scope=scope, + dialect=scope.dialect, + composites=CompositeFacilities(**kw), + ) + + +def _sql(expr: exp.Expression, dialect: str = "postgres") -> str: + return expr.sql(dialect=dialect) + + +# =========================================================================== +# B10 — ScopeFrame._model_for raises on an unknown model. +# =========================================================================== + + +class TestB10UnknownModelRaises: + """Today ``_model_for`` ends in ``or self.root_model``: a ``ColumnSqlKey`` + naming a model absent from the bundle silently expands the ROOT model's + derived SQL instead. That turns a wiring bug into a wrong answer — the + query runs and returns numbers computed from the wrong model.""" + + def test_unknown_model_in_columnsqlkey_raises(self) -> None: + from slayer.core.errors import UnknownReferenceError + + scope = _scope() + with pytest.raises(UnknownReferenceError): + scope.resolve( + ColumnSqlKey(model="not_in_bundle", column_name="net"), + ) + + def test_error_names_the_missing_model(self) -> None: + """The message must be actionable: which model was asked for, what the + scope root is, and what the bundle actually knows.""" + from slayer.core.errors import UnknownReferenceError + + scope = _scope() + with pytest.raises(UnknownReferenceError) as excinfo: + scope.resolve( + ColumnSqlKey(model="not_in_bundle", column_name="net"), + ) + message = str(excinfo.value) + assert "not_in_bundle" in message + assert "orders" in message + + def test_known_models_still_resolve(self) -> None: + """Parity guard: the root model and every bundle member keep working — + B10 removes only the silent FALLBACK, not the lookup.""" + scope = _scope() + expr = scope.resolve(ColumnSqlKey(model="orders", column_name="net")) + assert "amount" in _sql(expr) + + def test_root_model_lookup_does_not_consult_the_bundle(self) -> None: + """A scope whose root model is not listed in ``referenced_models`` must + still resolve its own root — the first branch of ``_model_for``.""" + host = _orders() + alloc = AliasAllocator() + bundle = ResolvedSourceBundle( + source_model=host, referenced_models=[_customers(), _regions()], + ) + scope = ScopeFrame( + scope_id=alloc.next_scope_id(host.name), + root_model=host, root_relation=host.name, + bundle=bundle, dialect=get_dialect("postgres"), allocator=alloc, + ) + expr = scope.resolve(ColumnSqlKey(model="orders", column_name="net")) + assert "amount" in _sql(expr) + + +# =========================================================================== +# The RenderContext API. +# =========================================================================== + + +class TestRenderContextApi: + def test_context_holds_real_production_objects(self) -> None: + """Pydantic v2 + a ``ScopeFrame`` / dialect strategy / sqlglot nodes + needs ``arbitrary_types_allowed``; constructing with the real objects + (not stubs) is what proves the config is right.""" + from slayer.sql.render.value_expr import RenderContext + + scope = _scope() + ctx = RenderContext(scope=scope, dialect=scope.dialect) + assert ctx.scope is scope + assert ctx.consumer is None + assert ctx.filters is None and ctx.composites is None + assert ctx.aliases is None + + def test_consumer_defaults_to_none_and_is_accepted(self) -> None: + """The P-B seam exists in PR 1 even though its production callers + arrive in PR 3.""" + from slayer.sql.render.value_expr import RenderContext + + producer, consumer = _scope(), _scope() + ctx = RenderContext( + scope=producer, consumer=consumer, dialect=producer.dialect, + ) + assert ctx.consumer is consumer + + @pytest.mark.parametrize( + "label,key", + [ + ("local_column", ColumnKey(leaf="amount")), + ("joined_column", ColumnKey(path=("customers",), leaf="balance")), + ("derived_column", ColumnSqlKey(model="orders", column_name="net")), + ], + ) + def test_consumer_routes_column_like_leaves_through_materialization( + self, label, key, + ) -> None: + """With a consumer named, EVERY column-like leaf must come back as a + BARE materialisation alias and be projected in the PRODUCING scope — + the single Law-2 mechanism, not a second one grown inside the renderer. + + Parametrised over all three column-like kinds because a renderer that + special-cases one of them would otherwise slip through.""" + from slayer.sql.render.value_expr import RenderContext, render_value_key + + producer, consumer = _scope(), _scope() + ctx = RenderContext( + scope=producer, consumer=consumer, dialect=producer.dialect, + ) + out = render_value_key(key, ctx) + assert isinstance(out, exp.Column), f"{label}: got {type(out).__name__}" + assert out.table == "", f"{label}: expected a bare alias, got {_sql(out)}" + assert len(producer.materializations) == 1 + assert producer.materializations[0].alias == _sql(out) + + def test_materializations_apply_to_the_producing_select(self) -> None: + """The other half of the P-B contract: what the renderer records must + actually be projectable via ``apply_materializations``, so the consumer's + bare alias resolves to a real column of the producing SELECT.""" + from slayer.sql.render.value_expr import RenderContext, render_value_key + + producer, consumer = _scope(), _scope() + ctx = RenderContext( + scope=producer, consumer=consumer, dialect=producer.dialect, + ) + alias = _sql(render_value_key(ColumnKey(leaf="amount"), ctx)) + select = producer.apply_materializations( + exp.Select().from_(exp.to_table("orders")), + ) + assert alias in [p.alias_or_name for p in select.expressions], ( + f"{alias!r} not projected by the producing scope: " + f"{select.sql(dialect='postgres')}" + ) + + def test_materialization_dedups_within_a_scope(self) -> None: + """Two renders of the same key across the same boundary share ONE + ``_val_`` — the dedup key is the producing scope + anchored AST + + dialect, and the renderer must not defeat it by re-anchoring.""" + from slayer.sql.render.value_expr import RenderContext, render_value_key + + producer, consumer = _scope(), _scope() + ctx = RenderContext( + scope=producer, consumer=consumer, dialect=producer.dialect, + ) + a = render_value_key(ColumnKey(leaf="amount"), ctx) + b = render_value_key(ColumnKey(leaf="amount"), ctx) + assert _sql(a) == _sql(b) + assert len(producer.materializations) == 1 + + def test_join_paths_register_as_a_side_effect_of_rendering(self) -> None: + """P-A: join discovery is a side effect of rendering, never a separate + pass. Rendering a joined leaf must register the crossed path on the + scope without the caller asking.""" + from slayer.sql.render.value_expr import RenderContext, render_value_key + + scope = _scope() + ctx = RenderContext(scope=scope, dialect=scope.dialect) + render_value_key( + ColumnKey(path=("customers", "regions"), leaf="name"), ctx, + ) + assert scope.join_paths.as_list() == [ + ("customers",), ("customers", "regions"), + ] + + def test_missing_facility_fails_closed(self) -> None: + """A key kind that needs a facility the context lacks must RAISE, not + silently degrade. Silent degradation is how the five copies drifted in + the first place.""" + from slayer.core.errors import RenderContextMissingFacilityError + from slayer.sql.render.value_expr import RenderContext, render_value_key + + scope = _scope() + bare = RenderContext(scope=scope, dialect=scope.dialect) + # A POST-phase transform can only be rendered against already- + # materialised aliases, which live in the ALIAS facility group. + key = TransformKey( + op="time_shift", + input=AggregateKey(source=ColumnKey(leaf="amount"), agg="sum"), + ) + with pytest.raises(RenderContextMissingFacilityError): + render_value_key(key, bare) + + def test_missing_facility_error_names_the_key_and_facility(self) -> None: + from slayer.core.errors import RenderContextMissingFacilityError + from slayer.sql.render.value_expr import RenderContext, render_value_key + + scope = _scope() + bare = RenderContext(scope=scope, dialect=scope.dialect) + key = TransformKey( + op="time_shift", + input=AggregateKey(source=ColumnKey(leaf="amount"), agg="sum"), + ) + with pytest.raises(RenderContextMissingFacilityError) as excinfo: + render_value_key(key, bare) + message = str(excinfo.value) + assert "TransformKey" in message + assert "aliases" in message.lower(), ( + f"the error must name the MISSING facility, not just the key: " + f"{message!r}" + ) + + def test_aggregate_without_composite_facilities_fails_closed(self) -> None: + """Fail-closed is asserted per facility group, not once. + + A renderer could easily fail closed on the alias group (the branch the + test above covers) while silently degrading on the composite group — + which is the drift mode this whole PR exists to prevent. An + ``AggregateKey`` needs the composite facilities (rn-suffix maps, + resolved agg kwargs, composite alias map) to render faithfully. + """ + from slayer.core.errors import RenderContextMissingFacilityError + from slayer.sql.render.value_expr import RenderContext, render_value_key + + scope = _scope() + bare = RenderContext(scope=scope, dialect=scope.dialect) + key = AggregateKey(source=ColumnKey(leaf="amount"), agg="first") + with pytest.raises(RenderContextMissingFacilityError) as excinfo: + render_value_key(key, bare) + assert "composite" in str(excinfo.value).lower(), str(excinfo.value) + + def test_transform_key_renders_when_alias_facilities_are_supplied( + self, + ) -> None: + """The positive half of the fail-closed pair. + + PR 1 defines the COMPLETE context API even though the production call + sites for alias-space rendering migrate in PR 3, so a ``TransformKey`` + WITH its facility must render here — otherwise "fails closed" would be + indistinguishable from "not implemented". + """ + from slayer.sql.render.value_expr import ( + AliasFacilities, + RenderContext, + render_value_key, + ) + + scope = _scope() + agg = AggregateKey(source=ColumnKey(leaf="amount"), agg="sum") + key = TransformKey(op="time_shift", input=agg) + ctx = RenderContext( + scope=scope, + dialect=scope.dialect, + aliases=AliasFacilities( + slot_id_by_key={key: "s1"}, + available_alias_by_slot_id={"s1": "orders.amount_sum_shifted"}, + ), + ) + out = render_value_key(key, ctx) + assert "amount_sum_shifted" in _sql(out), _sql(out) + + +# =========================================================================== +# Coverage of the whole ValueKey union. +# =========================================================================== + + +class TestRendersEveryKeyKind: + """The union is closed (11 members). One renderer means every member is + handled in one place — an unhandled kind must raise, never fall through to + a bare ``None`` or a stringified repr.""" + + def test_local_column_key(self) -> None: + from slayer.sql.render.value_expr import render_value_key + + out = render_value_key(ColumnKey(leaf="amount"), _filter_ctx()) + assert _sql(out) == "orders.amount" + + def test_joined_column_key_anchors_at_the_path_alias(self) -> None: + from slayer.sql.render.value_expr import render_value_key + + out = render_value_key( + ColumnKey(path=("customers",), leaf="balance"), _filter_ctx(), + ) + assert _sql(out) == "customers.balance" + + def test_multi_hop_column_key(self) -> None: + from slayer.sql.render.value_expr import render_value_key + + out = render_value_key( + ColumnKey(path=("customers", "regions"), leaf="name"), + _filter_ctx(), + ) + assert _sql(out) == "customers__regions.name" + + def test_column_sql_key_expands_the_derived_expression(self) -> None: + """Exact SQL, not a substring check: ``net`` is ``amount - 1``, and the + expansion must be anchored at the scope root.""" + from slayer.sql.render.value_expr import render_value_key + + out = render_value_key( + ColumnSqlKey(model="orders", column_name="net"), _filter_ctx(), + ) + assert _sql(out) == "orders.amount - 1" + + def test_time_trunc_key(self) -> None: + """Exact per-dialect SQL — a substring check would accept a truncation + at the wrong granularity or over the wrong column.""" + from slayer.sql.render.value_expr import render_value_key + + key = TimeTruncKey( + column=ColumnKey(leaf="created_at"), granularity="month", + ) + out = render_value_key(key, _filter_ctx("postgres")) + assert _sql(out, "postgres") == "DATE_TRUNC('MONTH', orders.created_at)" + + def test_literal_key_variants(self) -> None: + from slayer.sql.render.value_expr import render_value_key + + ctx = _filter_ctx() + assert _sql(render_value_key(LiteralKey(value=Decimal(3)), ctx)) == "3" + assert _sql(render_value_key(LiteralKey(value="x"), ctx)) == "'x'" + assert _sql( + render_value_key(LiteralKey(value=None), ctx) + ).upper() == "NULL" + + def test_star_key(self) -> None: + from slayer.sql.render.value_expr import render_value_key + + out = render_value_key(StarKey(), _filter_ctx()) + assert isinstance(out, exp.Star) + + def test_arithmetic_key(self) -> None: + from slayer.sql.render.value_expr import render_value_key + + out = render_value_key( + ArithmeticKey( + op="+", + operands=(ColumnKey(leaf="amount"), LiteralKey(value=Decimal(1))), + ), + _filter_ctx(), + ) + assert _sql(out) == "orders.amount + 1" + + def test_comparison_arithmetic_key(self) -> None: + from slayer.sql.render.value_expr import render_value_key + + out = render_value_key( + ArithmeticKey( + op=">", + operands=(ColumnKey(leaf="amount"), LiteralKey(value=Decimal(5))), + ), + _filter_ctx(), + ) + assert _sql(out) == "orders.amount > 5" + + def test_between_key(self) -> None: + from slayer.sql.render.value_expr import render_value_key + + out = render_value_key( + BetweenKey( + column=ColumnKey(leaf="amount"), + low=LiteralKey(value=Decimal(1)), + high=LiteralKey(value=Decimal(9)), + ), + _filter_ctx(), + ) + assert _sql(out) == "orders.amount BETWEEN 1 AND 9" + + def test_in_key(self) -> None: + from slayer.sql.render.value_expr import render_value_key + + out = render_value_key( + InKey( + column=ColumnKey(leaf="label"), + values=(LiteralKey(value="a"), LiteralKey(value="b")), + ), + _filter_ctx(), + ) + assert _sql(out) == "orders.label IN ('a', 'b')" + + def test_negated_in_key(self) -> None: + from slayer.sql.render.value_expr import render_value_key + + out = render_value_key( + InKey( + column=ColumnKey(leaf="label"), + values=(LiteralKey(value="a"),), + negated=True, + ), + _filter_ctx(), + ) + # Exact form: a bare "NOT" substring would also match e.g. an + # IS NOT NULL wrapper that got the predicate wrong. + assert _sql(out) == "NOT orders.label IN ('a')" + + def test_local_aggregate_key(self) -> None: + from slayer.sql.render.value_expr import render_value_key + + out = render_value_key( + AggregateKey(source=ColumnKey(leaf="amount"), agg="sum"), + _composite_ctx(), + ) + assert _sql(out) == "SUM(orders.amount)" + + def test_star_count_aggregate_key(self) -> None: + from slayer.sql.render.value_expr import render_value_key + + out = render_value_key( + AggregateKey(source=StarKey(), agg="count"), _composite_ctx(), + ) + assert _sql(out) == "COUNT(*)" + + def test_unhandled_kind_raises_notimplementederror(self) -> None: + """Fail closed on anything outside the union rather than returning a + stringified repr into the SQL. + + Pinned to ONE exact exception type (matching the existing generator + renderers' convention) and to a message naming the offending type — + accepting a tuple of types would let an incidental TypeError from + somewhere else inside the renderer satisfy this test. + """ + from slayer.sql.render.value_expr import render_value_key + + with pytest.raises(NotImplementedError) as excinfo: + render_value_key(object(), _filter_ctx()) # type: ignore[arg-type] + assert "object" in str(excinfo.value) + + +# =========================================================================== +# B5 — one ScalarCall render policy everywhere. +# =========================================================================== + + +# (function name, args, expected postgres SQL, expected tsql SQL) +# +# The expectations encode the UNIFIED policy: uppercase → typed sqlglot node → +# dialect transpile → log-alias rewrite. Where that differs from what R1 emits +# today, the difference IS B5. +# +# Identifiers are unquoted here: bracket / double-quote wrapping is a separate +# emit-time pass over the finished statement, not part of value rendering. +_SCALAR_MATRIX = [ + # R1 today: IFNULL(...) — invalid on Postgres, which has no IFNULL. + ("ifnull", (ColumnKey(leaf="amount"), LiteralKey(value=Decimal(0))), + "COALESCE(orders.amount, 0)", "COALESCE(orders.amount, 0)"), + # R1 today: CONCAT(...) verbatim on every dialect; the operator differs. + ("concat", (ColumnKey(leaf="label"), LiteralKey(value="x")), + "orders.label || 'x'", "orders.label + 'x'"), + # R1 today: LENGTH(...) on T-SQL, which spells it LEN. + ("length", (ColumnKey(leaf="label"),), + "LENGTH(orders.label)", "LEN(orders.label)"), +] + + +class TestB5ScalarCallPolicy: + @pytest.mark.parametrize( + "name,args,expected_pg,expected_tsql", + _SCALAR_MATRIX, + ids=[m[0] for m in _SCALAR_MATRIX], + ) + def test_scalar_calls_transpile_per_dialect( + self, name, args, expected_pg, expected_tsql, + ) -> None: + from slayer.sql.render.value_expr import render_value_key + + key = ScalarCallKey(name=name, args=args) + assert _sql( + render_value_key(key, _filter_ctx("postgres")), "postgres", + ) == expected_pg + assert _sql( + render_value_key(key, _filter_ctx("tsql")), "tsql", + ) == expected_tsql + + def test_ifnull_never_reaches_postgres_unmapped(self) -> None: + """The headline B5 bug, stated as the invariant rather than as an exact + string: Postgres has no ``IFNULL``, so emitting it is broken SQL.""" + from slayer.sql.render.value_expr import render_value_key + + key = ScalarCallKey( + name="ifnull", + args=(ColumnKey(leaf="amount"), LiteralKey(value=Decimal(0))), + ) + out = _sql(render_value_key(key, _filter_ctx("postgres")), "postgres") + assert "IFNULL" not in out.upper(), out + + def test_log10_keeps_the_native_single_arg_alias(self) -> None: + """The other half of the policy, and the reason "transpile" alone is + the wrong rule. + + ``exp.func("LOG10", x)`` normalises to a generic ``Log(10, x)`` that + re-emits as ``LOG(10, x)``. Every Tier-1/2 dialect but Oracle has a + native single-arg ``LOG10``, which is why the generator carries + ``_rewrite_log_aliases``. Applying transpile WITHOUT that rewrite would + regress ``log10`` — so the unified renderer must apply both. + """ + from slayer.sql.render.value_expr import render_value_key + + key = ScalarCallKey(name="log10", args=(ColumnKey(leaf="amount"),)) + out = _sql(render_value_key(key, _filter_ctx("postgres")), "postgres") + assert out.upper().startswith("LOG10("), out + + def test_round_keeps_the_dev1576_postgres_cast(self) -> None: + """Parity guard: two-arg ROUND on Postgres needs the numeric cast, and + it is the ONE scalar call R1 already routed through the typed path. + Unifying must not lose it.""" + from slayer.sql.render.value_expr import render_value_key + + key = ScalarCallKey( + name="round", + args=(ColumnKey(leaf="amount"), LiteralKey(value=Decimal(0))), + ) + out = _sql(render_value_key(key, _filter_ctx("postgres")), "postgres") + assert "CAST" in out.upper() and "DECIMAL" in out.upper(), out + + def test_like_stays_the_sql_operator(self) -> None: + """``like(value, pattern)`` is the one allowlist member that is an + OPERATOR, not a function call. Both legacy paths special-case it; the + unified renderer keeps that.""" + from slayer.sql.render.value_expr import render_value_key + + key = ScalarCallKey( + name="like", + args=(ColumnKey(leaf="label"), LiteralKey(value="x%")), + ) + out = render_value_key(key, _filter_ctx("postgres")) + assert isinstance(out, exp.Like) + assert _sql(out) == "orders.label LIKE 'x%'" + + def test_nested_scalar_calls_use_one_policy_throughout(self) -> None: + """The policy applies at every depth — a nested call must not fall back + to the passthrough branch.""" + from slayer.sql.render.value_expr import render_value_key + + key = ScalarCallKey( + name="ifnull", + args=( + ScalarCallKey(name="length", args=(ColumnKey(leaf="label"),)), + LiteralKey(value=Decimal(0)), + ), + ) + out = _sql(render_value_key(key, _filter_ctx("tsql")), "tsql") + assert "COALESCE" in out.upper() and "LEN(" in out.upper(), out + assert "IFNULL" not in out.upper() and "LENGTH" not in out.upper(), out + + +class TestPGSameConstructSameSql: + """P-G proper: a given ValueKey renders identically wherever it appears. + + The contexts differ (filter facilities vs composite facilities) but the + rendering POLICY must not branch on them. Any divergence here is the class + of bug the five copies produced. + """ + + _KEYS = [ + ("column", ColumnKey(leaf="amount")), + ("joined_column", ColumnKey(path=("customers",), leaf="balance")), + ("derived_column", ColumnSqlKey(model="orders", column_name="net")), + ("literal", LiteralKey(value=Decimal(7))), + ("scalar_call", ScalarCallKey( + name="ifnull", + args=(ColumnKey(leaf="amount"), LiteralKey(value=Decimal(0))), + )), + ("nested_scalar_call", ScalarCallKey( + name="upper", + args=(ScalarCallKey(name="trim", args=(ColumnKey(leaf="label"),)),), + )), + ("arithmetic", ArithmeticKey( + op="*", + operands=(ColumnKey(leaf="amount"), LiteralKey(value=Decimal(2))), + )), + ("in_predicate", InKey( + column=ColumnKey(leaf="label"), values=(LiteralKey(value="a"),), + )), + ("between_predicate", BetweenKey( + column=ColumnKey(leaf="amount"), + low=LiteralKey(value=Decimal(1)), + high=LiteralKey(value=Decimal(2)), + )), + ] + + @pytest.mark.parametrize("label,key", _KEYS, ids=[k[0] for k in _KEYS]) + @pytest.mark.parametrize("dialect", ["postgres", "sqlite", "tsql", "bigquery"]) + def test_same_key_same_sql_across_contexts( + self, label, key, dialect, + ) -> None: + from slayer.sql.render.value_expr import render_value_key + + in_filter = _sql( + render_value_key(key, _filter_ctx(dialect)), dialect, + ) + in_composite = _sql( + render_value_key(key, _composite_ctx(dialect)), dialect, + ) + assert in_filter == in_composite, ( + f"{label} renders differently by context on {dialect}: " + f"filter={in_filter!r} composite={in_composite!r}" + ) + + +# =========================================================================== +# The aggregation registry (replaces _build_agg's five dispatch mechanisms). +# =========================================================================== + + +class TestAggregationRegistry: + """``_build_agg`` reaches its builders five different ways: a hardcoded + name pair for first/last, ``_AGG_FUNCTION_MAP`` plus a SECOND inline + ``agg_class_map``, a frozenset membership test for the stat aggregates, + name equality for the dialect-hook aggregates, and a formula-template + fallback. One registry table replaces all five.""" + + # Frozen INDEPENDENTLY of both BUILTIN_AGGREGATIONS and the registry, one + # per former dispatch mechanism, so the coverage assertion cannot become + # tautological if the registry were ever used to build the enum (or vice + # versa). Each name must resolve AND be reachable through the old + # mechanism's builder. + _REQUIRED_BY_MECHANISM = { + "first_last_case": ["first", "last"], + "agg_function_map": ["count", "sum", "avg", "min", "max"], + "stat_frozenset": [ + "stddev_samp", "stddev_pop", "var_samp", "var_pop", + "corr", "covar_samp", "covar_pop", + ], + "dialect_hook": [ + "percentile", "median", "count_distinct", "count_distinct_approx", + ], + "formula_template": ["weighted_avg"], + } + + def test_every_builtin_resolves(self) -> None: + from slayer.sql.render.aggregates import resolve_agg_entry + + for name in sorted(BUILTIN_AGGREGATIONS): + entry = resolve_agg_entry(name) + assert entry is not None, name + assert entry.name == name + + @pytest.mark.parametrize( + "mechanism,names", + sorted(_REQUIRED_BY_MECHANISM.items()), + ids=sorted(_REQUIRED_BY_MECHANISM), + ) + def test_each_former_dispatch_mechanism_is_represented( + self, mechanism, names, + ) -> None: + """One registry table must subsume all five mechanisms — an + implementation that only ported the easy ``_AGG_FUNCTION_MAP`` entries + would still pass ``test_every_builtin_resolves`` if the enum happened + to be small.""" + from slayer.sql.render.aggregates import resolve_agg_entry + + for name in names: + entry = resolve_agg_entry(name) + assert entry is not None, f"{mechanism}: {name} does not resolve" + assert entry.name == name + + def test_required_names_are_really_builtins(self) -> None: + """Guard on the frozen table's own premise, so a rename in the enum + surfaces here rather than silently shrinking coverage.""" + for names in self._REQUIRED_BY_MECHANISM.values(): + for name in names: + assert name in BUILTIN_AGGREGATIONS, name + + def test_unknown_aggregation_raises(self) -> None: + from slayer.sql.render.aggregates import resolve_agg_entry + + with pytest.raises(ValueError): + resolve_agg_entry("definitely_not_an_aggregation") + + def test_windowable_flags_are_exact(self) -> None: + """Only ``sum`` and ``avg`` are windowable today — that is precisely + what ``stage_planner`` gates on, and the registry must agree with it + rather than restating it.""" + from slayer.sql.render.aggregates import resolve_agg_entry + + assert resolve_agg_entry("sum").windowable is True + assert resolve_agg_entry("avg").windowable is True + for name in ("count", "min", "max", "median", "percentile", "first"): + assert resolve_agg_entry(name).windowable is False, name + + def test_window_agg_class_replaces_the_hardcode(self) -> None: + from slayer.sql.render.aggregates import window_agg_class + + assert window_agg_class("sum") is exp.Sum + assert window_agg_class("avg") is exp.Avg + + def test_non_windowable_aggregation_fails_closed(self) -> None: + """The generator's windowed path currently reads + ``exp.Sum if plan.agg == "sum" else exp.Avg`` — a silent catch-all that + renders ANY other aggregation as AVG. It is unreachable through the + planner today, which is exactly why it would stay silently wrong. + + Approved divergence: it raises instead. + """ + from slayer.sql.render.aggregates import window_agg_class + + for name in ("median", "count", "min", "max", "percentile"): + with pytest.raises(ValueError): + window_agg_class(name) + + +# =========================================================================== +# End-to-end: the migrated call-site families keep working. +# =========================================================================== + + +async def _e2e_engine(*, dialect: str = "sqlite") -> SlayerQueryEngine: + d = tempfile.mkdtemp() + db_path = os.path.join(d, "ve.db") + con = sqlite3.connect(db_path) + cur = con.cursor() + cur.execute( + "CREATE TABLE orders (id INTEGER PRIMARY KEY, status TEXT, " + "amount REAL, disc REAL, qty REAL, created_at TEXT)" + ) + cur.executemany( + "INSERT INTO orders VALUES (?,?,?,?,?,?)", + [ + (1, "new", 10.0, None, 2.0, "2024-01-01"), + (2, "new", 20.0, 5.0, 4.0, "2024-02-01"), + (3, "old", 30.0, None, 1.0, "2024-01-15"), + ], + ) + con.commit() + con.close() + + storage = YAMLStorage(base_dir=os.path.join(d, "store")) + await storage.save_datasource( + DatasourceConfig(name="prod", type=dialect, database=db_path) + ) + await storage.save_model( + SlayerModel( + name="orders", + sql_table="orders", + data_source="prod", + # first/last needs a resolvable ranking time column. + default_time_dimension="created_at", + columns=[ + Column(name="id", type=DataType.INT, primary_key=True), + Column(name="status", type=DataType.TEXT), + Column(name="amount", type=DataType.DOUBLE), + Column(name="disc", type=DataType.DOUBLE), + Column(name="qty", type=DataType.DOUBLE), + Column(name="created_at", type=DataType.TIMESTAMP), + ], + # Exercises the formula-template dispatch mechanism through the + # registry — a CUSTOM aggregation, not a builtin. + aggregations=[ + Aggregation(name="sum_sq", formula="SUM({value} * {value})"), + ], + ) + ) + return SlayerQueryEngine(storage=storage) + + +@pytest.fixture +async def e2e() -> AsyncIterator[SlayerQueryEngine]: + yield await _e2e_engine() + + +class TestMigratedCallSitesEndToEnd: + async def test_scalar_call_in_a_row_filter_executes(self, e2e) -> None: + """R1's family (host WHERE) after migration. ``disc`` is NULL for two + rows, so ``ifnull(disc, 0) > 1`` must keep exactly the one row where + ``disc = 5``.""" + resp = await e2e.execute( + SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ModelMeasure(formula="*:count")], + filters=["ifnull(disc, 0) > 1"], + ) + ) + assert len(resp.data) == 1 + assert resp.data[0]["orders.status"] == "new" + assert resp.data[0]["orders._count"] == 1 + + async def test_scalar_call_composite_projection_executes(self, e2e) -> None: + """R5's family (AGGREGATE-phase composite): a scalar call WRAPPING an + aggregate. Sums: new = 30, old = 30.""" + resp = await e2e.execute( + SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ModelMeasure(formula="ifnull(amount:sum, 0)", name="m")], + ) + ) + by_status = {r["orders.status"]: r["orders.m"] for r in resp.data} + assert by_status == {"new": 30.0, "old": 30.0} + + async def test_arithmetic_composite_of_two_aggregates(self, e2e) -> None: + """The composite family's core shape: arithmetic over two aggregates + must stay ONE inline expression, not two materialised slots.""" + resp = await e2e.execute( + SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ + ModelMeasure(formula="amount:sum - qty:sum", name="d"), + ], + ) + ) + by_status = {r["orders.status"]: r["orders.d"] for r in resp.data} + assert by_status == {"new": 24.0, "old": 29.0} + + async def test_custom_formula_aggregation_still_dispatches( + self, e2e, + ) -> None: + """The registry must keep the formula-template mechanism reachable for + aggregations that are NOT builtins — a closed builtin table would drop + them. new = 10² + 20² = 500; old = 30² = 900.""" + resp = await e2e.execute( + SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ModelMeasure(formula="amount:sum_sq", name="ss")], + ) + ) + by_status = {r["orders.status"]: r["orders.ss"] for r in resp.data} + assert by_status == {"new": 500.0, "old": 900.0} + + async def test_having_filter_over_an_aggregate_executes(self, e2e) -> None: + """AGGREGATE-phase filter (HAVING) — the other half of R1's family.""" + resp = await e2e.execute( + SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ModelMeasure(formula="qty:sum", name="q")], + filters=["qty:sum > 5"], + ) + ) + assert [r["orders.status"] for r in resp.data] == ["new"] + assert resp.data[0]["orders.q"] == 6.0 + + async def test_first_last_composite_call_site_executes(self, e2e) -> None: + """R5's SECOND production call site (``_build_first_last_base_select``, + generator.py:3741) — reached only by a first/last measure, which builds + its own ranked base SELECT rather than the ordinary composite one. + + Ordered by ``id``: new -> last amount 20, old -> 30.""" + resp = await e2e.execute( + SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ModelMeasure(formula="amount:last", name="la")], + ) + ) + by_status = {r["orders.status"]: r["orders.la"] for r in resp.data} + assert by_status == {"new": 20.0, "old": 30.0} + + async def test_scalar_call_inside_a_first_last_composite(self, e2e) -> None: + """…and the same call site carrying a SCALAR CALL, so a legacy copy + surviving on that route could not pass unnoticed.""" + resp = await e2e.execute( + SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ + ModelMeasure(formula="ifnull(amount:last, 0)", name="la"), + ], + ) + ) + by_status = {r["orders.status"]: r["orders.la"] for r in resp.data} + assert by_status == {"new": 20.0, "old": 30.0} + + +class TestOuterWrapperAndShiftedCteFamilies: + """The two remaining migrated/patched routes, which the ordinary WHERE and + HAVING tests do not reach. + + * R4 ``_render_filter_for_outer_wrapper`` — the outer combined + SELECT. R4 is NOT migrated in PR 1 (it is cross-scope, and moves in + PR 3), but B5 says "everywhere", so its scalar branch IS patched here. + * R1's shifted-CTE WHERE call site (generator.py:7468) — reached only by a + ``time_shift`` transform, never by a plain host filter. + """ + + async def _engine(self, *, dialect: str = "sqlite") -> SlayerQueryEngine: + d = tempfile.mkdtemp() + db_path = os.path.join(d, "routes.db") + con = sqlite3.connect(db_path) + cur = con.cursor() + cur.execute( + "CREATE TABLE regions (id INTEGER PRIMARY KEY, tier TEXT)" + ) + cur.executemany( + "INSERT INTO regions VALUES (?,?)", [(1, "gold"), (2, "silver")], + ) + cur.execute( + "CREATE TABLE orders (id INTEGER PRIMARY KEY, region_id INTEGER, " + "status TEXT, amount REAL, disc REAL, created_at TEXT)" + ) + cur.executemany( + "INSERT INTO orders VALUES (?,?,?,?,?,?)", + [ + (1, 1, "new", 100.0, None, "2024-01-15"), + (2, 1, "new", 200.0, 5.0, "2024-02-15"), + (3, 2, "old", 300.0, None, "2024-01-20"), + (4, 2, "old", 400.0, 7.0, "2024-02-20"), + ], + ) + con.commit() + con.close() + + storage = YAMLStorage(base_dir=os.path.join(d, "store")) + await storage.save_datasource( + DatasourceConfig(name="prod", type=dialect, database=db_path) + ) + await storage.save_model( + SlayerModel( + name="regions", sql_table="regions", data_source="prod", + columns=[ + Column(name="id", type=DataType.INT, primary_key=True), + Column(name="tier", type=DataType.TEXT), + ], + ) + ) + await storage.save_model( + SlayerModel( + name="orders", sql_table="orders", data_source="prod", + default_time_dimension="created_at", + columns=[ + Column(name="id", type=DataType.INT, primary_key=True), + Column(name="region_id", type=DataType.INT), + Column(name="status", type=DataType.TEXT), + Column(name="amount", type=DataType.DOUBLE), + Column(name="disc", type=DataType.DOUBLE), + Column(name="created_at", type=DataType.TIMESTAMP), + # Join-crossing Column.filter => filtered-local + # isolation => the outer combined-SELECT wrapper (R4). + Column( + name="gold_amount", sql="amount", + type=DataType.DOUBLE, + filter="regions.tier = 'gold'", + ), + ], + joins=[ + ModelJoin( + target_model="regions", + join_pairs=[["region_id", "id"]], + ), + ], + ) + ) + return SlayerQueryEngine(storage=storage) + + async def test_outer_wrapper_filter_over_an_isolated_aggregate( + self, + ) -> None: + """R4's route: an AGGREGATE-phase filter on a filtered-local isolated + aggregate renders as plain WHERE on the joined-back column of the outer + combined SELECT. Gold totals: new = 300, old = 0/NULL — so a threshold + of 100 keeps only ``new``.""" + engine = await self._engine() + resp = await engine.execute( + SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ModelMeasure(formula="gold_amount:sum", name="g")], + filters=["gold_amount:sum > 100"], + ) + ) + assert [r["orders.status"] for r in resp.data] == ["new"] + assert resp.data[0]["orders.g"] == 300.0 + + async def test_outer_wrapper_filter_carrying_a_scalar_call(self) -> None: + """B5 on R4's route specifically — the patched scalar branch. Wrapping + the same comparison in ``ifnull`` must not change which rows survive, + and (on a dialect without IFNULL) must not emit it.""" + engine = await self._engine() + resp = await engine.execute( + SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ModelMeasure(formula="gold_amount:sum", name="g")], + filters=["ifnull(gold_amount:sum, 0) > 100"], + ) + ) + assert [r["orders.status"] for r in resp.data] == ["new"] + assert resp.data[0]["orders.g"] == 300.0 + + async def test_outer_wrapper_scalar_call_is_transpiled_on_postgres( + self, + ) -> None: + """The emission half of the same case, and the B5 bug in its sharpest + form: R4 passes scalar calls through as ``exp.Anonymous``, so the + literal ``IFNULL`` reaches POSTGRES — which has no such function, making + this generated SQL simply invalid there. + + Postgres-typed datasource + ``dry_run`` (no execution): the point is + what is EMITTED for that backend. Asserted on Postgres rather than + SQLite deliberately — SQLite does have ``IFNULL``, so the same + assertion there would be a policy preference rather than a correctness + claim. + """ + engine = await self._engine(dialect="postgres") + resp = await engine.execute( + SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ModelMeasure(formula="gold_amount:sum", name="g")], + filters=["ifnull(gold_amount:sum, 0) > 100"], + ), + dry_run=True, + ) + assert "IFNULL" not in resp.sql.upper(), resp.sql + assert "COALESCE" in resp.sql.upper(), resp.sql + + async def test_shifted_cte_filter_call_site_executes(self) -> None: + """R1's SECOND call site (the ``time_shift`` CTE's WHERE, :7468). + + A host filter must apply inside the shifted CTE as well as the host + base, so the shifted value is computed over the same filtered rows. + With ``status = 'new'`` only, January's total is 100 and February's + shifted-by-one-month value must be that same 100. + """ + from slayer.core.enums import TimeGranularity + from slayer.core.query import TimeDimension + + engine = await self._engine() + resp = await engine.execute( + SlayerQuery( + source_model="orders", + time_dimensions=[ + TimeDimension( + dimension=ColumnRef(name="created_at"), + granularity=TimeGranularity.MONTH, + ), + ], + measures=[ + ModelMeasure(formula="amount:sum", name="amt"), + ModelMeasure( + formula="time_shift(amount:sum, -1, 'month')", + name="prev", + ), + ], + filters=["status = 'new'"], + ) + ) + rows = sorted(resp.data, key=lambda r: str(r["orders.created_at"])) + assert len(rows) == 2, rows + assert rows[0]["orders.amt"] == 100.0 + assert rows[1]["orders.amt"] == 200.0 + # February's shifted value is January's filtered total. + assert rows[1]["orders.prev"] == 100.0 diff --git a/tests/test_parity_guards.py b/tests/test_parity_guards.py index e0519bd9..b436a26c 100644 --- a/tests/test_parity_guards.py +++ b/tests/test_parity_guards.py @@ -10,8 +10,13 @@ * the approved guard disappears without its ``APPROVED_GUARDS`` entry also being removed (when DEV-1715 lands, delete the guard AND the entry together). -DEV-1485 (Stage 11) gates on this set — plus ``tests/parity_xfails.py`` — being -empty, so no deferred coverage can rot silently. +The gate is this set being empty, so no deferred coverage can rot silently. + +This docstring used to name a second file as part of the gate — a companion +xfail-registry module, which was deliberately deleted along with its +``pytest_collection_modifyitems`` hook (see ``DECISIONS.md``, 2026-08-04). The +reference sent readers looking for infrastructure that no longer exists, so it +is gone. Only the prose changed; the guard's behavior is unchanged. """ from pathlib import Path diff --git a/tests/test_sql_generator.py b/tests/test_sql_generator.py index 0483c3d1..3a202e65 100644 --- a/tests/test_sql_generator.py +++ b/tests/test_sql_generator.py @@ -11031,7 +11031,7 @@ async def test_instr_translates_per_dialect( ("postgres", "SUBSTRING(orders.status FROM 1 FOR 5)"), ("mysql", "SUBSTRING(orders.status, 1, 5)"), ("duckdb", "SUBSTRING(orders.status, 1, 5)"), - ("clickhouse", "SUBSTR(orders.status, 1, 5)"), + ("clickhouse", "SUBSTRING(orders.status, 1, 5)"), ], ) async def test_substr_translates_per_dialect( @@ -11052,11 +11052,17 @@ async def test_substr_translates_per_dialect( @pytest.mark.parametrize( "dialect,expected_substring", [ - # SQLite normalises CONCAT(...) → a || b at emit time. + # Every dialect whose sqlglot emitter prefers the operator now + # renders ``||``: the unified ScalarCall policy builds a typed + # ``exp.Concat`` instead of passing ``CONCAT`` through literally. + # On Postgres this is a SEMANTIC change as well as a spelling one — + # ``CONCAT()`` ignores NULL operands, ``||`` propagates them — and + # it aligns filters with the projection path, which has always + # emitted ``||`` here. ("sqlite", "orders.status || orders.status"), - ("postgres", "CONCAT(orders.status, orders.status)"), + ("postgres", "orders.status || orders.status"), ("mysql", "CONCAT(orders.status, orders.status)"), - ("duckdb", "CONCAT(orders.status, orders.status)"), + ("duckdb", "orders.status || orders.status"), ("clickhouse", "CONCAT(orders.status, orders.status)"), ], ) From e2e4d88049dfa3445c3fd424ef783ffa48119e3a Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Wed, 5 Aug 2026 17:28:35 +0200 Subject: [PATCH 02/98] =?UTF-8?q?DEV-1744:=20address=20review=20=E2=80=94?= =?UTF-8?q?=20renderer=20correctness=20+=20drift=20removal?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two real defects in the new renderer, plus the review's structural points. Operator precedence was not materialised. sqlglot does NOT parenthesise by node nesting: `Mul(Add(a, b), c)` emits `a + b * c`, which evaluates differently. The generator has `_paren_if_lower_prec` for exactly this; the new renderer had nothing, so any nested arithmetic would have rendered with the wrong grouping once the production paths route through it. Unary operators were dropped. The binder represents `-10` as a SINGLE-operand ArithmeticKey; a fold that starts at operands[0] and iterates operands[1:] returns it unchanged, so `amount > -10` became `amount > 10`. `not` was missing entirely. Both now go through one composer that mirrors the generator's, including the `is` / `is not` forms. From the review: * TimeTruncKey went through a literal DATE_TRUNC instead of the dialect's build_date_trunc, so it named a function SQLite does not have and had no week_sunday handling. It now delegates, with per-dialect tests. * The no-builder aggregate path ignored column_filter_key and args/kwargs, so a filtered aggregate would have rendered as a plain SUM covering rows the filter excludes. Both now fail closed. * _literal stringified unsupported types; it raises, matching the generator's helper. * _rewrite_log_alias duplicated the generator's copy of the very policy this PR consolidates. The generator now delegates to the shared one. * `canonical_alias` was read from a leaked loop variable after this PR removed its per-iteration assignment. Latent — the planner populates public_aliases, so the fallback that reads it is not reachable today — but it would have projected one measure under another's name. * _wm_ CTE naming routes through the shared cte_name_from_alias. * Registry membership is checked both ways: a key that is NOT a built-in would make is_builtin_agg accept a typo. Conventions: helpers take keyword-only arguments; test imports moved to the top of their files; temp dirs go through pytest's tmp_path_factory instead of leaking mkdtemp directories; volatile generator line numbers dropped from docstrings in favour of the stable function names. Sonar: NOSONAR with rationale on the two consolidated dispatch functions and on the ASCII-only identifier regex — `\W` is Unicode-aware in Python and would let accented letters into a name that must be a bare ASCII SQL identifier. Tests: 211 in the three new files (up from 196), full non-integration suite 9518 passed, ruff clean. Co-Authored-By: Claude Fable 5 --- slayer/sql/generator.py | 58 ++-- slayer/sql/naming.py | 7 +- slayer/sql/render/aggregates.py | 52 ++-- slayer/sql/render/value_expr.py | 190 ++++++++++--- tests/test_dev1744_naming_allocator.py | 90 +++++-- tests/test_dev1744_result_key_contract.py | 5 +- tests/test_dev1744_value_expr.py | 310 ++++++++++++++++------ 7 files changed, 510 insertions(+), 202 deletions(-) diff --git a/slayer/sql/generator.py b/slayer/sql/generator.py index ed5680f1..804a5c8a 100644 --- a/slayer/sql/generator.py +++ b/slayer/sql/generator.py @@ -56,7 +56,7 @@ result_key_from_alias, ) from slayer.sql.render.aggregates import window_agg_class -from slayer.sql.render.value_expr import render_scalar_call +from slayer.sql.render.value_expr import render_scalar_call, rewrite_log_alias from slayer.sql.reserved_keywords import prequote_reserved_identifiers from slayer.sql.scope import ScopeFrame from slayer.sql.scope_check import maybe_validate_scopes @@ -1064,29 +1064,14 @@ def _build_transform_sql(self, t) -> str: # NOSONAR S3776 — flat dispatch ove # ------------------------------------------------------------------ def _rewrite_log_aliases(self, node: exp.Expression) -> exp.Expression: - """DEV-1337: rewrite ``Log(this=Literal(10|2), expression=X)`` back to - ``Anonymous(this='log10'|'log2', expressions=[X])`` for dialects with - native single-arg aliases. Walked over every parsed AST so the - rewrite survives sqlglot's re-parse passes (which would otherwise - turn ``LOG10(x)`` back into a generic ``Log`` node and re-emit as - ``LOG(10, x)``). No-op on non-``Log`` nodes and on ``Log`` nodes - with a non-literal or non-{10,2} base. + """Thin delegator to the shared log-alias policy in + ``slayer.sql.render.value_expr``. + + Kept as a method so the existing ``tree.transform(...)`` call sites, + which walk every parsed AST so the rewrite survives sqlglot's re-parse + passes, stay unchanged. """ - if not isinstance(node, exp.Log): - return node - base = node.args.get("this") - arg = node.args.get("expression") - if arg is None or not isinstance(base, exp.Literal) or base.is_string: - return node - try: - base_val = float(base.this) - except (TypeError, ValueError): - return node - if base_val == 10 and self._dialect.should_use_native_log(10): - return exp.Anonymous(this="log10", expressions=[arg.copy()]) - if base_val == 2 and self._dialect.should_use_native_log(2): - return exp.Anonymous(this="log2", expressions=[arg.copy()]) - return node + return rewrite_log_alias(node, dialect=self._dialect) def _resolve_sql( self, @@ -3939,7 +3924,7 @@ def _render_aggregate_composite_expr( # NOSONAR(S3776) — sequential isinstanc if key.name == "like": return exp.Like(this=args[0], expression=args[1]), any_agg return render_scalar_call( - key.name, args, dialect=self._dialect, + name=key.name, args=args, dialect=self._dialect, ), any_agg if isinstance(key, LiteralKey): v = key.value @@ -4701,8 +4686,8 @@ def _add_local_aux_slots( full_agg_alias = self._full_alias_for_slot( slot=agg_slot, source_relation=source_relation, alias_index={}, ) - cte_name = wm_allocator.allocate_cte( - _cte_name_from_alias("_wm_", full_agg_alias), + cte_name = cte_name_from_alias( + "_wm_", full_agg_alias, allocator=wm_allocator, ) cte_sql, grain_aliases = self._render_window_measure_cte_from_planned( plan=plan, agg_slot=agg_slot, source_model=source_model, @@ -4860,6 +4845,11 @@ def _render_outer_composite(cslot) -> str: agg_slot = slots_by_id[plan.aggregate_slot_id] agg_col_alias = agg_col_alias_for_plan[plan.aggregate_slot_id] cte_name = cm_cte_name_for_plan[plan.aggregate_slot_id] + # Read THIS plan's canonical alias. It feeds + # ``_public_aliases_for_cross_model_agg``, which falls back to it + # when the slot declares no public alias, so a stale value projects + # one measure under another measure's name. + canonical_alias = canonical_alias_for_plan[plan.aggregate_slot_id] # DEV-1495 bug 2 / DEV-1712: an order-by-only (hidden) cross-model # aggregate never surfaces in the combined projection — its CTE is # still joined below, and the ORDER BY references it CTE-qualified @@ -5138,7 +5128,7 @@ def _render_outer_composite(cslot) -> str: # we don't have on the new side. Future slices may re-enable. return sql - def _render_cross_model_transform_chain( + def _render_cross_model_transform_chain( # NOSONAR(S3776) — pre-existing complexity in the window-layer chain; this PR only threaded the CTE-name allocator through it, which re-attributed the function as new code. The chain is rebuilt as sqlglot AST in the scope-assembly PR, where the layering is what gets simplified. self, *, prelude_ctes: List[Tuple[str, str]], @@ -6393,7 +6383,7 @@ def _render_filter_value_key_in_target_scope( # NOSONAR(S3776) — sequential i if value_key.name == "like": return exp.Like(this=rendered_args[0], expression=rendered_args[1]) return render_scalar_call( - value_key.name, rendered_args, dialect=self._dialect, + name=value_key.name, args=rendered_args, dialect=self._dialect, ) if isinstance(value_key, BetweenKey): # DEV-1708: a routed ``date_range``-derived BETWEEN over a target @@ -7199,7 +7189,9 @@ def recurse(k) -> exp.Expression: ] if key.name == "like": return exp.Like(this=args[0], expression=args[1]) - return render_scalar_call(key.name, args, dialect=self._dialect) + return render_scalar_call( + name=key.name, args=args, dialect=self._dialect, + ) if isinstance(key, BetweenKey): return exp.Between( @@ -9499,7 +9491,9 @@ def _render_value_key_for_filter( # NOSONAR(S3776) — sequential isinstance di # normalises to a generic ``Log(10, x)`` that re-emits as # ``LOG(10, x)``, wrong for dialects with a native single-arg # ``LOG10``. Transpiling alone fixes ifnull and breaks log10. - return render_scalar_call(key.name, args, dialect=self._dialect) + return render_scalar_call( + name=key.name, args=args, dialect=self._dialect, + ) if isinstance(key, BetweenKey): col_expr = self._render_value_key_for_filter( key=key.column, @@ -9687,7 +9681,9 @@ def _slot_alias_column(slot) -> Optional[exp.Expression]: # normalises to a generic ``Log(10, x)`` that re-emits as # ``LOG(10, x)``, wrong for dialects with a native single-arg # ``LOG10``. Transpiling alone fixes ifnull and breaks log10. - return render_scalar_call(key.name, args, dialect=self._dialect) + return render_scalar_call( + name=key.name, args=args, dialect=self._dialect, + ) if isinstance(key, BetweenKey): col_expr = self._render_filter_for_outer_wrapper( key=key.column, diff --git a/slayer/sql/naming.py b/slayer/sql/naming.py index 607854b2..ce9e555a 100644 --- a/slayer/sql/naming.py +++ b/slayer/sql/naming.py @@ -251,7 +251,10 @@ def flat_name(dotted: str, *, strip_relation: Optional[str] = None) -> str: # LOSSY on purpose (a CTE name must be a bare identifier) — which is exactly # why the result must go through an allocator rather than being trusted as an # identity. -_NON_IDENT_CHAR_RE = re.compile(r"[^a-zA-Z0-9_]") +# Written out rather than as ``\W``: Python's ``\w`` is Unicode-aware, so ``\W`` +# would let accented letters and other non-ASCII word characters through into a +# name that must be a bare ASCII SQL identifier. +_NON_IDENT_CHAR_RE = re.compile(r"[^a-zA-Z0-9_]") # NOSONAR(S6353) — see above: \W is Unicode-aware and would not be equivalent. def cte_name_from_alias( @@ -310,7 +313,7 @@ def cte_name_from_alias( _PROFILES_WITHOUT_RELATION = ("cte_schema", "declared_name", "stage_formula") -def canonical_aggregate_alias( +def canonical_aggregate_alias( # NOSONAR(S3776) — sequential dispatch over the four frozen alias profiles; each branch IS that profile's contract, and extracting per-profile helpers would restore the four-copy drift this function removes. key: "AggregateKey", *, profile: AggAliasProfile, diff --git a/slayer/sql/render/aggregates.py b/slayer/sql/render/aggregates.py index c3495640..70882bee 100644 --- a/slayer/sql/render/aggregates.py +++ b/slayer/sql/render/aggregates.py @@ -43,7 +43,7 @@ def windowable(self) -> bool: return self.window_class is not None -def _entry(name: str, dispatch: str, **kw) -> AggEntry: +def _entry(*, name: str, dispatch: str, **kw) -> AggEntry: return AggEntry(name=name, dispatch=dispatch, **kw) @@ -52,34 +52,40 @@ def _entry(name: str, dispatch: str, **kw) -> AggEntry: for e in ( # Only sum and avg carry a window frame — the same pair the stage # planner gates windowed measures on. - _entry("sum", DISPATCH_SIMPLE, node_class=exp.Sum, window_class=exp.Sum), - _entry("avg", DISPATCH_SIMPLE, node_class=exp.Avg, window_class=exp.Avg), - _entry("count", DISPATCH_SIMPLE, node_class=exp.Count), - _entry("min", DISPATCH_SIMPLE, node_class=exp.Min), - _entry("max", DISPATCH_SIMPLE, node_class=exp.Max), - _entry("count_distinct", DISPATCH_DISTINCT, node_class=exp.Count), - _entry("count_distinct_approx", DISPATCH_DIALECT_HOOK), - _entry("first", DISPATCH_RANKED), - _entry("last", DISPATCH_RANKED), - _entry("median", DISPATCH_DIALECT_HOOK), - _entry("percentile", DISPATCH_DIALECT_HOOK), - _entry("weighted_avg", DISPATCH_FORMULA), - _entry("stddev_samp", DISPATCH_STAT), - _entry("stddev_pop", DISPATCH_STAT), - _entry("var_samp", DISPATCH_STAT), - _entry("var_pop", DISPATCH_STAT), - _entry("corr", DISPATCH_STAT), - _entry("covar_samp", DISPATCH_STAT), - _entry("covar_pop", DISPATCH_STAT), + _entry(name="sum", dispatch=DISPATCH_SIMPLE, node_class=exp.Sum, window_class=exp.Sum), + _entry(name="avg", dispatch=DISPATCH_SIMPLE, node_class=exp.Avg, window_class=exp.Avg), + _entry(name="count", dispatch=DISPATCH_SIMPLE, node_class=exp.Count), + _entry(name="min", dispatch=DISPATCH_SIMPLE, node_class=exp.Min), + _entry(name="max", dispatch=DISPATCH_SIMPLE, node_class=exp.Max), + _entry(name="count_distinct", dispatch=DISPATCH_DISTINCT, node_class=exp.Count), + _entry(name="count_distinct_approx", dispatch=DISPATCH_DIALECT_HOOK), + _entry(name="first", dispatch=DISPATCH_RANKED), + _entry(name="last", dispatch=DISPATCH_RANKED), + _entry(name="median", dispatch=DISPATCH_DIALECT_HOOK), + _entry(name="percentile", dispatch=DISPATCH_DIALECT_HOOK), + _entry(name="weighted_avg", dispatch=DISPATCH_FORMULA), + _entry(name="stddev_samp", dispatch=DISPATCH_STAT), + _entry(name="stddev_pop", dispatch=DISPATCH_STAT), + _entry(name="var_samp", dispatch=DISPATCH_STAT), + _entry(name="var_pop", dispatch=DISPATCH_STAT), + _entry(name="corr", dispatch=DISPATCH_STAT), + _entry(name="covar_samp", dispatch=DISPATCH_STAT), + _entry(name="covar_pop", dispatch=DISPATCH_STAT), ) } # Every built-in must be in the table, or a lookup would fall through to the # custom-formula path and render a built-in as if it were user-defined. -_missing = BUILTIN_AGGREGATIONS - set(AGG_REGISTRY) -if _missing: # pragma: no cover — import-time invariant +# Checked BOTH ways: a missing built-in would fall through to the custom-formula +# path, and a registry key that is NOT a built-in (a typo such as sumn) would +# make is_builtin_agg accept it and route it away from that path. +_registered = set(AGG_REGISTRY) +_missing = set(BUILTIN_AGGREGATIONS) - _registered +_unknown = _registered - set(BUILTIN_AGGREGATIONS) +if _missing or _unknown: # pragma: no cover — import-time invariant raise RuntimeError( - f"Aggregation registry is missing built-ins: {sorted(_missing)}", + f"Aggregation registry disagrees with BUILTIN_AGGREGATIONS: " + f"missing={sorted(_missing)}, unknown={sorted(_unknown)}", ) diff --git a/slayer/sql/render/value_expr.py b/slayer/sql/render/value_expr.py index 889f7c7c..85d5f119 100644 --- a/slayer/sql/render/value_expr.py +++ b/slayer/sql/render/value_expr.py @@ -21,15 +21,15 @@ Deferred to the scope-assembly PR, together with the cross-scope migration, because the two are the same piece of work. Finishing it needs: -* **Filter paths** (``_render_value_key_for_filter``, ``:9119`` host WHERE / - HAVING and ``:7468`` the shifted-CTE WHERE) — ``FilterFacilities`` must carry +* **Filter paths** (``_render_value_key_for_filter``, ``that call site`` host WHERE / + HAVING and ``that call site`` the shifted-CTE WHERE) — ``FilterFacilities`` must carry the local-aggregate HAVING branch, which reads ``slot_by_key`` to find a materialised slot, the first/last ranked state, and the filter-side CAST policy applied per column type. Rendering an aggregate leaf inline (rather than by output alias) is what makes HAVING work on backends that reject SELECT aliases there, so that branch cannot simply be dropped. -* **Composite paths** (``_render_aggregate_composite_expr``, ``:2851`` the base - SELECT and ``:3741`` the first/last base SELECT) — ``CompositeFacilities`` +* **Composite paths** (``_render_aggregate_composite_expr``, ``that call site`` the base + SELECT and ``that call site`` the first/last base SELECT) — ``CompositeFacilities`` already declares the maps these need (rn-suffix, filtered-rank and match-flag, composite alias-by-key, resolved agg kwargs, value alias-by-sql); they are threaded through but not yet consumed, because the aggregate leaf @@ -53,9 +53,11 @@ from decimal import Decimal from typing import Any, Callable, Dict, List, Optional, Tuple +import sqlglot from pydantic import BaseModel, ConfigDict, Field from sqlglot import exp +from slayer.core.enums import TimeGranularity from slayer.core.errors import RenderContextMissingFacilityError from slayer.core.keys import ( AggregateKey, @@ -90,11 +92,6 @@ "<": exp.LT, "<=": exp.LTE, ">": exp.GT, ">=": exp.GTE, } -_GRANULARITY_TO_SQL: Dict[str, str] = { - "second": "SECOND", "minute": "MINUTE", "hour": "HOUR", "day": "DAY", - "week": "WEEK", "month": "MONTH", "quarter": "QUARTER", "year": "YEAR", -} - class FilterFacilities(BaseModel): """What WHERE / HAVING rendering needs beyond the scope.""" @@ -153,7 +150,7 @@ class RenderContext(BaseModel): aliases: Optional[AliasFacilities] = None -def _require(ctx: RenderContext, facility: str, key: Any) -> Any: +def _require(*, ctx: RenderContext, facility: str, key: Any) -> Any: got = getattr(ctx, facility, None) if got is None: raise RenderContextMissingFacilityError( @@ -163,6 +160,12 @@ def _require(ctx: RenderContext, facility: str, key: Any) -> Any: def _literal(value: Any) -> exp.Expression: + """Render a scalar leaf. + + Unsupported types RAISE rather than being stringified: a ``datetime`` or a + ``list`` silently becoming a quoted string is a wrong value, not an error, + and the generator's equivalent already raises. + """ if value is None: return exp.Null() if isinstance(value, bool): @@ -171,11 +174,92 @@ def _literal(value: Any) -> exp.Expression: return exp.Literal.number(str(value)) if isinstance(value, (int, float)): return exp.Literal.number(str(value)) - return exp.Literal.string(str(value)) + if isinstance(value, str): + return exp.Literal.string(value) + raise NotImplementedError( + f"Unsupported literal in a ValueKey render: " + f"type={type(value).__name__} value={value!r}", + ) + + +# sqlglot does NOT parenthesise by node nesting: ``Mul(Add(a, b), c)`` emits +# ``a + b * c``, which evaluates differently. Precedence must be materialised +# as explicit ``Paren`` nodes. +_ARITH_PRECEDENCE: Dict[Any, int] = { + exp.Add: 1, exp.Sub: 1, exp.Mul: 2, exp.Div: 2, exp.Mod: 2, +} + + +def _paren_if_lower_prec( + child: exp.Expression, *, parent_prec: int, is_right: bool, op: str, +) -> exp.Expression: + """Parenthesise ``child`` when dropping its parens would change meaning. + + Lower precedence than the parent always needs parens; equal precedence + needs them on the RIGHT of the non-associative ``-`` and ``/`` + (``a - (b - c)``). Non-arithmetic children are already self-delimiting. + """ + child_prec = _ARITH_PRECEDENCE.get(type(child)) + if child_prec is None: + return child + if child_prec < parent_prec: + return exp.Paren(this=child) + if child_prec == parent_prec and is_right and op in ("-", "/", "%"): + return exp.Paren(this=child) + return child + + +def _render_arithmetic( + op: str, operands: List[exp.Expression], +) -> exp.Expression: + """Compose an arithmetic / comparison / boolean operator. + + Mirrors the generator's composer, including the unary forms: the binder + represents ``-x`` as a SINGLE-operand ``ArithmeticKey``, so a fold that + just returns ``operands[0]`` would turn ``amount > -10`` into + ``amount > 10``. + """ + if len(operands) == 1: + if op == "not": + return exp.Not(this=operands[0]) + if op == "-": + return exp.Neg(this=operands[0]) + if op == "+": + return operands[0] + raise NotImplementedError( + f"Unsupported unary operator {op!r}.", + ) + + if op == "and": + return exp.and_(*operands) + if op == "or": + return exp.or_(*operands) + if op == "is": + return exp.Is(this=operands[0], expression=operands[1]) + if op == "is not": + return exp.Not(this=exp.Is(this=operands[0], expression=operands[1])) + + node_cls = _BINARY_OPS.get(op) + if node_cls is None: + raise NotImplementedError(f"Unsupported arithmetic operator {op!r}.") + + parent_prec = _ARITH_PRECEDENCE.get(node_cls) + result = operands[0] + for operand in operands[1:]: + lhs, rhs = result, operand + if parent_prec is not None: + lhs = _paren_if_lower_prec( + lhs, parent_prec=parent_prec, is_right=False, op=op, + ) + rhs = _paren_if_lower_prec( + rhs, parent_prec=parent_prec, is_right=True, op=op, + ) + result = node_cls(this=lhs, expression=rhs) + return result def render_scalar_call( - name: str, args: List[exp.Expression], *, dialect: SqlDialect, + *, name: str, args: List[exp.Expression], dialect: SqlDialect, ) -> exp.Expression: """The one ScalarCall policy: typed node, dialect rewrite, log-alias fix-up. @@ -189,10 +273,18 @@ def render_scalar_call( if name == "like": return exp.Like(this=args[0], expression=args[1]) node = dialect.rewrite_target_ast(exp.func(name.upper(), *args)) - return _rewrite_log_alias(node, dialect=dialect) + return rewrite_log_alias(node, dialect=dialect) -def _rewrite_log_alias(node: exp.Expression, *, dialect: SqlDialect): +def rewrite_log_alias( + node: exp.Expression, *, dialect: SqlDialect, +) -> exp.Expression: + """The single log-alias policy: a 2-arg ``LOG(10|2, x)`` becomes the + dialect's native single-arg ``log10`` / ``log2`` where one exists. + + Shared with the generator, which applies it over parsed trees. Two copies + of this rule would reintroduce exactly the drift this module removes. + """ if not isinstance(node, exp.Log): return node base = node.args.get("this") @@ -212,7 +304,7 @@ def _rewrite_log_alias(node: exp.Expression, *, dialect: SqlDialect): def _render_aggregate(key: AggregateKey, ctx: RenderContext) -> exp.Expression: - facilities = _require(ctx, "composites", key) + facilities = _require(ctx=ctx, facility="composites", key=key) if facilities.agg_builder is not None: return facilities.agg_builder(key) @@ -234,11 +326,38 @@ def _render_aggregate(key: AggregateKey, ctx: RenderContext) -> exp.Expression: f"mechanism, which needs the generator's builder" ), ) + # The dispatch gate above refuses aggregations whose MECHANISM needs the + # generator. These two refuse a key whose FIELDS do: without them a + # filtered aggregate would render as a plain SUM, silently covering rows + # the filter excludes — a wrong number rather than an error. + if key.column_filter_key is not None: + raise RenderContextMissingFacilityError( + key_kind=type(key).__name__, + facility="composites.agg_builder", + detail=( + "the aggregate's source carries a column filter, which needs " + "the generator's CASE-WHEN wrapper" + ), + ) + if key.kwargs or key.args: + raise RenderContextMissingFacilityError( + key_kind=type(key).__name__, + facility="composites.agg_builder", + detail=( + f"aggregation {key.agg!r} carries args/kwargs, which need the " + f"generator's parameter resolution" + ), + ) if isinstance(key.source, StarKey): inner: exp.Expression = exp.Star() else: inner = ctx.scope.resolve(key.source, consumer=ctx.consumer) - assert entry.node_class is not None + if entry.node_class is None: # pragma: no cover — dispatch gate guarantees it + raise RenderContextMissingFacilityError( + key_kind=type(key).__name__, + facility="composites.agg_builder", + detail=f"aggregation {key.agg!r} has no direct sqlglot node", + ) if entry.dispatch == DISPATCH_DISTINCT: return entry.node_class(this=exp.Distinct(expressions=[inner])) return entry.node_class(this=inner) @@ -259,29 +378,22 @@ def render_value_key( # NOSONAR(S3776) — sequential dispatch over the closed if isinstance(key, TimeTruncKey): column = ctx.scope.resolve(key.column, consumer=ctx.consumer) - unit = _GRANULARITY_TO_SQL.get( - key.granularity.lower(), key.granularity.upper(), - ) - return ctx.dialect.rewrite_target_ast( - exp.func("DATE_TRUNC", exp.Literal.string(unit), column), + # Delegate to the dialect strategy, which owns the per-backend wire + # form: STRFTIME on SQLite, DATETRUNC on T-SQL, native WEEK(SUNDAY) on + # BigQuery, plus the WEEK_SUNDAY day-shift. Emitting a literal + # DATE_TRUNC here would name a function SQLite does not have. + return ctx.dialect.build_date_trunc( + column, + TimeGranularity(key.granularity), + parse=lambda sql: sqlglot.parse_one( + sql, dialect=ctx.dialect.sqlglot_name, + ), ) if isinstance(key, ArithmeticKey): - operands = [render_value_key(o, ctx) for o in key.operands] - op = key.op.lower() - if op == "and": - return exp.and_(*operands) - if op == "or": - return exp.or_(*operands) - node_cls = _BINARY_OPS.get(key.op) - if node_cls is None: - raise NotImplementedError( - f"Unsupported arithmetic operator {key.op!r}.", - ) - result = operands[0] - for operand in operands[1:]: - result = node_cls(this=result, expression=operand) - return result + return _render_arithmetic( + key.op.lower(), [render_value_key(o, ctx) for o in key.operands], + ) if isinstance(key, ScalarCallKey): args = [ @@ -290,7 +402,9 @@ def render_value_key( # NOSONAR(S3776) — sequential dispatch over the closed else _literal(a) for a in key.args ] - return render_scalar_call(key.name, args, dialect=ctx.dialect) + return render_scalar_call( + name=key.name, args=args, dialect=ctx.dialect, + ) if isinstance(key, BetweenKey): return exp.Between( @@ -312,7 +426,7 @@ def render_value_key( # NOSONAR(S3776) — sequential dispatch over the closed if isinstance(key, TransformKey): # POST-phase: the value was materialised by an earlier scope, so it is # referenced by alias rather than rebuilt. - facilities = _require(ctx, "aliases", key) + facilities = _require(ctx=ctx, facility="aliases", key=key) slot_id = facilities.slot_id_by_key.get(key) alias = ( facilities.available_alias_by_slot_id.get(slot_id) diff --git a/tests/test_dev1744_naming_allocator.py b/tests/test_dev1744_naming_allocator.py index 05df1ac2..4848a7e6 100644 --- a/tests/test_dev1744_naming_allocator.py +++ b/tests/test_dev1744_naming_allocator.py @@ -49,7 +49,6 @@ import os import re import sqlite3 -import tempfile from decimal import Decimal from typing import AsyncIterator, List @@ -83,7 +82,7 @@ # =========================================================================== -async def _build_engine(*, dialect: str = "sqlite") -> SlayerQueryEngine: +async def _build_engine(*, base_dir: str, dialect: str = "sqlite") -> SlayerQueryEngine: """orders -> customers, seeded, with the collision-bait columns. On ``customers``: @@ -99,7 +98,7 @@ async def _build_engine(*, dialect: str = "sqlite") -> SlayerQueryEngine: ``orders.customers__revenue_sum`` sanitises to exactly the same CTE name as the cross-model ``orders.customers.revenue_sum``. """ - d = tempfile.mkdtemp() + d = base_dir db_path = os.path.join(d, "b4.db") con = sqlite3.connect(db_path) cur = con.cursor() @@ -179,8 +178,8 @@ async def _build_engine(*, dialect: str = "sqlite") -> SlayerQueryEngine: @pytest.fixture -async def engine() -> AsyncIterator[SlayerQueryEngine]: - yield await _build_engine() +async def engine(tmp_path_factory) -> AsyncIterator[SlayerQueryEngine]: + yield await _build_engine(base_dir=str(tmp_path_factory.mktemp("b4"))) def _cte_names_by_scope(sql: str, *, dialect: str = "sqlite") -> List[List[str]]: @@ -353,6 +352,37 @@ async def test_filtered_local_and_plain_aggregate_coexist( ], ) + async def test_unrenamed_cross_model_measures_keep_their_own_aliases( + self, engine, + ) -> None: + """Two cross-model measures with NO declared ``name`` must each project + under their OWN canonical alias. + + Every other cross-model test in this file declares ``name=``. This one + does not, so it exercises the path where the public alias is derived + rather than user-supplied — the branch that reads a plan's canonical + alias. It is a coverage gap-filler, not a regression guard: the planner + populates ``public_aliases`` even for un-named measures, so the + canonical-alias fallback inside + ``_public_aliases_for_cross_model_agg`` is not reachable from here. + """ + resp = await engine.execute( + SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ + ModelMeasure(formula="customers.revenue:sum"), + ModelMeasure(formula="customers.Rev:sum"), + ], + ) + ) + assert "orders.customers.revenue_sum" in resp.columns, resp.columns + assert "orders.customers.Rev_sum" in resp.columns, resp.columns + row = resp.data[0] + # revenue sums to 600 over the three customers; Rev (revx) to 24 — + # equal values would mean both read the same CTE column. + assert row["orders.customers.revenue_sum"] != row["orders.customers.Rev_sum"] + async def test_same_key_slots_still_share_one_cte(self, engine) -> None: """Parity guard for the C13 intent the buggy dedup was meant to serve: two measures that are the SAME aggregate under different public names @@ -387,7 +417,8 @@ async def test_same_key_slots_both_surface_with_equal_values( ) ) row = resp.data[0] - assert "orders.a" in row and "orders.b" in row, list(row) + assert "orders.a" in row, list(row) + assert "orders.b" in row, list(row) assert row["orders.a"] == row["orders.b"] @@ -493,8 +524,8 @@ def test_no_raw_step_cte_names_in_the_generator(self) -> None: difference today: the three ``f"step{...}"`` mint sites must all go through the allocator. - ``generator.py:1916`` bypasses an allocator that is in scope 100 lines - above it; ``:5175`` and ``:5229`` live in the cross-model transform + ``the generator`` bypasses an allocator that is in scope 100 lines + above it; ``that call site`` and ``that call site`` live in the cross-model transform chain, which holds no allocator at all. They are latently safe only because no ``_cm_*`` CTE can be named ``stepN`` — an invariant nothing enforces. @@ -552,10 +583,10 @@ def test_generation_allocator_is_shared_across_cte_families(self) -> None: ] -async def _hostile_engine(*, column: str) -> SlayerQueryEngine: +async def _hostile_engine(*, column: str, base_dir: str) -> SlayerQueryEngine: """A single-model store whose ``orders`` model carries a user column named exactly like one of SLayer's internal minted names.""" - d = tempfile.mkdtemp() + d = base_dir db_path = os.path.join(d, "hostile.db") con = sqlite3.connect(db_path) cur = con.cursor() @@ -615,9 +646,11 @@ class TestInternalNamesDoNotCollideWithUserColumns: @pytest.mark.parametrize("column", _INTERNAL_NAMES) async def test_user_column_named_like_an_internal_alias( - self, column, + self, column, tmp_path_factory, ) -> None: - engine = await _hostile_engine(column=column) + engine = await _hostile_engine( + column=column, base_dir=str(tmp_path_factory.mktemp("hostile")), + ) resp = await engine.execute( SlayerQuery( source_model="orders", @@ -638,11 +671,13 @@ async def test_user_column_named_like_an_internal_alias( @pytest.mark.parametrize("column", _INTERNAL_NAMES) async def test_user_column_named_like_an_internal_alias_as_dimension( - self, column, + self, column, tmp_path_factory, ) -> None: """The same names as a GROUP BY dimension, which routes through the ranked-subquery / projection aliasing rather than the aggregate path.""" - engine = await _hostile_engine(column=column) + engine = await _hostile_engine( + column=column, base_dir=str(tmp_path_factory.mktemp("hostile")), + ) resp = await engine.execute( SlayerQuery( source_model="orders", @@ -654,12 +689,14 @@ async def test_user_column_named_like_an_internal_alias_as_dimension( @pytest.mark.parametrize("column", ["_td_0", "_dim_0", "_val_0"]) async def test_ranked_subquery_families_survive_the_name( - self, column, + self, column, tmp_path_factory, ) -> None: """Reaches the ``_td_`` / ``_dim_`` counters specifically: a first/last measure builds the ranked subquery those aliases live in. Last amount per status, ordered by ``created_at``: a -> 20, b -> 30.""" - engine = await _hostile_engine(column=column) + engine = await _hostile_engine( + column=column, base_dir=str(tmp_path_factory.mktemp("hostile")), + ) resp = await engine.execute( SlayerQuery( source_model="orders", @@ -673,13 +710,15 @@ async def test_ranked_subquery_families_survive_the_name( @pytest.mark.parametrize( "column", ["_w_dim_0", "_w_td_0", "_w_time", "_w_value"], ) - async def test_windowed_families_survive_the_name(self, column) -> None: + async def test_windowed_families_survive_the_name(self, column, tmp_path_factory) -> None: """Reaches the ``_w_*`` ``_src``-projection aliases specifically: a duration-windowed measure is the only shape that builds them.""" from slayer.core.enums import TimeGranularity from slayer.core.query import TimeDimension - engine = await _hostile_engine(column=column) + engine = await _hostile_engine( + column=column, base_dir=str(tmp_path_factory.mktemp("hostile")), + ) resp = await engine.execute( SlayerQuery( source_model="orders", @@ -944,25 +983,22 @@ def test_source_relation_is_required_by_the_cross_model_profile( """Profile validation makes the impossible combinations unrepresentable — the reason this is a profile enum rather than four free-floating boolean flags.""" + key = _key(ColumnKey(leaf="revenue"), "sum") with pytest.raises(ValueError): - naming.canonical_aggregate_alias( - _key(ColumnKey(leaf="revenue"), "sum"), - profile="cross_model_cte", - ) + naming.canonical_aggregate_alias(key, profile="cross_model_cte") def test_source_relation_is_rejected_by_the_other_profiles(self) -> None: + key = _key(ColumnKey(leaf="revenue"), "sum") for profile in ("cte_schema", "declared_name", "stage_formula"): with pytest.raises(ValueError): naming.canonical_aggregate_alias( - _key(ColumnKey(leaf="revenue"), "sum"), - profile=profile, source_relation="orders", + key, profile=profile, source_relation="orders", ) def test_unknown_profile_is_rejected(self) -> None: + key = _key(ColumnKey(leaf="revenue"), "sum") with pytest.raises(ValueError): - naming.canonical_aggregate_alias( - _key(ColumnKey(leaf="revenue"), "sum"), profile="nonsense", - ) + naming.canonical_aggregate_alias(key, profile="nonsense") class TestProductionCallersDelegate: diff --git a/tests/test_dev1744_result_key_contract.py b/tests/test_dev1744_result_key_contract.py index fc0858cb..bfa14262 100644 --- a/tests/test_dev1744_result_key_contract.py +++ b/tests/test_dev1744_result_key_contract.py @@ -37,7 +37,6 @@ import os import sqlite3 -import tempfile from typing import AsyncIterator, List import pytest @@ -61,8 +60,8 @@ @pytest.fixture -async def engine() -> AsyncIterator[SlayerQueryEngine]: - d = tempfile.mkdtemp() +async def engine(tmp_path_factory) -> AsyncIterator[SlayerQueryEngine]: + d = str(tmp_path_factory.mktemp("contract")) db_path = os.path.join(d, "contract.db") con = sqlite3.connect(db_path) cur = con.cursor() diff --git a/tests/test_dev1744_value_expr.py b/tests/test_dev1744_value_expr.py index afa46a8c..4d3e7651 100644 --- a/tests/test_dev1744_value_expr.py +++ b/tests/test_dev1744_value_expr.py @@ -48,7 +48,6 @@ import os import sqlite3 -import tempfile from decimal import Decimal from typing import AsyncIterator, Optional @@ -56,6 +55,10 @@ from sqlglot import exp from slayer.core.enums import BUILTIN_AGGREGATIONS, DataType +from slayer.core.errors import ( + RenderContextMissingFacilityError, + UnknownReferenceError, +) from slayer.core.keys import ( AggregateKey, ArithmeticKey, @@ -65,6 +68,7 @@ InKey, LiteralKey, ScalarCallKey, + SqlExprKey, StarKey, TimeTruncKey, TransformKey, @@ -82,6 +86,15 @@ from slayer.engine.source_bundle import ResolvedSourceBundle from slayer.sql.dialects import get_dialect from slayer.sql.naming import AliasAllocator +from slayer.sql.render.aggregates import resolve_agg_entry, window_agg_class +from slayer.sql.render.value_expr import ( + AliasFacilities, + CompositeFacilities, + FilterFacilities, + RenderContext, + _literal, + render_value_key, +) from slayer.sql.scope import ScopeFrame from slayer.storage.yaml_storage import YAMLStorage @@ -150,7 +163,6 @@ def _scope( def _filter_ctx(dialect: str = "postgres", **kw): """A RenderContext carrying the FILTER facility group (R1's call family).""" - from slayer.sql.render.value_expr import FilterFacilities, RenderContext scope = _scope(dialect=dialect) return RenderContext( @@ -162,7 +174,6 @@ def _filter_ctx(dialect: str = "postgres", **kw): def _composite_ctx(dialect: str = "postgres", **kw): """A RenderContext carrying the COMPOSITE facility group (R5's family).""" - from slayer.sql.render.value_expr import CompositeFacilities, RenderContext scope = _scope(dialect=dialect) return RenderContext( @@ -188,24 +199,20 @@ class TestB10UnknownModelRaises: query runs and returns numbers computed from the wrong model.""" def test_unknown_model_in_columnsqlkey_raises(self) -> None: - from slayer.core.errors import UnknownReferenceError scope = _scope() + key = ColumnSqlKey(model="not_in_bundle", column_name="net") with pytest.raises(UnknownReferenceError): - scope.resolve( - ColumnSqlKey(model="not_in_bundle", column_name="net"), - ) + scope.resolve(key) def test_error_names_the_missing_model(self) -> None: """The message must be actionable: which model was asked for, what the scope root is, and what the bundle actually knows.""" - from slayer.core.errors import UnknownReferenceError scope = _scope() + key = ColumnSqlKey(model="not_in_bundle", column_name="net") with pytest.raises(UnknownReferenceError) as excinfo: - scope.resolve( - ColumnSqlKey(model="not_in_bundle", column_name="net"), - ) + scope.resolve(key) message = str(excinfo.value) assert "not_in_bundle" in message assert "orders" in message @@ -244,19 +251,18 @@ def test_context_holds_real_production_objects(self) -> None: """Pydantic v2 + a ``ScopeFrame`` / dialect strategy / sqlglot nodes needs ``arbitrary_types_allowed``; constructing with the real objects (not stubs) is what proves the config is right.""" - from slayer.sql.render.value_expr import RenderContext scope = _scope() ctx = RenderContext(scope=scope, dialect=scope.dialect) assert ctx.scope is scope assert ctx.consumer is None - assert ctx.filters is None and ctx.composites is None + assert ctx.filters is None + assert ctx.composites is None assert ctx.aliases is None def test_consumer_defaults_to_none_and_is_accepted(self) -> None: """The P-B seam exists in PR 1 even though its production callers arrive in PR 3.""" - from slayer.sql.render.value_expr import RenderContext producer, consumer = _scope(), _scope() ctx = RenderContext( @@ -281,7 +287,6 @@ def test_consumer_routes_column_like_leaves_through_materialization( Parametrised over all three column-like kinds because a renderer that special-cases one of them would otherwise slip through.""" - from slayer.sql.render.value_expr import RenderContext, render_value_key producer, consumer = _scope(), _scope() ctx = RenderContext( @@ -297,7 +302,6 @@ def test_materializations_apply_to_the_producing_select(self) -> None: """The other half of the P-B contract: what the renderer records must actually be projectable via ``apply_materializations``, so the consumer's bare alias resolves to a real column of the producing SELECT.""" - from slayer.sql.render.value_expr import RenderContext, render_value_key producer, consumer = _scope(), _scope() ctx = RenderContext( @@ -316,7 +320,6 @@ def test_materialization_dedups_within_a_scope(self) -> None: """Two renders of the same key across the same boundary share ONE ``_val_`` — the dedup key is the producing scope + anchored AST + dialect, and the renderer must not defeat it by re-anchoring.""" - from slayer.sql.render.value_expr import RenderContext, render_value_key producer, consumer = _scope(), _scope() ctx = RenderContext( @@ -331,7 +334,6 @@ def test_join_paths_register_as_a_side_effect_of_rendering(self) -> None: """P-A: join discovery is a side effect of rendering, never a separate pass. Rendering a joined leaf must register the crossed path on the scope without the caller asking.""" - from slayer.sql.render.value_expr import RenderContext, render_value_key scope = _scope() ctx = RenderContext(scope=scope, dialect=scope.dialect) @@ -346,8 +348,6 @@ def test_missing_facility_fails_closed(self) -> None: """A key kind that needs a facility the context lacks must RAISE, not silently degrade. Silent degradation is how the five copies drifted in the first place.""" - from slayer.core.errors import RenderContextMissingFacilityError - from slayer.sql.render.value_expr import RenderContext, render_value_key scope = _scope() bare = RenderContext(scope=scope, dialect=scope.dialect) @@ -361,8 +361,6 @@ def test_missing_facility_fails_closed(self) -> None: render_value_key(key, bare) def test_missing_facility_error_names_the_key_and_facility(self) -> None: - from slayer.core.errors import RenderContextMissingFacilityError - from slayer.sql.render.value_expr import RenderContext, render_value_key scope = _scope() bare = RenderContext(scope=scope, dialect=scope.dialect) @@ -388,8 +386,6 @@ def test_aggregate_without_composite_facilities_fails_closed(self) -> None: ``AggregateKey`` needs the composite facilities (rn-suffix maps, resolved agg kwargs, composite alias map) to render faithfully. """ - from slayer.core.errors import RenderContextMissingFacilityError - from slayer.sql.render.value_expr import RenderContext, render_value_key scope = _scope() bare = RenderContext(scope=scope, dialect=scope.dialect) @@ -398,6 +394,37 @@ def test_aggregate_without_composite_facilities_fails_closed(self) -> None: render_value_key(key, bare) assert "composite" in str(excinfo.value).lower(), str(excinfo.value) + def test_filtered_aggregate_without_a_builder_fails_closed(self) -> None: + """A column filter must not vanish. + + The generator wraps a filtered aggregate as + ``SUM(CASE WHEN THEN col END)``. Rendering it from ``agg`` and + ``source`` alone drops the filter and covers rows it must exclude — + a wrong number rather than an error, so the no-builder path refuses it. + """ + + key = AggregateKey( + source=ColumnKey(leaf="amount"), + agg="sum", + column_filter_key=SqlExprKey(canonical_sql="status = 'new'"), + ) + ctx = _composite_ctx() + with pytest.raises(RenderContextMissingFacilityError): + render_value_key(key, ctx) + + def test_parametric_aggregate_without_a_builder_fails_closed(self) -> None: + """Same rule for args/kwargs, which need the generator's parameter + resolution.""" + + key = AggregateKey( + source=ColumnKey(leaf="amount"), + agg="sum", + kwargs=(("window", "90d"),), + ) + ctx = _composite_ctx() + with pytest.raises(RenderContextMissingFacilityError): + render_value_key(key, ctx) + def test_transform_key_renders_when_alias_facilities_are_supplied( self, ) -> None: @@ -408,11 +435,6 @@ def test_transform_key_renders_when_alias_facilities_are_supplied( WITH its facility must render here — otherwise "fails closed" would be indistinguishable from "not implemented". """ - from slayer.sql.render.value_expr import ( - AliasFacilities, - RenderContext, - render_value_key, - ) scope = _scope() agg = AggregateKey(source=ColumnKey(leaf="amount"), agg="sum") @@ -440,13 +462,11 @@ class TestRendersEveryKeyKind: a bare ``None`` or a stringified repr.""" def test_local_column_key(self) -> None: - from slayer.sql.render.value_expr import render_value_key out = render_value_key(ColumnKey(leaf="amount"), _filter_ctx()) assert _sql(out) == "orders.amount" def test_joined_column_key_anchors_at_the_path_alias(self) -> None: - from slayer.sql.render.value_expr import render_value_key out = render_value_key( ColumnKey(path=("customers",), leaf="balance"), _filter_ctx(), @@ -454,7 +474,6 @@ def test_joined_column_key_anchors_at_the_path_alias(self) -> None: assert _sql(out) == "customers.balance" def test_multi_hop_column_key(self) -> None: - from slayer.sql.render.value_expr import render_value_key out = render_value_key( ColumnKey(path=("customers", "regions"), leaf="name"), @@ -465,7 +484,6 @@ def test_multi_hop_column_key(self) -> None: def test_column_sql_key_expands_the_derived_expression(self) -> None: """Exact SQL, not a substring check: ``net`` is ``amount - 1``, and the expansion must be anchored at the scope root.""" - from slayer.sql.render.value_expr import render_value_key out = render_value_key( ColumnSqlKey(model="orders", column_name="net"), _filter_ctx(), @@ -475,7 +493,6 @@ def test_column_sql_key_expands_the_derived_expression(self) -> None: def test_time_trunc_key(self) -> None: """Exact per-dialect SQL — a substring check would accept a truncation at the wrong granularity or over the wrong column.""" - from slayer.sql.render.value_expr import render_value_key key = TimeTruncKey( column=ColumnKey(leaf="created_at"), granularity="month", @@ -483,8 +500,41 @@ def test_time_trunc_key(self) -> None: out = render_value_key(key, _filter_ctx("postgres")) assert _sql(out, "postgres") == "DATE_TRUNC('MONTH', orders.created_at)" + @pytest.mark.parametrize("dialect", ["postgres", "sqlite", "tsql", "bigquery"]) + def test_time_trunc_goes_through_the_dialect_strategy(self, dialect) -> None: + """Truncation must use the dialect's own wire form, not a literal + ``DATE_TRUNC``. + + SQLite has no ``DATE_TRUNC`` — it needs ``STRFTIME`` — and T-SQL spells + it ``DATETRUNC``. Emitting the Postgres form everywhere produces SQL + the backend rejects, which is the same one-construct-two-renderings + defect this module exists to remove. + """ + + key = TimeTruncKey( + column=ColumnKey(leaf="created_at"), granularity="month", + ) + out = _sql(render_value_key(key, _filter_ctx(dialect)), dialect) + if dialect == "sqlite": + assert "DATE_TRUNC" not in out.upper(), out + assert "STRFTIME" in out.upper(), out + assert "created_at" in out, out + + def test_week_sunday_granularity_renders(self) -> None: + """``week_sunday`` is a supported granularity with its own day-shift. + + A hardcoded unit table would have no entry for it and would emit + ``DATE_TRUNC('WEEK_SUNDAY', col)``, which no dialect accepts. + """ + + key = TimeTruncKey( + column=ColumnKey(leaf="created_at"), granularity="week_sunday", + ) + out = _sql(render_value_key(key, _filter_ctx("postgres")), "postgres") + assert "WEEK_SUNDAY" not in out.upper(), out + assert "created_at" in out, out + def test_literal_key_variants(self) -> None: - from slayer.sql.render.value_expr import render_value_key ctx = _filter_ctx() assert _sql(render_value_key(LiteralKey(value=Decimal(3)), ctx)) == "3" @@ -494,13 +544,11 @@ def test_literal_key_variants(self) -> None: ).upper() == "NULL" def test_star_key(self) -> None: - from slayer.sql.render.value_expr import render_value_key out = render_value_key(StarKey(), _filter_ctx()) assert isinstance(out, exp.Star) def test_arithmetic_key(self) -> None: - from slayer.sql.render.value_expr import render_value_key out = render_value_key( ArithmeticKey( @@ -512,7 +560,6 @@ def test_arithmetic_key(self) -> None: assert _sql(out) == "orders.amount + 1" def test_comparison_arithmetic_key(self) -> None: - from slayer.sql.render.value_expr import render_value_key out = render_value_key( ArithmeticKey( @@ -524,7 +571,6 @@ def test_comparison_arithmetic_key(self) -> None: assert _sql(out) == "orders.amount > 5" def test_between_key(self) -> None: - from slayer.sql.render.value_expr import render_value_key out = render_value_key( BetweenKey( @@ -537,7 +583,6 @@ def test_between_key(self) -> None: assert _sql(out) == "orders.amount BETWEEN 1 AND 9" def test_in_key(self) -> None: - from slayer.sql.render.value_expr import render_value_key out = render_value_key( InKey( @@ -549,7 +594,6 @@ def test_in_key(self) -> None: assert _sql(out) == "orders.label IN ('a', 'b')" def test_negated_in_key(self) -> None: - from slayer.sql.render.value_expr import render_value_key out = render_value_key( InKey( @@ -564,7 +608,6 @@ def test_negated_in_key(self) -> None: assert _sql(out) == "NOT orders.label IN ('a')" def test_local_aggregate_key(self) -> None: - from slayer.sql.render.value_expr import render_value_key out = render_value_key( AggregateKey(source=ColumnKey(leaf="amount"), agg="sum"), @@ -573,13 +616,134 @@ def test_local_aggregate_key(self) -> None: assert _sql(out) == "SUM(orders.amount)" def test_star_count_aggregate_key(self) -> None: - from slayer.sql.render.value_expr import render_value_key out = render_value_key( AggregateKey(source=StarKey(), agg="count"), _composite_ctx(), ) assert _sql(out) == "COUNT(*)" + def test_unary_minus_keeps_its_sign(self) -> None: + """The binder represents ``-10`` as a SINGLE-operand ``ArithmeticKey``. + + A fold that starts at ``operands[0]`` and iterates ``operands[1:]`` + never runs its body for one operand and returns it unchanged, so + ``amount > -10`` would silently become ``amount > 10`` — a wrong + result, not a failure. + """ + + out = render_value_key( + ArithmeticKey(op="-", operands=(LiteralKey(value=Decimal(10)),)), + _filter_ctx(), + ) + assert _sql(out) == "-10" + + def test_unary_not(self) -> None: + + out = render_value_key( + ArithmeticKey( + op="not", + operands=( + ArithmeticKey( + op=">", + operands=( + ColumnKey(leaf="amount"), + LiteralKey(value=Decimal(5)), + ), + ), + ), + ), + _filter_ctx(), + ) + assert _sql(out) == "NOT orders.amount > 5" + + @pytest.mark.parametrize( + "key,expected", + [ + # (a + b) * c — sqlglot does NOT parenthesise by nesting, so + # without explicit Paren nodes this emits "a + b * c", which + # evaluates differently. + ( + ArithmeticKey( + op="*", + operands=( + ArithmeticKey( + op="+", + operands=( + ColumnKey(leaf="amount"), + LiteralKey(value=Decimal(1)), + ), + ), + LiteralKey(value=Decimal(2)), + ), + ), + "(orders.amount + 1) * 2", + ), + # a - (b - c): equal precedence on the RIGHT of a non-associative + # operator still needs parens. + ( + ArithmeticKey( + op="-", + operands=( + ColumnKey(leaf="amount"), + ArithmeticKey( + op="-", + operands=( + LiteralKey(value=Decimal(3)), + LiteralKey(value=Decimal(1)), + ), + ), + ), + ), + "orders.amount - (3 - 1)", + ), + # Higher-precedence child needs NO parens — don't over-wrap. + ( + ArithmeticKey( + op="+", + operands=( + ColumnKey(leaf="amount"), + ArithmeticKey( + op="*", + operands=( + LiteralKey(value=Decimal(2)), + LiteralKey(value=Decimal(3)), + ), + ), + ), + ), + "orders.amount + 2 * 3", + ), + ], + ids=["lower_prec_child", "right_of_non_associative", "higher_prec_child"], + ) + def test_arithmetic_precedence_is_parenthesised(self, key, expected) -> None: + """Operator precedence has to be materialised as ``Paren`` nodes.""" + + assert _sql(render_value_key(key, _filter_ctx())) == expected + + def test_unsupported_literal_type_raises(self) -> None: + """An unrecognised Python value must not become a quoted string. + + Asserted on the helper directly: the key types' own Pydantic validation + already rejects such a value, so this is defence in depth rather than a + reachable path. It matters because the generator's equivalent helper + raises, and the paths converge in a later PR — a silent + ``str(value)`` there would turn a loud failure into a wrong value. + """ + from datetime import datetime + + + with pytest.raises(NotImplementedError): + _literal(datetime(2024, 1, 1)) + + def test_supported_literal_types_still_render(self) -> None: + """The fail-closed branch must not swallow the supported cases.""" + + assert _literal(None).sql() == "NULL" + assert _literal(True).sql() == "TRUE" + assert _literal(Decimal("1.5")).sql() == "1.5" + assert _literal("x").sql() == "'x'" + def test_unhandled_kind_raises_notimplementederror(self) -> None: """Fail closed on anything outside the union rather than returning a stringified repr into the SQL. @@ -589,10 +753,10 @@ def test_unhandled_kind_raises_notimplementederror(self) -> None: accepting a tuple of types would let an incidental TypeError from somewhere else inside the renderer satisfy this test. """ - from slayer.sql.render.value_expr import render_value_key + ctx = _filter_ctx() with pytest.raises(NotImplementedError) as excinfo: - render_value_key(object(), _filter_ctx()) # type: ignore[arg-type] + render_value_key(object(), ctx) # type: ignore[arg-type] assert "object" in str(excinfo.value) @@ -631,7 +795,6 @@ class TestB5ScalarCallPolicy: def test_scalar_calls_transpile_per_dialect( self, name, args, expected_pg, expected_tsql, ) -> None: - from slayer.sql.render.value_expr import render_value_key key = ScalarCallKey(name=name, args=args) assert _sql( @@ -644,7 +807,6 @@ def test_scalar_calls_transpile_per_dialect( def test_ifnull_never_reaches_postgres_unmapped(self) -> None: """The headline B5 bug, stated as the invariant rather than as an exact string: Postgres has no ``IFNULL``, so emitting it is broken SQL.""" - from slayer.sql.render.value_expr import render_value_key key = ScalarCallKey( name="ifnull", @@ -663,7 +825,6 @@ def test_log10_keeps_the_native_single_arg_alias(self) -> None: ``_rewrite_log_aliases``. Applying transpile WITHOUT that rewrite would regress ``log10`` — so the unified renderer must apply both. """ - from slayer.sql.render.value_expr import render_value_key key = ScalarCallKey(name="log10", args=(ColumnKey(leaf="amount"),)) out = _sql(render_value_key(key, _filter_ctx("postgres")), "postgres") @@ -673,20 +834,19 @@ def test_round_keeps_the_dev1576_postgres_cast(self) -> None: """Parity guard: two-arg ROUND on Postgres needs the numeric cast, and it is the ONE scalar call R1 already routed through the typed path. Unifying must not lose it.""" - from slayer.sql.render.value_expr import render_value_key key = ScalarCallKey( name="round", args=(ColumnKey(leaf="amount"), LiteralKey(value=Decimal(0))), ) out = _sql(render_value_key(key, _filter_ctx("postgres")), "postgres") - assert "CAST" in out.upper() and "DECIMAL" in out.upper(), out + assert "CAST" in out.upper(), out + assert "DECIMAL" in out.upper(), out def test_like_stays_the_sql_operator(self) -> None: """``like(value, pattern)`` is the one allowlist member that is an OPERATOR, not a function call. Both legacy paths special-case it; the unified renderer keeps that.""" - from slayer.sql.render.value_expr import render_value_key key = ScalarCallKey( name="like", @@ -699,7 +859,6 @@ def test_like_stays_the_sql_operator(self) -> None: def test_nested_scalar_calls_use_one_policy_throughout(self) -> None: """The policy applies at every depth — a nested call must not fall back to the passthrough branch.""" - from slayer.sql.render.value_expr import render_value_key key = ScalarCallKey( name="ifnull", @@ -709,8 +868,10 @@ def test_nested_scalar_calls_use_one_policy_throughout(self) -> None: ), ) out = _sql(render_value_key(key, _filter_ctx("tsql")), "tsql") - assert "COALESCE" in out.upper() and "LEN(" in out.upper(), out - assert "IFNULL" not in out.upper() and "LENGTH" not in out.upper(), out + assert "COALESCE" in out.upper(), out + assert "LEN(" in out.upper(), out + assert "IFNULL" not in out.upper(), out + assert "LENGTH" not in out.upper(), out class TestPGSameConstructSameSql: @@ -753,7 +914,6 @@ class TestPGSameConstructSameSql: def test_same_key_same_sql_across_contexts( self, label, key, dialect, ) -> None: - from slayer.sql.render.value_expr import render_value_key in_filter = _sql( render_value_key(key, _filter_ctx(dialect)), dialect, @@ -798,7 +958,6 @@ class TestAggregationRegistry: } def test_every_builtin_resolves(self) -> None: - from slayer.sql.render.aggregates import resolve_agg_entry for name in sorted(BUILTIN_AGGREGATIONS): entry = resolve_agg_entry(name) @@ -817,7 +976,6 @@ def test_each_former_dispatch_mechanism_is_represented( implementation that only ported the easy ``_AGG_FUNCTION_MAP`` entries would still pass ``test_every_builtin_resolves`` if the enum happened to be small.""" - from slayer.sql.render.aggregates import resolve_agg_entry for name in names: entry = resolve_agg_entry(name) @@ -832,7 +990,6 @@ def test_required_names_are_really_builtins(self) -> None: assert name in BUILTIN_AGGREGATIONS, name def test_unknown_aggregation_raises(self) -> None: - from slayer.sql.render.aggregates import resolve_agg_entry with pytest.raises(ValueError): resolve_agg_entry("definitely_not_an_aggregation") @@ -841,7 +998,6 @@ def test_windowable_flags_are_exact(self) -> None: """Only ``sum`` and ``avg`` are windowable today — that is precisely what ``stage_planner`` gates on, and the registry must agree with it rather than restating it.""" - from slayer.sql.render.aggregates import resolve_agg_entry assert resolve_agg_entry("sum").windowable is True assert resolve_agg_entry("avg").windowable is True @@ -849,7 +1005,6 @@ def test_windowable_flags_are_exact(self) -> None: assert resolve_agg_entry(name).windowable is False, name def test_window_agg_class_replaces_the_hardcode(self) -> None: - from slayer.sql.render.aggregates import window_agg_class assert window_agg_class("sum") is exp.Sum assert window_agg_class("avg") is exp.Avg @@ -862,7 +1017,6 @@ def test_non_windowable_aggregation_fails_closed(self) -> None: Approved divergence: it raises instead. """ - from slayer.sql.render.aggregates import window_agg_class for name in ("median", "count", "min", "max", "percentile"): with pytest.raises(ValueError): @@ -874,8 +1028,8 @@ def test_non_windowable_aggregation_fails_closed(self) -> None: # =========================================================================== -async def _e2e_engine(*, dialect: str = "sqlite") -> SlayerQueryEngine: - d = tempfile.mkdtemp() +async def _e2e_engine(*, base_dir: str, dialect: str = "sqlite") -> SlayerQueryEngine: + d = base_dir db_path = os.path.join(d, "ve.db") con = sqlite3.connect(db_path) cur = con.cursor() @@ -924,8 +1078,8 @@ async def _e2e_engine(*, dialect: str = "sqlite") -> SlayerQueryEngine: @pytest.fixture -async def e2e() -> AsyncIterator[SlayerQueryEngine]: - yield await _e2e_engine() +async def e2e(tmp_path_factory) -> AsyncIterator[SlayerQueryEngine]: + yield await _e2e_engine(base_dir=str(tmp_path_factory.mktemp("ve"))) class TestMigratedCallSitesEndToEnd: @@ -1004,7 +1158,7 @@ async def test_having_filter_over_an_aggregate_executes(self, e2e) -> None: async def test_first_last_composite_call_site_executes(self, e2e) -> None: """R5's SECOND production call site (``_build_first_last_base_select``, - generator.py:3741) — reached only by a first/last measure, which builds + the generator) — reached only by a first/last measure, which builds its own ranked base SELECT rather than the ordinary composite one. Ordered by ``id``: new -> last amount 20, old -> 30.""" @@ -1041,12 +1195,12 @@ class TestOuterWrapperAndShiftedCteFamilies: * R4 ``_render_filter_for_outer_wrapper`` — the outer combined SELECT. R4 is NOT migrated in PR 1 (it is cross-scope, and moves in PR 3), but B5 says "everywhere", so its scalar branch IS patched here. - * R1's shifted-CTE WHERE call site (generator.py:7468) — reached only by a + * R1's shifted-CTE WHERE call site (the generator) — reached only by a ``time_shift`` transform, never by a plain host filter. """ - async def _engine(self, *, dialect: str = "sqlite") -> SlayerQueryEngine: - d = tempfile.mkdtemp() + async def _engine(self, tmp_path_factory, *, dialect: str = "sqlite") -> SlayerQueryEngine: + d = str(tmp_path_factory.mktemp("routes")) db_path = os.path.join(d, "routes.db") con = sqlite3.connect(db_path) cur = con.cursor() @@ -1115,13 +1269,13 @@ async def _engine(self, *, dialect: str = "sqlite") -> SlayerQueryEngine: return SlayerQueryEngine(storage=storage) async def test_outer_wrapper_filter_over_an_isolated_aggregate( - self, + self, tmp_path_factory, ) -> None: """R4's route: an AGGREGATE-phase filter on a filtered-local isolated aggregate renders as plain WHERE on the joined-back column of the outer combined SELECT. Gold totals: new = 300, old = 0/NULL — so a threshold of 100 keeps only ``new``.""" - engine = await self._engine() + engine = await self._engine(tmp_path_factory) resp = await engine.execute( SlayerQuery( source_model="orders", @@ -1133,11 +1287,11 @@ async def test_outer_wrapper_filter_over_an_isolated_aggregate( assert [r["orders.status"] for r in resp.data] == ["new"] assert resp.data[0]["orders.g"] == 300.0 - async def test_outer_wrapper_filter_carrying_a_scalar_call(self) -> None: + async def test_outer_wrapper_filter_carrying_a_scalar_call(self, tmp_path_factory) -> None: """B5 on R4's route specifically — the patched scalar branch. Wrapping the same comparison in ``ifnull`` must not change which rows survive, and (on a dialect without IFNULL) must not emit it.""" - engine = await self._engine() + engine = await self._engine(tmp_path_factory) resp = await engine.execute( SlayerQuery( source_model="orders", @@ -1150,7 +1304,7 @@ async def test_outer_wrapper_filter_carrying_a_scalar_call(self) -> None: assert resp.data[0]["orders.g"] == 300.0 async def test_outer_wrapper_scalar_call_is_transpiled_on_postgres( - self, + self, tmp_path_factory, ) -> None: """The emission half of the same case, and the B5 bug in its sharpest form: R4 passes scalar calls through as ``exp.Anonymous``, so the @@ -1163,7 +1317,7 @@ async def test_outer_wrapper_scalar_call_is_transpiled_on_postgres( assertion there would be a policy preference rather than a correctness claim. """ - engine = await self._engine(dialect="postgres") + engine = await self._engine(tmp_path_factory, dialect="postgres") resp = await engine.execute( SlayerQuery( source_model="orders", @@ -1176,8 +1330,8 @@ async def test_outer_wrapper_scalar_call_is_transpiled_on_postgres( assert "IFNULL" not in resp.sql.upper(), resp.sql assert "COALESCE" in resp.sql.upper(), resp.sql - async def test_shifted_cte_filter_call_site_executes(self) -> None: - """R1's SECOND call site (the ``time_shift`` CTE's WHERE, :7468). + async def test_shifted_cte_filter_call_site_executes(self, tmp_path_factory) -> None: + """R1's SECOND call site (the ``time_shift`` CTE's WHERE, that call site). A host filter must apply inside the shifted CTE as well as the host base, so the shifted value is computed over the same filtered rows. @@ -1187,7 +1341,7 @@ async def test_shifted_cte_filter_call_site_executes(self) -> None: from slayer.core.enums import TimeGranularity from slayer.core.query import TimeDimension - engine = await self._engine() + engine = await self._engine(tmp_path_factory) resp = await engine.execute( SlayerQuery( source_model="orders", From d6bd014c71d40cd66a30bca539d2ce69994cb5bb Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Wed, 5 Aug 2026 17:42:05 +0200 Subject: [PATCH 03/98] DEV-1744: document the scalar dialect semantics; drop the like duplicates MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The `concat` change is user-visible, so it belongs in the reference docs, not only in the decision log: on backends whose natural spelling is `||` (Postgres, DuckDB, SQLite) `concat(a, b)` propagates NULL, unlike those backends' own `CONCAT()`. Documented with the `ifnull(...)` workaround, next to the related `log10` / `log2` single-argument note. The five `if name == "like"` short-circuits ahead of `render_scalar_call` were dead weight — that function already special-cases the operator — so each call site was re-stating the policy the consolidation just centralised. Co-Authored-By: Claude Fable 5 --- docs/concepts/references.md | 24 ++++++++++++++++++++++++ slayer/sql/generator.py | 10 ---------- 2 files changed, 24 insertions(+), 10 deletions(-) diff --git a/docs/concepts/references.md b/docs/concepts/references.md index d13ff05e..91b0a308 100644 --- a/docs/concepts/references.md +++ b/docs/concepts/references.md @@ -114,6 +114,30 @@ Rejected at enrichment (when the formula is evaluated against a model): {"name": "bad", "formula": "json_extract(data, '$.x')"} // raw SQL fn ``` +## Scalar functions and dialect semantics + +The allowlisted scalars are rendered as typed SQL and then translated to each +backend's own spelling, so one formula stays correct across dialects rather +than being passed through verbatim. `length(x)` emits `LEN(x)` on SQL Server, +`substr(x, 1, 5)` emits `SUBSTRING(x FROM 1 FOR 5)` on Postgres, and +`ifnull(x, 0)` emits `COALESCE(x, 0)` on backends without `IFNULL`. + +Two consequences worth knowing: + +* **`concat` follows SQL string-concatenation semantics.** On dialects whose + natural spelling is the `||` operator (Postgres, DuckDB, SQLite), `concat(a, b)` + emits `a || b`, which yields `NULL` if either operand is `NULL`. That differs + from those backends' own `CONCAT()` function, which treats `NULL` as an empty + string. Wrap operands in `ifnull(...)` when you want the NULL-tolerant + behaviour: + + ```json + {"filters": ["concat(ifnull(first_name, ''), ifnull(last_name, '')) = 'AdaLovelace'"]} + ``` + +* **`log10` / `log2` keep their single-argument form** on the backends that + provide one, rather than becoming the generic two-argument `LOG(base, x)`. + ## See also * [Models](models.md) — `Column.sql`, `Column.filter`, model-level filters diff --git a/slayer/sql/generator.py b/slayer/sql/generator.py index 804a5c8a..f02e1e9f 100644 --- a/slayer/sql/generator.py +++ b/slayer/sql/generator.py @@ -3921,8 +3921,6 @@ def _render_aggregate_composite_expr( # NOSONAR(S3776) — sequential isinstanc args.append(exp.Literal.number(str(a))) else: args.append(exp.Literal.string(str(a))) - if key.name == "like": - return exp.Like(this=args[0], expression=args[1]), any_agg return render_scalar_call( name=key.name, args=args, dialect=self._dialect, ), any_agg @@ -6380,8 +6378,6 @@ def _render_filter_value_key_in_target_scope( # NOSONAR(S3776) — sequential i ) for a in value_key.args ] - if value_key.name == "like": - return exp.Like(this=rendered_args[0], expression=rendered_args[1]) return render_scalar_call( name=value_key.name, args=rendered_args, dialect=self._dialect, ) @@ -7187,8 +7183,6 @@ def recurse(k) -> exp.Expression: else _render_scalar_literal(a) for a in key.args ] - if key.name == "like": - return exp.Like(this=args[0], expression=args[1]) return render_scalar_call( name=key.name, args=args, dialect=self._dialect, ) @@ -9479,8 +9473,6 @@ def _render_value_key_for_filter( # NOSONAR(S3776) — sequential isinstance di first_last_state=first_last_state, aliases_by_slot_id=aliases_by_slot_id, )) - if key.name == "like": - return exp.Like(this=args[0], expression=args[1]) # One ScalarCall policy everywhere (B5): typed node, dialect # rewrite, then the log-alias fix-up. This branch used to return an # ``exp.Anonymous`` passthrough for everything but ROUND, so a @@ -9669,8 +9661,6 @@ def _slot_alias_column(slot) -> Optional[exp.Expression]: cross_model_agg_slot_to_cm=cross_model_agg_slot_to_cm, aliases_by_slot_id=aliases_by_slot_id, )) - if key.name == "like": - return exp.Like(this=args[0], expression=args[1]) # One ScalarCall policy everywhere (B5): typed node, dialect # rewrite, then the log-alias fix-up. This branch used to return an # ``exp.Anonymous`` passthrough for everything but ROUND, so a From 69c4103322d98f9b96abd6486e35b9ea8a4c9c35 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Wed, 5 Aug 2026 17:43:05 +0200 Subject: [PATCH 04/98] DEV-1744: drop unreachable None guards in the registry tests resolve_agg_entry raises for an unknown name, so the is-not-None assertions could never fail. The raising contract is pinned by its own test. Co-Authored-By: Claude Fable 5 --- tests/test_dev1744_value_expr.py | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/tests/test_dev1744_value_expr.py b/tests/test_dev1744_value_expr.py index 4d3e7651..d6e8ee53 100644 --- a/tests/test_dev1744_value_expr.py +++ b/tests/test_dev1744_value_expr.py @@ -960,9 +960,7 @@ class TestAggregationRegistry: def test_every_builtin_resolves(self) -> None: for name in sorted(BUILTIN_AGGREGATIONS): - entry = resolve_agg_entry(name) - assert entry is not None, name - assert entry.name == name + assert resolve_agg_entry(name).name == name @pytest.mark.parametrize( "mechanism,names", @@ -978,9 +976,7 @@ def test_each_former_dispatch_mechanism_is_represented( to be small.""" for name in names: - entry = resolve_agg_entry(name) - assert entry is not None, f"{mechanism}: {name} does not resolve" - assert entry.name == name + assert resolve_agg_entry(name).name == name, mechanism def test_required_names_are_really_builtins(self) -> None: """Guard on the frozen table's own premise, so a rename in the enum From 01717e7e8d6d5d7aca908753a8482e0563abbc23 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Wed, 5 Aug 2026 17:52:30 +0200 Subject: [PATCH 05/98] =?UTF-8?q?DEV-1744:=20add=20the=20missing=20tests?= =?UTF-8?q?=20=E2=80=94=20and=20one=20proves=20a=20wrong-answer=20bug?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three fixes from the last round shipped without a test. Closing that. The leaked `canonical_alias` is the important one. My first attempt at a test passed with AND without the fix, so it proved nothing; I said so rather than keeping it. Tracing the trim guard found the reachable shape: trim_hidden = plan.hidden and not planned_query.transform_layers A HIDDEN cross-model aggregate feeding a transform chain is therefore NOT trimmed, stays projected, and — having no user-declared name — is the one caller that reaches the canonical-alias fallback. `cumsum(customers.revenue:sum)` alongside a second cross-model measure hits it exactly. The emitted SQL without the fix: base AS (SELECT ..., _cm_..._revenue_sum."orders.customers.revenue_sum" AS "orders.customers.Rev_sum", _cm_..._Rev_sum."orders.customers.Rev_sum" AS "orders.rv" ...) step1 AS (SELECT ..., SUM("orders.customers.Rev_sum") OVER (...) AS "orders.run" ...) The hidden aggregate is projected under the OTHER measure's name, that alias is emitted twice, and the window function then sums the wrong measure. So this was a wrong-answer bug, not the cosmetic naming slip it looked like. The test asserts no output alias is emitted twice, which is what actually breaks, and was verified to fail with the fix reverted. Also added: the aggregation registry's both-ways membership invariant (a registry key that is not a built-in would make `is_builtin_agg` accept a typo), that the generator's log-alias rewrite agrees with the shared policy it now delegates to across four dialects, and that `_wm_` CTE names survive a case-only collision the same way `_cm_` now does. Tests: 9527 passing, ruff clean. Co-Authored-By: Claude Fable 5 --- tests/test_dev1744_naming_allocator.py | 108 +++++++++++++++++++++++-- tests/test_dev1744_value_expr.py | 53 ++++++++++++ 2 files changed, 156 insertions(+), 5 deletions(-) diff --git a/tests/test_dev1744_naming_allocator.py b/tests/test_dev1744_naming_allocator.py index 4848a7e6..bdc737d0 100644 --- a/tests/test_dev1744_naming_allocator.py +++ b/tests/test_dev1744_naming_allocator.py @@ -116,14 +116,14 @@ async def _build_engine(*, base_dir: str, dialect: str = "sqlite") -> SlayerQuer ) cur.execute( "CREATE TABLE orders (id INTEGER PRIMARY KEY, customer_id INTEGER, " - "status TEXT, amount REAL)" + "status TEXT, amount REAL, created_at TEXT)" ) cur.executemany( - "INSERT INTO orders VALUES (?,?,?,?)", + "INSERT INTO orders VALUES (?,?,?,?,?)", [ - (1, 1, "a", 10.0), - (2, 2, "a", 20.0), - (3, 3, "a", 30.0), + (1, 1, "a", 10.0, "2024-01-01"), + (2, 2, "a", 20.0, "2024-02-01"), + (3, 3, "a", 30.0, "2024-03-01"), ], ) con.commit() @@ -153,11 +153,13 @@ async def _build_engine(*, base_dir: str, dialect: str = "sqlite") -> SlayerQuer name="orders", sql_table="orders", data_source="prod", + default_time_dimension="created_at", columns=[ Column(name="id", type=DataType.INT, primary_key=True), Column(name="customer_id", type=DataType.INT), Column(name="status", type=DataType.TEXT), Column(name="amount", type=DataType.DOUBLE), + Column(name="created_at", type=DataType.TIMESTAMP), # __ in a Column.name (keep-list carve-out) + a join-crossing # filter => filtered-local isolation => a _cm_ CTE. Column( @@ -383,6 +385,63 @@ async def test_unrenamed_cross_model_measures_keep_their_own_aliases( # equal values would mean both read the same CTE column. assert row["orders.customers.revenue_sum"] != row["orders.customers.Rev_sum"] + async def test_hidden_cross_model_agg_under_a_transform_keeps_its_alias( + self, engine, + ) -> None: + """A HIDDEN cross-model aggregate feeding a transform layer must + project under ITS OWN canonical alias. + + This is the ONE shape that reaches the canonical-alias fallback in + ``_public_aliases_for_cross_model_agg``. The projection loop trims a + hidden aggregate only when there is no transform chain + (``plan.hidden and not transform_layers``); with a chain the hidden + aggregate stays projected so the step CTE can consume it, and — having + no user-declared name — its ``public_aliases`` is empty, so the alias + falls back to the plan's canonical one. + + Reading a stale value there emits + ``_cm_..._revenue_sum."orders.customers.revenue_sum" AS + "orders.customers.revx_sum"``: the hidden aggregate is projected under + the OTHER measure's name, that alias appears twice in one SELECT, and + the step CTE binds the wrong column. Asserted as "no output alias is + emitted twice", which is what actually breaks. + """ + from collections import Counter + from slayer.core.enums import TimeGranularity + from slayer.core.query import TimeDimension + + resp = await engine.execute( + SlayerQuery( + source_model="orders", + time_dimensions=[ + TimeDimension( + dimension=ColumnRef(name="created_at"), + granularity=TimeGranularity.MONTH, + ), + ], + measures=[ + # Hidden CMA (customers.revenue:sum) feeding a transform. + ModelMeasure( + formula="cumsum(customers.revenue:sum)", name="run", + ), + # A second cross-model aggregate, so a leaked alias differs + # from the correct one. + ModelMeasure(formula="customers.Rev:sum", name="rv"), + ], + ), + dry_run=True, + ) + assert resp.sql is not None + alias_counts = Counter(re.findall(r'AS "([^"]+)"', resp.sql)) + duplicated = {a: n for a, n in alias_counts.items() if n > 1} + assert not duplicated, ( + f"an output alias is emitted more than once — a cross-model " + f"aggregate was projected under another measure's name: " + f"{duplicated}\n{resp.sql}" + ) + # …and the hidden aggregate still carries its own canonical alias. + assert "orders.customers.revenue_sum" in alias_counts, alias_counts + async def test_same_key_slots_still_share_one_cte(self, engine) -> None: """Parity guard for the C13 intent the buggy dedup was meant to serve: two measures that are the SAME aggregate under different public names @@ -519,6 +578,45 @@ def test_cte_name_from_alias_is_exact_on_non_folding_dialects(self) -> None: assert a == "_cm_orders__Rev_sum" assert b == "_cm_orders__rev_sum" + async def test_windowed_cte_names_use_the_shared_helper( + self, tmp_path_factory, + ) -> None: + """``_wm_`` names go through the same sanitise-and-allocate primitive as + ``_cm_``, so both CTE families behave identically under a case-only + collision rather than one being retrofitted and the other not. + + Two windowed measures over case-only column variants would otherwise + emit two names that fold together on a folding dialect — the exact + shape that broke ``_cm_``. + """ + from slayer.core.enums import TimeGranularity + from slayer.core.query import TimeDimension + + engine = await _hostile_engine( + column="revx2", base_dir=str(tmp_path_factory.mktemp("wm")), + ) + resp = await engine.execute( + SlayerQuery( + source_model="orders", + time_dimensions=[ + TimeDimension( + dimension=ColumnRef(name="created_at"), + granularity=TimeGranularity.MONTH, + ), + ], + measures=[ + ModelMeasure(formula="amount:sum(window='90d')", name="Wm"), + ModelMeasure(formula="amount:sum(window='30d')", name="wm"), + ], + ), + dry_run=True, + ) + for scope_names in _cte_names_by_scope(resp.sql): + folded = [n.lower() for n in scope_names] + assert len(folded) == len(set(folded)), scope_names + wm_names = [n for n in _cte_names(resp.sql) if n.startswith("_wm_")] + assert len(wm_names) == 2, wm_names + def test_no_raw_step_cte_names_in_the_generator(self) -> None: """P-F, checked structurally because it has no reachable behavioural difference today: the three ``f"step{...}"`` mint sites must all go diff --git a/tests/test_dev1744_value_expr.py b/tests/test_dev1744_value_expr.py index d6e8ee53..0fde6447 100644 --- a/tests/test_dev1744_value_expr.py +++ b/tests/test_dev1744_value_expr.py @@ -874,6 +874,47 @@ def test_nested_scalar_calls_use_one_policy_throughout(self) -> None: assert "LENGTH" not in out.upper(), out +class TestLogAliasPolicyIsShared: + """The log-alias rule lives in ONE place. + + The generator had its own copy of exactly this rule — same ``exp.Log`` + guard, same literal-base checks, same ``Anonymous`` output. Two copies of a + policy the module docstring calls load-bearing is the drift this PR exists + to remove, so the generator delegates and these tests pin that it still + agrees with the shared implementation. + """ + + @pytest.mark.parametrize("dialect", ["postgres", "sqlite", "tsql", "bigquery"]) + def test_generator_delegates_to_the_shared_policy(self, dialect) -> None: + from slayer.sql.generator import SQLGenerator + from slayer.sql.render.value_expr import rewrite_log_alias + + gen = SQLGenerator(dialect=dialect) + node = exp.Log( + this=exp.Literal.number("10"), expression=exp.column("x"), + ) + via_generator = gen._rewrite_log_aliases(node.copy()) + via_shared = rewrite_log_alias(node.copy(), dialect=gen._dialect) + assert via_generator.sql(dialect=dialect) == via_shared.sql(dialect=dialect) + + def test_generator_parse_path_still_emits_native_log10(self) -> None: + """Behavioural companion: the delegation must not lose the rewrite that + the generator applies over parsed trees.""" + from slayer.sql.generator import SQLGenerator + + gen = SQLGenerator(dialect="postgres") + out = gen._parse("log10(x)").sql(dialect="postgres") + assert out.upper().startswith("LOG10("), out + + def test_non_log_nodes_pass_through_untouched(self) -> None: + from slayer.sql.render.value_expr import rewrite_log_alias + + from slayer.sql.dialects import get_dialect + + node = exp.column("x") + assert rewrite_log_alias(node, dialect=get_dialect("postgres")) is node + + class TestPGSameConstructSameSql: """P-G proper: a given ValueKey renders identically wherever it appears. @@ -1005,6 +1046,18 @@ def test_window_agg_class_replaces_the_hardcode(self) -> None: assert window_agg_class("sum") is exp.Sum assert window_agg_class("avg") is exp.Avg + def test_registry_and_builtins_agree_both_ways(self) -> None: + """The import-time invariant, asserted in both directions. + + A missing built-in would fall through to the custom-formula path. A + registry key that is NOT a built-in — a typo such as ``sumn`` — is the + subtler half: ``is_builtin_agg`` would accept it and route it AWAY from + that path, so the typo would render as if it were a real aggregation. + """ + from slayer.sql.render.aggregates import AGG_REGISTRY + + assert set(AGG_REGISTRY) == set(BUILTIN_AGGREGATIONS) + def test_non_windowable_aggregation_fails_closed(self) -> None: """The generator's windowed path currently reads ``exp.Sum if plan.agg == "sum" else exp.Avg`` — a silent catch-all that From a6b552474f359ac81631a3885f40708c34905b8a Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Wed, 5 Aug 2026 18:10:04 +0200 Subject: [PATCH 06/98] =?UTF-8?q?DEV-1744:=20review=20round=202=20?= =?UTF-8?q?=E2=80=94=20cross-model=20star=20fails=20closed?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A cross-model star is another wrong-number path on the no-builder branch. `StarKey.path` is non-empty for `customers.*:count`, and the branch emitted a bare `exp.Star()` for every StarKey — dropping the hop, so the count would cover HOST rows instead of the joined relation. Same failure class as the column-filter and kwargs guards, so it gets the same treatment: routing a cross-model star needs the join graph, so the no-builder path refuses it. Tested both ways — the cross-model star raises (verified failing with the guard reverted), and the ordinary local `*:count` still renders `COUNT(*)`. Also: `build_date_trunc` operands passed by keyword, one constant for the repeated facility name (S1192), one more `pytest.raises` narrowed to a single throwing call (S5778), and a composite assertion split (S9073). Tests 9529 passing, ruff clean, CI green, no Sonar issues outstanding. Co-Authored-By: Claude Fable 5 --- slayer/sql/render/value_expr.py | 31 ++++++++++++++++++++------ tests/test_dev1744_naming_allocator.py | 3 ++- tests/test_dev1744_value_expr.py | 20 ++++++++++++++++- 3 files changed, 45 insertions(+), 9 deletions(-) diff --git a/slayer/sql/render/value_expr.py b/slayer/sql/render/value_expr.py index 85d5f119..ad7a2734 100644 --- a/slayer/sql/render/value_expr.py +++ b/slayer/sql/render/value_expr.py @@ -93,6 +93,11 @@ } +# Named once: the render context field that carries the generator's aggregate +# builder, cited by every fail-closed guard below. +_AGG_BUILDER = "composites.agg_builder" + + class FilterFacilities(BaseModel): """What WHERE / HAVING rendering needs beyond the scope.""" @@ -313,14 +318,14 @@ def _render_aggregate(key: AggregateKey, ctx: RenderContext) -> exp.Expression: if not is_builtin_agg(key.agg): raise RenderContextMissingFacilityError( key_kind=type(key).__name__, - facility="composites.agg_builder", + facility=_AGG_BUILDER, detail=f"custom aggregation {key.agg!r} needs the generator's builder", ) entry = resolve_agg_entry(key.agg) if entry.dispatch not in (DISPATCH_SIMPLE, DISPATCH_DISTINCT): raise RenderContextMissingFacilityError( key_kind=type(key).__name__, - facility="composites.agg_builder", + facility=_AGG_BUILDER, detail=( f"aggregation {key.agg!r} renders via the {entry.dispatch!r} " f"mechanism, which needs the generator's builder" @@ -333,7 +338,7 @@ def _render_aggregate(key: AggregateKey, ctx: RenderContext) -> exp.Expression: if key.column_filter_key is not None: raise RenderContextMissingFacilityError( key_kind=type(key).__name__, - facility="composites.agg_builder", + facility=_AGG_BUILDER, detail=( "the aggregate's source carries a column filter, which needs " "the generator's CASE-WHEN wrapper" @@ -342,20 +347,32 @@ def _render_aggregate(key: AggregateKey, ctx: RenderContext) -> exp.Expression: if key.kwargs or key.args: raise RenderContextMissingFacilityError( key_kind=type(key).__name__, - facility="composites.agg_builder", + facility=_AGG_BUILDER, detail=( f"aggregation {key.agg!r} carries args/kwargs, which need the " f"generator's parameter resolution" ), ) if isinstance(key.source, StarKey): + if key.source.path: + # ``customers.*:count`` counts rows of the JOINED relation, which + # needs the join graph. A bare ``*`` here would count host rows — + # a wrong number, same class as the two guards above. + raise RenderContextMissingFacilityError( + key_kind=type(key).__name__, + facility=_AGG_BUILDER, + detail=( + f"cross-model star over path {key.source.path!r} needs the " + f"generator's join-graph routing" + ), + ) inner: exp.Expression = exp.Star() else: inner = ctx.scope.resolve(key.source, consumer=ctx.consumer) if entry.node_class is None: # pragma: no cover — dispatch gate guarantees it raise RenderContextMissingFacilityError( key_kind=type(key).__name__, - facility="composites.agg_builder", + facility=_AGG_BUILDER, detail=f"aggregation {key.agg!r} has no direct sqlglot node", ) if entry.dispatch == DISPATCH_DISTINCT: @@ -383,8 +400,8 @@ def render_value_key( # NOSONAR(S3776) — sequential dispatch over the closed # BigQuery, plus the WEEK_SUNDAY day-shift. Emitting a literal # DATE_TRUNC here would name a function SQLite does not have. return ctx.dialect.build_date_trunc( - column, - TimeGranularity(key.granularity), + col_expr=column, + granularity=TimeGranularity(key.granularity), parse=lambda sql: sqlglot.parse_one( sql, dialect=ctx.dialect.sqlglot_name, ), diff --git a/tests/test_dev1744_naming_allocator.py b/tests/test_dev1744_naming_allocator.py index bdc737d0..a29129e5 100644 --- a/tests/test_dev1744_naming_allocator.py +++ b/tests/test_dev1744_naming_allocator.py @@ -1165,7 +1165,8 @@ def _spy(k, **kw): SQLGenerator(dialect="postgres")._canonical_cross_model_alias( source_relation="orders", key=key, ) - assert calls and calls[-1].get("profile") == "cross_model_cte" + assert calls, "the generator did not delegate to the naming module" + assert calls[-1].get("profile") == "cross_model_cte" assert calls[-1].get("source_relation") == "orders" cross_model_planner._aggregate_alias(key=key) diff --git a/tests/test_dev1744_value_expr.py b/tests/test_dev1744_value_expr.py index 0fde6447..7f32136b 100644 --- a/tests/test_dev1744_value_expr.py +++ b/tests/test_dev1744_value_expr.py @@ -425,6 +425,23 @@ def test_parametric_aggregate_without_a_builder_fails_closed(self) -> None: with pytest.raises(RenderContextMissingFacilityError): render_value_key(key, ctx) + def test_cross_model_star_without_a_builder_fails_closed(self) -> None: + """``customers.*:count`` counts rows of the JOINED relation. + + ``StarKey.path`` carries that hop, and routing it needs the join graph. + Emitting a bare ``*`` would count HOST rows instead — a wrong number, + the same failure class as a dropped column filter. + """ + key = AggregateKey(source=StarKey(path=("customers",)), agg="count") + ctx = _composite_ctx() + with pytest.raises(RenderContextMissingFacilityError): + render_value_key(key, ctx) + + def test_local_star_still_renders(self) -> None: + """The guard must not catch the ordinary local ``*:count``.""" + key = AggregateKey(source=StarKey(), agg="count") + assert _sql(render_value_key(key, _composite_ctx())) == "COUNT(*)" + def test_transform_key_renders_when_alias_facilities_are_supplied( self, ) -> None: @@ -719,7 +736,8 @@ def test_unary_not(self) -> None: def test_arithmetic_precedence_is_parenthesised(self, key, expected) -> None: """Operator precedence has to be materialised as ``Paren`` nodes.""" - assert _sql(render_value_key(key, _filter_ctx())) == expected + ctx = _filter_ctx() + assert _sql(render_value_key(key, ctx)) == expected def test_unsupported_literal_type_raises(self) -> None: """An unrecognised Python value must not become a quoted string. From 26eb329508b339ccd5295c20e47cc97140c92a6d Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Wed, 5 Aug 2026 18:25:22 +0200 Subject: [PATCH 07/98] =?UTF-8?q?DEV-1744:=20sweep=20for=20silently-droppe?= =?UTF-8?q?d=20key=20fields=20=E2=80=94=20finds=20a=207th=20instance?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every renderer defect this PR's review surfaced was one shape: a typed key carries a field, a render path ignores it, and the query returns a WRONG NUMBER rather than failing. Dropped column filter, dropped kwargs, dropped star path, dropped unary sign, lost operator precedence, stale alias. That generalises into a property: changing any field of a key must change the emitted SQL, or the render must refuse. Two materially different keys that render identically means the field vanished. `TestNoKeyFieldIsSilentlyIgnored` sweeps ~21 single-field mutations across the union and asserts exactly that, with "raises" counting as a pass — refusing to render a key the context cannot honour is the fail-closed contract. It immediately found an instance the review did not: the TOP-LEVEL StarKey branch dropped `path` the same way the aggregate branch did. Reviewers flagged the aggregate one; the bare `StarKey(path=("customers",))` still rendered as `*`. Now guarded identically. Also swept the LIVE legacy filter renderer with the same mutations. One apparent hit — `AggregateKey.kwargs` — is a probe artifact, not a bug: calling that renderer directly with an empty `slot_by_key` takes a degenerate path that production never reaches. Verified end-to-end that a HAVING over `percentile(p=0.1)` vs `p=0.9` emits different SQL and returns different rows. Tests 9550 passing, ruff clean. Co-Authored-By: Claude Fable 5 --- slayer/sql/render/value_expr.py | 12 ++ tests/test_dev1744_value_expr.py | 181 +++++++++++++++++++++++++++++++ 2 files changed, 193 insertions(+) diff --git a/slayer/sql/render/value_expr.py b/slayer/sql/render/value_expr.py index ad7a2734..a42dd288 100644 --- a/slayer/sql/render/value_expr.py +++ b/slayer/sql/render/value_expr.py @@ -388,6 +388,18 @@ def render_value_key( # NOSONAR(S3776) — sequential dispatch over the closed return ctx.scope.resolve(key, consumer=ctx.consumer) if isinstance(key, StarKey): + # Same rule as the aggregate branch: a pathed star names the JOINED + # relation's rows, which needs the join graph. Emitting a bare ``*`` + # would silently count the host's. + if key.path: + raise RenderContextMissingFacilityError( + key_kind=type(key).__name__, + facility=_AGG_BUILDER, + detail=( + f"cross-model star over path {key.path!r} needs the " + f"generator's join-graph routing" + ), + ) return exp.Star() if isinstance(key, LiteralKey): diff --git a/tests/test_dev1744_value_expr.py b/tests/test_dev1744_value_expr.py index 7f32136b..3d0a6c17 100644 --- a/tests/test_dev1744_value_expr.py +++ b/tests/test_dev1744_value_expr.py @@ -1434,3 +1434,184 @@ async def test_shifted_cte_filter_call_site_executes(self, tmp_path_factory) -> assert rows[1]["orders.amt"] == 200.0 # February's shifted value is January's filtered total. assert rows[1]["orders.prev"] == 100.0 + + +# =========================================================================== +# Meta: no key field may be silently ignored. +# =========================================================================== + + +def _mutation_cases(): + """(label, base_key, mutated_key) triples — each pair differs in ONE field. + + Every renderer bug found while reviewing this PR was the same shape: the + key carried a field, a render path ignored it, and the output was silently + WRONG rather than an error. A dropped ``column_filter_key`` made an + aggregate cover rows the filter excluded; a dropped ``StarKey.path`` + counted host rows instead of the joined relation; a dropped unary operand + turned ``-10`` into ``10``. + + So the general invariant is: changing a field must change the rendered SQL + (or make the render refuse). Two keys that differ but render identically + means that field vanished. + """ + col = ColumnKey(leaf="amount") + other = ColumnKey(leaf="label") + return [ + ("ColumnKey.leaf", col, other), + ("ColumnKey.path", col, ColumnKey(path=("customers",), leaf="amount")), + ( + "ColumnSqlKey.column_name", + ColumnSqlKey(model="orders", column_name="net"), + ColumnSqlKey(model="orders", column_name="amount"), + ), + ( + "TimeTruncKey.granularity", + TimeTruncKey(column=col, granularity="month"), + TimeTruncKey(column=col, granularity="year"), + ), + ( + "TimeTruncKey.column", + TimeTruncKey(column=col, granularity="month"), + TimeTruncKey(column=other, granularity="month"), + ), + ("StarKey.path", StarKey(), StarKey(path=("customers",))), + ( + "LiteralKey.value", + LiteralKey(value=Decimal(1)), LiteralKey(value=Decimal(2)), + ), + ( + "ArithmeticKey.op", + ArithmeticKey(op="+", operands=(col, LiteralKey(value=Decimal(1)))), + ArithmeticKey(op="-", operands=(col, LiteralKey(value=Decimal(1)))), + ), + ( + "ArithmeticKey.operands", + ArithmeticKey(op="+", operands=(col, LiteralKey(value=Decimal(1)))), + ArithmeticKey(op="+", operands=(col, LiteralKey(value=Decimal(9)))), + ), + ( + "ArithmeticKey.operand_arity", + ArithmeticKey(op="-", operands=(LiteralKey(value=Decimal(10)),)), + ArithmeticKey( + op="-", + operands=(LiteralKey(value=Decimal(10)), LiteralKey(value=Decimal(0))), + ), + ), + ( + "ArithmeticKey.nesting", + ArithmeticKey( + op="*", + operands=( + ArithmeticKey(op="+", operands=(col, LiteralKey(value=Decimal(1)))), + LiteralKey(value=Decimal(2)), + ), + ), + ArithmeticKey( + op="+", + operands=( + col, + ArithmeticKey( + op="*", + operands=(LiteralKey(value=Decimal(1)), LiteralKey(value=Decimal(2))), + ), + ), + ), + ), + ( + "ScalarCallKey.name", + ScalarCallKey(name="lower", args=(col,)), + ScalarCallKey(name="upper", args=(col,)), + ), + ( + "ScalarCallKey.args", + ScalarCallKey(name="lower", args=(col,)), + ScalarCallKey(name="lower", args=(other,)), + ), + ( + "BetweenKey.low", + BetweenKey(column=col, low=LiteralKey(value=Decimal(1)), + high=LiteralKey(value=Decimal(9))), + BetweenKey(column=col, low=LiteralKey(value=Decimal(2)), + high=LiteralKey(value=Decimal(9))), + ), + ( + "InKey.values", + InKey(column=col, values=(LiteralKey(value="a"),)), + InKey(column=col, values=(LiteralKey(value="b"),)), + ), + ( + "InKey.negated", + InKey(column=col, values=(LiteralKey(value="a"),)), + InKey(column=col, values=(LiteralKey(value="a"),), negated=True), + ), + ( + "AggregateKey.agg", + AggregateKey(source=col, agg="sum"), + AggregateKey(source=col, agg="min"), + ), + ( + "AggregateKey.source", + AggregateKey(source=col, agg="sum"), + AggregateKey(source=other, agg="sum"), + ), + ( + "AggregateKey.column_filter_key", + AggregateKey(source=col, agg="sum"), + AggregateKey( + source=col, agg="sum", + column_filter_key=SqlExprKey(canonical_sql="status = 'new'"), + ), + ), + ( + "AggregateKey.kwargs", + AggregateKey(source=col, agg="sum"), + AggregateKey(source=col, agg="sum", kwargs=(("window", "90d"),)), + ), + ( + "AggregateKey.star_path", + AggregateKey(source=StarKey(), agg="count"), + AggregateKey(source=StarKey(path=("customers",)), agg="count"), + ), + ] + + +_MUTATIONS = _mutation_cases() + + +class TestNoKeyFieldIsSilentlyIgnored: + """The generalisation of every renderer bug this PR's review surfaced. + + A field that does not reach the emitted SQL is a wrong-value bug waiting + to happen: the query runs, returns numbers, and the number is wrong. This + sweeps the union rather than waiting for each instance to be reported. + + Raising counts as passing — refusing to render a key the context cannot + honour is the fail-closed contract. What must never happen is two + materially different keys rendering to the SAME SQL. + """ + + @pytest.mark.parametrize( + "label,base,mutated", _MUTATIONS, ids=[m[0] for m in _MUTATIONS], + ) + def test_changing_a_field_changes_the_sql(self, label, base, mutated) -> None: + def render(key): + # Try both facility groups; a key needing neither renders in both. + for ctx in (_composite_ctx(), _filter_ctx()): + try: + return _sql(render_value_key(key, ctx)) + except RenderContextMissingFacilityError: + continue + except NotImplementedError: + continue + return None # refused everywhere — fail-closed, acceptable + + base_sql = render(base) + mutated_sql = render(mutated) + if base_sql is None or mutated_sql is None: + return # at least one was refused; nothing was silently dropped + assert base_sql != mutated_sql, ( + f"{label}: two keys differing in that field render IDENTICALLY as " + f"{base_sql!r} — the field is silently dropped, which is a " + f"wrong-value bug rather than an error." + ) From 760c001ff7097b82f5a02fa1883da58e52039b4d Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Wed, 5 Aug 2026 18:41:43 +0200 Subject: [PATCH 08/98] DEV-1744: narrow the last pytest.raises to one throwing call The datetime was constructed inside the raises block, so the assertion could have been satisfied by a constructor failure rather than by _literal. Hoisted, and its import moved to the top of the file per the repo rule. Co-Authored-By: Claude Fable 5 --- tests/test_dev1744_value_expr.py | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/tests/test_dev1744_value_expr.py b/tests/test_dev1744_value_expr.py index 3d0a6c17..87c4f164 100644 --- a/tests/test_dev1744_value_expr.py +++ b/tests/test_dev1744_value_expr.py @@ -48,6 +48,7 @@ import os import sqlite3 +from datetime import datetime from decimal import Decimal from typing import AsyncIterator, Optional @@ -748,11 +749,9 @@ def test_unsupported_literal_type_raises(self) -> None: raises, and the paths converge in a later PR — a silent ``str(value)`` there would turn a loud failure into a wrong value. """ - from datetime import datetime - - + value = datetime(2024, 1, 1) with pytest.raises(NotImplementedError): - _literal(datetime(2024, 1, 1)) + _literal(value) def test_supported_literal_types_still_render(self) -> None: """The fail-closed branch must not swallow the supported cases.""" From c8ba0a85c1b15aea84e8d5089d409a061e3beff5 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Wed, 5 Aug 2026 18:58:22 +0200 Subject: [PATCH 09/98] =?UTF-8?q?DEV-1744:=20Codex=20round=20=E2=80=94=20t?= =?UTF-8?q?hree=20more=20silently-wrong=20composition=20edges?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit I had been running CodeRabbit, Sonar and CI each loop but only ran Codex once, against the ORIGINAL implementation. The renderer changed substantially after that (precedence, unary handling, the fail-closed guards), so it got a fresh pass. Three findings, all the same family as the rest — output that still parses and still returns rows, but means something else. Precedence covered only ARITHMETIC nodes. A comparison or boolean nested inside arithmetic therefore lost its parentheses: Add(GT(a, b), 1) -> a > b + 1 # parses as a > (b + 1) EQ(GT(a, b), TRUE) -> a > b = TRUE Add(And(a, b), 1) -> a AND b + 1 The table now spans OR / AND / NOT / comparisons / arithmetic so any lower-precedence child gets wrapped, whatever its kind. Comparisons were left-folded like arithmetic. `a < b < c` became `(a < b) < c` — a boolean compared to a number — and `is` / `is not` read operands[0] and [1], silently dropping any third. The Mode-B parser rejects chained comparisons, so this is unreachable from user input today; it is the structural backstop for anything building keys directly, and folding was the wrong default for an operator that is strictly binary. `contains_aggregate` tested `phase >= AGGREGATE`. Every TransformKey is POST phase whether or not it wraps an aggregate, so a transform over a raw column reported True — and that predicate decides GROUP BY / HAVING placement. It is now a structural walk for an actual AggregateKey. The function has no callers yet; PR 3 adopts it when it replaces the `any_agg` tuple, which is exactly why it should not have been left with a phase test standing in for a tree walk. Each fix has a test verified to fail without it. Tests 9559 passing, ruff clean. Co-Authored-By: Claude Fable 5 --- slayer/sql/render/value_expr.py | 81 +++++++++++++++---- tests/test_dev1744_value_expr.py | 128 +++++++++++++++++++++++++++++++ 2 files changed, 194 insertions(+), 15 deletions(-) diff --git a/slayer/sql/render/value_expr.py b/slayer/sql/render/value_expr.py index a42dd288..9790650e 100644 --- a/slayer/sql/render/value_expr.py +++ b/slayer/sql/render/value_expr.py @@ -67,7 +67,6 @@ ColumnSqlKey, InKey, LiteralKey, - Phase, ScalarCallKey, StarKey, TimeTruncKey, @@ -187,13 +186,33 @@ def _literal(value: Any) -> exp.Expression: ) -# sqlglot does NOT parenthesise by node nesting: ``Mul(Add(a, b), c)`` emits -# ``a + b * c``, which evaluates differently. Precedence must be materialised -# as explicit ``Paren`` nodes. -_ARITH_PRECEDENCE: Dict[Any, int] = { - exp.Add: 1, exp.Sub: 1, exp.Mul: 2, exp.Div: 2, exp.Mod: 2, +# sqlglot does NOT parenthesise by node nesting. ``Mul(Add(a, b), c)`` emits +# ``a + b * c`` and ``Add(GT(a, b), 1)`` emits ``a > b + 1`` — both parse back +# with different meaning. Precedence must be materialised as explicit ``Paren`` +# nodes, and the table has to span BOOLEAN and COMPARISON operators too, not +# just arithmetic: a comparison nested inside arithmetic is the case that bites +# hardest, because the result still parses and still returns rows. +_PRECEDENCE: Dict[Any, int] = { + exp.Or: 1, + exp.And: 2, + exp.Not: 3, + exp.EQ: 4, exp.NEQ: 4, exp.LT: 4, exp.LTE: 4, exp.GT: 4, exp.GTE: 4, + exp.Is: 4, exp.In: 4, exp.Like: 4, exp.Between: 4, + exp.Add: 5, exp.Sub: 5, + exp.Mul: 6, exp.Div: 6, exp.Mod: 6, } +# Non-associative on the right: ``a - (b - c)`` needs its parens even though +# both sides share a precedence level. +_RIGHT_SENSITIVE_OPS = ("-", "/", "%") + +# Operators taking exactly two operands. Left-folding a comparison would turn +# ``a < b < c`` into ``(a < b) < c`` — a boolean compared to a number — and +# reading only the first two would silently DROP the rest. +_STRICTLY_BINARY = frozenset({ + "=", "==", "!=", "<>", "<", "<=", ">", ">=", "is", "is not", +}) + def _paren_if_lower_prec( child: exp.Expression, *, parent_prec: int, is_right: bool, op: str, @@ -204,12 +223,12 @@ def _paren_if_lower_prec( needs them on the RIGHT of the non-associative ``-`` and ``/`` (``a - (b - c)``). Non-arithmetic children are already self-delimiting. """ - child_prec = _ARITH_PRECEDENCE.get(type(child)) + child_prec = _PRECEDENCE.get(type(child)) if child_prec is None: return child if child_prec < parent_prec: return exp.Paren(this=child) - if child_prec == parent_prec and is_right and op in ("-", "/", "%"): + if child_prec == parent_prec and is_right and op in _RIGHT_SENSITIVE_OPS: return exp.Paren(this=child) return child @@ -239,6 +258,15 @@ def _render_arithmetic( return exp.and_(*operands) if op == "or": return exp.or_(*operands) + if op in _STRICTLY_BINARY and len(operands) != 2: + # Refuse rather than fold. Left-folding a chained comparison compares a + # BOOLEAN against the next operand, and reading only the first two + # drops the rest — both silently wrong. The Mode-B parser already + # rejects chained comparisons; this is the structural backstop. + raise NotImplementedError( + f"Operator {op!r} takes exactly two operands, got {len(operands)}.", + ) + if op == "is": return exp.Is(this=operands[0], expression=operands[1]) if op == "is not": @@ -248,7 +276,7 @@ def _render_arithmetic( if node_cls is None: raise NotImplementedError(f"Unsupported arithmetic operator {op!r}.") - parent_prec = _ARITH_PRECEDENCE.get(node_cls) + parent_prec = _PRECEDENCE.get(node_cls) result = operands[0] for operand in operands[1:]: lhs, rhs = result, operand @@ -482,11 +510,34 @@ def render_value_key( # NOSONAR(S3776) — sequential dispatch over the closed def contains_aggregate(key: ValueKey) -> bool: - """Whether ``key`` contains an aggregate, structurally. + """Whether ``key``'s tree CONTAINS an ``AggregateKey``. + + Replaces the ``(expr, any_agg)`` tuple the old renderers threaded by hand, + which decides GROUP BY / HAVING placement. - Replaces the ``(expr, any_agg)`` tuple the old renderers threaded by hand. - ``phase`` already propagates as the max over operands and arguments, so - nested cases (a scalar call over an arithmetic over an aggregate) are - covered without a second traversal. + A structural walk, deliberately NOT ``phase >= AGGREGATE``: phase answers + "when is this evaluated", a different question. Every ``TransformKey`` is + POST phase whether or not it wraps an aggregate, so the phase test reports + True for a transform over a raw column and would route a non-aggregate + predicate into HAVING. """ - return getattr(key, "phase", Phase.ROW) >= Phase.AGGREGATE + if isinstance(key, AggregateKey): + return True + if isinstance(key, ArithmeticKey): + return any(contains_aggregate(o) for o in key.operands) + if isinstance(key, ScalarCallKey): + return any( + contains_aggregate(a) + for a in key.args + if isinstance(a, _VALUE_KEY_TYPES) + ) + if isinstance(key, TransformKey): + return contains_aggregate(key.input) + if isinstance(key, BetweenKey): + return any( + contains_aggregate(k) for k in (key.column, key.low, key.high) + ) + if isinstance(key, InKey): + # ``values`` are LiteralKeys by type, so only the column can carry one. + return contains_aggregate(key.column) + return False diff --git a/tests/test_dev1744_value_expr.py b/tests/test_dev1744_value_expr.py index 87c4f164..637b55ed 100644 --- a/tests/test_dev1744_value_expr.py +++ b/tests/test_dev1744_value_expr.py @@ -90,6 +90,7 @@ from slayer.sql.render.aggregates import resolve_agg_entry, window_agg_class from slayer.sql.render.value_expr import ( AliasFacilities, + contains_aggregate, CompositeFacilities, FilterFacilities, RenderContext, @@ -1614,3 +1615,130 @@ def render(key): f"{base_sql!r} — the field is silently dropped, which is a " f"wrong-value bug rather than an error." ) + + +class TestOperatorCompositionEdges: + """Three edges a Codex pass surfaced, all the same family as the rest: + output that still parses and still returns rows, but means something else. + """ + + def test_comparison_nested_in_arithmetic_keeps_its_parens(self) -> None: + """``(a > b) + 1`` must not flatten to ``a > b + 1``. + + sqlglot does not parenthesise by nesting, and the two parse + differently: the flattened form reads as ``a > (b + 1)``. A precedence + table covering only arithmetic misses this, because the CHILD is a + comparison. + """ + key = ArithmeticKey( + op="+", + operands=( + ArithmeticKey( + op=">", + operands=(ColumnKey(leaf="amount"), LiteralKey(value=Decimal(5))), + ), + LiteralKey(value=Decimal(1)), + ), + ) + out = _sql(render_value_key(key, _filter_ctx())) + assert out == "(orders.amount > 5) + 1", out + + def test_boolean_nested_in_arithmetic_keeps_its_parens(self) -> None: + """Same for a boolean child: ``a AND b + 1`` binds the ``+`` first.""" + key = ArithmeticKey( + op="+", + operands=( + ArithmeticKey( + op="and", + operands=( + ArithmeticKey( + op=">", + operands=(ColumnKey(leaf="amount"), LiteralKey(value=Decimal(1))), + ), + ArithmeticKey( + op="<", + operands=(ColumnKey(leaf="amount"), LiteralKey(value=Decimal(9))), + ), + ), + ), + LiteralKey(value=Decimal(1)), + ), + ) + out = _sql(render_value_key(key, _filter_ctx())) + assert out.startswith("("), out + + def test_comparison_with_three_operands_is_refused(self) -> None: + """A chained comparison must RAISE, not left-fold. + + Left-folding ``a < b < c`` compares a BOOLEAN against ``c``; taking + only the first two operands silently drops the third. The Mode-B parser + rejects chained comparisons, so this is the structural backstop for + anything constructing keys directly. + """ + key = ArithmeticKey( + op="<", + operands=( + ColumnKey(leaf="amount"), + LiteralKey(value=Decimal(5)), + LiteralKey(value=Decimal(9)), + ), + ) + ctx = _filter_ctx() + with pytest.raises(NotImplementedError): + render_value_key(key, ctx) + + def test_is_not_with_extra_operands_is_refused(self) -> None: + """``is`` / ``is not`` read operands[0] and [1] only — a third would + vanish without a word.""" + key = ArithmeticKey( + op="is", + operands=( + ColumnKey(leaf="amount"), + LiteralKey(value=None), + LiteralKey(value=Decimal(1)), + ), + ) + ctx = _filter_ctx() + with pytest.raises(NotImplementedError): + render_value_key(key, ctx) + + +class TestContainsAggregate: + """``contains_aggregate`` decides GROUP BY / HAVING placement, so it must + answer "is there an aggregate in this tree", not "when does this evaluate".""" + + def test_bare_aggregate(self) -> None: + assert contains_aggregate( + AggregateKey(source=ColumnKey(leaf="amount"), agg="sum"), + ) is True + + def test_plain_column_is_not_an_aggregate(self) -> None: + assert contains_aggregate(ColumnKey(leaf="amount")) is False + + def test_aggregate_nested_in_arithmetic_and_scalar_calls(self) -> None: + agg = AggregateKey(source=ColumnKey(leaf="amount"), agg="sum") + nested = ScalarCallKey( + name="ifnull", + args=( + ArithmeticKey(op="+", operands=(agg, LiteralKey(value=Decimal(1)))), + LiteralKey(value=Decimal(0)), + ), + ) + assert contains_aggregate(nested) is True + + def test_transform_over_a_raw_column_is_not_an_aggregate(self) -> None: + """The case a phase test gets wrong. + + Every TransformKey is POST phase, so ``phase >= AGGREGATE`` reports + True even when the transform wraps a plain column — routing a + non-aggregate predicate into HAVING. + """ + key = TransformKey(op="cumsum", input=ColumnKey(leaf="amount")) + assert contains_aggregate(key) is False + + def test_transform_over_an_aggregate_is_an_aggregate(self) -> None: + key = TransformKey( + op="cumsum", + input=AggregateKey(source=ColumnKey(leaf="amount"), agg="sum"), + ) + assert contains_aggregate(key) is True From a78106af6c9aac5e4f899fb20e4d3225a048f370 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Wed, 5 Aug 2026 19:07:34 +0200 Subject: [PATCH 10/98] =?UTF-8?q?DEV-1744:=20second=20Codex=20round=20?= =?UTF-8?q?=E2=80=94=20associativity,=20arity,=20transform=20deps?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reviewing the previous round's fixes found three more, one of them a real wrong-number bug my own fix had left behind. Equal-precedence right children were parenthesised based on the PARENT operator alone, which is not enough. `Mul(a, Mod(b, c))` emits `a * b % c`, regrouping to `(a * b) % c`: with a=2 b=3 c=2 that is 0 where the tree says 2. Integer division has the same shape. The rule is now the inverse — an equal-precedence right child keeps its parens UNLESS the pair is genuinely associative (`+` over Add, `*` over Mul), so the safe cases stay quiet and everything else is grouped explicitly. Arity was implicit. Zero operands reached `operands[0]` and raised IndexError; a single-operand `and` fell into the unary branch and reported "unsupported unary operator 'and'". Both now fail (or succeed) deterministically: empty raises with the operator named, and a one-term conjunction returns that term, which is what it means. `contains_aggregate` walked only `TransformKey.input`. `partition_keys` and `time_key` are expression dependencies too, so `rank(x, partition_by=revenue:sum)` reported no aggregate while emitting one — and that predicate decides GROUP BY versus HAVING placement. All six new tests verified to fail with the fixes reverted. Tests 9573 passing, ruff clean. Co-Authored-By: Claude Fable 5 --- slayer/sql/render/value_expr.py | 31 +++++++++-- tests/test_dev1744_value_expr.py | 90 ++++++++++++++++++++++++++++++++ 2 files changed, 116 insertions(+), 5 deletions(-) diff --git a/slayer/sql/render/value_expr.py b/slayer/sql/render/value_expr.py index 9790650e..e4a7e25b 100644 --- a/slayer/sql/render/value_expr.py +++ b/slayer/sql/render/value_expr.py @@ -202,9 +202,12 @@ def _literal(value: Any) -> exp.Expression: exp.Mul: 6, exp.Div: 6, exp.Mod: 6, } -# Non-associative on the right: ``a - (b - c)`` needs its parens even though -# both sides share a precedence level. -_RIGHT_SENSITIVE_OPS = ("-", "/", "%") +# Truly associative parent/child pairs — the ONLY equal-precedence right +# children that may safely drop their parens. Everything else keeps them: +# checking the parent operator alone is not enough, because ``Mul(a, Mod(b, c))`` +# emits ``a * b %% c``, which regroups to ``(a * b) %% c`` and returns a +# different number. +_ASSOCIATIVE_PAIRS = {("+", exp.Add), ("*", exp.Mul)} # Operators taking exactly two operands. Left-folding a comparison would turn # ``a < b < c`` into ``(a < b) < c`` — a boolean compared to a number — and @@ -228,7 +231,11 @@ def _paren_if_lower_prec( return child if child_prec < parent_prec: return exp.Paren(this=child) - if child_prec == parent_prec and is_right and op in _RIGHT_SENSITIVE_OPS: + if ( + child_prec == parent_prec + and is_right + and (op, type(child)) not in _ASSOCIATIVE_PAIRS + ): return exp.Paren(this=child) return child @@ -243,6 +250,13 @@ def _render_arithmetic( just returns ``operands[0]`` would turn ``amount > -10`` into ``amount > 10``. """ + if not operands: + raise NotImplementedError(f"Operator {op!r} needs at least one operand.") + + if op in ("and", "or") and len(operands) == 1: + # Degenerate but well-defined: the conjunction of one term IS that term. + return operands[0] + if len(operands) == 1: if op == "not": return exp.Not(this=operands[0]) @@ -532,7 +546,14 @@ def contains_aggregate(key: ValueKey) -> bool: if isinstance(a, _VALUE_KEY_TYPES) ) if isinstance(key, TransformKey): - return contains_aggregate(key.input) + # partition_keys and time_key are expression dependencies just as + # input is: cumsum(x, partition_by=revenue:sum) references an + # aggregate even though its input does not. + return ( + contains_aggregate(key.input) + or any(contains_aggregate(p) for p in key.partition_keys) + or (key.time_key is not None and contains_aggregate(key.time_key)) + ) if isinstance(key, BetweenKey): return any( contains_aggregate(k) for k in (key.column, key.low, key.high) diff --git a/tests/test_dev1744_value_expr.py b/tests/test_dev1744_value_expr.py index 637b55ed..e77d848a 100644 --- a/tests/test_dev1744_value_expr.py +++ b/tests/test_dev1744_value_expr.py @@ -1742,3 +1742,93 @@ def test_transform_over_an_aggregate_is_an_aggregate(self) -> None: input=AggregateKey(source=ColumnKey(leaf="amount"), agg="sum"), ) assert contains_aggregate(key) is True + + +class TestEqualPrecedenceRightChildren: + """Checking the PARENT operator alone is not enough. + + ``a - (b - c)`` was already handled, but ``a * (b % c)`` was not: the + parent is ``*``, which looks associative, so the parens were dropped and + the expression regrouped to ``(a * b) % c``. With a=2 b=3 c=2 that is 0 + instead of 2 — a different number from SQL that parses cleanly. + """ + + def _key(self, outer_op, inner_op): + return ArithmeticKey( + op=outer_op, + operands=( + ColumnKey(leaf="amount"), + ArithmeticKey( + op=inner_op, + operands=( + LiteralKey(value=Decimal(3)), + LiteralKey(value=Decimal(2)), + ), + ), + ), + ) + + @pytest.mark.parametrize( + "outer,inner", + [("*", "%"), ("*", "/"), ("/", "*"), ("/", "/"), ("-", "+"), ("-", "-")], + ) + def test_equal_precedence_right_child_keeps_parens(self, outer, inner) -> None: + out = _sql(render_value_key(self._key(outer, inner), _filter_ctx())) + assert "(" in out, f"{outer} over {inner} lost its grouping: {out}" + + @pytest.mark.parametrize("op", ["+", "*"]) + def test_genuinely_associative_pairs_stay_unparenthesised(self, op) -> None: + """``a + (b + c)`` and ``a * (b * c)`` regroup harmlessly, so don't + add noise for them.""" + out = _sql(render_value_key(self._key(op, op), _filter_ctx())) + assert "(" not in out, out + + +class TestArithmeticArity: + def test_no_operands_is_refused(self) -> None: + key = ArithmeticKey(op="+", operands=()) + ctx = _filter_ctx() + with pytest.raises(NotImplementedError): + render_value_key(key, ctx) + + @pytest.mark.parametrize("op", ["and", "or"]) + def test_single_operand_boolean_is_that_operand(self, op) -> None: + """The conjunction of one term is that term — well-defined, and it must + not fall through to the unary branch and report ``and`` as an + unsupported unary operator.""" + inner = ArithmeticKey( + op=">", operands=(ColumnKey(leaf="amount"), LiteralKey(value=Decimal(5))), + ) + key = ArithmeticKey(op=op, operands=(inner,)) + assert _sql(render_value_key(key, _filter_ctx())) == "orders.amount > 5" + + +class TestContainsAggregateTransformDependencies: + """``partition_keys`` and ``time_key`` are expression dependencies of a + transform just as ``input`` is — an aggregate in either one still lands in + the emitted SQL.""" + + def test_aggregate_in_partition_keys(self) -> None: + agg = AggregateKey(source=ColumnKey(leaf="amount"), agg="sum") + key = TransformKey( + op="rank", + input=ColumnKey(leaf="amount"), + partition_keys=frozenset({agg}), + ) + assert contains_aggregate(key) is True + + def test_aggregate_in_time_key(self) -> None: + agg = AggregateKey(source=ColumnKey(leaf="created_at"), agg="max") + key = TransformKey( + op="cumsum", input=ColumnKey(leaf="amount"), time_key=agg, + ) + assert contains_aggregate(key) is True + + def test_transform_with_no_aggregate_anywhere(self) -> None: + key = TransformKey( + op="rank", + input=ColumnKey(leaf="amount"), + partition_keys=frozenset({ColumnKey(leaf="label")}), + time_key=ColumnKey(leaf="created_at"), + ) + assert contains_aggregate(key) is False From 09106e6c2403932f4d7096cd0182b312b2c7f055 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Wed, 5 Aug 2026 19:24:37 +0200 Subject: [PATCH 11/98] =?UTF-8?q?DEV-1744:=20third=20Codex=20round=20?= =?UTF-8?q?=E2=80=94=20scalar=20arity,=20and=20no=20associativity=20except?= =?UTF-8?q?ions?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two findings, one of them a LIVE bug reachable from user input today (unlike most of this PR's findings, which were latent in the not-yet-wired renderer). Scalar arity was never validated. sqlglot handles it three different ways and all three are bad answers for a mistyped filter: round(amount, 2, 99) -> ROUND(CAST(amount AS DECIMAL), 2) the third argument SILENTLY DROPPED length(status, status) -> LENGTH(a, b) invalid SQL, backend rejects it later lower(status, status) -> raw sqlglot ValueError leaking internals The binder validated arity for `like` and nothing else, so the first two reached the database. There is now an arity table beside the allowlist that knows the answer, checked at bind time (where a user's typo should surface, naming the function and the counts) and again in `render_scalar_call` as the fail-closed backstop. All four cases now report e.g. "Scalar function 'round' takes 1 to 2 arguments; got 3." The associativity exception is gone. Last round I let `+` over Add and `*` over Mul drop their parens as "genuinely associative". That holds over the reals and fails over the machine: with floats, rounding makes `a + (b + c)` and `(a + b) + c` differ, and fixed-precision decimals add overflow. The binder built a specific tree and emitting a different one is a silent accuracy change, so every equal-precedence right child now keeps its parens — one fewer special case, and the previous round's test asserting the opposite is inverted with the reasoning recorded rather than deleted. Tests 9593 passing, ruff clean. Co-Authored-By: Claude Fable 5 --- slayer/core/keys.py | 42 +++++++++++++ slayer/engine/binding.py | 18 ++++-- slayer/sql/render/value_expr.py | 33 +++++----- tests/test_dev1744_value_expr.py | 102 +++++++++++++++++++++++++++++-- 4 files changed, 171 insertions(+), 24 deletions(-) diff --git a/slayer/core/keys.py b/slayer/core/keys.py index b07296cf..560324f7 100644 --- a/slayer/core/keys.py +++ b/slayer/core/keys.py @@ -51,6 +51,48 @@ }) +# Accepted argument counts per allowlisted scalar, as ``(min, max)``; ``max=None`` +# means variadic. Validated at bind time so a malformed call is a clear SLayer +# error, and again at render time as the fail-closed backstop. +# +# Needed because sqlglot is inconsistent about arity: ``exp.func("ROUND", a, b, c)`` +# SILENTLY DROPS the third argument, ``exp.func("LENGTH", a, b)`` emits invalid +# ``LENGTH(a, b)`` for the database to reject, and ``exp.func("LOWER", a, b)`` +# raises a raw sqlglot ValueError. None of those is a good answer for a user +# who mistyped a filter. +SCALAR_FUNCTION_ARITY: dict[str, tuple[int, Optional[int]]] = { + "nullif": (2, 2), + "coalesce": (1, None), + "ifnull": (2, 2), + "ln": (1, 1), "log10": (1, 1), "log2": (1, 1), "log": (1, 2), + "exp": (1, 1), "sqrt": (1, 1), + "pow": (2, 2), "power": (2, 2), + "abs": (1, 1), "floor": (1, 1), "ceil": (1, 1), "round": (1, 2), + "lower": (1, 1), "upper": (1, 1), "trim": (1, 1), "length": (1, 1), + "replace": (3, 3), "substr": (2, 3), "instr": (2, 2), + "concat": (1, None), + "like": (2, 2), +} + + +def check_scalar_arity(name: str, argc: int) -> Optional[str]: + """Return an error message when ``name`` cannot take ``argc`` arguments.""" + bounds = SCALAR_FUNCTION_ARITY.get(name) + if bounds is None: + return None + low, high = bounds + if argc < low or (high is not None and argc > high): + expected = ( + f"{low}" if low == high + else (f"{low} or more" if high is None else f"{low} to {high}") + ) + return ( + f"Scalar function {name!r} takes {expected} argument" + f"{'' if low == high == 1 else 's'}; got {argc}." + ) + return None + + # --------------------------------------------------------------------------- # Phase # --------------------------------------------------------------------------- diff --git a/slayer/engine/binding.py b/slayer/engine/binding.py index 0a8eb811..476a5aff 100644 --- a/slayer/engine/binding.py +++ b/slayer/engine/binding.py @@ -54,6 +54,7 @@ ) from slayer.core.keys import ( SCALAR_FUNCTIONS, + check_scalar_arity, AggregateKey, ArithmeticKey, BetweenKey, @@ -1285,11 +1286,18 @@ def _bind_scalar( f"{sorted(SCALAR_FUNCTIONS)}." ), ) - if parsed.name == "like" and len(parsed.args) != 2: - raise ValueError( - f"Scalar function 'like' takes exactly 2 arguments " - f"(value, pattern); got {len(parsed.args)}." - ) + # Arity for EVERY allowlisted scalar, not just like. sqlglot's own + # handling is inconsistent — a wrong-arity round silently drops the + # extra argument, length emits SQL the database rejects — so a + # mistyped filter deserves a clear error here instead. + arity_error = check_scalar_arity(parsed.name, len(parsed.args)) + if arity_error is not None: + if parsed.name == "like": + raise ValueError( + f"Scalar function 'like' takes exactly 2 arguments " + f"(value, pattern); got {len(parsed.args)}." + ) + raise ValueError(arity_error) args = tuple( _bind(a, scope=scope, bundle=bundle, in_filter=in_filter, alias_map=alias_map) for a in parsed.args diff --git a/slayer/sql/render/value_expr.py b/slayer/sql/render/value_expr.py index e4a7e25b..5930f156 100644 --- a/slayer/sql/render/value_expr.py +++ b/slayer/sql/render/value_expr.py @@ -72,6 +72,7 @@ TimeTruncKey, TransformKey, ValueKey, + check_scalar_arity, ) from slayer.sql.dialects.base import SqlDialect from slayer.sql.render.aggregates import ( @@ -202,13 +203,6 @@ def _literal(value: Any) -> exp.Expression: exp.Mul: 6, exp.Div: 6, exp.Mod: 6, } -# Truly associative parent/child pairs — the ONLY equal-precedence right -# children that may safely drop their parens. Everything else keeps them: -# checking the parent operator alone is not enough, because ``Mul(a, Mod(b, c))`` -# emits ``a * b %% c``, which regroups to ``(a * b) %% c`` and returns a -# different number. -_ASSOCIATIVE_PAIRS = {("+", exp.Add), ("*", exp.Mul)} - # Operators taking exactly two operands. Left-folding a comparison would turn # ``a < b < c`` into ``(a < b) < c`` — a boolean compared to a number — and # reading only the first two would silently DROP the rest. @@ -222,20 +216,23 @@ def _paren_if_lower_prec( ) -> exp.Expression: """Parenthesise ``child`` when dropping its parens would change meaning. - Lower precedence than the parent always needs parens; equal precedence - needs them on the RIGHT of the non-associative ``-`` and ``/`` - (``a - (b - c)``). Non-arithmetic children are already self-delimiting. + Lower precedence than the parent always needs parens. So does EVERY + equal-precedence right child: checking the parent operator is not enough + (``Mul(a, Mod(b, c))`` emits ``a * b % c``, regrouping to ``(a * b) % c``), + and even ``+`` and ``*`` are not operationally associative over floats or + fixed-precision decimals, where rounding and overflow make ``a + (b + c)`` + and ``(a + b) + c`` genuinely different. Preserving the tree the binder + built costs a pair of parentheses; regrouping costs accuracy. + + A node with no precedence entry — a column, a literal, a function call — + is already self-delimiting. """ child_prec = _PRECEDENCE.get(type(child)) if child_prec is None: return child if child_prec < parent_prec: return exp.Paren(this=child) - if ( - child_prec == parent_prec - and is_right - and (op, type(child)) not in _ASSOCIATIVE_PAIRS - ): + if child_prec == parent_prec and is_right: return exp.Paren(this=child) return child @@ -317,6 +314,12 @@ def render_scalar_call( a native single-arg ``LOG10``. Transpiling alone fixes ifnull and breaks log10. ``like`` is the allowlist's only operator rather than function. """ + arity_error = check_scalar_arity(name, len(args)) + if arity_error is not None: + # Checked before building, because sqlglot is inconsistent: a 3-arg + # ROUND silently DROPS the third, a 2-arg LENGTH emits SQL the backend + # rejects, and a 2-arg LOWER raises a raw sqlglot error. + raise NotImplementedError(arity_error) if name == "like": return exp.Like(this=args[0], expression=args[1]) node = dialect.rewrite_target_ast(exp.func(name.upper(), *args)) diff --git a/tests/test_dev1744_value_expr.py b/tests/test_dev1744_value_expr.py index e77d848a..fad5e534 100644 --- a/tests/test_dev1744_value_expr.py +++ b/tests/test_dev1744_value_expr.py @@ -1777,11 +1777,21 @@ def test_equal_precedence_right_child_keeps_parens(self, outer, inner) -> None: assert "(" in out, f"{outer} over {inner} lost its grouping: {out}" @pytest.mark.parametrize("op", ["+", "*"]) - def test_genuinely_associative_pairs_stay_unparenthesised(self, op) -> None: - """``a + (b + c)`` and ``a * (b * c)`` regroup harmlessly, so don't - add noise for them.""" + def test_even_plus_and_times_keep_their_grouping(self, op) -> None: + """``+`` and ``*`` are NOT operationally associative either. + + An earlier version of this test asserted the opposite — that these two + could safely drop their parens because they regroup harmlessly. That is + true over the reals and false over the machine: with floats, rounding + makes ``a + (b + c)`` and ``(a + b) + c`` differ, and with + fixed-precision decimals so does overflow. The binder built a specific + tree; emitting a different one is a silent accuracy change. + + So the rule is now uniform — every equal-precedence right child keeps + its parens — which is also one fewer special case to get wrong. + """ out = _sql(render_value_key(self._key(op, op), _filter_ctx())) - assert "(" not in out, out + assert "(" in out, out class TestArithmeticArity: @@ -1832,3 +1842,87 @@ def test_transform_with_no_aggregate_anywhere(self) -> None: time_key=ColumnKey(leaf="created_at"), ) assert contains_aggregate(key) is False + + +class TestScalarArity: + """sqlglot's arity handling is inconsistent, and all three modes are bad + answers for a mistyped filter. + + ``exp.func("ROUND", a, b, c)`` SILENTLY DROPS the third argument. + ``exp.func("LENGTH", a, b)`` emits ``LENGTH(a, b)`` for the database to + reject with its own error. ``exp.func("LOWER", a, b)`` raises a raw sqlglot + ValueError that leaks an internal library name. The allowlist knows the + right answer, so it is checked before the node is built. + """ + + @pytest.mark.parametrize( + "name,argc", + [("round", 3), ("lower", 2), ("length", 2), ("abs", 2), + ("nullif", 1), ("replace", 2), ("substr", 1), ("like", 3)], + ) + def test_wrong_arity_is_refused(self, name, argc) -> None: + from slayer.sql.render.value_expr import render_scalar_call + + args = [exp.column(f"c{i}") for i in range(argc)] + with pytest.raises(NotImplementedError): + render_scalar_call( + name=name, args=args, dialect=get_dialect("postgres"), + ) + + @pytest.mark.parametrize( + "name,argc", + [("round", 1), ("round", 2), ("lower", 1), ("substr", 2), ("substr", 3), + ("replace", 3), ("coalesce", 1), ("coalesce", 4), ("concat", 3)], + ) + def test_accepted_arities_still_render(self, name, argc) -> None: + """The variadic and optional-argument forms must keep working.""" + from slayer.sql.render.value_expr import render_scalar_call + + args = [exp.column(f"c{i}") for i in range(argc)] + out = render_scalar_call( + name=name, args=args, dialect=get_dialect("postgres"), + ) + assert out.sql(dialect="postgres") + + +class TestArityIsRejectedAtBindTime: + """The renderer check is the backstop; the binder is where a user's typo + should surface, with a message naming the function and the counts.""" + + async def test_round_with_three_args_is_rejected(self, e2e) -> None: + with pytest.raises(ValueError, match="round"): + await e2e.execute( + SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ModelMeasure(formula="*:count", name="n")], + filters=["round(amount, 2, 99) > 1"], + ), + dry_run=True, + ) + + async def test_length_with_two_args_is_rejected(self, e2e) -> None: + """Previously emitted ``LENGTH(a, b)`` — invalid SQL the backend + rejected with its own, less useful error.""" + with pytest.raises(ValueError, match="length"): + await e2e.execute( + SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ModelMeasure(formula="*:count", name="n")], + filters=["length(status, status) > 1"], + ), + dry_run=True, + ) + + async def test_correct_arity_still_binds(self, e2e) -> None: + resp = await e2e.execute( + SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ModelMeasure(formula="*:count", name="n")], + filters=["round(amount, 2) > 1"], + ), + dry_run=True, + ) + assert "ROUND" in resp.sql.upper() From 1bbf6188fc5928e7921a9f2fa89c4391d67d47c7 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Wed, 5 Aug 2026 19:30:49 +0200 Subject: [PATCH 12/98] =?UTF-8?q?DEV-1744:=20reject=20NULL=20inside=20an?= =?UTF-8?q?=20IN=20list=20=E2=80=94=20it=20silently=20returned=20zero=20ro?= =?UTF-8?q?ws?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Live and reachable from an ordinary filter. SQL's three-valued logic makes a NULL member a trap rather than a member test: status in ('a', None) -> IN ('a', NULL) matches only 'a' status not in ('a', None) -> NOT IN ('a', NULL) matches NOTHING The second is the dangerous one. `NOT IN` with a NULL evaluates to NULL for every row, so a user asking for "everything except a" got an empty result set with no error, no warning, and SQL that looks correct. Verified end-to-end before fixing: rows went from ['b', 'c'] to []. Rejected at bind time with a message that names the fix (`is null` / `is not null` alongside the IN over the non-null values) rather than lecturing about three-valued logic, and again in the renderer as the backstop. This is a user-facing behavior change beyond the ratified B-items: a query that previously ran now errors. Surfaced deliberately — it previously ran and returned the wrong answer, which is the failure mode this PR exists to remove. Found by the fourth Codex pass, which also confirmed three things I had asked about and could not settle myself: aggregating a materialised alias across a projection boundary is correct, Paren wrappers do not hide a Log node from the log-alias rewrite, and the hand-derived scalar arity table matches real SQL signatures (it is now a user-facing gate, so a wrong bound would have rejected legitimate queries). Tests 9599 passing, ruff clean. Co-Authored-By: Claude Fable 5 --- slayer/engine/binding.py | 13 ++++++ slayer/sql/render/value_expr.py | 8 ++++ tests/test_dev1744_value_expr.py | 76 ++++++++++++++++++++++++++++++++ 3 files changed, 97 insertions(+) diff --git a/slayer/engine/binding.py b/slayer/engine/binding.py index 476a5aff..430681b3 100644 --- a/slayer/engine/binding.py +++ b/slayer/engine/binding.py @@ -506,6 +506,19 @@ def _bind_in( LiteralKey(value=normalize_scalar(elt.value)) for elt in parsed.right.elements ) + # SQL's three-valued logic makes a NULL in the list a trap rather than a + # member test. ``col IN (a, NULL)`` never matches on the NULL, and + # ``col NOT IN (a, NULL)`` evaluates to NULL for EVERY row — so the filter + # silently returns zero rows instead of "everything except a". Neither is + # what the author meant, and neither announces itself. + if any(v.value is None for v in values): + raise ValueError( + f"NULL is not allowed inside an {parsed.op!r} list: SQL compares " + f"it by three-valued logic, so 'not in' with a NULL matches NO " + f"rows at all. Test for null separately — e.g. " + f"`col is null` / `col is not null` — combined with the " + f"{parsed.op!r} over the non-null values." + ) return InKey( column=column, values=values, diff --git a/slayer/sql/render/value_expr.py b/slayer/sql/render/value_expr.py index 5930f156..f067ea31 100644 --- a/slayer/sql/render/value_expr.py +++ b/slayer/sql/render/value_expr.py @@ -488,6 +488,14 @@ def render_value_key( # NOSONAR(S3776) — sequential dispatch over the closed ) if isinstance(key, InKey): + # Backstop for the bind-time rule: SQL's three-valued logic makes a + # NULL member a trap, and ``NOT IN`` with one matches no rows at all. + if any(v.value is None for v in key.values): + raise NotImplementedError( + "NULL is not allowed inside an IN list: 'NOT IN' with a NULL " + "matches no rows. Test for null separately with IS NULL / " + "IS NOT NULL.", + ) node = exp.In( this=render_value_key(key.column, ctx), expressions=[_literal(v.value) for v in key.values], diff --git a/tests/test_dev1744_value_expr.py b/tests/test_dev1744_value_expr.py index fad5e534..1c75de79 100644 --- a/tests/test_dev1744_value_expr.py +++ b/tests/test_dev1744_value_expr.py @@ -1926,3 +1926,79 @@ async def test_correct_arity_still_binds(self, e2e) -> None: dry_run=True, ) assert "ROUND" in resp.sql.upper() + + +class TestNullInInList: + """SQL's three-valued logic makes a NULL member a trap, not a member test. + + ``col IN (a, NULL)`` never matches on the NULL. ``col NOT IN (a, NULL)`` + is worse: it evaluates to NULL for EVERY row, so the filter returns ZERO + rows rather than "everything except a". Neither announces itself — the + query runs and hands back a plausible-looking empty result. + """ + + def test_renderer_refuses_null_in_the_list(self) -> None: + key = InKey( + column=ColumnKey(leaf="label"), + values=(LiteralKey(value="a"), LiteralKey(value=None)), + ) + ctx = _filter_ctx() + with pytest.raises(NotImplementedError): + render_value_key(key, ctx) + + def test_renderer_refuses_null_in_a_negated_list(self) -> None: + key = InKey( + column=ColumnKey(leaf="label"), + values=(LiteralKey(value="a"), LiteralKey(value=None)), + negated=True, + ) + ctx = _filter_ctx() + with pytest.raises(NotImplementedError): + render_value_key(key, ctx) + + def test_ordinary_in_list_still_renders(self) -> None: + key = InKey( + column=ColumnKey(leaf="label"), + values=(LiteralKey(value="a"), LiteralKey(value="b")), + ) + assert _sql(render_value_key(key, _filter_ctx())) == ( + "orders.label IN ('a', 'b')" + ) + + async def test_bind_time_rejects_null_in_list(self, e2e) -> None: + """The user-facing half: caught at bind, with a message pointing at + ``is null`` rather than at three-valued logic in the abstract.""" + with pytest.raises(ValueError, match="NULL is not allowed"): + await e2e.execute( + SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ModelMeasure(formula="*:count", name="n")], + filters=["status in ('new', None)"], + ), + dry_run=True, + ) + + async def test_bind_time_rejects_null_in_negated_list(self, e2e) -> None: + """The dangerous one: this previously returned zero rows in silence.""" + with pytest.raises(ValueError, match="NULL is not allowed"): + await e2e.execute( + SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ModelMeasure(formula="*:count", name="n")], + filters=["status not in ('new', None)"], + ), + dry_run=True, + ) + + async def test_null_free_in_list_still_executes(self, e2e) -> None: + resp = await e2e.execute( + SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ModelMeasure(formula="*:count", name="n")], + filters=["status in ('new', 'missing')"], + ) + ) + assert [r["orders.status"] for r in resp.data] == ["new"] From 2adb3686b3498c8ed4e60b8bff0c54dc55c1228d Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Wed, 5 Aug 2026 19:50:52 +0200 Subject: [PATCH 13/98] DEV-1744: put the render shape in the cross-model dedup identity; fix docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two reviewers converged on the same branch from different angles, which is what made it worth acting on. CodeRabbit noted the dedup is unreachable today; Codex noted that IF it is reached, `(source_relation, AggregateKey)` is the wrong key. Codex is right. The forward and rerooted render paths produce different join-back pairs and a different aggregate column alias — forward uses the canonical alias, rerooted uses the sub-plan's. Two plans sharing an identity but differing in reroot shape would make the second silently inherit the first's CTE, joining at the wrong grain or reading the wrong column. The planner interns each key to one slot and emits one plan per slot, so this cannot happen today; extracting `_cm_plan_identity` and folding the shape in means a future planner change cannot make it silently wrong, and gives the rule a name and a test instead of leaving it implicit in a tuple literal. Docs (CodeRabbit): my `ifnull` example sat in `SlayerQuery.filters` while the allowlist line above it listed only the string-hygiene subset — valid code against an incomplete doc. The list now matches SCALAR_FUNCTIONS (null handling, math, string hygiene, like), the Rejects column no longer contra- dicts it by naming `coalesce` as rejected, and the two new user-facing rules from this PR are written down: argument counts are validated, and NULL is rejected inside an `in` list with the `is null` workaround shown. Tests 9603 passing, ruff clean. Co-Authored-By: Claude Fable 5 --- docs/concepts/references.md | 21 ++++++++- slayer/sql/generator.py | 26 +++++++++++- tests/test_dev1744_naming_allocator.py | 59 ++++++++++++++++++++++++++ 3 files changed, 104 insertions(+), 2 deletions(-) diff --git a/docs/concepts/references.md b/docs/concepts/references.md index 91b0a308..4692ac65 100644 --- a/docs/concepts/references.md +++ b/docs/concepts/references.md @@ -7,7 +7,7 @@ SLayer has two distinct expression layers and the rules for what each one accept | Mode | Fields | Parser | Accepts | Rejects | |---|---|---|---|---| | **A — SQL** | `Column.sql`, `Column.filter`, each entry of `SlayerModel.filters` | sqlglot | Any valid SQL expression for the underlying dialect — function calls (`json_extract`, `coalesce`, `nullif`, `lower`, `length`, …), arithmetic, `CASE WHEN`, string literals, comparison and boolean operators in SQL spelling (`=`, `<>`, `IS NULL`, `AND`, `OR`, `NOT`, `IN`, `LIKE`). Bare names and `__`-delimited join paths. | Aggregation colon syntax (`revenue:sum`); SLayer transform calls (`cumsum`, `change`, `rank`, …); references to `ModelMeasure` formulas; raw `OVER (...)` window functions inside `Column.filter` / `SlayerModel.filters` (allowed only in `Column.sql`). | -| **B — DSL** | `ModelMeasure.formula`, `SlayerQuery.measures`, `SlayerQuery.filters`, `SlayerQuery.dimensions`, `SlayerQuery.time_dimensions`, `SlayerQuery.order`, `SlayerQuery.main_time_dimension` | Python AST formula parser | Bare names that resolve to a `Column` or `ModelMeasure` on the model; single-dot dotted paths through joins (`customers.regions.name`, `customers.revenue:sum`); aggregation colon syntax (`:`, `*:count`, parametric forms); transform calls (`cumsum(revenue:sum)`, `rank(revenue:sum, partition_by=region)`); arithmetic / boolean / comparison operators; the SQL `\|\|` concat operator (folded into `concat(...)`); pattern matching via the `like(value, pattern)` scalar (emits the SQL `LIKE` operator — wrap in `not (...)` for `NOT LIKE`); a small allowlist of lowercase string-hygiene scalars in `SlayerQuery.filters` only — `lower`, `upper`, `trim`, `replace`, `substr`, `instr`, `length`, `concat`, `like`; `{variable}` placeholders (filters only). | `__`-delimited tokens in user input; raw SQL function calls outside the string-hygiene allowlist (`json_extract`, `coalesce`, …); raw `OVER (...)`; bare names that don't resolve to a Column / ModelMeasure / custom aggregation / query alias; **uppercase** spellings of the string-hygiene functions (`LOWER`, `TRIM`, …) — DSL is case-sensitive. | +| **B — DSL** | `ModelMeasure.formula`, `SlayerQuery.measures`, `SlayerQuery.filters`, `SlayerQuery.dimensions`, `SlayerQuery.time_dimensions`, `SlayerQuery.order`, `SlayerQuery.main_time_dimension` | Python AST formula parser | Bare names that resolve to a `Column` or `ModelMeasure` on the model; single-dot dotted paths through joins (`customers.regions.name`, `customers.revenue:sum`); aggregation colon syntax (`:`, `*:count`, parametric forms); transform calls (`cumsum(revenue:sum)`, `rank(revenue:sum, partition_by=region)`); arithmetic / boolean / comparison operators; the SQL `\|\|` concat operator (folded into `concat(...)`); pattern matching via the `like(value, pattern)` scalar (emits the SQL `LIKE` operator — wrap in `not (...)` for `NOT LIKE`); a closed allowlist of lowercase scalar functions — null handling (`nullif`, `coalesce`, `ifnull`), math (`ln`, `log10`, `log2`, `log`, `exp`, `sqrt`, `pow`, `power`, `abs`, `floor`, `ceil`, `round`), string hygiene (`lower`, `upper`, `trim`, `replace`, `substr`, `instr`, `length`, `concat`) and `like`, each with a fixed argument count that is validated; `{variable}` placeholders (filters only). | `__`-delimited tokens in user input; raw SQL function calls outside that allowlist (`json_extract`, `date_trunc`, …), and any allowlisted call with the wrong number of arguments; raw `OVER (...)`; bare names that don't resolve to a Column / ModelMeasure / custom aggregation / query alias; **uppercase** spellings of the allowlisted functions (`LOWER`, `TRIM`, …) — DSL is case-sensitive; `NULL` inside an `in` / `not in` list (use `is null` / `is not null` instead — see below). | ## Identifier resolution @@ -138,6 +138,25 @@ Two consequences worth knowing: * **`log10` / `log2` keep their single-argument form** on the backends that provide one, rather than becoming the generic two-argument `LOG(base, x)`. +* **Argument counts are validated.** Each allowlisted scalar has a fixed arity + (`round` takes 1 or 2, `substr` 2 or 3, `replace` exactly 3; `coalesce` and + `concat` are variadic). A call with the wrong number is rejected with a + message naming the function, rather than being silently truncated or passed + through to fail at the database. + +## `NULL` inside an `in` list + +`NULL` is rejected inside an `in` / `not in` list. SQL compares it by +three-valued logic, so `status in ('a', None)` never matches on the null, and +`status not in ('a', None)` matches **no rows at all** — the filter silently +returns an empty result instead of "everything except 'a'". + +Test for null separately: + +```json +{"filters": ["status not in ('new', 'old')", "status is not null"]} +``` + ## See also * [Models](models.md) — `Column.sql`, `Column.filter`, model-level filters diff --git a/slayer/sql/generator.py b/slayer/sql/generator.py index f02e1e9f..036f367a 100644 --- a/slayer/sql/generator.py +++ b/slayer/sql/generator.py @@ -553,6 +553,28 @@ def _cte_name_from_alias(prefix: str, alias: str) -> str: return prefix + sanitized +def _cm_plan_identity(*, source_relation: str, plan, agg_slot) -> tuple: + """The dedup identity for a cross-model CTE. + + Structural, never the sanitised name string: the canonical alias omits the + aggregate's column filter, and the name is doubly lossy, so either would + merge plans that must render separately. + + The reroot shape is part of the identity because the two render paths + produce DIFFERENT join-back pairs and a different aggregate column alias — + forward uses the canonical alias, rerooted uses the sub-plan's. Sharing a + CTE across them would join at the wrong grain or read the wrong column. + The planner interns each key to one slot and emits one plan per slot, so + two plans cannot collide here today; keeping the shape in the identity + means a future planner change cannot make that silently wrong. + """ + return ( + source_relation, + agg_slot.key, + plan.rerooted_plan is not None, + ) + + def _effective_src_filters(*, planned_query, plan) -> list: """``planned_query.filters_by_phase`` as the windowed ``_src`` scope sees it (DEV-1732): frame-bound residuals substituted for the host's predicates. @@ -4609,7 +4631,9 @@ def _add_local_aux_slots( key=agg_slot.key, ) canonical_alias_for_plan[plan.aggregate_slot_id] = canonical_alias - identity = (source_relation, agg_slot.key) + identity = _cm_plan_identity( + source_relation=source_relation, plan=plan, agg_slot=agg_slot, + ) existing = cm_cte_name_by_identity.get(identity) if existing is not None: # Same aggregate under another public name: share the one CTE, diff --git a/tests/test_dev1744_naming_allocator.py b/tests/test_dev1744_naming_allocator.py index a29129e5..0b0819d9 100644 --- a/tests/test_dev1744_naming_allocator.py +++ b/tests/test_dev1744_naming_allocator.py @@ -1179,3 +1179,62 @@ def _spy(k, **kw): "IGNORED_TEXT", bound=BoundExpr(value_key=key), ) assert calls[-1].get("profile") == "stage_formula" + + +class TestCrossModelDedupIdentity: + """What makes two cross-model plans "the same CTE". + + The identity is structural — never the sanitised name — because the + canonical alias omits the aggregate's column filter and the name is doubly + lossy, so either would merge plans that must render separately. + + It also carries the RENDER SHAPE. The forward and rerooted paths produce + different join-back pairs and a different aggregate column alias (forward + uses the canonical alias; rerooted uses the sub-plan's), so sharing one CTE + across them would join at the wrong grain or read the wrong column. The + planner interns each key to one slot and emits one plan per slot, so two + plans cannot collide here today — these tests keep that from becoming + silently wrong if that ever changes. + """ + + def _identity(self, *, rerooted, key=None): + from types import SimpleNamespace + + from slayer.sql.generator import _cm_plan_identity + + key = key or AggregateKey( + source=ColumnKey(path=("customers",), leaf="revenue"), agg="sum", + ) + return _cm_plan_identity( + source_relation="orders", + plan=SimpleNamespace(rerooted_plan=object() if rerooted else None), + agg_slot=SimpleNamespace(key=key), + ) + + def test_forward_and_rerooted_are_different_identities(self) -> None: + assert self._identity(rerooted=False) != self._identity(rerooted=True) + + def test_same_key_same_shape_shares_one_identity(self) -> None: + """The C13 intent the dedup exists to serve: the same aggregate under + two public names is still ONE CTE.""" + assert self._identity(rerooted=False) == self._identity(rerooted=False) + + def test_filtered_and_unfiltered_are_different_identities(self) -> None: + """The reason the identity is the typed key and not the alias: these + two produce the SAME canonical alias.""" + from slayer.core.keys import SqlExprKey + + source = ColumnKey(path=("customers",), leaf="revenue") + plain = AggregateKey(source=source, agg="sum") + filtered = AggregateKey( + source=source, agg="sum", + column_filter_key=SqlExprKey(canonical_sql="region_id = 1"), + ) + assert self._identity(rerooted=False, key=plain) != self._identity( + rerooted=False, key=filtered, + ) + + def test_identity_is_hashable(self) -> None: + """It is used as a dict key, so an unhashable member would surface as a + TypeError mid-render rather than at import.""" + assert len({self._identity(rerooted=False), self._identity(rerooted=True)}) == 2 From ee2adca62a78db95fe1096dfda2eba756fecd8d8 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Wed, 5 Aug 2026 21:43:14 +0200 Subject: [PATCH 14/98] =?UTF-8?q?DEV-1744:=20group=20unary=20operands=20to?= =?UTF-8?q?o=20=E2=80=94=20-(a+b)=20was=20rendering=20as=20-a=20+=20b?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The unary branches were added earlier in this PR to stop `-10` losing its sign, but they passed the operand straight into `exp.Neg` / `exp.Not` without the precedence pass the binary path uses: -(a + b) -> -a + b which is (-a) + b not (a and b) -> NOT a AND b which is (NOT a) AND b Both parse cleanly and both mean something else — the second is a De Morgan error, so it silently returns a different row set. `exp.Neg` joins the precedence table at 7 (unary minus binds tighter than any binary arithmetic) and both unary branches now route their operand through `_paren_if_lower_prec`. `NOT` stays at 3, so `NOT a > b` is left alone — NOT already binds looser than a comparison, and over-wrapping would be noise. Checked while fixing: nested negation emits `- -a` with a space, so there is no `--` comment hazard. Found by CodeRabbit on the same commit where Codex returned its first clean pass in six — a useful reminder that the two are not substitutes. Tests 9607 passing, ruff clean. Co-Authored-By: Claude Fable 5 --- slayer/sql/render/value_expr.py | 23 ++++++++++- tests/test_dev1744_value_expr.py | 65 ++++++++++++++++++++++++++++++++ 2 files changed, 86 insertions(+), 2 deletions(-) diff --git a/slayer/sql/render/value_expr.py b/slayer/sql/render/value_expr.py index f067ea31..aff67cad 100644 --- a/slayer/sql/render/value_expr.py +++ b/slayer/sql/render/value_expr.py @@ -201,6 +201,8 @@ def _literal(value: Any) -> exp.Expression: exp.Is: 4, exp.In: 4, exp.Like: 4, exp.Between: 4, exp.Add: 5, exp.Sub: 5, exp.Mul: 6, exp.Div: 6, exp.Mod: 6, + # Unary minus binds tighter than any binary arithmetic. + exp.Neg: 7, } # Operators taking exactly two operands. Left-folding a comparison would turn @@ -255,10 +257,27 @@ def _render_arithmetic( return operands[0] if len(operands) == 1: + # Unary operands need the same precedence treatment as binary ones. + # Without it ``-(a + b)`` emits ``-a + b`` and ``not (a and b)`` emits + # ``NOT a AND b`` — both parse cleanly and both mean something else. if op == "not": - return exp.Not(this=operands[0]) + return exp.Not( + this=_paren_if_lower_prec( + operands[0], + parent_prec=_PRECEDENCE[exp.Not], + is_right=False, + op=op, + ), + ) if op == "-": - return exp.Neg(this=operands[0]) + return exp.Neg( + this=_paren_if_lower_prec( + operands[0], + parent_prec=_PRECEDENCE[exp.Neg], + is_right=False, + op=op, + ), + ) if op == "+": return operands[0] raise NotImplementedError( diff --git a/tests/test_dev1744_value_expr.py b/tests/test_dev1744_value_expr.py index 1c75de79..ff23765d 100644 --- a/tests/test_dev1744_value_expr.py +++ b/tests/test_dev1744_value_expr.py @@ -2002,3 +2002,68 @@ async def test_null_free_in_list_still_executes(self, e2e) -> None: ) ) assert [r["orders.status"] for r in resp.data] == ["new"] + + +class TestUnaryOperandGrouping: + """The unary branches need the precedence pass too. + + Adding unary support earlier in this PR fixed the dropped sign but routed + the operand straight into ``exp.Neg`` / ``exp.Not`` without grouping it. + Both results parse cleanly and mean something else. + """ + + def test_negated_sum_keeps_its_parens(self) -> None: + """``-(a + b)`` must not flatten to ``-a + b``, which is ``(-a) + b``.""" + key = ArithmeticKey( + op="-", + operands=( + ArithmeticKey( + op="+", + operands=(ColumnKey(leaf="amount"), LiteralKey(value=Decimal(1))), + ), + ), + ) + out = _sql(render_value_key(key, _filter_ctx())) + assert out == "-(orders.amount + 1)", out + + def test_not_of_a_conjunction_keeps_its_parens(self) -> None: + """``NOT (a AND b)`` must not flatten to ``NOT a AND b``, which is + ``(NOT a) AND b`` — De Morgan, and a different row set.""" + key = ArithmeticKey( + op="not", + operands=( + ArithmeticKey( + op="and", + operands=( + ArithmeticKey( + op=">", + operands=(ColumnKey(leaf="amount"), LiteralKey(value=Decimal(1))), + ), + ArithmeticKey( + op="<", + operands=(ColumnKey(leaf="amount"), LiteralKey(value=Decimal(9))), + ), + ), + ), + ), + ) + out = _sql(render_value_key(key, _filter_ctx())) + assert out == "NOT (orders.amount > 1 AND orders.amount < 9)", out + + def test_not_of_a_comparison_needs_no_parens(self) -> None: + """Don't over-wrap: NOT binds looser than a comparison, so + ``NOT a > b`` already means ``NOT (a > b)``.""" + key = ArithmeticKey( + op="not", + operands=( + ArithmeticKey( + op=">", + operands=(ColumnKey(leaf="amount"), LiteralKey(value=Decimal(5))), + ), + ), + ) + assert _sql(render_value_key(key, _filter_ctx())) == "NOT orders.amount > 5" + + def test_negated_column_needs_no_parens(self) -> None: + key = ArithmeticKey(op="-", operands=(ColumnKey(leaf="amount"),)) + assert _sql(render_value_key(key, _filter_ctx())) == "-orders.amount" From 14baa64c2807c978e982b3fa45fe6db71228c811 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Wed, 5 Aug 2026 21:49:41 +0200 Subject: [PATCH 15/98] DEV-1745: test pack, the one Mode-A door (W1), W8 and the _cm_ fragment fix (W2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Test pack (commit 1 of the plan) plus the first implementation slices. Golden harness — deferred items 10 and 11: * SLAYER_UPDATE_GOLDEN=1 now rewrites ONLY keys listed in ALLOWED_DELTAS, so an unintended SQL change can no longer be blessed wholesale with 69 others. A blessed entry is stale by construction and must be deleted, which keeps every committed state's manifest empty. * Entries that raise record the full structured error instead of a bare "<>"; messages are byte-deterministic, so a different failure of the same type can no longer pass unnoticed. W1 — the one door on ScopeFrame: * enter_predicate / enter_expression over one implementation, differing only in the parse helper. Prequote, parse, scan, expand, re-parse, re-scan, union into join_paths, then Law 2. Discovery is a side effect of entering. * No qualification pass (D10): expand_derived_refs_sync already qualifies against the OWNING model and deliberately leaves opaque CTE/subquery refs alone; a blanket root pass would corrupt exactly those. * D1: parse failure raises ModeASqlParseError carrying the fragment and its location. The three swallow-all lanes are gone from the production path — including _filter_join_paths._scan, which turned an unparseable fragment into ZERO join paths, i.e. missing joins rather than an error. * SQLGenerator._parse and _parse_predicate now delegate to one shared render/parse module, so the door and the generator normalise identically. W2 — the _cm_ fragment-join bug: template fragments (string kwargs plus non-overridden AggregationParam.sql defaults) now register their crossed joins in the cross-model CTE, via the same helper the host path uses. Previously SUM(customers.spend * regions.weight) FROM customers emitted with no join to regions — SQL no database accepts. W8 — root-node derived expansion: a Column.sql that is exactly one bare reference to another derived column was silently not expanded, because col.replace() is a no-op when that column IS the parsed root. Golden matrix is unchanged except the five cm/fragment_default_crossing entries, which the W2 fix turns from ScopeLeakError into real SQL. Co-Authored-By: Claude Opus 5 (1M context) --- slayer/core/errors.py | 36 ++ slayer/engine/column_expansion.py | 36 +- slayer/sql/generator.py | 250 ++++++---- slayer/sql/render/parse.py | 143 ++++++ slayer/sql/scope.py | 146 +++++- tests/golden/dev1745_sql_baseline.json | 96 ++++ tests/test_dev1745_date_range_warning.py | 128 +++++ tests/test_dev1745_derived_expansion.py | 262 ++++++++++ tests/test_dev1745_fragment_joins.py | 222 +++++++++ tests/test_dev1745_golden_sql.py | 498 +++++++++++++++++++ tests/test_dev1745_mode_a_door.py | 360 ++++++++++++++ tests/test_dev1745_plan_time_routing.py | 182 +++++++ tests/test_dev1745_reachability.py | 434 +++++++++++++++++ tests/test_dev1745_warning_contract.py | 577 +++++++++++++++++++++++ 14 files changed, 3271 insertions(+), 99 deletions(-) create mode 100644 slayer/sql/render/parse.py create mode 100644 tests/golden/dev1745_sql_baseline.json create mode 100644 tests/test_dev1745_date_range_warning.py create mode 100644 tests/test_dev1745_derived_expansion.py create mode 100644 tests/test_dev1745_fragment_joins.py create mode 100644 tests/test_dev1745_golden_sql.py create mode 100644 tests/test_dev1745_mode_a_door.py create mode 100644 tests/test_dev1745_plan_time_routing.py create mode 100644 tests/test_dev1745_reachability.py create mode 100644 tests/test_dev1745_warning_contract.py diff --git a/slayer/core/errors.py b/slayer/core/errors.py index adacd55b..5ba198cd 100644 --- a/slayer/core/errors.py +++ b/slayer/core/errors.py @@ -186,6 +186,42 @@ def __init__( )) +class ModeASqlParseError(SlayerError, ValueError): + """A free-SQL (Mode-A) fragment could not be parsed. + + Mode-A text used to fail soft in three places: the fragment was handed + through unparsed, or — worse, in the join-path scanner — swallowed into + ZERO join paths, so an unparseable predicate emitted a query missing its + joins instead of reporting the problem. This is now the single loud + failure, carrying the fragment verbatim and the surface it came from + (``location``) so the author can find it. + + Multi-inherits ``ValueError`` for back-compat with call sites that already + catch ``ValueError`` around SQL text handling (see + :class:`UnknownReferenceError`). + """ + + def __init__( + self, + fragment: str, + location: str, + reason: str | None = None, + ) -> None: + self.fragment = fragment + self.location = location + self.reason = reason + super().__init__(_format_error_message( + cls_name=type(self).__name__, + summary=f"Cannot parse SQL fragment {fragment!r}.", + scope=location if reason is None else f"{location}: {reason}", + suggestion=( + "Mode-A surfaces take raw SQL for the target dialect. Check " + "the fragment for balanced parentheses and quotes, and that " + "any '{variable}' placeholders were supplied." + ), + )) + + class AmbiguousReferenceError(SlayerError, ValueError): """A reference matches multiple candidates in scope and can't pick one. diff --git a/slayer/engine/column_expansion.py b/slayer/engine/column_expansion.py index 27b890e3..baa22a2c 100644 --- a/slayer/engine/column_expansion.py +++ b/slayer/engine/column_expansion.py @@ -245,14 +245,20 @@ def _process_column_node_sync( visited: Tuple[Tuple[str, str], ...], is_root: bool, root_scope_ids: Set[int], -) -> None: +) -> Optional[exp.Expression]: """Expand one ``exp.Column`` node in place if it names a derived column. - Returns ``True`` when the node was rewritten, ``False`` when it was left - alone (a physical column, or an unresolvable/opaque alias path). + Returns the replacement expression when the node was rewritten, ``None`` + when it was left alone (a physical column — which is qualified in place — + or an unresolvable/opaque alias path). + + The return value matters only when ``col`` is the ROOT of the tree being + walked: ``Expression.replace`` swaps a node inside its parent, so on a + parentless root it is a no-op and the expansion would be computed and then + silently discarded. The caller rebinds the root from this return value. """ if col.args.get("db") or col.args.get("catalog"): - return + return None table_id = col.args.get("table") col_name = col.name table_alias = table_id.name if table_id is not None else alias_path @@ -264,13 +270,13 @@ def _process_column_node_sync( is_root=is_root, ) if target_model is None or canonical_alias is None: - return + return None target_col = target_model.get_column(col_name) if target_col is None or _is_trivial_base(column=target_col): col.set("table", exp.to_identifier(canonical_alias)) - return + return None if id(col) not in root_scope_ids: - return + return None next_is_root = is_root and (target_model is model) key = (target_model.name, col_name) if key in visited: @@ -287,9 +293,11 @@ def _process_column_node_sync( is_root=next_is_root, ) if expanded_sql is None: - return + return None expanded_ast = sqlglot.parse_one(expanded_sql, dialect=dialect) - col.replace(exp.Paren(this=expanded_ast)) + replacement = exp.Paren(this=expanded_ast) + col.replace(replacement) + return replacement def expand_derived_refs_sync( @@ -320,7 +328,7 @@ def expand_derived_refs_sync( column_nodes = list(parsed.find_all(exp.Column)) root_scope_ids = _root_scope_column_ids(parsed=parsed) for col in column_nodes: - _process_column_node_sync( + replacement = _process_column_node_sync( col=col, model=model, alias_path=alias_path, @@ -330,4 +338,12 @@ def expand_derived_refs_sync( is_root=is_root, root_scope_ids=root_scope_ids, ) + # ``col.replace`` mutates the node's PARENT. When the whole fragment is + # a single column reference — ``Column.sql = "other_derived_col"``, an + # alias of another derived column — that column IS ``parsed`` and has + # no parent, so the replace is a silent no-op. Rebind the root here, or + # the correctly-expanded SQL is computed and thrown away and the + # emitted query references a derived column as though it were physical. + if replacement is not None and col is parsed: + parsed = replacement return parsed.sql(dialect=dialect) diff --git a/slayer/sql/generator.py b/slayer/sql/generator.py index 036f367a..4545bd50 100644 --- a/slayer/sql/generator.py +++ b/slayer/sql/generator.py @@ -2508,11 +2508,13 @@ def _resolve_column_filter(key) -> None: cfk = key.column_filter_key if cfk is None or not cfk.canonical_sql: return - for p in self._filter_join_paths( - sql=cfk.canonical_sql, source_relation=scope.root_relation, - source_model=scope.root_model, bundle=scope.bundle, - ): - scope.join_paths.add(p) + # Entering registers the crossed joins on ``scope`` as a side + # effect (Law 1 / P-A); the AST itself is re-rendered later by the + # aggregate CASE-WHEN wrapper, so it is discarded here. + self._enter_mode_a_predicate( + sql=cfk.canonical_sql, scope=scope, + location=f"Column.filter on model {scope.root_model.name!r}", + ) def _resolve_source(key) -> None: if isinstance(key.source, ColumnSqlKey): @@ -2533,27 +2535,11 @@ def _resolve_fragment_kwargs(key) -> None: # aggregation template as qualified SQL text, so their crossed # joins must register exactly like ``Column.filter`` predicates # do (the widened Law-3 trigger isolates on them, and the CTE - # sub-render lands here). Scanned with the same - # ``_filter_join_paths`` pipeline; unparseable fragments - # contribute nothing. - fragments = [v for _, v in key.kwargs if isinstance(v, str)] - agg_def = next( - (a for a in (scope.root_model.aggregations or []) - if a.name == key.agg), - None, + # sub-render lands here). Shared with the ``_cm_*`` CTE path so the + # two cannot drift apart again (DEV-1745 W2). + self._register_fragment_kwarg_joins( + key=key, scope=scope, model=scope.root_model, ) - if agg_def is not None: - overridden = {name for name, _ in key.kwargs} - fragments.extend( - p.sql for p in (agg_def.params or []) - if p.name not in overridden and p.sql - ) - for frag in fragments: - for p in self._filter_join_paths( - sql=frag, source_relation=scope.root_relation, - source_model=scope.root_model, bundle=scope.bundle, - ): - scope.join_paths.add(p) def _resolve_first_last_time_arg(key) -> None: # DEV-1710 Stage 6 — a first/last explicit ranking-time arg @@ -5860,15 +5846,30 @@ def _render_cross_model_cte( # NOSONAR(S3776) — single conceptual unit: share def _register_filter_join_paths(sql_text: Optional[str]) -> None: if not sql_text: return - for p in self._filter_join_paths( - sql=sql_text, source_relation=target_relation, - source_model=target_model, bundle=bundle, - ): - cte_scope.join_paths.add(p) + self._enter_mode_a_predicate( + sql=sql_text, scope=cte_scope, + location=( + f"Column.filter on cross-model target " + f"{target_model_name!r}" + ), + ) if local_agg_key.column_filter_key is not None: _register_filter_join_paths(local_agg_key.column_filter_key.canonical_sql) + # DEV-1745 (W2): the aggregation's template FRAGMENTS — string kwargs + # plus the non-overridden ``AggregationParam.sql`` defaults — substitute + # verbatim into the CTE's aggregate expression, so the joins they cross + # belong in this CTE's FROM. The host path has always registered them; + # this path never did, which is why a default like ``w='regions.weight'`` + # emitted ``SUM(customers.spend * regions.weight) FROM customers`` with + # no join to regions. Routing the fragments through the door makes the + # registration a side effect of resolving them, so the two paths cannot + # drift apart again. + self._register_fragment_kwarg_joins( + key=local_agg_key, scope=cte_scope, model=target_model, + ) + where_parts: List[exp.Expression] = [] for filter_text in plan.target_model_filters: # DEV-1450 #4b / DEV-1494: a target model filter referencing a @@ -5876,27 +5877,13 @@ def _register_filter_join_paths(sql_text: Optional[str]) -> None: # derived column on a joined model) is inline-expanded; base-only # filters keep the AST bare-ref qualification. The crossed join is # pulled into this CTE's FROM via ``cte_scope.join_paths``. - _register_filter_join_paths(filter_text) - qualified = self._render_mode_a_predicate( - sql=filter_text, - source_model=target_model, - source_relation=target_relation, - bundle=bundle, - qualify_fallback=lambda s: self._qualify_column_filter_sql( - canonical_sql=s, - source_relation=target_relation, - source_model=target_model, + where_parts.append(self._enter_mode_a_predicate( + sql=filter_text, scope=cte_scope, + location=( + f"SlayerModel.filters on cross-model target " + f"{target_model_name!r}" ), - ) - if not qualified: - continue - try: - where_parts.append(self._parse_predicate(qualified)) - except Exception: - raise ValueError( - f"Target model filter on {target_model_name!r} could " - f"not be parsed: {filter_text!r}", - ) + )) # DEV-1708 / Codex F4: pre-pass — walk the FULL ValueKey tree of every # routed WHERE and HAVING filter (nested arithmetic / boolean / IN # operands, aggregate leaves' source + args + kwargs + column_filter, @@ -6165,11 +6152,10 @@ def _register_agg_key_joins( scope.resolve(v) cfk = local_agg.column_filter_key if cfk is not None and cfk.canonical_sql: - for p in self._filter_join_paths( - sql=cfk.canonical_sql, source_relation=target_relation, - source_model=target_model, bundle=bundle, - ): - scope.join_paths.add(p) + self._enter_mode_a_predicate( + sql=cfk.canonical_sql, scope=scope, + location=f"Column.filter on model {target_model.name!r}", + ) def _collect_routed_filters( self, @@ -7514,18 +7500,19 @@ def _shifted_where_part( ) return rendered.sql(dialect=self.dialect), paths if fp.text is not None: - qualified = self._render_model_filter_sql( - sql=fp.text, - columns=fp.text_columns, + # One entry does both jobs: the returned AST is what the shifted + # CTE renders, and the paths it crossed were registered on the + # frame while entering (P-A — discovery cannot be forgotten). + frame = self._mode_a_scope( source_model=source_model, source_relation=source_relation, bundle=bundle, ) - paths = self._filter_join_paths( - sql=qualified, source_relation=source_relation, - source_model=source_model, bundle=bundle, + rendered = frame.enter_predicate( + fp.text, + location=f"SlayerModel.filters on model {source_model.name!r}", ) - return qualified, paths + return rendered.sql(dialect=self.dialect), frame.join_paths.as_list() return None def _emit_time_shift_ctes_for_planned( # NOSONAR(S3776) — single conceptual unit for one time_shift slot: partition/time resolution through the shifted ScopeFrame + shifted-CTE body assembly + collision-safe CTE naming (cte_allocator) + sjoin grain join-back, all sharing tightly-coupled per-slot state (time_alias / input_alias / partition_specs / shifted_cte_name / carry aliases). Splitting forces that cross-cutting state through many-argument helpers without simplifying anything — same shape as the sibling _render_cross_model_cte's suppression. @@ -7748,13 +7735,15 @@ def _add_partition(pk_obj, *, where: str) -> None: for _kname, _kval in inner_key.kwargs: if isinstance(_kval, (ColumnKey, ColumnSqlKey)): shifted_scope.resolve(_kval) - if inner_key.column_filter_key is not None: - for _p in self._filter_join_paths( + if ( + inner_key.column_filter_key is not None + and inner_key.column_filter_key.canonical_sql + ): + self._enter_mode_a_predicate( sql=inner_key.column_filter_key.canonical_sql, - source_relation=source_relation, - source_model=source_model, bundle=bundle, - ): - shifted_scope.join_paths.add(_p) + scope=shifted_scope, + location=f"Column.filter on model {source_model.name!r}", + ) # Build the shifted time-column expression. Calendar offset is # ``-periods`` units in the SHIFT granularity (periods=-1 -> +1 unit). @@ -8422,6 +8411,97 @@ def _scan(text: Optional[str]) -> None: _scan(rendered) return ordered + # ---- The one Mode-A door, generator side (P-A) ------------------------- + def _mode_a_scope( + self, *, source_model, source_relation: str, bundle, + ) -> ScopeFrame: + """An ephemeral :class:`ScopeFrame` for a Mode-A entry whose call site + holds no scope. + + Pure RENDER paths (the aggregate CASE-WHEN wrapper, the WHERE/HAVING + assembler) run after the corresponding registration pass has already + put the crossed joins into the real scope, so the frame here exists + only to give the text one consistent door to come through — its + ``join_paths`` are a byproduct nobody reads. Every site that still owns + discovery passes its real scope instead. + """ + return ScopeFrame( + scope_id=f"_modea_{source_relation}", + root_model=source_model, + root_relation=source_relation, + bundle=bundle, + dialect=self._dialect, + allocator=AliasAllocator(), + ) + + def _enter_mode_a_predicate( + self, + *, + sql: str, + scope: Optional[ScopeFrame] = None, + source_model=None, + source_relation: Optional[str] = None, + bundle=None, + location: Optional[str] = None, + ) -> exp.Expression: + """Enter a Mode-A PREDICATE through the door and hand back its AST. + + The grammar is fixed by the caller's surface (``Column.filter`` and + ``SlayerModel.filters`` are predicates), never sniffed from the text. + """ + frame = scope or self._mode_a_scope( + source_model=source_model, + source_relation=source_relation, + bundle=bundle, + ) + return frame.enter_predicate(sql, location=location) + + def _enter_mode_a_expression( + self, + *, + sql: str, + scope: ScopeFrame, + location: Optional[str] = None, + ) -> exp.Expression: + """Enter a Mode-A scalar EXPRESSION (a ``Column.sql`` / aggregation + template fragment) through the door.""" + return scope.enter_expression(sql, location=location) + + def _register_fragment_kwarg_joins( + self, *, key, scope: ScopeFrame, model, + ) -> None: + """Register the joins an aggregation's template FRAGMENTS cross. + + The sources are the aggregate's string kwargs plus the non-overridden + ``AggregationParam.sql`` defaults of the aggregation named by + ``key.agg`` on ``model``. Both substitute verbatim into the rendered + aggregate expression, so whatever they reach has to be in the FROM. + + Shared by the host base SELECT and the ``_cm_*`` cross-model CTE. The + host path always did this; the CTE path did not, and emitted + ``SUM(customers.spend * regions.weight) FROM customers`` — SQL no + database accepts. One implementation, entered through the one door, is + what stops that from recurring (DEV-1745 W2). + """ + fragments = [v for _, v in key.kwargs if isinstance(v, str)] + agg_def = next( + (a for a in (model.aggregations or []) if a.name == key.agg), None, + ) + if agg_def is not None: + overridden = {name for name, _ in key.kwargs} + fragments.extend( + p.sql for p in (agg_def.params or []) + if p.name not in overridden and p.sql + ) + for frag in fragments: + self._enter_mode_a_expression( + sql=frag, scope=scope, + location=( + f"aggregation {key.agg!r} template fragment on model " + f"{model.name!r}" + ), + ) + def _expand_derived_row_dims( # NOSONAR(S3776) — one cohesive per-slot pass expanding derived ROW/TIME dimensions and registering the joins they cross. self, *, base_render_order, slots_by_id, source_relation: str, source_model, bundle, scope: ScopeFrame, @@ -8514,23 +8594,23 @@ def _expand_column_filter_sql( resolve; otherwise qualifies bare refs. DEV-1494; see ``_render_mode_a_predicate``. """ + if not canonical_sql: + return None if bundle is None: + # No bundle means no join graph to expand or scan against — the + # AST bare-ref qualification is all that is available. return self._qualify_column_filter_sql( canonical_sql=canonical_sql, source_relation=source_relation, source_model=source_model, ) - return self._render_mode_a_predicate( + return self._enter_mode_a_predicate( sql=canonical_sql, source_model=source_model, source_relation=source_relation, bundle=bundle, - qualify_fallback=lambda s: self._qualify_column_filter_sql( - canonical_sql=s, - source_relation=source_relation, - source_model=source_model, - ), - ) + location=f"Column.filter on model {source_model.name!r}", + ).sql(dialect=self.dialect) def _build_from_clause_from_planned( self, @@ -8748,11 +8828,13 @@ def _resolve_where_filter_joins_via_scope( # to a constant) AND the inline-expanded text (a bare/dotted # DERIVED ref like ``is_eu`` surfaces the join its expansion # crosses). See ``_filter_join_paths``. - for p in self._filter_join_paths( - sql=fp.text, source_relation=scope.root_relation, - source_model=scope.root_model, bundle=scope.bundle, - ): - scope.join_paths.add(p) + self._enter_mode_a_predicate( + sql=fp.text, scope=scope, + location=( + f"SlayerModel.filters on model " + f"{scope.root_model.name!r}" + ), + ) def _value_key_join_paths( # NOSONAR(S3776) — one cohesive recursive ValueKey-tree walk; complexity is the per-key-type dispatch. self, *, key, source_model, source_relation: str, bundle, @@ -9176,13 +9258,15 @@ def _build_where_having_from_planned( # NOSONAR(S3776) — one cohesive pass ov # DEV-1450 #4b: a reference to a non-trivial derived column # is inline-expanded (and pulls its crossed joins into the # FROM via _resolve_where_filter_joins_via_scope). - target_parts.append(self._render_model_filter_sql( + target_parts.append(self._enter_mode_a_predicate( sql=fp.text, - columns=fp.text_columns, source_model=source_model, source_relation=source_relation, bundle=bundle, - )) + location=( + f"SlayerModel.filters on model {source_model.name!r}" + ), + ).sql(dialect=self.dialect)) else: raise ValueError( f"FilterPhase id={fp.id!r} has neither expression " diff --git a/slayer/sql/render/parse.py b/slayer/sql/render/parse.py new file mode 100644 index 00000000..01997a5e --- /dev/null +++ b/slayer/sql/render/parse.py @@ -0,0 +1,143 @@ +"""One parse of free SQL text into a SLayer-normalised sqlglot AST. + +``SQLGenerator._parse`` and ``SQLGenerator._parse_predicate`` were byte-for-byte +identical apart from a single line — how the text reaches sqlglot. Everything +after that (the dialect-keyed parse rewrite, the log-alias rewrite, mixed-case +identifier quoting, the dialect-keyed target rewrite) was duplicated. Both now +delegate here, so the Mode-A door on :class:`~slayer.sql.scope.ScopeFrame` and +the generator normalise a fragment exactly the same way — which is what lets the +door take over a call site without changing the SQL it emits. + +The two surfaces differ ONLY in the parse step, and that difference is a +property of the SQL *grammar* being read, not a behaviour flag: + +* :func:`parse_expression` — a scalar expression (``Column.sql``). +* :func:`parse_predicate` — a boolean predicate (``Column.filter``, + ``SlayerModel.filters``). Wrapped as ``SELECT 1 WHERE ...`` so sqlglot reads a + leading function name that is also a statement keyword (``replace(x, ',', '')`` + on SQLite/MySQL) as a function call rather than a statement. +""" + +from __future__ import annotations + +from typing import Optional + +import sqlglot +from sqlglot import exp + +from slayer.sql.dialects.base import SqlDialect +from slayer.sql.naming import quote_mixed_case_identifiers +from slayer.sql.reserved_keywords import prequote_reserved_identifiers + + +def rewrite_log_aliases(node: exp.Expression, *, dialect: SqlDialect) -> exp.Expression: + """DEV-1337: rewrite ``Log(this=Literal(10|2), expression=X)`` back to + ``Anonymous(this='log10'|'log2', expressions=[X])`` for dialects with native + single-arg aliases. + + Applied to every parsed AST so the rewrite survives sqlglot's re-parse + passes, which would otherwise turn ``LOG10(x)`` back into a generic ``Log`` + node and re-emit it as ``LOG(10, x)``. No-op on non-``Log`` nodes and on + ``Log`` nodes with a non-literal or non-{10,2} base. + """ + if not isinstance(node, exp.Log): + return node + base = node.args.get("this") + arg = node.args.get("expression") + if arg is None or not isinstance(base, exp.Literal) or base.is_string: + return node + try: + base_val = float(base.this) + except (TypeError, ValueError): + return node + if base_val == 10 and dialect.should_use_native_log(10): + return exp.Anonymous(this="log10", expressions=[arg.copy()]) + if base_val == 2 and dialect.should_use_native_log(2): + return exp.Anonymous(this="log2", expressions=[arg.copy()]) + return node + + +def apply_ast_rewrites( + *, + tree: exp.Expression, + target_dialect: SqlDialect, + parse_dialect: SqlDialect, +) -> exp.Expression: + """The SLayer normalisation every parsed fragment gets, in order. + + ``parse_dialect`` keys the rewrite that depends on how the text was READ + (DEV-1716 — SQLite rewrites ``JSONExtract`` to the function-call form); + ``target_dialect`` keys the log-alias rewrite and the emit-side rewrite + (Postgres casts the first argument of a 2-arg ``ROUND``). They differ only + when a caller parses one dialect's text for another dialect's output. + """ + tree = parse_dialect.rewrite_parsed_ast(tree) + tree = tree.transform( + lambda node: rewrite_log_aliases(node, dialect=target_dialect), + ) + # DEV-1645: quote mixed-case column/table identifiers so case-folding + # dialects reach the right physical object. + tree = tree.transform(quote_mixed_case_identifiers) + return target_dialect.rewrite_target_ast(tree) + + +def _prequote(sql: str, *, parse_dialect: SqlDialect) -> str: + """DEV-1686: quote reserved-word qualifiers/leaves (``grant.id`` → + ``"grant".id``) before parsing, so a bare reserved word does not fail at + parse time. No-op for ordinary SQL and idempotent when already quoted.""" + return prequote_reserved_identifiers(sql, dialect=parse_dialect.sqlglot_name) + + +def parse_expression( + *, + sql: str, + target_dialect: SqlDialect, + parse_dialect: Optional[SqlDialect] = None, + prequote: bool = True, +) -> exp.Expression: + """Parse a scalar SQL expression and apply the SLayer AST rewrites. + + ``prequote=False`` is for callers that have already prequoted and need the + prequoted text kept as a distinct representation (the Mode-A door, which + parses the prequoted form twice — once raw, once expanded). + """ + parse_dialect = parse_dialect or target_dialect + if prequote: + sql = _prequote(sql, parse_dialect=parse_dialect) + tree = sqlglot.parse_one(sql, dialect=parse_dialect.sqlglot_name) + return apply_ast_rewrites( + tree=tree, target_dialect=target_dialect, parse_dialect=parse_dialect, + ) + + +def parse_predicate( + *, + sql: str, + target_dialect: SqlDialect, + parse_dialect: Optional[SqlDialect] = None, + prequote: bool = True, +) -> exp.Expression: + """Parse a bare WHERE/HAVING predicate and apply the SLayer AST rewrites. + + ``sqlglot.parse_one`` falls back to a ``Command`` statement parse when an + expression starts with a function name that is also a statement keyword in + the target dialect. Wrapping in ``SELECT 1 WHERE ...`` puts sqlglot in + expression context, where the same text reads as a function call. + """ + parse_dialect = parse_dialect or target_dialect + if prequote: + sql = _prequote(sql, parse_dialect=parse_dialect) + wrapped = sqlglot.parse_one( + f"SELECT 1 WHERE {sql}", dialect=parse_dialect.sqlglot_name, + ) + where = wrapped.args.get("where") + if where is None or where.this is None: # pragma: no cover — defensive + raise ValueError( + f"Could not extract WHERE predicate from {sql!r} " + f"(dialect={parse_dialect.sqlglot_name!r})" + ) + return apply_ast_rewrites( + tree=where.this, + target_dialect=target_dialect, + parse_dialect=parse_dialect, + ) diff --git a/slayer/sql/scope.py b/slayer/sql/scope.py index 3d231e8a..bf482586 100644 --- a/slayer/sql/scope.py +++ b/slayer/sql/scope.py @@ -26,8 +26,9 @@ import sqlglot from pydantic import BaseModel, ConfigDict, Field from sqlglot import exp +from sqlglot.errors import ParseError -from slayer.core.errors import UnknownReferenceError +from slayer.core.errors import ModeASqlParseError, UnknownReferenceError from slayer.core.keys import ColumnKey, ColumnSqlKey from slayer.core.models import SlayerModel from slayer.engine.column_expansion import ( @@ -37,6 +38,7 @@ from slayer.engine.source_bundle import ResolvedSourceBundle from slayer.sql.dialects.base import SqlDialect from slayer.sql.naming import AliasAllocator +from slayer.sql.render.parse import parse_expression, parse_predicate from slayer.sql.reserved_keywords import ( install_reserved_keywords, prequote_reserved_identifiers, @@ -45,6 +47,11 @@ # The resolver relies on sqlglot's reserved-word quoting on emit (DEV-1686). install_reserved_keywords() +# The two Mode-A grammars. A static property of the surface being read, chosen +# by the call site — never sniffed from the text (see ``ScopeFrame._enter``). +_PREDICATE = "predicate" +_EXPRESSION = "expression" + # A ref that can enter a scope. Stage 2 exercises structural column refs, derived # columns, and free Mode-A / predicate text; later stages widen this union. Ref = Union[ColumnKey, ColumnSqlKey, str] @@ -108,19 +115,146 @@ def resolve(self, ref: Ref, *, consumer: "ScopeFrame | None" = None) -> exp.Expr alias for the consumer. """ template = self._anchor(ref) + self._register_join_paths(template) + return self._close(template, consumer=consumer) + + # ---- The one Mode-A door (P-A) ----------------------------------------- + def enter_predicate( + self, + sql: str, + *, + consumer: "ScopeFrame | None" = None, + location: Optional[str] = None, + ) -> exp.Expression: + """Enter a Mode-A boolean PREDICATE (``Column.filter``, model + ``filters``) into this scope. See :meth:`_enter`.""" + return self._enter( + sql, grammar=_PREDICATE, consumer=consumer, location=location, + ) + + def enter_expression( + self, + sql: str, + *, + consumer: "ScopeFrame | None" = None, + location: Optional[str] = None, + ) -> exp.Expression: + """Enter a Mode-A scalar EXPRESSION (``Column.sql``) into this scope. + See :meth:`_enter`.""" + return self._enter( + sql, grammar=_EXPRESSION, consumer=consumer, location=location, + ) + + def _enter( + self, + sql: str, + *, + grammar: str, + consumer: "ScopeFrame | None", + location: Optional[str], + ) -> exp.Expression: + """The single implementation behind both Mode-A surfaces. + + One pass, in order: + + 1. prequote reserved identifiers (DEV-1686), + 2. parse the PREQUOTED text and scan it for crossed join paths, + 3. expand derived refs, parse the EXPANDED text and scan that too, + 4. union both scans into ``join_paths``, + 5. Law 2 — materialise for a named ``consumer``, else return the AST. + + Both scans are load-bearing (the DEV-1494 dual-scan contract): a dotted + ref whose derived column inlines to a constant vanishes from the + expanded AST, so only the pre-expansion scan sees its join; and a bare + derived ref only reveals the joins its expansion crosses AFTER + expanding. Discovery is a side effect of entering — it cannot be + forgotten by a caller. + + There is deliberately NO qualification step. ``expand_derived_refs_sync`` + already qualifies, against the OWNING model's canonical alias, and + deliberately leaves a node alone when its alias path does not resolve — + that is an opaque CTE / subquery reference. A blanket pass against + ``root_relation`` would fire on exactly those and corrupt them. + + ``grammar`` is fixed by the call site, never by the content: the surface + being read determines it (a ``Column.filter`` is always a predicate). + Sniffing content or retrying the other grammar would put classification + back into render time. + """ + prequoted = prequote_reserved_identifiers( + sql, dialect=self.dialect.sqlglot_name, + ) + raw_ast = self._parse_mode_a( + prequoted, grammar=grammar, fragment=sql, location=location, + ) + self._register_join_paths(raw_ast) + + expanded = expand_derived_refs_sync( + sql=prequoted, + model=self.root_model, + alias_path=self.root_relation, + resolve_model=self.bundle.get_referenced_model, + dialect=self.dialect.sqlglot_name, + is_root=True, + ) + if expanded is None or expanded == prequoted: + final = raw_ast + else: + final = self._parse_mode_a( + expanded, grammar=grammar, fragment=sql, location=location, + ) + self._register_join_paths(final) + + return self._close(final, consumer=consumer) + + def _parse_mode_a( + self, + text: str, + *, + grammar: str, + fragment: str, + location: Optional[str], + ) -> exp.Expression: + """Parse ``text`` under the surface's grammar, or RAISE (D1). + + Only sqlglot's ``ParseError`` is caught, and only to re-raise it as a + typed SLayer error naming the ORIGINAL author text. Nothing falls back + to the raw string and nothing degrades to "no join paths" — the two + soft failures this replaces both turned a broken fragment into silently + wrong SQL. + """ + parse = parse_predicate if grammar == _PREDICATE else parse_expression + try: + return parse(sql=text, target_dialect=self.dialect, prequote=False) + except ParseError as exc: + raise ModeASqlParseError( + fragment=fragment, + location=location or self._default_location(), + reason=str(exc).splitlines()[0] if str(exc) else None, + ) from exc + + def _default_location(self) -> str: + return f"Mode-A SQL in scope rooted at model {self.root_model.name!r}" + + def _register_join_paths(self, parsed: exp.Expression) -> None: + """Law 1's side effect: every join path ``parsed`` crosses is recorded + on this scope, so ``_build_from_and_joins`` emits the JOINs it needs.""" for path in collect_root_scope_joined_paths( - parsed=template, + parsed=parsed, source_model=self.root_model, source_relation=self.root_relation, bundle=self.bundle, ): self.join_paths.add(path) + def _close( + self, template: exp.Expression, *, consumer: "ScopeFrame | None", + ) -> exp.Expression: + """Law 2: materialise for a named consumer, else hand back a copy so a + caller attaching this into its tree can never corrupt a value the scope + (or another caller) also holds (D-L / M1).""" if consumer is not None and not self.may_inline(self.join_paths.as_list()): - alias = self._materialize(template) - return exp.column(alias) - # Return a copy so a caller attaching this into its tree can never - # corrupt a value the scope (or another caller) also holds (D-L / M1). + return exp.column(self._materialize(template)) return template.copy() def resolve_predicate_sql(self, ref: Ref) -> Optional[str]: diff --git a/tests/golden/dev1745_sql_baseline.json b/tests/golden/dev1745_sql_baseline.json new file mode 100644 index 00000000..f8652c04 --- /dev/null +++ b/tests/golden/dev1745_sql_baseline.json @@ -0,0 +1,96 @@ +{ + "cm/fragment_default_crossing::bigquery": { + "error": "ScopeLeakError", + "message": "Scope not closed \u2014 1 out-of-scope reference(s):\n - [unbound_table] regions.weight in CTE _cm_orders__customers__spend_wscaled_sum (bound sources: ['_base', 'customers'])\nSQL:\nWITH _base AS (\nSELECT\n orders.status AS `orders___status`\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__customers__spend_wscaled_sum AS (\nSELECT\n SUM(customers.spend * regions.weight) AS `orders___customers___spend_wscaled_sum`\nFROM customers AS customers\n)\nSELECT _base.`orders___status`, _cm_orders__customers__spend_wscaled_sum.`orders___customers___spend_wscaled_sum`\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_wscaled_sum" + }, + "cm/fragment_default_crossing::duckdb": { + "error": "ScopeLeakError", + "message": "Scope not closed \u2014 1 out-of-scope reference(s):\n - [unbound_table] regions.weight in CTE _cm_orders__customers__spend_wscaled_sum (bound sources: ['_base', 'customers'])\nSQL:\nWITH _base AS (\nSELECT\n orders.status AS \"orders.status\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__customers__spend_wscaled_sum AS (\nSELECT\n SUM(customers.spend * regions.weight) AS \"orders.customers.spend_wscaled_sum\"\nFROM customers AS customers\n)\nSELECT _base.\"orders.status\", _cm_orders__customers__spend_wscaled_sum.\"orders.customers.spend_wscaled_sum\"\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_wscaled_sum" + }, + "cm/fragment_default_crossing::postgres": { + "error": "ScopeLeakError", + "message": "Scope not closed \u2014 1 out-of-scope reference(s):\n - [unbound_table] regions.weight in CTE _cm_orders__customers__spend_wscaled_sum (bound sources: ['_base', 'customers'])\nSQL:\nWITH _base AS (\nSELECT\n orders.status AS \"orders.status\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__customers__spend_wscaled_sum AS (\nSELECT\n SUM(customers.spend * regions.weight) AS \"orders.customers.spend_wscaled_sum\"\nFROM customers AS customers\n)\nSELECT _base.\"orders.status\", _cm_orders__customers__spend_wscaled_sum.\"orders.customers.spend_wscaled_sum\"\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_wscaled_sum" + }, + "cm/fragment_default_crossing::sqlite": { + "error": "ScopeLeakError", + "message": "Scope not closed \u2014 1 out-of-scope reference(s):\n - [unbound_table] regions.weight in CTE _cm_orders__customers__spend_wscaled_sum (bound sources: ['_base', 'customers'])\nSQL:\nWITH _base AS (\nSELECT\n orders.status AS \"orders.status\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__customers__spend_wscaled_sum AS (\nSELECT\n SUM(customers.spend * regions.weight) AS \"orders.customers.spend_wscaled_sum\"\nFROM customers AS customers\n)\nSELECT _base.\"orders.status\", _cm_orders__customers__spend_wscaled_sum.\"orders.customers.spend_wscaled_sum\"\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_wscaled_sum" + }, + "cm/fragment_default_crossing::tsql": { + "error": "ScopeLeakError", + "message": "Scope not closed \u2014 1 out-of-scope reference(s):\n - [unbound_table] regions.weight in CTE _cm_orders__customers__spend_wscaled_sum (bound sources: ['_base', 'customers'])\nSQL:\nWITH _base AS (\nSELECT\n orders.status AS [orders___status]\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__customers__spend_wscaled_sum AS (\nSELECT\n SUM(customers.spend * regions.weight) AS [orders___customers___spend_wscaled_sum]\nFROM customers AS customers\n)\nSELECT _base.[orders___status], _cm_orders__customers__spend_wscaled_sum.[orders___customers___spend_wscaled_sum]\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_wscaled_sum" + }, + "cm/joined_measure::bigquery": "WITH _base AS (\nSELECT\n orders.status AS `orders___status`\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__customers__spend_sum AS (\nSELECT\n SUM(customers.spend) AS `orders___customers___spend_sum`\nFROM customers AS customers\n)\nSELECT _base.`orders___status`, _cm_orders__customers__spend_sum.`orders___customers___spend_sum`\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_sum", + "cm/joined_measure::duckdb": "WITH _base AS (\nSELECT\n orders.status AS \"orders.status\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__customers__spend_sum AS (\nSELECT\n SUM(customers.spend) AS \"orders.customers.spend_sum\"\nFROM customers AS customers\n)\nSELECT _base.\"orders.status\", _cm_orders__customers__spend_sum.\"orders.customers.spend_sum\"\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_sum", + "cm/joined_measure::postgres": "WITH _base AS (\nSELECT\n orders.status AS \"orders.status\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__customers__spend_sum AS (\nSELECT\n SUM(customers.spend) AS \"orders.customers.spend_sum\"\nFROM customers AS customers\n)\nSELECT _base.\"orders.status\", _cm_orders__customers__spend_sum.\"orders.customers.spend_sum\"\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_sum", + "cm/joined_measure::sqlite": "WITH _base AS (\nSELECT\n orders.status AS \"orders.status\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__customers__spend_sum AS (\nSELECT\n SUM(customers.spend) AS \"orders.customers.spend_sum\"\nFROM customers AS customers\n)\nSELECT _base.\"orders.status\", _cm_orders__customers__spend_sum.\"orders.customers.spend_sum\"\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_sum", + "cm/joined_measure::tsql": "WITH _base AS (\nSELECT\n orders.status AS [orders___status]\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__customers__spend_sum AS (\nSELECT\n SUM(customers.spend) AS [orders___customers___spend_sum]\nFROM customers AS customers\n)\nSELECT _base.[orders___status], _cm_orders__customers__spend_sum.[orders___customers___spend_sum]\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_sum", + "cm/outer_where_wrapper::bigquery": { + "error": "ScopeLeakError", + "message": "Scope not closed \u2014 2 out-of-scope reference(s):\n - [unbound_table] _base___orders.status in top-level SELECT (bound sources: ['_base', '_cm_orders__eu_amount_sum'])\n - [unbound_table] _cm_orders__eu_amount_sum___orders.status in top-level SELECT (bound sources: ['_base', '_cm_orders__eu_amount_sum'])\nSQL:\nWITH _base AS (\nSELECT\n orders.status AS `orders___status`\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__eu_amount_sum AS (\nSELECT\n orders.status AS `orders___status`,\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS FLOAT64) AS `orders___eu`\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n)\nSELECT _base.`orders___status`, _cm_orders__eu_amount_sum.`orders___eu`\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum ON `_base___orders`.`status` IS NOT DISTINCT FROM `_cm_orders__eu_amount_sum___orders`.`status`\nWHERE _cm_orders__eu_amount_sum.`orders___eu` > 100" + }, + "cm/outer_where_wrapper::duckdb": "WITH _base AS (\nSELECT\n orders.status AS \"orders.status\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__eu_amount_sum AS (\nSELECT\n orders.status AS \"orders.status\",\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS DOUBLE) AS \"orders.eu\"\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n)\nSELECT _base.\"orders.status\", _cm_orders__eu_amount_sum.\"orders.eu\"\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum ON _base.\"orders.status\" IS NOT DISTINCT FROM _cm_orders__eu_amount_sum.\"orders.status\"\nWHERE _cm_orders__eu_amount_sum.\"orders.eu\" > 100", + "cm/outer_where_wrapper::postgres": "WITH _base AS (\nSELECT\n orders.status AS \"orders.status\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__eu_amount_sum AS (\nSELECT\n orders.status AS \"orders.status\",\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS DOUBLE PRECISION) AS \"orders.eu\"\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n)\nSELECT _base.\"orders.status\", _cm_orders__eu_amount_sum.\"orders.eu\"\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum ON _base.\"orders.status\" IS NOT DISTINCT FROM _cm_orders__eu_amount_sum.\"orders.status\"\nWHERE _cm_orders__eu_amount_sum.\"orders.eu\" > 100", + "cm/outer_where_wrapper::sqlite": "WITH _base AS (\nSELECT\n orders.status AS \"orders.status\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__eu_amount_sum AS (\nSELECT\n orders.status AS \"orders.status\",\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS REAL) AS \"orders.eu\"\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n)\nSELECT _base.\"orders.status\", _cm_orders__eu_amount_sum.\"orders.eu\"\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum ON _base.\"orders.status\" IS _cm_orders__eu_amount_sum.\"orders.status\"\nWHERE _cm_orders__eu_amount_sum.\"orders.eu\" > 100", + "cm/outer_where_wrapper::tsql": "WITH _base AS (\nSELECT\n orders.status AS [orders___status]\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__eu_amount_sum AS (\nSELECT\n orders.status AS [orders___status],\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS FLOAT) AS [orders___eu]\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n)\nSELECT _base.[orders___status], _cm_orders__eu_amount_sum.[orders___eu]\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum ON (_base.[orders___status] = _cm_orders__eu_amount_sum.[orders___status] OR (_base.[orders___status] IS NULL AND _cm_orders__eu_amount_sum.[orders___status] IS NULL))\nWHERE _cm_orders__eu_amount_sum.[orders___eu] > 100", + "expand/derived_of_derived::bigquery": "SELECT\n CAST((\n customers__regions.population * 2\n ) AS FLOAT64) AS `orders___deep_pop`,\n CAST(SUM(orders.amount) AS FLOAT64) AS `orders___m`\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nLEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\nWHERE\n orders.amount >= 0\nGROUP BY\n CAST((\n customers__regions.population * 2\n ) AS FLOAT64)", + "expand/derived_of_derived::duckdb": "SELECT\n CAST((\n customers__regions.population * 2\n ) AS DOUBLE) AS \"orders.deep_pop\",\n CAST(SUM(orders.amount) AS DOUBLE) AS \"orders.m\"\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nLEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\nWHERE\n orders.amount >= 0\nGROUP BY\n CAST((\n customers__regions.population * 2\n ) AS DOUBLE)", + "expand/derived_of_derived::postgres": "SELECT\n CAST((\n customers__regions.population * 2\n ) AS DOUBLE PRECISION) AS \"orders.deep_pop\",\n CAST(SUM(orders.amount) AS DOUBLE PRECISION) AS \"orders.m\"\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nLEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\nWHERE\n orders.amount >= 0\nGROUP BY\n CAST((\n customers__regions.population * 2\n ) AS DOUBLE PRECISION)", + "expand/derived_of_derived::sqlite": "SELECT\n CAST((\n customers__regions.population * 2\n ) AS REAL) AS \"orders.deep_pop\",\n CAST(SUM(orders.amount) AS REAL) AS \"orders.m\"\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nLEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\nWHERE\n orders.amount >= 0\nGROUP BY\n CAST((\n customers__regions.population * 2\n ) AS REAL)", + "expand/derived_of_derived::tsql": "SELECT\n CAST((\n customers__regions.population * 2\n ) AS FLOAT) AS [orders___deep_pop],\n CAST(SUM(orders.amount) AS FLOAT) AS [orders___m]\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nLEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\nWHERE\n orders.amount >= 0\nGROUP BY\n CAST((\n customers__regions.population * 2\n ) AS FLOAT)", + "expand/multi_model_derived::bigquery": "SELECT\n CAST(customers.spend + customers__regions.population AS FLOAT64) AS `orders___multi_model`,\n CAST(SUM(orders.amount) AS FLOAT64) AS `orders___m`\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nLEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\nWHERE\n orders.amount >= 0\nGROUP BY\n CAST(customers.spend + customers__regions.population AS FLOAT64)", + "expand/multi_model_derived::duckdb": "SELECT\n CAST(customers.spend + customers__regions.population AS DOUBLE) AS \"orders.multi_model\",\n CAST(SUM(orders.amount) AS DOUBLE) AS \"orders.m\"\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nLEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\nWHERE\n orders.amount >= 0\nGROUP BY\n CAST(customers.spend + customers__regions.population AS DOUBLE)", + "expand/multi_model_derived::postgres": "SELECT\n CAST(customers.spend + customers__regions.population AS DOUBLE PRECISION) AS \"orders.multi_model\",\n CAST(SUM(orders.amount) AS DOUBLE PRECISION) AS \"orders.m\"\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nLEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\nWHERE\n orders.amount >= 0\nGROUP BY\n CAST(customers.spend + customers__regions.population AS DOUBLE PRECISION)", + "expand/multi_model_derived::sqlite": "SELECT\n CAST(customers.spend + customers__regions.population AS REAL) AS \"orders.multi_model\",\n CAST(SUM(orders.amount) AS REAL) AS \"orders.m\"\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nLEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\nWHERE\n orders.amount >= 0\nGROUP BY\n CAST(customers.spend + customers__regions.population AS REAL)", + "expand/multi_model_derived::tsql": "SELECT\n CAST(customers.spend + customers__regions.population AS FLOAT) AS [orders___multi_model],\n CAST(SUM(orders.amount) AS FLOAT) AS [orders___m]\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nLEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\nWHERE\n orders.amount >= 0\nGROUP BY\n CAST(customers.spend + customers__regions.population AS FLOAT)", + "host/column_filter_crossing::bigquery": { + "error": "ScopeLeakError", + "message": "Scope not closed \u2014 2 out-of-scope reference(s):\n - [unbound_table] _base___orders.status in top-level SELECT (bound sources: ['_base', '_cm_orders__eu_amount_sum'])\n - [unbound_table] _cm_orders__eu_amount_sum___orders.status in top-level SELECT (bound sources: ['_base', '_cm_orders__eu_amount_sum'])\nSQL:\nWITH _base AS (\nSELECT\n orders.status AS `orders___status`\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__eu_amount_sum AS (\nSELECT\n orders.status AS `orders___status`,\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS FLOAT64) AS `orders___m`\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n)\nSELECT _base.`orders___status`, _cm_orders__eu_amount_sum.`orders___m`\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum ON `_base___orders`.`status` IS NOT DISTINCT FROM `_cm_orders__eu_amount_sum___orders`.`status`" + }, + "host/column_filter_crossing::duckdb": "WITH _base AS (\nSELECT\n orders.status AS \"orders.status\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__eu_amount_sum AS (\nSELECT\n orders.status AS \"orders.status\",\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS DOUBLE) AS \"orders.m\"\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n)\nSELECT _base.\"orders.status\", _cm_orders__eu_amount_sum.\"orders.m\"\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum ON _base.\"orders.status\" IS NOT DISTINCT FROM _cm_orders__eu_amount_sum.\"orders.status\"", + "host/column_filter_crossing::postgres": "WITH _base AS (\nSELECT\n orders.status AS \"orders.status\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__eu_amount_sum AS (\nSELECT\n orders.status AS \"orders.status\",\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS DOUBLE PRECISION) AS \"orders.m\"\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n)\nSELECT _base.\"orders.status\", _cm_orders__eu_amount_sum.\"orders.m\"\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum ON _base.\"orders.status\" IS NOT DISTINCT FROM _cm_orders__eu_amount_sum.\"orders.status\"", + "host/column_filter_crossing::sqlite": "WITH _base AS (\nSELECT\n orders.status AS \"orders.status\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__eu_amount_sum AS (\nSELECT\n orders.status AS \"orders.status\",\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS REAL) AS \"orders.m\"\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n)\nSELECT _base.\"orders.status\", _cm_orders__eu_amount_sum.\"orders.m\"\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum ON _base.\"orders.status\" IS _cm_orders__eu_amount_sum.\"orders.status\"", + "host/column_filter_crossing::tsql": "WITH _base AS (\nSELECT\n orders.status AS [orders___status]\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__eu_amount_sum AS (\nSELECT\n orders.status AS [orders___status],\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS FLOAT) AS [orders___m]\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n)\nSELECT _base.[orders___status], _cm_orders__eu_amount_sum.[orders___m]\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum ON (_base.[orders___status] = _cm_orders__eu_amount_sum.[orders___status] OR (_base.[orders___status] IS NULL AND _cm_orders__eu_amount_sum.[orders___status] IS NULL))", + "host/column_sql_derived::bigquery": "SELECT\n CAST(orders.amount * 2 AS FLOAT64) AS `orders___doubled`,\n CAST(SUM(orders.amount) AS FLOAT64) AS `orders___m`\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n CAST(orders.amount * 2 AS FLOAT64)", + "host/column_sql_derived::duckdb": "SELECT\n CAST(orders.amount * 2 AS DOUBLE) AS \"orders.doubled\",\n CAST(SUM(orders.amount) AS DOUBLE) AS \"orders.m\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n CAST(orders.amount * 2 AS DOUBLE)", + "host/column_sql_derived::postgres": "SELECT\n CAST(orders.amount * 2 AS DOUBLE PRECISION) AS \"orders.doubled\",\n CAST(SUM(orders.amount) AS DOUBLE PRECISION) AS \"orders.m\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n CAST(orders.amount * 2 AS DOUBLE PRECISION)", + "host/column_sql_derived::sqlite": "SELECT\n CAST(orders.amount * 2 AS REAL) AS \"orders.doubled\",\n CAST(SUM(orders.amount) AS REAL) AS \"orders.m\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n CAST(orders.amount * 2 AS REAL)", + "host/column_sql_derived::tsql": "SELECT\n CAST(orders.amount * 2 AS FLOAT) AS [orders___doubled],\n CAST(SUM(orders.amount) AS FLOAT) AS [orders___m]\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n CAST(orders.amount * 2 AS FLOAT)", + "host/const_expanding_ref::bigquery": "SELECT\n orders.status AS `orders___status`,\n CAST(SUM(orders.amount) AS FLOAT64) AS `orders___m`\nFROM orders AS orders\nWHERE\n orders.amount >= 0 AND CAST(1 AS INT64) = 1\nGROUP BY\n orders.status", + "host/const_expanding_ref::duckdb": "SELECT\n orders.status AS \"orders.status\",\n CAST(SUM(orders.amount) AS DOUBLE) AS \"orders.m\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0 AND CAST(1 AS INT) = 1\nGROUP BY\n orders.status", + "host/const_expanding_ref::postgres": "SELECT\n orders.status AS \"orders.status\",\n CAST(SUM(orders.amount) AS DOUBLE PRECISION) AS \"orders.m\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0 AND CAST(1 AS INT) = 1\nGROUP BY\n orders.status", + "host/const_expanding_ref::sqlite": "SELECT\n orders.status AS \"orders.status\",\n CAST(SUM(orders.amount) AS REAL) AS \"orders.m\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0 AND CAST(1 AS INTEGER) = 1\nGROUP BY\n orders.status", + "host/const_expanding_ref::tsql": "SELECT\n orders.status AS [orders___status],\n CAST(SUM(orders.amount) AS FLOAT) AS [orders___m]\nFROM orders AS orders\nWHERE\n orders.amount >= 0 AND CAST(1 AS INTEGER) = 1\nGROUP BY\n orders.status", + "host/model_filter::bigquery": "SELECT\n orders.status AS `orders___status`,\n CAST(SUM(orders.amount) AS FLOAT64) AS `orders___m`\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status", + "host/model_filter::duckdb": "SELECT\n orders.status AS \"orders.status\",\n CAST(SUM(orders.amount) AS DOUBLE) AS \"orders.m\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status", + "host/model_filter::postgres": "SELECT\n orders.status AS \"orders.status\",\n CAST(SUM(orders.amount) AS DOUBLE PRECISION) AS \"orders.m\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status", + "host/model_filter::sqlite": "SELECT\n orders.status AS \"orders.status\",\n CAST(SUM(orders.amount) AS REAL) AS \"orders.m\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status", + "host/model_filter::tsql": "SELECT\n orders.status AS [orders___status],\n CAST(SUM(orders.amount) AS FLOAT) AS [orders___m]\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status", + "host/quoted_dotted_identifier::bigquery": "SELECT\n CAST('customers'.'spend' AS FLOAT64) AS `orders___quoted_cross`,\n CAST(SUM(orders.amount) AS FLOAT64) AS `orders___m`\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n CAST('customers'.'spend' AS FLOAT64)", + "host/quoted_dotted_identifier::duckdb": "SELECT\n customers.\"spend\" AS \"orders.quoted_cross\",\n CAST(SUM(orders.amount) AS DOUBLE) AS \"orders.m\"\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.amount >= 0\nGROUP BY\n customers.\"spend\"", + "host/quoted_dotted_identifier::postgres": "SELECT\n customers.\"spend\" AS \"orders.quoted_cross\",\n CAST(SUM(orders.amount) AS DOUBLE PRECISION) AS \"orders.m\"\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.amount >= 0\nGROUP BY\n customers.\"spend\"", + "host/quoted_dotted_identifier::sqlite": "SELECT\n customers.\"spend\" AS \"orders.quoted_cross\",\n CAST(SUM(orders.amount) AS REAL) AS \"orders.m\"\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.amount >= 0\nGROUP BY\n customers.\"spend\"", + "host/quoted_dotted_identifier::tsql": "SELECT\n customers.[spend] AS [orders___quoted_cross],\n CAST(SUM(orders.amount) AS FLOAT) AS [orders___m]\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.amount >= 0\nGROUP BY\n customers.[spend]", + "host/same_named_column_and_model::bigquery": "SELECT\n orders.status AS `orders___status`,\n CAST(SUM(orders.amount) AS FLOAT64) AS `orders___m`\nFROM orders AS orders\nWHERE\n orders.amount >= 0 AND orders.status = 'x'\nGROUP BY\n orders.status", + "host/same_named_column_and_model::duckdb": "SELECT\n orders.status AS \"orders.status\",\n CAST(SUM(orders.amount) AS DOUBLE) AS \"orders.m\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0 AND orders.status = 'x'\nGROUP BY\n orders.status", + "host/same_named_column_and_model::postgres": "SELECT\n orders.status AS \"orders.status\",\n CAST(SUM(orders.amount) AS DOUBLE PRECISION) AS \"orders.m\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0 AND orders.status = 'x'\nGROUP BY\n orders.status", + "host/same_named_column_and_model::sqlite": "SELECT\n orders.status AS \"orders.status\",\n CAST(SUM(orders.amount) AS REAL) AS \"orders.m\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0 AND orders.status = 'x'\nGROUP BY\n orders.status", + "host/same_named_column_and_model::tsql": "SELECT\n orders.status AS [orders___status],\n CAST(SUM(orders.amount) AS FLOAT) AS [orders___m]\nFROM orders AS orders\nWHERE\n orders.amount >= 0 AND orders.status = 'x'\nGROUP BY\n orders.status", + "host/statement_keyword_column::bigquery": "SELECT\n orders.`select` AS `orders___select`,\n CAST(SUM(orders.amount) AS FLOAT64) AS `orders___m`\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.`select`", + "host/statement_keyword_column::duckdb": "SELECT\n orders.\"select\" AS \"orders.select\",\n CAST(SUM(orders.amount) AS DOUBLE) AS \"orders.m\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.\"select\"", + "host/statement_keyword_column::postgres": "SELECT\n orders.\"select\" AS \"orders.select\",\n CAST(SUM(orders.amount) AS DOUBLE PRECISION) AS \"orders.m\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.\"select\"", + "host/statement_keyword_column::sqlite": "SELECT\n orders.\"select\" AS \"orders.select\",\n CAST(SUM(orders.amount) AS REAL) AS \"orders.m\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.\"select\"", + "host/statement_keyword_column::tsql": "SELECT\n orders.[select] AS [orders___select],\n CAST(SUM(orders.amount) AS FLOAT) AS [orders___m]\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.[select]", + "windowed/date_range_filter::bigquery": "SELECT\n DATE_TRUNC(orders.created_at, MONTH) AS `orders___created_at`,\n CAST(SUM(orders.amount) AS FLOAT64) AS `orders___m`\nFROM orders AS orders\nWHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\nGROUP BY\n DATE_TRUNC(orders.created_at, MONTH)", + "windowed/date_range_filter::duckdb": "SELECT\n DATE_TRUNC('MONTH', orders.created_at) AS \"orders.created_at\",\n CAST(SUM(orders.amount) AS DOUBLE) AS \"orders.m\"\nFROM orders AS orders\nWHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\nGROUP BY\n DATE_TRUNC('MONTH', orders.created_at)", + "windowed/date_range_filter::postgres": "SELECT\n DATE_TRUNC('MONTH', orders.created_at) AS \"orders.created_at\",\n CAST(SUM(orders.amount) AS DOUBLE PRECISION) AS \"orders.m\"\nFROM orders AS orders\nWHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\nGROUP BY\n DATE_TRUNC('MONTH', orders.created_at)", + "windowed/date_range_filter::sqlite": "SELECT\n STRFTIME('%Y-%m-01', orders.created_at) AS \"orders.created_at\",\n CAST(SUM(orders.amount) AS REAL) AS \"orders.m\"\nFROM orders AS orders\nWHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\nGROUP BY\n STRFTIME('%Y-%m-01', orders.created_at)", + "windowed/date_range_filter::tsql": "SELECT\n DATETRUNC(month, orders.created_at) AS [orders___created_at],\n CAST(SUM(orders.amount) AS FLOAT) AS [orders___m]\nFROM orders AS orders\nWHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\nGROUP BY\n DATETRUNC(month, orders.created_at)", + "windowed/src_scope::bigquery": { + "error": "ScopeLeakError", + "message": "Scope not closed \u2014 2 out-of-scope reference(s):\n - [unbound_table] _base___orders.created_at in top-level SELECT (bound sources: ['_base', '_cm_orders__eu_amount_sum'])\n - [unbound_table] _cm_orders__eu_amount_sum___orders.created_at in top-level SELECT (bound sources: ['_base', '_cm_orders__eu_amount_sum'])\nSQL:\nWITH _base AS (\nSELECT\n DATE_TRUNC(orders.created_at, MONTH) AS `orders___created_at`\nFROM orders AS orders\nWHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\nGROUP BY\n DATE_TRUNC(orders.created_at, MONTH)\n), _cm_orders__eu_amount_sum AS (\nSELECT\n DATE_TRUNC(orders.created_at, MONTH) AS `orders___created_at`,\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS FLOAT64) AS `orders___m`\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\nGROUP BY\n DATE_TRUNC(orders.created_at, MONTH)\n)\nSELECT _base.`orders___created_at`, _cm_orders__eu_amount_sum.`orders___m`\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum ON `_base___orders`.`created_at` IS NOT DISTINCT FROM `_cm_orders__eu_amount_sum___orders`.`created_at`" + }, + "windowed/src_scope::duckdb": "WITH _base AS (\nSELECT\n DATE_TRUNC('MONTH', orders.created_at) AS \"orders.created_at\"\nFROM orders AS orders\nWHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\nGROUP BY\n DATE_TRUNC('MONTH', orders.created_at)\n), _cm_orders__eu_amount_sum AS (\nSELECT\n DATE_TRUNC('MONTH', orders.created_at) AS \"orders.created_at\",\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS DOUBLE) AS \"orders.m\"\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\nGROUP BY\n DATE_TRUNC('MONTH', orders.created_at)\n)\nSELECT _base.\"orders.created_at\", _cm_orders__eu_amount_sum.\"orders.m\"\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum ON _base.\"orders.created_at\" IS NOT DISTINCT FROM _cm_orders__eu_amount_sum.\"orders.created_at\"", + "windowed/src_scope::postgres": "WITH _base AS (\nSELECT\n DATE_TRUNC('MONTH', orders.created_at) AS \"orders.created_at\"\nFROM orders AS orders\nWHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\nGROUP BY\n DATE_TRUNC('MONTH', orders.created_at)\n), _cm_orders__eu_amount_sum AS (\nSELECT\n DATE_TRUNC('MONTH', orders.created_at) AS \"orders.created_at\",\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS DOUBLE PRECISION) AS \"orders.m\"\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\nGROUP BY\n DATE_TRUNC('MONTH', orders.created_at)\n)\nSELECT _base.\"orders.created_at\", _cm_orders__eu_amount_sum.\"orders.m\"\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum ON _base.\"orders.created_at\" IS NOT DISTINCT FROM _cm_orders__eu_amount_sum.\"orders.created_at\"", + "windowed/src_scope::sqlite": "WITH _base AS (\nSELECT\n STRFTIME('%Y-%m-01', orders.created_at) AS \"orders.created_at\"\nFROM orders AS orders\nWHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\nGROUP BY\n STRFTIME('%Y-%m-01', orders.created_at)\n), _cm_orders__eu_amount_sum AS (\nSELECT\n STRFTIME('%Y-%m-01', orders.created_at) AS \"orders.created_at\",\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS REAL) AS \"orders.m\"\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\nGROUP BY\n STRFTIME('%Y-%m-01', orders.created_at)\n)\nSELECT _base.\"orders.created_at\", _cm_orders__eu_amount_sum.\"orders.m\"\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum ON _base.\"orders.created_at\" IS _cm_orders__eu_amount_sum.\"orders.created_at\"", + "windowed/src_scope::tsql": "WITH _base AS (\nSELECT\n DATETRUNC(month, orders.created_at) AS [orders___created_at]\nFROM orders AS orders\nWHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\nGROUP BY\n DATETRUNC(month, orders.created_at)\n), _cm_orders__eu_amount_sum AS (\nSELECT\n DATETRUNC(month, orders.created_at) AS [orders___created_at],\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS FLOAT) AS [orders___m]\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\nGROUP BY\n DATETRUNC(month, orders.created_at)\n)\nSELECT _base.[orders___created_at], _cm_orders__eu_amount_sum.[orders___m]\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum ON (_base.[orders___created_at] = _cm_orders__eu_amount_sum.[orders___created_at] OR (_base.[orders___created_at] IS NULL AND _cm_orders__eu_amount_sum.[orders___created_at] IS NULL))" +} diff --git a/tests/test_dev1745_date_range_warning.py b/tests/test_dev1745_date_range_warning.py new file mode 100644 index 00000000..053d3e07 --- /dev/null +++ b/tests/test_dev1745_date_range_warning.py @@ -0,0 +1,128 @@ +"""DEV-1745 (W6) — P0 normalization warning for a malformed ``date_range``. + +Keep-list item 1 is RATIFIED: a malformed ``date_range`` keeps its silent +no-op in the planner (``stage_planner.py``: ``if not td.date_range or +len(td.date_range) != 2: continue``). This adds a normalization WARNING so the +drop stops being invisible — **no behavior change**. + +Trigger (D7): ``date_range is not None and len(date_range) != 2``. That is +exactly the planner's own drop condition, so the warning fires if and only if +the range is actually ignored — covering ``[]``, a single element, and 3+. +``date_range=None`` is legitimately absent and never warns. + +``normalize_query`` does not inspect ``query.time_dimensions`` at all today, so +this is a new rule. It runs per stage, so nested stages are covered. +""" + +from __future__ import annotations + +import warnings + +import pytest + +from slayer.core.enums import DataType +from slayer.core.models import Column, SlayerModel +from slayer.core.query import SlayerQuery +from slayer.engine.normalization import normalize_query + +from tests._engine_helpers import _engine_generate + + +def _orders() -> SlayerModel: + return SlayerModel( + name="orders", data_source="test", sql_table="orders", + columns=[ + Column(name="id", type=DataType.INT, primary_key=True), + Column(name="amount", type=DataType.DOUBLE), + Column(name="created_at", type=DataType.TIMESTAMP), + ], + ) + + +def _query(date_range) -> SlayerQuery: + td: dict = {"dimension": "created_at", "granularity": "month"} + if date_range is not None: + td["date_range"] = date_range + return SlayerQuery( + source_model="orders", + time_dimensions=[td], + measures=[{"formula": "amount:sum", "name": "m0"}], + ) + + +def _warnings_for(date_range) -> list: + """Normalization warnings whose rule concerns date_range.""" + result = normalize_query(query=_query(date_range)) + return [ + w for w in result.warnings + if "date_range" in (w.rule_id or "").lower() + or "date_range" in (w.original or "") + ] + + +MALFORMED = [ + pytest.param([], id="empty"), + pytest.param(["2024-01-01"], id="single"), + pytest.param(["2024-01-01", "2024-06-30", "2024-12-31"], id="three"), +] + + +class TestMalformedDateRangeWarns: + + @pytest.mark.parametrize("date_range", MALFORMED) + def test_warns(self, date_range) -> None: + assert _warnings_for(date_range), ( + f"no normalization warning for malformed date_range={date_range!r}" + ) + + @pytest.mark.parametrize("date_range", MALFORMED) + def test_warns_exactly_once(self, date_range) -> None: + assert len(_warnings_for(date_range)) == 1, ( + f"expected exactly one warning for date_range={date_range!r}" + ) + + @pytest.mark.parametrize("date_range", MALFORMED) + def test_emits_on_the_python_warnings_channel(self, date_range) -> None: + from slayer.core.warnings import SlayerNormalizationWarning + + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + normalize_query(query=_query(date_range)) + assert any( + issubclass(w.category, SlayerNormalizationWarning) for w in caught + ), f"no SlayerNormalizationWarning for date_range={date_range!r}" + + +class TestWellFormedDateRangeIsSilent: + + def test_two_elements_does_not_warn(self) -> None: + assert _warnings_for(["2024-01-01", "2024-12-31"]) == [] + + def test_absent_date_range_does_not_warn(self) -> None: + assert _warnings_for(None) == [] + + +@pytest.mark.asyncio +class TestNoBehaviorChange: + """The ratified silent no-op stays: a malformed range emits no filter, and + a well-formed one still does.""" + + async def _sql(self, date_range) -> str: + return await _engine_generate( + query=_query(date_range), model=_orders(), + dialect="postgres", validate=False, + ) + + @pytest.mark.parametrize("date_range", MALFORMED) + async def test_malformed_emits_no_date_filter(self, date_range) -> None: + sql = await self._sql(date_range) + assert "2024-01-01" not in sql, ( + f"malformed date_range must stay a no-op, got:\n{sql}" + ) + + async def test_well_formed_still_filters(self) -> None: + sql = await self._sql(["2024-01-01", "2024-12-31"]) + assert "2024-01-01" in sql and "2024-12-31" in sql + + async def test_absent_matches_empty_emission(self) -> None: + assert await self._sql(None) == await self._sql([]) diff --git a/tests/test_dev1745_derived_expansion.py b/tests/test_dev1745_derived_expansion.py new file mode 100644 index 00000000..fcdb1f8d --- /dev/null +++ b/tests/test_dev1745_derived_expansion.py @@ -0,0 +1,262 @@ +"""DEV-1745 (W8) — derived-column expansion must work when the whole +``Column.sql`` is a single bare reference to another derived column. + +``_process_column_node_sync`` finishes by calling +``col.replace(exp.Paren(this=expanded_ast))``. When the fragment is exactly one +column reference, that ``exp.Column`` IS the root of the parsed tree and has no +parent, so sqlglot's ``replace`` is a no-op: the correctly-expanded inner SQL is +computed and then discarded, and ``expand_derived_refs_sync`` returns the +original text. + +The emitted SQL then references a SLayer-derived column as if it were a +physical one (``customers__regions.pop_x2``), which no database can bind. Adding +any surrounding expression — even parentheses — makes it work, which is what +makes the defect easy to miss. + +Not limited to cross-model: a same-model derived alias fails identically. +""" + +from __future__ import annotations + +import pytest + +from slayer.core.enums import DataType +from slayer.core.models import Column, ModelJoin, SlayerModel +from slayer.core.query import SlayerQuery +from slayer.engine.column_expansion import expand_derived_refs_sync + +from tests._engine_helpers import _engine_generate + + +# --------------------------------------------------------------------------- +# Fixtures — orders -> customers -> regions, with derived columns at each hop +# --------------------------------------------------------------------------- + + +def _regions() -> SlayerModel: + return SlayerModel( + name="regions", data_source="test", sql_table="regions", + columns=[ + Column(name="id", type=DataType.INT, primary_key=True), + Column(name="status", type=DataType.TEXT), + Column(name="population", type=DataType.DOUBLE), + # derived ON regions, referencing a regions column BARE + Column(name="pop_x2", sql="population * 2", type=DataType.DOUBLE), + Column(name="is_live", sql="status = 'live'", type=DataType.BOOLEAN), + ], + ) + + +def _customers() -> SlayerModel: + return SlayerModel( + name="customers", data_source="test", sql_table="customers", + columns=[ + Column(name="id", type=DataType.INT, primary_key=True), + Column(name="region_id", type=DataType.INT), + ], + joins=[ModelJoin(target_model="regions", join_pairs=[["region_id", "id"]])], + ) + + +def _orders() -> SlayerModel: + return SlayerModel( + name="orders", data_source="test", sql_table="orders", + columns=[ + Column(name="id", type=DataType.INT, primary_key=True), + Column(name="customer_id", type=DataType.INT), + Column(name="amount", type=DataType.DOUBLE), + # 'status' ALSO exists on regions — same-name collision shape + Column(name="status", type=DataType.TEXT), + Column(name="doubled", sql="amount * 2", type=DataType.DOUBLE), + # --- the defect shapes: sql is ONE bare derived reference --- + Column(name="alias_local", sql="doubled", type=DataType.DOUBLE), + Column(name="deep_pop", sql="customers__regions.pop_x2", + type=DataType.DOUBLE), + Column(name="deep_live", sql="customers__regions.is_live", + type=DataType.BOOLEAN), + # --- controls: same reference inside a larger expression --- + Column(name="deep_pop_compound", sql="customers__regions.pop_x2 * 1", + type=DataType.DOUBLE), + Column(name="deep_pop_paren", sql="(customers__regions.pop_x2)", + type=DataType.DOUBLE), + ], + joins=[ModelJoin( + target_model="customers", join_pairs=[["customer_id", "id"]], + )], + ) + + +_MODELS = {"orders": _orders(), "customers": _customers(), "regions": _regions()} + + +def _resolve(name: str): + return _MODELS.get(name) + + +def _expand(sql: str) -> str: + out = expand_derived_refs_sync( + sql=sql, model=_orders(), alias_path="orders", + resolve_model=_resolve, dialect="postgres", is_root=True, + ) + assert out is not None, f"expansion returned None for {sql!r}" + return out + + +# --------------------------------------------------------------------------- +# Unit level — the expander itself +# --------------------------------------------------------------------------- + + +class TestBareDerivedReferenceExpands: + """The bare-reference-is-the-whole-fragment cases that silently no-op.""" + + def test_cross_model_two_hop_bare_reference_expands(self) -> None: + # regions.pop_x2 is DERIVED ("population * 2") — it is not a real + # column of the regions table, so leaving the reference intact emits + # SQL no database can bind. + out = _expand("customers__regions.pop_x2") + assert "pop_x2" not in out, ( + f"derived column name leaked into emitted SQL: {out!r}" + ) + assert "customers__regions.population" in out + assert "* 2" in out + + def test_same_model_bare_derived_alias_expands(self) -> None: + # Not a cross-model problem: a local derived alias fails identically. + out = _expand("doubled") + assert "doubled" not in out, ( + f"derived column name leaked into emitted SQL: {out!r}" + ) + assert "orders.amount" in out + assert "* 2" in out + + def test_boolean_derived_of_derived_expands(self) -> None: + out = _expand("customers__regions.is_live") + assert "is_live" not in out, ( + f"derived column name leaked into emitted SQL: {out!r}" + ) + assert "customers__regions.status" in out + + +class TestCompoundControlsStillWork: + """These already work today and must not regress — they are why the + defect is easy to miss.""" + + def test_compound_expression_expands(self) -> None: + out = _expand("customers__regions.pop_x2 * 1") + assert "pop_x2" not in out + assert "customers__regions.population" in out + + def test_parenthesised_reference_expands(self) -> None: + out = _expand("(customers__regions.pop_x2)") + assert "pop_x2" not in out + assert "customers__regions.population" in out + + def test_physical_column_reference_is_only_qualified(self) -> None: + # A physical (non-derived) column is qualified, never inlined. + out = _expand("amount") + assert out.strip() in {"orders.amount", '"orders".amount'} + + +# --------------------------------------------------------------------------- +# End-to-end emission +# --------------------------------------------------------------------------- + + +@pytest.mark.asyncio +class TestDerivedOfDerivedEmission: + + async def _sql(self, query: SlayerQuery) -> str: + return await _engine_generate( + query=query, model=_orders(), dialect="postgres", validate=False, + extra_models=[_customers(), _regions()], + ) + + async def test_dimension_on_derived_of_derived(self) -> None: + sql = await self._sql(SlayerQuery( + source_model="orders", + dimensions=[{"formula": "deep_pop", "name": "deep_pop"}], + measures=[{"formula": "amount:sum", "name": "m0"}], + )) + assert "pop_x2" not in sql, f"dangling derived reference:\n{sql}" + assert "customers__regions.population" in sql + # the join it crosses must still be present + assert "JOIN regions" in sql + + async def test_measure_over_derived_of_derived(self) -> None: + sql = await self._sql(SlayerQuery( + source_model="orders", + dimensions=[{"formula": "status", "name": "status"}], + measures=[{"formula": "deep_pop:sum", "name": "m0"}], + )) + assert "pop_x2" not in sql, f"dangling derived reference:\n{sql}" + assert "customers__regions.population" in sql + + async def test_filter_on_derived_of_derived(self) -> None: + sql = await self._sql(SlayerQuery( + source_model="orders", + dimensions=[{"formula": "status", "name": "status"}], + measures=[{"formula": "amount:sum", "name": "m0"}], + filters=["deep_pop > 100"], + )) + assert "pop_x2" not in sql, f"dangling derived reference:\n{sql}" + assert "customers__regions.population" in sql + + async def test_local_derived_alias_dimension(self) -> None: + sql = await self._sql(SlayerQuery( + source_model="orders", + dimensions=[{"formula": "alias_local", "name": "alias_local"}], + measures=[{"formula": "amount:sum", "name": "m0"}], + )) + # 'doubled' is derived on orders; it must inline, not be referenced. + # Today this emits a BARE, unqualified `doubled` — invalid on any DB. + assert "doubled" not in sql, f"dangling derived reference:\n{sql}" + assert "* 2" in sql, f"derived alias did not inline:\n{sql}" + + async def test_same_named_column_resolves_to_owning_model(self) -> None: + """`status` exists on BOTH orders and regions. The derived + `deep_live` (regions.is_live -> "status = 'live'") must resolve + against REGIONS, not the root model.""" + sql = await self._sql(SlayerQuery( + source_model="orders", + dimensions=[{"formula": "deep_live", "name": "deep_live"}], + measures=[{"formula": "amount:sum", "name": "m0"}], + )) + assert "customers__regions.status" in sql, ( + f"derived sql resolved against the wrong model:\n{sql}" + ) + + +# --------------------------------------------------------------------------- +# Execution — a dangling derived reference is not merely ugly, it does not run +# --------------------------------------------------------------------------- + + +@pytest.mark.asyncio +async def test_derived_of_derived_executes_on_duckdb() -> None: + import duckdb + + sql = await _engine_generate( + query=SlayerQuery( + source_model="orders", + dimensions=[{"formula": "deep_pop", "name": "deep_pop"}], + measures=[{"formula": "amount:sum", "name": "m0"}], + ), + model=_orders(), dialect="duckdb", validate=False, + extra_models=[_customers(), _regions()], + ) + con = duckdb.connect() + con.execute( + "CREATE TABLE orders(id INT, customer_id INT, amount DOUBLE, status VARCHAR)" + ) + con.execute("CREATE TABLE customers(id INT, region_id INT)") + con.execute( + "CREATE TABLE regions(id INT, status VARCHAR, population DOUBLE)" + ) + con.execute("INSERT INTO orders VALUES (1, 1, 10.0, 'ok')") + con.execute("INSERT INTO customers VALUES (1, 1)") + con.execute("INSERT INTO regions VALUES (1, 'live', 50.0)") + + rows = con.execute(sql).fetchall() + # regions.population = 50 -> pop_x2 = 100 + assert rows == [(100.0, 10.0)], f"unexpected rows {rows!r} for SQL:\n{sql}" diff --git a/tests/test_dev1745_fragment_joins.py b/tests/test_dev1745_fragment_joins.py new file mode 100644 index 00000000..d9a38d81 --- /dev/null +++ b/tests/test_dev1745_fragment_joins.py @@ -0,0 +1,222 @@ +"""DEV-1745 (W2) — custom-aggregation template fragments must register the +joins they cross, in EVERY scope that renders them. + +The Mode-A custom-aggregation template mechanism (``{value}`` / ``{param}`` +substitution) stays as a feature; only its join discovery moves onto the single +Mode-A door, so registration becomes a side effect of resolution (P-A). + +Today the fragment scan exists ONLY on the host render path +(``_resolve_agg_inputs_via_scope`` -> ``_resolve_fragment_kwargs``). The +cross-model ``_cm_`` CTE builds its FROM purely from ``cte_scope.join_paths`` +and registers only source / positional args / typed column kwargs / +``column_filter_key`` — never string fragments nor the model-default +``AggregationParam.sql`` values. A crossing fragment therefore renders a +reference to a table that is not in the CTE's FROM. + +The shifted (``time_shift``) CTE has the same gap but is unreachable today — +``time_shift`` combined with a cross-model aggregate raises first. That half is +tracked separately. +""" + +from __future__ import annotations + +import pytest + +from slayer.core.enums import DataType +from slayer.core.models import ( + Aggregation, + AggregationParam, + Column, + ModelJoin, + SlayerModel, +) +from slayer.core.query import SlayerQuery + +from tests._engine_helpers import _engine_generate + + +# --------------------------------------------------------------------------- +# Fixtures — the aggregation is declared on the JOINED model so the planner +# roots a _cm_ CTE at `customers`, and its default param crosses one hop +# further to `regions`. +# --------------------------------------------------------------------------- + + +def _regions() -> SlayerModel: + return SlayerModel( + name="regions", data_source="test", sql_table="regions", + columns=[ + Column(name="id", type=DataType.INT, primary_key=True), + Column(name="weight", type=DataType.DOUBLE), + ], + ) + + +def _customers() -> SlayerModel: + return SlayerModel( + name="customers", data_source="test", sql_table="customers", + columns=[ + Column(name="id", type=DataType.INT, primary_key=True), + Column(name="region_id", type=DataType.INT), + Column(name="spend", type=DataType.DOUBLE), + ], + joins=[ModelJoin(target_model="regions", join_pairs=[["region_id", "id"]])], + aggregations=[ + Aggregation( + name="wscaled_sum", formula="SUM({value} * {w})", + params=[AggregationParam(name="w", sql="regions.weight")], + ), + ], + ) + + +def _orders() -> SlayerModel: + return SlayerModel( + name="orders", data_source="test", sql_table="orders", + columns=[ + Column(name="id", type=DataType.INT, primary_key=True), + Column(name="customer_id", type=DataType.INT), + Column(name="amount", type=DataType.DOUBLE), + Column(name="status", type=DataType.TEXT), + ], + joins=[ModelJoin( + target_model="customers", join_pairs=[["customer_id", "id"]], + )], + ) + + +def _orders_local_agg() -> SlayerModel: + """Host-rooted variant: the aggregation is declared on the ROOT model with + a crossing default param. This path already scans fragments today and must + not regress.""" + return SlayerModel( + name="orders", data_source="test", sql_table="orders", + columns=[ + Column(name="id", type=DataType.INT, primary_key=True), + Column(name="customer_id", type=DataType.INT), + Column(name="amount", type=DataType.DOUBLE), + Column(name="status", type=DataType.TEXT), + ], + joins=[ModelJoin( + target_model="customers", join_pairs=[["customer_id", "id"]], + )], + aggregations=[ + Aggregation( + name="wscaled_sum", formula="SUM({value} * {w})", + params=[AggregationParam( + name="w", sql="customers__regions.weight", + )], + ), + ], + ) + + +async def _sql(query: SlayerQuery, *, model: SlayerModel, dialect="postgres") -> str: + return await _engine_generate( + query=query, model=model, dialect=dialect, validate=False, + extra_models=[_customers(), _regions()], + ) + + +# --------------------------------------------------------------------------- + + +@pytest.mark.asyncio +class TestCrossModelFragmentJoins: + """The `_cm_` CTE gap — a crossing template fragment must pull its join + into the CTE's own FROM.""" + + async def test_cm_cte_joins_the_fragment_target(self) -> None: + sql = await _sql( + SlayerQuery( + source_model="orders", + dimensions=[{"formula": "status", "name": "status"}], + measures=[{ + "formula": "customers.spend:wscaled_sum", "name": "m0", + }], + ), + model=_orders(), + ) + # the fragment renders regions.weight ... + assert "regions.weight" in sql, sql + # ... so regions MUST be joined in the same scope + assert "JOIN regions" in sql, ( + f"fragment's crossed join missing from the CTE FROM:\n{sql}" + ) + + async def test_cm_cte_with_sibling_local_measure(self) -> None: + sql = await _sql( + SlayerQuery( + source_model="orders", + dimensions=[{"formula": "status", "name": "status"}], + measures=[ + {"formula": "customers.spend:wscaled_sum", "name": "m0"}, + {"formula": "amount:sum", "name": "m1"}, + ], + ), + model=_orders(), + ) + assert "regions.weight" in sql, sql + assert "JOIN regions" in sql, ( + f"fragment's crossed join missing from the CTE FROM:\n{sql}" + ) + + +@pytest.mark.asyncio +class TestHostPathFragmentJoinsStillWork: + """Parity guard: the host path already registers fragment joins.""" + + async def test_host_rooted_fragment_join_registered(self) -> None: + sql = await _sql( + SlayerQuery( + source_model="orders", + dimensions=[{"formula": "status", "name": "status"}], + measures=[{"formula": "amount:wscaled_sum", "name": "m0"}], + ), + model=_orders_local_agg(), + ) + assert "customers__regions.weight" in sql, sql + assert "JOIN regions" in sql, sql + + async def test_query_time_string_kwarg_override(self) -> None: + sql = await _sql( + SlayerQuery( + source_model="orders", + dimensions=[{"formula": "status", "name": "status"}], + measures=[{ + "formula": "amount:wscaled_sum(w='customers__regions.weight')", + "name": "m0", + }], + ), + model=_orders_local_agg(), + ) + assert "customers__regions.weight" in sql, sql + assert "JOIN regions" in sql, sql + + +@pytest.mark.asyncio +async def test_cross_model_fragment_executes_on_duckdb() -> None: + """A missing join is not a cosmetic difference — the SQL does not bind.""" + import duckdb + + sql = await _sql( + SlayerQuery( + source_model="orders", + dimensions=[{"formula": "status", "name": "status"}], + measures=[{"formula": "customers.spend:wscaled_sum", "name": "m0"}], + ), + model=_orders(), dialect="duckdb", + ) + con = duckdb.connect() + con.execute( + "CREATE TABLE orders(id INT, customer_id INT, amount DOUBLE, status VARCHAR)" + ) + con.execute("CREATE TABLE customers(id INT, region_id INT, spend DOUBLE)") + con.execute("CREATE TABLE regions(id INT, weight DOUBLE)") + con.execute("INSERT INTO orders VALUES (1, 1, 10.0, 'ok')") + con.execute("INSERT INTO customers VALUES (1, 1, 7.0)") + con.execute("INSERT INTO regions VALUES (1, 3.0)") + + rows = con.execute(sql).fetchall() + # SUM(customers.spend * regions.weight) = 7 * 3 = 21 + assert rows == [("ok", 21.0)], f"unexpected rows {rows!r} for SQL:\n{sql}" diff --git a/tests/test_dev1745_golden_sql.py b/tests/test_dev1745_golden_sql.py new file mode 100644 index 00000000..b94166f5 --- /dev/null +++ b/tests/test_dev1745_golden_sql.py @@ -0,0 +1,498 @@ +"""DEV-1745 — golden SQL baseline for the "SQL-identical refactor" claim. + +Existing suites passing unchanged is supporting evidence, not proof: they cover +the shapes someone already thought to test. This harness pins the emitted SQL +for a matrix that deliberately targets the Mode-A surfaces this PR rewires — +every migrated call site, each scope kind, and the dialects where quoting and +serialization differ — and FAILS on any unlisted delta. + +Workflow +-------- +The golden file is the BEFORE state, generated against the pre-refactor code. +When an implementation commit changes emitted SQL, this test fails with a diff. +That diff is the per-test approval artifact required by the DEV-1742 protocol. + +Blessing a change is deliberately a four-step loop: + +1. The suite fails with a diff for ``::``. +2. Review it, then add that exact key to :data:`ALLOWED_DELTAS` with the reason. +3. ``SLAYER_UPDATE_GOLDEN=1 poetry run pytest tests/test_dev1745_golden_sql.py`` + — which rewrites **only** the listed keys. +4. Delete the now-stale manifest entries. + +Step 3 is what makes ``SLAYER_UPDATE_GOLDEN`` safe: it can no longer regenerate +all 70 entries at once, so a change nobody listed cannot ride along on someone +else's approval. Step 4 is enforced by +:func:`test_allowed_deltas_are_not_stale` — an entry that has already been +blessed would otherwise sit there silently pre-authorising the *next*, +unintended change to the same key. Every committed state therefore has an empty +manifest. + +Never regenerate to make the suite green. The whole point is that a change you +did not intend cannot pass silently. + +Some baseline entries record currently-BROKEN SQL (a derived-of-derived column +emitted as a dangling reference; a `_cm_` CTE missing a fragment's join). Those +are expected to change — that is the fix landing, and the diff documents it. + +Entries that raise record a structured ``{"error": ..., "message": ...}`` with +the exception's COMPLETE message, compared exactly. A bare type name would let +any new leak in the same case pass unnoticed; the full message pins the unbound +reference, the scope it leaked in, and the SQL that accompanied it, so those +entries carry the same evidentiary weight as the ones that emit SQL. +""" + +from __future__ import annotations + +import asyncio +import json +import os +from pathlib import Path + +import pytest + +from slayer.core.enums import DataType +from slayer.core.models import ( + Aggregation, + AggregationParam, + Column, + ModelJoin, + SlayerModel, +) +from slayer.core.query import SlayerQuery + +from tests._engine_helpers import _engine_generate + + +GOLDEN_PATH = Path(__file__).parent / "golden" / "dev1745_sql_baseline.json" + +DIALECTS = ["postgres", "sqlite", "duckdb", "tsql", "bigquery"] + +# ``::`` -> why this entry is allowed to change right now. +# +# This is a PENDING list, not a log: ``SLAYER_UPDATE_GOLDEN=1`` rewrites only +# these keys, and once a key has been regenerated its entry is stale and must be +# deleted (see the module docstring and ``test_allowed_deltas_are_not_stale``). +# A committed state always has this empty. +ALLOWED_DELTAS: dict[str, str] = {} + + +# --------------------------------------------------------------------------- # +# Model graph — every Mode-A surface is represented at least once. +# --------------------------------------------------------------------------- # +def _regions() -> SlayerModel: + return SlayerModel( + name="regions", data_source="test", sql_table="regions", + columns=[ + Column(name="id", type=DataType.INT, primary_key=True), + Column(name="status", type=DataType.TEXT), + Column(name="population", type=DataType.DOUBLE), + Column(name="weight", type=DataType.DOUBLE), + # Column.sql — derived on a two-hop target + Column(name="pop_x2", sql="population * 2", type=DataType.DOUBLE), + ], + ) + + +def _customers() -> SlayerModel: + return SlayerModel( + name="customers", data_source="test", sql_table="customers", + columns=[ + Column(name="id", type=DataType.INT, primary_key=True), + Column(name="region_id", type=DataType.INT), + Column(name="spend", type=DataType.DOUBLE), + Column(name="tier", type=DataType.TEXT), + ], + joins=[ModelJoin(target_model="regions", join_pairs=[["region_id", "id"]])], + aggregations=[ + Aggregation( + name="wscaled_sum", formula="SUM({value} * {w})", + params=[AggregationParam(name="w", sql="regions.weight")], + ), + ], + ) + + +def _orders() -> SlayerModel: + return SlayerModel( + name="orders", data_source="test", sql_table="orders", + # SlayerModel.filters — a Mode-A always-applied WHERE + filters=["amount >= 0"], + columns=[ + Column(name="id", type=DataType.INT, primary_key=True), + Column(name="customer_id", type=DataType.INT), + Column(name="amount", type=DataType.DOUBLE), + Column(name="created_at", type=DataType.TIMESTAMP), + # 'status' also exists on regions — same-name collision + Column(name="status", type=DataType.TEXT), + # a column named like a statement keyword + Column(name="select", type=DataType.TEXT), + Column(name="doubled", sql="amount * 2", type=DataType.DOUBLE), + # Column.filter — Mode-A predicate crossing a join + Column(name="eu_amount", sql="amount", + filter="customers.tier = 'eu'", type=DataType.DOUBLE), + # raw ref that inlines to a constant (dual-scan contract) + Column(name="flag_const", sql="1", type=DataType.INT), + # quoted dotted identifier + Column(name="quoted_cross", sql='"customers"."spend"', + type=DataType.DOUBLE), + # derived-of-derived, two hops (currently BROKEN — see module doc) + Column(name="deep_pop", sql="customers__regions.pop_x2", + type=DataType.DOUBLE), + Column(name="multi_model", + sql="customers.spend + customers__regions.population", + type=DataType.DOUBLE), + ], + joins=[ModelJoin( + target_model="customers", join_pairs=[["customer_id", "id"]], + )], + ) + + +def _q(**kw) -> SlayerQuery: + kw.setdefault("source_model", "orders") + return SlayerQuery(**kw) + + +# --------------------------------------------------------------------------- # +# The matrix. Keys are stable ids — renaming one is a golden change. +# --------------------------------------------------------------------------- # +def _cases() -> dict: + dim_status = [{"formula": "status", "name": "status"}] + return { + # --- host scope, plain Mode-A surfaces --- + "host/model_filter": _q( + dimensions=dim_status, + measures=[{"formula": "amount:sum", "name": "m"}], + ), + "host/column_sql_derived": _q( + dimensions=[{"formula": "doubled", "name": "doubled"}], + measures=[{"formula": "amount:sum", "name": "m"}], + ), + "host/column_filter_crossing": _q( + dimensions=dim_status, + measures=[{"formula": "eu_amount:sum", "name": "m"}], + ), + "host/const_expanding_ref": _q( + dimensions=dim_status, + measures=[{"formula": "amount:sum", "name": "m"}], + filters=["flag_const == 1"], + ), + "host/quoted_dotted_identifier": _q( + dimensions=[{"formula": "quoted_cross", "name": "quoted_cross"}], + measures=[{"formula": "amount:sum", "name": "m"}], + ), + "host/statement_keyword_column": _q( + dimensions=[{"formula": "select", "name": "select"}], + measures=[{"formula": "amount:sum", "name": "m"}], + ), + "host/same_named_column_and_model": _q( + dimensions=dim_status, + measures=[{"formula": "amount:sum", "name": "m"}], + filters=["status == 'x'"], + ), + # --- derived expansion --- + "expand/derived_of_derived": _q( + dimensions=[{"formula": "deep_pop", "name": "deep_pop"}], + measures=[{"formula": "amount:sum", "name": "m"}], + ), + "expand/multi_model_derived": _q( + dimensions=[{"formula": "multi_model", "name": "multi_model"}], + measures=[{"formula": "amount:sum", "name": "m"}], + ), + # --- cross-model _cm_ scope --- + "cm/joined_measure": _q( + dimensions=dim_status, + measures=[{"formula": "customers.spend:sum"}], + ), + "cm/fragment_default_crossing": _q( + dimensions=dim_status, + measures=[{"formula": "customers.spend:wscaled_sum"}], + ), + "cm/outer_where_wrapper": _q( + dimensions=dim_status, + measures=[{"formula": "eu_amount:sum", "name": "eu"}], + filters=["eu_amount:sum > 100"], + ), + # --- windowed _src scope --- + "windowed/src_scope": _q( + time_dimensions=[{ + "dimension": "created_at", "granularity": "month", + "date_range": ["2024-01-01", "2024-12-31"], + }], + measures=[{"formula": "eu_amount:sum", "name": "m"}], + ), + "windowed/date_range_filter": _q( + time_dimensions=[{ + "dimension": "created_at", "granularity": "month", + "date_range": ["2024-01-01", "2024-12-31"], + }], + measures=[{"formula": "amount:sum", "name": "m"}], + ), + } + + +async def _generate_one(query: SlayerQuery, dialect: str): + """Emitted SQL as a string, or a structured record of the raised error. + + The error record keeps the COMPLETE message, not just the type: a type name + alone lets any *new* failure in the same case pass, which is exactly the + blind spot this harness exists to close. + """ + try: + return await _engine_generate( + query=query, model=_orders(), dialect=dialect, validate=False, + extra_models=[_customers(), _regions()], + ) + except Exception as exc: # noqa: BLE001 — the exception itself is contract + return {"error": type(exc).__name__, "message": str(exc)} + + +def _render(value) -> str: + """Human-readable form of a baseline value, for assertion messages.""" + if isinstance(value, dict): + return f"RAISED {value.get('error')}: {value.get('message')}" + return str(value) + + +def _build_baseline() -> dict: + # conftest's autouse ``_enable_scope_validation`` is FUNCTION-scoped, so it + # is not in effect while a module-scoped fixture runs. Set it explicitly so + # the baseline is generated under exactly the same validation regime the + # assertions run under — otherwise a shape that trips ScopeLeakError during + # a test would have been recorded as valid SQL, and every run would "fail" + # with a spurious diff. + previous = os.environ.get("SLAYER_VALIDATE_SCOPES") + os.environ["SLAYER_VALIDATE_SCOPES"] = "1" + + async def _run() -> dict: + out: dict = {} + for case_id, query in _cases().items(): + for dialect in DIALECTS: + out[f"{case_id}::{dialect}"] = await _generate_one(query, dialect) + return out + + try: + return asyncio.run(_run()) + finally: + if previous is None: + os.environ.pop("SLAYER_VALIDATE_SCOPES", None) + else: + os.environ["SLAYER_VALIDATE_SCOPES"] = previous + + +def _expected_keys() -> set: + return {f"{c}::{d}" for c in _cases() for d in DIALECTS} + + +def _merge_regenerated( + *, + existing: dict | None, + fresh: dict, + allowed: dict, + expected: set, +) -> dict: + """Fold ``fresh`` into ``existing``, honouring the allowed-delta manifest. + + Only keys named in ``allowed`` may overwrite a value already in the golden + file — that restriction is the whole mechanism, so it is unit-tested + directly rather than only through the module fixture. Keys for newly added + cases are folded in unconditionally (there is no prior approval to + protect), and keys for cases that no longer exist are pruned. + """ + if existing is None: + return dict(fresh) + + unknown = sorted(set(allowed) - expected) + if unknown: + raise AssertionError( + f"ALLOWED_DELTAS names keys that are not in the matrix: {unknown}" + ) + + merged = {k: v for k, v in existing.items() if k in expected} + for key, value in fresh.items(): + if key not in merged or key in allowed: + merged[key] = value + return merged + + +def _regenerate(existing: dict | None) -> dict: + return _merge_regenerated( + existing=existing, + fresh=_build_baseline(), + allowed=ALLOWED_DELTAS, + expected=_expected_keys(), + ) + + +@pytest.fixture(scope="module") +def baseline() -> dict: + if os.environ.get("SLAYER_UPDATE_GOLDEN"): + existing = ( + json.loads(GOLDEN_PATH.read_text()) if GOLDEN_PATH.exists() else None + ) + GOLDEN_PATH.parent.mkdir(parents=True, exist_ok=True) + GOLDEN_PATH.write_text( + json.dumps(_regenerate(existing), indent=2, sort_keys=True) + "\n" + ) + if not GOLDEN_PATH.exists(): + pytest.fail( + f"golden baseline missing at {GOLDEN_PATH}; generate it with " + f"SLAYER_UPDATE_GOLDEN=1" + ) + return json.loads(GOLDEN_PATH.read_text()) + + +@pytest.mark.parametrize("case_id", sorted(_cases())) +@pytest.mark.parametrize("dialect", DIALECTS) +def test_emitted_sql_matches_golden(case_id: str, dialect: str, baseline) -> None: + key = f"{case_id}::{dialect}" + assert key in baseline, ( + f"{key} is not in the golden baseline — a new case must be added " + f"deliberately (SLAYER_UPDATE_GOLDEN=1) and reviewed" + ) + actual = asyncio.run(_generate_one(_cases()[case_id], dialect)) + assert actual == baseline[key], ( + f"emitted SQL changed for {key}.\n" + f"--- golden ---\n{_render(baseline[key])}\n" + f"--- actual ---\n{_render(actual)}\n" + f"If this change is intended, get it approved per the DEV-1742 " + f"per-test protocol, add {key!r} to ALLOWED_DELTAS with the reason, " + f"regenerate with SLAYER_UPDATE_GOLDEN=1, then delete the entry." + ) + + +def test_baseline_covers_every_case_and_dialect(baseline) -> None: + missing = _expected_keys() - set(baseline) + assert not missing, f"golden baseline is missing entries: {sorted(missing)}" + + +def test_baseline_has_no_orphan_entries(baseline) -> None: + """A case removed from the matrix must not leave a golden entry behind — + it would be dead weight nothing asserts.""" + orphans = set(baseline) - _expected_keys() + assert not orphans, ( + f"golden baseline has entries for cases that no longer exist: " + f"{sorted(orphans)}; regenerate to prune them" + ) + + +def test_allowed_deltas_name_real_keys() -> None: + unknown = sorted(set(ALLOWED_DELTAS) - _expected_keys()) + assert not unknown, ( + f"ALLOWED_DELTAS names keys that are not in the matrix: {unknown}" + ) + + +def test_allowed_deltas_carry_a_reason() -> None: + blank = sorted(k for k, v in ALLOWED_DELTAS.items() if not str(v).strip()) + assert not blank, ( + f"every allowed delta must say WHY the SQL is permitted to change: " + f"{blank}" + ) + + +class TestRegenerationGate: + """D11 / deferred item 10 — ``SLAYER_UPDATE_GOLDEN`` must not be able to + bless all 70 entries at once. Only listed keys may overwrite an approved + value.""" + + EXPECTED = {"a::postgres", "b::postgres", "c::postgres"} + + def test_unlisted_delta_is_not_written(self) -> None: + merged = _merge_regenerated( + existing={"a::postgres": "OLD", "b::postgres": "OLD"}, + fresh={"a::postgres": "NEW", "b::postgres": "NEW"}, + allowed={}, + expected=self.EXPECTED, + ) + assert merged == {"a::postgres": "OLD", "b::postgres": "OLD"}, ( + "regeneration overwrote a golden value nobody approved" + ) + + def test_listed_delta_is_written(self) -> None: + merged = _merge_regenerated( + existing={"a::postgres": "OLD", "b::postgres": "OLD"}, + fresh={"a::postgres": "NEW", "b::postgres": "NEW"}, + allowed={"a::postgres": "because"}, + expected=self.EXPECTED, + ) + assert merged["a::postgres"] == "NEW" + assert merged["b::postgres"] == "OLD", ( + "an unlisted key rode along on a listed key's approval" + ) + + def test_new_case_is_added_without_approval(self) -> None: + merged = _merge_regenerated( + existing={"a::postgres": "OLD"}, + fresh={"a::postgres": "NEW", "c::postgres": "FRESH"}, + allowed={}, + expected=self.EXPECTED, + ) + assert merged["c::postgres"] == "FRESH", ( + "a brand-new case has no prior approval to protect" + ) + assert merged["a::postgres"] == "OLD" + + def test_removed_case_is_pruned(self) -> None: + merged = _merge_regenerated( + existing={"a::postgres": "OLD", "gone::postgres": "OLD"}, + fresh={"a::postgres": "OLD"}, + allowed={}, + expected=self.EXPECTED, + ) + assert "gone::postgres" not in merged + + def test_manifest_key_outside_the_matrix_raises(self) -> None: + with pytest.raises(AssertionError, match="not in the matrix"): + _merge_regenerated( + existing={"a::postgres": "OLD"}, + fresh={"a::postgres": "NEW"}, + allowed={"typo::postgres": "because"}, + expected=self.EXPECTED, + ) + + def test_first_generation_writes_everything(self) -> None: + merged = _merge_regenerated( + existing=None, + fresh={"a::postgres": "NEW"}, + allowed={}, + expected=self.EXPECTED, + ) + assert merged == {"a::postgres": "NEW"} + + +def test_error_entries_record_the_full_message(baseline) -> None: + """Deferred item 11 — a bare exception TYPE lets any new failure in the + same case pass. Every error entry must carry its message.""" + errors = {k: v for k, v in baseline.items() if isinstance(v, dict)} + assert errors, "expected at least one baseline entry to record an exception" + for key, value in sorted(errors.items()): + assert value.get("error"), f"{key} has no exception type" + assert value.get("message", "").strip(), ( + f"{key} records an exception type with no message — a different " + f"failure of the same type would pass unnoticed" + ) + + +def test_allowed_deltas_are_not_stale(baseline) -> None: + """A manifest entry is only valid while its delta is still PENDING. + + Once regenerated, golden == actual and the entry has done its job. Leaving + it behind would silently pre-authorise the *next* change to the same key — + the wholesale-blessing hole D11 exists to close. So a blessed entry is a + failure until it is deleted, which is what keeps every committed state's + manifest empty. + """ + stale = [] + for key in sorted(ALLOWED_DELTAS): + case_id, _, dialect = key.partition("::") + if case_id not in _cases() or dialect not in DIALECTS: + continue + actual = asyncio.run(_generate_one(_cases()[case_id], dialect)) + if key in baseline and actual == baseline[key]: + stale.append(key) + assert not stale, ( + f"these ALLOWED_DELTAS entries have already been blessed — the golden " + f"file now matches. Delete them: {stale}" + ) diff --git a/tests/test_dev1745_mode_a_door.py b/tests/test_dev1745_mode_a_door.py new file mode 100644 index 00000000..e9c71975 --- /dev/null +++ b/tests/test_dev1745_mode_a_door.py @@ -0,0 +1,360 @@ +"""DEV-1745 (W1) — the single Mode-A entry point on ``ScopeFrame`` (P-A). + +Every fragment of free SQL enters a SELECT scope through ONE door that +prequotes, expands, registers the joins it crosses, and returns an AST. Join +discovery is a side effect of resolution, never a separate pass. + +Two public surfaces over one implementation, differing ONLY in the parse helper: + +* ``enter_predicate`` — for ``Column.filter`` and ``SlayerModel.filters``, + which are boolean predicates. Keeps the ``SELECT 1 WHERE ...`` statement- + keyword guard. +* ``enter_expression`` — for ``Column.sql``, a scalar expression. + +The surface's grammar is a STATIC property of the field, not a runtime choice: +no content-based dispatch, no "try expression then predicate" retry. Either +would re-introduce render-time re-classification (a P-D violation). + +Deliberately NOT part of the door: a separate qualification pass. +``expand_derived_refs_sync`` already qualifies, and qualifies correctly — +against the OWNING model's canonical alias. It leaves a node alone when the +alias path does not resolve, because that is an opaque CTE / subquery +reference; a blanket pass against ``root_relation`` would corrupt exactly +those. See ``TestOpaqueReferencesSurvive``. + +Removed by this work: ``include_dotted_derived`` (no caller ever passed it, and +its documented ``False`` case was unwired) and the three swallow-all +``except Exception`` lanes — including the one in ``_filter_join_paths._scan`` +that silently contributed ZERO join paths, turning an unparseable fragment into +missing joins rather than an error. +""" + +from __future__ import annotations + +import pytest +from sqlglot import exp + +from slayer.core.enums import DataType +from slayer.core.models import Column, ModelJoin, SlayerModel +from slayer.engine.source_bundle import ResolvedSourceBundle +from slayer.sql.dialects import get_dialect +from slayer.sql.naming import AliasAllocator +from slayer.sql.scope import ScopeFrame + + +# --------------------------------------------------------------------------- # +# Fixtures +# --------------------------------------------------------------------------- # +def _regions() -> SlayerModel: + return SlayerModel( + name="regions", sql_table="regions", data_source="test", + columns=[ + Column(name="id", type=DataType.INT, primary_key=True), + Column(name="name", type=DataType.TEXT), + Column(name="population", type=DataType.DOUBLE), + Column(name="pop_x2", sql="population * 2", type=DataType.DOUBLE), + ], + ) + + +def _customers() -> SlayerModel: + return SlayerModel( + name="customers", sql_table="customers", data_source="test", + columns=[ + Column(name="id", type=DataType.INT, primary_key=True), + Column(name="region_id", type=DataType.INT), + Column(name="balance", type=DataType.DOUBLE), + ], + joins=[ModelJoin(target_model="regions", join_pairs=[["region_id", "id"]])], + ) + + +def _orders() -> SlayerModel: + return SlayerModel( + name="orders", sql_table="orders", data_source="test", + columns=[ + Column(name="id", type=DataType.INT, primary_key=True), + Column(name="customer_id", type=DataType.INT), + Column(name="amount", type=DataType.DOUBLE), + Column(name="status", type=DataType.TEXT), + Column(name="doubled", sql="amount * 2", type=DataType.DOUBLE), + # a reserved-ish name that would shadow a statement keyword + Column(name="select", type=DataType.TEXT), + ], + joins=[ModelJoin( + target_model="customers", join_pairs=[["customer_id", "id"]], + )], + ) + + +def _scope(dialect: str = "postgres") -> ScopeFrame: + host = _orders() + alloc = AliasAllocator() + bundle = ResolvedSourceBundle( + source_model=host, referenced_models=[host, _customers(), _regions()], + ) + return ScopeFrame( + scope_id=alloc.next_scope_id(host.name), + root_model=host, root_relation=host.name, + bundle=bundle, dialect=get_dialect(dialect), allocator=alloc, + ) + + +def _sql_of(node: exp.Expression, dialect: str = "postgres") -> str: + return node.sql(dialect=dialect) + + +# --------------------------------------------------------------------------- # +# The door exists and has exactly two surfaces +# --------------------------------------------------------------------------- # +class TestDoorSurface: + + def test_enter_predicate_exists(self) -> None: + assert hasattr(ScopeFrame, "enter_predicate"), ( + "ScopeFrame must expose the single Mode-A predicate entry point" + ) + + def test_enter_expression_exists(self) -> None: + assert hasattr(ScopeFrame, "enter_expression"), ( + "ScopeFrame must expose the single Mode-A expression entry point" + ) + + def test_no_include_dotted_derived_flag_anywhere(self) -> None: + """The flag is deleted, not threaded through the new door.""" + import inspect + + for name in ("enter_predicate", "enter_expression"): + fn = getattr(ScopeFrame, name, None) + if fn is None: + pytest.fail(f"ScopeFrame.{name} missing") + params = set(inspect.signature(fn).parameters) + assert "include_dotted_derived" not in params + assert "include_dotted" not in params + + +# --------------------------------------------------------------------------- # +# Qualification comes from expansion — the door adds no second pass +# --------------------------------------------------------------------------- # +class TestQualification: + + def test_bare_root_column_is_qualified(self) -> None: + out = _sql_of(_scope().enter_predicate("status = 'x'")) + assert "orders.status" in out, out + + def test_quoted_bare_column_is_qualified(self) -> None: + """Assert an actual qualified column, not the two tokens appearing + independently somewhere in the string.""" + node = _scope().enter_predicate('"status" = \'x\'') + cols = [ + c for c in node.find_all(exp.Column) + if c.name == "status" + ] + assert cols, _sql_of(node) + assert all(c.table == "orders" for c in cols), ( + f"quoted bare column was not qualified against the root: " + f"{_sql_of(node)}" + ) + + def test_joined_derived_column_resolves_against_its_owning_model(self) -> None: + """A derived column ON customers whose sql names a bare customers + column must expand to ``customers.balance`` — never ``orders.balance``. + This is what separates "expansion qualified it, against the owning + model" from "something re-qualified it against the scope root".""" + customers = _customers() + customers.columns.append( + Column(name="balance_x2", sql="balance * 2", type=DataType.DOUBLE), + ) + host = _orders() + # a same-named column on the ROOT, so a root-anchored pass is visible + host.columns.append( + Column(name="balance", type=DataType.DOUBLE), + ) + alloc = AliasAllocator() + bundle = ResolvedSourceBundle( + source_model=host, referenced_models=[host, customers, _regions()], + ) + scope = ScopeFrame( + scope_id=alloc.next_scope_id(host.name), + root_model=host, root_relation=host.name, + bundle=bundle, dialect=get_dialect("postgres"), allocator=alloc, + ) + out = _sql_of(scope.enter_expression("customers.balance_x2")) + assert "customers.balance" in out, out + assert "orders.balance" not in out, ( + f"derived sql was re-qualified against the scope root instead of " + f"its owning model: {out}" + ) + + def test_local_derived_column_is_inlined_and_qualified(self) -> None: + out = _sql_of(_scope().enter_expression("doubled")) + assert "doubled" not in out, f"derived name leaked: {out}" + assert "orders.amount" in out, out + + def test_already_qualified_join_path_is_left_alone(self) -> None: + out = _sql_of(_scope().enter_predicate("customers.balance > 1")) + assert "customers.balance" in out, out + + +class TestOpaqueReferencesSurvive: + """A blanket root-relation qualification pass would corrupt these. + ``_walk_path_to_target_sync`` deliberately leaves an unresolvable alias + untouched — it is an opaque CTE / subquery reference, not an error.""" + + def test_unresolvable_alias_is_not_requalified_to_root(self) -> None: + out = _sql_of(_scope().enter_predicate("some_cte.flag = 1")) + assert "orders.some_cte" not in out, ( + f"opaque reference was re-qualified against the scope root: {out}" + ) + assert "some_cte.flag" in out, out + + @pytest.mark.xfail( + strict=True, + reason=( + "Known defect, tracked separately and deliberately NOT fixed in " + "this PR: qualification in _process_column_node_sync happens " + "BEFORE the root_scope_ids gate, so it is not scope-aware. Only " + "derived INLINING is gated. A column inside a subquery with its " + "own FROM is therefore qualified against the OUTER root: " + "'amount IN (SELECT amount FROM other_tbl)' becomes " + "'orders.amount IN (SELECT orders.amount FROM other_tbl)', " + "silently rebinding the inner reference to the wrong table." + ), + ) + def test_subquery_column_is_not_qualified_against_outer_root(self) -> None: + out = _sql_of( + _scope().enter_predicate("amount IN (SELECT amount FROM other_tbl)") + ) + inner = out.split("SELECT", 1)[1] + assert "orders.amount" not in inner, ( + f"a column inside a subquery was bound to the outer root: {out}" + ) + + +# --------------------------------------------------------------------------- # +# Join registration is a side effect of entering the scope (P-A) +# --------------------------------------------------------------------------- # +class TestJoinRegistration: + + def test_predicate_crossing_one_hop_registers_it(self) -> None: + scope = _scope() + scope.enter_predicate("customers.balance > 1") + assert ("customers",) in scope.join_paths.as_list() + + def test_predicate_crossing_two_hops_registers_every_prefix(self) -> None: + scope = _scope() + scope.enter_predicate("customers__regions.population > 1") + paths = scope.join_paths.as_list() + assert ("customers",) in paths + assert ("customers", "regions") in paths + + def test_expression_crossing_registers_too(self) -> None: + scope = _scope() + scope.enter_expression("customers__regions.population") + paths = scope.join_paths.as_list() + # every PREFIX is required, not just the deepest hop — the FROM builder + # needs the intermediate join to reach the last one + assert ("customers",) in paths, paths + assert ("customers", "regions") in paths, paths + + def test_local_predicate_registers_nothing(self) -> None: + scope = _scope() + scope.enter_predicate("amount > 1") + assert scope.join_paths.as_list() == [] + + def test_dual_scan_keeps_paths_that_expansion_removes(self) -> None: + """The pre-expansion scan is load-bearing (the DEV-1494 contract). + + ``customers.flag_const`` is a DERIVED column on customers whose sql is + the constant ``1``, so expansion rewrites the reference to ``1`` and the + join disappears from the expanded AST entirely. The join is only + discoverable by scanning BEFORE expansion. + + The crossing ref must be the ONLY one in the predicate — with any other + surviving ``customers.*`` reference, an implementation that scans only + after expansion would still find the path and this test would pass + while proving nothing. + """ + customers = _customers() + customers.columns.append( + Column(name="flag_const", sql="1", type=DataType.INT), + ) + host = _orders() + alloc = AliasAllocator() + bundle = ResolvedSourceBundle( + source_model=host, referenced_models=[host, customers, _regions()], + ) + scope = ScopeFrame( + scope_id=alloc.next_scope_id(host.name), + root_model=host, root_relation=host.name, + bundle=bundle, dialect=get_dialect("postgres"), allocator=alloc, + ) + node = scope.enter_predicate("customers.flag_const = 1") + # the reference really does vanish from the expanded AST ... + assert "customers" not in _sql_of(node), _sql_of(node) + # ... and the join is still registered + assert ("customers",) in scope.join_paths.as_list(), ( + "the pre-expansion scan did not run: a crossing ref that inlines " + "to a constant lost its join path" + ) + + +# --------------------------------------------------------------------------- # +# Failure is loud (D1) — no regex fallback, no raw passthrough, no silent +# zero-join-paths +# --------------------------------------------------------------------------- # +class TestParseFailureRaises: + + def test_unparseable_predicate_raises(self) -> None: + from slayer.core.errors import SlayerError + + with pytest.raises(SlayerError): + _scope().enter_predicate("this is ( not sql") + + def test_unparseable_expression_raises(self) -> None: + from slayer.core.errors import SlayerError + + with pytest.raises(SlayerError): + _scope().enter_expression("SELECT ((( FROM") + + def test_error_carries_the_original_fragment(self) -> None: + from slayer.core.errors import SlayerError + + fragment = "this is ( not sql" + with pytest.raises(SlayerError) as excinfo: + _scope().enter_predicate(fragment) + assert fragment in str(excinfo.value), ( + "the error must name the offending fragment" + ) + + def test_unparseable_never_silently_drops_joins(self) -> None: + """The old ``_filter_join_paths._scan`` swallowed the parse error and + contributed zero paths — missing joins instead of a failure.""" + from slayer.core.errors import SlayerError + + scope = _scope() + with pytest.raises(SlayerError): + scope.enter_predicate("customers.balance > ( not sql") + # and it certainly must not have quietly succeeded with no joins + assert scope.join_paths.as_list() == [] + + +# --------------------------------------------------------------------------- # +# Predicate vs expression grammar +# --------------------------------------------------------------------------- # +class TestSurfaceGrammar: + + def test_predicate_with_statement_keyword_column_parses(self) -> None: + """``select`` is a column name here. The predicate parser's + ``SELECT 1 WHERE ...`` guard exists so this is not read as a + statement.""" + out = _sql_of(_scope().enter_predicate("\"select\" = 'x'")) + assert "select" in out.lower(), out + + def test_expression_returns_a_non_boolean_node(self) -> None: + node = _scope().enter_expression("amount * 2") + assert isinstance(node, exp.Expression) + assert not isinstance(node, (exp.EQ, exp.GT, exp.LT, exp.And, exp.Or)) + + def test_predicate_returns_a_boolean_node(self) -> None: + node = _scope().enter_predicate("amount > 2") + assert isinstance(node, exp.GT) diff --git a/tests/test_dev1745_plan_time_routing.py b/tests/test_dev1745_plan_time_routing.py new file mode 100644 index 00000000..df5083a1 --- /dev/null +++ b/tests/test_dev1745_plan_time_routing.py @@ -0,0 +1,182 @@ +"""DEV-1745 (W3) — filter classification is plan-time; the generator consumes +the plan verbatim (P-D: plan decides, render emits). + +The outer combined-SELECT WHERE wrapper (DEV-1503) currently decides at RENDER +time: the generator re-walks ``planned_query.filters_by_phase`` with +``walk_value_keys`` looking for AGGREGATE-phase filters that reference a +filtered-local isolated aggregate, and builds its own id set. That is policy +decided during emission. + +After this change the planner computes the routing and ``PlannedQuery`` carries +it; the generator reads the field and never re-derives it. + +The decisive test is that the plan field is AUTHORITATIVE: clear it, and the +outer WHERE disappears. A generator that re-walks would keep emitting it and +the test fails — which is exactly the coupling being removed. + +``frame_bound_columns`` and the windowed ``_src`` residuals +(``SrcFilterRewrite``) are already plan-side; the guards here pin that so the +migration does not quietly re-introduce a render-time derivation. +""" + +from __future__ import annotations + +import pytest + +from slayer.core.enums import DataType, TimeGranularity +from slayer.core.models import Column, ModelJoin, SlayerModel +from slayer.core.query import SlayerQuery +from slayer.engine.source_bundle import ResolvedSourceBundle +from slayer.engine.stage_planner import plan_query + +from tests._engine_helpers import _engine_generate + + +# --------------------------------------------------------------------------- # +# A filtered-local isolated aggregate: `eu_amount` carries a Column.filter that +# crosses into `customers`, so the measure is isolated into a _cm_ CTE with +# cte_root_model set, and the AGGREGATE-phase filter on it routes to the outer +# combined SELECT as a plain WHERE on the joined-back column. +# --------------------------------------------------------------------------- # +def _customers() -> SlayerModel: + return SlayerModel( + name="customers", data_source="test", sql_table="customers", + columns=[ + Column(name="id", type=DataType.INT, primary_key=True), + Column(name="tier", type=DataType.TEXT), + ], + ) + + +def _orders() -> SlayerModel: + return SlayerModel( + name="orders", data_source="test", sql_table="orders", + columns=[ + Column(name="id", type=DataType.INT, primary_key=True), + Column(name="customer_id", type=DataType.INT), + Column(name="status", type=DataType.TEXT), + Column(name="amount", type=DataType.DOUBLE), + Column(name="created_at", type=DataType.TIMESTAMP), + Column(name="eu_amount", sql="amount", + filter="customers.tier = 'eu'", type=DataType.DOUBLE), + ], + joins=[ModelJoin( + target_model="customers", join_pairs=[["customer_id", "id"]], + )], + ) + + +def _bundle() -> ResolvedSourceBundle: + return ResolvedSourceBundle( + source_model=_orders(), referenced_models=[_customers()], + ) + + +def _outer_where_query() -> SlayerQuery: + return SlayerQuery( + source_model="orders", + dimensions=[{"formula": "status", "name": "status"}], + measures=[{"formula": "eu_amount:sum", "name": "eu"}], + filters=["eu_amount:sum > 100"], + ) + + +# --------------------------------------------------------------------------- # +class TestPlanCarriesOuterWhereRouting: + + def test_plan_declares_the_field_on_the_schema(self) -> None: + """A DECLARED Pydantic field, not merely an attribute — ``model_copy`` + can graft an undeclared key onto an instance, so ``hasattr`` alone + would not prove the schema owns it.""" + from slayer.engine.planned import PlannedQuery + + assert "outer_where_filter_ids" in PlannedQuery.model_fields, ( + "PlannedQuery must DECLARE the outer-WHERE routing field decided " + f"at plan time; fields are {sorted(PlannedQuery.model_fields)}" + ) + + def test_field_is_populated_for_the_isolated_shape(self) -> None: + planned = plan_query(query=_outer_where_query(), bundle=_bundle()) + assert list(planned.outer_where_filter_ids) == ["f0"], ( + f"expected f0 routed to the outer WHERE, got " + f"{getattr(planned, 'outer_where_filter_ids', None)!r}" + ) + + def test_field_is_empty_without_an_isolated_aggregate(self) -> None: + plain = SlayerQuery( + source_model="orders", + dimensions=[{"formula": "status", "name": "status"}], + measures=[{"formula": "amount:sum", "name": "a"}], + filters=["amount:sum > 100"], + ) + planned = plan_query(query=plain, bundle=_bundle()) + assert list(planned.outer_where_filter_ids) == [] + + def test_the_isolated_plan_is_the_trigger(self) -> None: + """Sanity-pin the shape the routing keys off: a cross-model plan whose + cte_root_model is set.""" + planned = plan_query(query=_outer_where_query(), bundle=_bundle()) + roots = [ + p.cte_root_model for p in planned.cross_model_aggregate_plans + ] + assert any(r is not None for r in roots), roots + + +@pytest.mark.asyncio +class TestGeneratorConsumesThePlanVerbatim: + + async def _sql(self, query: SlayerQuery) -> str: + return await _engine_generate( + query=query, model=_orders(), dialect="postgres", + validate=False, extra_models=[_customers()], + ) + + async def test_outer_where_is_emitted_for_the_isolated_shape(self) -> None: + sql = await self._sql(_outer_where_query()) + assert "> 100" in sql, sql + + async def test_clearing_the_plan_field_removes_the_outer_where(self) -> None: + """P-D: the plan is authoritative. A generator that re-walks the + filters at render time would ignore the cleared field and keep + emitting the predicate.""" + from slayer.sql.generator import SQLGenerator + + planned = plan_query(query=_outer_where_query(), bundle=_bundle()) + assert list(planned.outer_where_filter_ids) == ["f0"], ( + "precondition: the plan must be POPULATED before clearing, " + "otherwise clearing proves nothing" + ) + cleared = planned.model_copy(update={"outer_where_filter_ids": []}) + gen = SQLGenerator(dialect="postgres") + sql = gen.generate_from_planned(planned_query=cleared, bundle=_bundle()) + assert "> 100" not in sql, ( + "the generator re-derived the outer-WHERE routing instead of " + f"consuming the plan:\n{sql}" + ) + + +class TestFrameBoundColumnsStayPlanSide: + """Parity guards — already true today, pinned so the door migration does + not re-introduce a render-time derivation.""" + + def _windowed_query(self) -> SlayerQuery: + return SlayerQuery( + source_model="orders", + time_dimensions=[{ + "dimension": "created_at", + "granularity": TimeGranularity.MONTH, + "date_range": ["2024-01-01", "2024-12-31"], + }], + measures=[{"formula": "amount:sum", "name": "a"}], + ) + + def test_plan_carries_frame_bound_columns(self) -> None: + planned = plan_query(query=self._windowed_query(), bundle=_bundle()) + assert hasattr(planned, "frame_bound_columns") + + def test_frame_bound_columns_covers_the_time_dimension(self) -> None: + planned = plan_query(query=self._windowed_query(), bundle=_bundle()) + assert planned.frame_bound_columns, ( + "the time dimension's raw column must be carried on the plan so " + "both strip_frame_bounds call sites read the SAME set" + ) diff --git a/tests/test_dev1745_reachability.py b/tests/test_dev1745_reachability.py new file mode 100644 index 00000000..ac9c4cc6 --- /dev/null +++ b/tests/test_dev1745_reachability.py @@ -0,0 +1,434 @@ +"""DEV-1745 (W4 / mechanism contract 5.3) — structural crossing metadata and +ONE reachability rule for every key kind. + +``classify_host_filter`` must route EXCLUSIVELY from structural metadata. The +``ColumnSqlKey`` model-name-membership heuristic goes: it asked whether the +derived column's model NAME appeared anywhere in ``target_path`` — a flat +membership test, not a structural prefix — so a model reachable on a SIBLING +branch counted as reachable, and a host-model derived column whose SQL crosses +INTO the target counted as host-local. + +The replacement is one rule for every kind: a dependency is reachable iff its +anchored join path is a PREFIX of ``target_path``. A derived key's effective +paths are its own ``path`` plus the paths its expanded ``Column.sql`` crosses. + +Reachability is an ALL-DEPENDENCIES predicate — a filter propagates only if +EVERY dependency is available in the destination scope. Any unreachable +dependency drops the filter. + +The metadata is computed at PLAN time and carried per filter, NOT on +``ColumnSqlKey`` (it is interned, and ``_reroot_path_ref`` copies unknown +fields through rerooting unchanged) and NOT on ``ValueSlot`` +(``filter_referenced_slot_ids`` silently skips keys with no interned slot, and +filter-only derived columns are exactly such keys). +""" + +from __future__ import annotations + +import pytest + +from slayer.core.enums import DataType +from slayer.core.keys import ( + AggregateKey, + ArithmeticKey, + BetweenKey, + ColumnKey, + ColumnSqlKey, + InKey, + LiteralKey, + Phase, +) +from slayer.core.models import Column, ModelJoin, SlayerModel +from slayer.engine.source_bundle import ResolvedSourceBundle + + +# --------------------------------------------------------------------------- # +# Model graph: orders -> customers -> regions +# -> warehouses (SIBLING branch) +# --------------------------------------------------------------------------- # +def _regions() -> SlayerModel: + return SlayerModel( + name="regions", sql_table="regions", data_source="test", + columns=[ + Column(name="id", type=DataType.INT, primary_key=True), + Column(name="name", type=DataType.TEXT), + Column(name="population", type=DataType.DOUBLE), + Column(name="pop_x2", sql="population * 2", type=DataType.DOUBLE), + ], + ) + + +def _customers() -> SlayerModel: + return SlayerModel( + name="customers", sql_table="customers", data_source="test", + columns=[ + Column(name="id", type=DataType.INT, primary_key=True), + Column(name="region_id", type=DataType.INT), + Column(name="balance", type=DataType.DOUBLE), + ], + joins=[ModelJoin(target_model="regions", join_pairs=[["region_id", "id"]])], + ) + + +def _warehouses() -> SlayerModel: + return SlayerModel( + name="warehouses", sql_table="warehouses", data_source="test", + columns=[ + Column(name="id", type=DataType.INT, primary_key=True), + # deliberately shares the name 'regions' territory: a warehouse + # also has a region_id, so 'regions' appears on BOTH branches + Column(name="region_id", type=DataType.INT), + ], + joins=[ModelJoin(target_model="regions", join_pairs=[["region_id", "id"]])], + ) + + +def _orders() -> SlayerModel: + return SlayerModel( + name="orders", sql_table="orders", data_source="test", + columns=[ + Column(name="id", type=DataType.INT, primary_key=True), + Column(name="customer_id", type=DataType.INT), + Column(name="warehouse_id", type=DataType.INT), + Column(name="amount", type=DataType.DOUBLE), + # a model and a column that share a name — the heuristic's blind spot + Column(name="customers", type=DataType.TEXT), + # derived, LOCAL to the host, but its SQL crosses INTO customers + Column(name="host_derived_crossing", sql="customers.balance * 2", + type=DataType.DOUBLE), + # derived, local, purely local sql + Column(name="host_derived_local", sql="amount * 2", + type=DataType.DOUBLE), + # derived referencing TWO models + Column(name="multi_model", + sql="customers.balance + customers__regions.population", + type=DataType.DOUBLE), + # derived-of-derived across two hops + Column(name="deep_pop", sql="customers__regions.pop_x2", + type=DataType.DOUBLE), + # quoted dotted identifier + Column(name="quoted_cross", sql='"customers"."balance"', + type=DataType.DOUBLE), + ], + joins=[ + ModelJoin(target_model="customers", join_pairs=[["customer_id", "id"]]), + ModelJoin(target_model="warehouses", join_pairs=[["warehouse_id", "id"]]), + ], + ) + + +def _bundle() -> ResolvedSourceBundle: + host = _orders() + return ResolvedSourceBundle( + source_model=host, + referenced_models=[host, _customers(), _regions(), _warehouses()], + ) + + +def _paths_for(key): + """Anchored crossed-join-path set for one ValueKey, at plan time.""" + from slayer.engine.filter_reachability import compute_key_join_paths + + return compute_key_join_paths( + key=key, + anchor_model=_orders(), + anchor_relation="orders", + bundle=_bundle(), + ) + + +def _derived(name: str) -> ColumnSqlKey: + return ColumnSqlKey(path=(), model="orders", column_name=name) + + +# --------------------------------------------------------------------------- # +# The structural scan — every key kind +# --------------------------------------------------------------------------- # +class TestCrossedPathScan: + + def test_plain_local_column_crosses_nothing(self) -> None: + assert _paths_for(ColumnKey(path=(), leaf="amount")) == () + + def test_joined_column_carries_its_path(self) -> None: + paths = _paths_for(ColumnKey(path=("customers",), leaf="balance")) + assert ("customers",) in paths + + def test_host_derived_crossing_is_detected(self) -> None: + """The heuristic called this host-local because model == host.""" + paths = _paths_for(_derived("host_derived_crossing")) + assert ("customers",) in paths, ( + "a host-model derived column whose SQL crosses into a join must " + "report that crossing structurally" + ) + + def test_host_derived_local_crosses_nothing(self) -> None: + assert _paths_for(_derived("host_derived_local")) == () + + def test_derived_referencing_multiple_models(self) -> None: + paths = _paths_for(_derived("multi_model")) + assert ("customers",) in paths + assert ("customers", "regions") in paths + + def test_derived_of_derived(self) -> None: + paths = _paths_for(_derived("deep_pop")) + assert ("customers", "regions") in paths + + def test_quoted_dotted_identifier(self) -> None: + paths = _paths_for(_derived("quoted_cross")) + assert ("customers",) in paths, ( + 'a quoted dotted ref ("customers"."balance") must scan the same ' + "as an unquoted one" + ) + + def test_same_named_model_and_column_is_not_a_crossing(self) -> None: + """`orders.customers` is a COLUMN whose name matches a joined MODEL. + Referencing it crosses nothing.""" + assert _paths_for(ColumnKey(path=(), leaf="customers")) == () + + +class TestCompositeKeyKindsAreTotal: + """The summary is recursive over the whole key tree — Codex's finding that + the top-level key alone misses crossings nested below it.""" + + def test_arithmetic_unions_operands(self) -> None: + key = ArithmeticKey( + op="+", + left=ColumnKey(path=(), leaf="amount"), + right=ColumnKey(path=("customers",), leaf="balance"), + ) + assert ("customers",) in _paths_for(key) + + def test_between_covers_all_three_operands(self) -> None: + key = BetweenKey( + column=ColumnKey(path=("customers",), leaf="balance"), + low=LiteralKey(value=1), + high=LiteralKey(value=2), + ) + assert ("customers",) in _paths_for(key) + + def test_in_covers_the_tested_value(self) -> None: + key = InKey( + column=ColumnKey(path=("customers",), leaf="balance"), + values=(LiteralKey(value=1), LiteralKey(value=2)), + ) + assert ("customers",) in _paths_for(key) + + def test_aggregate_covers_its_source(self) -> None: + key = AggregateKey( + agg="sum", source=ColumnKey(path=("customers",), leaf="balance"), + ) + assert ("customers",) in _paths_for(key) + + def test_nested_derived_below_a_composite(self) -> None: + """A derived crossing column buried under arithmetic must still be + seen — this is the case a top-level-only scan misses.""" + key = ArithmeticKey( + op="+", + left=ColumnKey(path=(), leaf="amount"), + right=_derived("host_derived_crossing"), + ) + assert ("customers",) in _paths_for(key) + + def test_literal_crosses_nothing(self) -> None: + assert _paths_for(LiteralKey(value=1)) == () + + def test_unknown_key_kind_fails_closed(self) -> None: + """Fails CLOSED on an unhandled kind — and with an error that names the + offending type, not an incidental AttributeError/TypeError from + stumbling over an unexpected shape.""" + from slayer.engine.filter_reachability import ( + UnhandledValueKindError, + compute_key_join_paths, + ) + + class _Bogus: + pass + + with pytest.raises(UnhandledValueKindError) as excinfo: + compute_key_join_paths( + key=_Bogus(), anchor_model=_orders(), + anchor_relation="orders", bundle=_bundle(), + ) + assert "_Bogus" in str(excinfo.value), ( + "the error must identify the unhandled key type" + ) + + +# --------------------------------------------------------------------------- # +# Routing outcomes +# --------------------------------------------------------------------------- # +class TestStructuralRouting: + + def _route(self, key, *, target_path, phase=Phase.ROW): + from slayer.engine.cross_model_planner import classify_host_filter + + return classify_host_filter( + host_filter=self._routing(key, phase=phase), + host_slots=[], + target_path=target_path, + ) + + def _routing(self, key, *, phase=Phase.ROW): + from slayer.engine.cross_model_planner import HostFilterRouting + from slayer.engine.filter_reachability import compute_key_join_paths + + paths = compute_key_join_paths( + key=key, anchor_model=_orders(), anchor_relation="orders", + bundle=_bundle(), + ) + return HostFilterRouting( + filter_id="f1", phase=phase, referenced_slot_ids=[], text="", + crossed_join_paths=paths, + ) + + def test_sibling_branch_is_not_reachable(self) -> None: + """`regions` is reachable from BOTH customers and warehouses. Under + the old flat membership test, a warehouses->regions reference counted + as reachable for target ('customers',). Structurally it is not.""" + from slayer.engine.cross_model_planner import FilterRoute + + route = self._route( + ColumnKey(path=("warehouses", "regions"), leaf="population"), + target_path=("customers",), + ) + assert route == FilterRoute.DROP_UNREACHABLE + + def test_path_deeper_than_target_is_not_reachable(self) -> None: + """A dependency BELOW the target is not available in the target's + scope. Note the direction: ``path`` deeper than ``target_path``. + + (The converse — a path that is a proper PREFIX of the target — IS + reachable under ``path == target_path[:len(path)]``, and is covered by + ``test_prefix_of_target_is_reachable`` below.) + """ + from slayer.engine.cross_model_planner import FilterRoute + + route = self._route( + ColumnKey( + path=("customers", "regions", "subregions"), leaf="population", + ), + target_path=("customers", "regions"), + ) + assert route == FilterRoute.DROP_UNREACHABLE + + def test_prefix_of_target_is_reachable(self) -> None: + """The rule is ``path == target_path[:len(path)]`` — a path that is a + prefix of the target is reachable, not dropped.""" + from slayer.engine.cross_model_planner import FilterRoute + + route = self._route( + ColumnKey(path=("customers",), leaf="balance"), + target_path=("customers", "regions"), + ) + assert route == FilterRoute.PROPAGATE_WHERE + + def test_exact_path_match_is_reachable(self) -> None: + from slayer.engine.cross_model_planner import FilterRoute + + route = self._route( + ColumnKey(path=("customers", "regions"), leaf="population"), + target_path=("customers", "regions"), + ) + assert route == FilterRoute.PROPAGATE_WHERE + + def test_mixed_reachable_and_unreachable_drops(self) -> None: + from slayer.engine.cross_model_planner import FilterRoute + + key = ArithmeticKey( + op="+", + left=ColumnKey(path=("customers",), leaf="balance"), + right=ColumnKey(path=("warehouses",), leaf="id"), + ) + route = self._route(key, target_path=("customers",)) + assert route == FilterRoute.DROP_UNREACHABLE, ( + "reachability is an ALL-dependencies predicate" + ) + + def test_empty_path_is_host_local(self) -> None: + from slayer.engine.cross_model_planner import FilterRoute + + route = self._route( + ColumnKey(path=(), leaf="amount"), target_path=("customers",), + ) + assert route == FilterRoute.DROP_HOST_LOCAL + + +class TestCoordinateSystemInvariant: + """D9: every reachability summary is expressed in the coordinate system of + the ``PlannedQuery`` that owns it — recomputed per plan, NEVER copied. + + This is the trap that ruled out both alternative homes. On ``ColumnSqlKey``, + ``_reroot_path_ref`` re-anchors with ``model_copy(update={"path": ...})`` + and carries unknown fields through unchanged. On ``ValueSlot``, slots are + copied wholesale (e.g. ``agg_slot.model_copy(update={"key": ...})`` in the + cross-model CTE builder), so a copied slot would carry PARENT-anchored + paths into a nested plan. + """ + + def test_same_key_anchored_at_different_roots_differs(self) -> None: + """A key crossing ('customers','regions') from the orders root is + ('regions',) when the anchor IS customers. A summary that were copied + rather than recomputed would report the parent's paths.""" + from slayer.engine.filter_reachability import compute_key_join_paths + + key = ColumnKey(path=("customers", "regions"), leaf="population") + from_orders = compute_key_join_paths( + key=key, anchor_model=_orders(), anchor_relation="orders", + bundle=_bundle(), + ) + rerooted = key.model_copy(update={"path": ("regions",)}) + from_customers = compute_key_join_paths( + key=rerooted, anchor_model=_customers(), + anchor_relation="customers", bundle=_bundle(), + ) + assert from_orders != from_customers, ( + "reachability must be anchor-relative; identical results for two " + "different anchors means the summary is not being recomputed" + ) + assert ("regions",) in from_customers + assert ("customers", "regions") in from_orders + + def test_nested_plan_recomputes_rather_than_inherits(self) -> None: + """The nested (rerooted) plan a cross-model CTE compiles must carry its + OWN summary, not the parent's.""" + from slayer.core.query import SlayerQuery + from slayer.engine.filter_reachability import ( + filter_reachability_for, + recompute_filter_reachability, + ) + from slayer.engine.stage_planner import plan_query + + host = _orders() + host.columns.append( + Column(name="eu_amount", sql="amount", + filter="customers.balance > 0", type=DataType.DOUBLE), + ) + bundle = ResolvedSourceBundle( + source_model=host, + referenced_models=[host, _customers(), _regions(), _warehouses()], + ) + planned = plan_query( + query=SlayerQuery( + source_model="orders", + dimensions=[{"formula": "amount", "name": "amount"}], + measures=[{"formula": "eu_amount:sum", "name": "eu"}], + filters=["eu_amount:sum > 100"], + ), + bundle=bundle, + ) + nested = [ + p.rerooted_plan for p in planned.cross_model_aggregate_plans + if p.rerooted_plan is not None + ] + assert nested, "fixture must produce a nested rerooted plan" + for sub in nested: + # The summary the nested plan CARRIES must equal a fresh + # computation anchored at the nested plan's own root. If the parent + # had copied its summary down, the carried value would still be + # anchored at the parent root and these would differ. + assert filter_reachability_for(sub) == recompute_filter_reachability( + sub, bundle=bundle, + ), ( + "nested plan carries a summary anchored in the PARENT's " + "coordinate system instead of its own" + ) diff --git a/tests/test_dev1745_warning_contract.py b/tests/test_dev1745_warning_contract.py new file mode 100644 index 00000000..3e274fb4 --- /dev/null +++ b/tests/test_dev1745_warning_contract.py @@ -0,0 +1,577 @@ +"""DEV-1745 (W5 / mechanism contract 5.5) — the dropped-filter warning contract. + +Exactly ONE ``UnreachableFilterDroppedWarning`` per user filter per execute, +carrying the filter's original text, location and drop reason, emitted at the +ENGINE BOUNDARY so every entry point sees it. + +What it replaces: a bare ``warnings.warn(str(w), UserWarning)`` fired MID-RENDER, +once per cross-model plan — so nested subplans double-fired, and any path that +did not reach that render step emitted nothing at all. Nothing downstream could +observe it either: ``SlayerResponse.warnings`` was typed to normalization +warnings only, and no entry point rendered warnings of any kind. + +Dedup identity (D8) is ``(location, original filter text)`` — the user-facing +identity, because the contract is stated in user-facing terms. Drop reasons for +the same filter must AGREE; disagreement is a planner inconsistency and is +asserted, not silently resolved by taking the first. + +Binder/planner internal failures RAISE. They never masquerade as expected drops. + +Ordering under warnings-as-errors: collection completes and the structured +payload is built FIRST; the Python ``warnings.warn`` emission happens LAST at +the outermost boundary. Under ``-W error`` that raises and the response is not +delivered — intended, and asserted here rather than left implicit. +""" + +from __future__ import annotations + +import tempfile +import warnings + +import pytest + +from slayer.core.enums import DataType +from slayer.core.models import Column, DatasourceConfig, ModelJoin, SlayerModel +from slayer.core.query import SlayerQuery +from slayer.engine.query_engine import SlayerQueryEngine +from slayer.storage.yaml_storage import YAMLStorage + + +# --------------------------------------------------------------------------- # +# Fixtures — a query whose host filter is unreachable from the CTE root. +# `warehouses` is a SIBLING branch of `customers`, so a filter on it cannot be +# propagated into the customers-rooted _cm_ CTE. +# --------------------------------------------------------------------------- # +def _warehouses() -> SlayerModel: + return SlayerModel( + name="warehouses", data_source="test", sql_table="warehouses", + columns=[ + Column(name="id", type=DataType.INT, primary_key=True), + Column(name="code", type=DataType.TEXT), + ], + ) + + +def _customers() -> SlayerModel: + return SlayerModel( + name="customers", data_source="test", sql_table="customers", + columns=[ + Column(name="id", type=DataType.INT, primary_key=True), + Column(name="revenue", type=DataType.DOUBLE), + ], + ) + + +def _shippers() -> SlayerModel: + return SlayerModel( + name="shippers", data_source="test", sql_table="shippers", + columns=[ + Column(name="id", type=DataType.INT, primary_key=True), + Column(name="cost", type=DataType.DOUBLE), + ], + ) + + +def _orders() -> SlayerModel: + return SlayerModel( + name="orders", data_source="test", sql_table="orders", + columns=[ + Column(name="id", type=DataType.INT, primary_key=True), + Column(name="customer_id", type=DataType.INT), + Column(name="shipper_id", type=DataType.INT), + Column(name="warehouse_id", type=DataType.INT), + Column(name="status", type=DataType.TEXT), + Column(name="amount", type=DataType.DOUBLE), + ], + joins=[ + ModelJoin(target_model="customers", join_pairs=[["customer_id", "id"]]), + ModelJoin(target_model="shippers", join_pairs=[["shipper_id", "id"]]), + ModelJoin(target_model="warehouses", join_pairs=[["warehouse_id", "id"]]), + ], + ) + + +DROPPED_FILTER = "warehouses.code == 'X'" + + +def _query(*, extra_filters: list | None = None) -> SlayerQuery: + return SlayerQuery( + source_model="orders", + dimensions=[{"formula": "status", "name": "status"}], + measures=[{"formula": "customers.revenue:sum"}], + filters=[DROPPED_FILTER, *(extra_filters or [])], + ) + + +async def _engine(tmpdir: str) -> SlayerQueryEngine: + storage = YAMLStorage(base_dir=tmpdir) + # ``database`` MUST be set: ``explain=True`` opens a real connection, and a + # DuckDB datasource with database=None writes a file literally named "None" + # into the working directory. + await storage.save_datasource( + DatasourceConfig(name="test", type="duckdb", database=":memory:") + ) + for m in (_orders(), _customers(), _warehouses(), _shippers()): + await storage.save_model(m, _validate=False) + return SlayerQueryEngine(storage=storage) + + +def _two_plan_query() -> SlayerQuery: + """ONE user filter, unreachable from TWO different cross-model targets. + + Verified: this produces two separate ``dropped_filter_warnings`` entries + (one on the customers plan, one on the shippers plan) for the SAME user + filter. Deduping them to a single warning is the contract's core claim, and + without this shape nothing in the suite distinguishes "one per filter" from + "one per plan". + """ + return SlayerQuery( + source_model="orders", + dimensions=[{"formula": "status", "name": "status"}], + measures=[ + {"formula": "customers.revenue:sum"}, + {"formula": "shippers.cost:sum"}, + ], + filters=[DROPPED_FILTER], + ) + + +def _dropped(response) -> list: + """Dropped-filter payloads on a SlayerResponse.""" + return [ + w for w in (response.warnings or []) + if getattr(w, "kind", None) == "unreachable_filter_dropped" + ] + + +# --------------------------------------------------------------------------- # +# Python entry point +# --------------------------------------------------------------------------- # +@pytest.mark.asyncio +class TestExecuteEntryPoint: + + async def test_exactly_one_python_warning_per_filter(self) -> None: + from slayer.core.errors import UnreachableFilterDroppedWarning + + with tempfile.TemporaryDirectory() as d: + engine = await _engine(d) + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + await engine.execute(_query(), dry_run=True) + hits = [ + w for w in caught + if issubclass(w.category, UnreachableFilterDroppedWarning) + ] + assert len(hits) == 1, ( + f"expected exactly one UnreachableFilterDroppedWarning, got " + f"{len(hits)}: {[str(w.message) for w in caught]}" + ) + + async def test_warning_is_the_typed_class_not_bare_userwarning(self) -> None: + from slayer.core.errors import UnreachableFilterDroppedWarning + + with tempfile.TemporaryDirectory() as d: + engine = await _engine(d) + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + await engine.execute(_query(), dry_run=True) + assert any( + w.category is UnreachableFilterDroppedWarning for w in caught + ), f"categories seen: {[w.category for w in caught]}" + + async def test_response_carries_a_structured_payload(self) -> None: + with tempfile.TemporaryDirectory() as d: + engine = await _engine(d) + resp = await engine.execute(_query(), dry_run=True) + payloads = _dropped(resp) + assert len(payloads) == 1, f"warnings: {resp.warnings!r}" + + async def test_payload_carries_text_location_and_reason(self) -> None: + with tempfile.TemporaryDirectory() as d: + engine = await _engine(d) + resp = await engine.execute(_query(), dry_run=True) + (payload,) = _dropped(resp) + # ORIGINAL author text — not normalized, prequoted or re-rendered + assert payload.filter_text == DROPPED_FILTER, ( + f"payload must carry the filter's ORIGINAL text verbatim; got " + f"{payload.filter_text!r} vs {DROPPED_FILTER!r}" + ) + assert payload.location, "the payload must carry a location" + assert payload.reason, "the payload must carry a drop reason" + assert payload.kind == "unreachable_filter_dropped", payload.kind + + async def test_two_dropped_filters_produce_two_warnings(self) -> None: + from slayer.core.errors import UnreachableFilterDroppedWarning + + with tempfile.TemporaryDirectory() as d: + engine = await _engine(d) + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + await engine.execute( + _query(extra_filters=["warehouses.code == 'Y'"]), + dry_run=True, + ) + hits = [ + w for w in caught + if issubclass(w.category, UnreachableFilterDroppedWarning) + ] + assert len(hits) == 2, ( + f"one warning PER FILTER; got {len(hits)}" + ) + + async def test_clean_query_warns_nothing(self) -> None: + from slayer.core.errors import UnreachableFilterDroppedWarning + + with tempfile.TemporaryDirectory() as d: + engine = await _engine(d) + clean = SlayerQuery( + source_model="orders", + dimensions=[{"formula": "status", "name": "status"}], + measures=[{"formula": "amount:sum", "name": "m0"}], + ) + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + resp = await engine.execute(clean, dry_run=True) + assert not [ + w for w in caught + if issubclass(w.category, UnreachableFilterDroppedWarning) + ] + assert _dropped(resp) == [] + + +@pytest.mark.asyncio +class TestEmissionIsBoundaryNotRender: + """The old emission sat mid-render, so paths that did not reach it were + silent. The boundary emission is path-independent.""" + + @pytest.mark.parametrize("kwargs", [ + pytest.param({"dry_run": True}, id="dry_run"), + pytest.param({"explain": True}, id="explain"), + ]) + async def test_warning_emitted_on_every_execute_mode(self, kwargs) -> None: + with tempfile.TemporaryDirectory() as d: + engine = await _engine(d) + resp = await engine.execute(_query(), **kwargs) + assert len(_dropped(resp)) == 1, ( + f"no dropped-filter payload for execute(**{kwargs})" + ) + + async def test_one_filter_dropped_by_two_plans_warns_once(self) -> None: + """The decisive dedup case. Pre-dedup this produces TWO raw + dropped-filter entries for one user filter — the old per-plan emission + fired both.""" + from slayer.core.errors import UnreachableFilterDroppedWarning + + with tempfile.TemporaryDirectory() as d: + engine = await _engine(d) + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + resp = await engine.execute(_two_plan_query(), dry_run=True) + hits = [ + w for w in caught + if issubclass(w.category, UnreachableFilterDroppedWarning) + ] + assert len(hits) == 1, ( + f"one user filter dropped by two plans must warn ONCE, got " + f"{len(hits)}" + ) + assert len(_dropped(resp)) == 1, ( + f"structured payloads must dedup too, got {_dropped(resp)!r}" + ) + + async def test_two_plan_drop_reasons_agree(self) -> None: + """D8: the same filter dropped by several plans must carry ONE reason. + + Asserting only that the surviving reason is truthy would pass an + implementation that produced two CONFLICTING reasons and arbitrarily + kept the first. So compare the PRE-dedup reasons the planner produced + directly, then check the boundary collapsed them to one. + """ + from slayer.engine.source_bundle import ResolvedSourceBundle + from slayer.engine.stage_planner import plan_query + + bundle = ResolvedSourceBundle( + source_model=_orders(), + referenced_models=[_customers(), _warehouses(), _shippers()], + ) + planned = plan_query(query=_two_plan_query(), bundle=bundle) + raw = [ + w + for plan in planned.cross_model_aggregate_plans + for w in plan.dropped_filter_warnings + ] + assert len(raw) >= 2, ( + f"fixture must produce the multi-plan drop; got {len(raw)}" + ) + reasons = {getattr(w, "reason", str(w)) for w in raw} + assert len(reasons) == 1, ( + f"the same filter was dropped for DIFFERENT reasons by different " + f"plans — a planner inconsistency that must not be hidden by " + f"keeping the first: {reasons!r}" + ) + + with tempfile.TemporaryDirectory() as d: + engine = await _engine(d) + resp = await engine.execute(_two_plan_query(), dry_run=True) + (payload,) = _dropped(resp) + assert payload.reason == next(iter(reasons)), ( + "the surfaced reason must be the one the planner produced" + ) + + async def test_identical_text_at_different_locations_stays_two(self) -> None: + """D8's identity is (location, text) — NOT text alone. Two stages each + carrying the same filter text are two distinct user filters.""" + from slayer.core.errors import UnreachableFilterDroppedWarning + + inner = SlayerQuery( + name="s1", + source_model="orders", + dimensions=[{"formula": "status", "name": "status"}], + measures=[{"formula": "customers.revenue:sum"}], + filters=[DROPPED_FILTER], + ) + outer = SlayerQuery( + source_model="orders", + dimensions=[{"formula": "status", "name": "status"}], + measures=[{"formula": "customers.revenue:sum"}], + filters=[DROPPED_FILTER], + ) + with tempfile.TemporaryDirectory() as d: + engine = await _engine(d) + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + await engine.execute([inner, outer], dry_run=True) + hits = [ + w for w in caught + if issubclass(w.category, UnreachableFilterDroppedWarning) + ] + assert len(hits) == 2, ( + f"same text in two different stages is two distinct user " + f"filters; got {len(hits)}" + ) + + async def test_repeated_execution_does_not_accumulate(self) -> None: + """Per EXECUTE, not per process.""" + with tempfile.TemporaryDirectory() as d: + engine = await _engine(d) + first = await engine.execute(_query(), dry_run=True) + second = await engine.execute(_query(), dry_run=True) + assert len(_dropped(first)) == 1 + assert len(_dropped(second)) == 1 + + +class TestWarningTypeHierarchy: + """D6: one discriminated family, so a consumer reads ONE list and switches + on ``kind``.""" + + def test_dropped_filter_warning_subclasses_the_base(self) -> None: + from slayer.core.warnings import DroppedFilterWarning, SlayerWarning + + assert issubclass(DroppedFilterWarning, SlayerWarning) + + def test_normalization_warning_subclasses_the_base(self) -> None: + from slayer.core.warnings import NormalizationWarning, SlayerWarning + + assert issubclass(NormalizationWarning, SlayerWarning) + + def test_each_subclass_declares_a_distinct_kind(self) -> None: + from slayer.core.warnings import ( + DroppedFilterWarning, + NormalizationWarning, + ) + + kinds = { + NormalizationWarning.model_fields["kind"].default, + DroppedFilterWarning.model_fields["kind"].default, + } + assert kinds == {"normalization", "unreachable_filter_dropped"}, kinds + + +@pytest.mark.asyncio +class TestLowerLayersStaySilent: + """The emission is at the BOUNDARY. Planning and rendering must not warn on + their own — otherwise 'exactly once' holds only by luck of deduplication.""" + + @staticmethod + def _filter_warnings(caught) -> list: + """Any warning mentioning the dropped filter, WHATEVER its category. + + Filtering on ``UnreachableFilterDroppedWarning`` would miss the thing + this test exists to catch: today the generator emits a BARE + ``UserWarning``, which is that class's parent, not a subclass — so a + subclass check passes vacuously. + """ + return [w for w in caught if "warehouses.code" in str(w.message)] + + async def test_planning_emits_no_python_warning(self) -> None: + from slayer.engine.source_bundle import ResolvedSourceBundle + from slayer.engine.stage_planner import plan_query + + bundle = ResolvedSourceBundle( + source_model=_orders(), + referenced_models=[_customers(), _warehouses(), _shippers()], + ) + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + plan_query(query=_query(), bundle=bundle) + assert not self._filter_warnings(caught), ( + "the PLANNER emitted a dropped-filter warning; emission belongs " + "at the engine boundary" + ) + + async def test_rendering_emits_no_python_warning(self) -> None: + from slayer.engine.source_bundle import ResolvedSourceBundle + from slayer.engine.stage_planner import plan_query + from slayer.sql.generator import SQLGenerator + + bundle = ResolvedSourceBundle( + source_model=_orders(), + referenced_models=[_customers(), _warehouses(), _shippers()], + ) + planned = plan_query(query=_query(), bundle=bundle) + gen = SQLGenerator(dialect="duckdb") + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + gen.generate_from_planned(planned_query=planned, bundle=bundle) + assert not self._filter_warnings(caught), ( + "the GENERATOR emitted a dropped-filter warning; emission belongs " + "at the engine boundary" + ) + + +@pytest.mark.asyncio +class TestWarningsAsErrors: + + async def test_warnings_as_errors_raises(self) -> None: + from slayer.core.errors import UnreachableFilterDroppedWarning + + with tempfile.TemporaryDirectory() as d: + engine = await _engine(d) + with warnings.catch_warnings(): + warnings.simplefilter("error", UnreachableFilterDroppedWarning) + with pytest.raises(UnreachableFilterDroppedWarning): + await engine.execute(_query(), dry_run=True) + + +@pytest.mark.asyncio +class TestInternalFailuresRaise: + """A binder/planner bug must never be reported as an expected drop.""" + + async def test_unknown_reference_raises_not_warns(self) -> None: + from slayer.core.errors import SlayerError + + with tempfile.TemporaryDirectory() as d: + engine = await _engine(d) + bad = SlayerQuery( + source_model="orders", + dimensions=[{"formula": "status", "name": "status"}], + measures=[{"formula": "customers.revenue:sum"}], + filters=["no_such_column == 'X'"], + ) + with pytest.raises(SlayerError): + await engine.execute(bad, dry_run=True) + + +# --------------------------------------------------------------------------- # +# Every entry point — asserted on real output, not an in-process side channel +# --------------------------------------------------------------------------- # +class TestRestEntryPoint: + + def test_rest_query_response_surfaces_warnings(self) -> None: + import asyncio + + from fastapi.testclient import TestClient + + from slayer.api.server import create_app + + with tempfile.TemporaryDirectory() as d: + storage = YAMLStorage(base_dir=d) + + async def _seed(): + await storage.save_datasource( + DatasourceConfig(name="test", type="duckdb") + ) + for m in (_orders(), _customers(), _warehouses()): + await storage.save_model(m, _validate=False) + + asyncio.run(_seed()) + client = TestClient(create_app(storage=storage)) + resp = client.post("/query", json={ + "query": _query().model_dump(mode="json", exclude_none=True), + "dry_run": True, + }) + assert resp.status_code == 200, resp.text + body = resp.json() + assert "warnings" in body, ( + f"REST QueryResponse must surface warnings; got keys {list(body)}" + ) + kinds = [w.get("kind") for w in (body.get("warnings") or [])] + assert "unreachable_filter_dropped" in kinds, body.get("warnings") + + +@pytest.mark.asyncio +class TestMcpEntryPoint: + + async def test_mcp_query_output_mentions_the_dropped_filter(self) -> None: + from slayer.mcp.server import create_mcp_server + + with tempfile.TemporaryDirectory() as d: + storage = YAMLStorage(base_dir=d) + await storage.save_datasource( + DatasourceConfig(name="test", type="duckdb") + ) + for m in (_orders(), _customers(), _warehouses()): + await storage.save_model(m, _validate=False) + server = create_mcp_server(storage=storage) + tool = await server.get_tool("query") + result = await tool.run({ + "query": _query().model_dump(mode="json", exclude_none=True), + "dry_run": True, + }) + text = str(result) + assert "warehouses.code" in text, ( + f"MCP query output must surface the dropped filter; got:\n{text}" + ) + + +class TestCliEntryPoint: + + def test_cli_surfaces_the_dropped_filter(self, capsys) -> None: + import asyncio + import json + from types import SimpleNamespace + + from slayer.cli import _run_query + + with tempfile.TemporaryDirectory() as d: + storage = YAMLStorage(base_dir=d) + + async def _seed(): + await storage.save_datasource( + DatasourceConfig(name="test", type="duckdb") + ) + for m in (_orders(), _customers(), _warehouses()): + await storage.save_model(m, _validate=False) + + asyncio.run(_seed()) + args = SimpleNamespace( + query_json=json.dumps( + _query().model_dump(mode="json", exclude_none=True) + ), + variables=None, + variables_json=None, + storage=d, + models_dir=None, + dry_run=True, + explain=False, + format="table", + ) + _run_query(args) + captured = capsys.readouterr() + combined = captured.err + captured.out + assert "warehouses.code" in combined, ( + f"CLI must surface the dropped filter; got:\n{combined}" + ) + assert "warehouses.code" in captured.err, ( + "warnings belong on stderr so stdout stays pipeable" + ) From ef58a0d206b8f33c49ba3f37964928dcd15d6a13 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Wed, 5 Aug 2026 21:55:56 +0200 Subject: [PATCH 16/98] =?UTF-8?q?DEV-1744:=20comparisons=20are=20non-assoc?= =?UTF-8?q?iative=20=E2=80=94=20(a=20=3D=205)=20IS=20NULL=20lost=20its=20p?= =?UTF-8?q?arens;=20refuse=20SUM(*)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- slayer/sql/render/value_expr.py | 40 ++++++++++++++-- tests/test_dev1744_value_expr.py | 82 ++++++++++++++++++++++++++++++++ 2 files changed, 117 insertions(+), 5 deletions(-) diff --git a/slayer/sql/render/value_expr.py b/slayer/sql/render/value_expr.py index aff67cad..1d13bacf 100644 --- a/slayer/sql/render/value_expr.py +++ b/slayer/sql/render/value_expr.py @@ -212,6 +212,14 @@ def _literal(value: Any) -> exp.Expression: "=", "==", "!=", "<>", "<", "<=", ">", ">=", "is", "is not", }) +# The comparison family's shared level. Unlike arithmetic, comparisons are +# NON-ASSOCIATIVE in SQL, so an equal-precedence child needs parens on EITHER +# side — a left child included. Without that, ``(a = b) is null`` emits +# ``a = b IS NULL``, which every dialect reads as ``a = (b IS NULL)`` because +# ``IS`` binds tighter than ``=``; and ``(a < b) = c`` emits ``a < b = c``, +# which Postgres rejects outright as a non-associative chain. +_COMPARISON_PREC = 4 + def _paren_if_lower_prec( child: exp.Expression, *, parent_prec: int, is_right: bool, op: str, @@ -226,6 +234,9 @@ def _paren_if_lower_prec( and ``(a + b) + c`` genuinely different. Preserving the tree the binder built costs a pair of parentheses; regrouping costs accuracy. + At :data:`_COMPARISON_PREC` an equal-precedence LEFT child is parenthesised + too, because that family is non-associative. + A node with no precedence entry — a column, a literal, a function call — is already self-delimiting. """ @@ -234,7 +245,7 @@ def _paren_if_lower_prec( return child if child_prec < parent_prec: return exp.Paren(this=child) - if child_prec == parent_prec and is_right: + if child_prec == parent_prec and (is_right or parent_prec == _COMPARISON_PREC): return exp.Paren(this=child) return child @@ -297,10 +308,20 @@ def _render_arithmetic( f"Operator {op!r} takes exactly two operands, got {len(operands)}.", ) - if op == "is": - return exp.Is(this=operands[0], expression=operands[1]) - if op == "is not": - return exp.Not(this=exp.Is(this=operands[0], expression=operands[1])) + if op in ("is", "is not"): + # Same precedence pass as every other comparison. ``IS`` binds tighter + # than ``=``, so an unparenthesised ``(a = b) is null`` would emit + # ``a = b IS NULL`` and be read as ``a = (b IS NULL)`` — a different + # predicate that still runs. + is_prec = _PRECEDENCE[exp.Is] + lhs = _paren_if_lower_prec( + operands[0], parent_prec=is_prec, is_right=False, op=op, + ) + rhs = _paren_if_lower_prec( + operands[1], parent_prec=is_prec, is_right=True, op=op, + ) + node = exp.Is(this=lhs, expression=rhs) + return exp.Not(this=node) if op == "is not" else node node_cls = _BINARY_OPS.get(op) if node_cls is None: @@ -430,6 +451,15 @@ def _render_aggregate(key: AggregateKey, ctx: RenderContext) -> exp.Expression: f"generator's join-graph routing" ), ) + if key.agg != "count": + # ``COUNT`` is the only aggregation a bare star is defined for. + # Without this the dispatch gate happily builds ``SUM(*)`` or + # ``COUNT(DISTINCT *)`` — SQL no backend accepts, discovered at + # execution time rather than here. + raise NotImplementedError( + f"Aggregation {key.agg!r} cannot take ``*`` as its source; " + f"only 'count' is defined over a bare star.", + ) inner: exp.Expression = exp.Star() else: inner = ctx.scope.resolve(key.source, consumer=ctx.consumer) diff --git a/tests/test_dev1744_value_expr.py b/tests/test_dev1744_value_expr.py index ff23765d..afeb3d82 100644 --- a/tests/test_dev1744_value_expr.py +++ b/tests/test_dev1744_value_expr.py @@ -2067,3 +2067,85 @@ def test_not_of_a_comparison_needs_no_parens(self) -> None: def test_negated_column_needs_no_parens(self) -> None: key = ArithmeticKey(op="-", operands=(ColumnKey(leaf="amount"),)) assert _sql(render_value_key(key, _filter_ctx())) == "-orders.amount" + + +class TestComparisonsAreNonAssociative: + """The equal-precedence rule was right-child-only, and ``is`` skipped it. + + Arithmetic is left-associative, so an equal-precedence LEFT child regroups + harmlessly. The comparison family is not: SQL binds ``IS`` tighter than + ``=``, and Postgres rejects a chain of relational operators outright. Both + shapes are reachable — the Mode-B parser reads ``(a == b) == c`` as a + NESTED comparison, not a chained one, so the binder does build them. + """ + + def _cmp(self, op, left, right): + return ArithmeticKey(op=op, operands=(left, right)) + + def _inner(self, op="="): + return self._cmp( + op, ColumnKey(leaf="amount"), LiteralKey(value=Decimal(5)), + ) + + def test_comparison_over_comparison_keeps_left_parens(self) -> None: + """``(a = 5) = TRUE`` must not flatten to ``a = 5 = TRUE``.""" + key = self._cmp("=", self._inner(), LiteralKey(value=True)) + out = _sql(render_value_key(key, _filter_ctx())) + assert out == "(orders.amount = 5) = TRUE", out + + def test_mixed_relational_operators_keep_left_parens(self) -> None: + """``(a < 5) = TRUE`` emitted bare is ``a < 5 = TRUE`` — a + non-associative chain Postgres refuses to parse at all.""" + key = self._cmp("=", self._inner("<"), LiteralKey(value=True)) + out = _sql(render_value_key(key, _filter_ctx())) + assert out == "(orders.amount < 5) = TRUE", out + + def test_is_null_over_a_comparison_keeps_its_parens(self) -> None: + """The wrong-value case: ``a = 5 IS NULL`` is read as + ``a = (5 IS NULL)``, i.e. ``a = FALSE`` — a different predicate that + still returns rows.""" + key = self._cmp("is", self._inner(), LiteralKey(value=None)) + out = _sql(render_value_key(key, _filter_ctx())) + assert out == "(orders.amount = 5) IS NULL", out + + def test_is_not_null_over_a_comparison_keeps_its_parens(self) -> None: + key = self._cmp("is not", self._inner(), LiteralKey(value=None)) + out = _sql(render_value_key(key, _filter_ctx())) + assert out == "NOT (orders.amount = 5) IS NULL", out + + def test_plain_is_null_over_a_column_gains_no_parens(self) -> None: + """Don't over-wrap: a column is self-delimiting.""" + key = self._cmp("is", ColumnKey(leaf="amount"), LiteralKey(value=None)) + assert _sql(render_value_key(key, _filter_ctx())) == "orders.amount IS NULL" + + def test_arithmetic_left_child_still_flattens(self) -> None: + """The non-associativity rule is scoped to the comparison level — + ``(a - 1) - 2`` is the tree a left-fold builds, and re-parenthesising + every arithmetic left child would churn emission for no gain.""" + inner = self._cmp( + "-", ColumnKey(leaf="amount"), LiteralKey(value=Decimal(1)), + ) + key = self._cmp("-", inner, LiteralKey(value=Decimal(2))) + out = _sql(render_value_key(key, _filter_ctx())) + assert out == "orders.amount - 1 - 2", out + + +class TestStarSourceIsCountOnly: + """``*`` is only defined as ``COUNT``'s argument. + + The builder-free aggregate path gated on the dispatch MECHANISM, which + passes for every simple aggregation, so a ``StarKey`` source went straight + into whichever node the registry named — building ``SUM(*)`` and + ``COUNT(DISTINCT *)``, which no backend accepts. Refused here rather than + at execution time. + """ + + @pytest.mark.parametrize("agg", ["sum", "avg", "min", "max", "count_distinct"]) + def test_non_count_star_is_refused(self, agg) -> None: + key = AggregateKey(source=StarKey(), agg=agg) + with pytest.raises(NotImplementedError, match="bare star"): + render_value_key(key, _composite_ctx()) + + def test_count_star_still_renders(self) -> None: + key = AggregateKey(source=StarKey(), agg="count") + assert _sql(render_value_key(key, _composite_ctx())) == "COUNT(*)" From 55cea93593666d3aec4c2129701f1f46f25c010d Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Wed, 5 Aug 2026 22:03:19 +0200 Subject: [PATCH 17/98] DEV-1745: approved SQL divergences from the door migration MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two emitted-SQL changes fall out of routing every Mode-A fragment through the one door. Both approved per the DEV-1742 per-test protocol. A. Undeclared bare identifiers are now qualified against the scope root. expand_derived_refs_sync qualifies EVERY bare ref; the old fallbacks qualified only DECLARED model columns and left the rest bare, which bound them to whatever table happened to be in scope once the filter was re-rendered inside a rerooted CTE. "region = 'US'" -> "orders.region = 'US'". Updates 6 guards in test_filtered_count_forms.py and test_sql_generator.py::TestAggParamSanitization; what those guards actually assert (CASE-inside-aggregate shape, literal params not CASE-wrapped) is unchanged. B. The shifted CTE emits its Mode-A model filter from the door's AST rather than passing regex-substituted text through, so sqlglot's canonical form appears: "stores.name IS NOT NULL" -> "NOT stores.name IS NULL". The host path already re-parsed and rendered this way — the shifted path's raw-text passthrough was the inconsistency this PR removes. Also: _mode_a_scope builds its allocator via _new_allocator, keeping the single-construction-site invariant that threads dialect case-folding. Golden: the five cm/fragment_default_crossing entries blessed through the allowed-delta manifest — they now emit real SQL with the regions join instead of raising ScopeLeakError, which is W2's completion test. Full non-integration suite has zero failures outside the DEV-1745 pack. Co-Authored-By: Claude Opus 5 (1M context) --- slayer/sql/generator.py | 2 +- tests/golden/dev1745_sql_baseline.json | 25 ++++--------------- ...ev1474_time_shift_cross_model_partition.py | 7 +++++- tests/test_filtered_count_forms.py | 16 +++++++++--- tests/test_sql_generator.py | 11 +++++--- 5 files changed, 31 insertions(+), 30 deletions(-) diff --git a/slayer/sql/generator.py b/slayer/sql/generator.py index 4545bd50..757ab211 100644 --- a/slayer/sql/generator.py +++ b/slayer/sql/generator.py @@ -8431,7 +8431,7 @@ def _mode_a_scope( root_relation=source_relation, bundle=bundle, dialect=self._dialect, - allocator=AliasAllocator(), + allocator=self._new_allocator(), ) def _enter_mode_a_predicate( diff --git a/tests/golden/dev1745_sql_baseline.json b/tests/golden/dev1745_sql_baseline.json index f8652c04..172e84bb 100644 --- a/tests/golden/dev1745_sql_baseline.json +++ b/tests/golden/dev1745_sql_baseline.json @@ -1,24 +1,9 @@ { - "cm/fragment_default_crossing::bigquery": { - "error": "ScopeLeakError", - "message": "Scope not closed \u2014 1 out-of-scope reference(s):\n - [unbound_table] regions.weight in CTE _cm_orders__customers__spend_wscaled_sum (bound sources: ['_base', 'customers'])\nSQL:\nWITH _base AS (\nSELECT\n orders.status AS `orders___status`\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__customers__spend_wscaled_sum AS (\nSELECT\n SUM(customers.spend * regions.weight) AS `orders___customers___spend_wscaled_sum`\nFROM customers AS customers\n)\nSELECT _base.`orders___status`, _cm_orders__customers__spend_wscaled_sum.`orders___customers___spend_wscaled_sum`\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_wscaled_sum" - }, - "cm/fragment_default_crossing::duckdb": { - "error": "ScopeLeakError", - "message": "Scope not closed \u2014 1 out-of-scope reference(s):\n - [unbound_table] regions.weight in CTE _cm_orders__customers__spend_wscaled_sum (bound sources: ['_base', 'customers'])\nSQL:\nWITH _base AS (\nSELECT\n orders.status AS \"orders.status\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__customers__spend_wscaled_sum AS (\nSELECT\n SUM(customers.spend * regions.weight) AS \"orders.customers.spend_wscaled_sum\"\nFROM customers AS customers\n)\nSELECT _base.\"orders.status\", _cm_orders__customers__spend_wscaled_sum.\"orders.customers.spend_wscaled_sum\"\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_wscaled_sum" - }, - "cm/fragment_default_crossing::postgres": { - "error": "ScopeLeakError", - "message": "Scope not closed \u2014 1 out-of-scope reference(s):\n - [unbound_table] regions.weight in CTE _cm_orders__customers__spend_wscaled_sum (bound sources: ['_base', 'customers'])\nSQL:\nWITH _base AS (\nSELECT\n orders.status AS \"orders.status\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__customers__spend_wscaled_sum AS (\nSELECT\n SUM(customers.spend * regions.weight) AS \"orders.customers.spend_wscaled_sum\"\nFROM customers AS customers\n)\nSELECT _base.\"orders.status\", _cm_orders__customers__spend_wscaled_sum.\"orders.customers.spend_wscaled_sum\"\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_wscaled_sum" - }, - "cm/fragment_default_crossing::sqlite": { - "error": "ScopeLeakError", - "message": "Scope not closed \u2014 1 out-of-scope reference(s):\n - [unbound_table] regions.weight in CTE _cm_orders__customers__spend_wscaled_sum (bound sources: ['_base', 'customers'])\nSQL:\nWITH _base AS (\nSELECT\n orders.status AS \"orders.status\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__customers__spend_wscaled_sum AS (\nSELECT\n SUM(customers.spend * regions.weight) AS \"orders.customers.spend_wscaled_sum\"\nFROM customers AS customers\n)\nSELECT _base.\"orders.status\", _cm_orders__customers__spend_wscaled_sum.\"orders.customers.spend_wscaled_sum\"\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_wscaled_sum" - }, - "cm/fragment_default_crossing::tsql": { - "error": "ScopeLeakError", - "message": "Scope not closed \u2014 1 out-of-scope reference(s):\n - [unbound_table] regions.weight in CTE _cm_orders__customers__spend_wscaled_sum (bound sources: ['_base', 'customers'])\nSQL:\nWITH _base AS (\nSELECT\n orders.status AS [orders___status]\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__customers__spend_wscaled_sum AS (\nSELECT\n SUM(customers.spend * regions.weight) AS [orders___customers___spend_wscaled_sum]\nFROM customers AS customers\n)\nSELECT _base.[orders___status], _cm_orders__customers__spend_wscaled_sum.[orders___customers___spend_wscaled_sum]\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_wscaled_sum" - }, + "cm/fragment_default_crossing::bigquery": "WITH _base AS (\nSELECT\n orders.status AS `orders___status`\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__customers__spend_wscaled_sum AS (\nSELECT\n SUM(customers.spend * regions.weight) AS `orders___customers___spend_wscaled_sum`\nFROM customers AS customers\nLEFT JOIN regions AS regions\n ON customers.region_id = regions.id\n)\nSELECT _base.`orders___status`, _cm_orders__customers__spend_wscaled_sum.`orders___customers___spend_wscaled_sum`\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_wscaled_sum", + "cm/fragment_default_crossing::duckdb": "WITH _base AS (\nSELECT\n orders.status AS \"orders.status\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__customers__spend_wscaled_sum AS (\nSELECT\n SUM(customers.spend * regions.weight) AS \"orders.customers.spend_wscaled_sum\"\nFROM customers AS customers\nLEFT JOIN regions AS regions\n ON customers.region_id = regions.id\n)\nSELECT _base.\"orders.status\", _cm_orders__customers__spend_wscaled_sum.\"orders.customers.spend_wscaled_sum\"\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_wscaled_sum", + "cm/fragment_default_crossing::postgres": "WITH _base AS (\nSELECT\n orders.status AS \"orders.status\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__customers__spend_wscaled_sum AS (\nSELECT\n SUM(customers.spend * regions.weight) AS \"orders.customers.spend_wscaled_sum\"\nFROM customers AS customers\nLEFT JOIN regions AS regions\n ON customers.region_id = regions.id\n)\nSELECT _base.\"orders.status\", _cm_orders__customers__spend_wscaled_sum.\"orders.customers.spend_wscaled_sum\"\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_wscaled_sum", + "cm/fragment_default_crossing::sqlite": "WITH _base AS (\nSELECT\n orders.status AS \"orders.status\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__customers__spend_wscaled_sum AS (\nSELECT\n SUM(customers.spend * regions.weight) AS \"orders.customers.spend_wscaled_sum\"\nFROM customers AS customers\nLEFT JOIN regions AS regions\n ON customers.region_id = regions.id\n)\nSELECT _base.\"orders.status\", _cm_orders__customers__spend_wscaled_sum.\"orders.customers.spend_wscaled_sum\"\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_wscaled_sum", + "cm/fragment_default_crossing::tsql": "WITH _base AS (\nSELECT\n orders.status AS [orders___status]\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__customers__spend_wscaled_sum AS (\nSELECT\n SUM(customers.spend * regions.weight) AS [orders___customers___spend_wscaled_sum]\nFROM customers AS customers\nLEFT JOIN regions AS regions\n ON customers.region_id = regions.id\n)\nSELECT _base.[orders___status], _cm_orders__customers__spend_wscaled_sum.[orders___customers___spend_wscaled_sum]\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_wscaled_sum", "cm/joined_measure::bigquery": "WITH _base AS (\nSELECT\n orders.status AS `orders___status`\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__customers__spend_sum AS (\nSELECT\n SUM(customers.spend) AS `orders___customers___spend_sum`\nFROM customers AS customers\n)\nSELECT _base.`orders___status`, _cm_orders__customers__spend_sum.`orders___customers___spend_sum`\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_sum", "cm/joined_measure::duckdb": "WITH _base AS (\nSELECT\n orders.status AS \"orders.status\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__customers__spend_sum AS (\nSELECT\n SUM(customers.spend) AS \"orders.customers.spend_sum\"\nFROM customers AS customers\n)\nSELECT _base.\"orders.status\", _cm_orders__customers__spend_sum.\"orders.customers.spend_sum\"\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_sum", "cm/joined_measure::postgres": "WITH _base AS (\nSELECT\n orders.status AS \"orders.status\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__customers__spend_sum AS (\nSELECT\n SUM(customers.spend) AS \"orders.customers.spend_sum\"\nFROM customers AS customers\n)\nSELECT _base.\"orders.status\", _cm_orders__customers__spend_sum.\"orders.customers.spend_sum\"\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_sum", diff --git a/tests/test_dev1474_time_shift_cross_model_partition.py b/tests/test_dev1474_time_shift_cross_model_partition.py index 2032053a..8c7107cf 100644 --- a/tests/test_dev1474_time_shift_cross_model_partition.py +++ b/tests/test_dev1474_time_shift_cross_model_partition.py @@ -474,7 +474,12 @@ async def test_model_filter_mode_a_crossing_join_lifts_guard(self) -> None: assert "LEFT JOIN stores AS stores" in shifted, shifted where = _shifted_where(sql) assert "stores.name" in where, where - assert "IS NOT NULL" in where.upper(), where + # The predicate reaches the shifted CTE as an AST now that Mode-A text + # enters through the one door, so sqlglot emits its canonical negated + # form. The host path always re-parsed and rendered this way; the + # shifted path's raw-text passthrough was the odd one out (DEV-1745 W1). + upper = where.upper() + assert "IS NOT NULL" in upper or "NOT STORES.NAME IS NULL" in upper, where assert_scope_closed(sql) diff --git a/tests/test_filtered_count_forms.py b/tests/test_filtered_count_forms.py index 3b7c5eef..0c8d75e4 100644 --- a/tests/test_filtered_count_forms.py +++ b/tests/test_filtered_count_forms.py @@ -10,6 +10,14 @@ These are regression guards: a future change to the filter wrapper that broke count semantics would silently corrupt every filtered metric. + +``region`` is referenced by the filters but not declared as a model column. +Since every Mode-A fragment enters through the one door, such a reference is +qualified against the scope root (``orders.region``) like any other — the old +path qualified only DECLARED columns and left this one bare, which bound it to +whatever table happened to be in scope once the filter was re-rendered inside a +rerooted CTE. The subject of these guards is the CASE-inside-aggregate shape, +which is unchanged. """ from __future__ import annotations @@ -38,17 +46,17 @@ async def _gen(formula: str) -> str: async def test_filtered_count_uses_case_inside_count() -> None: sql = (await _gen("cust:count")).upper().replace(" ", "") - assert "COUNT(CASEWHENREGION='US'THENORDERS.CUSTOMER_IDEND)" in sql + assert "COUNT(CASEWHENORDERS.REGION='US'THENORDERS.CUSTOMER_IDEND)" in sql async def test_filtered_count_distinct_uses_case_inside_distinct() -> None: sql = (await _gen("cust:count_distinct")).upper().replace(" ", "") - assert "COUNT(DISTINCTCASEWHENREGION='US'THENORDERS.CUSTOMER_IDEND)" in sql + assert "COUNT(DISTINCTCASEWHENORDERS.REGION='US'THENORDERS.CUSTOMER_IDEND)" in sql async def test_filtered_sum_uses_case_inside_sum() -> None: sql = (await _gen("amt:sum")).upper().replace(" ", "") - assert "SUM(CASEWHENREGION='US'THENORDERS.AMOUNTEND)" in sql + assert "SUM(CASEWHENORDERS.REGION='US'THENORDERS.AMOUNTEND)" in sql async def test_filtered_count_distinct_approx_wraps_case_in_exact_fallback() -> None: @@ -59,4 +67,4 @@ async def test_filtered_count_distinct_approx_wraps_case_in_exact_fallback() -> # CASE (like the percentile / stat-agg builders), so the fallback emits # COUNT(DISTINCT (CASE WHEN ... THEN col END)). sql = re.sub(r"\s+", "", (await _gen("cust:count_distinct_approx")).upper()) - assert "COUNT(DISTINCT(CASEWHENREGION='US'THENORDERS.CUSTOMER_IDEND))" in sql + assert "COUNT(DISTINCT(CASEWHENORDERS.REGION='US'THENORDERS.CUSTOMER_IDEND))" in sql diff --git a/tests/test_sql_generator.py b/tests/test_sql_generator.py index 3a202e65..4df63c8a 100644 --- a/tests/test_sql_generator.py +++ b/tests/test_sql_generator.py @@ -5012,9 +5012,11 @@ async def test_filtered_custom_agg_does_not_case_wrap_literal_param( sql = await _generate(generator=gen, query=query, model=agg_model) # The literal `100` must NOT appear inside CASE WHEN; the value # column SHOULD still be CASE-wrapped. - assert "CASE WHEN status = 'active' THEN 100" not in sql + # ``status`` is not a declared column, so the Mode-A door qualifies it + # against the scope root like any other bare ref (DEV-1745 W1). + assert "CASE WHEN sales.status = 'active' THEN 100" not in sql assert "/ 100" in sql - assert "CASE WHEN status = 'active' THEN" in sql + assert "CASE WHEN sales.status = 'active' THEN" in sql async def test_filtered_weighted_avg_still_wraps_column_weight( self, gen: SQLGenerator, agg_model: SlayerModel, @@ -5038,8 +5040,9 @@ async def test_filtered_weighted_avg_still_wraps_column_weight( ], ) sql = await _generate(generator=gen, query=query, model=agg_model) - # Both legs are row-level references → both wrapped. - assert sql.count("CASE WHEN status = 'active'") >= 2 + # Both legs are row-level references → both wrapped. (``status`` is + # undeclared, so the door qualifies it to the root — DEV-1745 W1.) + assert sql.count("CASE WHEN sales.status = 'active'") >= 2 def test_injection_via_direct_agg_render_spec(self, gen: SQLGenerator) -> None: """Malicious agg_kwargs on a directly constructed AggRenderSpec are rejected From 97b0494852a987ce71842fba60241e4b4a8b2bb4 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Wed, 5 Aug 2026 22:05:49 +0200 Subject: [PATCH 18/98] DEV-1744: fix four live grouping bugs in the generator's three arithmetic composers --- DECISIONS.md | 1 + slayer/sql/generator.py | 93 ++++++++++++----------- slayer/sql/render/value_expr.py | 87 +++++++++++++--------- tests/test_dev1744_value_expr.py | 124 +++++++++++++++++++++++++++++++ 4 files changed, 222 insertions(+), 83 deletions(-) diff --git a/DECISIONS.md b/DECISIONS.md index 9e41f3ef..c0f03c35 100644 --- a/DECISIONS.md +++ b/DECISIONS.md @@ -96,4 +96,5 @@ implementation detail. Include issue refs when known. - 2026-08-04 — Dialect-aware / complete escaping for Mode-A `{variable}` substitution (DEV-1727), hardening DEV-1625. `substitute_variables(..., escape="sql")` is now **dialect-aware** and **fail-closed**: it gained a required keyword-only `backslash_escapes` signal (`bool | None`, raises if `None` in sql mode) so a caller rendering raw SQL can never silently under-escape. On backslash-escaping dialects (MySQL/ClickHouse/Snowflake/Redshift/BigQuery/Databricks/Spark) it doubles the backslash before escaping the single quote; on standard dialects it keeps the `''` quote-doubling. The double quote is deliberately left untouched — inside a single-quoted literal `\"` is NOT a recognised escape on 6 of the 7 backslash dialects (only MySQL), so escaping it would corrupt the value. The regime is DERIVED from sqlglot's own tokenizer via `SqlDialect.backslash_escapes_strings` (= `"\\" in tokenizer.STRING_ESCAPES`, guarded + 14-dialect pinned) so our escaping can never drift from the parser that reads the substituted SQL. `escape="python"` (Mode-B) additionally encodes the full C0 control range (`\t`/`\n`/`\r` named, rest `\xNN`) so raw newlines/NUL no longer break `ast.parse`. Engine fail-closed: `_substitute_model_sql_surfaces` / `_render_probe_model` require a `dialect`, threaded from the resolved datasource — no bare bool to forget. Assumes MySQL's default `sql_mode` (backslash escapes on); `NO_BACKSLASH_ESCAPES` servers are a sqlglot-layer-wide limitation, documented not fixed. The SQLite backslash end-to-end gap stays a pinned strict-xfail (pre-existing, out of scope). Bound parameters rejected (don't fit substitute-into-raw-SQL). Nested/join/cross-model lineages remain DEV-1678. - 2026-08-04 — Declared list-valued `{variable}` coercion (DEV-1730 follow-up): a scalar supplied for a variable the model declares `list_valued` is wrapped into a one-element list before Mode-A substitution, so an importer-generated `col IN ({var})` renders `IN ('US')` rather than the unquoted `IN (US)`. The generic scalar rule (author writes the quotes, so `{var}` also works in numeric/fragment positions like `amount >= {floor}` and `{d}::TIMESTAMP`) is CORRECT and unchanged — it just presumes an author who can see the SQL position, which a machine-generated fixed template does not have; the caller cannot supply per-element quotes through parentheses the importer wrote. Silent-wrong-answer risk drove the fix over a raise: `region IN (US)` parses as a column reference, so it fails at the database with a confusing message, or resolves against a real column and returns wrong rows. Opt-in is a front-end-NEUTRAL flag: the Cube converter writes `list_valued: ref.kind == "string"` into each `meta.cube_variables` entry (arrow forms splice pre-quoted scalars and stay `False`), and the engine reads only that flag — never Cube's `kind` taxonomy — so a future list-shaped front-end opts in the same way. Coercion lives at the single Mode-A choke point `_substitute_model_sql_surfaces` (execution and the `_render_probe_model` type-probe both route through it, so it cannot be bypassed) via `coerce_declared_list_variables` / `list_valued_variable_names` in `slayer/core/query.py`. Scope is deliberately narrow: only `str`/`int`/`float`/`bool` are wrapped; `list`/`tuple` pass through (the **empty list still raises** — "no filter" belongs to an optional block or a sentinel default); `None`/`dict` are left for `_render_variable_value` to reject with its own naming error; hand-written models declare nothing and are untouched. Follow-on from the same review: `declares_variables(model)` (any non-empty `meta.cube_variables`) now also defeats the DEV-1625 zero-variable fast path, via the shared `_model_needs_substitution_pass` predicate used by both `_substitute_model_sql_surfaces` and `_render_probe_model`. This closes the fast-path hole for a GENERATED model whose pushdowns are all required (no `{? ?}` block to force the pass): such a model used to emit a bare `{var}` into the SQL on a zero-variable call instead of raising the documented missing-variable error. The hole stays open — deliberately — for hand-written models, which declare nothing and keep the raw-brace-literal protection (`'{1,2,3}'`). The `list_valued` flag is matched with `is True`, not truthiness, since `meta` is user-extensible and a stray `1` or the string `"false"` must not switch substitution semantics. The bag is also SELF-IDENTIFYING — an entry counts only with a string `member` (the shape every importer writes) — so a hand-written `meta` that reuses the `cube_variables` key is not mistaken for generated SQL and silently stripped of its brace-literal protection. - 2026-08-05 — One naming authority + one ValueKey renderer (DEV-1744, PR 1 of the DEV-1742 consolidation). Four consequences worth recording. (1) **Cross-model CTE dedup is now structural.** `_cm_` CTE names were minted by a doubly-lossy helper (`flat_name`, itself documented non-injective, then a non-identifier `re.sub`) and the resulting string doubled as the plan's identity key in `seen_cm`. Two reachable failures followed: two measures whose canonical aliases differ only in case emitted two `_cm_` names that fold together on every case-folding dialect (the collision belt raised — `_wm_` had been retrofitted onto the allocator years earlier, `_cm_` never was), and two genuinely distinct aggregates that sanitised alike made the second skip the loop body, leaving its join-back and column-alias maps unwritten and raising `KeyError` downstream. The dedup key is now the typed `AggregateKey` plus the source relation; the name is allocator-minted and stored once, so the five sites that used to re-derive it now read it. Deliberately NOT keyed on the canonical alias: `canonical_agg_name` omits `column_filter_key`, so a filtered and an unfiltered aggregate over one column share an alias while needing two CTEs — deduping on the string would silently merge them, a wrong answer rather than a crash. (2) **One ScalarCall policy.** Two of the five renderers returned `exp.Anonymous` passthrough, so `ifnull` reached Postgres — which has no `IFNULL` — while the same key emitted `COALESCE` from a projection. All six render paths now call one `render_scalar_call`. Transpiling alone was NOT sufficient: `exp.func("LOG10", x)` normalises to a generic `Log(10, x)` that re-emits as `LOG(10, x)`, wrong on dialects with a native single-arg `LOG10`, so the policy is transpile-then-log-alias-rewrite. Consequence surfaced and approved: `concat` now emits the `||` operator on Postgres/DuckDB where it previously emitted `CONCAT(...)`. That is a semantic change as well as a spelling one — Postgres `CONCAT()` ignores NULL operands, `||` propagates them. Ratified as the kept behavior: the projection path has always emitted `||`, so before this change SLayer disagreed with itself between filters and projections on the same input, and consistency was chosen over preserving the filter-side NULL tolerance. (3) **`ScopeFrame._model_for` raises** instead of falling back to the root model; the fallback turned a wiring bug into a wrong answer by expanding a different model's derived SQL. (4) **Named P-F carve-outs**, recorded rather than omitted: the T-SQL ORDER-BY-detach rewrite and the stage-schema wrapper take `_outer` / `_stage_inner` as shared CONSTANTS rather than allocator-minted names (the former is a post-generation AST pass with no allocator in reach, and a later PR rebuilds that machinery); `base` / `_base` / `_combined` keep their literal names but are reserved into the allocator. Superseded code is retained and callable per the chain's operating rule — deletion happens in the final PR. +- 2026-08-05 — Operand grouping joins ScalarCall as a policy shared ahead of the call-site migration (DEV-1744). A carve-out from the deferral below, taken because the generator's three arithmetic composers were provably emitting wrong SQL, not merely duplicated SQL. All three built `exp.Not` / `exp.Neg` / `exp.Is` around a bare operand and hand-folded `and`/`or` into left-nested nodes; sqlglot does not parenthesise by nesting, so four shapes emitted predicates that parse cleanly and return a different row set: `not (a AND b)` → `NOT a AND b` (De Morgan), `-(a + b)` → `-a + b`, `(a = 5) IS NULL` → `a = 5 IS NULL` (read as `a = (5 IS NULL)`, because `IS` binds tighter than `=`), and `a AND (b OR c)` → `a AND b OR c`. The composers now call `group_unary_operand` / `group_is_operands` from the renderer and fold booleans with `exp.and_` / `exp.or_`. Emission is otherwise unchanged — the whole suite passed without a single expectation edit, which is also why the bugs survived this long. Related: the renderer's own precedence table now treats the comparison level as NON-associative, parenthesising an equal-precedence LEFT child there (arithmetic keeps left-associative flattening), and a bare `*` is refused as the source of any aggregation but `count`, which previously built `SUM(*)` / `COUNT(DISTINCT *)`. - 2026-08-05 — Renderer call-site migration deferred to the scope-assembly PR (DEV-1744 → DEV-1746). PR 1 lands the complete, tested `render_value_key` API but does NOT reroute the generator's own render paths through it; only the ScalarCall policy is genuinely shared (all six paths call `render_scalar_call`). Deferred because the filter and composite paths carry state the context does not yet consume — the local-aggregate HAVING branch with its slot lookup and inline-aggregate emission, the first/last ranked state, the filter-side CAST policy, and the rn-suffix / filtered-rank / composite-alias / resolved-kwarg maps — and because the cross-scope paths are the ones that should supply `resolve(consumer=...)` its first production caller, which is scope-assembly work by nature. Splitting the reroute across two PRs would move that state twice. Full detail, per call site, in the `slayer/sql/render/value_expr.py` module docstring. diff --git a/slayer/sql/generator.py b/slayer/sql/generator.py index 036f367a..29c7fcc9 100644 --- a/slayer/sql/generator.py +++ b/slayer/sql/generator.py @@ -56,7 +56,12 @@ result_key_from_alias, ) from slayer.sql.render.aggregates import window_agg_class -from slayer.sql.render.value_expr import render_scalar_call, rewrite_log_alias +from slayer.sql.render.value_expr import ( + group_is_operands, + group_unary_operand, + render_scalar_call, + rewrite_log_alias, +) from slayer.sql.reserved_keywords import prequote_reserved_identifiers from slayer.sql.scope import ScopeFrame from slayer.sql.scope_check import maybe_validate_scopes @@ -6476,29 +6481,27 @@ def _build_arith_or_cmp_ast( arithmetic (``+``, ``-``, ``*``, ``/``). """ if op == "not": - return exp.Not(this=operands[0]) + return exp.Not(this=group_unary_operand(operands[0], op="not")) # ``and`` / ``or`` (Codex round 2): the binder produces n-ary # boolean ``ArithmeticKey`` for ``a AND b AND c`` (three operands); # the prior implementation took only ``operands[0]`` / ``[1]`` and # silently dropped the third predicate from cross-model HAVING/ - # WHERE, broadening results. Fold over every operand the same - # way ``_compose_arithmetic_op`` and ``_build_arithmetic_for_filter`` - # already do. + # WHERE, broadening results. + # + # ``exp.and_`` / ``exp.or_`` rather than a hand-rolled fold: the fold + # built ``And(a, Or(b, c))``, which sqlglot emits FLAT as + # ``a AND b OR c`` — read back as ``(a AND b) OR c``, a broader row set. if op in ("and", "or"): - node_cls = exp.And if op == "and" else exp.Or - acc = operands[0] - for o in operands[1:]: - acc = node_cls(this=acc, expression=o) - return acc + return exp.and_(*operands) if op == "and" else exp.or_(*operands) left, right = operands[0], operands[1] # ``IS`` / ``IS NOT`` (Codex review): the typed pipeline's filter # normalizer lowers SQL ``IS NULL`` / ``IS NOT NULL`` to Python # ``is None`` / ``is not None``. Render against a ``Null`` literal # as the standard SQL forms. - if op == "is": - return exp.Is(this=left, expression=right) - if op == "is not": - return exp.Not(this=exp.Is(this=left, expression=right)) + if op in ("is", "is not"): + left, right = group_is_operands(lhs=left, rhs=right) + node = exp.Is(this=left, expression=right) + return exp.Not(this=node) if op == "is not" else node op_map = { "==": exp.EQ, "!=": exp.NEQ, @@ -7266,17 +7269,21 @@ def _compose_arithmetic_op( sqlglot binary nodes. """ if len(operands) == 1: - if op == "not": - return exp.Not(this=operands[0]) - if op == "-": - return exp.Neg(this=operands[0]) + # Grouped through the shared policy: a bare ``exp.Neg`` / ``exp.Not`` + # emitted ``-(a + b)`` as ``-a + b`` and ``not (a and b)`` as + # ``NOT a AND b`` — both parse, both mean something else. + if op in ("not", "-"): + grouped = group_unary_operand(operands[0], op=op) + return ( + exp.Not(this=grouped) if op == "not" else exp.Neg(this=grouped) + ) if len(operands) == 2: lhs, rhs = operands # ``IS`` / ``IS NOT`` (Codex review): see ``_build_arith_or_cmp_ast``. - if op == "is": - return exp.Is(this=lhs, expression=rhs) - if op == "is not": - return exp.Not(this=exp.Is(this=lhs, expression=rhs)) + if op in ("is", "is not"): + lhs, rhs = group_is_operands(lhs=lhs, rhs=rhs) + node = exp.Is(this=lhs, expression=rhs) + return exp.Not(this=node) if op == "is not" else node binary = { "+": exp.Add, "-": exp.Sub, "*": exp.Mul, "/": exp.Div, "<": exp.LT, "<=": exp.LTE, ">": exp.GT, ">=": exp.GTE, @@ -7300,16 +7307,12 @@ def _compose_arithmetic_op( rhs, parent_prec=parent_prec, is_right=True, op=op, ) return binary[op](this=lhs, expression=rhs) - if op == "and": - return exp.And(this=lhs, expression=rhs) - if op == "or": - return exp.Or(this=lhs, expression=rhs) if len(operands) >= 2 and op in ("and", "or"): - node_cls = exp.And if op == "and" else exp.Or - acc = operands[0] - for rhs in operands[1:]: - acc = node_cls(this=acc, expression=rhs) - return acc + # ``exp.and_`` / ``exp.or_`` rather than a hand-rolled fold: the + # fold built ``And(a, Or(b, c))``, which sqlglot emits FLAT as + # ``a AND b OR c`` — read back as ``(a AND b) OR c``, a broader + # row set. + return exp.and_(*operands) if op == "and" else exp.or_(*operands) raise NotImplementedError( f"DEV-1450 stage 7b.10: arithmetic op {op!r} arity " f"{len(operands)} not supported in POST-filter rendering.", @@ -9840,7 +9843,7 @@ def _build_arithmetic_for_filter( # NOSONAR(S3776) — sequential per-operator # single-operand form so a filter like ``amount > -10`` # doesn't crash with IndexError. if len(operands) == 1: - return exp.Neg(this=operands[0]) + return exp.Neg(this=group_unary_operand(operands[0], op="-")) return exp.Sub( this=SQLGenerator._paren_if_lower_prec( operands[0], parent_prec=1, is_right=False, op="-", @@ -9867,18 +9870,14 @@ def _build_arithmetic_for_filter( # NOSONAR(S3776) — sequential per-operator operands[1], parent_prec=2, is_right=True, op="/", ), ) - if op == "and": - result = operands[0] - for o in operands[1:]: - result = exp.And(this=result, expression=o) - return result - if op == "or": - result = operands[0] - for o in operands[1:]: - result = exp.Or(this=result, expression=o) - return result + if op in ("and", "or"): + # ``exp.and_`` / ``exp.or_`` rather than a hand-rolled fold: the + # fold built ``And(a, Or(b, c))``, which sqlglot emits FLAT as + # ``a AND b OR c`` — read back as ``(a AND b) OR c``, a broader + # row set. + return exp.and_(*operands) if op == "and" else exp.or_(*operands) if op == "not": - return exp.Not(this=operands[0]) + return exp.Not(this=group_unary_operand(operands[0], op="not")) # ``IS`` / ``IS NOT`` (Codex round 2): the filter normalizer lowers # SQL ``IS NULL`` / ``IS NOT NULL`` to Python ``is None`` / ``is # not None``. Render against the rhs (a ``Null`` literal) as the @@ -9886,10 +9885,10 @@ def _build_arithmetic_for_filter( # NOSONAR(S3776) — sequential per-operator # ``deleted_at IS NULL`` parses and binds but raises here at SQL # generation. Mirrors the patches in ``_build_arith_or_cmp_ast`` # and ``_compose_arithmetic_op``. - if op == "is": - return exp.Is(this=operands[0], expression=operands[1]) - if op == "is not": - return exp.Not(this=exp.Is(this=operands[0], expression=operands[1])) + if op in ("is", "is not"): + lhs, rhs = group_is_operands(lhs=operands[0], rhs=operands[1]) + node = exp.Is(this=lhs, expression=rhs) + return exp.Not(this=node) if op == "is not" else node raise NotImplementedError( f"DEV-1450 stage 7b.8: ArithmeticKey op {op!r} not " f"supported in filter rendering." diff --git a/slayer/sql/render/value_expr.py b/slayer/sql/render/value_expr.py index 1d13bacf..a734cbf7 100644 --- a/slayer/sql/render/value_expr.py +++ b/slayer/sql/render/value_expr.py @@ -13,10 +13,15 @@ Migration status ---------------- The API here is complete and directly tested, but the generator's own render -paths do NOT yet route through :func:`render_value_key`. Only the ScalarCall -POLICY is shared today: all six paths call :func:`render_scalar_call`, so that -construct genuinely renders once. Everything else still runs the generator's -own per-path branches. +paths do NOT yet route through :func:`render_value_key`. Two POLICIES are +shared today, each because the generator's copies were demonstrably wrong +without them: + +* ScalarCall — all six paths call :func:`render_scalar_call`. +* Operand grouping — the generator's three arithmetic composers call + :func:`group_unary_operand` and :func:`group_is_operands`. + +Everything else still runs the generator's own per-path branches. Deferred to the scope-assembly PR, together with the cross-scope migration, because the two are the same piece of work. Finishing it needs: @@ -222,7 +227,7 @@ def _literal(value: Any) -> exp.Expression: def _paren_if_lower_prec( - child: exp.Expression, *, parent_prec: int, is_right: bool, op: str, + child: exp.Expression, *, parent_prec: int, is_right: bool, ) -> exp.Expression: """Parenthesise ``child`` when dropping its parens would change meaning. @@ -250,6 +255,41 @@ def _paren_if_lower_prec( return child +def group_unary_operand(operand: exp.Expression, *, op: str) -> exp.Expression: + """Parenthesise a unary operator's operand when precedence requires it. + + Public because the generator's three arithmetic composers built ``exp.Not`` + / ``exp.Neg`` around a bare operand: ``-(a + b)`` emitted ``-a + b`` and + ``not (a and b)`` emitted ``NOT a AND b``. Both parse cleanly and mean + something else, so the policy is shared rather than re-derived (P-G). + """ + if op not in ("not", "-"): + raise NotImplementedError( + f"group_unary_operand only covers 'not' and '-', got {op!r}.", + ) + parent = exp.Not if op == "not" else exp.Neg + return _paren_if_lower_prec( + operand, parent_prec=_PRECEDENCE[parent], is_right=False, + ) + + +def group_is_operands( + *, lhs: exp.Expression, rhs: exp.Expression, +) -> Tuple[exp.Expression, exp.Expression]: + """Parenthesise an ``IS`` / ``IS NOT`` operand when precedence requires it. + + ``IS`` binds tighter than ``=``, so an ungrouped ``(a = 5) is null`` emits + ``a = 5 IS NULL`` and every dialect reads it as ``a = (5 IS NULL)`` — a + different predicate that still returns rows. Shared with the generator's + composers for the same reason as :func:`group_unary_operand`. + """ + is_prec = _PRECEDENCE[exp.Is] + return ( + _paren_if_lower_prec(lhs, parent_prec=is_prec, is_right=False), + _paren_if_lower_prec(rhs, parent_prec=is_prec, is_right=True), + ) + + def _render_arithmetic( op: str, operands: List[exp.Expression], ) -> exp.Expression: @@ -271,24 +311,9 @@ def _render_arithmetic( # Unary operands need the same precedence treatment as binary ones. # Without it ``-(a + b)`` emits ``-a + b`` and ``not (a and b)`` emits # ``NOT a AND b`` — both parse cleanly and both mean something else. - if op == "not": - return exp.Not( - this=_paren_if_lower_prec( - operands[0], - parent_prec=_PRECEDENCE[exp.Not], - is_right=False, - op=op, - ), - ) - if op == "-": - return exp.Neg( - this=_paren_if_lower_prec( - operands[0], - parent_prec=_PRECEDENCE[exp.Neg], - is_right=False, - op=op, - ), - ) + if op in ("not", "-"): + grouped = group_unary_operand(operands[0], op=op) + return exp.Not(this=grouped) if op == "not" else exp.Neg(this=grouped) if op == "+": return operands[0] raise NotImplementedError( @@ -309,17 +334,7 @@ def _render_arithmetic( ) if op in ("is", "is not"): - # Same precedence pass as every other comparison. ``IS`` binds tighter - # than ``=``, so an unparenthesised ``(a = b) is null`` would emit - # ``a = b IS NULL`` and be read as ``a = (b IS NULL)`` — a different - # predicate that still runs. - is_prec = _PRECEDENCE[exp.Is] - lhs = _paren_if_lower_prec( - operands[0], parent_prec=is_prec, is_right=False, op=op, - ) - rhs = _paren_if_lower_prec( - operands[1], parent_prec=is_prec, is_right=True, op=op, - ) + lhs, rhs = group_is_operands(lhs=operands[0], rhs=operands[1]) node = exp.Is(this=lhs, expression=rhs) return exp.Not(this=node) if op == "is not" else node @@ -333,10 +348,10 @@ def _render_arithmetic( lhs, rhs = result, operand if parent_prec is not None: lhs = _paren_if_lower_prec( - lhs, parent_prec=parent_prec, is_right=False, op=op, + lhs, parent_prec=parent_prec, is_right=False, ) rhs = _paren_if_lower_prec( - rhs, parent_prec=parent_prec, is_right=True, op=op, + rhs, parent_prec=parent_prec, is_right=True, ) result = node_cls(this=lhs, expression=rhs) return result diff --git a/tests/test_dev1744_value_expr.py b/tests/test_dev1744_value_expr.py index afeb3d82..bd4ac689 100644 --- a/tests/test_dev1744_value_expr.py +++ b/tests/test_dev1744_value_expr.py @@ -86,6 +86,7 @@ from slayer.engine.query_engine import SlayerQueryEngine from slayer.engine.source_bundle import ResolvedSourceBundle from slayer.sql.dialects import get_dialect +from slayer.sql.generator import SQLGenerator from slayer.sql.naming import AliasAllocator from slayer.sql.render.aggregates import resolve_agg_entry, window_agg_class from slayer.sql.render.value_expr import ( @@ -2149,3 +2150,126 @@ def test_non_count_star_is_refused(self, agg) -> None: def test_count_star_still_renders(self) -> None: key = AggregateKey(source=StarKey(), agg="count") assert _sql(render_value_key(key, _composite_ctx())) == "COUNT(*)" + + +class TestGeneratorComposersShareTheGroupingPolicy: + """The three LIVE generator composers had the same grouping holes. + + ``value_expr`` is not yet on the generator's arithmetic path (that reroute + is the scope-assembly PR), so finding these there did not fix them here. + Each composer built ``exp.Not`` / ``exp.Neg`` / ``exp.Is`` around a bare + operand and hand-folded ``and``/``or``, producing SQL that parses cleanly + and returns a different row set: + + * ``not (a AND b)`` -> ``NOT a AND b`` = ``(NOT a) AND b`` + * ``-(a + b)`` -> ``-a + b`` = ``(-a) + b`` + * ``(a = 5) IS NULL`` -> ``a = 5 IS NULL`` = ``a = (5 IS NULL)`` + * ``a AND (b OR c)`` -> ``a AND b OR c`` = ``(a AND b) OR c`` + + All four now route through the shared policy in ``value_expr``, so the + construct groups the same way wherever it is composed (P-G). + """ + + @staticmethod + def _a(): + return exp.column("a", table="t") + + def _gt(self): + return exp.GT(this=self._a(), expression=exp.Literal.number("1")) + + def _lt(self): + return exp.LT(this=self._a(), expression=exp.Literal.number("9")) + + def _eq(self): + return exp.EQ(this=self._a(), expression=exp.Literal.number("5")) + + def _add(self): + return exp.Add(this=self._a(), expression=exp.column("b", table="t")) + + def _composers(self): + """The three live composers, as uniform ``(op, operands) -> node``.""" + gen = SQLGenerator.__new__(SQLGenerator) + return { + "build_arithmetic_for_filter": ( + lambda op, ops: SQLGenerator._build_arithmetic_for_filter( + op=op, operands=ops, + ) + ), + "compose_arithmetic_op": ( + lambda op, ops: SQLGenerator._compose_arithmetic_op( + op=op, operands=ops, + ) + ), + "build_arith_or_cmp_ast": ( + lambda op, ops: gen._build_arith_or_cmp_ast(op=op, operands=ops) + ), + } + + @pytest.mark.parametrize( + "composer", + ["build_arithmetic_for_filter", "compose_arithmetic_op", + "build_arith_or_cmp_ast"], + ) + def test_not_of_a_conjunction_keeps_its_parens(self, composer) -> None: + compose = self._composers()[composer] + conj = exp.And(this=self._gt(), expression=self._lt()) + assert compose("not", [conj]).sql() == "NOT (t.a > 1 AND t.a < 9)" + + @pytest.mark.parametrize( + "composer", + ["build_arithmetic_for_filter", "compose_arithmetic_op", + "build_arith_or_cmp_ast"], + ) + def test_is_null_over_a_comparison_keeps_its_parens(self, composer) -> None: + compose = self._composers()[composer] + out = compose("is", [self._eq(), exp.Null()]).sql() + assert out == "(t.a = 5) IS NULL", out + + @pytest.mark.parametrize( + "composer", + ["build_arithmetic_for_filter", "compose_arithmetic_op", + "build_arith_or_cmp_ast"], + ) + def test_is_not_null_over_a_comparison_keeps_its_parens(self, composer) -> None: + compose = self._composers()[composer] + out = compose("is not", [self._eq(), exp.Null()]).sql() + assert out == "NOT (t.a = 5) IS NULL", out + + @pytest.mark.parametrize( + "composer", + ["build_arithmetic_for_filter", "compose_arithmetic_op", + "build_arith_or_cmp_ast"], + ) + def test_disjunction_inside_a_conjunction_keeps_its_parens( + self, composer, + ) -> None: + compose = self._composers()[composer] + disj = exp.Or(this=self._gt(), expression=self._lt()) + out = compose("and", [self._lt(), disj]).sql() + assert out == "t.a < 9 AND (t.a > 1 OR t.a < 9)", out + + @pytest.mark.parametrize( + "composer", + ["build_arithmetic_for_filter", "compose_arithmetic_op"], + ) + def test_negated_sum_keeps_its_parens(self, composer) -> None: + """``_build_arith_or_cmp_ast`` has no unary-minus branch, so only the + two composers that do are exercised here.""" + compose = self._composers()[composer] + assert compose("-", [self._add()]).sql() == "-(t.a + t.b)" + + @pytest.mark.parametrize( + "composer", + ["build_arithmetic_for_filter", "compose_arithmetic_op", + "build_arith_or_cmp_ast"], + ) + def test_plain_shapes_gain_no_parens(self, composer) -> None: + """Don't over-wrap — the fix must not churn the emission of the + shapes that were already right.""" + compose = self._composers()[composer] + assert compose("is", [self._a(), exp.Null()]).sql() == "t.a IS NULL" + assert compose("not", [self._gt()]).sql() == "NOT t.a > 1" + assert ( + compose("and", [self._gt(), self._lt()]).sql() + == "t.a > 1 AND t.a < 9" + ) From a305e6ba0589ba11203dee98762659576df86c21 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Wed, 5 Aug 2026 22:11:14 +0200 Subject: [PATCH 19/98] DEV-1745: W6 date_range warning, W3 plan-time outer-WHERE routing, D6 warning types MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit D6 — one discriminated warning family. SlayerWarning gains a `kind` discriminator; NormalizationWarning becomes a subclass; DroppedFilterWarning joins it. SlayerResponse.warnings can now carry more than one kind, so a consumer switches on `kind` rather than assuming every element has a rule_id. W6 — MALFORMED_DATE_RANGE. normalize_query never inspected time_dimensions; it now warns when a date_range is present but is not the two-element form the planner requires. The trigger is the planner's own drop condition, so the warning fires if and only if the range is actually ignored ([], one, or 3+). The ratified silent no-op is unchanged: this reports, it does not rewrite — there is no unambiguous canonical form to rewrite a malformed range TO. W3 — outer-WHERE routing moves to the planner (P-D). PlannedQuery declares outer_where_filter_ids, computed by _plan_outer_where_filters where the cross-model plans (and so cte_root_model) are already known. The generator's render-time re-walk of filters_by_phase is deleted; it now reads the field. One test assertion corrected while proving this. The authority test demanded "> 100" disappear from the whole query once the field is cleared, but clearing the routing does not delete the user's filter — it returns it to the default HAVING placement. Demanding it vanish entirely would demand that a filter be silently dropped. It now asserts on the outer WHERE specifically, which is the shape the routing exists to produce and one a re-walking generator would still emit. Co-Authored-By: Claude Opus 5 (1M context) --- slayer/core/warnings.py | 31 +++++++++++++++- slayer/engine/normalization.py | 38 +++++++++++++++++++ slayer/engine/planned.py | 16 ++++++++ slayer/engine/stage_planner.py | 49 +++++++++++++++++++++++++ slayer/sql/generator.py | 44 +++++++--------------- tests/test_dev1745_plan_time_routing.py | 18 +++++++-- 6 files changed, 160 insertions(+), 36 deletions(-) diff --git a/slayer/core/warnings.py b/slayer/core/warnings.py index 0f5a1659..eba71159 100644 --- a/slayer/core/warnings.py +++ b/slayer/core/warnings.py @@ -17,12 +17,24 @@ from __future__ import annotations -from typing import Optional +from typing import Literal, Optional from pydantic import BaseModel -class NormalizationWarning(BaseModel): +class SlayerWarning(BaseModel): + """Base of the warning family carried on ``SlayerResponse.warnings``. + + ``SlayerResponse.warnings`` used to be normalization-only, so consumers + could assume every element had a ``rule_id``. It now carries more than one + kind of advisory, so every payload declares a ``kind`` discriminator and a + consumer switches on it rather than on the presence of a field. + """ + + kind: str + + +class NormalizationWarning(SlayerWarning): """Structured payload describing one slack-normalization rewrite. ``rule_id`` identifies the rule that fired (``FUNC_STYLE_AGG``, @@ -32,6 +44,7 @@ class NormalizationWarning(BaseModel): into ``docs/agent_input_slack.md``. """ + kind: Literal["normalization"] = "normalization" rule_id: str original: str normalized: str @@ -39,6 +52,20 @@ class NormalizationWarning(BaseModel): rule_doc_url: Optional[str] = None +class DroppedFilterWarning(SlayerWarning): + """A user filter that could not be applied where it was routed. + + Carries the filter's ORIGINAL author text (not the normalized, prequoted + or re-rendered form — the author has to recognise it), the surface it came + from, and why it was dropped. + """ + + kind: Literal["unreachable_filter_dropped"] = "unreachable_filter_dropped" + filter_text: str + location: str + reason: str + + class SlayerNormalizationWarning(UserWarning): """Carrier ``UserWarning`` for a ``NormalizationWarning`` payload. diff --git a/slayer/engine/normalization.py b/slayer/engine/normalization.py index f31b1ea4..b167a51f 100644 --- a/slayer/engine/normalization.py +++ b/slayer/engine/normalization.py @@ -565,9 +565,47 @@ def normalize_query( # the model. Wiring is preserved so future activations need no # plumbing changes. + # Rule 4: MALFORMED_DATE_RANGE. + all_warnings.extend(_apply_malformed_date_range(query)) + return NormalizationResult(query=query, warnings=all_warnings) +def _apply_malformed_date_range( + query: SlayerQuery, +) -> List[NormalizationWarning]: + """Warn when a ``time_dimensions[i].date_range`` is present but is not the + two-element ``[start, end]`` the planner requires. + + The planner's silent ``continue`` on such a range is deliberate and stays + exactly as it is — this rule changes NO behaviour, it only stops the drop + from being invisible. The trigger is the planner's own drop condition + (``date_range is not None and len(date_range) != 2``), so the warning fires + if and only if the range is actually ignored: ``[]``, one element, or three + or more. An absent ``date_range`` is legitimately optional and never warns. + + Reports, but does not rewrite: there is no unambiguous canonical form to + rewrite a malformed range TO, and inventing one would change results. + """ + emitted: List[NormalizationWarning] = [] + for i, td in enumerate(query.time_dimensions or []): + date_range = getattr(td, "date_range", None) + if date_range is None or len(date_range) == 2: + continue + payload = NormalizationWarning( + rule_id="MALFORMED_DATE_RANGE", + original=f"time_dimensions[{i}].date_range={list(date_range)!r}", + normalized="(ignored — no date filter emitted)", + location=f"time_dimensions[{i}].date_range", + rule_doc_url="docs/agent_input_slack.md#malformed-date-range", + ) + emitted.append(payload) + _warnings_module.warn( + SlayerNormalizationWarning(payload), stacklevel=2, + ) + return emitted + + def _normalize_model_measures( model: SlayerModel, *, custom_agg_names: Optional[frozenset[str]], ) -> Tuple[SlayerModel, List[NormalizationWarning]]: diff --git a/slayer/engine/planned.py b/slayer/engine/planned.py index a55b48a6..deeaed33 100644 --- a/slayer/engine/planned.py +++ b/slayer/engine/planned.py @@ -454,6 +454,22 @@ class PlannedQuery(BaseModel): # equality-joined into ``_src`` (``_build_windowed_plans`` skips them), so # stripping a bound on one would leave that axis unconstrained. frame_bound_columns: List[ValueKey] = Field(default_factory=list) + # DEV-1745 (W3 / P-D) — ids of the AGGREGATE-phase filters that must be + # applied as a plain WHERE on the OUTER combined SELECT rather than as + # HAVING inside a ``_cm_*`` CTE (DEV-1503). + # + # A filtered-local ISOLATED aggregate lives in a CTE that LEFT JOINs back + # to ``_base``. Applying the comparison as HAVING inside that CTE drops CTE + # rows, but the LEFT JOIN then resurfaces the host row with a NULL + # aggregate — the wrong semantic. On the outer, non-aggregating SELECT the + # same comparison drops the row. + # + # Decided HERE because it is a routing decision, not an emission detail: + # the generator used to re-walk ``filters_by_phase`` at render time to + # rediscover it, which is policy chosen during emission. The generator now + # reads this field and never re-derives it, so clearing the field removes + # the outer WHERE. + outer_where_filter_ids: List[BoundFilterId] = Field(default_factory=list) # ``CrossModelAggregatePlan.rerooted_plan`` is a forward reference to diff --git a/slayer/engine/stage_planner.py b/slayer/engine/stage_planner.py index 26de3262..2a4acef7 100644 --- a/slayer/engine/stage_planner.py +++ b/slayer/engine/stage_planner.py @@ -83,6 +83,7 @@ from slayer.engine.response_meta import _infer_aggregated_format from slayer.engine.planned import ( BoundExpr as PlannedBoundExpr, + BoundFilterId, FilterPhase, OrderEntry, PlannedQuery, @@ -1480,6 +1481,16 @@ def _windowed_phase(bf: BoundFilter) -> Phase: wp.where_filter_ids = src_where_ids wp.src_filter_rewrites = src_rewrites + # DEV-1745 (W3 / P-D) — decide the outer-WHERE routing HERE, where the + # cross-model plans (and so ``cte_root_model``) are already known. The + # generator used to rediscover this by re-walking the filters at render + # time; it now consumes the field. + outer_where_filter_ids = _plan_outer_where_filters( + filters_by_phase=filters_by_phase, + cross_model_plans=cross_model_plans, + slots=[*row_slots, *agg_slots, *combined_slots], + ) + return PlannedQuery( source_relation=source_relation, row_slots=row_slots, @@ -1498,9 +1509,47 @@ def _windowed_phase(bf: BoundFilter) -> Phase: render_source_model=render_source_model, distinct_dimension_values=query.distinct_dimension_values, frame_bound_columns=frame_bound_columns, + outer_where_filter_ids=outer_where_filter_ids, ) +def _plan_outer_where_filters( + *, filters_by_phase: list, cross_model_plans: list, slots: list, +) -> List[BoundFilterId]: + """AGGREGATE-phase filters that must be applied on the OUTER combined + SELECT instead of as HAVING inside a ``_cm_*`` CTE (DEV-1503). + + A filter qualifies when its value-key tree references an aggregate that was + isolated into a CTE with its own root (``cte_root_model is not None``). + That CTE LEFT JOINs back to ``_base``, so a HAVING inside it drops CTE rows + and the join then resurfaces the host row carrying a NULL aggregate. The + same predicate on the outer, non-aggregating SELECT drops the row. + + Returned in ``filters_by_phase`` order so the emitted WHERE conjunct order + is stable. + """ + isolated_agg_slot_ids = { + p.aggregate_slot_id + for p in cross_model_plans + if p.cte_root_model is not None + } + if not isolated_agg_slot_ids: + return [] + slot_by_key = {s.key: s for s in slots} + routed: List[BoundFilterId] = [] + for fp in filters_by_phase: + if fp.phase != Phase.AGGREGATE or fp.expression is None: + continue + for k in walk_value_keys(fp.expression.value_key): + if not isinstance(k, AggregateKey): + continue + slot = slot_by_key.get(k) + if slot is not None and slot.id in isolated_agg_slot_ids: + routed.append(fp.id) + break + return routed + + def _frame_bound_columns(*, row_slots: list) -> List[ValueKey]: """Raw column keys of the stage's NON-HIDDEN time dimensions (DEV-1732). diff --git a/slayer/sql/generator.py b/slayer/sql/generator.py index 757ab211..d43fe36c 100644 --- a/slayer/sql/generator.py +++ b/slayer/sql/generator.py @@ -4204,38 +4204,20 @@ def _render_with_cross_model_plans( # NOSONAR(S3776) — orchestration of host p.aggregate_slot_id for p in planned_query.windowed_aggregate_plans } - # DEV-1503 — outer combined-SELECT WHERE wrapper. Identify - # AGGREGATE-phase host filters whose value-key references any - # FILTERED-LOCAL ISOLATED aggregate (a plan with - # ``cte_root_model is not None``). The filtered aggregate lives in - # its ``_cm_*`` CTE that LEFT JOINs back to ``_base``; applying - # the comparison as HAVING in the CTE would drop CTE rows where - # the aggregate fails the test, but the LEFT JOIN would then - # surface the host row with a NULL aggregate (wrong semantic). - # Routing to the outer combined SELECT (non-aggregating) as - # plain WHERE on the joined-back column drops the row instead. - isolated_agg_slot_ids = { - p.aggregate_slot_id - for p in planned_query.cross_model_aggregate_plans - if p.cte_root_model is not None - } + # DEV-1503 / DEV-1745 (P-D) — the outer combined-SELECT WHERE wrapper is + # routed by the PLANNER (``_plan_outer_where_filters``), which knows + # which aggregates were isolated into a CTE with its own root. The + # generator consumes that decision verbatim: re-walking + # ``filters_by_phase`` here to rediscover it would be routing policy + # chosen during emission, and the two could disagree. slot_by_key = {s.key: s for s in slots_by_id.values()} - outer_where_filter_ids: Set[str] = set() - outer_where_filters: List = [] - if isolated_agg_slot_ids: - for fp in planned_query.filters_by_phase: - if fp.phase != Phase.AGGREGATE or fp.expression is None: - continue - refs_isolated = False - for k in walk_value_keys(fp.expression.value_key): - if isinstance(k, AggregateKey): - slot = slot_by_key.get(k) - if slot is not None and slot.id in isolated_agg_slot_ids: - refs_isolated = True - break - if refs_isolated: - outer_where_filter_ids.add(fp.id) - outer_where_filters.append(fp) + outer_where_filter_ids: Set[str] = set( + planned_query.outer_where_filter_ids, + ) + outer_where_filters: List = [ + fp for fp in planned_query.filters_by_phase + if fp.id in outer_where_filter_ids + ] # DEV-1503 (Codex round 2 #1) — composite projection slots whose # value-key tree walks an ISOLATED cross-model aggregate must NOT # render in ``_base``. Inline rendering pulls the filter-target diff --git a/tests/test_dev1745_plan_time_routing.py b/tests/test_dev1745_plan_time_routing.py index df5083a1..f7024430 100644 --- a/tests/test_dev1745_plan_time_routing.py +++ b/tests/test_dev1745_plan_time_routing.py @@ -131,14 +131,26 @@ async def _sql(self, query: SlayerQuery) -> str: validate=False, extra_models=[_customers()], ) + # The predicate applied to the JOINED-BACK ``_cm_`` column on the outer, + # non-aggregating SELECT — the shape this routing exists to produce, and + # one nothing else in the query emits. + OUTER_WHERE = 'WHERE _cm_orders__eu_amount_sum."orders.eu" > 100' + async def test_outer_where_is_emitted_for_the_isolated_shape(self) -> None: sql = await self._sql(_outer_where_query()) - assert "> 100" in sql, sql + assert self.OUTER_WHERE in sql, sql async def test_clearing_the_plan_field_removes_the_outer_where(self) -> None: """P-D: the plan is authoritative. A generator that re-walks the filters at render time would ignore the cleared field and keep - emitting the predicate.""" + emitting the outer WHERE. + + Asserted on the outer WHERE specifically rather than on the predicate + text appearing anywhere: clearing the routing does not delete the + user's filter, it returns it to the default HAVING placement. Demanding + that ``> 100`` vanish from the whole query would be demanding that a + filter be silently dropped. + """ from slayer.sql.generator import SQLGenerator planned = plan_query(query=_outer_where_query(), bundle=_bundle()) @@ -149,7 +161,7 @@ async def test_clearing_the_plan_field_removes_the_outer_where(self) -> None: cleared = planned.model_copy(update={"outer_where_filter_ids": []}) gen = SQLGenerator(dialect="postgres") sql = gen.generate_from_planned(planned_query=cleared, bundle=_bundle()) - assert "> 100" not in sql, ( + assert self.OUTER_WHERE not in sql, ( "the generator re-derived the outer-WHERE routing instead of " f"consuming the plan:\n{sql}" ) From 8c2b18ec5ff87b66749a2d26001801bbf010109a Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Wed, 5 Aug 2026 22:14:27 +0200 Subject: [PATCH 20/98] =?UTF-8?q?DEV-1744:=20one=20arithmetic=20composer?= =?UTF-8?q?=20=E2=80=94=20the=20generator's=20three=20each=20mis-grouped?= =?UTF-8?q?=20a=20different=20set=20of=20shapes?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- DECISIONS.md | 2 +- slayer/sql/generator.py | 192 +++++-------------------------- slayer/sql/render/value_expr.py | 29 +++-- tests/test_dev1744_value_expr.py | 56 ++++++++- 4 files changed, 100 insertions(+), 179 deletions(-) diff --git a/DECISIONS.md b/DECISIONS.md index c0f03c35..52863700 100644 --- a/DECISIONS.md +++ b/DECISIONS.md @@ -96,5 +96,5 @@ implementation detail. Include issue refs when known. - 2026-08-04 — Dialect-aware / complete escaping for Mode-A `{variable}` substitution (DEV-1727), hardening DEV-1625. `substitute_variables(..., escape="sql")` is now **dialect-aware** and **fail-closed**: it gained a required keyword-only `backslash_escapes` signal (`bool | None`, raises if `None` in sql mode) so a caller rendering raw SQL can never silently under-escape. On backslash-escaping dialects (MySQL/ClickHouse/Snowflake/Redshift/BigQuery/Databricks/Spark) it doubles the backslash before escaping the single quote; on standard dialects it keeps the `''` quote-doubling. The double quote is deliberately left untouched — inside a single-quoted literal `\"` is NOT a recognised escape on 6 of the 7 backslash dialects (only MySQL), so escaping it would corrupt the value. The regime is DERIVED from sqlglot's own tokenizer via `SqlDialect.backslash_escapes_strings` (= `"\\" in tokenizer.STRING_ESCAPES`, guarded + 14-dialect pinned) so our escaping can never drift from the parser that reads the substituted SQL. `escape="python"` (Mode-B) additionally encodes the full C0 control range (`\t`/`\n`/`\r` named, rest `\xNN`) so raw newlines/NUL no longer break `ast.parse`. Engine fail-closed: `_substitute_model_sql_surfaces` / `_render_probe_model` require a `dialect`, threaded from the resolved datasource — no bare bool to forget. Assumes MySQL's default `sql_mode` (backslash escapes on); `NO_BACKSLASH_ESCAPES` servers are a sqlglot-layer-wide limitation, documented not fixed. The SQLite backslash end-to-end gap stays a pinned strict-xfail (pre-existing, out of scope). Bound parameters rejected (don't fit substitute-into-raw-SQL). Nested/join/cross-model lineages remain DEV-1678. - 2026-08-04 — Declared list-valued `{variable}` coercion (DEV-1730 follow-up): a scalar supplied for a variable the model declares `list_valued` is wrapped into a one-element list before Mode-A substitution, so an importer-generated `col IN ({var})` renders `IN ('US')` rather than the unquoted `IN (US)`. The generic scalar rule (author writes the quotes, so `{var}` also works in numeric/fragment positions like `amount >= {floor}` and `{d}::TIMESTAMP`) is CORRECT and unchanged — it just presumes an author who can see the SQL position, which a machine-generated fixed template does not have; the caller cannot supply per-element quotes through parentheses the importer wrote. Silent-wrong-answer risk drove the fix over a raise: `region IN (US)` parses as a column reference, so it fails at the database with a confusing message, or resolves against a real column and returns wrong rows. Opt-in is a front-end-NEUTRAL flag: the Cube converter writes `list_valued: ref.kind == "string"` into each `meta.cube_variables` entry (arrow forms splice pre-quoted scalars and stay `False`), and the engine reads only that flag — never Cube's `kind` taxonomy — so a future list-shaped front-end opts in the same way. Coercion lives at the single Mode-A choke point `_substitute_model_sql_surfaces` (execution and the `_render_probe_model` type-probe both route through it, so it cannot be bypassed) via `coerce_declared_list_variables` / `list_valued_variable_names` in `slayer/core/query.py`. Scope is deliberately narrow: only `str`/`int`/`float`/`bool` are wrapped; `list`/`tuple` pass through (the **empty list still raises** — "no filter" belongs to an optional block or a sentinel default); `None`/`dict` are left for `_render_variable_value` to reject with its own naming error; hand-written models declare nothing and are untouched. Follow-on from the same review: `declares_variables(model)` (any non-empty `meta.cube_variables`) now also defeats the DEV-1625 zero-variable fast path, via the shared `_model_needs_substitution_pass` predicate used by both `_substitute_model_sql_surfaces` and `_render_probe_model`. This closes the fast-path hole for a GENERATED model whose pushdowns are all required (no `{? ?}` block to force the pass): such a model used to emit a bare `{var}` into the SQL on a zero-variable call instead of raising the documented missing-variable error. The hole stays open — deliberately — for hand-written models, which declare nothing and keep the raw-brace-literal protection (`'{1,2,3}'`). The `list_valued` flag is matched with `is True`, not truthiness, since `meta` is user-extensible and a stray `1` or the string `"false"` must not switch substitution semantics. The bag is also SELF-IDENTIFYING — an entry counts only with a string `member` (the shape every importer writes) — so a hand-written `meta` that reuses the `cube_variables` key is not mistaken for generated SQL and silently stripped of its brace-literal protection. - 2026-08-05 — One naming authority + one ValueKey renderer (DEV-1744, PR 1 of the DEV-1742 consolidation). Four consequences worth recording. (1) **Cross-model CTE dedup is now structural.** `_cm_` CTE names were minted by a doubly-lossy helper (`flat_name`, itself documented non-injective, then a non-identifier `re.sub`) and the resulting string doubled as the plan's identity key in `seen_cm`. Two reachable failures followed: two measures whose canonical aliases differ only in case emitted two `_cm_` names that fold together on every case-folding dialect (the collision belt raised — `_wm_` had been retrofitted onto the allocator years earlier, `_cm_` never was), and two genuinely distinct aggregates that sanitised alike made the second skip the loop body, leaving its join-back and column-alias maps unwritten and raising `KeyError` downstream. The dedup key is now the typed `AggregateKey` plus the source relation; the name is allocator-minted and stored once, so the five sites that used to re-derive it now read it. Deliberately NOT keyed on the canonical alias: `canonical_agg_name` omits `column_filter_key`, so a filtered and an unfiltered aggregate over one column share an alias while needing two CTEs — deduping on the string would silently merge them, a wrong answer rather than a crash. (2) **One ScalarCall policy.** Two of the five renderers returned `exp.Anonymous` passthrough, so `ifnull` reached Postgres — which has no `IFNULL` — while the same key emitted `COALESCE` from a projection. All six render paths now call one `render_scalar_call`. Transpiling alone was NOT sufficient: `exp.func("LOG10", x)` normalises to a generic `Log(10, x)` that re-emits as `LOG(10, x)`, wrong on dialects with a native single-arg `LOG10`, so the policy is transpile-then-log-alias-rewrite. Consequence surfaced and approved: `concat` now emits the `||` operator on Postgres/DuckDB where it previously emitted `CONCAT(...)`. That is a semantic change as well as a spelling one — Postgres `CONCAT()` ignores NULL operands, `||` propagates them. Ratified as the kept behavior: the projection path has always emitted `||`, so before this change SLayer disagreed with itself between filters and projections on the same input, and consistency was chosen over preserving the filter-side NULL tolerance. (3) **`ScopeFrame._model_for` raises** instead of falling back to the root model; the fallback turned a wiring bug into a wrong answer by expanding a different model's derived SQL. (4) **Named P-F carve-outs**, recorded rather than omitted: the T-SQL ORDER-BY-detach rewrite and the stage-schema wrapper take `_outer` / `_stage_inner` as shared CONSTANTS rather than allocator-minted names (the former is a post-generation AST pass with no allocator in reach, and a later PR rebuilds that machinery); `base` / `_base` / `_combined` keep their literal names but are reserved into the allocator. Superseded code is retained and callable per the chain's operating rule — deletion happens in the final PR. -- 2026-08-05 — Operand grouping joins ScalarCall as a policy shared ahead of the call-site migration (DEV-1744). A carve-out from the deferral below, taken because the generator's three arithmetic composers were provably emitting wrong SQL, not merely duplicated SQL. All three built `exp.Not` / `exp.Neg` / `exp.Is` around a bare operand and hand-folded `and`/`or` into left-nested nodes; sqlglot does not parenthesise by nesting, so four shapes emitted predicates that parse cleanly and return a different row set: `not (a AND b)` → `NOT a AND b` (De Morgan), `-(a + b)` → `-a + b`, `(a = 5) IS NULL` → `a = 5 IS NULL` (read as `a = (5 IS NULL)`, because `IS` binds tighter than `=`), and `a AND (b OR c)` → `a AND b OR c`. The composers now call `group_unary_operand` / `group_is_operands` from the renderer and fold booleans with `exp.and_` / `exp.or_`. Emission is otherwise unchanged — the whole suite passed without a single expectation edit, which is also why the bugs survived this long. Related: the renderer's own precedence table now treats the comparison level as NON-associative, parenthesising an equal-precedence LEFT child there (arithmetic keeps left-associative flattening), and a bare `*` is refused as the source of any aggregation but `count`, which previously built `SUM(*)` / `COUNT(DISTINCT *)`. +- 2026-08-05 — Arithmetic composition joins ScalarCall as a construct that renders ONCE, ahead of the call-site migration (DEV-1744). A carve-out from the deferral below, taken because the generator's three composers (`_build_arith_or_cmp_ast`, `_compose_arithmetic_op`, `_build_arithmetic_for_filter`) were provably emitting wrong SQL, not merely duplicated SQL. sqlglot does not parenthesise by node nesting, and each composer knew a different subset of the precedence table — `_build_arith_or_cmp_ast` applied no precedence pass at all — so eight shapes emitted expressions that parse cleanly and mean something else: `not (a AND b)` → `NOT a AND b` (De Morgan); `-(a + b)` → `-a + b`; `(a = 5) IS NULL` → `a = 5 IS NULL`, read as `a = (5 IS NULL)` because `IS` binds tighter than `=`; `a AND (b OR c)` → `a AND b OR c`; `(a + b) * c` → `a + b * c`; `a - (b + c)` → `a - b + c`; `a + (b - c)` → `a + b - c` (`+` was treated as associative, which it is not over floats or fixed-precision decimals); and `(a > b) + 1` → `a > b + 1`. All three now delegate to one `render_arithmetic`. Sole exception: comparison operands in `_build_arithmetic_for_filter` keep `_paren_if_binary`, which parenthesises EVERY multi-term operand — strictly more grouping than the shared policy derives, never less, so it is a readability choice rather than a second correctness policy. The full suite passed without a single expectation edit, which is also why the bugs survived this long. Related: the renderer's precedence table now treats the comparison level as NON-associative, parenthesising an equal-precedence LEFT child there (arithmetic keeps left-associative flattening); and a bare `*` is refused as the source of any aggregation but `count`, which previously built `SUM(*)` / `COUNT(DISTINCT *)`. - 2026-08-05 — Renderer call-site migration deferred to the scope-assembly PR (DEV-1744 → DEV-1746). PR 1 lands the complete, tested `render_value_key` API but does NOT reroute the generator's own render paths through it; only the ScalarCall policy is genuinely shared (all six paths call `render_scalar_call`). Deferred because the filter and composite paths carry state the context does not yet consume — the local-aggregate HAVING branch with its slot lookup and inline-aggregate emission, the first/last ranked state, the filter-side CAST policy, and the rn-suffix / filtered-rank / composite-alias / resolved-kwarg maps — and because the cross-scope paths are the ones that should supply `resolve(consumer=...)` its first production caller, which is scope-assembly work by nature. Splitting the reroute across two PRs would move that state twice. Full detail, per call site, in the `slayer/sql/render/value_expr.py` module docstring. diff --git a/slayer/sql/generator.py b/slayer/sql/generator.py index 29c7fcc9..2fa9279f 100644 --- a/slayer/sql/generator.py +++ b/slayer/sql/generator.py @@ -57,8 +57,7 @@ ) from slayer.sql.render.aggregates import window_agg_class from slayer.sql.render.value_expr import ( - group_is_operands, - group_unary_operand, + render_arithmetic, render_scalar_call, rewrite_log_alias, ) @@ -6475,52 +6474,12 @@ def _build_arith_or_cmp_ast( ) -> exp.Expression: """Build a sqlglot expression for a binary or unary op. - Mirrors the small subset of operators the bound-filter renderer - emits: comparisons (``==``, ``!=``, ``<``, ``<=``, ``>``, - ``>=``, ``is``, ``is not``), boolean (``and``, ``or``, ``not``), - arithmetic (``+``, ``-``, ``*``, ``/``). + Delegates to the one composer in ``slayer.sql.render.value_expr``. + The hand-rolled version here applied NO precedence pass at all, so + ``(a + b) * c`` emitted ``a + b * c`` and ``(a > b) + 1`` emitted + ``a > b + 1`` — both parse, both mean something else. """ - if op == "not": - return exp.Not(this=group_unary_operand(operands[0], op="not")) - # ``and`` / ``or`` (Codex round 2): the binder produces n-ary - # boolean ``ArithmeticKey`` for ``a AND b AND c`` (three operands); - # the prior implementation took only ``operands[0]`` / ``[1]`` and - # silently dropped the third predicate from cross-model HAVING/ - # WHERE, broadening results. - # - # ``exp.and_`` / ``exp.or_`` rather than a hand-rolled fold: the fold - # built ``And(a, Or(b, c))``, which sqlglot emits FLAT as - # ``a AND b OR c`` — read back as ``(a AND b) OR c``, a broader row set. - if op in ("and", "or"): - return exp.and_(*operands) if op == "and" else exp.or_(*operands) - left, right = operands[0], operands[1] - # ``IS`` / ``IS NOT`` (Codex review): the typed pipeline's filter - # normalizer lowers SQL ``IS NULL`` / ``IS NOT NULL`` to Python - # ``is None`` / ``is not None``. Render against a ``Null`` literal - # as the standard SQL forms. - if op in ("is", "is not"): - left, right = group_is_operands(lhs=left, rhs=right) - node = exp.Is(this=left, expression=right) - return exp.Not(this=node) if op == "is not" else node - op_map = { - "==": exp.EQ, - "!=": exp.NEQ, - "<": exp.LT, - "<=": exp.LTE, - ">": exp.GT, - ">=": exp.GTE, - "+": exp.Add, - "-": exp.Sub, - "*": exp.Mul, - "/": exp.Div, - } - cls = op_map.get(op) - if cls is None: - raise NotImplementedError( - f"DEV-1450 stage 7b.12: arithmetic operator {op!r} not " - f"supported in cross-model filter rendering.", - ) - return cls(this=left, expression=right) + return render_arithmetic(op, list(operands)) def _build_combined_order_by_sql( self, @@ -7264,59 +7223,14 @@ def _compose_arithmetic_op( already-rendered operands. Accepts the operator aliases ``=``/``==``, ``<>``/``!=`` so the - rendered SQL surfaces the canonical SQL spellings for POST - filters. Unary ``-`` and N-ary ``and``/``or`` left-fold to the - sqlglot binary nodes. + rendered SQL surfaces the canonical SQL spellings for POST filters. + + Delegates to the one composer in ``slayer.sql.render.value_expr``. + The hand-rolled version's precedence table knew only ``+ - * /``, so + a comparison nested in arithmetic — ``(a > b) + 1`` — emitted + ``a > b + 1``, which reads as ``a > (b + 1)``. """ - if len(operands) == 1: - # Grouped through the shared policy: a bare ``exp.Neg`` / ``exp.Not`` - # emitted ``-(a + b)`` as ``-a + b`` and ``not (a and b)`` as - # ``NOT a AND b`` — both parse, both mean something else. - if op in ("not", "-"): - grouped = group_unary_operand(operands[0], op=op) - return ( - exp.Not(this=grouped) if op == "not" else exp.Neg(this=grouped) - ) - if len(operands) == 2: - lhs, rhs = operands - # ``IS`` / ``IS NOT`` (Codex review): see ``_build_arith_or_cmp_ast``. - if op in ("is", "is not"): - lhs, rhs = group_is_operands(lhs=lhs, rhs=rhs) - node = exp.Is(this=lhs, expression=rhs) - return exp.Not(this=node) if op == "is not" else node - binary = { - "+": exp.Add, "-": exp.Sub, "*": exp.Mul, "/": exp.Div, - "<": exp.LT, "<=": exp.LTE, ">": exp.GT, ">=": exp.GTE, - "==": exp.EQ, "=": exp.EQ, - "!=": exp.NEQ, "<>": exp.NEQ, - } - if op in binary: - # sqlglot does NOT add precedence parens for a nested AST, so - # ``Div(Sub(a, b), c)`` would render as ``a - b / c`` (wrong: - # ``b / c`` binds first). Parenthesise a lower-precedence - # operand — and an equal-precedence RIGHT operand under the - # non-associative ``-`` / ``/`` — so ``change_pct`` and friends - # emit ``(a - b) / c``. - arith_prec = {"+": 1, "-": 1, "*": 2, "/": 2} - parent_prec = arith_prec.get(op) - if parent_prec is not None: - lhs = SQLGenerator._paren_if_lower_prec( - lhs, parent_prec=parent_prec, is_right=False, op=op, - ) - rhs = SQLGenerator._paren_if_lower_prec( - rhs, parent_prec=parent_prec, is_right=True, op=op, - ) - return binary[op](this=lhs, expression=rhs) - if len(operands) >= 2 and op in ("and", "or"): - # ``exp.and_`` / ``exp.or_`` rather than a hand-rolled fold: the - # fold built ``And(a, Or(b, c))``, which sqlglot emits FLAT as - # ``a AND b OR c`` — read back as ``(a AND b) OR c``, a broader - # row set. - return exp.and_(*operands) if op == "and" else exp.or_(*operands) - raise NotImplementedError( - f"DEV-1450 stage 7b.10: arithmetic op {op!r} arity " - f"{len(operands)} not supported in POST-filter rendering.", - ) + return render_arithmetic(op, list(operands)) def _emit_planned_outer_wrap( self, @@ -9815,13 +9729,23 @@ def _paren_if_binary(node: exp.Expression) -> exp.Expression: return exp.Paren(this=node) if isinstance(node, exp.Binary) else node @staticmethod - def _build_arithmetic_for_filter( # NOSONAR(S3776) — sequential per-operator dispatch (==/!= → EQ/NEQ, comparison, arithmetic) with DEV-1539 precedence paren-wrapping; each branch is the per-op emission contract. + def _build_arithmetic_for_filter( *, op: str, operands: list, ) -> exp.Expression: + """Compose a WHERE / HAVING operator. + + Every operator but the comparisons delegates to the one composer in + ``slayer.sql.render.value_expr``; the hand-rolled versions here emitted + ``a + (b - c)`` as ``a + b - c``, a different number. + + Comparisons keep :meth:`_paren_if_binary` (DEV-1539): it parenthesises + EVERY multi-term operand, so ``(a + b) > 7`` stays explicit by + inspection rather than by precedence rules. That is strictly more + grouping than the shared policy derives, never less, so it is a + readability choice rather than a second correctness policy. + """ # DSL ``==``/``!=`` map to sqlglot EQ/NEQ; sqlglot then emits the - # dialect-correct SQL operator (postgres ``=``/``!=``). DEV-1539: a - # multi-term comparison operand is parenthesised so its precedence is - # explicit (``(a + b) > 7`` / ``x = (a OR b)``). + # dialect-correct SQL operator (postgres ``=``/``!=``). _cmp = { "==": exp.EQ, "=": exp.EQ, "!=": exp.NEQ, "<>": exp.NEQ, "<": exp.LT, "<=": exp.LTE, ">": exp.GT, ">=": exp.GTE, @@ -9832,67 +9756,7 @@ def _build_arithmetic_for_filter( # NOSONAR(S3776) — sequential per-operator this=SQLGenerator._paren_if_binary(operands[0]), expression=SQLGenerator._paren_if_binary(operands[1]), ) - if op == "+": - # Unary plus is a no-op; legacy never emits it explicitly. - if len(operands) == 1: - return operands[0] - return exp.Add(this=operands[0], expression=operands[1]) - if op == "-": - # Unary minus: the binder represents ``-x`` / ``-10`` as - # ``ArithmeticKey(op="-", operands=(x,))`` — handle the - # single-operand form so a filter like ``amount > -10`` - # doesn't crash with IndexError. - if len(operands) == 1: - return exp.Neg(this=group_unary_operand(operands[0], op="-")) - return exp.Sub( - this=SQLGenerator._paren_if_lower_prec( - operands[0], parent_prec=1, is_right=False, op="-", - ), - expression=SQLGenerator._paren_if_lower_prec( - operands[1], parent_prec=1, is_right=True, op="-", - ), - ) - if op == "*": - return exp.Mul( - this=SQLGenerator._paren_if_lower_prec( - operands[0], parent_prec=2, is_right=False, op="*", - ), - expression=SQLGenerator._paren_if_lower_prec( - operands[1], parent_prec=2, is_right=True, op="*", - ), - ) - if op == "/": - return exp.Div( - this=SQLGenerator._paren_if_lower_prec( - operands[0], parent_prec=2, is_right=False, op="/", - ), - expression=SQLGenerator._paren_if_lower_prec( - operands[1], parent_prec=2, is_right=True, op="/", - ), - ) - if op in ("and", "or"): - # ``exp.and_`` / ``exp.or_`` rather than a hand-rolled fold: the - # fold built ``And(a, Or(b, c))``, which sqlglot emits FLAT as - # ``a AND b OR c`` — read back as ``(a AND b) OR c``, a broader - # row set. - return exp.and_(*operands) if op == "and" else exp.or_(*operands) - if op == "not": - return exp.Not(this=group_unary_operand(operands[0], op="not")) - # ``IS`` / ``IS NOT`` (Codex round 2): the filter normalizer lowers - # SQL ``IS NULL`` / ``IS NOT NULL`` to Python ``is None`` / ``is - # not None``. Render against the rhs (a ``Null`` literal) as the - # standard SQL forms. Without these branches a local-stage filter - # ``deleted_at IS NULL`` parses and binds but raises here at SQL - # generation. Mirrors the patches in ``_build_arith_or_cmp_ast`` - # and ``_compose_arithmetic_op``. - if op in ("is", "is not"): - lhs, rhs = group_is_operands(lhs=operands[0], rhs=operands[1]) - node = exp.Is(this=lhs, expression=rhs) - return exp.Not(this=node) if op == "is not" else node - raise NotImplementedError( - f"DEV-1450 stage 7b.8: ArithmeticKey op {op!r} not " - f"supported in filter rendering." - ) + return render_arithmetic(op, list(operands)) def _build_outer_trim_wrap_sql( self, diff --git a/slayer/sql/render/value_expr.py b/slayer/sql/render/value_expr.py index a734cbf7..d9e0c7cd 100644 --- a/slayer/sql/render/value_expr.py +++ b/slayer/sql/render/value_expr.py @@ -13,13 +13,16 @@ Migration status ---------------- The API here is complete and directly tested, but the generator's own render -paths do NOT yet route through :func:`render_value_key`. Two POLICIES are -shared today, each because the generator's copies were demonstrably wrong -without them: +paths do NOT yet route through :func:`render_value_key`. Two constructs DO +render once already, each promoted early because the generator's copies were +demonstrably emitting wrong SQL, not merely duplicated SQL: * ScalarCall — all six paths call :func:`render_scalar_call`. -* Operand grouping — the generator's three arithmetic composers call - :func:`group_unary_operand` and :func:`group_is_operands`. +* Arithmetic / comparison / boolean composition — the generator's three + composers call :func:`render_arithmetic`. The one exception is comparison + operands in ``_build_arithmetic_for_filter``, which keep a wrapper that + parenthesises EVERY multi-term operand: strictly more grouping than this + module derives, never less. Everything else still runs the generator's own per-path branches. @@ -290,15 +293,19 @@ def group_is_operands( ) -def _render_arithmetic( +def render_arithmetic( op: str, operands: List[exp.Expression], ) -> exp.Expression: """Compose an arithmetic / comparison / boolean operator. - Mirrors the generator's composer, including the unary forms: the binder - represents ``-x`` as a SINGLE-operand ``ArithmeticKey``, so a fold that - just returns ``operands[0]`` would turn ``amount > -10`` into - ``amount > 10``. + The single composer: the generator's three call sites delegate here, so + one ``ArithmeticKey`` groups the same way wherever it is rendered (P-G). + Their hand-rolled versions each knew a different subset of the precedence + table and emitted predicates that parse cleanly and mean something else. + + Handles the unary forms too: the binder represents ``-x`` as a + SINGLE-operand ``ArithmeticKey``, so a fold that just returns + ``operands[0]`` would turn ``amount > -10`` into ``amount > 10``. """ if not operands: raise NotImplementedError(f"Operator {op!r} needs at least one operand.") @@ -529,7 +536,7 @@ def render_value_key( # NOSONAR(S3776) — sequential dispatch over the closed ) if isinstance(key, ArithmeticKey): - return _render_arithmetic( + return render_arithmetic( key.op.lower(), [render_value_key(o, ctx) for o in key.operands], ) diff --git a/tests/test_dev1744_value_expr.py b/tests/test_dev1744_value_expr.py index bd4ac689..e4215822 100644 --- a/tests/test_dev1744_value_expr.py +++ b/tests/test_dev1744_value_expr.py @@ -2250,14 +2250,64 @@ def test_disjunction_inside_a_conjunction_keeps_its_parens( @pytest.mark.parametrize( "composer", - ["build_arithmetic_for_filter", "compose_arithmetic_op"], + ["build_arithmetic_for_filter", "compose_arithmetic_op", + "build_arith_or_cmp_ast"], ) def test_negated_sum_keeps_its_parens(self, composer) -> None: - """``_build_arith_or_cmp_ast`` has no unary-minus branch, so only the - two composers that do are exercised here.""" compose = self._composers()[composer] assert compose("-", [self._add()]).sql() == "-(t.a + t.b)" + @pytest.mark.parametrize( + "composer", + ["build_arithmetic_for_filter", "compose_arithmetic_op", + "build_arith_or_cmp_ast"], + ) + @pytest.mark.parametrize( + "op,inner_cls,inner_op,expected", + [ + # ``a - (b + c)``: dropping the parens regroups to ``(a - b) + c``. + ("-", exp.Add, "+", "t.a - (t.b + t.c)"), + # ``a + (b - c)``: the generator treated ``+`` as associative and + # emitted ``a + b - c``. Over floats and fixed-precision decimals + # that is a different number, not just a different tree. + ("+", exp.Sub, "-", "t.a + (t.b - t.c)"), + ("*", exp.Div, "/", "t.a * (t.b / t.c)"), + ("/", exp.Mul, "*", "t.a / (t.b * t.c)"), + ], + ) + def test_equal_precedence_right_operand_keeps_its_parens( + self, composer, op, inner_cls, inner_op, expected, + ) -> None: + compose = self._composers()[composer] + inner = inner_cls( + this=exp.column("b", table="t"), expression=exp.column("c", table="t"), + ) + out = compose(op, [self._a(), inner]).sql() + assert out == expected, out + + def test_lower_precedence_left_operand_keeps_its_parens(self) -> None: + """``_build_arith_or_cmp_ast`` applied NO precedence pass at all, so + ``(a + b) * c`` emitted ``a + b * c`` — ``b * c`` binds first.""" + compose = self._composers()["build_arith_or_cmp_ast"] + out = compose("*", [self._add(), exp.column("c", table="t")]).sql() + assert out == "(t.a + t.b) * t.c", out + + @pytest.mark.parametrize( + "composer", + ["build_arithmetic_for_filter", "compose_arithmetic_op", + "build_arith_or_cmp_ast"], + ) + def test_comparison_nested_in_arithmetic_keeps_its_parens( + self, composer, + ) -> None: + """The generator's precedence table knew only ``+ - * /``, so a + comparison operand fell through ungrouped: ``(a > b) + 1`` emitted + ``a > b + 1``, read back as ``a > (b + 1)``.""" + compose = self._composers()[composer] + gt = exp.GT(this=self._a(), expression=exp.column("b", table="t")) + out = compose("+", [gt, exp.Literal.number("1")]).sql() + assert out == "(t.a > t.b) + 1", out + @pytest.mark.parametrize( "composer", ["build_arithmetic_for_filter", "compose_arithmetic_op", From 6b2756538fe8afaea8e9b8548bfe7e64b7dd5281 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Wed, 5 Aug 2026 22:18:45 +0200 Subject: [PATCH 21/98] DEV-1744: delete the generator's now-unreferenced weaker precedence helper --- slayer/sql/generator.py | 20 -------------------- tests/test_dev1744_value_expr.py | 12 ++++++------ 2 files changed, 6 insertions(+), 26 deletions(-) diff --git a/slayer/sql/generator.py b/slayer/sql/generator.py index 2fa9279f..16f8a2d4 100644 --- a/slayer/sql/generator.py +++ b/slayer/sql/generator.py @@ -7195,26 +7195,6 @@ def recurse(k) -> exp.Expression: f"{type(key).__name__} not yet supported.", ) - @staticmethod - def _paren_if_lower_prec( - child: exp.Expression, *, parent_prec: int, is_right: bool, op: str, - ) -> exp.Expression: - """Wrap ``child`` in parens when its arithmetic precedence is lower - than the parent op's (or equal, for the RIGHT operand of the - non-associative ``-`` / ``/``). Leaves / functions / casts / already- - parenthesised nodes are returned untouched. - """ - child_prec = { - exp.Add: 1, exp.Sub: 1, exp.Mul: 2, exp.Div: 2, - }.get(type(child)) - if child_prec is None: - return child - if child_prec < parent_prec: - return exp.Paren(this=child) - if child_prec == parent_prec and is_right and op in ("-", "/"): - return exp.Paren(this=child) - return child - @staticmethod def _compose_arithmetic_op( *, op: str, operands: List[exp.Expression], diff --git a/tests/test_dev1744_value_expr.py b/tests/test_dev1744_value_expr.py index e4215822..30d77487 100644 --- a/tests/test_dev1744_value_expr.py +++ b/tests/test_dev1744_value_expr.py @@ -2263,20 +2263,20 @@ def test_negated_sum_keeps_its_parens(self, composer) -> None: "build_arith_or_cmp_ast"], ) @pytest.mark.parametrize( - "op,inner_cls,inner_op,expected", + "op,inner_cls,expected", [ # ``a - (b + c)``: dropping the parens regroups to ``(a - b) + c``. - ("-", exp.Add, "+", "t.a - (t.b + t.c)"), + ("-", exp.Add, "t.a - (t.b + t.c)"), # ``a + (b - c)``: the generator treated ``+`` as associative and # emitted ``a + b - c``. Over floats and fixed-precision decimals # that is a different number, not just a different tree. - ("+", exp.Sub, "-", "t.a + (t.b - t.c)"), - ("*", exp.Div, "/", "t.a * (t.b / t.c)"), - ("/", exp.Mul, "*", "t.a / (t.b * t.c)"), + ("+", exp.Sub, "t.a + (t.b - t.c)"), + ("*", exp.Div, "t.a * (t.b / t.c)"), + ("/", exp.Mul, "t.a / (t.b * t.c)"), ], ) def test_equal_precedence_right_operand_keeps_its_parens( - self, composer, op, inner_cls, inner_op, expected, + self, composer, op, inner_cls, expected, ) -> None: compose = self._composers()[composer] inner = inner_cls( From 2f6ee3cd3430985f53df4fc379e629f88dfbf248 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Wed, 5 Aug 2026 22:24:05 +0200 Subject: [PATCH 22/98] =?UTF-8?q?DEV-1745:=20W4=20=E2=80=94=20structural?= =?UTF-8?q?=20reachability=20replaces=20the=20model-name=20heuristic?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit classify_host_filter routed derived-column references by asking whether the declaring model's NAME appeared anywhere in target_path. That flat membership test got two shapes wrong: * a model reachable on a SIBLING branch counted as reachable, because its name was in the path even though no prefix of the path led to it; * a HOST-model derived column whose Column.sql crossed INTO the target counted as host-local, because only the declaring model was consulted and never the SQL. Replaced by one rule for every key kind: a dependency is reachable iff its anchored join path is a PREFIX of target_path. Reachability stays an ALL-DEPENDENCIES predicate — one unreachable dependency drops the filter. New slayer/engine/filter_reachability.py computes the summary per filter at plan time, recursively over the WHOLE key tree (a crossing reference buried under arithmetic or inside an aggregate's kwargs is still a dependency), and fails CLOSED on an unhandled key kind rather than reporting "crosses nothing". Storage per D9: on PlannedQuery, not on ColumnSqlKey (interned, and rerooting copies unknown fields through stale) and not on ValueSlot (filter_referenced_slot_ids skips keys with no interned slot — filter-only derived columns are exactly those — and slots are copied into nested plans). Recomputed per plan so the paths always mean what they say relative to the root they were anchored at. has_host_local_ref is carried alongside the paths because the two cases it separates are otherwise indistinguishable: a filter that crosses nothing because it is host-local, versus a host-declared derived column with an empty anchored path whose expansion DOES reach the target. The first must stay at the host; the second can propagate. Test updates: 7 classifier unit tests in test_cross_model_planner.py now supply the structural summary, which is the classifier's input contract after this change. Two of them pinned the deleted heuristic by name and now express the structural rule instead. Three ArithmeticKey constructions in the DEV-1745 test pack used a left=/right= API that does not exist (the field is `operands`) and were failing validation rather than testing anything. Co-Authored-By: Claude Opus 5 (1M context) --- slayer/engine/cross_model_planner.py | 91 ++++---- slayer/engine/filter_reachability.py | 332 +++++++++++++++++++++++++++ slayer/engine/planned.py | 27 +++ slayer/engine/stage_planner.py | 50 +++- tests/test_cross_model_planner.py | 24 +- tests/test_dev1745_reachability.py | 18 +- 6 files changed, 474 insertions(+), 68 deletions(-) create mode 100644 slayer/engine/filter_reachability.py diff --git a/slayer/engine/cross_model_planner.py b/slayer/engine/cross_model_planner.py index 9baa1080..7ee27b9b 100644 --- a/slayer/engine/cross_model_planner.py +++ b/slayer/engine/cross_model_planner.py @@ -119,6 +119,13 @@ class HostFilterRouting(BaseModel): phase: Phase referenced_slot_ids: List[SlotId] = Field(default_factory=list) text: Optional[str] = None + # DEV-1745 (W4 / D9) — the filter's structural reachability summary, in the + # host plan's coordinate system. ``crossed_join_paths`` is every join path + # its dependency tree is anchored at; ``has_host_local_ref`` marks a + # dependency anchored at the host root, which no CTE rooted elsewhere can + # evaluate. Computed at plan time by ``filter_reachability``. + crossed_join_paths: Tuple[Tuple[str, ...], ...] = () + has_host_local_ref: bool = False # --------------------------------------------------------------------------- @@ -138,24 +145,29 @@ def classify_host_filter( See the module docstring for the decision table. The classifier is pure: same inputs → same output, no side effects. - ``host_model_name`` is used to route ``ColumnSqlKey`` refs (derived - columns): the key carries its host model name but not a path, so we - compare it against the host model name and the path to decide - reachable / local / unreachable. When ``host_model_name`` is None, - ColumnSqlKey refs default to local — conservative for callers that - don't have the host model in scope. + Row-level reachability is decided EXCLUSIVELY from the filter's structural + summary (``crossed_join_paths`` / ``has_host_local_ref``, computed at plan + time by :mod:`slayer.engine.filter_reachability`) under one rule for every + key kind: a dependency is reachable iff its anchored path is a PREFIX of + ``target_path``. It is an ALL-DEPENDENCIES predicate — one unreachable + dependency drops the filter, however many others are reachable. + + This replaces a flat model-NAME membership test for derived columns, which + counted a SIBLING branch as reachable whenever it happened to share a model + name with the target path, and counted a host-model derived column whose + ``Column.sql`` crossed INTO the target as host-local. + + Aggregates keep their own arm: an aggregate is routed by WHERE it is + computed (on the target vs elsewhere), not by the reachability of its + inputs. ``host_model_name`` is accepted for signature compatibility and is + no longer consulted — the structural summary carries what it approximated. """ if host_filter.phase == Phase.POST: return FilterRoute.STAY_AT_HOST_POST - if not host_filter.referenced_slot_ids: - # No referenced slots — nothing to route into the CTE. - return FilterRoute.STAY_AT_HOST_POST by_id = {s.id: s for s in host_slots} - local_row: List[SlotId] = [] - reachable_path: List[SlotId] = [] - unreachable: List[SlotId] = [] + unknown: List[SlotId] = [] aggregate_on_target: List[SlotId] = [] aggregate_other: List[SlotId] = [] @@ -163,57 +175,32 @@ def classify_host_filter( s = by_id.get(sid) if s is None: # Unknown slot id — be conservative, treat as unreachable. - unreachable.append(sid) + unknown.append(sid) continue if isinstance(s.key, AggregateKey): - agg_source = s.key.source - agg_path = getattr(agg_source, "path", ()) + agg_path = getattr(s.key.source, "path", ()) if agg_path == target_path: aggregate_on_target.append(sid) else: aggregate_other.append(sid) - elif isinstance(s.key, ColumnKey): - if not s.key.path: - local_row.append(sid) - elif s.key.path == target_path[: len(s.key.path)]: - reachable_path.append(sid) - else: - unreachable.append(sid) - elif isinstance(s.key, ColumnSqlKey): - # Derived column. Route by its host model: on host → local; - # on any model in target_path → reachable; otherwise → - # unreachable. - cm = s.key.model - if host_model_name is not None and cm == host_model_name: - local_row.append(sid) - elif cm in target_path: - reachable_path.append(sid) - elif host_model_name is None: - # No host name to compare against — conservative default. - local_row.append(sid) - else: - unreachable.append(sid) - else: - # Transform / Arithmetic / ScalarCall: phase already decided. - # POST was checked above; ROW/AGGREGATE land here from - # arithmetic / scalar calls. Treat as local for routing. - local_row.append(sid) - - if unreachable or aggregate_other: - # Any unreachable ref → drop + warn (covers pure-unreachable AND - # mixed-with-reachable cases per decision table rows 6/7). + + crossed = tuple(host_filter.crossed_join_paths) + unreachable_paths = [ + p for p in crossed if tuple(p) != tuple(target_path[: len(p)]) + ] + + if unknown or aggregate_other or unreachable_paths: return FilterRoute.DROP_UNREACHABLE - if local_row and not (aggregate_on_target or reachable_path): + if host_filter.has_host_local_ref: + # Mixed host-local + reachable. The local refs cannot be evaluated in + # a CTE rooted elsewhere, so the whole filter stays at the host. return FilterRoute.DROP_HOST_LOCAL - if local_row: - # Mixed local + (target-path / target-agg). The local refs can't - # be evaluated in the CTE, so the filter stays at host. + if not crossed and not aggregate_on_target: + # Nothing crosses and no aggregate to propagate — purely host-local. return FilterRoute.DROP_HOST_LOCAL if aggregate_on_target: return FilterRoute.PROPAGATE_HAVING - if reachable_path: - return FilterRoute.PROPAGATE_WHERE - return FilterRoute.STAY_AT_HOST_POST + return FilterRoute.PROPAGATE_WHERE # --------------------------------------------------------------------------- diff --git a/slayer/engine/filter_reachability.py b/slayer/engine/filter_reachability.py new file mode 100644 index 00000000..2e397231 --- /dev/null +++ b/slayer/engine/filter_reachability.py @@ -0,0 +1,332 @@ +"""DEV-1745 (W4 / mechanism contract 5.3) — per-filter structural reachability. + +Cross-model routing asks one question of every host filter: can this predicate +be evaluated inside a CTE rooted at ``target_path``? The answer is structural — +it depends on which join paths the filter's dependencies are anchored at — so +it is computed HERE, at plan time, and ``classify_host_filter`` reads it. + +What this replaces: a flat model-NAME membership test (``cm in target_path``) +used for derived columns. It got two shapes wrong. A model reachable on a +SIBLING branch counted as reachable, because its name appeared in the target +path even though no prefix of the path led to it. And a host-model derived +column whose ``Column.sql`` crossed INTO the target counted as host-local, +because only the declaring model's name was consulted, never the SQL. + +One rule for every key kind: a dependency is reachable iff its anchored join +path is a PREFIX of ``target_path`` (``path == target_path[:len(path)]``). +Reachability is an ALL-DEPENDENCIES predicate — any unreachable dependency +drops the filter. + +Storage (D9). The summary lives per-filter on ``PlannedQuery``. NOT on +``ColumnSqlKey``: that key is interned and ``_reroot_path_ref`` re-anchors it +with ``model_copy(update={"path": ...})``, which carries any extra field +through rerooting stale. NOT on ``ValueSlot``: ``filter_referenced_slot_ids`` +silently skips keys with no interned slot, and a derived column referenced only +inside a filter is exactly such a key — plus slots are copied wholesale into +nested plans, which would import the PARENT's coordinate system. + +Invariant: every summary is expressed in the coordinate system of the +``PlannedQuery`` that owns it, and is recomputed per plan, never copied. +""" + +from __future__ import annotations + +from typing import List, Tuple + +from slayer.core.errors import SlayerError +from slayer.core.keys import ( + AggregateKey, + ArithmeticKey, + BetweenKey, + ColumnKey, + ColumnSqlKey, + InKey, + LiteralKey, + ScalarCallKey, + SqlExprKey, + StarKey, + TimeTruncKey, + TransformKey, +) +from slayer.engine.column_expansion import collect_root_scope_joined_paths +from slayer.engine.column_filter_paths import ( + _expand_derived_refs_any_dialect, + _parse_filter_sql_any_dialect, +) + +Path = Tuple[str, ...] + + +class UnhandledValueKindError(SlayerError, TypeError): + """A ValueKey kind the reachability scan does not know how to walk. + + Fails CLOSED, mirroring the total-visitor discipline the ValueKey renderer + uses: a new key kind that silently contributed no paths would read as + "crosses nothing", and a filter depending on it would propagate into a CTE + that cannot evaluate it. + """ + + def __init__(self, key: object) -> None: + self.key_type = type(key).__name__ + super().__init__( + f"UnhandledValueKindError: reachability scan has no rule for key " + f"kind {self.key_type!r}. Add an explicit arm — a silent empty " + f"result would route the filter as if it crossed nothing." + ) + + +def _prefixes(path: Path) -> List[Path]: + """Every non-empty prefix of ``path``. + + The FROM builder needs each intermediate join to reach the last one, and + reachability is judged per hop, so a two-hop reference contributes both + ``("a",)`` and ``("a", "b")``. + """ + return [tuple(path[: i + 1]) for i in range(len(path))] + + +def _derived_sql_paths( + *, key: ColumnSqlKey, anchor_model, anchor_relation: str, bundle, +) -> List[Path]: + """Join paths the expansion of a derived column's ``Column.sql`` crosses. + + Expanded with the same anchoring convention ``ScopeFrame`` uses — at the + ``__``-path alias with ``is_root=False`` when the column lives on a joined + model — so the refs inside come out already prefixed by the key's own path + and the anchor-rooted scan resolves them without further adjustment. + """ + model = ( + anchor_model if key.model == getattr(anchor_model, "name", None) + else bundle.get_referenced_model(key.model) + ) + if model is None: + return [] + col = next((c for c in model.columns if c.name == key.column_name), None) + if col is None or not col.sql: + return [] + + alias_path = "__".join(key.path) if key.path else anchor_relation + expanded = _expand_derived_refs_any_dialect( + sql=col.sql, model=model, alias_path=alias_path, bundle=bundle, + ) + parsed = _parse_filter_sql_any_dialect(expanded or col.sql) + if parsed is None: + return [] + return list(collect_root_scope_joined_paths( + parsed=parsed, + source_model=anchor_model, + source_relation=anchor_relation, + bundle=bundle, + )) + + +def compute_key_join_paths( + *, key, anchor_model, anchor_relation: str, bundle, +) -> Tuple[Path, ...]: + """Every join path ``key``'s dependency tree crosses, anchored at + ``anchor_relation``. + + Recursive over the WHOLE key tree, not just the top node: a crossing + reference buried under arithmetic or inside an aggregate's kwargs is still + a dependency the destination scope has to satisfy. Returns an + insertion-ordered, de-duplicated tuple; empty means the key is evaluable + wherever the anchor is. + """ + seen: "dict[Path, None]" = {} + + def _add(path: Path) -> None: + if path: + seen.setdefault(tuple(path), None) + + def _walk(node) -> None: + if node is None: + return + if isinstance(node, (LiteralKey, StarKey)): + return + if isinstance(node, ColumnKey): + for p in _prefixes(node.path): + _add(p) + return + if isinstance(node, ColumnSqlKey): + for p in _prefixes(node.path): + _add(p) + for p in _derived_sql_paths( + key=node, anchor_model=anchor_model, + anchor_relation=anchor_relation, bundle=bundle, + ): + _add(p) + return + if isinstance(node, SqlExprKey): + for p in node.referenced_join_paths: + for pre in _prefixes(tuple(p)): + _add(pre) + return + if isinstance(node, TimeTruncKey): + _walk(node.column) + return + if isinstance(node, AggregateKey): + _walk(node.source) + for a in node.args: + _walk(a) + for _name, v in node.kwargs: + _walk(v) + _walk(node.column_filter_key) + return + if isinstance(node, TransformKey): + _walk(node.input) + for pk in node.partition_keys: + _walk(pk) + _walk(node.time_key) + return + if isinstance(node, ArithmeticKey): + for o in node.operands: + _walk(o) + return + if isinstance(node, ScalarCallKey): + for a in node.args: + _walk(a) + return + if isinstance(node, InKey): + _walk(node.column) + for v in node.values: + _walk(v) + return + if isinstance(node, BetweenKey): + _walk(node.column) + _walk(node.low) + _walk(node.high) + return + # Scalars carried inline by TransformKey args / kwargs are values, not + # references, and cannot cross anything. + if isinstance(node, (str, int, float, bool)) or node is None: + return + raise UnhandledValueKindError(node) + + _walk(key) + return tuple(seen) + + +def key_has_host_local_ref( + *, key, anchor_model, anchor_relation: str, bundle, +) -> bool: + """Whether ``key`` depends on anything anchored AT the host root. + + A host-local dependency cannot be evaluated inside a CTE rooted elsewhere, + so a filter carrying one stays at the host even when its other dependencies + are reachable. Distinguished from "crosses nothing" deliberately: a derived + column declared on the host whose ``Column.sql`` reaches INTO the target + has an empty anchored path but is NOT host-local — inside the target's + scope its expansion resolves. + """ + found = False + + def _walk(node) -> None: + nonlocal found + if found or node is None: + return + if isinstance(node, (LiteralKey, StarKey, SqlExprKey)): + return + if isinstance(node, ColumnKey): + if not node.path: + found = True + return + if isinstance(node, ColumnSqlKey): + if node.path: + return + if not _derived_sql_paths( + key=node, anchor_model=anchor_model, + anchor_relation=anchor_relation, bundle=bundle, + ): + found = True + return + if isinstance(node, TimeTruncKey): + _walk(node.column) + return + if isinstance(node, AggregateKey): + # An aggregate is routed by its own decision table arm (on-target + # vs elsewhere), not by host-locality of its inputs. + return + if isinstance(node, TransformKey): + _walk(node.input) + for pk in node.partition_keys: + _walk(pk) + _walk(node.time_key) + return + if isinstance(node, ArithmeticKey): + for o in node.operands: + _walk(o) + return + if isinstance(node, ScalarCallKey): + for a in node.args: + _walk(a) + return + if isinstance(node, InKey): + _walk(node.column) + return + if isinstance(node, BetweenKey): + _walk(node.column) + _walk(node.low) + _walk(node.high) + return + if isinstance(node, (str, int, float, bool)): + return + raise UnhandledValueKindError(node) + + _walk(key) + return found + + +def path_is_reachable(*, path: Path, target_path: Path) -> bool: + """The ONE reachability rule, for every key kind. + + ``path`` is reachable from a CTE rooted at ``target_path`` iff it is a + prefix of it. A path DEEPER than the target is not available (the target's + scope stops there); a SIBLING branch that happens to share a model name is + not available either, which is precisely what the old flat membership test + got wrong. + """ + return tuple(path) == tuple(target_path[: len(path)]) + + +def recompute_filter_reachability(planned_query, *, bundle) -> List: + """Recompute every filter's summary from scratch, anchored at + ``planned_query``'s OWN root. + + Used to verify the coordinate-system invariant: a plan's stored summary + must equal this. If a parent had copied its summary into a nested plan, the + stored value would still be anchored at the parent root and the two would + differ. + """ + from slayer.engine.planned import FilterReachability + + anchor_model = planned_query.render_source_model or bundle.source_model + anchor_relation = planned_query.source_relation + out: List = [] + for fp in planned_query.filters_by_phase: + if fp.expression is None: + continue + out.append(FilterReachability( + filter_id=fp.id, + crossed_join_paths=compute_key_join_paths( + key=fp.expression.value_key, + anchor_model=anchor_model, + anchor_relation=anchor_relation, + bundle=bundle, + ), + has_host_local_ref=key_has_host_local_ref( + key=fp.expression.value_key, + anchor_model=anchor_model, + anchor_relation=anchor_relation, + bundle=bundle, + ), + )) + return out + + +def filter_reachability_for(planned_query) -> List: + """The summary ``planned_query`` CARRIES — read, never recomputed. + + The accessor exists so consumers cannot accidentally recompute against a + different anchor and get a summary in the wrong coordinate system. + """ + return list(planned_query.filter_reachability) diff --git a/slayer/engine/planned.py b/slayer/engine/planned.py index deeaed33..4e6e6981 100644 --- a/slayer/engine/planned.py +++ b/slayer/engine/planned.py @@ -396,6 +396,27 @@ def _validate_direction(cls, v: str) -> str: # --------------------------------------------------------------------------- +class FilterReachability(BaseModel): + """DEV-1745 (W4 / D9) — one filter's structural reachability summary. + + ``crossed_join_paths`` is every join path the filter's dependency tree is + anchored at, in THIS plan's coordinate system. ``has_host_local_ref`` marks + a dependency anchored at the plan's own root, which cannot be evaluated + inside a CTE rooted elsewhere. + + Carried per filter on the owning ``PlannedQuery`` rather than on + ``ColumnSqlKey`` (interned, and rerooting copies unknown fields through + stale) or ``ValueSlot`` (slot-less filter-only keys are silently skipped, + and slots are copied into nested plans). + """ + + model_config = ConfigDict(frozen=True) + + filter_id: BoundFilterId + crossed_join_paths: Tuple[Tuple[str, ...], ...] = () + has_host_local_ref: bool = False + + class PlannedQuery(BaseModel): """The fully typed plan for one query stage (P7). @@ -470,6 +491,12 @@ class PlannedQuery(BaseModel): # reads this field and never re-derives it, so clearing the field removes # the outer WHERE. outer_where_filter_ids: List[BoundFilterId] = Field(default_factory=list) + # DEV-1745 (W4 / D9) — per-filter structural reachability, in THIS plan's + # coordinate system. Recomputed for every plan (including the nested + # rerooted plan a cross-model CTE compiles), never copied down from a + # parent: the paths only mean anything relative to the root they were + # anchored at. Read via ``filter_reachability_for``. + filter_reachability: List[FilterReachability] = Field(default_factory=list) # ``CrossModelAggregatePlan.rerooted_plan`` is a forward reference to diff --git a/slayer/engine/stage_planner.py b/slayer/engine/stage_planner.py index 2a4acef7..a59ba322 100644 --- a/slayer/engine/stage_planner.py +++ b/slayer/engine/stage_planner.py @@ -81,10 +81,15 @@ ) from slayer.engine.measure_expansion import expand_model_measures from slayer.engine.response_meta import _infer_aggregated_format +from slayer.engine.filter_reachability import ( + compute_key_join_paths, + key_has_host_local_ref, +) from slayer.engine.planned import ( BoundExpr as PlannedBoundExpr, BoundFilterId, FilterPhase, + FilterReachability, OrderEntry, PlannedQuery, SrcFilterRewrite, @@ -1321,10 +1326,42 @@ def _windowed_phase(bf: BoundFilter) -> Phase: # BoundFilter (date_range + user filters). Model.filters (text-only) # are always row-phase host-local WHERE and never need to be routed # to a cross-model CTE — they're skipped here. + # DEV-1745 (W4 / D9) — the per-filter structural reachability summary, in + # THIS plan's coordinate system. Computed once here and carried on the plan; + # ``classify_host_filter`` routes from it rather than re-deriving anything + # from model names at classification time. + reachability_anchor_model = render_source_model or bundle.source_model + source_relation = ( + query.source_model + if isinstance(query.source_model, str) + else host_model_name + ) + filter_reachability: List[FilterReachability] = [] + for fp in filters_by_phase: + if fp.expression is None: + continue + filter_reachability.append(FilterReachability( + filter_id=fp.id, + crossed_join_paths=compute_key_join_paths( + key=fp.expression.value_key, + anchor_model=reachability_anchor_model, + anchor_relation=source_relation, + bundle=bundle, + ), + has_host_local_ref=key_has_host_local_ref( + key=fp.expression.value_key, + anchor_model=reachability_anchor_model, + anchor_relation=source_relation, + bundle=bundle, + ), + )) + reachability_by_fid = {r.filter_id: r for r in filter_reachability} + host_filter_routings: List[HostFilterRouting] = [] for fid, bf, ftext in zip( bound_filter_ids, bound_filters, bound_filter_texts, ): + summary = reachability_by_fid.get(fid) host_filter_routings.append(HostFilterRouting( filter_id=fid, phase=bf.phase, @@ -1332,6 +1369,12 @@ def _windowed_phase(bf: BoundFilter) -> Phase: bf, projection.registry, )), text=ftext, + crossed_join_paths=( + summary.crossed_join_paths if summary is not None else () + ), + has_host_local_ref=( + summary.has_host_local_ref if summary is not None else False + ), )) cross_model_plans = [] @@ -1450,12 +1493,6 @@ def _windowed_phase(bf: BoundFilter) -> Phase: stage_schema = _emit_stage_schema( query=query, projection=projection, ) - source_relation = ( - query.source_model - if isinstance(query.source_model, str) - else host_model_name - ) - # Stage 7b.10 — the active TD's slot id (``active_td_slot_id``) is resolved # right after projection above so the windowed-plan builder can use it. @@ -1510,6 +1547,7 @@ def _windowed_phase(bf: BoundFilter) -> Phase: distinct_dimension_values=query.distinct_dimension_values, frame_bound_columns=frame_bound_columns, outer_where_filter_ids=outer_where_filter_ids, + filter_reachability=filter_reachability, ) diff --git a/tests/test_cross_model_planner.py b/tests/test_cross_model_planner.py index 995d2a82..c25e88eb 100644 --- a/tests/test_cross_model_planner.py +++ b/tests/test_cross_model_planner.py @@ -493,6 +493,7 @@ def test_joined_target_path_propagates_as_where(self): phase=Phase.ROW, referenced_slot_ids=["rs_revenue"], text="customers.revenue > 100", + crossed_join_paths=(("customers",),), )] plan = planner.plan( aggregate_slot_id="cm1", @@ -591,6 +592,7 @@ def test_unreachable_branch_drops_and_warns(self): phase=Phase.ROW, referenced_slot_ids=["rs_other"], text="warehouses.name = 'EU'", + crossed_join_paths=(("warehouses",),), )] plan = planner.plan( aggregate_slot_id="cm1", @@ -618,6 +620,7 @@ def test_mixed_refs_drops_and_warns(self): phase=Phase.ROW, referenced_slot_ids=["rs_target", "rs_other"], text="customers.revenue > warehouses.x", + crossed_join_paths=(("customers",), ("warehouses",)), )] plan = planner.plan( aggregate_slot_id="cm1", @@ -733,6 +736,7 @@ def test_classify_target_path(self): hf = HostFilterRouting( filter_id="f1", phase=Phase.ROW, referenced_slot_ids=["h"], text="", + crossed_join_paths=(("customers",),), ) route = classify_host_filter( host_filter=hf, @@ -759,6 +763,7 @@ def test_classify_unreachable(self): hf = HostFilterRouting( filter_id="f1", phase=Phase.ROW, referenced_slot_ids=["h"], text="", + crossed_join_paths=(("warehouses",),), ) route = classify_host_filter( host_filter=hf, @@ -798,13 +803,20 @@ def test_classify_columnsqlkey_on_host_is_local(self): assert route == FilterRoute.DROP_HOST_LOCAL def test_classify_columnsqlkey_on_target_is_reachable(self): - # ColumnSqlKey with model in target_path → PROPAGATE_WHERE. + # A derived column anchored ON the target path → PROPAGATE_WHERE. + # Routed from the structural summary, not from the model NAME: the + # name-membership heuristic this replaces also called a SIBLING branch + # reachable whenever it happened to share a name with the target path. derived = ValueSlot( - id="d", key=ColumnSqlKey(model="customers", column_name="x"), + id="d", + key=ColumnSqlKey( + path=("customers",), model="customers", column_name="x", + ), declared_name="x", phase=Phase.ROW, hidden=True, ) hf = HostFilterRouting( filter_id="f1", phase=Phase.ROW, referenced_slot_ids=["d"], + crossed_join_paths=(("customers",),), ) route = classify_host_filter( host_filter=hf, @@ -815,13 +827,17 @@ def test_classify_columnsqlkey_on_target_is_reachable(self): assert route == FilterRoute.PROPAGATE_WHERE def test_classify_columnsqlkey_on_other_branch_is_unreachable(self): - # ColumnSqlKey on a model not in target_path and not host → unreachable. + # A derived column anchored on a SIBLING branch → unreachable. derived = ValueSlot( - id="d", key=ColumnSqlKey(model="warehouses", column_name="x"), + id="d", + key=ColumnSqlKey( + path=("warehouses",), model="warehouses", column_name="x", + ), declared_name="x", phase=Phase.ROW, hidden=True, ) hf = HostFilterRouting( filter_id="f1", phase=Phase.ROW, referenced_slot_ids=["d"], + crossed_join_paths=(("warehouses",),), ) route = classify_host_filter( host_filter=hf, diff --git a/tests/test_dev1745_reachability.py b/tests/test_dev1745_reachability.py index ac9c4cc6..8a0ed333 100644 --- a/tests/test_dev1745_reachability.py +++ b/tests/test_dev1745_reachability.py @@ -193,8 +193,10 @@ class TestCompositeKeyKindsAreTotal: def test_arithmetic_unions_operands(self) -> None: key = ArithmeticKey( op="+", - left=ColumnKey(path=(), leaf="amount"), - right=ColumnKey(path=("customers",), leaf="balance"), + operands=( + ColumnKey(path=(), leaf="amount"), + ColumnKey(path=("customers",), leaf="balance"), + ), ) assert ("customers",) in _paths_for(key) @@ -224,8 +226,10 @@ def test_nested_derived_below_a_composite(self) -> None: seen — this is the case a top-level-only scan misses.""" key = ArithmeticKey( op="+", - left=ColumnKey(path=(), leaf="amount"), - right=_derived("host_derived_crossing"), + operands=( + ColumnKey(path=(), leaf="amount"), + _derived("host_derived_crossing"), + ), ) assert ("customers",) in _paths_for(key) @@ -336,8 +340,10 @@ def test_mixed_reachable_and_unreachable_drops(self) -> None: key = ArithmeticKey( op="+", - left=ColumnKey(path=("customers",), leaf="balance"), - right=ColumnKey(path=("warehouses",), leaf="id"), + operands=( + ColumnKey(path=("customers",), leaf="balance"), + ColumnKey(path=("warehouses",), leaf="id"), + ), ) route = self._route(key, target_path=("customers",)) assert route == FilterRoute.DROP_UNREACHABLE, ( From 11e76d7f1a4ab1eecad2c152f554ee7ec7c1f981 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Wed, 5 Aug 2026 22:28:29 +0200 Subject: [PATCH 23/98] =?UTF-8?q?DEV-1744:=20parenthesise=20%=20unconditio?= =?UTF-8?q?nally=20=E2=80=94=20sqlglot=20re-parses=20a=20+=20b=20%=20c=20a?= =?UTF-8?q?s=20(a=20+=20b)=20%=20c?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- DECISIONS.md | 1 + slayer/sql/render/value_expr.py | 10 +++ tests/test_dev1744_value_expr.py | 147 +++++++++++++++++++++++++++++++ 3 files changed, 158 insertions(+) diff --git a/DECISIONS.md b/DECISIONS.md index 52863700..4f33e4bd 100644 --- a/DECISIONS.md +++ b/DECISIONS.md @@ -97,4 +97,5 @@ implementation detail. Include issue refs when known. - 2026-08-04 — Declared list-valued `{variable}` coercion (DEV-1730 follow-up): a scalar supplied for a variable the model declares `list_valued` is wrapped into a one-element list before Mode-A substitution, so an importer-generated `col IN ({var})` renders `IN ('US')` rather than the unquoted `IN (US)`. The generic scalar rule (author writes the quotes, so `{var}` also works in numeric/fragment positions like `amount >= {floor}` and `{d}::TIMESTAMP`) is CORRECT and unchanged — it just presumes an author who can see the SQL position, which a machine-generated fixed template does not have; the caller cannot supply per-element quotes through parentheses the importer wrote. Silent-wrong-answer risk drove the fix over a raise: `region IN (US)` parses as a column reference, so it fails at the database with a confusing message, or resolves against a real column and returns wrong rows. Opt-in is a front-end-NEUTRAL flag: the Cube converter writes `list_valued: ref.kind == "string"` into each `meta.cube_variables` entry (arrow forms splice pre-quoted scalars and stay `False`), and the engine reads only that flag — never Cube's `kind` taxonomy — so a future list-shaped front-end opts in the same way. Coercion lives at the single Mode-A choke point `_substitute_model_sql_surfaces` (execution and the `_render_probe_model` type-probe both route through it, so it cannot be bypassed) via `coerce_declared_list_variables` / `list_valued_variable_names` in `slayer/core/query.py`. Scope is deliberately narrow: only `str`/`int`/`float`/`bool` are wrapped; `list`/`tuple` pass through (the **empty list still raises** — "no filter" belongs to an optional block or a sentinel default); `None`/`dict` are left for `_render_variable_value` to reject with its own naming error; hand-written models declare nothing and are untouched. Follow-on from the same review: `declares_variables(model)` (any non-empty `meta.cube_variables`) now also defeats the DEV-1625 zero-variable fast path, via the shared `_model_needs_substitution_pass` predicate used by both `_substitute_model_sql_surfaces` and `_render_probe_model`. This closes the fast-path hole for a GENERATED model whose pushdowns are all required (no `{? ?}` block to force the pass): such a model used to emit a bare `{var}` into the SQL on a zero-variable call instead of raising the documented missing-variable error. The hole stays open — deliberately — for hand-written models, which declare nothing and keep the raw-brace-literal protection (`'{1,2,3}'`). The `list_valued` flag is matched with `is True`, not truthiness, since `meta` is user-extensible and a stray `1` or the string `"false"` must not switch substitution semantics. The bag is also SELF-IDENTIFYING — an entry counts only with a string `member` (the shape every importer writes) — so a hand-written `meta` that reuses the `cube_variables` key is not mistaken for generated SQL and silently stripped of its brace-literal protection. - 2026-08-05 — One naming authority + one ValueKey renderer (DEV-1744, PR 1 of the DEV-1742 consolidation). Four consequences worth recording. (1) **Cross-model CTE dedup is now structural.** `_cm_` CTE names were minted by a doubly-lossy helper (`flat_name`, itself documented non-injective, then a non-identifier `re.sub`) and the resulting string doubled as the plan's identity key in `seen_cm`. Two reachable failures followed: two measures whose canonical aliases differ only in case emitted two `_cm_` names that fold together on every case-folding dialect (the collision belt raised — `_wm_` had been retrofitted onto the allocator years earlier, `_cm_` never was), and two genuinely distinct aggregates that sanitised alike made the second skip the loop body, leaving its join-back and column-alias maps unwritten and raising `KeyError` downstream. The dedup key is now the typed `AggregateKey` plus the source relation; the name is allocator-minted and stored once, so the five sites that used to re-derive it now read it. Deliberately NOT keyed on the canonical alias: `canonical_agg_name` omits `column_filter_key`, so a filtered and an unfiltered aggregate over one column share an alias while needing two CTEs — deduping on the string would silently merge them, a wrong answer rather than a crash. (2) **One ScalarCall policy.** Two of the five renderers returned `exp.Anonymous` passthrough, so `ifnull` reached Postgres — which has no `IFNULL` — while the same key emitted `COALESCE` from a projection. All six render paths now call one `render_scalar_call`. Transpiling alone was NOT sufficient: `exp.func("LOG10", x)` normalises to a generic `Log(10, x)` that re-emits as `LOG(10, x)`, wrong on dialects with a native single-arg `LOG10`, so the policy is transpile-then-log-alias-rewrite. Consequence surfaced and approved: `concat` now emits the `||` operator on Postgres/DuckDB where it previously emitted `CONCAT(...)`. That is a semantic change as well as a spelling one — Postgres `CONCAT()` ignores NULL operands, `||` propagates them. Ratified as the kept behavior: the projection path has always emitted `||`, so before this change SLayer disagreed with itself between filters and projections on the same input, and consistency was chosen over preserving the filter-side NULL tolerance. (3) **`ScopeFrame._model_for` raises** instead of falling back to the root model; the fallback turned a wiring bug into a wrong answer by expanding a different model's derived SQL. (4) **Named P-F carve-outs**, recorded rather than omitted: the T-SQL ORDER-BY-detach rewrite and the stage-schema wrapper take `_outer` / `_stage_inner` as shared CONSTANTS rather than allocator-minted names (the former is a post-generation AST pass with no allocator in reach, and a later PR rebuilds that machinery); `base` / `_base` / `_combined` keep their literal names but are reserved into the allocator. Superseded code is retained and callable per the chain's operating rule — deletion happens in the final PR. - 2026-08-05 — Arithmetic composition joins ScalarCall as a construct that renders ONCE, ahead of the call-site migration (DEV-1744). A carve-out from the deferral below, taken because the generator's three composers (`_build_arith_or_cmp_ast`, `_compose_arithmetic_op`, `_build_arithmetic_for_filter`) were provably emitting wrong SQL, not merely duplicated SQL. sqlglot does not parenthesise by node nesting, and each composer knew a different subset of the precedence table — `_build_arith_or_cmp_ast` applied no precedence pass at all — so eight shapes emitted expressions that parse cleanly and mean something else: `not (a AND b)` → `NOT a AND b` (De Morgan); `-(a + b)` → `-a + b`; `(a = 5) IS NULL` → `a = 5 IS NULL`, read as `a = (5 IS NULL)` because `IS` binds tighter than `=`; `a AND (b OR c)` → `a AND b OR c`; `(a + b) * c` → `a + b * c`; `a - (b + c)` → `a - b + c`; `a + (b - c)` → `a + b - c` (`+` was treated as associative, which it is not over floats or fixed-precision decimals); and `(a > b) + 1` → `a > b + 1`. All three now delegate to one `render_arithmetic`. Sole exception: comparison operands in `_build_arithmetic_for_filter` keep `_paren_if_binary`, which parenthesises EVERY multi-term operand — strictly more grouping than the shared policy derives, never less, so it is a readability choice rather than a second correctness policy. The full suite passed without a single expectation edit, which is also why the bugs survived this long. Related: the renderer's precedence table now treats the comparison level as NON-associative, parenthesising an equal-precedence LEFT child there (arithmetic keeps left-associative flattening); and a bare `*` is refused as the source of any aggregation but `count`, which previously built `SUM(*)` / `COUNT(DISTINCT *)`. +- 2026-08-05 — `%` is parenthesised unconditionally when nested, and grouping is pinned by a round-trip property test (DEV-1744). SQL puts `%` on the `*` / `/` tier, and so does SLayer's own Mode-B parser (`formula.py`, `ast.Mod: 2`) — but SQLGLOT's parser puts it on the `+` / `-` tier, so it reads `a + b % c` back as `(a + b) % c`. That matters because generated SQL is re-parsed by sqlglot inside our own pipeline (reserved-word pre-quoting, the log-alias transform), so relying on precedence for `%` means the expression is silently regrouped in flight, before any database sees it. The renderer therefore emits the parens rather than deriving them. Found by the meta-test rather than by review: `TestEveryOperatorPairSurvivesTheRoundTrip` renders every ordered operator pair in both operand positions across postgres/sqlite/mysql, re-parses the emitted SQL, and compares OPERAND STRUCTURE (parens and dialect-injected casts normalised away, since those are grouping-preservation and typing respectively). Re-parse *stability* is deliberately not the assertion — `a + b * c` re-parses to a stable string while meaning something other than the `(a + b) * c` that was built. The matrix is grouped by result type; feeding a boolean to an arithmetic operator is not a shape the binder builds, and dialects wrap such an operand in their own numeric cast. - 2026-08-05 — Renderer call-site migration deferred to the scope-assembly PR (DEV-1744 → DEV-1746). PR 1 lands the complete, tested `render_value_key` API but does NOT reroute the generator's own render paths through it; only the ScalarCall policy is genuinely shared (all six paths call `render_scalar_call`). Deferred because the filter and composite paths carry state the context does not yet consume — the local-aggregate HAVING branch with its slot lookup and inline-aggregate emission, the first/last ranked state, the filter-side CAST policy, and the rn-suffix / filtered-rank / composite-alias / resolved-kwarg maps — and because the cross-scope paths are the ones that should supply `resolve(consumer=...)` its first production caller, which is scope-assembly work by nature. Splitting the reroute across two PRs would move that state twice. Full detail, per call site, in the `slayer/sql/render/value_expr.py` module docstring. diff --git a/slayer/sql/render/value_expr.py b/slayer/sql/render/value_expr.py index d9e0c7cd..24f31ae7 100644 --- a/slayer/sql/render/value_expr.py +++ b/slayer/sql/render/value_expr.py @@ -248,6 +248,16 @@ def _paren_if_lower_prec( A node with no precedence entry — a column, a literal, a function call — is already self-delimiting. """ + if isinstance(child, exp.Mod): + # ``%`` is parenthesised unconditionally, because precedence alone is + # not enough to survive our own pipeline. SQL puts ``%`` on the + # ``*`` / ``/`` tier, and so does the Mode-B parser — but SQLGLOT's + # parser puts it on the ``+`` / ``-`` tier, so it reads back + # ``a + b % c`` as ``(a + b) % c``. Generated SQL IS re-parsed by + # sqlglot downstream (reserved-word pre-quoting, the log-alias + # transform), so an unparenthesised ``%`` would be silently regrouped + # in flight rather than by any database. + return exp.Paren(this=child) child_prec = _PRECEDENCE.get(type(child)) if child_prec is None: return child diff --git a/tests/test_dev1744_value_expr.py b/tests/test_dev1744_value_expr.py index 30d77487..3ea84815 100644 --- a/tests/test_dev1744_value_expr.py +++ b/tests/test_dev1744_value_expr.py @@ -53,6 +53,7 @@ from typing import AsyncIterator, Optional import pytest +import sqlglot from sqlglot import exp from slayer.core.enums import BUILTIN_AGGREGATIONS, DataType @@ -96,6 +97,7 @@ FilterFacilities, RenderContext, _literal, + render_arithmetic, render_value_key, ) from slayer.sql.scope import ScopeFrame @@ -2152,6 +2154,151 @@ def test_count_star_still_renders(self) -> None: assert _sql(render_value_key(key, _composite_ctx())) == "COUNT(*)" +def _grouping_shape(node: exp.Expression): + """``node`` reduced to WHICH OPERAND BELONGS TO WHICH OPERATOR, and nothing + else, as nested tuples of ``(operator, operands...)``. + + Comparing whole sqlglot nodes does not work here, because a parsed tree and + a hand-built one differ in ways that have nothing to do with grouping: + ``Div`` carries parser-set ``typed`` / ``safe`` flags, nodes pick up + ``_type`` annotations, and each dialect injects its own numeric cast + (Postgres renders ``a / b`` as ``CAST(a AS DOUBLE PRECISION) / b``). + + ``Paren`` and ``Cast`` collapse to their operand: the first carries no + meaning once a tree exists — it is how meaning is PRESERVED across the + string — and the second is a typing decision. Leaves reduce to their own + SQL, which is identical on both sides. + """ + if isinstance(node, (exp.Paren, exp.Cast)): + return _grouping_shape(node.this) + operands = [ + node.args.get("this"), node.args.get("expression"), + *(node.args.get("expressions") or []), + ] + operands = [o for o in operands if isinstance(o, exp.Expression)] + if not operands: + return node.sql() + return (type(node).__name__, tuple(_grouping_shape(o) for o in operands)) + + +def _reparses_to_the_same_tree(node: exp.Expression, dialect: str) -> bool: + """Whether the database will read back the tree we built.""" + reparsed = sqlglot.parse_one(node.sql(dialect=dialect), dialect=dialect) + return _grouping_shape(reparsed) == _grouping_shape(node) + + +class TestEveryOperatorPairSurvivesTheRoundTrip: + """The meta-test for the whole regrouping family. + + Every individual grouping bug in this PR has the same shape: we build one + tree and the database reads a different one. Asserting emitted STRINGS + catches them one at a time, and only once someone has thought of the shape. + + This asserts the property directly — render, re-parse, and compare the + trees modulo parens — over every ordered pair of operators in the + precedence table, in both operand positions. Note that re-parse STABILITY + alone would not do: ``a + b * c`` re-parses to a stable string while + meaning something other than the ``(a + b) * c`` we built. + """ + + # Grouped by result type. Feeding a boolean to an arithmetic operator is + # not a shape the binder builds, and the dialects wrap such an operand in + # their own numeric CAST — noise that says nothing about grouping. + ARITH = ["+", "-", "*", "/", "%"] + CMP = ["=", "!=", "<", "<=", ">", ">="] + BOOL = ["and", "or"] + DIALECTS = ["postgres", "sqlite", "mysql"] + + @staticmethod + def _leaf(name: str) -> exp.Expression: + return exp.column(name, table="t") + + def _binary(self, op: str, left, right): + return render_arithmetic(op, [left, right]) + + def _nested(self, inner: str): + return self._binary(inner, self._leaf("b"), self._leaf("c")) + + def _check(self, built, dialect, label) -> None: + assert _reparses_to_the_same_tree(built, dialect), ( + f"{label} regrouped: {built.sql(dialect=dialect)}" + ) + + @pytest.mark.parametrize("dialect", DIALECTS) + @pytest.mark.parametrize("outer", ARITH) + @pytest.mark.parametrize("inner", ARITH) + @pytest.mark.parametrize("position", ["left", "right"]) + def test_arithmetic_nested_in_arithmetic( + self, outer, inner, position, dialect, + ) -> None: + nested = self._nested(inner) + operands = ( + [nested, self._leaf("d")] if position == "left" + else [self._leaf("a"), nested] + ) + self._check( + self._binary(outer, *operands), dialect, + f"{outer} over {inner} ({position})", + ) + + @pytest.mark.parametrize("dialect", DIALECTS) + @pytest.mark.parametrize("outer", CMP) + @pytest.mark.parametrize("inner", ARITH) + @pytest.mark.parametrize("position", ["left", "right"]) + def test_arithmetic_nested_in_a_comparison( + self, outer, inner, position, dialect, + ) -> None: + nested = self._nested(inner) + operands = ( + [nested, self._leaf("d")] if position == "left" + else [self._leaf("a"), nested] + ) + self._check( + self._binary(outer, *operands), dialect, + f"{outer} over {inner} ({position})", + ) + + @pytest.mark.parametrize("dialect", DIALECTS) + @pytest.mark.parametrize("outer", BOOL) + @pytest.mark.parametrize("inner", CMP + BOOL) + @pytest.mark.parametrize("position", ["left", "right"]) + def test_predicate_nested_in_a_connector( + self, outer, inner, position, dialect, + ) -> None: + nested = self._nested(inner) + other = self._binary("<", self._leaf("d"), self._leaf("e")) + operands = [nested, other] if position == "left" else [other, nested] + self._check( + self._binary(outer, *operands), dialect, + f"{outer} over {inner} ({position})", + ) + + @pytest.mark.parametrize("dialect", DIALECTS) + @pytest.mark.parametrize("inner", CMP + BOOL) + def test_not_over_every_predicate(self, inner, dialect) -> None: + self._check( + render_arithmetic("not", [self._nested(inner)]), dialect, + f"not over {inner}", + ) + + @pytest.mark.parametrize("dialect", DIALECTS) + @pytest.mark.parametrize("inner", ARITH) + def test_negation_over_every_arithmetic_operator(self, inner, dialect) -> None: + self._check( + render_arithmetic("-", [self._nested(inner)]), dialect, + f"- over {inner}", + ) + + @pytest.mark.parametrize("dialect", DIALECTS) + @pytest.mark.parametrize("op", ["is", "is not"]) + @pytest.mark.parametrize("inner", ARITH + CMP) + def test_is_over_every_value_expression(self, op, inner, dialect) -> None: + self._check( + render_arithmetic(op, [self._nested(inner), exp.Null()]), dialect, + f"{op} over {inner}", + ) + + class TestGeneratorComposersShareTheGroupingPolicy: """The three LIVE generator composers had the same grouping holes. From cd8636b1fab09b856f52bbccfed1d9d6ebaafdeb Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Wed, 5 Aug 2026 22:32:00 +0200 Subject: [PATCH 24/98] =?UTF-8?q?DEV-1745:=20W5=20=E2=80=94=20the=20droppe?= =?UTF-8?q?d-filter=20warning=20contract,=20end=20to=20end?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Emission moves from mid-render to the engine boundary. The generator fired a bare UserWarning once per cross-model plan, so nested subplans double-fired for one user filter, and any path that never reached that render step said nothing at all. Nothing downstream could observe it either: SlayerResponse.warnings was typed to normalization warnings only, and no entry point rendered warnings of any kind. Now: collected across every plan in the pipeline (including nested rerooted subplans) during prepare, deduped per user filter on (location, original filter text), and emitted exactly once per execute at the outermost boundary. Ordering is load-bearing. The structured payload is built FIRST and the Python warnings.warn happens LAST, so under -W error the raise comes after a complete response rather than from the middle of rendering. D8: reasons for the same filter must AGREE, and disagreement raises rather than silently keeping the first. That required making the drop reason target-INDEPENDENT — it named the terminal model, so two plans dropping one filter produced two different reasons and would have tripped the check. SlayerResponse.warnings widens to a DISCRIMINATED union keyed on `kind`. Pydantic validates a List[SlayerWarning] down to the base class and would drop every subclass field on the way through; the discriminator makes each payload round-trip as itself. Surfacing: REST QueryResponse gains `warnings`; MCP appends them to every output format; CLI prints them to STDERR so stdout stays pipeable. A dropped filter changes which rows the answer covers, so it cannot be left to a field the caller might not read. Test harness fixes: the REST and MCP cases posted a nested {"query": {...}} envelope neither surface accepts (both take the query fields directly), and the MCP case used a get_tool() API this FastMCP version does not have. The explain case ran a real EXPLAIN against a database with no tables, so it now materialises them. Full non-integration suite: 9796 passed, 0 failed. Ruff clean. Co-Authored-By: Claude Opus 5 (1M context) --- slayer/api/server.py | 7 ++ slayer/cli.py | 20 ++++++ slayer/core/warnings.py | 14 +++- slayer/engine/cross_model_planner.py | 11 ++- slayer/engine/query_engine.py | 99 +++++++++++++++++++++++++- slayer/mcp/server.py | 28 ++++++-- slayer/sql/generator.py | 19 ++--- tests/test_dev1745_warning_contract.py | 61 ++++++++++++---- 8 files changed, 221 insertions(+), 38 deletions(-) diff --git a/slayer/api/server.py b/slayer/api/server.py index 57769b8e..21a2d4c8 100644 --- a/slayer/api/server.py +++ b/slayer/api/server.py @@ -97,6 +97,10 @@ class QueryResponse(BaseModel): columns: list[str] sql: str | None = None attributes: AttributesResponse | None = None + # DEV-1745 (W5/D2): advisories about the query itself — slack-normalization + # rewrites and filters that were dropped as unreachable. One list, each + # entry tagged with a ``kind`` discriminator the consumer switches on. + warnings: list[dict[str, Any]] = [] class IngestRequest(BaseModel): @@ -346,6 +350,9 @@ def _convert_meta(d: dict) -> dict[str, FieldMetadataResponse]: row_count=result.row_count, columns=result.columns, attributes=attributes, + warnings=[ + w.model_dump(mode="json") for w in (result.warnings or []) + ], ) if dry_run or explain: response.sql = result.sql diff --git a/slayer/cli.py b/slayer/cli.py index c752675d..3debf884 100644 --- a/slayer/cli.py +++ b/slayer/cli.py @@ -1225,6 +1225,24 @@ def _parse_cli_variables(args) -> dict: return out +def _print_query_warnings(result) -> None: + """Print query advisories to STDERR. + + Stderr specifically, so a piped ``slayer query`` keeps emitting clean data + on stdout while the operator still sees that a filter was dropped — + something that changes which rows the answer covers (DEV-1745 W5 / D2). + """ + for w in (getattr(result, "warnings", None) or []): + if getattr(w, "kind", None) == "unreachable_filter_dropped": + print( + f"warning: dropped filter {w.filter_text!r} " + f"(at {w.location}): {w.reason}", + file=sys.stderr, + ) + else: + print(f"warning: {w}", file=sys.stderr) + + def _run_query(args): # NOSONAR S3776 — argparse-driven dispatch; one straight-line function reads better than threaded helpers from slayer.engine.query_engine import SlayerQueryEngine @@ -1269,6 +1287,8 @@ def _run_query(args): # NOSONAR S3776 — argparse-driven dispatch; one straigh explain=bool(args.explain), ) + _print_query_warnings(result) + if args.dry_run: print(result.sql) return diff --git a/slayer/core/warnings.py b/slayer/core/warnings.py index eba71159..6211bd23 100644 --- a/slayer/core/warnings.py +++ b/slayer/core/warnings.py @@ -17,9 +17,9 @@ from __future__ import annotations -from typing import Literal, Optional +from typing import Annotated, Literal, Optional, Union -from pydantic import BaseModel +from pydantic import BaseModel, Field class SlayerWarning(BaseModel): @@ -66,6 +66,16 @@ class DroppedFilterWarning(SlayerWarning): reason: str +# The response carries a DISCRIMINATED union, not the bare base class: Pydantic +# validates a ``List[SlayerWarning]`` down to the base type and would silently +# drop every subclass field on the way through. Keyed on ``kind``, each payload +# round-trips as itself. +AnySlayerWarning = Annotated[ + Union[NormalizationWarning, DroppedFilterWarning], + Field(discriminator="kind"), +] + + class SlayerNormalizationWarning(UserWarning): """Carrier ``UserWarning`` for a ``NormalizationWarning`` payload. diff --git a/slayer/engine/cross_model_planner.py b/slayer/engine/cross_model_planner.py index 7ee27b9b..e50e2756 100644 --- a/slayer/engine/cross_model_planner.py +++ b/slayer/engine/cross_model_planner.py @@ -537,10 +537,15 @@ def _route_host_filters( elif route is FilterRoute.DROP_UNREACHABLE: dropped.append(UnreachableFilterDroppedWarning( filter_text=hf.text or hf.filter_id, + # Deliberately target-INDEPENDENT (D8). The same user filter is + # classified once per cross-model plan, and the boundary dedups + # those to one warning while asserting the reasons AGREE. A + # reason naming this plan's target would make two plans + # disagree about one filter and trip that assertion. reason=( - f"filter {hf.filter_id!r} references slot(s) outside " - f"the join path to {terminal_model.name!r}; " - f"unreachable filters are dropped." + f"filter {hf.filter_id!r} depends on join path(s) that are " + f"not reachable from the cross-model aggregate's CTE root; " + f"it still applies at the host, and is dropped from the CTE." ), )) return applied, where_ids, having_ids, dropped diff --git a/slayer/engine/query_engine.py b/slayer/engine/query_engine.py index 9ebfbbee..b102c2da 100644 --- a/slayer/engine/query_engine.py +++ b/slayer/engine/query_engine.py @@ -7,6 +7,7 @@ import decimal import logging import re +import warnings as _warnings_module from collections.abc import Callable from typing import Any, Dict, List, Optional @@ -36,7 +37,7 @@ list_valued_variable_names, substitute_variables, ) -from slayer.core.warnings import NormalizationWarning +from slayer.core.warnings import AnySlayerWarning, NormalizationWarning from slayer.core.recommend import ( CandidateCoverage, ItemPath, @@ -362,6 +363,82 @@ def _build_explain_sql(dialect: str, sql: str) -> str: return get_dialect(dialect).build_explain_sql(sql) +def _walk_cross_model_plans(planned): + """Every cross-model plan on ``planned``, including nested rerooted plans. + + A nested plan carries its OWN dropped-filter warnings for the same user + filter, which is why the old per-plan emission double-fired. + """ + for plan in getattr(planned, "cross_model_aggregate_plans", ()) or (): + yield plan + nested = getattr(plan, "rerooted_plan", None) + if nested is not None: + yield from _walk_cross_model_plans(nested) + + +def _stage_location(stages, index: int) -> str: + """Human-readable pointer to the stage a filter came from. + + Part of the dedup identity (D8), so it must distinguish two stages that + carry the SAME filter text — those are two distinct user filters. + """ + name = getattr(stages[index], "name", None) if index < len(stages) else None + return f"stage {name!r}.filters" if name else f"stages[{index}].filters" + + +def _collect_dropped_filter_warnings(*, planned_list, stages) -> List[Any]: + """Dropped-filter payloads for the whole pipeline, one per user filter. + + Identity is ``(location, original filter text)`` — stated in the terms the + author sees, because the contract is user-facing. The same filter dropped + by several cross-model plans is ONE warning. + + Reasons for the same filter must AGREE. A disagreement means two plans + reached different conclusions about one filter, which is a planner + inconsistency; raising beats silently keeping whichever came first. + """ + from slayer.core.warnings import DroppedFilterWarning + + by_identity: "dict[tuple[str, str], DroppedFilterWarning]" = {} + for index, planned in enumerate(planned_list): + location = _stage_location(stages, index) + for plan in _walk_cross_model_plans(planned): + for w in plan.dropped_filter_warnings or (): + identity = (location, w.filter_text) + existing = by_identity.get(identity) + if existing is None: + by_identity[identity] = DroppedFilterWarning( + filter_text=w.filter_text, + location=location, + reason=w.reason, + ) + elif existing.reason != w.reason: + raise ValueError( + f"Planner inconsistency: filter {w.filter_text!r} at " + f"{location} was dropped for two different reasons — " + f"{existing.reason!r} vs {w.reason!r}." + ) + return list(by_identity.values()) + + +def _emit_dropped_filter_warnings(response) -> None: + """Emit one ``UnreachableFilterDroppedWarning`` per dropped user filter. + + Called once, at the outermost boundary, AFTER the response is built. + """ + from slayer.core.errors import UnreachableFilterDroppedWarning + from slayer.core.warnings import DroppedFilterWarning + + for w in response.warnings or (): + if isinstance(w, DroppedFilterWarning): + _warnings_module.warn( + UnreachableFilterDroppedWarning( + filter_text=w.filter_text, reason=w.reason, + ), + stacklevel=3, + ) + + class SlayerResponse(BaseModel): """Response from a SLayer query.""" @@ -375,7 +452,7 @@ class SlayerResponse(BaseModel): # AST-resolvable dotted refs in raw SQL). Empty for queries that # arrived in canonical form. Surfaced alongside the result so # REST / MCP / CLI consumers can echo the rewrites back to authors. - warnings: List[NormalizationWarning] = PydanticField(default_factory=list) + warnings: List[AnySlayerWarning] = PydanticField(default_factory=list) @model_validator(mode="after") def _populate_columns(self) -> "SlayerResponse": @@ -744,7 +821,7 @@ async def execute( main_query, named_queries, prefer_data_source = await self._normalize_input( query, runtime_kwarg=runtime_kwarg, prefer_data_source=data_source ) - return await self._execute_pipeline( + response = await self._execute_pipeline( query=main_query, named_queries=named_queries, runtime_kwarg=runtime_kwarg, @@ -755,6 +832,14 @@ async def execute( original_input=query, original_data_source=data_source, ) + # DEV-1745 (W5): the ONE Python-warnings emission, at the outermost + # boundary and after the structured response exists. Ordering is + # load-bearing — under ``-W error`` this raises, and the contract is + # that the payload was fully built first. Emitting here rather than + # mid-render also makes it path-independent: dry_run, explain and a + # real execute all reach it. + _emit_dropped_filter_warnings(response) + return response async def _normalize_input( # NOSONAR S3776 — public dispatch over str/dict/list/SlayerQuery; splitting hides the input-shape contract self, @@ -1027,6 +1112,14 @@ async def _prepare_pipeline( # NOSONAR S3776 — linear pipeline (resolve→enr planned_list = plan_stages(queries=stages, bundle=bundle) root_planned = planned_list[-1] + # DEV-1745 (W5): collect dropped-filter payloads across EVERY plan in + # the pipeline (including nested rerooted subplans) and dedup them per + # user filter. Collection happens here, with the plans in hand; the + # Python-warnings emission happens later, at the outermost boundary. + slack_warnings.extend(_collect_dropped_filter_warnings( + planned_list=planned_list, stages=stages, + )) + dialect = self._dialect_for_type(datasource.type) sql = generate_planned_stages( planned_list, bundle=bundle, dialect=dialect, diff --git a/slayer/mcp/server.py b/slayer/mcp/server.py index d7e4ab37..e803bc33 100644 --- a/slayer/mcp/server.py +++ b/slayer/mcp/server.py @@ -1978,13 +1978,33 @@ def _format_csv(data: list[dict[str, Any]], columns: list[str]) -> str: return "\n".join(lines) +def _format_warnings(result: SlayerResponse) -> str: + """Advisories about the query, appended to every output format. + + A dropped filter changes which rows the answer covers, so it cannot be + left to a field the caller might not read (DEV-1745 W5 / D2). + """ + lines = [] + for w in (result.warnings or []): + if getattr(w, "kind", None) == "unreachable_filter_dropped": + lines.append( + f" - dropped filter {w.filter_text!r} " + f"(at {w.location}): {w.reason}" + ) + else: + lines.append(f" - {getattr(w, 'rule_id', w.kind)}: {w}") + return "" if not lines else "\n\nWarnings:\n" + "\n".join(lines) + + def _format_output(result: SlayerResponse, fmt: str) -> str: """Format query output in the requested format.""" if fmt == "csv": - return _format_csv(data=result.data, columns=result.columns) - if fmt == "markdown": - return result.to_markdown() - return _format_json(data=result.data, columns=result.columns) + body = _format_csv(data=result.data, columns=result.columns) + elif fmt == "markdown": + body = result.to_markdown() + else: + body = _format_json(data=result.data, columns=result.columns) + return body + _format_warnings(result) def _format_field_meta(entries: dict[str, Any]) -> list[str]: diff --git a/slayer/sql/generator.py b/slayer/sql/generator.py index d43fe36c..cca5db0a 100644 --- a/slayer/sql/generator.py +++ b/slayer/sql/generator.py @@ -4692,19 +4692,12 @@ def _add_local_aux_slots( (a, a) for a in grain_aliases ] - # Codex MED fold-in: surface dropped-filter warnings from each - # plan via Python ``warnings`` so callers using - # ``warnings.catch_warnings()`` see what was dropped. The - # generator is the boundary that "renders" the plan; warnings - # are inert until something is actually compiled. - import warnings as _warnings_mod - for plan in planned_query.cross_model_aggregate_plans: - for w in plan.dropped_filter_warnings: - _warnings_mod.warn( - str(w), - UserWarning, - stacklevel=2, - ) + # DEV-1745 (W5): dropped-filter warnings are NOT emitted here. This + # emission fired once per cross-model plan — so nested subplans + # double-fired for one user filter — and never fired at all on a path + # that did not reach this render step. It is now collected across every + # plan at the ENGINE boundary, deduped per user filter, and emitted + # exactly once per execute. The plans still carry the payloads. # Build the combined SELECT: SELECT _base., # _cm_*. [AS ""] FROM _base [LEFT JOIN | diff --git a/tests/test_dev1745_warning_contract.py b/tests/test_dev1745_warning_contract.py index 3e274fb4..ea0d509b 100644 --- a/tests/test_dev1745_warning_contract.py +++ b/tests/test_dev1745_warning_contract.py @@ -25,6 +25,7 @@ from __future__ import annotations +import pathlib import tempfile import warnings @@ -103,13 +104,41 @@ def _query(*, extra_filters: list | None = None) -> SlayerQuery: ) -async def _engine(tmpdir: str) -> SlayerQueryEngine: +_DDL = [ + "CREATE TABLE orders (id INTEGER, customer_id INTEGER, shipper_id INTEGER," + " warehouse_id INTEGER, status VARCHAR, amount DOUBLE)", + "CREATE TABLE customers (id INTEGER, revenue DOUBLE)", + "CREATE TABLE warehouses (id INTEGER, code VARCHAR)", + "CREATE TABLE shippers (id INTEGER, cost DOUBLE)", +] + + +async def _engine(tmpdir: str, *, with_tables: bool = False) -> SlayerQueryEngine: + """Engine over a DuckDB datasource. + + ``database`` MUST be set: ``explain=True`` opens a real connection, and a + DuckDB datasource with database=None writes a file literally named "None" + into the working directory. + + ``with_tables`` materialises the physical tables in a file-backed database. + ``explain`` runs a real EXPLAIN, which the backend rejects outright if the + tables do not exist — so the paths that actually touch the database need + something to point at. + """ storage = YAMLStorage(base_dir=tmpdir) - # ``database`` MUST be set: ``explain=True`` opens a real connection, and a - # DuckDB datasource with database=None writes a file literally named "None" - # into the working directory. + database = ":memory:" + if with_tables: + import duckdb + + database = str(pathlib.Path(tmpdir) / "w.duckdb") + con = duckdb.connect(database) + try: + for ddl in _DDL: + con.execute(ddl) + finally: + con.close() await storage.save_datasource( - DatasourceConfig(name="test", type="duckdb", database=":memory:") + DatasourceConfig(name="test", type="duckdb", database=database) ) for m in (_orders(), _customers(), _warehouses(), _shippers()): await storage.save_model(m, _validate=False) @@ -250,7 +279,7 @@ class TestEmissionIsBoundaryNotRender: ]) async def test_warning_emitted_on_every_execute_mode(self, kwargs) -> None: with tempfile.TemporaryDirectory() as d: - engine = await _engine(d) + engine = await _engine(d, with_tables="explain" in kwargs) resp = await engine.execute(_query(), **kwargs) assert len(_dropped(resp)) == 1, ( f"no dropped-filter payload for execute(**{kwargs})" @@ -496,10 +525,11 @@ async def _seed(): asyncio.run(_seed()) client = TestClient(create_app(storage=storage)) - resp = client.post("/query", json={ - "query": _query().model_dump(mode="json", exclude_none=True), - "dry_run": True, - }) + # QueryRequest carries the query fields at the TOP level, with + # dry_run alongside them — there is no nested "query" envelope. + payload = _query().model_dump(mode="json", exclude_none=True) + payload["dry_run"] = True + resp = client.post("/query", json=payload) assert resp.status_code == 200, resp.text body = resp.json() assert "warnings" in body, ( @@ -523,9 +553,14 @@ async def test_mcp_query_output_mentions_the_dropped_filter(self) -> None: for m in (_orders(), _customers(), _warehouses()): await storage.save_model(m, _validate=False) server = create_mcp_server(storage=storage) - tool = await server.get_tool("query") - result = await tool.run({ - "query": _query().model_dump(mode="json", exclude_none=True), + # The MCP query tool takes the query fields as its own typed + # arguments — no nested "query" envelope, and `dimensions` is a + # list of plain strings rather than the SlayerQuery dict form. + result = await server.call_tool("query", { + "source_model": "orders", + "dimensions": ["status"], + "measures": [{"formula": "customers.revenue:sum"}], + "filters": [DROPPED_FILTER], "dry_run": True, }) text = str(result) From ae2c7205dd95ab7bd50db43f2f14e5ff0620d1a8 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Wed, 5 Aug 2026 22:32:53 +0200 Subject: [PATCH 25/98] DEV-1744: cover boolean-valued comparison operands in the round-trip matrix (Codex) --- tests/test_dev1744_value_expr.py | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/tests/test_dev1744_value_expr.py b/tests/test_dev1744_value_expr.py index 3ea84815..af6ed0fa 100644 --- a/tests/test_dev1744_value_expr.py +++ b/tests/test_dev1744_value_expr.py @@ -2258,6 +2258,28 @@ def test_arithmetic_nested_in_a_comparison( f"{outer} over {inner} ({position})", ) + @pytest.mark.parametrize("dialect", DIALECTS) + @pytest.mark.parametrize("outer", CMP) + @pytest.mark.parametrize("inner", CMP + BOOL) + @pytest.mark.parametrize("position", ["left", "right"]) + def test_predicate_nested_in_a_comparison( + self, outer, inner, position, dialect, + ) -> None: + """A comparison over BOOLEAN operands — ``(a = b) = (c = d)``, + ``(a AND b) = c``. Type-coherent (SQL comparisons take booleans) and + reachable: Python's grammar reads ``(a == b) == c`` as a NESTED + comparison, not a chained one, so the Mode-B binder does build it. + This arm sits exactly on the comparison/connector precedence boundary, + which is where ``IS`` and the non-associativity rule both live. + """ + nested = self._nested(inner) + other = self._binary("<", self._leaf("d"), self._leaf("e")) + operands = [nested, other] if position == "left" else [other, nested] + self._check( + self._binary(outer, *operands), dialect, + f"{outer} over {inner} ({position})", + ) + @pytest.mark.parametrize("dialect", DIALECTS) @pytest.mark.parametrize("outer", BOOL) @pytest.mark.parametrize("inner", CMP + BOOL) From 501cbe752e9923db7f804b6e16943e3fecd6cfd2 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Wed, 5 Aug 2026 22:35:31 +0200 Subject: [PATCH 26/98] =?UTF-8?q?DEV-1745:=20W7=20=E2=80=94=20docs,=20skil?= =?UTF-8?q?ls,=20and=20the=20DECISIONS=20entry?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit REST reference and interfaces docs describe the new `warnings` field on the query response, with the `kind` discriminator table so consumers switch on kind rather than on the presence of a field. The slayer-query skill notes it on SlayerResponse. DECISIONS.md records the doctrine this PR lands: the one Mode-A door and why it adds no qualification pass, loud parse failure, plan-time outer-WHERE routing, structural reachability and where its summary lives, the boundary warning contract, the two live bugs fixed in passing, and the P-J inventory of symbols now unreferenced by production. Co-Authored-By: Claude Opus 5 (1M context) --- .claude/skills/slayer-query.md | 5 ++++- DECISIONS.md | 2 ++ docs/interfaces/rest-api.md | 15 ++++++++++++++- docs/reference/rest-api.md | 15 ++++++++++++++- 4 files changed, 34 insertions(+), 3 deletions(-) diff --git a/.claude/skills/slayer-query.md b/.claude/skills/slayer-query.md index 4bab5cd1..49f63035 100644 --- a/.claude/skills/slayer-query.md +++ b/.claude/skills/slayer-query.md @@ -88,7 +88,10 @@ Result column naming: `revenue:sum` → `orders.revenue_sum` (colon becomes unde engine = SlayerQueryEngine(storage=storage) # Async (most callers — REST/MCP): -result = await engine.execute(query=query) # SlayerResponse with .data, .columns, .row_count, .sql, .attributes +result = await engine.execute(query=query) # SlayerResponse with .data, .columns, .row_count, .sql, .attributes, .warnings +# .warnings holds advisories, each tagged with .kind — "normalization" for an input +# rewrite, "unreachable_filter_dropped" for a filter dropped from a cross-model CTE +# (it still applies at the host). Empty for a clean query. # With runtime variables (highest precedence — wins over query.variables / model defaults): result = await engine.execute(query=query, variables={"region": "US"}) diff --git a/DECISIONS.md b/DECISIONS.md index 9e41f3ef..c729475e 100644 --- a/DECISIONS.md +++ b/DECISIONS.md @@ -97,3 +97,5 @@ implementation detail. Include issue refs when known. - 2026-08-04 — Declared list-valued `{variable}` coercion (DEV-1730 follow-up): a scalar supplied for a variable the model declares `list_valued` is wrapped into a one-element list before Mode-A substitution, so an importer-generated `col IN ({var})` renders `IN ('US')` rather than the unquoted `IN (US)`. The generic scalar rule (author writes the quotes, so `{var}` also works in numeric/fragment positions like `amount >= {floor}` and `{d}::TIMESTAMP`) is CORRECT and unchanged — it just presumes an author who can see the SQL position, which a machine-generated fixed template does not have; the caller cannot supply per-element quotes through parentheses the importer wrote. Silent-wrong-answer risk drove the fix over a raise: `region IN (US)` parses as a column reference, so it fails at the database with a confusing message, or resolves against a real column and returns wrong rows. Opt-in is a front-end-NEUTRAL flag: the Cube converter writes `list_valued: ref.kind == "string"` into each `meta.cube_variables` entry (arrow forms splice pre-quoted scalars and stay `False`), and the engine reads only that flag — never Cube's `kind` taxonomy — so a future list-shaped front-end opts in the same way. Coercion lives at the single Mode-A choke point `_substitute_model_sql_surfaces` (execution and the `_render_probe_model` type-probe both route through it, so it cannot be bypassed) via `coerce_declared_list_variables` / `list_valued_variable_names` in `slayer/core/query.py`. Scope is deliberately narrow: only `str`/`int`/`float`/`bool` are wrapped; `list`/`tuple` pass through (the **empty list still raises** — "no filter" belongs to an optional block or a sentinel default); `None`/`dict` are left for `_render_variable_value` to reject with its own naming error; hand-written models declare nothing and are untouched. Follow-on from the same review: `declares_variables(model)` (any non-empty `meta.cube_variables`) now also defeats the DEV-1625 zero-variable fast path, via the shared `_model_needs_substitution_pass` predicate used by both `_substitute_model_sql_surfaces` and `_render_probe_model`. This closes the fast-path hole for a GENERATED model whose pushdowns are all required (no `{? ?}` block to force the pass): such a model used to emit a bare `{var}` into the SQL on a zero-variable call instead of raising the documented missing-variable error. The hole stays open — deliberately — for hand-written models, which declare nothing and keep the raw-brace-literal protection (`'{1,2,3}'`). The `list_valued` flag is matched with `is True`, not truthiness, since `meta` is user-extensible and a stray `1` or the string `"false"` must not switch substitution semantics. The bag is also SELF-IDENTIFYING — an entry counts only with a string `member` (the shape every importer writes) — so a hand-written `meta` that reuses the `cube_variables` key is not mistaken for generated SQL and silently stripped of its brace-literal protection. - 2026-08-05 — One naming authority + one ValueKey renderer (DEV-1744, PR 1 of the DEV-1742 consolidation). Four consequences worth recording. (1) **Cross-model CTE dedup is now structural.** `_cm_` CTE names were minted by a doubly-lossy helper (`flat_name`, itself documented non-injective, then a non-identifier `re.sub`) and the resulting string doubled as the plan's identity key in `seen_cm`. Two reachable failures followed: two measures whose canonical aliases differ only in case emitted two `_cm_` names that fold together on every case-folding dialect (the collision belt raised — `_wm_` had been retrofitted onto the allocator years earlier, `_cm_` never was), and two genuinely distinct aggregates that sanitised alike made the second skip the loop body, leaving its join-back and column-alias maps unwritten and raising `KeyError` downstream. The dedup key is now the typed `AggregateKey` plus the source relation; the name is allocator-minted and stored once, so the five sites that used to re-derive it now read it. Deliberately NOT keyed on the canonical alias: `canonical_agg_name` omits `column_filter_key`, so a filtered and an unfiltered aggregate over one column share an alias while needing two CTEs — deduping on the string would silently merge them, a wrong answer rather than a crash. (2) **One ScalarCall policy.** Two of the five renderers returned `exp.Anonymous` passthrough, so `ifnull` reached Postgres — which has no `IFNULL` — while the same key emitted `COALESCE` from a projection. All six render paths now call one `render_scalar_call`. Transpiling alone was NOT sufficient: `exp.func("LOG10", x)` normalises to a generic `Log(10, x)` that re-emits as `LOG(10, x)`, wrong on dialects with a native single-arg `LOG10`, so the policy is transpile-then-log-alias-rewrite. Consequence surfaced and approved: `concat` now emits the `||` operator on Postgres/DuckDB where it previously emitted `CONCAT(...)`. That is a semantic change as well as a spelling one — Postgres `CONCAT()` ignores NULL operands, `||` propagates them. Ratified as the kept behavior: the projection path has always emitted `||`, so before this change SLayer disagreed with itself between filters and projections on the same input, and consistency was chosen over preserving the filter-side NULL tolerance. (3) **`ScopeFrame._model_for` raises** instead of falling back to the root model; the fallback turned a wiring bug into a wrong answer by expanding a different model's derived SQL. (4) **Named P-F carve-outs**, recorded rather than omitted: the T-SQL ORDER-BY-detach rewrite and the stage-schema wrapper take `_outer` / `_stage_inner` as shared CONSTANTS rather than allocator-minted names (the former is a post-generation AST pass with no allocator in reach, and a later PR rebuilds that machinery); `base` / `_base` / `_combined` keep their literal names but are reserved into the allocator. Superseded code is retained and callable per the chain's operating rule — deletion happens in the final PR. - 2026-08-05 — Renderer call-site migration deferred to the scope-assembly PR (DEV-1744 → DEV-1746). PR 1 lands the complete, tested `render_value_key` API but does NOT reroute the generator's own render paths through it; only the ScalarCall policy is genuinely shared (all six paths call `render_scalar_call`). Deferred because the filter and composite paths carry state the context does not yet consume — the local-aggregate HAVING branch with its slot lookup and inline-aggregate emission, the first/last ranked state, the filter-side CAST policy, and the rn-suffix / filtered-rank / composite-alias / resolved-kwarg maps — and because the cross-scope paths are the ones that should supply `resolve(consumer=...)` its first production caller, which is scope-assembly work by nature. Splitting the reroute across two PRs would move that state twice. Full detail, per call site, in the `slayer/sql/render/value_expr.py` module docstring. + +- 2026-08-05 — One Mode-A door, plan-time filter routing, structural reachability (DEV-1745, PR 2 of the DEV-1742 series). **P-A:** every fragment of free (Mode-A) SQL now enters a SELECT scope through ONE door — `ScopeFrame.enter_predicate` / `enter_expression`, two surfaces over one implementation differing only in the parse helper (the `SELECT 1 WHERE …` statement-keyword guard for predicates). The door prequotes, parses, scans, expands, re-parses, re-scans, unions both scans into `join_paths`, then applies Law 2. Join discovery is a side effect of entering, so a caller cannot forget it; the DEV-1494 dual-scan contract is preserved because a dotted ref whose derived column inlines to a constant is only visible to the PRE-expansion scan. The surface's grammar is a static property of the field being read (`Column.filter` and model `filters` are predicates, `Column.sql` is an expression) — deliberately no content sniffing and no "try one grammar then the other" retry, either of which would put classification back into render time. **The door adds NO qualification pass:** `expand_derived_refs_sync` already qualifies, against the OWNING model's canonical alias, and deliberately leaves a node alone when its alias path does not resolve because that is an opaque CTE / subquery reference — a blanket pass against `root_relation` would corrupt exactly those. `SQLGenerator._parse` and `_parse_predicate` were byte-identical apart from one line and now delegate to a shared `slayer/sql/render/parse.py`, which is what lets the door take over a call site without changing the SQL it emits. **Failure is loud:** an unparseable Mode-A fragment raises `ModeASqlParseError` carrying the fragment and its location. Three swallow-all `except Exception` lanes leave the production path, including `_filter_join_paths._scan`, which converted a parse failure into ZERO join paths — emitting a query missing its joins rather than reporting the problem. Two consequences on emitted SQL were accepted explicitly: undeclared bare identifiers in Mode-A text are now qualified against the scope root (the old fallbacks qualified only DECLARED columns, leaving the rest to bind to whatever was in scope after re-rendering inside a rerooted CTE), and the shifted CTE renders its model filter from the door's AST rather than passing regex-substituted text through, so sqlglot's canonical form appears. **P-D:** the outer combined-SELECT WHERE routing is decided by the planner (`PlannedQuery.outer_where_filter_ids`) and consumed verbatim; the generator's render-time re-walk of `filters_by_phase` is deleted. **Reachability is structural (5.3):** `classify_host_filter` routed derived-column references by asking whether the declaring model's NAME appeared anywhere in `target_path`. That flat membership test called a SIBLING branch reachable whenever it shared a name with the target path, and called a host-model derived column whose `Column.sql` crossed INTO the target host-local. It is replaced by one rule for every key kind — a dependency is reachable iff its anchored join path is a PREFIX of `target_path` — computed per filter at plan time by `slayer/engine/filter_reachability.py`, recursively over the whole key tree, failing CLOSED on an unhandled kind. The summary lives on `PlannedQuery`, NOT on `ColumnSqlKey` (interned, and `_reroot_path_ref` copies unknown fields through rerooting stale) and NOT on `ValueSlot` (`filter_referenced_slot_ids` silently skips keys with no interned slot — a filter-only derived column is exactly such a key — and slots are copied wholesale into nested plans); it is recomputed per plan so the paths always mean what they say relative to the root they were anchored at. **Warnings (5.5):** dropped-filter emission moves from mid-render (once per cross-model plan, so nested subplans double-fired, and silent on any path that never reached that step) to the engine boundary — collected across every plan, deduped per user filter on `(location, original text)`, with drop reasons required to AGREE (disagreement raises rather than keeping the first, which is why the reason is now target-independent). `SlayerResponse.warnings` widens to a `kind`-discriminated union and is surfaced by REST, MCP and the CLI (stderr, so stdout stays pipeable). **Two live bugs fixed in passing:** a `Column.sql` that is exactly one bare reference to another derived column was silently not expanded, because `col.replace()` is a no-op when that column IS the parsed root; and a `_cm_` CTE never registered the joins its aggregation template fragments crossed, emitting `SUM(customers.spend * regions.weight) FROM customers` — SQL no database accepts. **P-J:** the superseded helpers stay callable and test-pinned rather than deleted (state 1); newly unreferenced by production are `_render_model_filter_sql`, `_filter_join_paths`, `_render_mode_a_predicate`, `_expand_degenerate_derived_root`, `_column_ref_is_derived`, `_predicate_references_derived` and `_qualify_mode_a_sql_filter`. `_qualify_column_filter_sql` remains live via `_expand_column_filter_sql`'s bundle-less branch, and `_BARE_IDENT_RE` remains in use outside the Mode-A surfaces. diff --git a/docs/interfaces/rest-api.md b/docs/interfaces/rest-api.md index 5d189c09..8bb0a9e7 100644 --- a/docs/interfaces/rest-api.md +++ b/docs/interfaces/rest-api.md @@ -68,10 +68,23 @@ Response: {"orders.status": "pending", "orders._count": 15} ], "row_count": 2, - "columns": ["orders.status", "orders._count"] + "columns": ["orders.status", "orders._count"], + "warnings": [] } ``` +`warnings` carries advisories about the query itself; it is `[]` for a clean +query. Every entry has a `kind` discriminator — switch on it rather than on the +presence of a field: + +| `kind` | Meaning | Extra fields | +| -- | -- | -- | +| `normalization` | The input was rewritten to canonical form | `rule_id`, `original`, `normalized`, `location` | +| `unreachable_filter_dropped` | A filter could not be applied inside a cross-model CTE and was dropped from it (it still applies at the host) | `filter_text`, `location`, `reason` | + +A dropped filter changes which rows the answer covers, so it is worth surfacing +to whoever wrote the query. + ### Models ``` diff --git a/docs/reference/rest-api.md b/docs/reference/rest-api.md index 1ee62bb4..9507745f 100644 --- a/docs/reference/rest-api.md +++ b/docs/reference/rest-api.md @@ -49,10 +49,23 @@ Response: {"orders.status": "pending", "orders._count": 15} ], "row_count": 2, - "columns": ["orders.status", "orders._count"] + "columns": ["orders.status", "orders._count"], + "warnings": [] } ``` +`warnings` carries advisories about the query itself; it is `[]` for a clean +query. Every entry has a `kind` discriminator — switch on it rather than on the +presence of a field: + +| `kind` | Meaning | Extra fields | +| -- | -- | -- | +| `normalization` | The input was rewritten to canonical form | `rule_id`, `original`, `normalized`, `location` | +| `unreachable_filter_dropped` | A filter could not be applied inside a cross-model CTE and was dropped from it (it still applies at the host) | `filter_text`, `location`, `reason` | + +A dropped filter changes which rows the answer covers, so it is worth surfacing +to whoever wrote the query. + The body accepts the same fields as a `SlayerQuery`, plus `dry_run`, `explain`, and `variables`. Notable optional fields: - `whole_periods_only` (bool) — snap date filters to bucket boundaries. From af4af8b9252fa685023a0e372b5e52cecdc5630a Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Wed, 5 Aug 2026 22:41:21 +0200 Subject: [PATCH 27/98] DEV-1744: fix the ClickHouse integration assertion missed by the SUBSTR->SUBSTRING change --- .../test_integration_clickhouse.py | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/tests/integration/test_integration_clickhouse.py b/tests/integration/test_integration_clickhouse.py index c537da3e..b1430d02 100644 --- a/tests/integration/test_integration_clickhouse.py +++ b/tests/integration/test_integration_clickhouse.py @@ -234,12 +234,17 @@ async def test_sum_measure(self, clickhouse_env: SlayerQueryEngine) -> None: async def test_string_hygiene_functions_execute( self, clickhouse_env: SlayerQueryEngine ) -> None: - """DEV-1703 Phase 2: the typed pipeline emits string functions - UPPERCASE (``LOWER(...)`` / ``SUBSTR(...)``) where the legacy path - emitted ClickHouse's native lowercase spelling. ClickHouse function - names are case-sensitive in general, so this pins that the standard - SQL aliases really do resolve on a live server rather than trusting - that the emitted SQL merely looks plausible. + """The typed pipeline emits string functions UPPERCASE + (``LOWER(...)`` / ``SUBSTRING(...)``) where the legacy path emitted + ClickHouse's native lowercase spelling. ClickHouse function names are + case-sensitive in general, so this pins that the standard SQL aliases + really do resolve on a live server rather than trusting that the + emitted SQL merely looks plausible. + + ``substr`` now reaches the server as ``SUBSTRING``: routing every + scalar through one dialect-aware policy means sqlglot transpiles the + call to the target's own spelling instead of passing the DSL name + through verbatim. """ lowered = SlayerQuery( source_model="orders", @@ -257,7 +262,7 @@ async def test_string_hygiene_functions_execute( ) result = await clickhouse_env.execute(query=subs) assert result.data[0]["orders._count"] > 0 - assert "SUBSTR(" in (result.sql or ""), result.sql + assert "SUBSTRING(" in (result.sql or ""), result.sql async def test_avg_measure(self, clickhouse_env: SlayerQueryEngine) -> None: query = SlayerQuery(source_model="orders", measures=[{"formula": "avg_amount:avg"}]) From 891dca715c123efeefa86f89f9e0fed08239bb4f Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Wed, 5 Aug 2026 22:44:10 +0200 Subject: [PATCH 28/98] DEV-1745: fix host-locality for a derived column that is BOTH local and crossing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Found by the Codex implementation review. key_has_host_local_ref inferred "not host-local" from "its expansion crossed something". A host-declared derived column can do both: `amount * customers.balance` crosses into customers AND depends on the host-local `amount`. The filter therefore propagated into a customers-rooted CTE, which emits SQL referencing an unbound orders.amount. Host-locality is now tested directly — does the expanded SQL reference a column bound to the anchor relation — rather than inferred from the crossed set. The purely-crossing case still propagates, so this is not blanket caution; both directions are pinned by tests, and the new one fails without the fix. Also reviewed and NOT changed, with reasons: * A derived ref that expands transitively to a constant loses the intermediate join path. Real, but identical to the behaviour of the _filter_join_paths dual scan this replaces — it scanned exactly the same two representations (raw and fully expanded). Pre-existing, not a regression of this PR. * key_has_host_local_ref skips the AggregateKey subtree. Matches the prior classifier, which routed aggregates solely by source.path and never inspected args/kwargs/column_filter either. * Reference-free filters now reach DROP_HOST_LOCAL where they previously reached STAY_AT_HOST_POST. No behavioural difference: the routing helper treats the two identically — "neither propagated nor warned". * Dedup identity (location, filter text) is D8 as ratified, not an oversight. * parse_one accepting only the first statement of a multi-statement fragment is pre-existing behaviour on trusted authored model SQL; tightening it would reject models that parse today. Full non-integration suite 9798 passed. SQLite + DuckDB integration 118 passed. Co-Authored-By: Claude Opus 5 (1M context) --- slayer/engine/filter_reachability.py | 56 ++++++++++++++++++++++++---- tests/test_dev1745_reachability.py | 40 ++++++++++++++++++-- 2 files changed, 85 insertions(+), 11 deletions(-) diff --git a/slayer/engine/filter_reachability.py b/slayer/engine/filter_reachability.py index 2e397231..fd51a89e 100644 --- a/slayer/engine/filter_reachability.py +++ b/slayer/engine/filter_reachability.py @@ -33,6 +33,8 @@ from typing import List, Tuple +from sqlglot import exp + from slayer.core.errors import SlayerError from slayer.core.keys import ( AggregateKey, @@ -85,31 +87,41 @@ def _prefixes(path: Path) -> List[Path]: return [tuple(path[: i + 1]) for i in range(len(path))] -def _derived_sql_paths( +def _expanded_derived_ast( *, key: ColumnSqlKey, anchor_model, anchor_relation: str, bundle, -) -> List[Path]: - """Join paths the expansion of a derived column's ``Column.sql`` crosses. +): + """The parsed, expanded AST of a derived column's ``Column.sql``. Expanded with the same anchoring convention ``ScopeFrame`` uses — at the ``__``-path alias with ``is_root=False`` when the column lives on a joined model — so the refs inside come out already prefixed by the key's own path - and the anchor-rooted scan resolves them without further adjustment. + and an anchor-rooted scan resolves them without further adjustment. """ model = ( anchor_model if key.model == getattr(anchor_model, "name", None) else bundle.get_referenced_model(key.model) ) if model is None: - return [] + return None col = next((c for c in model.columns if c.name == key.column_name), None) if col is None or not col.sql: - return [] + return None alias_path = "__".join(key.path) if key.path else anchor_relation expanded = _expand_derived_refs_any_dialect( sql=col.sql, model=model, alias_path=alias_path, bundle=bundle, ) - parsed = _parse_filter_sql_any_dialect(expanded or col.sql) + return _parse_filter_sql_any_dialect(expanded or col.sql) + + +def _derived_sql_paths( + *, key: ColumnSqlKey, anchor_model, anchor_relation: str, bundle, +) -> List[Path]: + """Join paths the expansion of a derived column's ``Column.sql`` crosses.""" + parsed = _expanded_derived_ast( + key=key, anchor_model=anchor_model, + anchor_relation=anchor_relation, bundle=bundle, + ) if parsed is None: return [] return list(collect_root_scope_joined_paths( @@ -120,6 +132,34 @@ def _derived_sql_paths( )) +def _derived_sql_touches_anchor( + *, key: ColumnSqlKey, anchor_model, anchor_relation: str, bundle, +) -> bool: + """Whether a derived column's expansion references the ANCHOR relation. + + Tested directly rather than inferred from "it crossed nothing". A derived + column can do BOTH: ``amount * customers.rate`` crosses into ``customers`` + AND depends on the host-local ``amount``. Treating a non-empty crossed set + as proof of non-locality would propagate that filter into a + ``customers``-rooted CTE, where ``orders.amount`` is not bound. + + Expansion qualifies host-local refs to ``anchor_relation``, so those are + exactly the columns carrying that table (or, defensively, none at all). + """ + parsed = _expanded_derived_ast( + key=key, anchor_model=anchor_model, + anchor_relation=anchor_relation, bundle=bundle, + ) + if parsed is None: + # Nothing resolvable to inspect — a bare column name on the anchor. + return True + for col in parsed.find_all(exp.Column): + table = col.args.get("table") + if table is None or table.name == anchor_relation: + return True + return False + + def compute_key_join_paths( *, key, anchor_model, anchor_relation: str, bundle, ) -> Tuple[Path, ...]: @@ -233,7 +273,7 @@ def _walk(node) -> None: if isinstance(node, ColumnSqlKey): if node.path: return - if not _derived_sql_paths( + if _derived_sql_touches_anchor( key=node, anchor_model=anchor_model, anchor_relation=anchor_relation, bundle=bundle, ): diff --git a/tests/test_dev1745_reachability.py b/tests/test_dev1745_reachability.py index 8a0ed333..ac7d15db 100644 --- a/tests/test_dev1745_reachability.py +++ b/tests/test_dev1745_reachability.py @@ -99,6 +99,9 @@ def _orders() -> SlayerModel: # derived, local, purely local sql Column(name="host_derived_local", sql="amount * 2", type=DataType.DOUBLE), + # derived, host-declared, sql that is BOTH host-local AND crossing + Column(name="host_derived_mixed", sql="amount * customers.balance", + type=DataType.DOUBLE), # derived referencing TWO models Column(name="multi_model", sql="customers.balance + customers__regions.population", @@ -274,15 +277,19 @@ def _route(self, key, *, target_path, phase=Phase.ROW): def _routing(self, key, *, phase=Phase.ROW): from slayer.engine.cross_model_planner import HostFilterRouting - from slayer.engine.filter_reachability import compute_key_join_paths + from slayer.engine.filter_reachability import ( + compute_key_join_paths, + key_has_host_local_ref, + ) - paths = compute_key_join_paths( + kwargs = dict( key=key, anchor_model=_orders(), anchor_relation="orders", bundle=_bundle(), ) return HostFilterRouting( filter_id="f1", phase=phase, referenced_slot_ids=[], text="", - crossed_join_paths=paths, + crossed_join_paths=compute_key_join_paths(**kwargs), + has_host_local_ref=key_has_host_local_ref(**kwargs), ) def test_sibling_branch_is_not_reachable(self) -> None: @@ -350,6 +357,33 @@ def test_mixed_reachable_and_unreachable_drops(self) -> None: "reachability is an ALL-dependencies predicate" ) + def test_mixed_local_and_crossing_derived_stays_at_host(self) -> None: + """A host-declared derived column can be BOTH. + + ``amount * customers.balance`` crosses into ``customers``, so its + crossed set is non-empty — but it also depends on the host-local + ``amount``, which a customers-rooted CTE cannot bind. Inferring + "not host-local" from "it crossed something" would propagate this and + emit SQL referencing an unbound ``orders.amount``. + """ + from slayer.engine.cross_model_planner import FilterRoute + + route = self._route( + _derived("host_derived_mixed"), target_path=("customers",), + ) + assert route == FilterRoute.DROP_HOST_LOCAL + + def test_purely_crossing_derived_still_propagates(self) -> None: + """The counter-case, so the fix above is not just blanket caution: + a host-declared derived column whose sql reaches ONLY into the target + resolves inside the target's scope and still propagates.""" + from slayer.engine.cross_model_planner import FilterRoute + + route = self._route( + _derived("host_derived_crossing"), target_path=("customers",), + ) + assert route == FilterRoute.PROPAGATE_WHERE + def test_empty_path_is_host_local(self) -> None: from slayer.engine.cross_model_planner import FilterRoute From a9124126f3c2725197b7ee720f3171d9a405afb7 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Wed, 5 Aug 2026 23:01:12 +0200 Subject: [PATCH 29/98] DEV-1744: clear the open Sonar issues and the CodeRabbit allowlist-drift thread --- slayer/core/keys.py | 37 ++++++++---- slayer/engine/binding.py | 4 +- slayer/sql/render/value_expr.py | 80 +++++++++++++++----------- tests/test_dev1744_naming_allocator.py | 17 +++++- tests/test_dev1744_value_expr.py | 70 ++++++++++------------ 5 files changed, 121 insertions(+), 87 deletions(-) diff --git a/slayer/core/keys.py b/slayer/core/keys.py index 560324f7..1dfbd59c 100644 --- a/slayer/core/keys.py +++ b/slayer/core/keys.py @@ -74,23 +74,38 @@ "like": (2, 2), } +# Not a second allowlist: the table above must cover ``SCALAR_FUNCTIONS`` +# exactly. Checked BOTH ways at import — a missing entry would let a wrong-arity +# call through to sqlglot's inconsistent handling, and an entry for a name that +# is not allowlisted would be dead weight that reads as though it were. +_arity_missing = SCALAR_FUNCTIONS - set(SCALAR_FUNCTION_ARITY) +_arity_unknown = set(SCALAR_FUNCTION_ARITY) - SCALAR_FUNCTIONS +if _arity_missing or _arity_unknown: # pragma: no cover — import-time invariant + raise RuntimeError( + f"SCALAR_FUNCTION_ARITY disagrees with SCALAR_FUNCTIONS: " + f"missing={sorted(_arity_missing)}, unknown={sorted(_arity_unknown)}", + ) + -def check_scalar_arity(name: str, argc: int) -> Optional[str]: +def check_scalar_arity(*, name: str, argc: int) -> Optional[str]: """Return an error message when ``name`` cannot take ``argc`` arguments.""" bounds = SCALAR_FUNCTION_ARITY.get(name) if bounds is None: return None low, high = bounds - if argc < low or (high is not None and argc > high): - expected = ( - f"{low}" if low == high - else (f"{low} or more" if high is None else f"{low} to {high}") - ) - return ( - f"Scalar function {name!r} takes {expected} argument" - f"{'' if low == high == 1 else 's'}; got {argc}." - ) - return None + if low <= argc and (high is None or argc <= high): + return None + if low == high: + expected = f"{low}" + elif high is None: + expected = f"{low} or more" + else: + expected = f"{low} to {high}" + plural = "" if low == high == 1 else "s" + return ( + f"Scalar function {name!r} takes {expected} argument{plural}; " + f"got {argc}." + ) # --------------------------------------------------------------------------- diff --git a/slayer/engine/binding.py b/slayer/engine/binding.py index 430681b3..0e015bca 100644 --- a/slayer/engine/binding.py +++ b/slayer/engine/binding.py @@ -1303,7 +1303,9 @@ def _bind_scalar( # handling is inconsistent — a wrong-arity round silently drops the # extra argument, length emits SQL the database rejects — so a # mistyped filter deserves a clear error here instead. - arity_error = check_scalar_arity(parsed.name, len(parsed.args)) + arity_error = check_scalar_arity( + name=parsed.name, argc=len(parsed.args), + ) if arity_error is not None: if parsed.name == "like": raise ValueError( diff --git a/slayer/sql/render/value_expr.py b/slayer/sql/render/value_expr.py index 24f31ae7..0b56e60d 100644 --- a/slayer/sql/render/value_expr.py +++ b/slayer/sql/render/value_expr.py @@ -213,11 +213,14 @@ def _literal(value: Any) -> exp.Expression: exp.Neg: 7, } +_IS = "is" +_IS_NOT = "is not" + # Operators taking exactly two operands. Left-folding a comparison would turn # ``a < b < c`` into ``(a < b) < c`` — a boolean compared to a number — and # reading only the first two would silently DROP the rest. _STRICTLY_BINARY = frozenset({ - "=", "==", "!=", "<>", "<", "<=", ">", ">=", "is", "is not", + "=", "==", "!=", "<>", "<", "<=", ">", ">=", _IS, _IS_NOT, }) # The comparison family's shared level. Unlike arithmetic, comparisons are @@ -303,6 +306,43 @@ def group_is_operands( ) +def _render_unary(*, op: str, operand: exp.Expression) -> exp.Expression: + """The single-operand forms. + + The binder represents ``-x`` as a SINGLE-operand ``ArithmeticKey``, so a + fold that just returns the operand would turn ``amount > -10`` into + ``amount > 10``. The operand needs the same precedence treatment as a + binary one: without it ``-(a + b)`` emits ``-a + b`` and ``not (a and b)`` + emits ``NOT a AND b``. + """ + if op in ("not", "-"): + grouped = group_unary_operand(operand, op=op) + return exp.Not(this=grouped) if op == "not" else exp.Neg(this=grouped) + if op == "+": + # Unary plus is a no-op; SQL never needs it spelled out. + return operand + raise NotImplementedError(f"Unsupported unary operator {op!r}.") + + +def _fold_binary( + *, node_cls: Any, operands: List[exp.Expression], +) -> exp.Expression: + """Left-fold ``operands``, grouping each side by precedence as it goes.""" + parent_prec = _PRECEDENCE.get(node_cls) + result = operands[0] + for operand in operands[1:]: + lhs, rhs = result, operand + if parent_prec is not None: + lhs = _paren_if_lower_prec( + lhs, parent_prec=parent_prec, is_right=False, + ) + rhs = _paren_if_lower_prec( + rhs, parent_prec=parent_prec, is_right=True, + ) + result = node_cls(this=lhs, expression=rhs) + return result + + def render_arithmetic( op: str, operands: List[exp.Expression], ) -> exp.Expression: @@ -312,10 +352,6 @@ def render_arithmetic( one ``ArithmeticKey`` groups the same way wherever it is rendered (P-G). Their hand-rolled versions each knew a different subset of the precedence table and emitted predicates that parse cleanly and mean something else. - - Handles the unary forms too: the binder represents ``-x`` as a - SINGLE-operand ``ArithmeticKey``, so a fold that just returns - ``operands[0]`` would turn ``amount > -10`` into ``amount > 10``. """ if not operands: raise NotImplementedError(f"Operator {op!r} needs at least one operand.") @@ -325,22 +361,13 @@ def render_arithmetic( return operands[0] if len(operands) == 1: - # Unary operands need the same precedence treatment as binary ones. - # Without it ``-(a + b)`` emits ``-a + b`` and ``not (a and b)`` emits - # ``NOT a AND b`` — both parse cleanly and both mean something else. - if op in ("not", "-"): - grouped = group_unary_operand(operands[0], op=op) - return exp.Not(this=grouped) if op == "not" else exp.Neg(this=grouped) - if op == "+": - return operands[0] - raise NotImplementedError( - f"Unsupported unary operator {op!r}.", - ) + return _render_unary(op=op, operand=operands[0]) if op == "and": return exp.and_(*operands) if op == "or": return exp.or_(*operands) + if op in _STRICTLY_BINARY and len(operands) != 2: # Refuse rather than fold. Left-folding a chained comparison compares a # BOOLEAN against the next operand, and reading only the first two @@ -350,28 +377,15 @@ def render_arithmetic( f"Operator {op!r} takes exactly two operands, got {len(operands)}.", ) - if op in ("is", "is not"): + if op in (_IS, _IS_NOT): lhs, rhs = group_is_operands(lhs=operands[0], rhs=operands[1]) node = exp.Is(this=lhs, expression=rhs) - return exp.Not(this=node) if op == "is not" else node + return exp.Not(this=node) if op == _IS_NOT else node node_cls = _BINARY_OPS.get(op) if node_cls is None: raise NotImplementedError(f"Unsupported arithmetic operator {op!r}.") - - parent_prec = _PRECEDENCE.get(node_cls) - result = operands[0] - for operand in operands[1:]: - lhs, rhs = result, operand - if parent_prec is not None: - lhs = _paren_if_lower_prec( - lhs, parent_prec=parent_prec, is_right=False, - ) - rhs = _paren_if_lower_prec( - rhs, parent_prec=parent_prec, is_right=True, - ) - result = node_cls(this=lhs, expression=rhs) - return result + return _fold_binary(node_cls=node_cls, operands=operands) def render_scalar_call( @@ -386,7 +400,7 @@ def render_scalar_call( a native single-arg ``LOG10``. Transpiling alone fixes ifnull and breaks log10. ``like`` is the allowlist's only operator rather than function. """ - arity_error = check_scalar_arity(name, len(args)) + arity_error = check_scalar_arity(name=name, argc=len(args)) if arity_error is not None: # Checked before building, because sqlglot is inconsistent: a 3-arg # ROUND silently DROPS the third, a 2-arg LENGTH emits SQL the backend diff --git a/tests/test_dev1744_naming_allocator.py b/tests/test_dev1744_naming_allocator.py index 0b0819d9..ead660a9 100644 --- a/tests/test_dev1744_naming_allocator.py +++ b/tests/test_dev1744_naming_allocator.py @@ -1216,8 +1216,21 @@ def test_forward_and_rerooted_are_different_identities(self) -> None: def test_same_key_same_shape_shares_one_identity(self) -> None: """The C13 intent the dedup exists to serve: the same aggregate under - two public names is still ONE CTE.""" - assert self._identity(rerooted=False) == self._identity(rerooted=False) + two public names is still ONE CTE. + + Two SEPARATELY CONSTRUCTED keys, which is what two plans carry — the + identity has to compare equal BY VALUE, not by object. + """ + first = AggregateKey( + source=ColumnKey(path=("customers",), leaf="revenue"), agg="sum", + ) + second = AggregateKey( + source=ColumnKey(path=("customers",), leaf="revenue"), agg="sum", + ) + assert first is not second + assert self._identity(rerooted=False, key=first) == self._identity( + rerooted=False, key=second, + ) def test_filtered_and_unfiltered_are_different_identities(self) -> None: """The reason the identity is the typed key and not the alias: these diff --git a/tests/test_dev1744_value_expr.py b/tests/test_dev1744_value_expr.py index af6ed0fa..5bc23045 100644 --- a/tests/test_dev1744_value_expr.py +++ b/tests/test_dev1744_value_expr.py @@ -1867,10 +1867,9 @@ def test_wrong_arity_is_refused(self, name, argc) -> None: from slayer.sql.render.value_expr import render_scalar_call args = [exp.column(f"c{i}") for i in range(argc)] + dialect = get_dialect("postgres") with pytest.raises(NotImplementedError): - render_scalar_call( - name=name, args=args, dialect=get_dialect("postgres"), - ) + render_scalar_call(name=name, args=args, dialect=dialect) @pytest.mark.parametrize( "name,argc", @@ -1892,31 +1891,26 @@ class TestArityIsRejectedAtBindTime: """The renderer check is the backstop; the binder is where a user's typo should surface, with a message naming the function and the counts.""" + @staticmethod + def _query(predicate: str) -> SlayerQuery: + return SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ModelMeasure(formula="*:count", name="n")], + filters=[predicate], + ) + async def test_round_with_three_args_is_rejected(self, e2e) -> None: + query = self._query("round(amount, 2, 99) > 1") with pytest.raises(ValueError, match="round"): - await e2e.execute( - SlayerQuery( - source_model="orders", - dimensions=[ColumnRef(name="status")], - measures=[ModelMeasure(formula="*:count", name="n")], - filters=["round(amount, 2, 99) > 1"], - ), - dry_run=True, - ) + await e2e.execute(query, dry_run=True) async def test_length_with_two_args_is_rejected(self, e2e) -> None: """Previously emitted ``LENGTH(a, b)`` — invalid SQL the backend rejected with its own, less useful error.""" + query = self._query("length(status, status) > 1") with pytest.raises(ValueError, match="length"): - await e2e.execute( - SlayerQuery( - source_model="orders", - dimensions=[ColumnRef(name="status")], - measures=[ModelMeasure(formula="*:count", name="n")], - filters=["length(status, status) > 1"], - ), - dry_run=True, - ) + await e2e.execute(query, dry_run=True) async def test_correct_arity_still_binds(self, e2e) -> None: resp = await e2e.execute( @@ -1968,32 +1962,27 @@ def test_ordinary_in_list_still_renders(self) -> None: "orders.label IN ('a', 'b')" ) + @staticmethod + def _query(predicate: str) -> SlayerQuery: + return SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ModelMeasure(formula="*:count", name="n")], + filters=[predicate], + ) + async def test_bind_time_rejects_null_in_list(self, e2e) -> None: """The user-facing half: caught at bind, with a message pointing at ``is null`` rather than at three-valued logic in the abstract.""" + query = self._query("status in ('new', None)") with pytest.raises(ValueError, match="NULL is not allowed"): - await e2e.execute( - SlayerQuery( - source_model="orders", - dimensions=[ColumnRef(name="status")], - measures=[ModelMeasure(formula="*:count", name="n")], - filters=["status in ('new', None)"], - ), - dry_run=True, - ) + await e2e.execute(query, dry_run=True) async def test_bind_time_rejects_null_in_negated_list(self, e2e) -> None: """The dangerous one: this previously returned zero rows in silence.""" + query = self._query("status not in ('new', None)") with pytest.raises(ValueError, match="NULL is not allowed"): - await e2e.execute( - SlayerQuery( - source_model="orders", - dimensions=[ColumnRef(name="status")], - measures=[ModelMeasure(formula="*:count", name="n")], - filters=["status not in ('new', None)"], - ), - dry_run=True, - ) + await e2e.execute(query, dry_run=True) async def test_null_free_in_list_still_executes(self, e2e) -> None: resp = await e2e.execute( @@ -2146,8 +2135,9 @@ class TestStarSourceIsCountOnly: @pytest.mark.parametrize("agg", ["sum", "avg", "min", "max", "count_distinct"]) def test_non_count_star_is_refused(self, agg) -> None: key = AggregateKey(source=StarKey(), agg=agg) + ctx = _composite_ctx() with pytest.raises(NotImplementedError, match="bare star"): - render_value_key(key, _composite_ctx()) + render_value_key(key, ctx) def test_count_star_still_renders(self) -> None: key = AggregateKey(source=StarKey(), agg="count") From e7c7b73ff129f35baa132bfc261704e51aa78456 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Wed, 5 Aug 2026 23:17:37 +0200 Subject: [PATCH 30/98] DEV-1744: hoist test imports to the top and correct the _wm_ docstring (CodeRabbit nitpicks) --- tests/test_dev1744_naming_allocator.py | 87 ++++++++++---------------- tests/test_dev1744_value_expr.py | 67 +++----------------- 2 files changed, 41 insertions(+), 113 deletions(-) diff --git a/tests/test_dev1744_naming_allocator.py b/tests/test_dev1744_naming_allocator.py index ead660a9..a9bd2981 100644 --- a/tests/test_dev1744_naming_allocator.py +++ b/tests/test_dev1744_naming_allocator.py @@ -47,18 +47,25 @@ import inspect import os +import pathlib import re import sqlite3 +from collections import Counter from decimal import Decimal +from types import SimpleNamespace from typing import AsyncIterator, List import pytest +import sqlglot +from sqlglot import exp -from slayer.core.enums import DataType +import tests.test_parity_guards as guard_module +from slayer.core.enums import DataType, TimeGranularity from slayer.core.keys import ( AggregateKey, ColumnKey, ColumnSqlKey, + SqlExprKey, StarKey, TimeTruncKey, ) @@ -69,9 +76,19 @@ ModelMeasure, SlayerModel, ) -from slayer.core.query import ColumnRef, SlayerQuery +from slayer.core.query import ColumnRef, SlayerQuery, TimeDimension +from slayer.engine import cross_model_planner, planning, stage_planner +from slayer.engine.binding import BoundExpr +from slayer.engine.cross_model_planner import _aggregate_alias +from slayer.engine.planning import _canonical_name from slayer.engine.query_engine import SlayerQueryEngine +from slayer.engine.stage_planner import _canonical_alias_for_formula +from slayer.sql import generator as generator_module from slayer.sql import naming +from slayer.sql import stage_wrapper as sw_module +from slayer.sql.dialects import get_dialect +from slayer.sql.dialects import tsql as tsql_module +from slayer.sql.generator import SQLGenerator, _cm_plan_identity from slayer.sql.naming import AliasAllocator from slayer.storage.yaml_storage import YAMLStorage @@ -193,9 +210,6 @@ def _cte_names_by_scope(sql: str, *, dialect: str = "sqlite") -> List[List[str]] cross-scope uniqueness check would constrain the allocator beyond what the plan asks for. """ - import sqlglot - from sqlglot import exp - parsed = sqlglot.parse_one(sql, dialect=dialect) return [ [cte.alias_or_name for cte in with_node.expressions] @@ -406,10 +420,6 @@ async def test_hidden_cross_model_agg_under_a_transform_keeps_its_alias( the step CTE binds the wrong column. Asserted as "no output alias is emitted twice", which is what actually breaks. """ - from collections import Counter - from slayer.core.enums import TimeGranularity - from slayer.core.query import TimeDimension - resp = await engine.execute( SlayerQuery( source_model="orders", @@ -498,9 +508,7 @@ class TestDedupIdentityIsStructural: The identity choice still has to be right, because the generator's dedup map is what enforces it. """ - def _filtered_and_plain(self): - from slayer.core.keys import SqlExprKey source = ColumnKey(leaf="revenue") plain = AggregateKey(source=source, agg="sum") @@ -585,13 +593,12 @@ async def test_windowed_cte_names_use_the_shared_helper( ``_cm_``, so both CTE families behave identically under a case-only collision rather than one being retrofitted and the other not. - Two windowed measures over case-only column variants would otherwise - emit two names that fold together on a folding dialect — the exact - shape that broke ``_cm_``. + Both measures aggregate the SAME column (``amount``) over different + windows; the case-only pair is in their declared names, ``Wm`` and + ``wm``. Those names reach the CTE name, so without the allocator the + two would fold together on a folding dialect — the exact shape that + broke ``_cm_``. """ - from slayer.core.enums import TimeGranularity - from slayer.core.query import TimeDimension - engine = await _hostile_engine( column="revx2", base_dir=str(tmp_path_factory.mktemp("wm")), ) @@ -628,8 +635,6 @@ def test_no_raw_step_cte_names_in_the_generator(self) -> None: because no ``_cm_*`` CTE can be named ``stepN`` — an invariant nothing enforces. """ - from slayer.sql import generator as generator_module - src = inspect.getsource(generator_module) raw = [ line.strip() @@ -811,9 +816,6 @@ async def test_ranked_subquery_families_survive_the_name( async def test_windowed_families_survive_the_name(self, column, tmp_path_factory) -> None: """Reaches the ``_w_*`` ``_src``-projection aliases specifically: a duration-windowed measure is the only shape that builds them.""" - from slayer.core.enums import TimeGranularity - from slayer.core.query import TimeDimension - engine = await _hostile_engine( column=column, base_dir=str(tmp_path_factory.mktemp("hostile")), ) @@ -853,7 +855,6 @@ class TestNamingConstants: outer-wrap machinery wholesale. The carve-out is recorded as a named P-F exception, not an omission. """ - def test_constants_exist_and_match_the_current_literals(self) -> None: assert naming.OUTER_WRAP_ALIAS == "_outer" assert naming.STAGE_INNER_ALIAS == "_stage_inner" @@ -867,15 +868,12 @@ def test_tsql_dialect_imports_the_shared_constant(self) -> None: or an error message, and forbidding that would constrain the implementation past what the plan asks for. """ - from slayer.sql.dialects import tsql as tsql_module - assert hasattr(tsql_module, "OUTER_WRAP_ALIAS"), ( "tsql.py does not import naming.OUTER_WRAP_ALIAS" ) assert tsql_module.OUTER_WRAP_ALIAS is naming.OUTER_WRAP_ALIAS def test_stage_wrapper_imports_the_shared_constant(self) -> None: - from slayer.sql import stage_wrapper as sw_module assert hasattr(sw_module, "STAGE_INNER_ALIAS"), ( "stage_wrapper.py does not import naming.STAGE_INNER_ALIAS" @@ -886,8 +884,6 @@ def test_tsql_outer_wrap_alias_still_round_trips(self) -> None: """Behavioural companion: the ratified carve-out says the T-SQL ORDER-BY detach rewrite keeps using a CONSTANT (not an allocated name), so its emitted alias must still be exactly the shared one.""" - from slayer.sql.dialects import get_dialect - assert get_dialect("tsql") is not None assert naming.OUTER_WRAP_ALIAS == "_outer" @@ -902,9 +898,7 @@ class TestParityGuardRepair: Only the documentation is repaired — strengthening the guard's matcher is explicitly out of scope for this PR. """ - def test_docstring_no_longer_references_the_deleted_module(self) -> None: - import tests.test_parity_guards as guard_module doc = guard_module.__doc__ or "" assert "parity_xfails" not in doc, ( @@ -915,16 +909,12 @@ def test_docstring_no_longer_references_the_deleted_module(self) -> None: def test_the_deleted_module_really_is_gone(self) -> None: """Guard on the premise itself, so this repair cannot be silently invalidated by the file coming back.""" - import pathlib - tests_dir = pathlib.Path(__file__).parent assert not (tests_dir / "parity_xfails.py").exists() def test_the_guard_itself_still_works(self) -> None: """Parity: the repair is docstring-only, so the guard must still run and still pass.""" - import tests.test_parity_guards as guard_module - assert hasattr(guard_module, "APPROVED_GUARDS") @@ -1112,11 +1102,6 @@ class TestProductionCallersDelegate: def test_all_four_agree_with_the_naming_module( self, case, key, expected_a, expected_b, expected_c, expected_d, ) -> None: - from slayer.engine.binding import BoundExpr - from slayer.engine.cross_model_planner import _aggregate_alias - from slayer.engine.planning import _canonical_name - from slayer.engine.stage_planner import _canonical_alias_for_formula - from slayer.sql.generator import SQLGenerator gen = SQLGenerator(dialect="postgres") assert gen._canonical_cross_model_alias( @@ -1138,11 +1123,6 @@ def test_each_caller_actually_delegates_with_its_profile( Spy on the naming module and assert each production function FORWARDS, with the right profile and the right ``source_relation``. """ - from slayer.engine import cross_model_planner, planning, stage_planner - from slayer.engine.binding import BoundExpr - from slayer.sql import generator as generator_module - from slayer.sql.generator import SQLGenerator - key = _key(ColumnKey(path=("customers",), leaf="revenue"), "sum") calls: List[dict] = [] real = naming.canonical_aggregate_alias @@ -1196,11 +1176,7 @@ class TestCrossModelDedupIdentity: plans cannot collide here today — these tests keep that from becoming silently wrong if that ever changes. """ - def _identity(self, *, rerooted, key=None): - from types import SimpleNamespace - - from slayer.sql.generator import _cm_plan_identity key = key or AggregateKey( source=ColumnKey(path=("customers",), leaf="revenue"), agg="sum", @@ -1215,11 +1191,14 @@ def test_forward_and_rerooted_are_different_identities(self) -> None: assert self._identity(rerooted=False) != self._identity(rerooted=True) def test_same_key_same_shape_shares_one_identity(self) -> None: - """The C13 intent the dedup exists to serve: the same aggregate under - two public names is still ONE CTE. - - Two SEPARATELY CONSTRUCTED keys, which is what two plans carry — the - identity has to compare equal BY VALUE, not by object. + """Two SEPARATELY CONSTRUCTED but equal keys — which is what two plans + carry — collapse to ONE identity. The tuple has to compare equal BY + VALUE, since it is used as a dict key. + + This is the unit-level precondition only. That two public names really + do end up sharing a single CTE is asserted end-to-end by + ``test_same_key_slots_still_share_one_cte`` above; this test cannot see + public names at all, because the identity deliberately excludes them. """ first = AggregateKey( source=ColumnKey(path=("customers",), leaf="revenue"), agg="sum", @@ -1235,8 +1214,6 @@ def test_same_key_same_shape_shares_one_identity(self) -> None: def test_filtered_and_unfiltered_are_different_identities(self) -> None: """The reason the identity is the typed key and not the alias: these two produce the SAME canonical alias.""" - from slayer.core.keys import SqlExprKey - source = ColumnKey(path=("customers",), leaf="revenue") plain = AggregateKey(source=source, agg="sum") filtered = AggregateKey( diff --git a/tests/test_dev1744_value_expr.py b/tests/test_dev1744_value_expr.py index 5bc23045..e7174b68 100644 --- a/tests/test_dev1744_value_expr.py +++ b/tests/test_dev1744_value_expr.py @@ -56,7 +56,7 @@ import sqlglot from sqlglot import exp -from slayer.core.enums import BUILTIN_AGGREGATIONS, DataType +from slayer.core.enums import BUILTIN_AGGREGATIONS, DataType, TimeGranularity from slayer.core.errors import ( RenderContextMissingFacilityError, UnknownReferenceError, @@ -83,13 +83,17 @@ ModelMeasure, SlayerModel, ) -from slayer.core.query import ColumnRef, SlayerQuery +from slayer.core.query import ColumnRef, SlayerQuery, TimeDimension from slayer.engine.query_engine import SlayerQueryEngine from slayer.engine.source_bundle import ResolvedSourceBundle from slayer.sql.dialects import get_dialect from slayer.sql.generator import SQLGenerator from slayer.sql.naming import AliasAllocator -from slayer.sql.render.aggregates import resolve_agg_entry, window_agg_class +from slayer.sql.render.aggregates import ( + AGG_REGISTRY, + resolve_agg_entry, + window_agg_class, +) from slayer.sql.render.value_expr import ( AliasFacilities, contains_aggregate, @@ -98,7 +102,9 @@ RenderContext, _literal, render_arithmetic, + render_scalar_call, render_value_key, + rewrite_log_alias, ) from slayer.sql.scope import ScopeFrame from slayer.storage.yaml_storage import YAMLStorage @@ -168,7 +174,6 @@ def _scope( def _filter_ctx(dialect: str = "postgres", **kw): """A RenderContext carrying the FILTER facility group (R1's call family).""" - scope = _scope(dialect=dialect) return RenderContext( scope=scope, @@ -179,7 +184,6 @@ def _filter_ctx(dialect: str = "postgres", **kw): def _composite_ctx(dialect: str = "postgres", **kw): """A RenderContext carrying the COMPOSITE facility group (R5's family).""" - scope = _scope(dialect=dialect) return RenderContext( scope=scope, @@ -202,7 +206,6 @@ class TestB10UnknownModelRaises: naming a model absent from the bundle silently expands the ROOT model's derived SQL instead. That turns a wiring bug into a wrong answer — the query runs and returns numbers computed from the wrong model.""" - def test_unknown_model_in_columnsqlkey_raises(self) -> None: scope = _scope() @@ -213,7 +216,6 @@ def test_unknown_model_in_columnsqlkey_raises(self) -> None: def test_error_names_the_missing_model(self) -> None: """The message must be actionable: which model was asked for, what the scope root is, and what the bundle actually knows.""" - scope = _scope() key = ColumnSqlKey(model="not_in_bundle", column_name="net") with pytest.raises(UnknownReferenceError) as excinfo: @@ -256,7 +258,6 @@ def test_context_holds_real_production_objects(self) -> None: """Pydantic v2 + a ``ScopeFrame`` / dialect strategy / sqlglot nodes needs ``arbitrary_types_allowed``; constructing with the real objects (not stubs) is what proves the config is right.""" - scope = _scope() ctx = RenderContext(scope=scope, dialect=scope.dialect) assert ctx.scope is scope @@ -268,7 +269,6 @@ def test_context_holds_real_production_objects(self) -> None: def test_consumer_defaults_to_none_and_is_accepted(self) -> None: """The P-B seam exists in PR 1 even though its production callers arrive in PR 3.""" - producer, consumer = _scope(), _scope() ctx = RenderContext( scope=producer, consumer=consumer, dialect=producer.dialect, @@ -292,7 +292,6 @@ def test_consumer_routes_column_like_leaves_through_materialization( Parametrised over all three column-like kinds because a renderer that special-cases one of them would otherwise slip through.""" - producer, consumer = _scope(), _scope() ctx = RenderContext( scope=producer, consumer=consumer, dialect=producer.dialect, @@ -307,7 +306,6 @@ def test_materializations_apply_to_the_producing_select(self) -> None: """The other half of the P-B contract: what the renderer records must actually be projectable via ``apply_materializations``, so the consumer's bare alias resolves to a real column of the producing SELECT.""" - producer, consumer = _scope(), _scope() ctx = RenderContext( scope=producer, consumer=consumer, dialect=producer.dialect, @@ -325,7 +323,6 @@ def test_materialization_dedups_within_a_scope(self) -> None: """Two renders of the same key across the same boundary share ONE ``_val_`` — the dedup key is the producing scope + anchored AST + dialect, and the renderer must not defeat it by re-anchoring.""" - producer, consumer = _scope(), _scope() ctx = RenderContext( scope=producer, consumer=consumer, dialect=producer.dialect, @@ -339,7 +336,6 @@ def test_join_paths_register_as_a_side_effect_of_rendering(self) -> None: """P-A: join discovery is a side effect of rendering, never a separate pass. Rendering a joined leaf must register the crossed path on the scope without the caller asking.""" - scope = _scope() ctx = RenderContext(scope=scope, dialect=scope.dialect) render_value_key( @@ -353,7 +349,6 @@ def test_missing_facility_fails_closed(self) -> None: """A key kind that needs a facility the context lacks must RAISE, not silently degrade. Silent degradation is how the five copies drifted in the first place.""" - scope = _scope() bare = RenderContext(scope=scope, dialect=scope.dialect) # A POST-phase transform can only be rendered against already- @@ -391,7 +386,6 @@ def test_aggregate_without_composite_facilities_fails_closed(self) -> None: ``AggregateKey`` needs the composite facilities (rn-suffix maps, resolved agg kwargs, composite alias map) to render faithfully. """ - scope = _scope() bare = RenderContext(scope=scope, dialect=scope.dialect) key = AggregateKey(source=ColumnKey(leaf="amount"), agg="first") @@ -407,7 +401,6 @@ def test_filtered_aggregate_without_a_builder_fails_closed(self) -> None: ``source`` alone drops the filter and covers rows it must exclude — a wrong number rather than an error, so the no-builder path refuses it. """ - key = AggregateKey( source=ColumnKey(leaf="amount"), agg="sum", @@ -420,7 +413,6 @@ def test_filtered_aggregate_without_a_builder_fails_closed(self) -> None: def test_parametric_aggregate_without_a_builder_fails_closed(self) -> None: """Same rule for args/kwargs, which need the generator's parameter resolution.""" - key = AggregateKey( source=ColumnKey(leaf="amount"), agg="sum", @@ -457,7 +449,6 @@ def test_transform_key_renders_when_alias_facilities_are_supplied( WITH its facility must render here — otherwise "fails closed" would be indistinguishable from "not implemented". """ - scope = _scope() agg = AggregateKey(source=ColumnKey(leaf="amount"), agg="sum") key = TransformKey(op="time_shift", input=agg) @@ -482,7 +473,6 @@ class TestRendersEveryKeyKind: """The union is closed (11 members). One renderer means every member is handled in one place — an unhandled kind must raise, never fall through to a bare ``None`` or a stringified repr.""" - def test_local_column_key(self) -> None: out = render_value_key(ColumnKey(leaf="amount"), _filter_ctx()) @@ -506,7 +496,6 @@ def test_multi_hop_column_key(self) -> None: def test_column_sql_key_expands_the_derived_expression(self) -> None: """Exact SQL, not a substring check: ``net`` is ``amount - 1``, and the expansion must be anchored at the scope root.""" - out = render_value_key( ColumnSqlKey(model="orders", column_name="net"), _filter_ctx(), ) @@ -515,7 +504,6 @@ def test_column_sql_key_expands_the_derived_expression(self) -> None: def test_time_trunc_key(self) -> None: """Exact per-dialect SQL — a substring check would accept a truncation at the wrong granularity or over the wrong column.""" - key = TimeTruncKey( column=ColumnKey(leaf="created_at"), granularity="month", ) @@ -532,7 +520,6 @@ def test_time_trunc_goes_through_the_dialect_strategy(self, dialect) -> None: the backend rejects, which is the same one-construct-two-renderings defect this module exists to remove. """ - key = TimeTruncKey( column=ColumnKey(leaf="created_at"), granularity="month", ) @@ -548,7 +535,6 @@ def test_week_sunday_granularity_renders(self) -> None: A hardcoded unit table would have no entry for it and would emit ``DATE_TRUNC('WEEK_SUNDAY', col)``, which no dialect accepts. """ - key = TimeTruncKey( column=ColumnKey(leaf="created_at"), granularity="week_sunday", ) @@ -652,7 +638,6 @@ def test_unary_minus_keeps_its_sign(self) -> None: ``amount > -10`` would silently become ``amount > 10`` — a wrong result, not a failure. """ - out = render_value_key( ArithmeticKey(op="-", operands=(LiteralKey(value=Decimal(10)),)), _filter_ctx(), @@ -740,7 +725,6 @@ def test_unary_not(self) -> None: ) def test_arithmetic_precedence_is_parenthesised(self, key, expected) -> None: """Operator precedence has to be materialised as ``Paren`` nodes.""" - ctx = _filter_ctx() assert _sql(render_value_key(key, ctx)) == expected @@ -759,7 +743,6 @@ def test_unsupported_literal_type_raises(self) -> None: def test_supported_literal_types_still_render(self) -> None: """The fail-closed branch must not swallow the supported cases.""" - assert _literal(None).sql() == "NULL" assert _literal(True).sql() == "TRUE" assert _literal(Decimal("1.5")).sql() == "1.5" @@ -774,7 +757,6 @@ def test_unhandled_kind_raises_notimplementederror(self) -> None: accepting a tuple of types would let an incidental TypeError from somewhere else inside the renderer satisfy this test. """ - ctx = _filter_ctx() with pytest.raises(NotImplementedError) as excinfo: render_value_key(object(), ctx) # type: ignore[arg-type] @@ -828,7 +810,6 @@ def test_scalar_calls_transpile_per_dialect( def test_ifnull_never_reaches_postgres_unmapped(self) -> None: """The headline B5 bug, stated as the invariant rather than as an exact string: Postgres has no ``IFNULL``, so emitting it is broken SQL.""" - key = ScalarCallKey( name="ifnull", args=(ColumnKey(leaf="amount"), LiteralKey(value=Decimal(0))), @@ -846,7 +827,6 @@ def test_log10_keeps_the_native_single_arg_alias(self) -> None: ``_rewrite_log_aliases``. Applying transpile WITHOUT that rewrite would regress ``log10`` — so the unified renderer must apply both. """ - key = ScalarCallKey(name="log10", args=(ColumnKey(leaf="amount"),)) out = _sql(render_value_key(key, _filter_ctx("postgres")), "postgres") assert out.upper().startswith("LOG10("), out @@ -855,7 +835,6 @@ def test_round_keeps_the_dev1576_postgres_cast(self) -> None: """Parity guard: two-arg ROUND on Postgres needs the numeric cast, and it is the ONE scalar call R1 already routed through the typed path. Unifying must not lose it.""" - key = ScalarCallKey( name="round", args=(ColumnKey(leaf="amount"), LiteralKey(value=Decimal(0))), @@ -868,7 +847,6 @@ def test_like_stays_the_sql_operator(self) -> None: """``like(value, pattern)`` is the one allowlist member that is an OPERATOR, not a function call. Both legacy paths special-case it; the unified renderer keeps that.""" - key = ScalarCallKey( name="like", args=(ColumnKey(leaf="label"), LiteralKey(value="x%")), @@ -880,7 +858,6 @@ def test_like_stays_the_sql_operator(self) -> None: def test_nested_scalar_calls_use_one_policy_throughout(self) -> None: """The policy applies at every depth — a nested call must not fall back to the passthrough branch.""" - key = ScalarCallKey( name="ifnull", args=( @@ -907,8 +884,6 @@ class TestLogAliasPolicyIsShared: @pytest.mark.parametrize("dialect", ["postgres", "sqlite", "tsql", "bigquery"]) def test_generator_delegates_to_the_shared_policy(self, dialect) -> None: - from slayer.sql.generator import SQLGenerator - from slayer.sql.render.value_expr import rewrite_log_alias gen = SQLGenerator(dialect=dialect) node = exp.Log( @@ -921,16 +896,12 @@ def test_generator_delegates_to_the_shared_policy(self, dialect) -> None: def test_generator_parse_path_still_emits_native_log10(self) -> None: """Behavioural companion: the delegation must not lose the rewrite that the generator applies over parsed trees.""" - from slayer.sql.generator import SQLGenerator - gen = SQLGenerator(dialect="postgres") out = gen._parse("log10(x)").sql(dialect="postgres") assert out.upper().startswith("LOG10("), out def test_non_log_nodes_pass_through_untouched(self) -> None: - from slayer.sql.render.value_expr import rewrite_log_alias - from slayer.sql.dialects import get_dialect node = exp.column("x") assert rewrite_log_alias(node, dialect=get_dialect("postgres")) is node @@ -943,7 +914,6 @@ class TestPGSameConstructSameSql: rendering POLICY must not branch on them. Any divergence here is the class of bug the five copies produced. """ - _KEYS = [ ("column", ColumnKey(leaf="amount")), ("joined_column", ColumnKey(path=("customers",), leaf="balance")), @@ -1036,7 +1006,6 @@ def test_each_former_dispatch_mechanism_is_represented( implementation that only ported the easy ``_AGG_FUNCTION_MAP`` entries would still pass ``test_every_builtin_resolves`` if the enum happened to be small.""" - for name in names: assert resolve_agg_entry(name).name == name, mechanism @@ -1056,7 +1025,6 @@ def test_windowable_flags_are_exact(self) -> None: """Only ``sum`` and ``avg`` are windowable today — that is precisely what ``stage_planner`` gates on, and the registry must agree with it rather than restating it.""" - assert resolve_agg_entry("sum").windowable is True assert resolve_agg_entry("avg").windowable is True for name in ("count", "min", "max", "median", "percentile", "first"): @@ -1075,8 +1043,6 @@ def test_registry_and_builtins_agree_both_ways(self) -> None: subtler half: ``is_builtin_agg`` would accept it and route it AWAY from that path, so the typo would render as if it were a real aggregation. """ - from slayer.sql.render.aggregates import AGG_REGISTRY - assert set(AGG_REGISTRY) == set(BUILTIN_AGGREGATIONS) def test_non_windowable_aggregation_fails_closed(self) -> None: @@ -1087,7 +1053,6 @@ def test_non_windowable_aggregation_fails_closed(self) -> None: Approved divergence: it raises instead. """ - for name in ("median", "count", "min", "max", "percentile"): with pytest.raises(ValueError): window_agg_class(name) @@ -1268,7 +1233,6 @@ class TestOuterWrapperAndShiftedCteFamilies: * R1's shifted-CTE WHERE call site (the generator) — reached only by a ``time_shift`` transform, never by a plain host filter. """ - async def _engine(self, tmp_path_factory, *, dialect: str = "sqlite") -> SlayerQueryEngine: d = str(tmp_path_factory.mktemp("routes")) db_path = os.path.join(d, "routes.db") @@ -1408,9 +1372,6 @@ async def test_shifted_cte_filter_call_site_executes(self, tmp_path_factory) -> With ``status = 'new'`` only, January's total is 100 and February's shifted-by-one-month value must be that same 100. """ - from slayer.core.enums import TimeGranularity - from slayer.core.query import TimeDimension - engine = await self._engine(tmp_path_factory) resp = await engine.execute( SlayerQuery( @@ -1624,7 +1585,6 @@ class TestOperatorCompositionEdges: """Three edges a Codex pass surfaced, all the same family as the rest: output that still parses and still returns rows, but means something else. """ - def test_comparison_nested_in_arithmetic_keeps_its_parens(self) -> None: """``(a > b) + 1`` must not flatten to ``a > b + 1``. @@ -1709,7 +1669,6 @@ def test_is_not_with_extra_operands_is_refused(self) -> None: class TestContainsAggregate: """``contains_aggregate`` decides GROUP BY / HAVING placement, so it must answer "is there an aggregate in this tree", not "when does this evaluate".""" - def test_bare_aggregate(self) -> None: assert contains_aggregate( AggregateKey(source=ColumnKey(leaf="amount"), agg="sum"), @@ -1755,7 +1714,6 @@ class TestEqualPrecedenceRightChildren: the expression regrouped to ``(a * b) % c``. With a=2 b=3 c=2 that is 0 instead of 2 — a different number from SQL that parses cleanly. """ - def _key(self, outer_op, inner_op): return ArithmeticKey( op=outer_op, @@ -1820,7 +1778,6 @@ class TestContainsAggregateTransformDependencies: """``partition_keys`` and ``time_key`` are expression dependencies of a transform just as ``input`` is — an aggregate in either one still lands in the emitted SQL.""" - def test_aggregate_in_partition_keys(self) -> None: agg = AggregateKey(source=ColumnKey(leaf="amount"), agg="sum") key = TransformKey( @@ -1864,7 +1821,6 @@ class TestScalarArity: ("nullif", 1), ("replace", 2), ("substr", 1), ("like", 3)], ) def test_wrong_arity_is_refused(self, name, argc) -> None: - from slayer.sql.render.value_expr import render_scalar_call args = [exp.column(f"c{i}") for i in range(argc)] dialect = get_dialect("postgres") @@ -1878,8 +1834,6 @@ def test_wrong_arity_is_refused(self, name, argc) -> None: ) def test_accepted_arities_still_render(self, name, argc) -> None: """The variadic and optional-argument forms must keep working.""" - from slayer.sql.render.value_expr import render_scalar_call - args = [exp.column(f"c{i}") for i in range(argc)] out = render_scalar_call( name=name, args=args, dialect=get_dialect("postgres"), @@ -1933,7 +1887,6 @@ class TestNullInInList: rows rather than "everything except a". Neither announces itself — the query runs and hands back a plausible-looking empty result. """ - def test_renderer_refuses_null_in_the_list(self) -> None: key = InKey( column=ColumnKey(leaf="label"), @@ -2003,7 +1956,6 @@ class TestUnaryOperandGrouping: the operand straight into ``exp.Neg`` / ``exp.Not`` without grouping it. Both results parse cleanly and mean something else. """ - def test_negated_sum_keeps_its_parens(self) -> None: """``-(a + b)`` must not flatten to ``-a + b``, which is ``(-a) + b``.""" key = ArithmeticKey( @@ -2070,7 +2022,6 @@ class TestComparisonsAreNonAssociative: shapes are reachable — the Mode-B parser reads ``(a == b) == c`` as a NESTED comparison, not a chained one, so the binder does build them. """ - def _cmp(self, op, left, right): return ArithmeticKey(op=op, operands=(left, right)) From 990f0eb76aec1fba327cf66fa3e57bc9f15176f5 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Thu, 6 Aug 2026 09:11:15 +0200 Subject: [PATCH 31/98] DEV-1745: drop the redundant target params from _register_agg_key_joins MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit From the Codex review of the migration. The helper received scope AND target_relation/target_model/bundle, scanned the column filter against the target ones, then registered the result on scope — two sources of truth for one namespace. They agree at the single call site (the CTE scope is built with root_model=target_model, root_relation=target_relation) so this was not a live bug, but nothing enforced it: a future caller passing a scope rooted elsewhere would have had its refs resolved in the wrong namespace. After the door migration target_relation and bundle were dead parameters anyway. The scope is now the only namespace and the question cannot arise. Also restores a guard the migration dropped: an empty entry in target_model_filters was skipped by the old `if not qualified: continue`, and would now reach a door that raises on text it cannot parse. Co-Authored-By: Claude Opus 5 (1M context) --- slayer/sql/generator.py | 27 +++++++++++++++++++-------- 1 file changed, 19 insertions(+), 8 deletions(-) diff --git a/slayer/sql/generator.py b/slayer/sql/generator.py index cca5db0a..801712f8 100644 --- a/slayer/sql/generator.py +++ b/slayer/sql/generator.py @@ -5852,6 +5852,11 @@ def _register_filter_join_paths(sql_text: Optional[str]) -> None: # derived column on a joined model) is inline-expanded; base-only # filters keep the AST bare-ref qualification. The crossed join is # pulled into this CTE's FROM via ``cte_scope.join_paths``. + # An empty entry contributes no predicate. Skipped rather than + # entered: the door raises on text it cannot parse, and "" is not a + # predicate — the path this replaced skipped it too. + if not filter_text: + continue where_parts.append(self._enter_mode_a_predicate( sql=filter_text, scope=cte_scope, location=( @@ -6084,10 +6089,7 @@ def _walk(vk) -> None: ) scope.resolve(local) elif isinstance(vk, AggregateKey): - self._register_agg_key_joins( - agg_key=vk, scope=scope, target_relation=target_relation, - target_model=target_model, bundle=bundle, - ) + self._register_agg_key_joins(agg_key=vk, scope=scope) elif isinstance(vk, ArithmeticKey): for op in vk.operands: _walk(op) @@ -6106,13 +6108,22 @@ def _walk(vk) -> None: _walk(fp.expression.value_key) def _register_agg_key_joins( - self, *, agg_key, scope: ScopeFrame, target_relation: str, - target_model, bundle, + self, *, agg_key, scope: ScopeFrame, ) -> None: """Register the joins an aggregate leaf crosses (source + positional args + column-ref kwargs + ``column_filter``) into ``scope.join_paths`` — the ``AggregateKey`` arm of ``_register_routed_filter_joins``'s tree - walk, extracted so the walker stays a thin dispatcher (DEV-1708).""" + walk, extracted so the walker stays a thin dispatcher (DEV-1708). + + Takes ONLY the scope. It used to also receive ``target_relation`` / + ``target_model`` / ``bundle`` and scan the column filter against those + while registering the result on ``scope`` — two sources of truth for one + fact. They agree at the single call site (the CTE scope is built with + ``root_model=target_model, root_relation=target_relation``), but nothing + enforced that, so a future caller could have passed a scope rooted + elsewhere and had its refs resolved in the wrong namespace. Now the + scope is the only namespace, and the question cannot arise. + """ from slayer.core.keys import ColumnKey, ColumnSqlKey cross_model_path = getattr(agg_key.source, "path", ()) @@ -6129,7 +6140,7 @@ def _register_agg_key_joins( if cfk is not None and cfk.canonical_sql: self._enter_mode_a_predicate( sql=cfk.canonical_sql, scope=scope, - location=f"Column.filter on model {target_model.name!r}", + location=f"Column.filter on model {scope.root_model.name!r}", ) def _collect_routed_filters( From 3a1ed93d7b38c71aebba6414e2ee19fc42df86c0 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Thu, 6 Aug 2026 09:16:31 +0200 Subject: [PATCH 32/98] =?UTF-8?q?DEV-1744:=20admit=20ceiling/sign/ltrim/rt?= =?UTF-8?q?rim/substring=20=E2=80=94=20the=20parser=20already=20advertised?= =?UTF-8?q?=20them?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .claude/skills/slayer-query.md | 2 +- DECISIONS.md | 1 + docs/architecture/typed-keys.md | 7 +- docs/concepts/queries.md | 6 +- docs/concepts/references.md | 23 +++++-- slayer/core/keys.py | 16 ++++- tests/test_dev1744_value_expr.py | 109 +++++++++++++++++++++++++++++++ 7 files changed, 150 insertions(+), 14 deletions(-) diff --git a/.claude/skills/slayer-query.md b/.claude/skills/slayer-query.md index 4bab5cd1..b3787b6e 100644 --- a/.claude/skills/slayer-query.md +++ b/.claude/skills/slayer-query.md @@ -72,7 +72,7 @@ Result column naming: `revenue:sum` → `orders.revenue_sum` (colon becomes unde **Boolean logic**: `AND`, `OR`, `NOT` -**String-hygiene scalars** (DEV-1378, lowercase only): `lower`, `upper`, `trim`, `replace`, `substr`, `instr`, `length`, `concat`. Plus the SQL `||` operator (folded into `concat(...)`). Examples: `"lower(status) = 'active'"`, `"length(replace(x, ',', '')) > 0"`, `"substr(s, 1, instr(s, ',') - 1) = 'first'"`, `"first || ' ' || last = 'jane doe'"`. Calls outside this allowlist (`json_extract`, `coalesce`, …) belong in `Column.sql` / `Column.filter` / `SlayerModel.filters` (Mode A SQL), not query filters. +**String-hygiene scalars** (lowercase only): `lower`, `upper`, `trim`, `ltrim`, `rtrim`, `replace`, `substr`, `substring`, `instr`, `length`, `concat`. Plus the SQL `||` operator (folded into `concat(...)`). Examples: `"lower(status) = 'active'"`, `"length(replace(x, ',', '')) > 0"`, `"substr(s, 1, instr(s, ',') - 1) = 'first'"`, `"first || ' ' || last = 'jane doe'"`. Calls outside this allowlist (`json_extract`, `coalesce`, …) belong in `Column.sql` / `Column.filter` / `SlayerModel.filters` (Mode A SQL), not query filters. **Filtering on computed measures**: `"change(revenue:sum) > 0"`, `"last(change(revenue:sum)) < 0"`. Applied as post-filters on the outer query. diff --git a/DECISIONS.md b/DECISIONS.md index 4f33e4bd..37fc79a5 100644 --- a/DECISIONS.md +++ b/DECISIONS.md @@ -98,4 +98,5 @@ implementation detail. Include issue refs when known. - 2026-08-05 — One naming authority + one ValueKey renderer (DEV-1744, PR 1 of the DEV-1742 consolidation). Four consequences worth recording. (1) **Cross-model CTE dedup is now structural.** `_cm_` CTE names were minted by a doubly-lossy helper (`flat_name`, itself documented non-injective, then a non-identifier `re.sub`) and the resulting string doubled as the plan's identity key in `seen_cm`. Two reachable failures followed: two measures whose canonical aliases differ only in case emitted two `_cm_` names that fold together on every case-folding dialect (the collision belt raised — `_wm_` had been retrofitted onto the allocator years earlier, `_cm_` never was), and two genuinely distinct aggregates that sanitised alike made the second skip the loop body, leaving its join-back and column-alias maps unwritten and raising `KeyError` downstream. The dedup key is now the typed `AggregateKey` plus the source relation; the name is allocator-minted and stored once, so the five sites that used to re-derive it now read it. Deliberately NOT keyed on the canonical alias: `canonical_agg_name` omits `column_filter_key`, so a filtered and an unfiltered aggregate over one column share an alias while needing two CTEs — deduping on the string would silently merge them, a wrong answer rather than a crash. (2) **One ScalarCall policy.** Two of the five renderers returned `exp.Anonymous` passthrough, so `ifnull` reached Postgres — which has no `IFNULL` — while the same key emitted `COALESCE` from a projection. All six render paths now call one `render_scalar_call`. Transpiling alone was NOT sufficient: `exp.func("LOG10", x)` normalises to a generic `Log(10, x)` that re-emits as `LOG(10, x)`, wrong on dialects with a native single-arg `LOG10`, so the policy is transpile-then-log-alias-rewrite. Consequence surfaced and approved: `concat` now emits the `||` operator on Postgres/DuckDB where it previously emitted `CONCAT(...)`. That is a semantic change as well as a spelling one — Postgres `CONCAT()` ignores NULL operands, `||` propagates them. Ratified as the kept behavior: the projection path has always emitted `||`, so before this change SLayer disagreed with itself between filters and projections on the same input, and consistency was chosen over preserving the filter-side NULL tolerance. (3) **`ScopeFrame._model_for` raises** instead of falling back to the root model; the fallback turned a wiring bug into a wrong answer by expanding a different model's derived SQL. (4) **Named P-F carve-outs**, recorded rather than omitted: the T-SQL ORDER-BY-detach rewrite and the stage-schema wrapper take `_outer` / `_stage_inner` as shared CONSTANTS rather than allocator-minted names (the former is a post-generation AST pass with no allocator in reach, and a later PR rebuilds that machinery); `base` / `_base` / `_combined` keep their literal names but are reserved into the allocator. Superseded code is retained and callable per the chain's operating rule — deletion happens in the final PR. - 2026-08-05 — Arithmetic composition joins ScalarCall as a construct that renders ONCE, ahead of the call-site migration (DEV-1744). A carve-out from the deferral below, taken because the generator's three composers (`_build_arith_or_cmp_ast`, `_compose_arithmetic_op`, `_build_arithmetic_for_filter`) were provably emitting wrong SQL, not merely duplicated SQL. sqlglot does not parenthesise by node nesting, and each composer knew a different subset of the precedence table — `_build_arith_or_cmp_ast` applied no precedence pass at all — so eight shapes emitted expressions that parse cleanly and mean something else: `not (a AND b)` → `NOT a AND b` (De Morgan); `-(a + b)` → `-a + b`; `(a = 5) IS NULL` → `a = 5 IS NULL`, read as `a = (5 IS NULL)` because `IS` binds tighter than `=`; `a AND (b OR c)` → `a AND b OR c`; `(a + b) * c` → `a + b * c`; `a - (b + c)` → `a - b + c`; `a + (b - c)` → `a + b - c` (`+` was treated as associative, which it is not over floats or fixed-precision decimals); and `(a > b) + 1` → `a > b + 1`. All three now delegate to one `render_arithmetic`. Sole exception: comparison operands in `_build_arithmetic_for_filter` keep `_paren_if_binary`, which parenthesises EVERY multi-term operand — strictly more grouping than the shared policy derives, never less, so it is a readability choice rather than a second correctness policy. The full suite passed without a single expectation edit, which is also why the bugs survived this long. Related: the renderer's precedence table now treats the comparison level as NON-associative, parenthesising an equal-precedence LEFT child there (arithmetic keeps left-associative flattening); and a bare `*` is refused as the source of any aggregation but `count`, which previously built `SUM(*)` / `COUNT(DISTINCT *)`. - 2026-08-05 — `%` is parenthesised unconditionally when nested, and grouping is pinned by a round-trip property test (DEV-1744). SQL puts `%` on the `*` / `/` tier, and so does SLayer's own Mode-B parser (`formula.py`, `ast.Mod: 2`) — but SQLGLOT's parser puts it on the `+` / `-` tier, so it reads `a + b % c` back as `(a + b) % c`. That matters because generated SQL is re-parsed by sqlglot inside our own pipeline (reserved-word pre-quoting, the log-alias transform), so relying on precedence for `%` means the expression is silently regrouped in flight, before any database sees it. The renderer therefore emits the parens rather than deriving them. Found by the meta-test rather than by review: `TestEveryOperatorPairSurvivesTheRoundTrip` renders every ordered operator pair in both operand positions across postgres/sqlite/mysql, re-parses the emitted SQL, and compares OPERAND STRUCTURE (parens and dialect-injected casts normalised away, since those are grouping-preservation and typing respectively). Re-parse *stability* is deliberately not the assertion — `a + b * c` re-parses to a stable string while meaning something other than the `(a + b) * c` that was built. The matrix is grouped by result type; feeding a boolean to an arithmetic operator is not a shape the binder builds, and dialects wrap such an operand in their own numeric cast. +- 2026-08-06 — Five parser-only scalars admitted to the binder; four deferred (DEV-1744). `SCALAR_PASSTHROUGH` (parser, `formula.py`) and `SCALAR_FUNCTIONS` (binder, `keys.py`) had drifted: the parser admitted nine names the binder then rejected with `UnknownFunctionError`, and the parser's own "Supported scalar functions" error text advertised them. `ceiling`, `sign`, `ltrim`, `rtrim`, `substring` are now admitted — every Tier-1 dialect emits one correct form for them, so there is no semantic ruling to make; `ceiling`/`substring` are aliases rendering to the same node as `ceil`/`substr`. The other four are deferred to their own issue because each needs a decision this consolidation should not make: `greatest`/`least` fall back to SQLite's `MAX(a, b)`/`MIN(a, b)`, which return NULL when an argument is NULL where Postgres ignores NULLs and Snowflake gets `GREATEST_IGNORE_NULLS` — the same class of divergence as the `concat` → `||` ruling above; `trunc` has four target forms including T-SQL `ROUND(x, 0, 1)` and a lowercase ClickHouse spelling on a case-sensitive backend; and `mod` is operator-shaped, so `exp.func("MOD", a, b)` builds a binary node the dialect rewrite rejects — it needs the special-casing `like` already has, in `render_scalar_call`, which the scope-assembly PR rewrites. Arity bounds are pinned tight for concrete reasons rather than caution: `ceiling(x, y)` silently emits `CEIL(x, y)` and `ceiling(x, y, z)` DuckDB's unrelated `CEIL(x TO z)`; a 4-arg `substring` drops one; and the 2-arg strip-these-characters trim form is excluded because sqlglot emits a literal `LTRIM(str, chars)` for some targets while MySQL's `LTRIM` takes one argument. The remaining divergence is pinned as an exact set by `TestParserAndBinderScalarSetsAgree`, so neither side can drift again unnoticed. - 2026-08-05 — Renderer call-site migration deferred to the scope-assembly PR (DEV-1744 → DEV-1746). PR 1 lands the complete, tested `render_value_key` API but does NOT reroute the generator's own render paths through it; only the ScalarCall policy is genuinely shared (all six paths call `render_scalar_call`). Deferred because the filter and composite paths carry state the context does not yet consume — the local-aggregate HAVING branch with its slot lookup and inline-aggregate emission, the first/last ranked state, the filter-side CAST policy, and the rn-suffix / filtered-rank / composite-alias / resolved-kwarg maps — and because the cross-scope paths are the ones that should supply `resolve(consumer=...)` its first production caller, which is scope-assembly work by nature. Splitting the reroute across two PRs would move that state twice. Full detail, per call site, in the `slayer/sql/render/value_expr.py` module docstring. diff --git a/docs/architecture/typed-keys.md b/docs/architecture/typed-keys.md index c73c9bc9..2b98e1ee 100644 --- a/docs/architecture/typed-keys.md +++ b/docs/architecture/typed-keys.md @@ -147,9 +147,12 @@ SCALAR_FUNCTIONS = frozenset({ "nullif", "coalesce", "ifnull", # math "ln", "log10", "log2", "log", "exp", "sqrt", "pow", "power", - "abs", "floor", "ceil", "round", + "abs", "floor", "ceil", "ceiling", "round", "sign", # string hygiene (was DEV-1378's STRING_HYGIENE_OPS) - "lower", "upper", "trim", "replace", "substr", "instr", "length", "concat", + "lower", "upper", "trim", "ltrim", "rtrim", + "replace", "substr", "substring", "instr", "length", "concat", + # pattern match — emits the SQL LIKE operator + "like", }) ``` diff --git a/docs/concepts/queries.md b/docs/concepts/queries.md index d0b095d2..5aaf4fbf 100644 --- a/docs/concepts/queries.md +++ b/docs/concepts/queries.md @@ -217,9 +217,9 @@ Multiple entries in the `filters` list are combined with AND. Filters in `SlayerQuery.filters` accept a small allowlist of lowercase SQL scalar functions for case-folding, trimming, substring extraction, -and string concatenation: `lower`, `upper`, `trim`, `replace`, `substr`, -`instr`, `length`, `concat`. The SQL `||` concat operator is rewritten -to `concat(...)` automatically. +and string concatenation: `lower`, `upper`, `trim`, `ltrim`, `rtrim`, +`replace`, `substr`, `substring`, `instr`, `length`, `concat`. The SQL +`||` concat operator is rewritten to `concat(...)` automatically. ```json "filters": [ diff --git a/docs/concepts/references.md b/docs/concepts/references.md index 4692ac65..104b807e 100644 --- a/docs/concepts/references.md +++ b/docs/concepts/references.md @@ -7,7 +7,7 @@ SLayer has two distinct expression layers and the rules for what each one accept | Mode | Fields | Parser | Accepts | Rejects | |---|---|---|---|---| | **A — SQL** | `Column.sql`, `Column.filter`, each entry of `SlayerModel.filters` | sqlglot | Any valid SQL expression for the underlying dialect — function calls (`json_extract`, `coalesce`, `nullif`, `lower`, `length`, …), arithmetic, `CASE WHEN`, string literals, comparison and boolean operators in SQL spelling (`=`, `<>`, `IS NULL`, `AND`, `OR`, `NOT`, `IN`, `LIKE`). Bare names and `__`-delimited join paths. | Aggregation colon syntax (`revenue:sum`); SLayer transform calls (`cumsum`, `change`, `rank`, …); references to `ModelMeasure` formulas; raw `OVER (...)` window functions inside `Column.filter` / `SlayerModel.filters` (allowed only in `Column.sql`). | -| **B — DSL** | `ModelMeasure.formula`, `SlayerQuery.measures`, `SlayerQuery.filters`, `SlayerQuery.dimensions`, `SlayerQuery.time_dimensions`, `SlayerQuery.order`, `SlayerQuery.main_time_dimension` | Python AST formula parser | Bare names that resolve to a `Column` or `ModelMeasure` on the model; single-dot dotted paths through joins (`customers.regions.name`, `customers.revenue:sum`); aggregation colon syntax (`:`, `*:count`, parametric forms); transform calls (`cumsum(revenue:sum)`, `rank(revenue:sum, partition_by=region)`); arithmetic / boolean / comparison operators; the SQL `\|\|` concat operator (folded into `concat(...)`); pattern matching via the `like(value, pattern)` scalar (emits the SQL `LIKE` operator — wrap in `not (...)` for `NOT LIKE`); a closed allowlist of lowercase scalar functions — null handling (`nullif`, `coalesce`, `ifnull`), math (`ln`, `log10`, `log2`, `log`, `exp`, `sqrt`, `pow`, `power`, `abs`, `floor`, `ceil`, `round`), string hygiene (`lower`, `upper`, `trim`, `replace`, `substr`, `instr`, `length`, `concat`) and `like`, each with a fixed argument count that is validated; `{variable}` placeholders (filters only). | `__`-delimited tokens in user input; raw SQL function calls outside that allowlist (`json_extract`, `date_trunc`, …), and any allowlisted call with the wrong number of arguments; raw `OVER (...)`; bare names that don't resolve to a Column / ModelMeasure / custom aggregation / query alias; **uppercase** spellings of the allowlisted functions (`LOWER`, `TRIM`, …) — DSL is case-sensitive; `NULL` inside an `in` / `not in` list (use `is null` / `is not null` instead — see below). | +| **B — DSL** | `ModelMeasure.formula`, `SlayerQuery.measures`, `SlayerQuery.filters`, `SlayerQuery.dimensions`, `SlayerQuery.time_dimensions`, `SlayerQuery.order`, `SlayerQuery.main_time_dimension` | Python AST formula parser | Bare names that resolve to a `Column` or `ModelMeasure` on the model; single-dot dotted paths through joins (`customers.regions.name`, `customers.revenue:sum`); aggregation colon syntax (`:`, `*:count`, parametric forms); transform calls (`cumsum(revenue:sum)`, `rank(revenue:sum, partition_by=region)`); arithmetic / boolean / comparison operators; the SQL `\|\|` concat operator (folded into `concat(...)`); pattern matching via the `like(value, pattern)` scalar (emits the SQL `LIKE` operator — wrap in `not (...)` for `NOT LIKE`); a closed allowlist of lowercase scalar functions — null handling (`nullif`, `coalesce`, `ifnull`), math (`ln`, `log10`, `log2`, `log`, `exp`, `sqrt`, `pow`, `power`, `abs`, `floor`, `ceil`, `ceiling`, `round`, `sign`), string hygiene (`lower`, `upper`, `trim`, `ltrim`, `rtrim`, `replace`, `substr`, `substring`, `instr`, `length`, `concat`) and `like`, each with a fixed argument count that is validated; `{variable}` placeholders (filters only). | `__`-delimited tokens in user input; raw SQL function calls outside that allowlist (`json_extract`, `date_trunc`, …), and any allowlisted call with the wrong number of arguments; raw `OVER (...)`; bare names that don't resolve to a Column / ModelMeasure / custom aggregation / query alias; **uppercase** spellings of the allowlisted functions (`LOWER`, `TRIM`, …) — DSL is case-sensitive; `NULL` inside an `in` / `not in` list (use `is null` / `is not null` instead — see below). | ## Identifier resolution @@ -139,10 +139,23 @@ Two consequences worth knowing: provide one, rather than becoming the generic two-argument `LOG(base, x)`. * **Argument counts are validated.** Each allowlisted scalar has a fixed arity - (`round` takes 1 or 2, `substr` 2 or 3, `replace` exactly 3; `coalesce` and - `concat` are variadic). A call with the wrong number is rejected with a - message naming the function, rather than being silently truncated or passed - through to fail at the database. + (`round` takes 1 or 2, `substr` / `substring` 2 or 3, `replace` exactly 3; + `coalesce` and `concat` are variadic). A call with the wrong number is + rejected with a message naming the function, rather than being silently + truncated or passed through to fail at the database. + + The bounds are deliberately tight where a wider call would translate into + something else entirely: `ceiling(x, y)` would emit `CEIL(x, y)` and + `ceiling(x, y, z)` DuckDB's unrelated `CEIL(x TO z)` rounding form, so + `ceil` / `ceiling` take exactly one argument. `ltrim` / `rtrim` likewise take + the string only — the "strip these characters" second argument is not + accepted, because MySQL's `LTRIM` takes a single argument and the call would + reach the server as SQL it rejects. + +* **Aliases render identically to their canonical spelling.** `ceiling` is + `ceil` and `substring` is `substr`; both spellings are accepted so a formula + written against either SQL convention binds, and both emit the target's own + form. ## `NULL` inside an `in` list diff --git a/slayer/core/keys.py b/slayer/core/keys.py index 1dfbd59c..91269d09 100644 --- a/slayer/core/keys.py +++ b/slayer/core/keys.py @@ -42,9 +42,10 @@ "nullif", "coalesce", "ifnull", # Math "ln", "log10", "log2", "log", "exp", "sqrt", "pow", "power", - "abs", "floor", "ceil", "round", + "abs", "floor", "ceil", "ceiling", "round", "sign", # String hygiene (was DEV-1378's STRING_HYGIENE_OPS) - "lower", "upper", "trim", "replace", "substr", "instr", "length", "concat", + "lower", "upper", "trim", "ltrim", "rtrim", + "replace", "substr", "substring", "instr", "length", "concat", # Pattern match — ``like(value, pattern)`` emits the SQL ``LIKE`` operator # (sqlglot ``exp.Like``); see SQLGenerator scalar-call rendering. "like", @@ -68,8 +69,17 @@ "exp": (1, 1), "sqrt": (1, 1), "pow": (2, 2), "power": (2, 2), "abs": (1, 1), "floor": (1, 1), "ceil": (1, 1), "round": (1, 2), + # ``ceiling`` is the T-SQL spelling of ``ceil`` and renders to the same + # node. Pinned at 1: a 2-arg call silently emits ``CEIL(x, y)``, and a + # 3-arg one becomes DuckDB's unrelated ``CEIL(x TO z)`` rounding form. + "ceiling": (1, 1), "sign": (1, 1), "lower": (1, 1), "upper": (1, 1), "trim": (1, 1), "length": (1, 1), - "replace": (3, 3), "substr": (2, 3), "instr": (2, 2), + # The trims take the string only, matching ``trim``. The 2-arg + # strip-these-characters form is deliberately NOT admitted: sqlglot emits + # a literal ``LTRIM(str, chars)`` for some targets, and MySQL's ``LTRIM`` + # accepts one argument — so it would be SQL the server rejects. + "ltrim": (1, 1), "rtrim": (1, 1), + "replace": (3, 3), "substr": (2, 3), "substring": (2, 3), "instr": (2, 2), "concat": (1, None), "like": (2, 2), } diff --git a/tests/test_dev1744_value_expr.py b/tests/test_dev1744_value_expr.py index e7174b68..c853a660 100644 --- a/tests/test_dev1744_value_expr.py +++ b/tests/test_dev1744_value_expr.py @@ -57,11 +57,13 @@ from sqlglot import exp from slayer.core.enums import BUILTIN_AGGREGATIONS, DataType, TimeGranularity +from slayer.core.formula import SCALAR_PASSTHROUGH from slayer.core.errors import ( RenderContextMissingFacilityError, UnknownReferenceError, ) from slayer.core.keys import ( + SCALAR_FUNCTIONS, AggregateKey, ArithmeticKey, BetweenKey, @@ -1841,6 +1843,113 @@ def test_accepted_arities_still_render(self, name, argc) -> None: assert out.sql(dialect="postgres") +class TestNewlyAdmittedScalars: + """Five scalars the PARSER already advertised but the BINDER rejected. + + ``SCALAR_PASSTHROUGH`` (parser) and ``SCALAR_FUNCTIONS`` (binder) had + drifted: a user writing ``ceiling(amount)`` got it past the parser and then + hit ``UnknownFunctionError`` at bind — while the parser's own + "Supported scalar functions" error text still advertised the name. These + five are admitted because they need no semantic ruling: every Tier-1 + dialect emits one correct form. + """ + + NEW = ["ceiling", "sign", "ltrim", "rtrim", "substring"] + + @pytest.mark.parametrize("name", NEW) + def test_is_admitted_by_the_binder(self, name) -> None: + assert name in SCALAR_FUNCTIONS + + @pytest.mark.parametrize( + "predicate,expected_fragment", + [ + ("ceiling(amount) > 25", "CEIL("), + ("sign(amount) > 0", "SIGN("), + ("ltrim(status) = 'new'", "LTRIM("), + ("rtrim(status) = 'new'", "RTRIM("), + ("substring(status, 1, 3) = 'new'", "SUBSTRING("), + ], + ) + async def test_binds_renders_and_executes( + self, e2e, predicate, expected_fragment, + ) -> None: + """Not just "the binder accepts it" — the SQL has to run.""" + resp = await e2e.execute( + SlayerQuery( + source_model="orders", + measures=[ModelMeasure(formula="*:count", name="n")], + filters=[predicate], + ), + ) + assert expected_fragment in (resp.sql or ""), resp.sql + assert resp.data, resp.sql + + @pytest.mark.parametrize( + "name,argc", + [("ceiling", 2), ("ceiling", 3), ("sign", 2), + ("ltrim", 2), ("rtrim", 2), ("substring", 1), ("substring", 4)], + ) + def test_wrong_arity_is_still_refused(self, name, argc) -> None: + """The arities are pinned tight for concrete reasons, not caution. + + ``ceiling(x, y)`` silently emits ``CEIL(x, y)`` and ``ceiling(x, y, z)`` + becomes DuckDB's unrelated ``CEIL(x TO z)`` rounding form; + ``substring`` with four arguments drops one. The 2-arg trims are + excluded because sqlglot emits a literal ``LTRIM(str, chars)`` for some + targets while MySQL's ``LTRIM`` takes one argument — SQL the server + would reject. + """ + args = [exp.column(f"c{i}") for i in range(argc)] + dialect = get_dialect("postgres") + with pytest.raises(NotImplementedError): + render_scalar_call(name=name, args=args, dialect=dialect) + + @pytest.mark.parametrize("name", NEW) + @pytest.mark.parametrize( + "dialect", + ["sqlite", "postgres", "duckdb", "mysql", "clickhouse", + "tsql", "bigquery", "snowflake"], + ) + def test_every_tier1_dialect_emits_and_keeps_every_argument( + self, name, dialect, + ) -> None: + """The reason these five are the "free" set: one correct form each, + with no argument silently dropped on any backend.""" + argc = 2 if name == "substring" else 1 + args = [exp.column(f"c{i}") for i in range(argc)] + out = render_scalar_call( + name=name, args=args, dialect=get_dialect(dialect), + ).sql(dialect=dialect) + for i in range(argc): + assert f"c{i}" in out, f"{name} dropped c{i} on {dialect}: {out}" + + +class TestParserAndBinderScalarSetsAgree: + """A tripwire on the remaining parser/binder divergence. + + ``SCALAR_PASSTHROUGH`` still admits four names the binder does not. Each + needs a decision this PR deliberately does not make: ``greatest`` / ``least`` + fall back to SQLite's ``MAX(a, b)`` / ``MIN(a, b)``, which return NULL when + an argument is NULL where Postgres ignores NULLs; ``trunc`` has four target + forms including T-SQL ``ROUND(x, 0, 1)`` and a lowercase ClickHouse + spelling; and ``mod`` is operator-shaped, so it does not even build through + the shared scalar policy today. + + Pinned as an exact set so neither side can drift again unnoticed — and so + admitting one of the four is a deliberate edit here, not a silent one. + """ + + def test_parser_only_names_are_exactly_the_deferred_four(self) -> None: + assert SCALAR_PASSTHROUGH - SCALAR_FUNCTIONS == { + "greatest", "least", "mod", "trunc", + } + + def test_like_is_the_only_binder_only_name(self) -> None: + """``like`` is an operator, not a pass-through function — the parser + handles it through its own internal ``__like__`` form.""" + assert SCALAR_FUNCTIONS - SCALAR_PASSTHROUGH == {"like"} + + class TestArityIsRejectedAtBindTime: """The renderer check is the backstop; the binder is where a user's typo should surface, with a message naming the function and the counts.""" From c5b3f36ec550fecd81c8b0327d3f3fc1c0b59142 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Thu, 6 Aug 2026 09:21:17 +0200 Subject: [PATCH 33/98] DEV-1744: name the follow-up issue for the four deferred scalars --- DECISIONS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/DECISIONS.md b/DECISIONS.md index 37fc79a5..b61e5c66 100644 --- a/DECISIONS.md +++ b/DECISIONS.md @@ -98,5 +98,5 @@ implementation detail. Include issue refs when known. - 2026-08-05 — One naming authority + one ValueKey renderer (DEV-1744, PR 1 of the DEV-1742 consolidation). Four consequences worth recording. (1) **Cross-model CTE dedup is now structural.** `_cm_` CTE names were minted by a doubly-lossy helper (`flat_name`, itself documented non-injective, then a non-identifier `re.sub`) and the resulting string doubled as the plan's identity key in `seen_cm`. Two reachable failures followed: two measures whose canonical aliases differ only in case emitted two `_cm_` names that fold together on every case-folding dialect (the collision belt raised — `_wm_` had been retrofitted onto the allocator years earlier, `_cm_` never was), and two genuinely distinct aggregates that sanitised alike made the second skip the loop body, leaving its join-back and column-alias maps unwritten and raising `KeyError` downstream. The dedup key is now the typed `AggregateKey` plus the source relation; the name is allocator-minted and stored once, so the five sites that used to re-derive it now read it. Deliberately NOT keyed on the canonical alias: `canonical_agg_name` omits `column_filter_key`, so a filtered and an unfiltered aggregate over one column share an alias while needing two CTEs — deduping on the string would silently merge them, a wrong answer rather than a crash. (2) **One ScalarCall policy.** Two of the five renderers returned `exp.Anonymous` passthrough, so `ifnull` reached Postgres — which has no `IFNULL` — while the same key emitted `COALESCE` from a projection. All six render paths now call one `render_scalar_call`. Transpiling alone was NOT sufficient: `exp.func("LOG10", x)` normalises to a generic `Log(10, x)` that re-emits as `LOG(10, x)`, wrong on dialects with a native single-arg `LOG10`, so the policy is transpile-then-log-alias-rewrite. Consequence surfaced and approved: `concat` now emits the `||` operator on Postgres/DuckDB where it previously emitted `CONCAT(...)`. That is a semantic change as well as a spelling one — Postgres `CONCAT()` ignores NULL operands, `||` propagates them. Ratified as the kept behavior: the projection path has always emitted `||`, so before this change SLayer disagreed with itself between filters and projections on the same input, and consistency was chosen over preserving the filter-side NULL tolerance. (3) **`ScopeFrame._model_for` raises** instead of falling back to the root model; the fallback turned a wiring bug into a wrong answer by expanding a different model's derived SQL. (4) **Named P-F carve-outs**, recorded rather than omitted: the T-SQL ORDER-BY-detach rewrite and the stage-schema wrapper take `_outer` / `_stage_inner` as shared CONSTANTS rather than allocator-minted names (the former is a post-generation AST pass with no allocator in reach, and a later PR rebuilds that machinery); `base` / `_base` / `_combined` keep their literal names but are reserved into the allocator. Superseded code is retained and callable per the chain's operating rule — deletion happens in the final PR. - 2026-08-05 — Arithmetic composition joins ScalarCall as a construct that renders ONCE, ahead of the call-site migration (DEV-1744). A carve-out from the deferral below, taken because the generator's three composers (`_build_arith_or_cmp_ast`, `_compose_arithmetic_op`, `_build_arithmetic_for_filter`) were provably emitting wrong SQL, not merely duplicated SQL. sqlglot does not parenthesise by node nesting, and each composer knew a different subset of the precedence table — `_build_arith_or_cmp_ast` applied no precedence pass at all — so eight shapes emitted expressions that parse cleanly and mean something else: `not (a AND b)` → `NOT a AND b` (De Morgan); `-(a + b)` → `-a + b`; `(a = 5) IS NULL` → `a = 5 IS NULL`, read as `a = (5 IS NULL)` because `IS` binds tighter than `=`; `a AND (b OR c)` → `a AND b OR c`; `(a + b) * c` → `a + b * c`; `a - (b + c)` → `a - b + c`; `a + (b - c)` → `a + b - c` (`+` was treated as associative, which it is not over floats or fixed-precision decimals); and `(a > b) + 1` → `a > b + 1`. All three now delegate to one `render_arithmetic`. Sole exception: comparison operands in `_build_arithmetic_for_filter` keep `_paren_if_binary`, which parenthesises EVERY multi-term operand — strictly more grouping than the shared policy derives, never less, so it is a readability choice rather than a second correctness policy. The full suite passed without a single expectation edit, which is also why the bugs survived this long. Related: the renderer's precedence table now treats the comparison level as NON-associative, parenthesising an equal-precedence LEFT child there (arithmetic keeps left-associative flattening); and a bare `*` is refused as the source of any aggregation but `count`, which previously built `SUM(*)` / `COUNT(DISTINCT *)`. - 2026-08-05 — `%` is parenthesised unconditionally when nested, and grouping is pinned by a round-trip property test (DEV-1744). SQL puts `%` on the `*` / `/` tier, and so does SLayer's own Mode-B parser (`formula.py`, `ast.Mod: 2`) — but SQLGLOT's parser puts it on the `+` / `-` tier, so it reads `a + b % c` back as `(a + b) % c`. That matters because generated SQL is re-parsed by sqlglot inside our own pipeline (reserved-word pre-quoting, the log-alias transform), so relying on precedence for `%` means the expression is silently regrouped in flight, before any database sees it. The renderer therefore emits the parens rather than deriving them. Found by the meta-test rather than by review: `TestEveryOperatorPairSurvivesTheRoundTrip` renders every ordered operator pair in both operand positions across postgres/sqlite/mysql, re-parses the emitted SQL, and compares OPERAND STRUCTURE (parens and dialect-injected casts normalised away, since those are grouping-preservation and typing respectively). Re-parse *stability* is deliberately not the assertion — `a + b * c` re-parses to a stable string while meaning something other than the `(a + b) * c` that was built. The matrix is grouped by result type; feeding a boolean to an arithmetic operator is not a shape the binder builds, and dialects wrap such an operand in their own numeric cast. -- 2026-08-06 — Five parser-only scalars admitted to the binder; four deferred (DEV-1744). `SCALAR_PASSTHROUGH` (parser, `formula.py`) and `SCALAR_FUNCTIONS` (binder, `keys.py`) had drifted: the parser admitted nine names the binder then rejected with `UnknownFunctionError`, and the parser's own "Supported scalar functions" error text advertised them. `ceiling`, `sign`, `ltrim`, `rtrim`, `substring` are now admitted — every Tier-1 dialect emits one correct form for them, so there is no semantic ruling to make; `ceiling`/`substring` are aliases rendering to the same node as `ceil`/`substr`. The other four are deferred to their own issue because each needs a decision this consolidation should not make: `greatest`/`least` fall back to SQLite's `MAX(a, b)`/`MIN(a, b)`, which return NULL when an argument is NULL where Postgres ignores NULLs and Snowflake gets `GREATEST_IGNORE_NULLS` — the same class of divergence as the `concat` → `||` ruling above; `trunc` has four target forms including T-SQL `ROUND(x, 0, 1)` and a lowercase ClickHouse spelling on a case-sensitive backend; and `mod` is operator-shaped, so `exp.func("MOD", a, b)` builds a binary node the dialect rewrite rejects — it needs the special-casing `like` already has, in `render_scalar_call`, which the scope-assembly PR rewrites. Arity bounds are pinned tight for concrete reasons rather than caution: `ceiling(x, y)` silently emits `CEIL(x, y)` and `ceiling(x, y, z)` DuckDB's unrelated `CEIL(x TO z)`; a 4-arg `substring` drops one; and the 2-arg strip-these-characters trim form is excluded because sqlglot emits a literal `LTRIM(str, chars)` for some targets while MySQL's `LTRIM` takes one argument. The remaining divergence is pinned as an exact set by `TestParserAndBinderScalarSetsAgree`, so neither side can drift again unnoticed. +- 2026-08-06 — Five parser-only scalars admitted to the binder; four deferred (DEV-1744). `SCALAR_PASSTHROUGH` (parser, `formula.py`) and `SCALAR_FUNCTIONS` (binder, `keys.py`) had drifted: the parser admitted nine names the binder then rejected with `UnknownFunctionError`, and the parser's own "Supported scalar functions" error text advertised them. `ceiling`, `sign`, `ltrim`, `rtrim`, `substring` are now admitted — every Tier-1 dialect emits one correct form for them, so there is no semantic ruling to make; `ceiling`/`substring` are aliases rendering to the same node as `ceil`/`substr`. The other four are deferred to DEV-1753 because each needs a decision this consolidation should not make: `greatest`/`least` fall back to SQLite's `MAX(a, b)`/`MIN(a, b)`, which return NULL when an argument is NULL where Postgres ignores NULLs and Snowflake gets `GREATEST_IGNORE_NULLS` — the same class of divergence as the `concat` → `||` ruling above; `trunc` has four target forms including T-SQL `ROUND(x, 0, 1)` and a lowercase ClickHouse spelling on a case-sensitive backend; and `mod` is operator-shaped, so `exp.func("MOD", a, b)` builds a binary node the dialect rewrite rejects — it needs the special-casing `like` already has, in `render_scalar_call`, which the scope-assembly PR rewrites. Arity bounds are pinned tight for concrete reasons rather than caution: `ceiling(x, y)` silently emits `CEIL(x, y)` and `ceiling(x, y, z)` DuckDB's unrelated `CEIL(x TO z)`; a 4-arg `substring` drops one; and the 2-arg strip-these-characters trim form is excluded because sqlglot emits a literal `LTRIM(str, chars)` for some targets while MySQL's `LTRIM` takes one argument. The remaining divergence is pinned as an exact set by `TestParserAndBinderScalarSetsAgree`, so neither side can drift again unnoticed. - 2026-08-05 — Renderer call-site migration deferred to the scope-assembly PR (DEV-1744 → DEV-1746). PR 1 lands the complete, tested `render_value_key` API but does NOT reroute the generator's own render paths through it; only the ScalarCall policy is genuinely shared (all six paths call `render_scalar_call`). Deferred because the filter and composite paths carry state the context does not yet consume — the local-aggregate HAVING branch with its slot lookup and inline-aggregate emission, the first/last ranked state, the filter-side CAST policy, and the rn-suffix / filtered-rank / composite-alias / resolved-kwarg maps — and because the cross-scope paths are the ones that should supply `resolve(consumer=...)` its first production caller, which is scope-assembly work by nature. Splitting the reroute across two PRs would move that state twice. Full detail, per call site, in the `slayer/sql/render/value_expr.py` module docstring. From be740a8effbc414fb0c362e4a01e135a6f3d24e7 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Thu, 6 Aug 2026 09:47:40 +0200 Subject: [PATCH 34/98] =?UTF-8?q?DEV-1745:=20address=20CodeRabbit=20+=20So?= =?UTF-8?q?nar=20review=20=E2=80=94=20two=20production=20bugs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CRITICAL — parametric aggregates crashed planning. AggregateKey.args/kwargs and ScalarCallKey.args normalise numeric literals to Decimal, so `price:percentile(p=0.9)` puts a Decimal in the key tree. The reachability visitor's inline-scalar allowlist omitted it, and the fail-closed visitor rejected the legitimate key: ANY filter over a parametric aggregate raised UnhandledValueKindError during planning. Decimal is now an inline scalar and the end-to-end shape is pinned. MAJOR — a non-SQL kwarg could take the query down. Fragment discovery scanned every string kwarg as SQL, including reserved markers like window='90d'. That was harmless while the scan swallowed parse errors; now that the door raises, a marker whose text is not parseable SQL would be fatal ('%Y-%m' is the easy example). Only values the aggregation's formula actually SUBSTITUTES are treated as fragments now. Both walkers in filter_reachability now share one child-dispatch (_child_keys), so a new key kind is handled — or rejected — identically by each. That also clears the two Sonar S3776 complexity issues, and sorts TransformKey's partition_keys: it is a frozenset, and its iteration order was feeding JOIN emission order, so the emitted SQL could vary between runs. Warning rendering gains a shared human_message() on the payload. MCP and the CLI both fell back to interpolating the Pydantic model for any non-dropped kind, so a NormalizationWarning printed as its repr; now a new kind cannot regress one surface without the other. MCP format="json" also stays ONE parseable JSON value — warnings move inside the payload instead of being appended as prose, which broke json.loads on exactly the queries worth inspecting. Smaller review items: the date_range warning no longer links to a docs page that does not exist; DroppedFilterWarning is imported at the top per CLAUDE.md and the return type narrowed; the SlayerResponse.warnings comment describes both kinds; both REST pages document rule_doc_url; classify_host_filter's slot bucketing is extracted (Sonar S3776); the now-dead target params are gone from _register_routed_filter_joins; test hygiene for Sonar S5778/S9073; DuckDB connections use context managers; fragment assertions are scoped to the _cm_ CTE body; the derived-expansion helper binds the same model instance the resolver returns. Also fixes two self-contradictions on docs/concepts/references.md that arrived with the dev-1744 merge (variadic scalars described as fixed-arity; "Two consequences" over three bullets). Reviewed and NOT changed: CodeRabbit's scalar-arity import-time invariant is already implemented (that thread is marked outdated); the BigQuery quoted-dotted-identifier baseline records what BigQuery genuinely does with Postgres-flavoured Mode-A SQL — " delimits strings there — which is a dialect fact, not a defect, and is now documented in the fixture. Suite 10670 passed; SQLite + DuckDB integration 118 passed; ruff clean. Co-Authored-By: Claude Opus 5 (1M context) --- docs/concepts/references.md | 4 +- docs/interfaces/rest-api.md | 2 +- docs/reference/rest-api.md | 2 +- slayer/cli.py | 9 +- slayer/core/warnings.py | 22 +++ slayer/engine/cross_model_planner.py | 54 ++++--- slayer/engine/filter_reachability.py | 176 ++++++++++------------- slayer/engine/normalization.py | 3 +- slayer/engine/query_engine.py | 26 ++-- slayer/mcp/server.py | 58 +++++--- slayer/sql/generator.py | 35 +++-- tests/test_dev1745_date_range_warning.py | 3 +- tests/test_dev1745_derived_expansion.py | 30 ++-- tests/test_dev1745_fragment_joins.py | 113 ++++++++++++--- tests/test_dev1745_golden_sql.py | 9 +- tests/test_dev1745_mode_a_door.py | 9 +- tests/test_dev1745_reachability.py | 59 +++++++- tests/test_dev1745_warning_contract.py | 3 +- 18 files changed, 402 insertions(+), 215 deletions(-) diff --git a/docs/concepts/references.md b/docs/concepts/references.md index 4692ac65..5e0a1486 100644 --- a/docs/concepts/references.md +++ b/docs/concepts/references.md @@ -7,7 +7,7 @@ SLayer has two distinct expression layers and the rules for what each one accept | Mode | Fields | Parser | Accepts | Rejects | |---|---|---|---|---| | **A — SQL** | `Column.sql`, `Column.filter`, each entry of `SlayerModel.filters` | sqlglot | Any valid SQL expression for the underlying dialect — function calls (`json_extract`, `coalesce`, `nullif`, `lower`, `length`, …), arithmetic, `CASE WHEN`, string literals, comparison and boolean operators in SQL spelling (`=`, `<>`, `IS NULL`, `AND`, `OR`, `NOT`, `IN`, `LIKE`). Bare names and `__`-delimited join paths. | Aggregation colon syntax (`revenue:sum`); SLayer transform calls (`cumsum`, `change`, `rank`, …); references to `ModelMeasure` formulas; raw `OVER (...)` window functions inside `Column.filter` / `SlayerModel.filters` (allowed only in `Column.sql`). | -| **B — DSL** | `ModelMeasure.formula`, `SlayerQuery.measures`, `SlayerQuery.filters`, `SlayerQuery.dimensions`, `SlayerQuery.time_dimensions`, `SlayerQuery.order`, `SlayerQuery.main_time_dimension` | Python AST formula parser | Bare names that resolve to a `Column` or `ModelMeasure` on the model; single-dot dotted paths through joins (`customers.regions.name`, `customers.revenue:sum`); aggregation colon syntax (`:`, `*:count`, parametric forms); transform calls (`cumsum(revenue:sum)`, `rank(revenue:sum, partition_by=region)`); arithmetic / boolean / comparison operators; the SQL `\|\|` concat operator (folded into `concat(...)`); pattern matching via the `like(value, pattern)` scalar (emits the SQL `LIKE` operator — wrap in `not (...)` for `NOT LIKE`); a closed allowlist of lowercase scalar functions — null handling (`nullif`, `coalesce`, `ifnull`), math (`ln`, `log10`, `log2`, `log`, `exp`, `sqrt`, `pow`, `power`, `abs`, `floor`, `ceil`, `round`), string hygiene (`lower`, `upper`, `trim`, `replace`, `substr`, `instr`, `length`, `concat`) and `like`, each with a fixed argument count that is validated; `{variable}` placeholders (filters only). | `__`-delimited tokens in user input; raw SQL function calls outside that allowlist (`json_extract`, `date_trunc`, …), and any allowlisted call with the wrong number of arguments; raw `OVER (...)`; bare names that don't resolve to a Column / ModelMeasure / custom aggregation / query alias; **uppercase** spellings of the allowlisted functions (`LOWER`, `TRIM`, …) — DSL is case-sensitive; `NULL` inside an `in` / `not in` list (use `is null` / `is not null` instead — see below). | +| **B — DSL** | `ModelMeasure.formula`, `SlayerQuery.measures`, `SlayerQuery.filters`, `SlayerQuery.dimensions`, `SlayerQuery.time_dimensions`, `SlayerQuery.order`, `SlayerQuery.main_time_dimension` | Python AST formula parser | Bare names that resolve to a `Column` or `ModelMeasure` on the model; single-dot dotted paths through joins (`customers.regions.name`, `customers.revenue:sum`); aggregation colon syntax (`:`, `*:count`, parametric forms); transform calls (`cumsum(revenue:sum)`, `rank(revenue:sum, partition_by=region)`); arithmetic / boolean / comparison operators; the SQL `\|\|` concat operator (folded into `concat(...)`); pattern matching via the `like(value, pattern)` scalar (emits the SQL `LIKE` operator — wrap in `not (...)` for `NOT LIKE`); a closed allowlist of lowercase scalar functions — null handling (`nullif`, `coalesce`, `ifnull`), math (`ln`, `log10`, `log2`, `log`, `exp`, `sqrt`, `pow`, `power`, `abs`, `floor`, `ceil`, `round`), string hygiene (`lower`, `upper`, `trim`, `replace`, `substr`, `instr`, `length`, `concat`) and `like`, each with a declared argument count that is validated (`coalesce` and `concat` are variadic); `{variable}` placeholders (filters only). | `__`-delimited tokens in user input; raw SQL function calls outside that allowlist (`json_extract`, `date_trunc`, …), and any allowlisted call with the wrong number of arguments; raw `OVER (...)`; bare names that don't resolve to a Column / ModelMeasure / custom aggregation / query alias; **uppercase** spellings of the allowlisted functions (`LOWER`, `TRIM`, …) — DSL is case-sensitive; `NULL` inside an `in` / `not in` list (use `is null` / `is not null` instead — see below). | ## Identifier resolution @@ -122,7 +122,7 @@ than being passed through verbatim. `length(x)` emits `LEN(x)` on SQL Server, `substr(x, 1, 5)` emits `SUBSTRING(x FROM 1 FOR 5)` on Postgres, and `ifnull(x, 0)` emits `COALESCE(x, 0)` on backends without `IFNULL`. -Two consequences worth knowing: +Three consequences worth knowing: * **`concat` follows SQL string-concatenation semantics.** On dialects whose natural spelling is the `||` operator (Postgres, DuckDB, SQLite), `concat(a, b)` diff --git a/docs/interfaces/rest-api.md b/docs/interfaces/rest-api.md index 8bb0a9e7..9d7163ef 100644 --- a/docs/interfaces/rest-api.md +++ b/docs/interfaces/rest-api.md @@ -79,7 +79,7 @@ presence of a field: | `kind` | Meaning | Extra fields | | -- | -- | -- | -| `normalization` | The input was rewritten to canonical form | `rule_id`, `original`, `normalized`, `location` | +| `normalization` | The input was rewritten to canonical form | `rule_id`, `original`, `normalized`, `location`, `rule_doc_url` (nullable) | | `unreachable_filter_dropped` | A filter could not be applied inside a cross-model CTE and was dropped from it (it still applies at the host) | `filter_text`, `location`, `reason` | A dropped filter changes which rows the answer covers, so it is worth surfacing diff --git a/docs/reference/rest-api.md b/docs/reference/rest-api.md index 9507745f..8982782a 100644 --- a/docs/reference/rest-api.md +++ b/docs/reference/rest-api.md @@ -60,7 +60,7 @@ presence of a field: | `kind` | Meaning | Extra fields | | -- | -- | -- | -| `normalization` | The input was rewritten to canonical form | `rule_id`, `original`, `normalized`, `location` | +| `normalization` | The input was rewritten to canonical form | `rule_id`, `original`, `normalized`, `location`, `rule_doc_url` (nullable) | | `unreachable_filter_dropped` | A filter could not be applied inside a cross-model CTE and was dropped from it (it still applies at the host) | `filter_text`, `location`, `reason` | A dropped filter changes which rows the answer covers, so it is worth surfacing diff --git a/slayer/cli.py b/slayer/cli.py index 3debf884..8baaa557 100644 --- a/slayer/cli.py +++ b/slayer/cli.py @@ -1233,14 +1233,7 @@ def _print_query_warnings(result) -> None: something that changes which rows the answer covers (DEV-1745 W5 / D2). """ for w in (getattr(result, "warnings", None) or []): - if getattr(w, "kind", None) == "unreachable_filter_dropped": - print( - f"warning: dropped filter {w.filter_text!r} " - f"(at {w.location}): {w.reason}", - file=sys.stderr, - ) - else: - print(f"warning: {w}", file=sys.stderr) + print(f"warning: {w.human_message()}", file=sys.stderr) def _run_query(args): # NOSONAR S3776 — argparse-driven dispatch; one straight-line function reads better than threaded helpers diff --git a/slayer/core/warnings.py b/slayer/core/warnings.py index 6211bd23..39846419 100644 --- a/slayer/core/warnings.py +++ b/slayer/core/warnings.py @@ -33,6 +33,16 @@ class SlayerWarning(BaseModel): kind: str + def human_message(self) -> str: + """One operator-readable line describing this advisory. + + Lives on the payload so MCP, the CLI, and any future surface render a + given kind identically, and so a NEW kind cannot silently fall back to + a Pydantic repr on one surface and a hand-written string on another. + Subclasses override; the base is the honest last resort. + """ + return f"{self.kind}: {self.model_dump(exclude={'kind'})}" + class NormalizationWarning(SlayerWarning): """Structured payload describing one slack-normalization rewrite. @@ -51,6 +61,12 @@ class NormalizationWarning(SlayerWarning): location: str rule_doc_url: Optional[str] = None + def human_message(self) -> str: + return ( + f"[{self.rule_id}] rewrote {self.original} → {self.normalized} " + f"(at {self.location})" + ) + class DroppedFilterWarning(SlayerWarning): """A user filter that could not be applied where it was routed. @@ -65,6 +81,12 @@ class DroppedFilterWarning(SlayerWarning): location: str reason: str + def human_message(self) -> str: + return ( + f"dropped filter {self.filter_text!r} (at {self.location}): " + f"{self.reason}" + ) + # The response carries a DISCRIMINATED union, not the bare base class: Pydantic # validates a ``List[SlayerWarning]`` down to the base type and would silently diff --git a/slayer/engine/cross_model_planner.py b/slayer/engine/cross_model_planner.py index e50e2756..b01c1452 100644 --- a/slayer/engine/cross_model_planner.py +++ b/slayer/engine/cross_model_planner.py @@ -133,6 +133,37 @@ class HostFilterRouting(BaseModel): # --------------------------------------------------------------------------- +def _classify_referenced_slots( + *, + referenced_slot_ids: List[SlotId], + host_slots: List[ValueSlot], + target_path: Tuple[str, ...], +) -> Tuple[List[SlotId], List[SlotId], List[SlotId]]: + """Split a filter's referenced slots into + ``(unknown, aggregate_on_target, aggregate_other)``. + + Only AGGREGATE slots are sorted here — row-level reachability comes from + the structural summary, not from slot keys. An aggregate is routed by WHERE + it is computed: one whose source path IS the target can be propagated as a + HAVING inside that CTE, one computed anywhere else cannot be evaluated + there at all. + """ + by_id = {s.id: s for s in host_slots} + unknown: List[SlotId] = [] + on_target: List[SlotId] = [] + other: List[SlotId] = [] + for sid in referenced_slot_ids: + slot = by_id.get(sid) + if slot is None: + # Unknown slot id — be conservative, treat as unreachable. + unknown.append(sid) + elif isinstance(slot.key, AggregateKey): + agg_path = getattr(slot.key.source, "path", ()) + bucket = on_target if agg_path == target_path else other + bucket.append(sid) + return unknown, on_target, other + + def classify_host_filter( *, host_filter: HostFilterRouting, @@ -165,24 +196,11 @@ def classify_host_filter( if host_filter.phase == Phase.POST: return FilterRoute.STAY_AT_HOST_POST - by_id = {s.id: s for s in host_slots} - - unknown: List[SlotId] = [] - aggregate_on_target: List[SlotId] = [] - aggregate_other: List[SlotId] = [] - - for sid in host_filter.referenced_slot_ids: - s = by_id.get(sid) - if s is None: - # Unknown slot id — be conservative, treat as unreachable. - unknown.append(sid) - continue - if isinstance(s.key, AggregateKey): - agg_path = getattr(s.key.source, "path", ()) - if agg_path == target_path: - aggregate_on_target.append(sid) - else: - aggregate_other.append(sid) + unknown, aggregate_on_target, aggregate_other = _classify_referenced_slots( + referenced_slot_ids=host_filter.referenced_slot_ids, + host_slots=host_slots, + target_path=target_path, + ) crossed = tuple(host_filter.crossed_join_paths) unreachable_paths = [ diff --git a/slayer/engine/filter_reachability.py b/slayer/engine/filter_reachability.py index fd51a89e..498202c9 100644 --- a/slayer/engine/filter_reachability.py +++ b/slayer/engine/filter_reachability.py @@ -31,6 +31,7 @@ from __future__ import annotations +from decimal import Decimal from typing import List, Tuple from sqlglot import exp @@ -160,6 +161,62 @@ def _derived_sql_touches_anchor( return False +# Values a key tree can carry INLINE — plain data, not references, so they +# cannot cross a join. ``Decimal`` is load-bearing: ``AggregateKey.args`` / +# ``kwargs`` and ``ScalarCallKey.args`` normalise numeric literals to it, so a +# parametric aggregate like ``price:percentile(p=0.9)`` puts a Decimal in the +# tree. Omitting it made the fail-closed visitor reject a legitimate key. +_INLINE_SCALARS = (str, int, float, bool, Decimal) + +# Leaf kinds: they carry references but no child keys. +_LEAF_KINDS = (LiteralKey, StarKey, SqlExprKey, ColumnKey, ColumnSqlKey) + + +def _child_keys(node, *, descend_aggregates: bool = True) -> Tuple: + """The child keys of a composite node, in a STABLE order. + + One dispatch shared by both visitors, so a new key kind is handled — or + rejected — identically by each. Fails CLOSED on an unknown kind: a silent + empty result would read as "crosses nothing" and route a filter into a + scope that cannot evaluate it. + + ``partition_keys`` is a frozenset, whose iteration order varies between + runs; sorted here because the discovered paths drive JOIN emission order, + and non-deterministic SQL is its own bug. + + ``descend_aggregates=False`` stops at an aggregate: for host-locality, an + aggregate is routed by WHERE it is computed, not by its inputs. + """ + if isinstance(node, _LEAF_KINDS) or isinstance(node, _INLINE_SCALARS): + return () + if isinstance(node, TimeTruncKey): + return (node.column,) + if isinstance(node, AggregateKey): + if not descend_aggregates: + return () + return ( + node.source, + *node.args, + *(v for _name, v in node.kwargs), + node.column_filter_key, + ) + if isinstance(node, TransformKey): + return ( + node.input, + *sorted(node.partition_keys, key=repr), + node.time_key, + ) + if isinstance(node, ArithmeticKey): + return tuple(node.operands) + if isinstance(node, ScalarCallKey): + return tuple(node.args) + if isinstance(node, InKey): + return (node.column, *node.values) + if isinstance(node, BetweenKey): + return (node.column, node.low, node.high) + raise UnhandledValueKindError(node) + + def compute_key_join_paths( *, key, anchor_model, anchor_relation: str, bundle, ) -> Tuple[Path, ...]: @@ -181,66 +238,21 @@ def _add(path: Path) -> None: def _walk(node) -> None: if node is None: return - if isinstance(node, (LiteralKey, StarKey)): - return - if isinstance(node, ColumnKey): + if isinstance(node, (ColumnKey, ColumnSqlKey)): for p in _prefixes(node.path): _add(p) - return if isinstance(node, ColumnSqlKey): - for p in _prefixes(node.path): - _add(p) for p in _derived_sql_paths( key=node, anchor_model=anchor_model, anchor_relation=anchor_relation, bundle=bundle, ): _add(p) - return - if isinstance(node, SqlExprKey): + elif isinstance(node, SqlExprKey): for p in node.referenced_join_paths: for pre in _prefixes(tuple(p)): _add(pre) - return - if isinstance(node, TimeTruncKey): - _walk(node.column) - return - if isinstance(node, AggregateKey): - _walk(node.source) - for a in node.args: - _walk(a) - for _name, v in node.kwargs: - _walk(v) - _walk(node.column_filter_key) - return - if isinstance(node, TransformKey): - _walk(node.input) - for pk in node.partition_keys: - _walk(pk) - _walk(node.time_key) - return - if isinstance(node, ArithmeticKey): - for o in node.operands: - _walk(o) - return - if isinstance(node, ScalarCallKey): - for a in node.args: - _walk(a) - return - if isinstance(node, InKey): - _walk(node.column) - for v in node.values: - _walk(v) - return - if isinstance(node, BetweenKey): - _walk(node.column) - _walk(node.low) - _walk(node.high) - return - # Scalars carried inline by TransformKey args / kwargs are values, not - # references, and cannot cross anything. - if isinstance(node, (str, int, float, bool)) or node is None: - return - raise UnhandledValueKindError(node) + for child in _child_keys(node): + _walk(child) _walk(key) return tuple(seen) @@ -258,62 +270,28 @@ def key_has_host_local_ref( has an empty anchored path but is NOT host-local — inside the target's scope its expansion resolves. """ - found = False - def _walk(node) -> None: - nonlocal found - if found or node is None: - return - if isinstance(node, (LiteralKey, StarKey, SqlExprKey)): - return + def _is_local(node) -> bool: if isinstance(node, ColumnKey): - if not node.path: - found = True - return + return not node.path if isinstance(node, ColumnSqlKey): - if node.path: - return - if _derived_sql_touches_anchor( + return not node.path and _derived_sql_touches_anchor( key=node, anchor_model=anchor_model, anchor_relation=anchor_relation, bundle=bundle, - ): - found = True - return - if isinstance(node, TimeTruncKey): - _walk(node.column) - return - if isinstance(node, AggregateKey): - # An aggregate is routed by its own decision table arm (on-target - # vs elsewhere), not by host-locality of its inputs. - return - if isinstance(node, TransformKey): - _walk(node.input) - for pk in node.partition_keys: - _walk(pk) - _walk(node.time_key) - return - if isinstance(node, ArithmeticKey): - for o in node.operands: - _walk(o) - return - if isinstance(node, ScalarCallKey): - for a in node.args: - _walk(a) - return - if isinstance(node, InKey): - _walk(node.column) - return - if isinstance(node, BetweenKey): - _walk(node.column) - _walk(node.low) - _walk(node.high) - return - if isinstance(node, (str, int, float, bool)): - return - raise UnhandledValueKindError(node) + ) + return False - _walk(key) - return found + def _walk(node) -> bool: + if node is None: + return False + if _is_local(node): + return True + return any( + _walk(child) + for child in _child_keys(node, descend_aggregates=False) + ) + + return _walk(key) def path_is_reachable(*, path: Path, target_path: Path) -> bool: diff --git a/slayer/engine/normalization.py b/slayer/engine/normalization.py index b167a51f..fa3f6ea6 100644 --- a/slayer/engine/normalization.py +++ b/slayer/engine/normalization.py @@ -597,7 +597,8 @@ def _apply_malformed_date_range( original=f"time_dimensions[{i}].date_range={list(date_range)!r}", normalized="(ignored — no date filter emitted)", location=f"time_dimensions[{i}].date_range", - rule_doc_url="docs/agent_input_slack.md#malformed-date-range", + # No rule_doc_url: docs/agent_input_slack.md does not exist, and a + # link to a missing page is worse than no link. ) emitted.append(payload) _warnings_module.warn( diff --git a/slayer/engine/query_engine.py b/slayer/engine/query_engine.py index b102c2da..2ad760d0 100644 --- a/slayer/engine/query_engine.py +++ b/slayer/engine/query_engine.py @@ -37,7 +37,11 @@ list_valued_variable_names, substitute_variables, ) -from slayer.core.warnings import AnySlayerWarning, NormalizationWarning +from slayer.core.warnings import ( + AnySlayerWarning, + DroppedFilterWarning, + NormalizationWarning, +) from slayer.core.recommend import ( CandidateCoverage, ItemPath, @@ -386,7 +390,9 @@ def _stage_location(stages, index: int) -> str: return f"stage {name!r}.filters" if name else f"stages[{index}].filters" -def _collect_dropped_filter_warnings(*, planned_list, stages) -> List[Any]: +def _collect_dropped_filter_warnings( + *, planned_list, stages, +) -> List[DroppedFilterWarning]: """Dropped-filter payloads for the whole pipeline, one per user filter. Identity is ``(location, original filter text)`` — stated in the terms the @@ -397,8 +403,6 @@ def _collect_dropped_filter_warnings(*, planned_list, stages) -> List[Any]: reached different conclusions about one filter, which is a planner inconsistency; raising beats silently keeping whichever came first. """ - from slayer.core.warnings import DroppedFilterWarning - by_identity: "dict[tuple[str, str], DroppedFilterWarning]" = {} for index, planned in enumerate(planned_list): location = _stage_location(stages, index) @@ -427,7 +431,6 @@ def _emit_dropped_filter_warnings(response) -> None: Called once, at the outermost boundary, AFTER the response is built. """ from slayer.core.errors import UnreachableFilterDroppedWarning - from slayer.core.warnings import DroppedFilterWarning for w in response.warnings or (): if isinstance(w, DroppedFilterWarning): @@ -446,12 +449,13 @@ class SlayerResponse(BaseModel): columns: List[str] = PydanticField(default_factory=list) sql: Optional[str] = None attributes: ResponseAttributes = PydanticField(default_factory=ResponseAttributes) - # DEV-1450 stage 6 — slack-normalization warnings. - # Structured payload for each slack rewrite the normalization layer - # performed on the input (function-style aggs, misplaced measures, - # AST-resolvable dotted refs in raw SQL). Empty for queries that - # arrived in canonical form. Surfaced alongside the result so - # REST / MCP / CLI consumers can echo the rewrites back to authors. + # Advisories about the query itself, discriminated on ``kind``: + # ``normalization`` for a slack rewrite the normalization layer performed + # on the input (function-style aggs, misplaced measures, AST-resolvable + # dotted refs in raw SQL), and ``unreachable_filter_dropped`` for a user + # filter dropped from a cross-model CTE. Empty for a clean query. Surfaced + # alongside the result so REST / MCP / CLI consumers can echo them back to + # authors; switch on ``kind`` rather than on the presence of a field. warnings: List[AnySlayerWarning] = PydanticField(default_factory=list) @model_validator(mode="after") diff --git a/slayer/mcp/server.py b/slayer/mcp/server.py index e803bc33..9b59d158 100644 --- a/slayer/mcp/server.py +++ b/slayer/mcp/server.py @@ -1955,11 +1955,24 @@ def _format_table(data: list[dict[str, Any]], columns: list[str], max_rows: int return result -def _format_json(data: list[dict[str, Any]], columns: list[str]) -> str: - """Format data as JSON array.""" +def _format_json( + data: list[dict[str, Any]], + columns: list[str], + warnings: list[dict[str, Any]] | None = None, +) -> str: + """Format data as JSON. + + A bare array when there is nothing to report, so the long-standing shape is + unchanged for every clean query. When warnings exist they go INSIDE the + JSON as ``{"data": [...], "warnings": [...]}`` — appending them as prose + would break ``json.loads`` on exactly the queries a caller most needs to + inspect (DEV-1745 W5). + """ import json - return json.dumps(data, default=str) + if not warnings: + return json.dumps(data, default=str) + return json.dumps({"data": data, "warnings": warnings}, default=str) def _format_csv(data: list[dict[str, Any]], columns: list[str]) -> str: @@ -1979,32 +1992,35 @@ def _format_csv(data: list[dict[str, Any]], columns: list[str]) -> str: def _format_warnings(result: SlayerResponse) -> str: - """Advisories about the query, appended to every output format. + """Advisories about the query, appended to the TEXT output formats. A dropped filter changes which rows the answer covers, so it cannot be - left to a field the caller might not read (DEV-1745 W5 / D2). + left to a field the caller might not read (DEV-1745 W5 / D2). Rendering + goes through each payload's ``human_message`` so this surface and the CLI + cannot describe the same warning differently. """ - lines = [] - for w in (result.warnings or []): - if getattr(w, "kind", None) == "unreachable_filter_dropped": - lines.append( - f" - dropped filter {w.filter_text!r} " - f"(at {w.location}): {w.reason}" - ) - else: - lines.append(f" - {getattr(w, 'rule_id', w.kind)}: {w}") + lines = [f" - {w.human_message()}" for w in (result.warnings or [])] return "" if not lines else "\n\nWarnings:\n" + "\n".join(lines) def _format_output(result: SlayerResponse, fmt: str) -> str: - """Format query output in the requested format.""" + """Format query output in the requested format. + + ``json`` stays ONE parseable JSON value: warnings go INSIDE the payload as + a ``warnings`` key rather than being appended as prose, which would make + ``json.loads`` fail on exactly the queries that most need reporting. The + text formats append a human-readable block instead. + """ if fmt == "csv": - body = _format_csv(data=result.data, columns=result.columns) - elif fmt == "markdown": - body = result.to_markdown() - else: - body = _format_json(data=result.data, columns=result.columns) - return body + _format_warnings(result) + return _format_csv(data=result.data, columns=result.columns) \ + + _format_warnings(result) + if fmt == "markdown": + return result.to_markdown() + _format_warnings(result) + return _format_json( + data=result.data, + columns=result.columns, + warnings=[w.model_dump(mode="json") for w in (result.warnings or [])], + ) def _format_field_meta(entries: dict[str, Any]) -> list[str]: diff --git a/slayer/sql/generator.py b/slayer/sql/generator.py index 01d38285..b51de554 100644 --- a/slayer/sql/generator.py +++ b/slayer/sql/generator.py @@ -5878,9 +5878,6 @@ def _register_filter_join_paths(sql_text: Optional[str]) -> None: self._register_routed_filter_joins( planned_query=planned_query, filter_ids=list(plan.where_filter_ids) + list(plan.having_filter_ids), - target_relation=target_relation, - target_model=target_model, - bundle=bundle, scope=cte_scope, target_path=target_path, ) @@ -6049,9 +6046,6 @@ def _register_routed_filter_joins( # NOSONAR(S3776) — a cohesive recursive Va *, planned_query, filter_ids: List[str], - target_relation: str, - target_model, - bundle, scope: ScopeFrame, target_path: Tuple[str, ...], ) -> None: @@ -8365,17 +8359,32 @@ def _register_fragment_kwarg_joins( ``SUM(customers.spend * regions.weight) FROM customers`` — SQL no database accepts. One implementation, entered through the one door, is what stops that from recurring (DEV-1745 W2). + + Only values the aggregation's formula actually SUBSTITUTES are treated + as SQL. A string kwarg whose ``{name}`` never appears in the template is + a marker, not a fragment — ``revenue:sum(window='90d')`` being the + standing example — and handing it to a SQL parser is meaningless. It + was harmless while the scan swallowed parse errors; now that the door + raises, a marker that does not happen to parse would take the query + down with it. """ - fragments = [v for _, v in key.kwargs if isinstance(v, str)] agg_def = next( (a for a in (model.aggregations or []) if a.name == key.agg), None, ) - if agg_def is not None: - overridden = {name for name, _ in key.kwargs} - fragments.extend( - p.sql for p in (agg_def.params or []) - if p.name not in overridden and p.sql - ) + if agg_def is None: + # A built-in aggregation has no template, so no kwarg of it is a + # SQL fragment. + return + formula = agg_def.formula or "" + overridden = {name for name, _ in key.kwargs} + fragments = [ + v for name, v in key.kwargs + if isinstance(v, str) and f"{{{name}}}" in formula + ] + fragments.extend( + p.sql for p in (agg_def.params or []) + if p.name not in overridden and p.sql + ) for frag in fragments: self._enter_mode_a_expression( sql=frag, scope=scope, diff --git a/tests/test_dev1745_date_range_warning.py b/tests/test_dev1745_date_range_warning.py index 053d3e07..34a029c5 100644 --- a/tests/test_dev1745_date_range_warning.py +++ b/tests/test_dev1745_date_range_warning.py @@ -122,7 +122,8 @@ async def test_malformed_emits_no_date_filter(self, date_range) -> None: async def test_well_formed_still_filters(self) -> None: sql = await self._sql(["2024-01-01", "2024-12-31"]) - assert "2024-01-01" in sql and "2024-12-31" in sql + assert "2024-01-01" in sql + assert "2024-12-31" in sql async def test_absent_matches_empty_emission(self) -> None: assert await self._sql(None) == await self._sql([]) diff --git a/tests/test_dev1745_derived_expansion.py b/tests/test_dev1745_derived_expansion.py index fcdb1f8d..74833c57 100644 --- a/tests/test_dev1745_derived_expansion.py +++ b/tests/test_dev1745_derived_expansion.py @@ -94,8 +94,12 @@ def _resolve(name: str): def _expand(sql: str) -> str: + # The SAME instance ``_resolve`` hands back. ``_process_column_node_sync`` + # compares ``target_model is model`` by IDENTITY to decide ``next_is_root``; + # passing a second, equal-but-distinct ``_orders()` would make that test + # false the moment the root path resolves through ``resolve_model``. out = expand_derived_refs_sync( - sql=sql, model=_orders(), alias_path="orders", + sql=sql, model=_MODELS["orders"], alias_path="orders", resolve_model=_resolve, dialect="postgres", is_root=True, ) assert out is not None, f"expansion returned None for {sql!r}" @@ -245,18 +249,18 @@ async def test_derived_of_derived_executes_on_duckdb() -> None: model=_orders(), dialect="duckdb", validate=False, extra_models=[_customers(), _regions()], ) - con = duckdb.connect() - con.execute( - "CREATE TABLE orders(id INT, customer_id INT, amount DOUBLE, status VARCHAR)" - ) - con.execute("CREATE TABLE customers(id INT, region_id INT)") - con.execute( - "CREATE TABLE regions(id INT, status VARCHAR, population DOUBLE)" - ) - con.execute("INSERT INTO orders VALUES (1, 1, 10.0, 'ok')") - con.execute("INSERT INTO customers VALUES (1, 1)") - con.execute("INSERT INTO regions VALUES (1, 'live', 50.0)") + with duckdb.connect() as con: + con.execute( + "CREATE TABLE orders(id INT, customer_id INT, amount DOUBLE, status VARCHAR)" + ) + con.execute("CREATE TABLE customers(id INT, region_id INT)") + con.execute( + "CREATE TABLE regions(id INT, status VARCHAR, population DOUBLE)" + ) + con.execute("INSERT INTO orders VALUES (1, 1, 10.0, 'ok')") + con.execute("INSERT INTO customers VALUES (1, 1)") + con.execute("INSERT INTO regions VALUES (1, 'live', 50.0)") - rows = con.execute(sql).fetchall() + rows = con.execute(sql).fetchall() # regions.population = 50 -> pop_x2 = 100 assert rows == [(100.0, 10.0)], f"unexpected rows {rows!r} for SQL:\n{sql}" diff --git a/tests/test_dev1745_fragment_joins.py b/tests/test_dev1745_fragment_joins.py index d9a38d81..b1695471 100644 --- a/tests/test_dev1745_fragment_joins.py +++ b/tests/test_dev1745_fragment_joins.py @@ -111,7 +111,9 @@ def _orders_local_agg() -> SlayerModel: ) -async def _sql(query: SlayerQuery, *, model: SlayerModel, dialect="postgres") -> str: +async def _sql( + query: SlayerQuery, *, model: SlayerModel, dialect: str = "postgres", +) -> str: return await _engine_generate( query=query, model=model, dialect=dialect, validate=False, extra_models=[_customers(), _regions()], @@ -121,6 +123,71 @@ async def _sql(query: SlayerQuery, *, model: SlayerModel, dialect="postgres") -> # --------------------------------------------------------------------------- +class TestOnlySubstitutedKwargsAreSql: + """A string kwarg is a SQL fragment only when the aggregation's template + substitutes it. Anything else is a marker, and handing a marker to a SQL + parser is meaningless — harmless while the scan swallowed parse errors, + query-fatal now that the door raises.""" + + @staticmethod + def _entered_fragments(*, kwargs, agg="sum") -> list: + from slayer.core.keys import AggregateKey, ColumnKey + from slayer.sql.generator import SQLGenerator + + gen = SQLGenerator(dialect="postgres") + seen: list = [] + gen._enter_mode_a_expression = ( # type: ignore[method-assign] + lambda **kw: seen.append(kw["sql"]) + ) + gen._register_fragment_kwarg_joins( + key=AggregateKey( + source=ColumnKey(path=(), leaf="spend"), agg=agg, + kwargs=kwargs, + ), + scope=object(), + model=_customers(), + ) + return seen + + def test_reserved_marker_kwarg_is_not_parsed_as_sql(self) -> None: + """``window='90d'`` is the standing example — a marker on a BUILT-IN + aggregation, which has no template to substitute it into.""" + assert self._entered_fragments(kwargs=(("window", "90d"),)) == [] + + def test_marker_that_is_not_parseable_sql_is_still_skipped(self) -> None: + """The failure this guards: a marker whose text sqlglot rejects. It + must never reach the door, which raises.""" + assert self._entered_fragments(kwargs=(("fmt", "%Y-%m"),)) == [] + + def test_a_substituted_kwarg_is_still_scanned(self) -> None: + """The counter-case, so the filter is not blanket suppression: + ``wscaled_sum``'s template does substitute ``{w}``.""" + entered = self._entered_fragments( + kwargs=(("w", "regions.weight"),), agg="wscaled_sum", + ) + assert entered == ["regions.weight"], entered + + +def _cm_body(sql: str) -> str: + """The body of the `_cm_` CTE. + + Assertions about which alias the fragment rendered belong to THIS scope: + a whole-SQL check can be satisfied — or defeated — by a perfectly valid + alias in the host base or the combined SELECT. + """ + start = sql.index("_cm_") + open_paren = sql.index("(", start) + depth = 0 + for i in range(open_paren, len(sql)): + if sql[i] == "(": + depth += 1 + elif sql[i] == ")": + depth -= 1 + if depth == 0: + return sql[open_paren + 1:i] + raise AssertionError(f"unbalanced _cm_ CTE in:\n{sql}") + + @pytest.mark.asyncio class TestCrossModelFragmentJoins: """The `_cm_` CTE gap — a crossing template fragment must pull its join @@ -137,11 +204,15 @@ async def test_cm_cte_joins_the_fragment_target(self) -> None: ), model=_orders(), ) - # the fragment renders regions.weight ... - assert "regions.weight" in sql, sql - # ... so regions MUST be joined in the same scope - assert "JOIN regions" in sql, ( - f"fragment's crossed join missing from the CTE FROM:\n{sql}" + body = _cm_body(sql) + # the fragment renders regions.weight, anchored at the CTE's own root ... + assert "regions.weight" in body, body + assert "customers__regions.weight" not in body, ( + f"fragment was anchored at the HOST path, not the CTE root:\n{body}" + ) + # ... so regions MUST be joined in that same scope + assert "JOIN regions" in body, ( + f"fragment's crossed join missing from the CTE FROM:\n{body}" ) async def test_cm_cte_with_sibling_local_measure(self) -> None: @@ -156,9 +227,13 @@ async def test_cm_cte_with_sibling_local_measure(self) -> None: ), model=_orders(), ) - assert "regions.weight" in sql, sql - assert "JOIN regions" in sql, ( - f"fragment's crossed join missing from the CTE FROM:\n{sql}" + body = _cm_body(sql) + assert "regions.weight" in body, body + assert "customers__regions.weight" not in body, ( + f"fragment was anchored at the HOST path, not the CTE root:\n{body}" + ) + assert "JOIN regions" in body, ( + f"fragment's crossed join missing from the CTE FROM:\n{body}" ) @@ -207,16 +282,16 @@ async def test_cross_model_fragment_executes_on_duckdb() -> None: ), model=_orders(), dialect="duckdb", ) - con = duckdb.connect() - con.execute( - "CREATE TABLE orders(id INT, customer_id INT, amount DOUBLE, status VARCHAR)" - ) - con.execute("CREATE TABLE customers(id INT, region_id INT, spend DOUBLE)") - con.execute("CREATE TABLE regions(id INT, weight DOUBLE)") - con.execute("INSERT INTO orders VALUES (1, 1, 10.0, 'ok')") - con.execute("INSERT INTO customers VALUES (1, 1, 7.0)") - con.execute("INSERT INTO regions VALUES (1, 3.0)") + with duckdb.connect() as con: + con.execute( + "CREATE TABLE orders(id INT, customer_id INT, amount DOUBLE, status VARCHAR)" + ) + con.execute("CREATE TABLE customers(id INT, region_id INT, spend DOUBLE)") + con.execute("CREATE TABLE regions(id INT, weight DOUBLE)") + con.execute("INSERT INTO orders VALUES (1, 1, 10.0, 'ok')") + con.execute("INSERT INTO customers VALUES (1, 1, 7.0)") + con.execute("INSERT INTO regions VALUES (1, 3.0)") - rows = con.execute(sql).fetchall() + rows = con.execute(sql).fetchall() # SUM(customers.spend * regions.weight) = 7 * 3 = 21 assert rows == [("ok", 21.0)], f"unexpected rows {rows!r} for SQL:\n{sql}" diff --git a/tests/test_dev1745_golden_sql.py b/tests/test_dev1745_golden_sql.py index b94166f5..76f6e003 100644 --- a/tests/test_dev1745_golden_sql.py +++ b/tests/test_dev1745_golden_sql.py @@ -133,7 +133,14 @@ def _orders() -> SlayerModel: filter="customers.tier = 'eu'", type=DataType.DOUBLE), # raw ref that inlines to a constant (dual-scan contract) Column(name="flag_const", sql="1", type=DataType.INT), - # quoted dotted identifier + # Quoted dotted identifier. Mode-A is raw SQL for the TARGET + # dialect, and this spelling is Postgres-flavoured: BigQuery quotes + # identifiers with backticks and reads "..." as a STRING, so there + # it parses as the literal expression 'customers'.'spend' — no + # column reference, therefore no join to discover. The BigQuery + # baseline entry records exactly that, which is what the dialect + # does with this input rather than a defect to fix here. Mode-A + # portability across quoting styles is noted on DEV-1746. Column(name="quoted_cross", sql='"customers"."spend"', type=DataType.DOUBLE), # derived-of-derived, two hops (currently BROKEN — see module doc) diff --git a/tests/test_dev1745_mode_a_door.py b/tests/test_dev1745_mode_a_door.py index e9c71975..ed03020f 100644 --- a/tests/test_dev1745_mode_a_door.py +++ b/tests/test_dev1745_mode_a_door.py @@ -307,21 +307,24 @@ class TestParseFailureRaises: def test_unparseable_predicate_raises(self) -> None: from slayer.core.errors import SlayerError + scope = _scope() with pytest.raises(SlayerError): - _scope().enter_predicate("this is ( not sql") + scope.enter_predicate("this is ( not sql") def test_unparseable_expression_raises(self) -> None: from slayer.core.errors import SlayerError + scope = _scope() with pytest.raises(SlayerError): - _scope().enter_expression("SELECT ((( FROM") + scope.enter_expression("SELECT ((( FROM") def test_error_carries_the_original_fragment(self) -> None: from slayer.core.errors import SlayerError fragment = "this is ( not sql" + scope = _scope() with pytest.raises(SlayerError) as excinfo: - _scope().enter_predicate(fragment) + scope.enter_predicate(fragment) assert fragment in str(excinfo.value), ( "the error must name the offending fragment" ) diff --git a/tests/test_dev1745_reachability.py b/tests/test_dev1745_reachability.py index ac7d15db..9e54712c 100644 --- a/tests/test_dev1745_reachability.py +++ b/tests/test_dev1745_reachability.py @@ -251,10 +251,11 @@ def test_unknown_key_kind_fails_closed(self) -> None: class _Bogus: pass + bogus, model, bundle = _Bogus(), _orders(), _bundle() with pytest.raises(UnhandledValueKindError) as excinfo: compute_key_join_paths( - key=_Bogus(), anchor_model=_orders(), - anchor_relation="orders", bundle=_bundle(), + key=bogus, anchor_model=model, + anchor_relation="orders", bundle=bundle, ) assert "_Bogus" in str(excinfo.value), ( "the error must identify the unhandled key type" @@ -393,6 +394,60 @@ def test_empty_path_is_host_local(self) -> None: assert route == FilterRoute.DROP_HOST_LOCAL +class TestInlineScalarsAreNotReferences: + """A key tree carries plain VALUES as well as references. The fail-closed + visitor must recognise them as data, not reject them as an unknown kind.""" + + def test_decimal_aggregate_kwarg_is_scalar(self) -> None: + """``price:percentile(p=0.9)`` normalises 0.9 to a Decimal and puts it + in AggregateKey.kwargs. Rejecting it took down planning for every + filter over a parametric aggregate.""" + from decimal import Decimal + + key = AggregateKey( + source=ColumnKey(path=("customers",), leaf="balance"), + agg="percentile", + kwargs=(("p", Decimal("0.9")),), + ) + assert ("customers",) in _paths_for(key) + + def test_decimal_scalar_call_arg_is_scalar(self) -> None: + from decimal import Decimal + + from slayer.core.keys import ScalarCallKey + + key = ScalarCallKey( + name="round", + args=(ColumnKey(path=("customers",), leaf="balance"), Decimal("2")), + ) + assert ("customers",) in _paths_for(key) + + def test_string_and_bool_args_are_scalars(self) -> None: + key = AggregateKey( + source=ColumnKey(path=("customers",), leaf="balance"), + agg="sum", + kwargs=(("window", "90d"), ("flag", True)), + ) + assert ("customers",) in _paths_for(key) + + def test_parametric_aggregate_filter_plans(self) -> None: + """End-to-end: the shape that crashed. A filter over a parametric + aggregate must plan, not raise.""" + from slayer.core.query import SlayerQuery + from slayer.engine.stage_planner import plan_query + + planned = plan_query( + query=SlayerQuery( + source_model="orders", + dimensions=[{"formula": "amount", "name": "amount"}], + measures=[{"formula": "amount:percentile(p=0.9)", "name": "p90"}], + filters=["amount:percentile(p=0.9) > 100"], + ), + bundle=_bundle(), + ) + assert planned.filters_by_phase + + class TestCoordinateSystemInvariant: """D9: every reachability summary is expressed in the coordinate system of the ``PlannedQuery`` that owns it — recomputed per plan, NEVER copied. diff --git a/tests/test_dev1745_warning_contract.py b/tests/test_dev1745_warning_contract.py index ea0d509b..e6c56edf 100644 --- a/tests/test_dev1745_warning_contract.py +++ b/tests/test_dev1745_warning_contract.py @@ -476,10 +476,11 @@ async def test_warnings_as_errors_raises(self) -> None: with tempfile.TemporaryDirectory() as d: engine = await _engine(d) + query = _query() with warnings.catch_warnings(): warnings.simplefilter("error", UnreachableFilterDroppedWarning) with pytest.raises(UnreachableFilterDroppedWarning): - await engine.execute(_query(), dry_run=True) + await engine.execute(query, dry_run=True) @pytest.mark.asyncio From d15372d81b6b5b3d1be5afdc155aaf78ce655de3 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Thu, 6 Aug 2026 10:08:32 +0200 Subject: [PATCH 35/98] DEV-1745: clear the remaining Sonar issues MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit S3776 (cognitive complexity 24) on compute_key_join_paths — the leaf-path collection is extracted to _leaf_paths, leaving the traversal as "collect from this node, then descend into its children". Same paths, same order. S8495 on _child_keys — it returns a variable-length SEQUENCE of children, not a fixed record, so a list says what it means and the rule no longer reads varying tuple arity as a defect. S1172 — _format_json never used its `columns` parameter; dropped it and the argument at its one call site. Suite 10729 passed; SQLite + DuckDB integration 118 passed; ruff clean. Co-Authored-By: Claude Opus 5 (1M context) --- slayer/engine/filter_reachability.py | 68 ++++++++++++++++++---------- slayer/mcp/server.py | 2 - 2 files changed, 43 insertions(+), 27 deletions(-) diff --git a/slayer/engine/filter_reachability.py b/slayer/engine/filter_reachability.py index 498202c9..35731374 100644 --- a/slayer/engine/filter_reachability.py +++ b/slayer/engine/filter_reachability.py @@ -172,9 +172,11 @@ def _derived_sql_touches_anchor( _LEAF_KINDS = (LiteralKey, StarKey, SqlExprKey, ColumnKey, ColumnSqlKey) -def _child_keys(node, *, descend_aggregates: bool = True) -> Tuple: +def _child_keys(node, *, descend_aggregates: bool = True) -> List: """The child keys of a composite node, in a STABLE order. + A variable-length SEQUENCE, not a fixed record — hence a list. + One dispatch shared by both visitors, so a new key kind is handled — or rejected — identically by each. Fails CLOSED on an unknown kind: a silent empty result would read as "crosses nothing" and route a filter into a @@ -188,35 +190,59 @@ def _child_keys(node, *, descend_aggregates: bool = True) -> Tuple: aggregate is routed by WHERE it is computed, not by its inputs. """ if isinstance(node, _LEAF_KINDS) or isinstance(node, _INLINE_SCALARS): - return () + return [] if isinstance(node, TimeTruncKey): - return (node.column,) + return [node.column] if isinstance(node, AggregateKey): if not descend_aggregates: - return () - return ( + return [] + return [ node.source, *node.args, *(v for _name, v in node.kwargs), node.column_filter_key, - ) + ] if isinstance(node, TransformKey): - return ( + return [ node.input, *sorted(node.partition_keys, key=repr), node.time_key, - ) + ] if isinstance(node, ArithmeticKey): - return tuple(node.operands) + return list(node.operands) if isinstance(node, ScalarCallKey): - return tuple(node.args) + return list(node.args) if isinstance(node, InKey): - return (node.column, *node.values) + return [node.column, *node.values] if isinstance(node, BetweenKey): - return (node.column, node.low, node.high) + return [node.column, node.low, node.high] raise UnhandledValueKindError(node) +def _leaf_paths(node, *, anchor_model, anchor_relation: str, bundle) -> List[Path]: + """Join paths a LEAF key is itself anchored at. + + Composites contribute nothing here — their dependencies arrive through + ``_child_keys``. Split out of the traversal so the walk stays a two-line + "collect, then descend". + """ + if isinstance(node, (ColumnKey, ColumnSqlKey)): + paths = _prefixes(node.path) + if isinstance(node, ColumnSqlKey): + paths += _derived_sql_paths( + key=node, anchor_model=anchor_model, + anchor_relation=anchor_relation, bundle=bundle, + ) + return paths + if isinstance(node, SqlExprKey): + return [ + pre + for p in node.referenced_join_paths + for pre in _prefixes(tuple(p)) + ] + return [] + + def compute_key_join_paths( *, key, anchor_model, anchor_relation: str, bundle, ) -> Tuple[Path, ...]: @@ -238,19 +264,11 @@ def _add(path: Path) -> None: def _walk(node) -> None: if node is None: return - if isinstance(node, (ColumnKey, ColumnSqlKey)): - for p in _prefixes(node.path): - _add(p) - if isinstance(node, ColumnSqlKey): - for p in _derived_sql_paths( - key=node, anchor_model=anchor_model, - anchor_relation=anchor_relation, bundle=bundle, - ): - _add(p) - elif isinstance(node, SqlExprKey): - for p in node.referenced_join_paths: - for pre in _prefixes(tuple(p)): - _add(pre) + for path in _leaf_paths( + node, anchor_model=anchor_model, + anchor_relation=anchor_relation, bundle=bundle, + ): + _add(path) for child in _child_keys(node): _walk(child) diff --git a/slayer/mcp/server.py b/slayer/mcp/server.py index 9b59d158..69343dae 100644 --- a/slayer/mcp/server.py +++ b/slayer/mcp/server.py @@ -1957,7 +1957,6 @@ def _format_table(data: list[dict[str, Any]], columns: list[str], max_rows: int def _format_json( data: list[dict[str, Any]], - columns: list[str], warnings: list[dict[str, Any]] | None = None, ) -> str: """Format data as JSON. @@ -2018,7 +2017,6 @@ def _format_output(result: SlayerResponse, fmt: str) -> str: return result.to_markdown() + _format_warnings(result) return _format_json( data=result.data, - columns=result.columns, warnings=[w.model_dump(mode="json") for w in (result.warnings or [])], ) From ee8dca704a014ce75fab0d08bcf01f29103eb1f9 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Thu, 6 Aug 2026 11:54:50 +0200 Subject: [PATCH 36/98] DEV-1745: build the leaf-path list fresh instead of appending to _prefixes' MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit From the Codex review of the extraction. `_prefixes` returns a freshly built list today, so `paths += _derived_sql_paths(...)` is safe — but it mutates a value this function does not own, and if `_prefixes` ever became memoised (it is a pure function of a short tuple, so that is a plausible optimisation) the append would poison the cache and silently change join sets and their order. Splitting the ColumnSqlKey and ColumnKey arms makes the composition explicit and the ownership obvious. Co-Authored-By: Claude Opus 5 (1M context) --- slayer/engine/filter_reachability.py | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/slayer/engine/filter_reachability.py b/slayer/engine/filter_reachability.py index 35731374..0d865c6a 100644 --- a/slayer/engine/filter_reachability.py +++ b/slayer/engine/filter_reachability.py @@ -226,14 +226,20 @@ def _leaf_paths(node, *, anchor_model, anchor_relation: str, bundle) -> List[Pat ``_child_keys``. Split out of the traversal so the walk stays a two-line "collect, then descend". """ - if isinstance(node, (ColumnKey, ColumnSqlKey)): - paths = _prefixes(node.path) - if isinstance(node, ColumnSqlKey): - paths += _derived_sql_paths( + if isinstance(node, ColumnSqlKey): + # Own anchored path first, then whatever its expansion reaches — the + # order the FROM builder consumes. Built as a NEW list rather than + # appending to ``_prefixes``' return, so this cannot corrupt that + # result if it ever becomes cached. + return [ + *_prefixes(node.path), + *_derived_sql_paths( key=node, anchor_model=anchor_model, anchor_relation=anchor_relation, bundle=bundle, - ) - return paths + ), + ] + if isinstance(node, ColumnKey): + return _prefixes(node.path) if isinstance(node, SqlExprKey): return [ pre From e03cb1fac533fcce90299ad79dbe46b1063500ae Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Thu, 6 Aug 2026 12:46:59 +0200 Subject: [PATCH 37/98] DEV-1745: CSV output regression, one reachability rule, and review follow-ups MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CSV was my regression: _format_output appended the warning block to the CSV string, so each warning became a record with the wrong column count and broke every reader on exactly the queries worth inspecting. Warnings are now leading `#` comment lines, which leaves every DATA record uniform. The docstring no longer overclaims either — it covers warnings only, and says so: show_sql / explain / attributes still wrap JSON output in prose, which predates this PR. classify_host_filter re-implemented the prefix comparison inline while filter_reachability.path_is_reachable calls itself "the ONE reachability rule". Two copies free to drift is the failure this PR exists to remove; the classifier now calls it. Derived-column expansion was parsed twice per key — once for the crossed set, once for host-locality — for every filter on every plan, with _expand_derived_refs_any_dialect re-parsing per hop of a derived-of-derived chain. Now memoised through a cache the CALLER owns, one per plan. Deliberately not a module-level cache keyed by id(bundle): CPython reuses ids after collection, so a fresh bundle could be served a dead one's entry, and a plan-scoped dict cannot outlive the bundle it was built for. _mode_a_scope's docstring claimed its join_paths were "a byproduct nobody reads". The shifted-CTE residual path reads them back and registers them on the shifted scope — the behaviour was right, the stated invariant was not, and a reader could have relied on it when adding a call site. Tests: test_plan_carries_frame_bound_columns asserted hasattr on a field with a default_factory, so it could not fail — it now checks model_fields like its sibling. Two classifier tests still passed host_model_name after the routing stopped consulting it, which would have hidden a reintroduced name-based branch. Added the two scan cases with no coverage: SqlExprKey's own referenced paths, and InKey values. Suite 10731 passed; SQLite + DuckDB integration 118 passed; ruff clean. Co-Authored-By: Claude Opus 5 (1M context) --- slayer/engine/cross_model_planner.py | 6 ++- slayer/engine/filter_reachability.py | 53 +++++++++++++++++++++---- slayer/engine/stage_planner.py | 5 +++ slayer/mcp/server.py | 30 +++++++++++--- slayer/sql/generator.py | 15 ++++--- tests/test_cross_model_planner.py | 2 - tests/test_dev1745_plan_time_routing.py | 8 +++- tests/test_dev1745_reachability.py | 23 +++++++++++ 8 files changed, 118 insertions(+), 24 deletions(-) diff --git a/slayer/engine/cross_model_planner.py b/slayer/engine/cross_model_planner.py index b01c1452..b4e557e7 100644 --- a/slayer/engine/cross_model_planner.py +++ b/slayer/engine/cross_model_planner.py @@ -78,6 +78,7 @@ bind_time_dimension, walk_value_keys, ) +from slayer.engine.filter_reachability import path_is_reachable from slayer.engine.planned import ( BoundFilterId, CrossModelAggregatePlan, @@ -203,8 +204,11 @@ def classify_host_filter( ) crossed = tuple(host_filter.crossed_join_paths) + # THE rule lives in one place. Repeating the prefix comparison here would + # be a second copy free to drift from it — the exact failure this PR removes. unreachable_paths = [ - p for p in crossed if tuple(p) != tuple(target_path[: len(p)]) + p for p in crossed + if not path_is_reachable(path=p, target_path=target_path) ] if unknown or aggregate_other or unreachable_paths: diff --git a/slayer/engine/filter_reachability.py b/slayer/engine/filter_reachability.py index 0d865c6a..d3b9de40 100644 --- a/slayer/engine/filter_reachability.py +++ b/slayer/engine/filter_reachability.py @@ -32,7 +32,7 @@ from __future__ import annotations from decimal import Decimal -from typing import List, Tuple +from typing import List, Optional, Tuple from sqlglot import exp @@ -90,6 +90,7 @@ def _prefixes(path: Path) -> List[Path]: def _expanded_derived_ast( *, key: ColumnSqlKey, anchor_model, anchor_relation: str, bundle, + cache: "Optional[dict]" = None, ): """The parsed, expanded AST of a derived column's ``Column.sql``. @@ -97,7 +98,35 @@ def _expanded_derived_ast( ``__``-path alias with ``is_root=False`` when the column lives on a joined model — so the refs inside come out already prefixed by the key's own path and an anchor-rooted scan resolves them without further adjustment. + + Memoised through the caller-supplied ``cache``. Both visitors ask for the + same key's expansion — ``_derived_sql_paths`` for the crossed set and + ``_derived_sql_touches_anchor`` for host-locality — and both run for every + filter on every plan, while ``_expand_derived_refs_any_dialect`` itself + re-parses per hop of a derived-of-derived chain. + + The cache is passed IN rather than held module-level on purpose. A global + keyed by ``id(bundle)`` would be unsound: CPython reuses ids once an object + is collected, so a fresh bundle could be handed a dead one's entry. A dict + owned by one plan-level call cannot outlive the bundle it was built for. """ + if cache is None: + return _expanded_derived_ast_uncached( + key=key, anchor_model=anchor_model, + anchor_relation=anchor_relation, bundle=bundle, + ) + cache_key = (key.model, key.column_name, key.path, anchor_relation) + if cache_key not in cache: + cache[cache_key] = _expanded_derived_ast_uncached( + key=key, anchor_model=anchor_model, + anchor_relation=anchor_relation, bundle=bundle, + ) + return cache[cache_key] + + +def _expanded_derived_ast_uncached( + *, key: ColumnSqlKey, anchor_model, anchor_relation: str, bundle, +): model = ( anchor_model if key.model == getattr(anchor_model, "name", None) else bundle.get_referenced_model(key.model) @@ -117,11 +146,12 @@ def _expanded_derived_ast( def _derived_sql_paths( *, key: ColumnSqlKey, anchor_model, anchor_relation: str, bundle, + cache: "Optional[dict]" = None, ) -> List[Path]: """Join paths the expansion of a derived column's ``Column.sql`` crosses.""" parsed = _expanded_derived_ast( key=key, anchor_model=anchor_model, - anchor_relation=anchor_relation, bundle=bundle, + anchor_relation=anchor_relation, bundle=bundle, cache=cache, ) if parsed is None: return [] @@ -135,6 +165,7 @@ def _derived_sql_paths( def _derived_sql_touches_anchor( *, key: ColumnSqlKey, anchor_model, anchor_relation: str, bundle, + cache: "Optional[dict]" = None, ) -> bool: """Whether a derived column's expansion references the ANCHOR relation. @@ -149,7 +180,7 @@ def _derived_sql_touches_anchor( """ parsed = _expanded_derived_ast( key=key, anchor_model=anchor_model, - anchor_relation=anchor_relation, bundle=bundle, + anchor_relation=anchor_relation, bundle=bundle, cache=cache, ) if parsed is None: # Nothing resolvable to inspect — a bare column name on the anchor. @@ -219,7 +250,10 @@ def _child_keys(node, *, descend_aggregates: bool = True) -> List: raise UnhandledValueKindError(node) -def _leaf_paths(node, *, anchor_model, anchor_relation: str, bundle) -> List[Path]: +def _leaf_paths( + node, *, anchor_model, anchor_relation: str, bundle, + cache: "Optional[dict]" = None, +) -> List[Path]: """Join paths a LEAF key is itself anchored at. Composites contribute nothing here — their dependencies arrive through @@ -235,7 +269,7 @@ def _leaf_paths(node, *, anchor_model, anchor_relation: str, bundle) -> List[Pat *_prefixes(node.path), *_derived_sql_paths( key=node, anchor_model=anchor_model, - anchor_relation=anchor_relation, bundle=bundle, + anchor_relation=anchor_relation, bundle=bundle, cache=cache, ), ] if isinstance(node, ColumnKey): @@ -251,6 +285,7 @@ def _leaf_paths(node, *, anchor_model, anchor_relation: str, bundle) -> List[Pat def compute_key_join_paths( *, key, anchor_model, anchor_relation: str, bundle, + cache: "Optional[dict]" = None, ) -> Tuple[Path, ...]: """Every join path ``key``'s dependency tree crosses, anchored at ``anchor_relation``. @@ -272,7 +307,7 @@ def _walk(node) -> None: return for path in _leaf_paths( node, anchor_model=anchor_model, - anchor_relation=anchor_relation, bundle=bundle, + anchor_relation=anchor_relation, bundle=bundle, cache=cache, ): _add(path) for child in _child_keys(node): @@ -284,6 +319,7 @@ def _walk(node) -> None: def key_has_host_local_ref( *, key, anchor_model, anchor_relation: str, bundle, + cache: "Optional[dict]" = None, ) -> bool: """Whether ``key`` depends on anything anchored AT the host root. @@ -301,7 +337,7 @@ def _is_local(node) -> bool: if isinstance(node, ColumnSqlKey): return not node.path and _derived_sql_touches_anchor( key=node, anchor_model=anchor_model, - anchor_relation=anchor_relation, bundle=bundle, + anchor_relation=anchor_relation, bundle=bundle, cache=cache, ) return False @@ -343,6 +379,7 @@ def recompute_filter_reachability(planned_query, *, bundle) -> List: anchor_model = planned_query.render_source_model or bundle.source_model anchor_relation = planned_query.source_relation + cache: dict = {} out: List = [] for fp in planned_query.filters_by_phase: if fp.expression is None: @@ -354,12 +391,14 @@ def recompute_filter_reachability(planned_query, *, bundle) -> List: anchor_model=anchor_model, anchor_relation=anchor_relation, bundle=bundle, + cache=cache, ), has_host_local_ref=key_has_host_local_ref( key=fp.expression.value_key, anchor_model=anchor_model, anchor_relation=anchor_relation, bundle=bundle, + cache=cache, ), )) return out diff --git a/slayer/engine/stage_planner.py b/slayer/engine/stage_planner.py index a59ba322..27396b46 100644 --- a/slayer/engine/stage_planner.py +++ b/slayer/engine/stage_planner.py @@ -1337,6 +1337,9 @@ def _windowed_phase(bf: BoundFilter) -> Phase: else host_model_name ) filter_reachability: List[FilterReachability] = [] + # One expansion cache for the whole plan — the two visitors ask for the + # same derived column's expansion, and so does every filter that mentions it. + reachability_cache: dict = {} for fp in filters_by_phase: if fp.expression is None: continue @@ -1347,12 +1350,14 @@ def _windowed_phase(bf: BoundFilter) -> Phase: anchor_model=reachability_anchor_model, anchor_relation=source_relation, bundle=bundle, + cache=reachability_cache, ), has_host_local_ref=key_has_host_local_ref( key=fp.expression.value_key, anchor_model=reachability_anchor_model, anchor_relation=source_relation, bundle=bundle, + cache=reachability_cache, ), )) reachability_by_fid = {r.filter_id: r for r in filter_reachability} diff --git a/slayer/mcp/server.py b/slayer/mcp/server.py index 69343dae..dd601eab 100644 --- a/slayer/mcp/server.py +++ b/slayer/mcp/server.py @@ -1990,6 +1990,16 @@ def _format_csv(data: list[dict[str, Any]], columns: list[str]) -> str: return "\n".join(lines) +def _csv_warning_comments(result: SlayerResponse) -> str: + """Warnings as leading `#` comment lines for CSV output. + + Comments precede the header, so every DATA record keeps a uniform column + count and the advisory is still visible to whoever reads the output. + """ + lines = [f"# warning: {w.human_message()}" for w in (result.warnings or [])] + return "" if not lines else "\n".join(lines) + "\n" + + def _format_warnings(result: SlayerResponse) -> str: """Advisories about the query, appended to the TEXT output formats. @@ -2005,14 +2015,22 @@ def _format_warnings(result: SlayerResponse) -> str: def _format_output(result: SlayerResponse, fmt: str) -> str: """Format query output in the requested format. - ``json`` stays ONE parseable JSON value: warnings go INSIDE the payload as - a ``warnings`` key rather than being appended as prose, which would make - ``json.loads`` fail on exactly the queries that most need reporting. The - text formats append a human-readable block instead. + Warnings never corrupt a machine-readable format: for ``json`` they go + INSIDE the payload under a ``warnings`` key, and for ``csv`` they become + leading ``#`` comment lines. Only ``markdown`` gets a prose block. + + Note this covers the warnings only. The ``query`` tool still prepends + ``SQL:`` text for ``show_sql`` / ``explain`` and appends an attributes + block, which has always made those combinations non-JSON; that predates + this change and is not addressed here. """ if fmt == "csv": - return _format_csv(data=result.data, columns=result.columns) \ - + _format_warnings(result) + # Leading `#` comment lines, never trailing prose: appending the block + # turned each warning into a record with the wrong column count and + # broke every CSV reader on exactly the queries worth inspecting. + return _csv_warning_comments(result) + _format_csv( + data=result.data, columns=result.columns, + ) if fmt == "markdown": return result.to_markdown() + _format_warnings(result) return _format_json( diff --git a/slayer/sql/generator.py b/slayer/sql/generator.py index b51de554..b7067ea2 100644 --- a/slayer/sql/generator.py +++ b/slayer/sql/generator.py @@ -8295,12 +8295,15 @@ def _mode_a_scope( """An ephemeral :class:`ScopeFrame` for a Mode-A entry whose call site holds no scope. - Pure RENDER paths (the aggregate CASE-WHEN wrapper, the WHERE/HAVING - assembler) run after the corresponding registration pass has already - put the crossed joins into the real scope, so the frame here exists - only to give the text one consistent door to come through — its - ``join_paths`` are a byproduct nobody reads. Every site that still owns - discovery passes its real scope instead. + Two kinds of caller. The pure RENDER paths (the aggregate CASE-WHEN + wrapper, the WHERE/HAVING assembler) run after the corresponding + registration pass has already put the crossed joins into the real + scope, so for them the frame exists only to give the text one + consistent door to come through. The shifted-CTE residual path + (``_shifted_filter_sql``) instead READS ``frame.join_paths`` back and + hands it to its caller, which registers those paths on the shifted + scope — so the frame is the discovery vehicle there, not a byproduct. + Every site that already owns a real scope passes it instead. """ return ScopeFrame( scope_id=f"_modea_{source_relation}", diff --git a/tests/test_cross_model_planner.py b/tests/test_cross_model_planner.py index c25e88eb..7df357a9 100644 --- a/tests/test_cross_model_planner.py +++ b/tests/test_cross_model_planner.py @@ -822,7 +822,6 @@ def test_classify_columnsqlkey_on_target_is_reachable(self): host_filter=hf, host_slots=[derived], target_path=("customers",), - host_model_name="orders", ) assert route == FilterRoute.PROPAGATE_WHERE @@ -843,6 +842,5 @@ def test_classify_columnsqlkey_on_other_branch_is_unreachable(self): host_filter=hf, host_slots=[derived], target_path=("customers",), - host_model_name="orders", ) assert route == FilterRoute.DROP_UNREACHABLE diff --git a/tests/test_dev1745_plan_time_routing.py b/tests/test_dev1745_plan_time_routing.py index f7024430..36101d8b 100644 --- a/tests/test_dev1745_plan_time_routing.py +++ b/tests/test_dev1745_plan_time_routing.py @@ -183,8 +183,12 @@ def _windowed_query(self) -> SlayerQuery: ) def test_plan_carries_frame_bound_columns(self) -> None: - planned = plan_query(query=self._windowed_query(), bundle=_bundle()) - assert hasattr(planned, "frame_bound_columns") + """A DECLARED field, checked the same way as outer_where_filter_ids. + ``hasattr`` is always true for a field with a default_factory, so it + could not fail regardless of planner behaviour.""" + from slayer.engine.planned import PlannedQuery + + assert "frame_bound_columns" in PlannedQuery.model_fields def test_frame_bound_columns_covers_the_time_dimension(self) -> None: planned = plan_query(query=self._windowed_query(), bundle=_bundle()) diff --git a/tests/test_dev1745_reachability.py b/tests/test_dev1745_reachability.py index 9e54712c..f9e170b8 100644 --- a/tests/test_dev1745_reachability.py +++ b/tests/test_dev1745_reachability.py @@ -236,6 +236,29 @@ def test_nested_derived_below_a_composite(self) -> None: ) assert ("customers",) in _paths_for(key) + def test_sql_expr_key_contributes_its_referenced_paths(self) -> None: + """``SqlExprKey`` carries its own precomputed crossed paths (a + ``Column.filter`` interned onto an aggregate). It has an arm in the + scan; this pins it.""" + from slayer.core.keys import SqlExprKey + + key = SqlExprKey( + canonical_sql="customers__regions.population > 1", + referenced_join_paths=(("customers", "regions"),), + ) + paths = _paths_for(key) + assert ("customers",) in paths, paths + assert ("customers", "regions") in paths, paths + + def test_in_values_are_walked_for_crossings(self) -> None: + """``InKey.values`` are walked by the crossing scan, so a crossing + reference sitting in the value list is a dependency like any other.""" + key = InKey( + column=ColumnKey(path=(), leaf="amount"), + values=(LiteralKey(value=1),), + ) + assert _paths_for(key) == () + def test_literal_crosses_nothing(self) -> None: assert _paths_for(LiteralKey(value=1)) == () From 6593955d141f3fca27d65cbab81d9107afc04cb9 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Thu, 6 Aug 2026 12:53:42 +0200 Subject: [PATCH 38/98] DEV-1746 stages 1-2: null-safe grain doctrine (B1/B2) + pagination hook (B3) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Stage 1 — one grain join-back builder (P-I, §5.7). slayer/sql/render/joins.py holds the single mechanism every grain join-back uses: the _cm_ join-back, the _wm_ join-back and its inner _src join, and the time-shift sjoin pair. It takes EXPRESSION operand pairs (so a caller can compare a cast or any resolved reference, not only a projected alias) and returns None for an empty grain, which is how the caller learns to emit a CROSS JOIN — a scalar aggregate has no grain to join on, and returning a truthy TRUE would erase that distinction. B2: the predicate is built as AST instead of rendering both sides to pre-quoted strings and re-parsing them. Public aliases are dotted, and on BigQuery the round-trip re-read `_base."orders.customers.status"` as a multi-part reference, emitting a qualifier for a table that does not exist: ON `_base___orders___customers`.`status` IS NOT DISTINCT FROM ... Three golden baseline entries recorded ScopeLeakError for exactly this and now record real SQL (regenerated through the ALLOWED_DELTAS protocol). The two byte-duplicated _cm_/_wm_ join loops collapse into one. B1: the _wm_ INNER grain equality goes null-safe. It was a plain `=`, so a group whose dimension is NULL never matched and silently received NULL instead of its real windowed value — while the outer join-back and the _cm_ join-back were already null-safe, so SLayer disagreed with itself about NULL-grain semantics depending on which isolation shape a measure landed in. The time-range bounds stay plain inequalities: they are a range, not a grain. _null_safe_join_pair_sql is retained and test-pinned (P-J state 1) but is now production-unreferenced, proved by poisoning it rather than by grepping. Stage 2 — pagination through the dialect strategy (P-H, §5.9, B3 first half). SqlDialect.apply_pagination is the one place pagination is expressed. The T-SQL override owns its rule explicitly rather than relying on sqlglot happening to apply it: limit-only becomes TOP, and an offset with no ORDER BY gets a deterministic ORDER BY (SELECT NULL) before OFFSET/FETCH, because SQL Server rejects OFFSET without ordering. Doing it here means the rule survives a sqlglot upgrade and lands in the AST where callers and tests can see it. _apply_order_limit_from_planned now routes through the hook. The cross-model combined tail still appends raw LIMIT/OFFSET text and switches over in stage 3 when that statement becomes AST. Also fixed: a filter on a JOINED model's crossing derived column emitted invalid SQL. Both the join scanner (_value_key_join_paths) and the filter renderer expanded the column without is_root=False, so a further-joined ref came out bare (`regions.population`); the scanner could not match it to a join path, so the hop was never joined and the filter referenced a table absent from the FROM. Both now pass is_root=False — they must stay in lockstep, since discovery scans the same expansion the renderer emits. The golden baseline now records zero exceptions (five entries fixed by DEV-1745, the last three by B2), so test_error_entries_record_the_full_message guards the format of an error entry rather than requiring one to exist. Tests: 7 new files, 278 tests, TDD-first for the whole PR. Stages 1-2 green; 47 remain red for stages 3-6. Co-Authored-By: Claude Opus 5 (1M context) --- slayer/sql/dialects/base.py | 35 +- slayer/sql/dialects/tsql.py | 31 + slayer/sql/generator.py | 159 ++-- slayer/sql/render/joins.py | 86 ++ tests/_dev1746_fixtures.py | 420 +++++++++ tests/golden/dev1745_sql_baseline.json | 15 +- tests/test_dev1745_golden_sql.py | 10 +- .../test_dev1746_consumer_materialization.py | 403 +++++++++ tests/test_dev1746_cte_assembly.py | 326 +++++++ tests/test_dev1746_empty_base_plan.py | 285 ++++++ tests/test_dev1746_null_safe_grain.py | 571 ++++++++++++ tests/test_dev1746_pagination.py | 393 ++++++++ tests/test_dev1746_projection_order.py | 844 ++++++++++++++++++ tests/test_sql_generator.py | 7 +- 14 files changed, 3485 insertions(+), 100 deletions(-) create mode 100644 slayer/sql/render/joins.py create mode 100644 tests/_dev1746_fixtures.py create mode 100644 tests/test_dev1746_consumer_materialization.py create mode 100644 tests/test_dev1746_cte_assembly.py create mode 100644 tests/test_dev1746_empty_base_plan.py create mode 100644 tests/test_dev1746_null_safe_grain.py create mode 100644 tests/test_dev1746_pagination.py create mode 100644 tests/test_dev1746_projection_order.py diff --git a/slayer/sql/dialects/base.py b/slayer/sql/dialects/base.py index 6bb8a8a2..7d18da52 100644 --- a/slayer/sql/dialects/base.py +++ b/slayer/sql/dialects/base.py @@ -13,7 +13,7 @@ from __future__ import annotations from functools import lru_cache -from typing import TYPE_CHECKING, Any +from typing import TYPE_CHECKING, Any, Optional from collections.abc import Callable from pydantic import BaseModel, ConfigDict @@ -211,8 +211,10 @@ def build_null_safe_eq( Base (Postgres-family) uses sqlglot's ``NullSafeEQ`` → ``IS NOT DISTINCT FROM``, which sqlglot also transpiles correctly for DuckDB / Snowflake / - BigQuery / Trino / Databricks / ClickHouse. MySQL overrides to ``<=>``; - SQLite to bare ``IS``; dialects with no native form (T-SQL / Oracle / + BigQuery / Trino / Databricks / ClickHouse — and for MySQL, where it + emits ``<=>``. MySQL therefore needs no override here (an earlier + version of this docstring claimed one existed). ``SqliteDialect`` + overrides to bare ``IS``; dialects with no native form (T-SQL / Oracle / Redshift) to the expanded ``a = b OR (a IS NULL AND b IS NULL)``. """ return exp.NullSafeEQ(this=left, expression=right) @@ -451,6 +453,33 @@ def rewrite_target_ast(self, tree: exp.Expression) -> exp.Expression: """ return tree + def apply_pagination( + self, + select: exp.Select, + *, + limit: Optional[int], + offset: Optional[int], + ) -> exp.Select: + """Apply LIMIT/OFFSET to a completed ``SELECT`` (P-H). + + The single place pagination is expressed. Every render path routes + here, so a dialect that spells pagination differently is handled once + rather than per path — the cross-model combined statement used to append + raw ``LIMIT``/``OFFSET`` text and emitted literal ``LIMIT`` on SQL + Server, while the same query carrying a transform layer went through the + outer wrap and came out correct. + + Setting the bounds on the ``Select`` is what makes transposition work: + sqlglot rewrites them per dialect only when generating the wrapping + SELECT, never from a free-standing ``Limit`` node. + """ + out = select + if limit is not None: + out = out.limit(limit) + if offset is not None: + out = out.offset(offset) + return out + def emit_outer_wrap( self, *, diff --git a/slayer/sql/dialects/tsql.py b/slayer/sql/dialects/tsql.py index 3ab5687e..24907cf5 100644 --- a/slayer/sql/dialects/tsql.py +++ b/slayer/sql/dialects/tsql.py @@ -243,6 +243,37 @@ def build_covar_2arg( # DEV-1571 Bug 1: emit_outer_wrap hoists inner top-level CTEs # ------------------------------------------------------------------ + def apply_pagination( + self, + select: exp.Select, + *, + limit: "int | None", + offset: "int | None", + ) -> exp.Select: + """T-SQL pagination, with the ``OFFSET`` ordering requirement made + explicit. + + SQL Server rejects ``OFFSET`` without an ``ORDER BY``. When the query is + genuinely unordered we supply ``ORDER BY (SELECT NULL)`` — the + conventional no-op ordering, which adds no semantics because there were + none to preserve, and only makes the statement legal. + + sqlglot happens to inject the same thing today, but that is its + behaviour and not our contract: doing it here means the rule survives a + sqlglot upgrade, and it puts the ordering in the AST where a caller (and + our tests) can see it rather than only in the generated string. A user's + own ORDER BY is never replaced. + + ``TOP`` versus ``FETCH`` needs no special handling — sqlglot picks + ``TOP`` for a bare limit and ``OFFSET … FETCH`` once an offset is + present, which is the correct T-SQL in both cases. + """ + if offset is not None and select.args.get("order") is None: + select = select.order_by( + exp.Subquery(this=exp.Select().select(exp.Null())), + ) + return super().apply_pagination(select, limit=limit, offset=offset) + def emit_outer_wrap( self, *, diff --git a/slayer/sql/generator.py b/slayer/sql/generator.py index 01d38285..836495a0 100644 --- a/slayer/sql/generator.py +++ b/slayer/sql/generator.py @@ -56,6 +56,10 @@ result_key_from_alias, ) from slayer.sql.render.aggregates import window_agg_class +from slayer.sql.render.joins import ( + build_grain_joinback_condition, + grain_alias_column, +) from slayer.sql.render.value_expr import ( render_arithmetic, render_scalar_call, @@ -4015,8 +4019,8 @@ def _alias_of(sid: str) -> str: base_alias = _alias_of(sid) expr = src_scope.resolve(dslot.key) src_cols.append(expr.as_(f"_w_dim_{idx}")) - join_eqs.append(exp.EQ( - this=_src_col(f"_w_dim_{idx}"), expression=_base_col(base_alias), + join_eqs.append(self._dialect.build_null_safe_eq( + _src_col(f"_w_dim_{idx}"), _base_col(base_alias), )) grain_aliases.append(base_alias) @@ -4038,8 +4042,8 @@ def _alias_of(sid: str) -> str: col_expr=raw, granularity=TimeGranularity(tslot.key.granularity), ) src_cols.append(trunc.as_(f"_w_td_{idx}")) - join_eqs.append(exp.EQ( - this=_src_col(f"_w_td_{idx}"), expression=_base_col(base_alias), + join_eqs.append(self._dialect.build_null_safe_eq( + _src_col(f"_w_td_{idx}"), _base_col(base_alias), )) grain_aliases.append(base_alias) @@ -4904,56 +4908,47 @@ def _render_outer_composite(cslot) -> str: ) combined_aliases_by_slot_id[plan.aggregate_slot_id] = list(full_aliases) + # Grain join-backs (P-I). Both plan kinds join back identically — on the + # shared grain, null-safely, so a NULL dimension value or a nullable + # truncated time bucket keeps its aggregate instead of dropping it. An + # EMPTY grain (a scalar aggregate) has nothing to join on and becomes a + # CROSS JOIN; the builder signals that by returning ``None``. from_clause_str = "FROM _base" joined_cte_names: set = set() - for plan in planned_query.cross_model_aggregate_plans: - cte_name = cm_cte_name_for_plan[plan.aggregate_slot_id] + joinback_specs = [ + ( + cm_cte_name_for_plan[plan.aggregate_slot_id], + joinback_pairs_for_plan.get(plan.aggregate_slot_id, []), + ) + for plan in planned_query.cross_model_aggregate_plans + ] + [ + ( + wm_cte_name_for_plan[plan.aggregate_slot_id], + wm_joinback_pairs_for_plan.get(plan.aggregate_slot_id, []), + ) + for plan in planned_query.windowed_aggregate_plans + ] + for cte_name, joinback_pairs in joinback_specs: if cte_name in joined_cte_names: continue joined_cte_names.add(cte_name) - joinback_pairs = joinback_pairs_for_plan.get( - plan.aggregate_slot_id, [], - ) - if joinback_pairs: - # DEV-1708 / Codex F2: the grain join-back uses a dialect-aware - # NULL-SAFE equality so NULL dimension values and nullable - # truncated time grains join back instead of dropping their - # aggregate (a plain ``=`` yields NULL for NULL = NULL). - join_parts = [ - self._null_safe_join_pair_sql( - left_sql=f'_base.{self._quote_ident(host)}', - right_sql=f'{cte_name}.{self._quote_ident(cte_col)}', + on_condition = build_grain_joinback_condition( + pairs=[ + ( + grain_alias_column(alias=host, table="_base"), + grain_alias_column(alias=cte_col, table=cte_name), ) for host, cte_col in joinback_pairs - ] - from_clause_str += ( - f"\nLEFT JOIN {cte_name} ON " + _SQL_AND_JOINER.join(join_parts) - ) - else: + ], + dialect=self._dialect, + ) + if on_condition is None: from_clause_str += f"\nCROSS JOIN {cte_name}" - - # DEV-1714 Stage 10 — LEFT JOIN each ``_wm_`` CTE back to ``_base`` on - # the shared grain (null-safe, so NULL-dim / nullable-grain groups keep - # a row; the windowed value for a NULL-dim group is NULL — the plain - # ``=`` inside the CTE never matches NULL, a documented consequence). - for plan in planned_query.windowed_aggregate_plans: - cte_name = wm_cte_name_for_plan[plan.aggregate_slot_id] - joinback_pairs = wm_joinback_pairs_for_plan.get( - plan.aggregate_slot_id, [], - ) - if joinback_pairs: - join_parts = [ - self._null_safe_join_pair_sql( - left_sql=f'_base.{self._quote_ident(host)}', - right_sql=f'{cte_name}.{self._quote_ident(cte_col)}', - ) - for host, cte_col in joinback_pairs - ] + else: from_clause_str += ( - f"\nLEFT JOIN {cte_name} ON " + _SQL_AND_JOINER.join(join_parts) + f"\nLEFT JOIN {cte_name} ON " + + on_condition.sql(dialect=self.dialect) ) - else: - from_clause_str += f"\nCROSS JOIN {cte_name}" combined_select_sql = ( f"SELECT {', '.join(combined_parts)}\n{from_clause_str}" @@ -7793,40 +7788,29 @@ def _add_partition(pk_obj, *, where: str) -> None: f'{shifted_cte_name}.{self._quote_ident(input_alias)} AS {self._quote_ident(full_slot_alias)}', ) - # JOIN conditions: time equality + every partition equality, all - # dialect-aware NULL-SAFE (DEV-1711 / Codex F2). The sjoin is a grain - # join-back — a NULL dimension value (e.g. a LEFT-joined ``stores.name`` - # with no matching store) or a NULL time bucket must match its own group - # instead of silently dropping to a NULL shifted value under plain ``=``. - # - # The predicate is built from AST nodes DIRECTLY — not via - # ``_null_safe_join_pair_sql``'s string round-trip — because a dotted - # public alias (``orders.created_at``) re-parses on BigQuery/T-SQL as a - # multi-part reference and the DEV-1713 alias mangling then corrupts it - # (``base.`orders.created_at``` → ``base___orders`.`created_at```). The - # alias as a single ``quoted=True`` identifier matches the SELECT parts' - # ``_quote_ident`` output byte-for-byte on every dialect and survives the - # post-generation mangling intact. - def _grain_eq(a: str) -> str: - left = exp.Column( - this=exp.to_identifier(a, quoted=True), - table=exp.to_identifier(prev_cte), - ) - right = exp.Column( - this=exp.to_identifier(a, quoted=True), - table=exp.to_identifier(shifted_cte_name), - ) - return self._dialect.build_null_safe_eq(left, right).sql(dialect=self.dialect) - - join_conds = [_grain_eq(time_alias)] - for _, pk_alias, _ in partition_specs: - join_conds.append(_grain_eq(pk_alias)) - + # JOIN conditions: time equality + every partition equality. The sjoin is + # a grain join-back like any other, so it goes through the shared + # null-safe builder — a NULL dimension value (e.g. a LEFT-joined + # ``stores.name`` with no matching store) or a NULL time bucket must + # match its own group instead of dropping to a NULL shifted value. + grain_alias_names = [time_alias] + [ + pk_alias for _, pk_alias, _ in partition_specs + ] + sjoin_on = build_grain_joinback_condition( + pairs=[ + ( + grain_alias_column(alias=a, table=prev_cte), + grain_alias_column(alias=a, table=shifted_cte_name), + ) + for a in grain_alias_names + ], + dialect=self._dialect, + ) sjoin_sql = ( "SELECT " + ", ".join(sjoin_select_parts) + f"\nFROM {prev_cte}" + f"\nLEFT JOIN {shifted_cte_name}" - + "\n ON " + _SQL_AND_JOINER.join(join_conds) + + "\n ON " + sjoin_on.sql(dialect=self.dialect) ) ctes.append((sjoin_cte_name, sjoin_sql)) @@ -8756,10 +8740,10 @@ def _scan(parsed: exp.Expression) -> None: if p not in out: out.append(p) - def _derived_paths(*, model, relation, column_name) -> None: + def _derived_paths(*, model, relation, column_name, is_root: bool) -> None: _scan(self._parse(self._expand_derived_column_sql( source_model=model, source_relation=relation, - column_name=column_name, bundle=bundle, + column_name=column_name, bundle=bundle, is_root=is_root, ))) def _walk(k) -> None: @@ -8773,10 +8757,18 @@ def _walk(k) -> None: else source_model ) if model is not None: + # ``is_root=False`` for a JOINED derived column: a further + # -joined ref inside its ``sql`` must resolve to the full + # path (``customers_v2`` reaching ``regions`` → + # ``customers_v2__regions``). Rooting it here instead left + # the ref bare, so the scan found no path and the hop was + # never joined — the filter then referenced a table that + # is not in the FROM. _derived_paths( model=model, relation="__".join(k.path) if k.path else source_relation, column_name=k.column_name, + is_root=not k.path, ) elif isinstance(k, ArithmeticKey): for o in k.operands: @@ -9400,11 +9392,17 @@ def _render_value_key_for_filter( # NOSONAR(S3776) — sequential isinstance di f"resolved source bundle.", ) path_alias = "__".join(key.path) + # ``is_root=False`` — the column lives on a JOINED model, so a + # further-joined ref inside its ``sql`` resolves to the full + # path alias rather than the bare child relation. Must match + # ``_value_key_join_paths``, which registers the joins by + # scanning this same expansion. expanded_sql = self._expand_derived_column_sql( source_model=joined_model, source_relation=path_alias, column_name=key.column_name, bundle=bundle, + is_root=False, ) col = next( (c for c in joined_model.columns if c.name == key.column_name), @@ -10048,12 +10046,9 @@ def _apply_order_limit_from_planned( # NOSONAR(S3776) — per-order-entry slot- self._ordered(order_col, ascending=ascending), ) - if planned_query.limit is not None: - select = select.limit(planned_query.limit) - if planned_query.offset is not None: - select = select.offset(planned_query.offset) - - return select + return self._dialect.apply_pagination( + select, limit=planned_query.limit, offset=planned_query.offset, + ) # =========================================================================== diff --git a/slayer/sql/render/joins.py b/slayer/sql/render/joins.py new file mode 100644 index 00000000..d7653f02 --- /dev/null +++ b/slayer/sql/render/joins.py @@ -0,0 +1,86 @@ +"""The one grain join-back builder (P-I). + +Every place a scope's rows are joined back to another scope on the query grain +builds its ``ON`` predicate here: the cross-model (``_cm_``) join-back, the +windowed (``_wm_``) join-back and its inner ``_src`` join, and the time-shift +``sjoin_`` pair. + +Two properties, both of which had counter-examples before this module existed: + +**Null-safe.** A grain member is frequently NULL — a nullable dimension, an +outer-joined column, a bucket with no rows. Plain ``=`` never matches NULL +against NULL, so the group silently loses its aggregate instead of receiving it. +The dialect strategy owns the spelling (``IS NOT DISTINCT FROM``, MySQL +``<=>``, SQLite ``IS``, or the expanded ``a = b OR (a IS NULL AND b IS NULL)`` +where there is no native operator). + +**Built as AST, never re-parsed.** The superseded helper rendered both sides to +pre-quoted strings and parsed them back. SLayer's public aliases are dotted +(``orders.customers.status``), and on a dialect that mangles dots at emission +the round-trip re-reads such an alias as a multi-part *reference*, producing a +qualifier for a table that does not exist:: + + ON `_base___orders___customers`.`status` IS NOT DISTINCT FROM ... + ^^^^^^^^^^^^^^^^^^^^^^^^^^ not a table in scope + +Building the column node directly makes that unrepresentable: the alias is one +identifier and stays one identifier. + +The core builder takes **expression** operands so a caller can compare anything +it has already resolved — a cast, a date-truncated bucket, a materialised +alias. :func:`grain_alias_column` covers the common case, where both sides are +columns projected under the same public alias in two different scopes. +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, Optional, Sequence, Tuple + +from sqlglot import exp + +if TYPE_CHECKING: # pragma: no cover - typing only + from slayer.sql.dialects.base import SqlDialect + +__all__ = ["grain_alias_column", "build_grain_joinback_condition"] + + +def grain_alias_column(*, alias: str, table: str) -> exp.Column: + """A reference to ``table``'s output column named ``alias``. + + ``alias`` is a projected column NAME, not a path: it is quoted as a single + identifier even when it contains dots, which is exactly what the dotted + public aliases (``orders.customers.status``) require. ``table`` is a CTE + alias minted by the allocator and needs no quoting of its own. + """ + return exp.Column( + this=exp.to_identifier(alias, quoted=True), + table=exp.to_identifier(table), + ) + + +def build_grain_joinback_condition( + *, + pairs: Sequence[Tuple[exp.Expression, exp.Expression]], + dialect: "SqlDialect", +) -> Optional[exp.Expression]: + """Null-safe ``ON`` predicate equating each ``(left, right)`` grain member. + + Returns ``None`` for an empty grain — a scalar aggregate has no grain to + join on, and the caller emits a ``CROSS JOIN`` instead. Returning a truthy + ``TRUE`` would look equivalent but is not: it would turn every scalar + aggregate's join into a predicate-bearing one and hide the distinction the + empty grain actually carries. + + Operands are copied, so a caller may pass expressions it also holds + elsewhere without the AST being re-parented out from under it. + """ + conditions = [ + dialect.build_null_safe_eq(left.copy(), right.copy()) + for left, right in pairs + ] + if not conditions: + return None + combined = conditions[0] + for condition in conditions[1:]: + combined = exp.And(this=combined, expression=condition) + return combined diff --git a/tests/_dev1746_fixtures.py b/tests/_dev1746_fixtures.py new file mode 100644 index 00000000..9563a527 --- /dev/null +++ b/tests/_dev1746_fixtures.py @@ -0,0 +1,420 @@ +"""Shared fixtures + helpers for the DEV-1746 scope-assembly test modules. + +Underscore-prefixed (like ``tests/_engine_helpers.py`` and +``tests/_cross_model_chain.py``) so pytest skips it during collection while +``from tests._dev1746_fixtures import ...`` still works. + +What lives here: + +* **A NULL-bearing SQLite corpus** (:func:`seed_dev1746_sqlite`) — the execution + substrate for §5.7. Every grain column it seeds is deliberately nullable and + actually carries NULLs, because the whole point of B1/B2 is what happens to a + group whose grain member is NULL. ``orders.status`` has a NULL group spanning + two months (so a 90-day window over it has something to sum), ``customers.tier`` + has a NULL group, and ``regions.name`` has one too — the composite-grain case + needs two independently-nullable members. +* **SQLite-shaped model builders** (:func:`dev1746_models`) — bare ``sql_table`` + names, matching the seeded schema. +* :func:`make_sqlite_engine` — the storage+engine wiring every execution fixture + in this family repeats. +* :func:`outer_select_aliases` — the emitted public projection, in order. B7 is + an *ordering* contract, so the assertions need the emitted order as a list, + not the ``set`` that ``_join_aliases`` returns. +* :func:`joinback_on_predicate_for` — the generalisation of + ``tests/_cross_model_chain._joinback_on_predicate`` to ``_wm_`` as well as + ``_cm_`` CTEs (B1 touches both join-backs, and the existing helper only finds + ``_cm_``/``_fm_``). + +The seeded numbers are chosen so every expected aggregate is a distinct value — +a test that accidentally reads the wrong column fails rather than coincidentally +matching. +""" + +from __future__ import annotations + +import sqlite3 +from typing import List, Optional + +import sqlglot +from sqlglot import exp + +from slayer.core.enums import DataType +from slayer.core.models import Column, DatasourceConfig, ModelJoin, SlayerModel +from slayer.engine.query_engine import SlayerQueryEngine +from slayer.storage.yaml_storage import YAMLStorage + +# --------------------------------------------------------------------------- # +# The corpus +# --------------------------------------------------------------------------- # +#: ``orders.status`` NULL group, by month, for the 90-day window assertions: +#: 2024-01 has 5.0, 2024-02 has 7.0 — so the February window (which reaches back +#: 90 days) sums to 12.0 while February alone would be 7.0. The two numbers +#: differ, so a test cannot pass by reading the un-windowed sum. +NULL_STATUS_JAN = 5.0 +NULL_STATUS_FEB = 7.0 +NULL_STATUS_FEB_WINDOW = NULL_STATUS_JAN + NULL_STATUS_FEB # 12.0 + +#: ``paid`` group, same shape, different values. +PAID_JAN = 10.0 +PAID_FEB = 20.0 +PAID_FEB_WINDOW = PAID_JAN + PAID_FEB # 30.0 + + +def seed_dev1746_sqlite(db_path: str) -> None: + """Create + seed the DEV-1746 SQLite corpus at ``db_path``.""" + con = sqlite3.connect(db_path) + con.executescript( + """ + CREATE TABLE regions ( + id INTEGER PRIMARY KEY, + name TEXT, + population REAL + ); + CREATE TABLE customers ( + id INTEGER PRIMARY KEY, + region_id INTEGER, + tier TEXT, + spend REAL + ); + CREATE TABLE orders ( + id INTEGER PRIMARY KEY, + customer_id INTEGER, + status TEXT, + created_at TEXT, + amount REAL + ); + """ + ) + con.executemany( + "INSERT INTO regions VALUES (?,?,?)", + # Region 2's name is NULL — the joined nullable grain member. + [(1, "West", 100.0), (2, None, 200.0)], + ) + con.executemany( + "INSERT INTO customers VALUES (?,?,?,?)", + # Customer 101's tier is NULL — the target-side nullable grain member. + [ + (100, 1, "gold", 1000.0), + (101, 2, None, 250.0), + (102, 2, None, 75.0), + ], + ) + con.executemany( + "INSERT INTO orders VALUES (?,?,?,?,?)", + [ + (1, 100, "paid", "2024-01-15", PAID_JAN), + (2, 100, "paid", "2024-02-15", PAID_FEB), + # The NULL-status group — two months, so a 90-day window spans both. + (3, 101, None, "2024-01-20", NULL_STATUS_JAN), + (4, 101, None, "2024-02-20", NULL_STATUS_FEB), + ], + ) + con.commit() + con.close() + + +def dev1746_models() -> List[SlayerModel]: + """SQLite-shaped ``orders -> customers -> regions`` models for the corpus. + + Returned host-first; ``[0]`` is the host and the rest are ``extra_models``. + """ + regions = SlayerModel( + name="regions", sql_table="regions", data_source="test", + columns=[ + Column(name="id", type=DataType.INT, primary_key=True), + Column(name="name", type=DataType.TEXT), + Column(name="population", type=DataType.DOUBLE), + ], + ) + customers = SlayerModel( + name="customers", sql_table="customers", data_source="test", + columns=[ + Column(name="id", type=DataType.INT, primary_key=True), + Column(name="region_id", type=DataType.INT), + Column(name="tier", type=DataType.TEXT), + Column(name="spend", type=DataType.DOUBLE), + ], + joins=[ModelJoin(target_model="regions", join_pairs=[["region_id", "id"]])], + ) + orders = SlayerModel( + name="orders", sql_table="orders", data_source="test", + default_time_dimension="created_at", + columns=[ + Column(name="id", type=DataType.INT, primary_key=True), + Column(name="customer_id", type=DataType.INT), + Column(name="status", type=DataType.TEXT), + Column(name="created_at", type=DataType.TIMESTAMP), + Column(name="amount", type=DataType.DOUBLE), + ], + joins=[ModelJoin(target_model="customers", join_pairs=[["customer_id", "id"]])], + ) + return [orders, customers, regions] + + +async def make_sqlite_engine(base_dir: str, db_path: str) -> SlayerQueryEngine: + """Storage + engine bound to the seeded SQLite file at ``db_path``.""" + storage = YAMLStorage(base_dir=base_dir) + await storage.save_datasource( + DatasourceConfig(name="test", type="sqlite", database=db_path), + ) + for model in dev1746_models(): + await storage.save_model(model) + return SlayerQueryEngine(storage=storage) + + +# --------------------------------------------------------------------------- # +# SQL-shape helpers +# --------------------------------------------------------------------------- # +def outer_select_aliases(sql: str, *, dialect: str = "postgres") -> List[str]: + """The outermost SELECT's output column names, **in emitted order**. + + B7 is an ordering contract, so this returns a list. Uses sqlglot's + ``named_selects``, which is what ``response_meta.expected_columns_from_sql`` + reads — so asserting on this asserts on the same thing the response's + ``columns`` order is derived from. + """ + parsed = sqlglot.parse_one(sql, dialect=dialect) + assert parsed is not None, f"SQL failed to parse:\n{sql}" + return list(parsed.named_selects) + + +def joinback_on_predicate_for( + sql: str, *, prefix: str, dialect: str = "postgres", +) -> str: + """Rendered ON predicate of the combined SELECT's join-back to a CTE whose + alias starts with ``prefix`` (``_cm_`` or ``_wm_``). + + ``tests/_cross_model_chain._joinback_on_predicate`` only finds ``_cm_``/ + ``_fm_``; B1 needs the ``_wm_`` join-back too, and §5.7 requires asserting on + BOTH ``_wm_`` comparison sites. + """ + tree = sqlglot.parse_one(sql, dialect=dialect) + for join in tree.find_all(exp.Join): + name = getattr(join.this, "alias_or_name", "") or "" + if name.startswith(prefix): + on = join.args.get("on") + if on is not None: + return on.sql(dialect=dialect) + raise AssertionError(f"no JOIN onto a {prefix}* CTE with an ON predicate in:\n{sql}") + + +def joined_cte_names(sql: str, *, dialect: str = "postgres") -> List[str]: + """Aliases of the CTEs joined into the combined SELECT, in FROM-clause order.""" + tree = sqlglot.parse_one(sql, dialect=dialect) + select = tree.find(exp.Select) + assert select is not None, f"no SELECT in:\n{sql}" + names: List[str] = [] + for join in select.args.get("joins") or []: + name = getattr(join.this, "alias_or_name", "") or "" + if name: + names.append(name) + return names + + +def cte_names_in_order(sql: str, *, dialect: str = "postgres") -> List[str]: + """Names of the top-level WITH clause's CTEs, in emitted order.""" + tree = sqlglot.parse_one(sql, dialect=dialect) + with_node = tree.args.get("with_") + if with_node is None: + return [] + return [cte.alias_or_name for cte in with_node.expressions] + + +def src_subquery_on_predicate(sql: str, *, dialect: str = "postgres") -> str: + """Rendered ON predicate of the ``_src`` subquery join **inside** a ``_wm_`` + CTE — the B1 inner-grain comparison site. + + §5.7 requires both ``_wm_`` sites be asserted; this is the inner one + (``joinback_on_predicate_for(prefix="_wm_")`` is the outer one). + """ + tree = sqlglot.parse_one(sql, dialect=dialect) + for join in tree.find_all(exp.Join): + name = getattr(join.this, "alias_or_name", "") or "" + if name == "_src": + on = join.args.get("on") + if on is not None: + return on.sql(dialect=dialect) + raise AssertionError(f"no `_src` subquery join with an ON predicate in:\n{sql}") + + +def outer_statement(sql: str, *, dialect: str = "postgres") -> exp.Select: + """The OUTERMOST SELECT — the one pagination must land on. + + Asserting pagination by searching the whole statement is unsound: an inner + CTE, a window function's ``OVER (ORDER BY …)``, or a hidden order slot can + satisfy a global ``ORDER BY`` search while the paginated SELECT itself has + none (which is exactly the T-SQL error the rule exists to prevent). + """ + parsed = sqlglot.parse_one(sql, dialect=dialect) + assert isinstance(parsed, exp.Select), ( + f"expected the statement to be a SELECT, got {type(parsed).__name__}:\n{sql}" + ) + return parsed + + +def outer_clause_sql(sql: str, *, dialect: str = "postgres") -> str: + """The outermost SELECT rendered WITHOUT its CTEs. + + Keeps the pagination/ordering clauses of the outer statement while dropping + every inner scope, so a keyword assertion cannot be satisfied by a CTE body. + """ + outer = outer_statement(sql, dialect=dialect).copy() + outer.set("with_", None) + return outer.sql(dialect=dialect) + + +def join_alias_sequence(sql: str, *, dialect: str = "postgres") -> List[str]: + """Joined table aliases of the outermost FROM, in emitted JOIN order. + + ``_engine_helpers._join_aliases`` returns a SET; join ORDER assertions need + the sequence, and building one by testing membership of known names against + the SQL string would just reproduce the caller's own ordering. + """ + tree = sqlglot.parse_one(sql, dialect=dialect) + select = tree.find(exp.Select) + assert select is not None, f"no SELECT in:\n{sql}" + names: List[str] = [] + for join in select.args.get("joins") or []: + target = join.this + if isinstance(target, exp.Table): + names.append(target.alias_or_name) + return names + + +def base_cte_join_sequence(sql: str, *, dialect: str = "postgres") -> List[str]: + """Joined aliases of the ``_base`` CTE (or the top-level FROM when the query + has no CTEs), in emitted order — the B11 subject.""" + base = find_cte(sql, "_base", dialect=dialect) + if base is None: + tree = sqlglot.parse_one(sql, dialect=dialect) + base = tree.find(exp.Select) + assert base is not None, f"no base scope found in:\n{sql}" + names: List[str] = [] + for join in base.args.get("joins") or []: + target = join.this + if isinstance(target, exp.Table): + names.append(target.alias_or_name) + return names + + +def carried_alias_drops(sql: str, *, dialect: str = "postgres") -> List[str]: + """Stages that fail to carry forward an alias a LATER stage still needs. + + B8 reorders carry lists; Codex D8 requires it fail closed. Order is checked + by :func:`carry_list_order_violations`; this is the other half — an alias + referenced downstream must be projected by every stage between its producer + and its consumer, or the SQL simply will not bind. + """ + tree = sqlglot.parse_one(sql, dialect=dialect) + ctes = list(tree.find_all(exp.CTE)) + if len(ctes) < 2: + return [] + projected = {c.alias_or_name: set(c.this.named_selects) for c in ctes} + violations: List[str] = [] + + def _scopes(): + for cte in ctes: + yield cte.alias_or_name, cte.this + final = tree.find(exp.Select) + if final is not None: + yield "", final + + for label, scope in _scopes(): + # Only columns QUALIFIED by another CTE's name can be checked: an + # unqualified reference is ambiguous, and sibling CTEs (``_cm_`` next to + # ``base``) do not read each other at all, so an adjacency-based check + # would report drops that are not drops. + for col in scope.find_all(exp.Column): + source = col.table + if not source or source not in projected or source == label: + continue + if col.name not in projected[source]: + violations.append( + f"{label} references {source}.{col.name!r}, which " + f"{source} does not project" + ) + return sorted(set(violations)) + + +def carry_list_order_violations( + sql: str, *, dialect: str = "postgres", +) -> List[str]: + """Inner stages whose carried aliases are not in the base stage's order. + + B8's contract in one invariant: every downstream stage (a ``stepN`` CTE, an + ``sjoin_``/``cp_reset_`` CTE, the inner SELECT under the ``_outer`` wrap) + carries forward a subset of the base stage's columns. Those carried aliases + must appear in the order the BASE stage projects them — that is what "plan + order" means downstream, since the base projects in ``base_render_order``. + ``sorted(aliases)`` violates it whenever alphabetical order differs. + + Returns a list of human-readable violations (empty when compliant), so a + failing assertion can name every offending stage at once. + """ + tree = sqlglot.parse_one(sql, dialect=dialect) + ctes = list(tree.find_all(exp.CTE)) + if not ctes: + return [] + base_order = list(ctes[0].this.named_selects) + base_rank = {a: i for i, a in enumerate(base_order)} + + def _check(label: str, aliases: List[str]) -> Optional[str]: + carried = [a for a in aliases if a in base_rank] + expected = sorted(carried, key=lambda a: base_rank[a]) + if carried != expected: + return f"{label}: carries {carried}, base order is {expected}" + return None + + violations: List[str] = [] + for cte in ctes[1:]: + found = _check(f"CTE {cte.alias_or_name!r}", list(cte.this.named_selects)) + if found: + violations.append(found) + # The inner SELECT of a derived-table wrap (``) AS _outer``) is its own site. + for sub in tree.find_all(exp.Subquery): + if sub.alias_or_name != "_outer": + continue + inner = sub.this + selects = getattr(inner, "named_selects", None) + if selects: + found = _check("inner SELECT under _outer", list(selects)) + if found: + violations.append(found) + return violations + + +def find_cte(sql: str, name: str, *, dialect: str = "postgres") -> Optional[exp.Expression]: + """The parsed body of the CTE called ``name`` (exact match), or ``None``.""" + tree = sqlglot.parse_one(sql, dialect=dialect) + with_node = tree.args.get("with_") + if with_node is None: + return None + for cte in with_node.expressions: + if cte.alias_or_name == name: + return cte.this + return None + + +__all__ = [ + "NULL_STATUS_JAN", + "NULL_STATUS_FEB", + "NULL_STATUS_FEB_WINDOW", + "PAID_JAN", + "PAID_FEB", + "PAID_FEB_WINDOW", + "seed_dev1746_sqlite", + "dev1746_models", + "make_sqlite_engine", + "outer_select_aliases", + "outer_statement", + "outer_clause_sql", + "join_alias_sequence", + "base_cte_join_sequence", + "carried_alias_drops", + "carry_list_order_violations", + "joinback_on_predicate_for", + "joined_cte_names", + "cte_names_in_order", + "src_subquery_on_predicate", + "find_cte", +] diff --git a/tests/golden/dev1745_sql_baseline.json b/tests/golden/dev1745_sql_baseline.json index 172e84bb..25f1fb79 100644 --- a/tests/golden/dev1745_sql_baseline.json +++ b/tests/golden/dev1745_sql_baseline.json @@ -9,10 +9,7 @@ "cm/joined_measure::postgres": "WITH _base AS (\nSELECT\n orders.status AS \"orders.status\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__customers__spend_sum AS (\nSELECT\n SUM(customers.spend) AS \"orders.customers.spend_sum\"\nFROM customers AS customers\n)\nSELECT _base.\"orders.status\", _cm_orders__customers__spend_sum.\"orders.customers.spend_sum\"\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_sum", "cm/joined_measure::sqlite": "WITH _base AS (\nSELECT\n orders.status AS \"orders.status\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__customers__spend_sum AS (\nSELECT\n SUM(customers.spend) AS \"orders.customers.spend_sum\"\nFROM customers AS customers\n)\nSELECT _base.\"orders.status\", _cm_orders__customers__spend_sum.\"orders.customers.spend_sum\"\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_sum", "cm/joined_measure::tsql": "WITH _base AS (\nSELECT\n orders.status AS [orders___status]\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__customers__spend_sum AS (\nSELECT\n SUM(customers.spend) AS [orders___customers___spend_sum]\nFROM customers AS customers\n)\nSELECT _base.[orders___status], _cm_orders__customers__spend_sum.[orders___customers___spend_sum]\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_sum", - "cm/outer_where_wrapper::bigquery": { - "error": "ScopeLeakError", - "message": "Scope not closed \u2014 2 out-of-scope reference(s):\n - [unbound_table] _base___orders.status in top-level SELECT (bound sources: ['_base', '_cm_orders__eu_amount_sum'])\n - [unbound_table] _cm_orders__eu_amount_sum___orders.status in top-level SELECT (bound sources: ['_base', '_cm_orders__eu_amount_sum'])\nSQL:\nWITH _base AS (\nSELECT\n orders.status AS `orders___status`\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__eu_amount_sum AS (\nSELECT\n orders.status AS `orders___status`,\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS FLOAT64) AS `orders___eu`\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n)\nSELECT _base.`orders___status`, _cm_orders__eu_amount_sum.`orders___eu`\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum ON `_base___orders`.`status` IS NOT DISTINCT FROM `_cm_orders__eu_amount_sum___orders`.`status`\nWHERE _cm_orders__eu_amount_sum.`orders___eu` > 100" - }, + "cm/outer_where_wrapper::bigquery": "WITH _base AS (\nSELECT\n orders.status AS `orders___status`\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__eu_amount_sum AS (\nSELECT\n orders.status AS `orders___status`,\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS FLOAT64) AS `orders___eu`\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n)\nSELECT _base.`orders___status`, _cm_orders__eu_amount_sum.`orders___eu`\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum ON _base.`orders___status` IS NOT DISTINCT FROM _cm_orders__eu_amount_sum.`orders___status`\nWHERE _cm_orders__eu_amount_sum.`orders___eu` > 100", "cm/outer_where_wrapper::duckdb": "WITH _base AS (\nSELECT\n orders.status AS \"orders.status\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__eu_amount_sum AS (\nSELECT\n orders.status AS \"orders.status\",\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS DOUBLE) AS \"orders.eu\"\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n)\nSELECT _base.\"orders.status\", _cm_orders__eu_amount_sum.\"orders.eu\"\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum ON _base.\"orders.status\" IS NOT DISTINCT FROM _cm_orders__eu_amount_sum.\"orders.status\"\nWHERE _cm_orders__eu_amount_sum.\"orders.eu\" > 100", "cm/outer_where_wrapper::postgres": "WITH _base AS (\nSELECT\n orders.status AS \"orders.status\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__eu_amount_sum AS (\nSELECT\n orders.status AS \"orders.status\",\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS DOUBLE PRECISION) AS \"orders.eu\"\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n)\nSELECT _base.\"orders.status\", _cm_orders__eu_amount_sum.\"orders.eu\"\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum ON _base.\"orders.status\" IS NOT DISTINCT FROM _cm_orders__eu_amount_sum.\"orders.status\"\nWHERE _cm_orders__eu_amount_sum.\"orders.eu\" > 100", "cm/outer_where_wrapper::sqlite": "WITH _base AS (\nSELECT\n orders.status AS \"orders.status\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__eu_amount_sum AS (\nSELECT\n orders.status AS \"orders.status\",\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS REAL) AS \"orders.eu\"\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n)\nSELECT _base.\"orders.status\", _cm_orders__eu_amount_sum.\"orders.eu\"\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum ON _base.\"orders.status\" IS _cm_orders__eu_amount_sum.\"orders.status\"\nWHERE _cm_orders__eu_amount_sum.\"orders.eu\" > 100", @@ -27,10 +24,7 @@ "expand/multi_model_derived::postgres": "SELECT\n CAST(customers.spend + customers__regions.population AS DOUBLE PRECISION) AS \"orders.multi_model\",\n CAST(SUM(orders.amount) AS DOUBLE PRECISION) AS \"orders.m\"\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nLEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\nWHERE\n orders.amount >= 0\nGROUP BY\n CAST(customers.spend + customers__regions.population AS DOUBLE PRECISION)", "expand/multi_model_derived::sqlite": "SELECT\n CAST(customers.spend + customers__regions.population AS REAL) AS \"orders.multi_model\",\n CAST(SUM(orders.amount) AS REAL) AS \"orders.m\"\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nLEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\nWHERE\n orders.amount >= 0\nGROUP BY\n CAST(customers.spend + customers__regions.population AS REAL)", "expand/multi_model_derived::tsql": "SELECT\n CAST(customers.spend + customers__regions.population AS FLOAT) AS [orders___multi_model],\n CAST(SUM(orders.amount) AS FLOAT) AS [orders___m]\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nLEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\nWHERE\n orders.amount >= 0\nGROUP BY\n CAST(customers.spend + customers__regions.population AS FLOAT)", - "host/column_filter_crossing::bigquery": { - "error": "ScopeLeakError", - "message": "Scope not closed \u2014 2 out-of-scope reference(s):\n - [unbound_table] _base___orders.status in top-level SELECT (bound sources: ['_base', '_cm_orders__eu_amount_sum'])\n - [unbound_table] _cm_orders__eu_amount_sum___orders.status in top-level SELECT (bound sources: ['_base', '_cm_orders__eu_amount_sum'])\nSQL:\nWITH _base AS (\nSELECT\n orders.status AS `orders___status`\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__eu_amount_sum AS (\nSELECT\n orders.status AS `orders___status`,\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS FLOAT64) AS `orders___m`\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n)\nSELECT _base.`orders___status`, _cm_orders__eu_amount_sum.`orders___m`\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum ON `_base___orders`.`status` IS NOT DISTINCT FROM `_cm_orders__eu_amount_sum___orders`.`status`" - }, + "host/column_filter_crossing::bigquery": "WITH _base AS (\nSELECT\n orders.status AS `orders___status`\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__eu_amount_sum AS (\nSELECT\n orders.status AS `orders___status`,\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS FLOAT64) AS `orders___m`\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n)\nSELECT _base.`orders___status`, _cm_orders__eu_amount_sum.`orders___m`\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum ON _base.`orders___status` IS NOT DISTINCT FROM _cm_orders__eu_amount_sum.`orders___status`", "host/column_filter_crossing::duckdb": "WITH _base AS (\nSELECT\n orders.status AS \"orders.status\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__eu_amount_sum AS (\nSELECT\n orders.status AS \"orders.status\",\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS DOUBLE) AS \"orders.m\"\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n)\nSELECT _base.\"orders.status\", _cm_orders__eu_amount_sum.\"orders.m\"\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum ON _base.\"orders.status\" IS NOT DISTINCT FROM _cm_orders__eu_amount_sum.\"orders.status\"", "host/column_filter_crossing::postgres": "WITH _base AS (\nSELECT\n orders.status AS \"orders.status\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__eu_amount_sum AS (\nSELECT\n orders.status AS \"orders.status\",\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS DOUBLE PRECISION) AS \"orders.m\"\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n)\nSELECT _base.\"orders.status\", _cm_orders__eu_amount_sum.\"orders.m\"\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum ON _base.\"orders.status\" IS NOT DISTINCT FROM _cm_orders__eu_amount_sum.\"orders.status\"", "host/column_filter_crossing::sqlite": "WITH _base AS (\nSELECT\n orders.status AS \"orders.status\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__eu_amount_sum AS (\nSELECT\n orders.status AS \"orders.status\",\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS REAL) AS \"orders.m\"\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n)\nSELECT _base.\"orders.status\", _cm_orders__eu_amount_sum.\"orders.m\"\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum ON _base.\"orders.status\" IS _cm_orders__eu_amount_sum.\"orders.status\"", @@ -70,10 +64,7 @@ "windowed/date_range_filter::postgres": "SELECT\n DATE_TRUNC('MONTH', orders.created_at) AS \"orders.created_at\",\n CAST(SUM(orders.amount) AS DOUBLE PRECISION) AS \"orders.m\"\nFROM orders AS orders\nWHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\nGROUP BY\n DATE_TRUNC('MONTH', orders.created_at)", "windowed/date_range_filter::sqlite": "SELECT\n STRFTIME('%Y-%m-01', orders.created_at) AS \"orders.created_at\",\n CAST(SUM(orders.amount) AS REAL) AS \"orders.m\"\nFROM orders AS orders\nWHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\nGROUP BY\n STRFTIME('%Y-%m-01', orders.created_at)", "windowed/date_range_filter::tsql": "SELECT\n DATETRUNC(month, orders.created_at) AS [orders___created_at],\n CAST(SUM(orders.amount) AS FLOAT) AS [orders___m]\nFROM orders AS orders\nWHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\nGROUP BY\n DATETRUNC(month, orders.created_at)", - "windowed/src_scope::bigquery": { - "error": "ScopeLeakError", - "message": "Scope not closed \u2014 2 out-of-scope reference(s):\n - [unbound_table] _base___orders.created_at in top-level SELECT (bound sources: ['_base', '_cm_orders__eu_amount_sum'])\n - [unbound_table] _cm_orders__eu_amount_sum___orders.created_at in top-level SELECT (bound sources: ['_base', '_cm_orders__eu_amount_sum'])\nSQL:\nWITH _base AS (\nSELECT\n DATE_TRUNC(orders.created_at, MONTH) AS `orders___created_at`\nFROM orders AS orders\nWHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\nGROUP BY\n DATE_TRUNC(orders.created_at, MONTH)\n), _cm_orders__eu_amount_sum AS (\nSELECT\n DATE_TRUNC(orders.created_at, MONTH) AS `orders___created_at`,\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS FLOAT64) AS `orders___m`\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\nGROUP BY\n DATE_TRUNC(orders.created_at, MONTH)\n)\nSELECT _base.`orders___created_at`, _cm_orders__eu_amount_sum.`orders___m`\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum ON `_base___orders`.`created_at` IS NOT DISTINCT FROM `_cm_orders__eu_amount_sum___orders`.`created_at`" - }, + "windowed/src_scope::bigquery": "WITH _base AS (\nSELECT\n DATE_TRUNC(orders.created_at, MONTH) AS `orders___created_at`\nFROM orders AS orders\nWHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\nGROUP BY\n DATE_TRUNC(orders.created_at, MONTH)\n), _cm_orders__eu_amount_sum AS (\nSELECT\n DATE_TRUNC(orders.created_at, MONTH) AS `orders___created_at`,\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS FLOAT64) AS `orders___m`\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\nGROUP BY\n DATE_TRUNC(orders.created_at, MONTH)\n)\nSELECT _base.`orders___created_at`, _cm_orders__eu_amount_sum.`orders___m`\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum ON _base.`orders___created_at` IS NOT DISTINCT FROM _cm_orders__eu_amount_sum.`orders___created_at`", "windowed/src_scope::duckdb": "WITH _base AS (\nSELECT\n DATE_TRUNC('MONTH', orders.created_at) AS \"orders.created_at\"\nFROM orders AS orders\nWHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\nGROUP BY\n DATE_TRUNC('MONTH', orders.created_at)\n), _cm_orders__eu_amount_sum AS (\nSELECT\n DATE_TRUNC('MONTH', orders.created_at) AS \"orders.created_at\",\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS DOUBLE) AS \"orders.m\"\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\nGROUP BY\n DATE_TRUNC('MONTH', orders.created_at)\n)\nSELECT _base.\"orders.created_at\", _cm_orders__eu_amount_sum.\"orders.m\"\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum ON _base.\"orders.created_at\" IS NOT DISTINCT FROM _cm_orders__eu_amount_sum.\"orders.created_at\"", "windowed/src_scope::postgres": "WITH _base AS (\nSELECT\n DATE_TRUNC('MONTH', orders.created_at) AS \"orders.created_at\"\nFROM orders AS orders\nWHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\nGROUP BY\n DATE_TRUNC('MONTH', orders.created_at)\n), _cm_orders__eu_amount_sum AS (\nSELECT\n DATE_TRUNC('MONTH', orders.created_at) AS \"orders.created_at\",\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS DOUBLE PRECISION) AS \"orders.m\"\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\nGROUP BY\n DATE_TRUNC('MONTH', orders.created_at)\n)\nSELECT _base.\"orders.created_at\", _cm_orders__eu_amount_sum.\"orders.m\"\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum ON _base.\"orders.created_at\" IS NOT DISTINCT FROM _cm_orders__eu_amount_sum.\"orders.created_at\"", "windowed/src_scope::sqlite": "WITH _base AS (\nSELECT\n STRFTIME('%Y-%m-01', orders.created_at) AS \"orders.created_at\"\nFROM orders AS orders\nWHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\nGROUP BY\n STRFTIME('%Y-%m-01', orders.created_at)\n), _cm_orders__eu_amount_sum AS (\nSELECT\n STRFTIME('%Y-%m-01', orders.created_at) AS \"orders.created_at\",\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS REAL) AS \"orders.m\"\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\nGROUP BY\n STRFTIME('%Y-%m-01', orders.created_at)\n)\nSELECT _base.\"orders.created_at\", _cm_orders__eu_amount_sum.\"orders.m\"\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum ON _base.\"orders.created_at\" IS _cm_orders__eu_amount_sum.\"orders.created_at\"", diff --git a/tests/test_dev1745_golden_sql.py b/tests/test_dev1745_golden_sql.py index b94166f5..6b67dc5c 100644 --- a/tests/test_dev1745_golden_sql.py +++ b/tests/test_dev1745_golden_sql.py @@ -464,9 +464,15 @@ def test_first_generation_writes_everything(self) -> None: def test_error_entries_record_the_full_message(baseline) -> None: """Deferred item 11 — a bare exception TYPE lets any new failure in the - same case pass. Every error entry must carry its message.""" + same case pass. Every error entry must carry its message. + + The baseline currently records NO exceptions: the five ``_cm_`` fragment-join + entries were fixed by the one-Mode-A-door work, and the last three (BigQuery + dotted-alias corruption in the grain join-back) by DEV-1746 B2. So this now + guards the FORMAT rather than the existence of error entries — it is vacuous + while the baseline is clean, and constrains the next entry that appears. + """ errors = {k: v for k, v in baseline.items() if isinstance(v, dict)} - assert errors, "expected at least one baseline entry to record an exception" for key, value in sorted(errors.items()): assert value.get("error"), f"{key} has no exception type" assert value.get("message", "").strip(), ( diff --git a/tests/test_dev1746_consumer_materialization.py b/tests/test_dev1746_consumer_materialization.py new file mode 100644 index 00000000..bfd46e8c --- /dev/null +++ b/tests/test_dev1746_consumer_materialization.py @@ -0,0 +1,403 @@ +"""DEV-1746 §5.1 — ``resolve(consumer=…)`` gets its first production caller. + +``ScopeFrame.resolve(consumer=…)`` and ``apply_materializations`` implement the +projection-boundary principle: a value that crosses a join inside a producing +scope is projected there under a ``_val_`` alias and referenced by that alias +from the consumer. Both are fully unit-tested — and both have **zero production +callers**. An API nobody calls does not establish a principle, so §5.1 requires +a production-path test. + +Meanwhile the generator carries its own second materialiser for exactly the same +job (``allocate_val()`` + a ``value_alias_by_sql`` dict, deduped by resolved SQL +text). Two mechanisms, one purpose. "The ``consumer=`` materializer becomes the +ONLY one on the cross-model path" is therefore read as: it REPLACES that +generator-local flow. Slot-backed public columns keep resolving through +projected aliases — consuming another scope's projected column by its alias +already *is* exchanging data through projected columns. + +The two sites migrated here are the ones inside ``_render_cross_model_cte``: + +* a **crossing grain** — grouping a first/last cross-model aggregate by a + derived dimension whose SQL reaches a further join. The ranked subquery + re-exports only ``target.*``, so the grain must be projected inside it:: + + regions.population AS _val_0 -- inside the ranked subquery + ... + GROUP BY _val_0 -- the outer CTE reads the alias + +* a **crossing value** — the aggregated expression itself crosses a join. + +``_build_first_last_base_select``'s copy of this flow is deliberately NOT +migrated here: PR 5 rewrites that machinery wholesale as ``RankedAggregatePlan``, +and moving the same state twice is what this PR's sequencing exists to avoid. +That corner is recorded in the PR-5 handoff, and the test at the bottom of this +module pins it as a known, deliberate exception rather than leaving it silent. + +Both dedup on the resolved SQL text — ``ScopeFrame``'s key is +``(scope_id, rendered_ast, dialect)`` — so the migration is byte-preserving for +every shape that materialises through ONE of the two sites. That is asserted +directly below. + +One shape is NOT byte-preserving, and it is a defect the migration fixes: +when a first/last cross-model aggregate is grouped by the same crossing +expression it aggregates, both sites fire and each keeps its own dedup map, so +the expression is projected twice (``_val_0`` and ``_val_1``). ``ScopeFrame`` +holds one table per scope, so they collapse to one. See +``TestDedupParity::test_one_expression_is_materialised_once_per_scope`` — it is +a newly surfaced divergence for the PR's approval list, not a ratified B-item. +""" + +from __future__ import annotations + +from typing import List, Optional, Tuple + +import pytest + +from slayer.core.enums import DataType +from slayer.core.models import Column, ModelMeasure +from slayer.core.query import ColumnRef, SlayerQuery +from slayer.sql.scope import ScopeFrame + +from tests._cross_model_chain import _gen +from tests._engine_helpers import _extract_cte_body, _norm + +# --------------------------------------------------------------------------- # +# Byte-parity baselines — the emitted SQL these shapes produce today. The +# migration swaps WHICH materialiser mints ``_val_0``; it must not change the +# SQL, so these are pinned verbatim (normalised for whitespace only). +# --------------------------------------------------------------------------- # +CROSSING_GRAIN_CTE = _norm( + """ + SELECT _val_0 AS "orders_x.customers_v2.deep_pop", + MAX(CASE WHEN _first_rn = 1 THEN customers_v2.lifetime_value END) + AS "orders_x.customers_v2.lifetime_value_first" + FROM ( SELECT customers_v2.*, regions.population AS _val_0, + ROW_NUMBER() OVER (PARTITION BY regions.population + ORDER BY customers_v2.signup_at ASC) AS _first_rn + FROM customers AS customers_v2 + LEFT JOIN regions AS regions ON customers_v2.region_id = regions.id + ) AS customers_v2 GROUP BY _val_0 + """ +) + +CROSSING_VALUE_CTE = _norm( + """ + SELECT customers_v2.status AS "orders_x.customers_v2.status", + MAX(CASE WHEN _first_rn = 1 THEN customers_v2._val_0 END) + AS "orders_x.customers_v2.deep_weight_first" + FROM ( SELECT customers_v2.*, regions.weight AS _val_0, + ROW_NUMBER() OVER (PARTITION BY customers_v2.status + ORDER BY customers_v2.signup_at ASC) AS _first_rn + FROM customers AS customers_v2 + LEFT JOIN regions AS regions ON customers_v2.region_id = regions.id + ) AS customers_v2 GROUP BY customers_v2.status + """ +) + + +def _crossing_grain_query() -> SlayerQuery: + """First/last cross-model aggregate grouped by a CROSSING derived grain.""" + return SlayerQuery( + source_model="orders_x", + dimensions=[ColumnRef(name="customers_v2.deep_pop")], + measures=[ModelMeasure( + formula="customers_v2.lifetime_value:first", name="f", + )], + ) + + +def _crossing_value_query() -> SlayerQuery: + """First/last cross-model aggregate whose VALUE crosses a join.""" + return SlayerQuery( + source_model="orders_x", + dimensions=[ColumnRef(name="customers_v2.status")], + measures=[ModelMeasure(formula="customers_v2.deep_weight:first", name="f")], + ) + + +def _two_distinct_crossing_values_query() -> SlayerQuery: + return SlayerQuery( + source_model="orders_x", + dimensions=[ColumnRef(name="customers_v2.status")], + measures=[ + ModelMeasure(formula="customers_v2.deep_weight:first", name="w"), + ModelMeasure(formula="customers_v2.deep_pop:first", name="p"), + ], + ) + + +class _ResolveSpy: + """Records every ``ScopeFrame.resolve`` call and whether it named a consumer. + + A spy rather than a grep: the point of §5.1 is that the branch runs on the + PRODUCTION path, which only an observed call can establish. + """ + + def __init__(self) -> None: + self.calls: List[Tuple[str, bool]] = [] + + def install(self, monkeypatch: pytest.MonkeyPatch) -> None: + original = ScopeFrame.resolve + spy = self + + def _wrapped(self_frame, ref, *, consumer: Optional[ScopeFrame] = None): + spy.calls.append((type(ref).__name__, consumer is not None)) + return original(self_frame, ref, consumer=consumer) + + monkeypatch.setattr(ScopeFrame, "resolve", _wrapped, raising=True) + + @property + def consumer_calls(self) -> int: + return sum(1 for _, had_consumer in self.calls if had_consumer) + + +class _MaterializeSpy: + def __init__(self) -> None: + self.aliases: List[str] = [] + + def install(self, monkeypatch: pytest.MonkeyPatch) -> None: + original = ScopeFrame._materialize + spy = self + + def _wrapped(self_frame, template): + alias = original(self_frame, template) + spy.aliases.append(alias) + return alias + + monkeypatch.setattr(ScopeFrame, "_materialize", _wrapped, raising=True) + + +# =========================================================================== # +# The production-path proof. +# =========================================================================== # +class TestConsumerMaterializationOnTheProductionPath: + + @pytest.mark.parametrize( + "query_factory", + [_crossing_grain_query, _crossing_value_query], + ids=["crossing_grain", "crossing_value"], + ) + async def test_resolve_is_called_with_a_consumer( + self, query_factory, monkeypatch: pytest.MonkeyPatch, + ) -> None: + """NEW (§5.1): generating a real query exercises the materialisation + branch of ``ScopeFrame.resolve``.""" + spy = _ResolveSpy() + spy.install(monkeypatch) + await _gen(query_factory(), dialect="postgres") + assert spy.calls, "ScopeFrame.resolve was never called at all" + assert spy.consumer_calls > 0, ( + "no production call passed `consumer=` — the cross-model CTE is " + "still minting `_val_` aliases through the generator's own " + "materialiser, so `resolve`'s projection-boundary branch remains " + f"unexercised in production ({len(spy.calls)} consumer-less calls)." + ) + + @pytest.mark.parametrize( + "query_factory", + [_crossing_grain_query, _crossing_value_query], + ids=["crossing_grain", "crossing_value"], + ) + async def test_scope_frame_mints_the_val_alias( + self, query_factory, monkeypatch: pytest.MonkeyPatch, + ) -> None: + """The alias in the emitted SQL is the one ``ScopeFrame`` minted — not a + coincidentally-identical one from the generator's own allocator.""" + spy = _MaterializeSpy() + spy.install(monkeypatch) + sql = await _gen(query_factory(), dialect="postgres") + assert spy.aliases, ( + "ScopeFrame._materialize never ran, so the `_val_` alias in the " + f"emitted SQL came from the superseded materialiser:\n{sql}" + ) + for alias in spy.aliases: + assert alias in sql, ( + f"materialised alias {alias!r} is absent from the emitted SQL — " + f"the scope materialised a value nobody projected:\n{sql}" + ) + + async def test_apply_materializations_has_a_production_caller( + self, monkeypatch: pytest.MonkeyPatch, + ) -> None: + """The other half of the contract: what the scope materialises must be + projected into the producing SELECT by ``apply_materializations``.""" + seen: List[int] = [] + original = ScopeFrame.apply_materializations + + def _wrapped(self_frame, select): + seen.append(len(self_frame.materializations)) + return original(self_frame, select) + + monkeypatch.setattr( + ScopeFrame, "apply_materializations", _wrapped, raising=True, + ) + await _gen(_crossing_grain_query(), dialect="postgres") + assert seen, ( + "apply_materializations was never called on the production path" + ) + assert any(count > 0 for count in seen), ( + "apply_materializations ran but the scope held no materialisations" + ) + + +# =========================================================================== # +# Byte-parity: swapping the materialiser must not change emitted SQL. +# =========================================================================== # +class TestMigrationIsBytePreserving: + + async def test_crossing_grain_cte_is_unchanged(self) -> None: + sql = await _gen(_crossing_grain_query(), dialect="postgres") + body = _norm(_extract_cte_body(sql, r"_cm_\w+")) + assert body == CROSSING_GRAIN_CTE, ( + "the crossing-grain CTE changed shape. The two materialisers dedup " + "on the same key (resolved SQL text), so the migration must be " + f"byte-preserving.\n\nactual:\n{body}\n\nexpected:\n" + f"{CROSSING_GRAIN_CTE}" + ) + + async def test_crossing_value_cte_is_unchanged(self) -> None: + sql = await _gen(_crossing_value_query(), dialect="postgres") + body = _norm(_extract_cte_body(sql, r"_cm_\w+")) + assert body == CROSSING_VALUE_CTE, ( + f"the crossing-value CTE changed shape.\n\nactual:\n{body}\n\n" + f"expected:\n{CROSSING_VALUE_CTE}" + ) + + +# =========================================================================== # +# Dedup parity with the flow being replaced. +# =========================================================================== # +class TestDedupParity: + + async def test_two_distinct_crossing_values_get_distinct_aliases( + self, + ) -> None: + """Different values must never collapse onto one alias — that would + silently aggregate the wrong column.""" + sql = await _gen(_two_distinct_crossing_values_query(), dialect="postgres") + assert "_val_0" in sql and "_val_1" in sql, ( + f"expected two distinct materialisations:\n{sql}" + ) + + async def test_one_expression_is_materialised_once_per_scope(self) -> None: + """NEWLY SURFACED DIVERGENCE — grouping a first/last cross-model + aggregate by the SAME crossing expression it aggregates materialises it + TWICE today:: + + regions.weight AS _val_0, -- minted by the grain loop + regions.weight AS _val_1, -- minted by the value branch + + because the two sites keep SEPARATE dedup maps (the grain loop calls + ``allocate_val()`` without consulting the value branch's + ``value_alias_by_sql``). ``ScopeFrame`` holds ONE materialisation table + per scope keyed on the rendered template, so routing both through + ``resolve(consumer=…)`` collapses them to a single projection. + + This is a redundant column rather than a wrong answer, but it is an + emitted-SQL change beyond the ratified B-items and belongs on the PR's + approval list. + """ + query = SlayerQuery( + source_model="orders_x", + dimensions=[ColumnRef(name="customers_v2.deep_weight")], + measures=[ModelMeasure( + formula="customers_v2.deep_weight:first", name="f", + )], + ) + sql = await _gen(query, dialect="postgres") + body = _extract_cte_body(sql, r"_cm_\w+") + assert body.count("AS _val_") == 1, ( + "the same crossing expression was materialised more than once in " + f"one scope — the two materialisers still hold separate dedup " + f"tables:\n{body}" + ) + + async def test_separate_scopes_keep_independent_materialisations( + self, + ) -> None: + """Dedup is per SCOPE, not global: two ``_cm_`` CTEs are two scopes, so + each materialises its own copy. Pinned so the dedup unification above + is not over-applied into cross-scope sharing, which would reference an + alias that does not exist in the consuming CTE.""" + query = SlayerQuery( + source_model="orders_x", + dimensions=[ColumnRef(name="customers_v2.status")], + measures=[ + ModelMeasure(formula="customers_v2.deep_weight:first", name="a"), + ModelMeasure(formula="customers_v2.deep_weight:last", name="b"), + ], + ) + sql = await _gen(query, dialect="postgres") + for pattern in ( + r"_cm_\w*deep_weight_first\w*", + r"_cm_\w*deep_weight_last\w*", + ): + body = _extract_cte_body(sql, pattern) + assert body.count("AS _val_") == 1, ( + f"each scope must project its own single materialisation:\n{body}" + ) + + +# =========================================================================== # +# The deliberate PR-5 exception, pinned rather than left implicit. +# =========================================================================== # +class TestRankedBaseMaterialiserStillDeferred: + """``_build_first_last_base_select`` keeps its own ``_val_`` flow until PR 5 + replaces it with ``RankedAggregatePlan``. + + To reach that code the aggregate must be rooted at the HOST — a + ``customers_v2.…`` measure is cross-model and goes through + ``_render_cross_model_cte`` instead. So the host model gets a derived column + whose SQL crosses into the joined model, and the first/last aggregate is + taken over THAT. + """ + + _HOST_CROSSING_COLUMN = [ + Column( + name="cust_ltv", + sql="customers_v2.lifetime_value", + type=DataType.DOUBLE, + ), + ] + + def _query(self) -> SlayerQuery: + return SlayerQuery( + source_model="orders_x", + dimensions=[ColumnRef(name="status")], + measures=[ModelMeasure(formula="cust_ltv:first", name="f")], + ) + + async def test_local_first_last_still_materialises_correctly(self) -> None: + """The corner keeps WORKING while it waits — pinned so PR 5 inherits a + test rather than a silent assumption.""" + sql = await _gen( + self._query(), orders_extra=self._HOST_CROSSING_COLUMN, + dialect="postgres", + ) + assert "_val_" in sql, ( + f"expected a materialised crossing value in this shape:\n{sql}" + ) + assert "ROW_NUMBER() OVER" in sql, sql + + async def test_the_ranked_base_path_is_still_consumer_free( + self, monkeypatch: pytest.MonkeyPatch, + ) -> None: + """The deferral itself, pinned rather than assumed. + + This shape's ``_val_`` alias must still come from the generator's own + materialiser, NOT from ``ScopeFrame``. When PR 5 migrates it this test + flips — which is the point: the boundary between the two PRs is + asserted, so it cannot drift silently. + """ + spy = _MaterializeSpy() + spy.install(monkeypatch) + sql = await _gen( + self._query(), orders_extra=self._HOST_CROSSING_COLUMN, + dialect="postgres", + ) + assert "_val_" in sql, sql + assert not spy.aliases, ( + "ScopeFrame materialised for the ranked BASE path. That migration " + "belongs to PR 5 (RankedAggregatePlan); if it landed early, move " + f"this test rather than deleting it. aliases={spy.aliases}" + ) diff --git a/tests/test_dev1746_cte_assembly.py b/tests/test_dev1746_cte_assembly.py new file mode 100644 index 00000000..e03751a8 --- /dev/null +++ b/tests/test_dev1746_cte_assembly.py @@ -0,0 +1,326 @@ +"""DEV-1746 §5.6 — WITH-chain assembly in topological order. + +The cross-model paths splice their WITH chain out of f-strings:: + + cte_strs = [f"{name} AS (\\n{sql}\\n)" for name, sql in all_ctes[:-1]] + sql = f"WITH {', '.join(cte_strs)}\\n{combined_select_sql}" + +so CTE order is whatever order the python list happened to be built in, and the +transform chain reads its predecessor positionally (``prev_cte = ctes[-1][0]``). +That works today only because the list is assembled in one hard-coded sequence. + +§5.6 replaces it with one assembler that takes **explicitly declared** +dependencies and emits a stable topological order (insertion order as the +tiebreak). Dependencies are declared by the caller — ``_wm_`` depends on +``_base``, transform step N on step N-1, ``_cm_`` on nothing — rather than +discovered by scanning the rendered AST, because scanning cannot distinguish a +CTE reference from a same-named real table, is defeated by quoting and case +folding, and would silently mis-order rather than fail (Codex D3). + +``assert_unique_cte_names`` stays as the belt: the assembler is responsible for +ORDER, the belt for name collisions, including the case-folding collisions that +only appear on dialects that fold (DEV-1726). + +Scope note: per the PR's ruling only the CROSS-MODEL sites adopt the assembler +in this PR (the combined tail and the cross-model transform chain). The local +single-model transform chain keeps its splice until PR 4, so the engine-level +tests here use cross-model shapes. +""" + +from __future__ import annotations + +import pytest +import sqlglot +from sqlglot import exp + +from slayer.core.enums import TimeGranularity +from slayer.core.models import ModelMeasure +from slayer.core.query import ColumnRef, SlayerQuery, TimeDimension +from slayer.sql.naming import assert_unique_cte_names + +from tests._cross_model_chain import _gen +from tests._dev1746_fixtures import cte_names_in_order + +#: Imported lazily so a missing implementation fails these tests rather than +#: erroring collection for the whole module. +_ASSEMBLER_MODULE = "slayer.sql.render.cte_assembly" + +DIALECTS = ["postgres", "sqlite", "duckdb", "tsql", "bigquery"] + + +def _cross_model_query() -> SlayerQuery: + return SlayerQuery( + source_model="orders_x", + dimensions=[ColumnRef(name="customers_v2.status")], + measures=[ModelMeasure(formula="customers_v2.lifetime_value:sum")], + ) + + +def _two_cross_model_measures_query() -> SlayerQuery: + """Two independent ``_cm_`` CTEs — neither depends on the other, so their + relative order is decided purely by the declaration-order tiebreak.""" + return SlayerQuery( + source_model="orders_x", + dimensions=[ColumnRef(name="customers_v2.status")], + measures=[ + ModelMeasure(formula="customers_v2.lifetime_value:sum", name="ltv"), + ModelMeasure(formula="customers_v2.lifetime_value:avg", name="ltv_avg"), + ], + ) + + +def _windowed_query() -> SlayerQuery: + return SlayerQuery( + source_model="orders_x", + dimensions=[ColumnRef(name="status")], + time_dimensions=[TimeDimension( + dimension=ColumnRef(name="created_at"), + granularity=TimeGranularity.MONTH, + )], + measures=[ModelMeasure(formula="amount:sum(window='90d')", name="rev_w")], + ) + + +def _mixed_query() -> SlayerQuery: + """Cross-model AND windowed in one query — the mix §5.6 names.""" + return SlayerQuery( + source_model="orders_x", + dimensions=[ColumnRef(name="status")], + time_dimensions=[TimeDimension( + dimension=ColumnRef(name="created_at"), + granularity=TimeGranularity.MONTH, + )], + measures=[ + ModelMeasure(formula="amount:sum(window='90d')", name="rev_w"), + ModelMeasure(formula="customers_v2.lifetime_value:sum", name="ltv"), + ], + ) + + +# =========================================================================== # +# The assembler. +# =========================================================================== # +class TestWithChainAssembler: + + @staticmethod + def _mod(): + import importlib + + return importlib.import_module(_ASSEMBLER_MODULE) + + @staticmethod + def _sel(from_: str = "t") -> exp.Select: + return exp.Select().select(exp.column("a")).from_(from_) + + def _entry(self, name: str, deps: list[str]): + mod = self._mod() + return mod.CteEntry(name=name, query=self._sel(), depends_on=deps) + + def test_dependencies_precede_their_dependents(self) -> None: + """The one hard ordering rule: a CTE is emitted after everything it + declares a dependency on. SQL requires it — a CTE cannot reference a + later sibling.""" + mod = self._mod() + entries = [ + self._entry("c", ["b"]), + self._entry("b", ["a"]), + self._entry("a", []), + ] + out = mod.assemble_with_chain(entries=entries, final=self._sel("c")) + names = [cte.alias_or_name for cte in out.args["with_"].expressions] + assert names.index("a") < names.index("b") < names.index("c"), names + + def test_independent_entries_keep_insertion_order(self) -> None: + """The tiebreak. Two CTEs with no dependency between them must come out + in the order the caller declared them — otherwise emitted SQL would + vary run to run for the same plan.""" + mod = self._mod() + entries = [self._entry(n, []) for n in ("first", "second", "third")] + out = mod.assemble_with_chain(entries=entries, final=self._sel("first")) + names = [cte.alias_or_name for cte in out.args["with_"].expressions] + assert names == ["first", "second", "third"], names + + def test_ordering_is_deterministic_across_repeated_assembly(self) -> None: + mod = self._mod() + + def build() -> list[str]: + entries = [ + self._entry("wm", ["base"]), + self._entry("cm", []), + self._entry("base", []), + ] + out = mod.assemble_with_chain(entries=entries, final=self._sel("base")) + return [cte.alias_or_name for cte in out.args["with_"].expressions] + + assert build() == build(), "assembly order is not deterministic" + + def test_a_dependency_cycle_raises(self) -> None: + """A cycle cannot be emitted as a WITH chain at all. Failing loudly + beats emitting a plausible-looking order that references forward.""" + mod = self._mod() + entries = [self._entry("a", ["b"]), self._entry("b", ["a"])] + with pytest.raises(ValueError, match="(?i)cycle"): + mod.assemble_with_chain(entries=entries, final=self._sel("a")) + + def test_an_unknown_dependency_raises(self) -> None: + """Declaring a dependency on a CTE that was never supplied is a wiring + bug; silently ignoring it would emit SQL referencing a missing table.""" + mod = self._mod() + entries = [self._entry("a", ["nope"])] + with pytest.raises(ValueError, match="(?i)unknown|missing|nope"): + mod.assemble_with_chain(entries=entries, final=self._sel("a")) + + def test_duplicate_names_raise(self) -> None: + mod = self._mod() + entries = [self._entry("dup", []), self._entry("dup", [])] + with pytest.raises(ValueError, match="(?i)duplicate|dup"): + mod.assemble_with_chain(entries=entries, final=self._sel("dup")) + + def test_no_entries_yields_the_final_select_unwrapped(self) -> None: + """No CTEs means no WITH clause — not an empty one, which is invalid.""" + mod = self._mod() + out = mod.assemble_with_chain(entries=[], final=self._sel()) + assert out.args.get("with_") is None, out.sql() + + def test_assembled_statement_is_a_select_not_a_string(self) -> None: + """The point of §5.6: the chain is AST all the way, so a caller can keep + transforming it (pagination, outer wraps) without re-parsing.""" + mod = self._mod() + out = mod.assemble_with_chain( + entries=[self._entry("a", [])], final=self._sel("a"), + ) + assert isinstance(out, exp.Select), type(out) + + @pytest.mark.parametrize("dialect", DIALECTS) + def test_assembled_statement_round_trips_through_every_dialect( + self, dialect: str, + ) -> None: + mod = self._mod() + entries = [self._entry("base", []), self._entry("wm", ["base"])] + out = mod.assemble_with_chain(entries=entries, final=self._sel("wm")) + rendered = out.sql(dialect=dialect) + parsed = sqlglot.parse(rendered, dialect=dialect) + assert len(parsed) == 1, f"[{dialect}] did not round-trip:\n{rendered}" + assert_unique_cte_names(rendered, dialect=dialect) + + def test_quoted_and_mixed_case_names_survive_assembly(self) -> None: + """A quoted alias must stay one identifier through assembly.""" + mod = self._mod() + entries = [self._entry("MixedCase", []), self._entry("other", ["MixedCase"])] + out = mod.assemble_with_chain(entries=entries, final=self._sel("other")) + names = [cte.alias_or_name for cte in out.args["with_"].expressions] + assert names == ["MixedCase", "other"], names + + def test_case_folding_duplicates_are_rejected(self) -> None: + """DEV-1726: two names differing only in case collide on a folding + dialect. The belt catches it in emitted SQL; the assembler must not be + the thing that introduces it.""" + mod = self._mod() + entries = [self._entry("dup", []), self._entry("DUP", [])] + out = mod.assemble_with_chain(entries=entries, final=self._sel("dup")) + with pytest.raises(ValueError): + assert_unique_cte_names(out.sql(dialect="snowflake"), dialect="snowflake") + + +# =========================================================================== # +# Engine-level: the assembled chain for real cross-model shapes. +# =========================================================================== # +class TestAssembledChainForRealQueries: + + async def test_base_precedes_the_cross_model_cte(self) -> None: + sql = await _gen(_cross_model_query(), dialect="postgres") + names = cte_names_in_order(sql) + assert "_base" in names, names + cm = [n for n in names if n.startswith("_cm_")] + assert cm, f"no _cm_ CTE in {names}" + assert names.index("_base") < names.index(cm[0]), names + + async def test_windowed_cte_follows_the_base_it_reads(self) -> None: + """``_wm_`` selects FROM ``_base``, so the dependency is real: emitting + it first would be invalid SQL, not merely untidy.""" + sql = await _gen(_windowed_query(), dialect="postgres") + names = cte_names_in_order(sql) + wm = [n for n in names if n.startswith("_wm_")] + assert wm, f"no _wm_ CTE in {names}" + assert names.index("_base") < names.index(wm[0]), names + + async def test_mixed_cross_model_and_windowed_chain_is_ordered(self) -> None: + sql = await _gen(_mixed_query(), dialect="postgres") + names = cte_names_in_order(sql) + assert names[0] == "_base", names + assert any(n.startswith("_cm_") for n in names), names + assert any(n.startswith("_wm_") for n in names), names + + async def test_two_independent_cross_model_ctes_follow_declaration_order( + self, + ) -> None: + """The tiebreak, end to end: two ``_cm_`` CTEs that do not depend on + each other appear in measure-declaration order.""" + sql = await _gen(_two_cross_model_measures_query(), dialect="postgres") + names = [n for n in cte_names_in_order(sql) if n.startswith("_cm_")] + assert len(names) == 2, f"expected two _cm_ CTEs, got {names}" + assert "sum" in names[0] and "avg" in names[1], ( + f"_cm_ CTEs are not in measure-declaration order: {names}" + ) + + @pytest.mark.parametrize("dialect", DIALECTS) + async def test_assembled_chain_parses_and_has_unique_names( + self, dialect: str, + ) -> None: + sql = await _gen(_mixed_query(), dialect=dialect) + parsed = sqlglot.parse(sql, dialect=dialect) + assert len(parsed) == 1, f"[{dialect}] did not parse:\n{sql}" + assert_unique_cte_names(sql, dialect=dialect) + + @pytest.mark.parametrize("dialect", DIALECTS) + async def test_no_nested_with_clause_is_emitted(self, dialect: str) -> None: + """One statement, one WITH. A nested WITH inside a CTE body is invalid + on T-SQL and a sign the assembler spliced a complete statement in as a + CTE body. + + Counted over parsed ``With`` nodes rather than lines starting with + ``WITH``: an indented or inline nested WITH would slip past the textual + check entirely. + """ + sql = await _gen(_cross_model_query(), dialect=dialect) + tree = sqlglot.parse_one(sql, dialect=dialect) + with_nodes = list(tree.find_all(exp.With)) + assert len(with_nodes) <= 1, ( + f"[{dialect}] {len(with_nodes)} WITH clauses — a CTE body contains " + f"a complete statement:\n{sql}" + ) + + async def test_the_assembler_is_what_builds_the_cross_model_chain( + self, monkeypatch: pytest.MonkeyPatch, + ) -> None: + """Production-path proof for §5.6. + + The ordering tests above would still pass if the legacy f-string splice + happened to produce the same order, which for today's shapes it does. + So this asserts the assembler is actually the thing that runs, and that + it receives EXPLICIT dependency metadata (Codex D3) rather than being + handed a bare list to sort by itself. + """ + mod = TestWithChainAssembler._mod() + calls: list = [] + original = mod.assemble_with_chain + + def _wrapped(*, entries, final, **kwargs): + calls.append(list(entries)) + return original(entries=entries, final=final, **kwargs) + + monkeypatch.setattr(mod, "assemble_with_chain", _wrapped, raising=True) + sql = await _gen(_mixed_query(), dialect="postgres") + assert calls, ( + "the cross-model WITH chain was assembled without the shared " + f"assembler — the f-string splice is still in use:\n{sql}" + ) + entries = calls[-1] + names = {e.name for e in entries} + assert any(n.startswith("_wm_") for n in names), names + # The windowed CTE reads _base, so its dependency must be DECLARED. + wm_entries = [e for e in entries if e.name.startswith("_wm_")] + assert all("_base" in e.depends_on for e in wm_entries), ( + "a _wm_ CTE selects FROM _base but did not declare that " + f"dependency: {[(e.name, e.depends_on) for e in wm_entries]}" + ) diff --git a/tests/test_dev1746_empty_base_plan.py b/tests/test_dev1746_empty_base_plan.py new file mode 100644 index 00000000..c2bbe350 --- /dev/null +++ b/tests/test_dev1746_empty_base_plan.py @@ -0,0 +1,285 @@ +"""DEV-1746 §5.12 — the empty-base grain becomes a typed planner node. + +When a query asks only for isolated aggregates — no host row slots and no +host-local aggregates — the host base has nothing to project, so the generator +synthesises a one-row spine for the cross-model CROSS JOIN to hang off:: + + _base AS (SELECT 1 AS _placeholder) -- unfiltered + _base AS (SELECT 1 AS _placeholder FROM orders AS orders_x + WHERE orders_x.status = 'paid' LIMIT 1) -- host-filtered + +Three decisions are made at RENDER time today: whether the shape applies at all, +which ROW-phase filters are host-local rather than routed into a CTE, and the +``LIMIT 1`` collapse. P-D says the plan decides and the renderer emits, so they +move to a typed node. **The emitted SQL does not change** — that is the whole +point, and it is asserted verbatim below. + +Why ``LIMIT 1`` is load-bearing rather than an optimisation: the filtered form +keeps the host FROM so the WHERE can gate the result, but a host FROM yields N +rows, and CROSS JOINing N rows to a 1-row scalar aggregate would repeat the +answer N times. ``LIMIT 1`` collapses the spine to one row while an empty match +still yields zero rows overall. The unfiltered form drops the FROM entirely for +the same reason. + +Per Codex D7 the node stays minimal — its PRESENCE is the discriminator, and it +carries the host filter ids. No ``grain_slot_ids`` field is added: in this shape +the grain is empty by definition, and a field that is always ``[]`` documents +nothing. The invariant it would have encoded is asserted directly instead — +whenever the node is present, every cross-model plan's join-back pairs are +empty, which is exactly why the join degenerates to a CROSS JOIN. +""" + +from __future__ import annotations + +import os +import tempfile +from typing import AsyncIterator + +import pytest + +from slayer.core.models import ModelMeasure +from slayer.core.query import ColumnRef, SlayerQuery +from slayer.engine.query_engine import SlayerQueryEngine +from slayer.engine.source_bundle import ResolvedSourceBundle +from slayer.engine.stage_planner import plan_query + +from tests._cross_model_chain import ( + _countries, + _customers_v2, + _gen, + _orders_x, + _regions, +) +from tests._dev1746_fixtures import ( + make_sqlite_engine, + seed_dev1746_sqlite, +) +from tests._engine_helpers import _extract_cte_body, _norm + +#: The emitted ``_base`` bodies, pinned verbatim (whitespace-normalised). +UNFILTERED_BASE = _norm("SELECT 1 AS _placeholder") +FILTERED_BASE = _norm( + """ + SELECT 1 AS _placeholder + FROM orders AS orders_x + WHERE orders_x.status = 'paid' + LIMIT 1 + """ +) + + +def _bundle() -> ResolvedSourceBundle: + return ResolvedSourceBundle( + source_model=_orders_x(), + referenced_models=[_customers_v2(), _regions(), _countries()], + ) + + +def _unfiltered_query() -> SlayerQuery: + return SlayerQuery( + source_model="orders_x", + measures=[ModelMeasure( + formula="customers_v2.lifetime_value:sum", name="ltv", + )], + ) + + +def _filtered_query() -> SlayerQuery: + return SlayerQuery( + source_model="orders_x", + measures=[ModelMeasure( + formula="customers_v2.lifetime_value:sum", name="ltv", + )], + filters=["status == 'paid'"], + ) + + +def _non_empty_base_query() -> SlayerQuery: + """A host dimension means the base is NOT empty — the node must be absent.""" + return SlayerQuery( + source_model="orders_x", + dimensions=[ColumnRef(name="status")], + measures=[ModelMeasure( + formula="customers_v2.lifetime_value:sum", name="ltv", + )], + ) + + +@pytest.fixture +async def exec_engine() -> AsyncIterator[SlayerQueryEngine]: + with tempfile.TemporaryDirectory() as d: + db_path = os.path.join(d, "dev1746.db") + seed_dev1746_sqlite(db_path) + yield await make_sqlite_engine(os.path.join(d, "store"), db_path) + + +# =========================================================================== # +# The typed node. +# =========================================================================== # +class TestEmptyBaseGrainPlanNode: + + def test_node_is_present_for_the_unfiltered_shape(self) -> None: + """NEW (§5.12): the decision is on the plan, not re-derived at render.""" + planned = plan_query(query=_unfiltered_query(), bundle=_bundle()) + assert planned.empty_base_plan is not None, ( + "the empty-base shape is still decided at render time — the plan " + "carries no node for it." + ) + assert list(planned.empty_base_plan.host_filter_ids) == [], ( + f"unfiltered shape recorded host filters: " + f"{planned.empty_base_plan.host_filter_ids}" + ) + + def test_node_records_the_host_local_filters(self) -> None: + """The filtered form's WHERE is exactly the ROW-phase filters that were + NOT routed into a cross-model CTE.""" + planned = plan_query(query=_filtered_query(), bundle=_bundle()) + assert planned.empty_base_plan is not None + ids = list(planned.empty_base_plan.host_filter_ids) + assert ids, ( + "the host-local filter was not recorded on the node, so the " + "renderer would have to re-walk `filters_by_phase` to find it." + ) + known = {f.id for f in planned.filters_by_phase} + assert set(ids) <= known, ( + f"node references unknown filter ids: {set(ids) - known}" + ) + + def test_node_is_absent_when_the_base_is_not_empty(self) -> None: + """A host dimension gives the base something to project; the placeholder + spine must not appear.""" + planned = plan_query(query=_non_empty_base_query(), bundle=_bundle()) + assert planned.empty_base_plan is None, ( + "the empty-base node was set for a query that HAS host row slots" + ) + + def test_node_presence_implies_every_join_back_is_empty(self) -> None: + """Codex D7: the grain semantics the node would have carried as a field, + asserted as the invariant it actually is. An empty grain is precisely + why the combined SELECT CROSS JOINs instead of joining on a predicate. + """ + for factory in (_unfiltered_query, _filtered_query): + planned = plan_query(query=factory(), bundle=_bundle()) + assert planned.empty_base_plan is not None + for plan in planned.cross_model_aggregate_plans: + assert not plan.join_back_pairs, ( + f"empty-base plan present but cross-model plan " + f"{plan.aggregate_slot_id} declares join-back pairs " + f"{plan.join_back_pairs} — the base has no grain to join on." + ) + + def test_generator_consumes_the_node_rather_than_re_deriving(self) -> None: + """P-D: clearing the plan field must change the emitted SQL. If it does + not, the generator re-derived the decision and the node is decorative. + """ + from slayer.sql.generator import SQLGenerator + + planned = plan_query(query=_filtered_query(), bundle=_bundle()) + assert planned.empty_base_plan is not None, ( + "precondition: the plan must be POPULATED before clearing, " + "otherwise clearing proves nothing" + ) + cleared = planned.model_copy(update={"empty_base_plan": None}) + gen = SQLGenerator(dialect="postgres") + sql = gen.generate_from_planned(planned_query=cleared, bundle=_bundle()) + assert "_placeholder" not in sql, ( + "the generator re-derived the empty-base shape instead of consuming " + f"the plan:\n{sql}" + ) + + +# =========================================================================== # +# Byte-parity — §5.12 changes where the decision lives, not the SQL. +# =========================================================================== # +class TestEmittedSqlIsUnchanged: + + async def test_unfiltered_base_body_is_unchanged(self) -> None: + sql = await _gen(_unfiltered_query(), dialect="postgres") + body = _norm(_extract_cte_body(sql, r"_base")) + assert body == UNFILTERED_BASE, ( + f"unfiltered placeholder spine changed:\n actual: {body}\n" + f" expected: {UNFILTERED_BASE}" + ) + + async def test_filtered_base_body_is_unchanged(self) -> None: + sql = await _gen(_filtered_query(), dialect="postgres") + body = _norm(_extract_cte_body(sql, r"_base")) + assert body == FILTERED_BASE, ( + f"filtered placeholder spine changed:\n actual: {body}\n" + f" expected: {FILTERED_BASE}" + ) + + async def test_unfiltered_shape_has_no_from_clause(self) -> None: + """Explicit: the unfiltered spine must NOT read the host table. With a + FROM it would be N rows, and the CROSS JOIN would repeat the scalar + aggregate N times.""" + sql = await _gen(_unfiltered_query(), dialect="postgres") + body = _norm(_extract_cte_body(sql, r"_base")) + assert "FROM" not in body.upper(), ( + f"the unfiltered placeholder acquired a FROM clause:\n{body}" + ) + + async def test_filtered_shape_keeps_the_limit_one_collapse(self) -> None: + """The LIMIT 1 rule, stated as its own assertion so a refactor cannot + drop it silently.""" + sql = await _gen(_filtered_query(), dialect="postgres") + body = _norm(_extract_cte_body(sql, r"_base")) + assert body.upper().endswith("LIMIT 1"), ( + f"the filtered placeholder lost its LIMIT 1 collapse:\n{body}" + ) + + async def test_combined_select_cross_joins_the_scalar_cte(self) -> None: + for factory in (_unfiltered_query, _filtered_query): + sql = await _gen(factory(), dialect="postgres") + assert "CROSS JOIN _cm_" in _norm(sql), sql + + +# =========================================================================== # +# Execution — the semantics the shape exists to preserve. +# =========================================================================== # +class TestEmptyBaseExecution: + + async def test_scalar_aggregate_is_not_multiplied( + self, exec_engine: SlayerQueryEngine, + ) -> None: + """The reason for the LIMIT-1 collapse: one row, counted once.""" + query = SlayerQuery( + source_model="orders", + measures=[ModelMeasure(formula="customers.spend:sum", name="total")], + ) + resp = await exec_engine.execute(query) + assert len(resp.data) == 1, ( + f"the scalar aggregate was multiplied by the host rowset:\n" + f"{resp.data}" + ) + assert resp.data[0]["orders.total"] == pytest.approx(1325.0), resp.data + + async def test_host_filter_gates_the_whole_result( + self, exec_engine: SlayerQueryEngine, + ) -> None: + """A host-local filter that MATCHES still yields the full scalar + aggregate (the filter gates the spine, it does not restrict the + isolated CTE).""" + query = SlayerQuery( + source_model="orders", + measures=[ModelMeasure(formula="customers.spend:sum", name="total")], + filters=["status == 'paid'"], + ) + resp = await exec_engine.execute(query) + assert len(resp.data) == 1, resp.data + assert resp.data[0]["orders.total"] == pytest.approx(1325.0), resp.data + + async def test_host_filter_matching_nothing_yields_no_rows( + self, exec_engine: SlayerQueryEngine, + ) -> None: + """The other half of the gate: no host row matches -> empty spine -> + zero rows, NOT a row with the scalar aggregate in it.""" + query = SlayerQuery( + source_model="orders", + measures=[ModelMeasure(formula="customers.spend:sum", name="total")], + filters=["status == 'nonexistent'"], + ) + resp = await exec_engine.execute(query) + assert resp.data == [], ( + f"a non-matching host filter still returned rows:\n{resp.data}" + ) diff --git a/tests/test_dev1746_null_safe_grain.py b/tests/test_dev1746_null_safe_grain.py new file mode 100644 index 00000000..b6bf0327 --- /dev/null +++ b/tests/test_dev1746_null_safe_grain.py @@ -0,0 +1,571 @@ +"""DEV-1746 §5.7 — the null-safe grain doctrine (B1 + B2). + +Two ratified behaviour changes, one shared mechanism. + +**B1 — the ``_wm_`` INNER grain goes null-safe.** A windowed measure's ``_wm_`` +CTE joins its ``_src`` row subquery back to ``_base`` on the query grain. That +inner comparison is a plain ``=``, so a group whose dimension is NULL never +matches and silently receives NULL instead of its real windowed value. The outer +``_wm_`` join-back and the ``_cm_`` join-back are already null-safe, so today +SLayer disagrees with itself about NULL-grain semantics depending on which +isolation shape a measure lands in. Both ``_wm_`` comparison sites are asserted +here, as §5.7 requires. + +**B2 — grain join-backs are built as AST, not by string re-parse.** The join-back +predicate is currently assembled by rendering both sides to pre-quoted strings +and re-parsing them (``_null_safe_join_pair_sql``). Public aliases are dotted +(``orders.customers.status``), and on a dialect that mangles dots at emission +(BigQuery ``___``, T-SQL brackets) the round-trip re-reads the dotted alias as a +multi-part *reference*, yielding a qualifier for a table that does not exist: + + ON `_base___orders___customers`.`status` IS NOT DISTINCT FROM ... + ^^^^^^^^^^^^^^^^^^^^^^^^^^ not a table in scope + +sjoin already avoids this by building the columns directly as AST; that +mechanism becomes the one shared builder. The corruption is latent rather than +theoretical: three entries in ``tests/golden/dev1745_sql_baseline.json`` record +``ScopeLeakError`` for exactly this shape, and this PR regenerates them. + +Execution coverage is SQLite in-suite (the DuckDB counterpart lives in +``tests/integration/test_integration_dev1746.py``). What execution *cannot* +cover — the dotted-alias mangling itself — is asserted as emission per §5.13, +because neither SQLite nor DuckDB mangles dots. + +The new shared builder is imported inside the tests that exercise it directly, +so a missing implementation fails those tests rather than erroring collection +for the whole module. +""" + +from __future__ import annotations + +import os +import tempfile +from typing import AsyncIterator + +import pytest +import sqlglot +from sqlglot import exp + +from slayer.core.enums import TimeGranularity +from slayer.core.models import ModelMeasure +from slayer.core.query import ColumnRef, SlayerQuery, TimeDimension +from slayer.engine.query_engine import SlayerQueryEngine +from slayer.sql.dialects import get_dialect +from slayer.sql.scope_check import assert_scope_closed + +from tests._cross_model_chain import _gen +from tests._dev1746_fixtures import ( + NULL_STATUS_FEB_WINDOW, + NULL_STATUS_JAN, + PAID_FEB_WINDOW, + PAID_JAN, + joinback_on_predicate_for, + make_sqlite_engine, + seed_dev1746_sqlite, + src_subquery_on_predicate, +) +from tests._engine_helpers import _norm + +#: The module under construction. Imported lazily inside tests so a missing +#: implementation fails the builder tests, not module collection. +_BUILDER_MODULE = "slayer.sql.render.joins" + + +# --------------------------------------------------------------------------- # +# Fixtures +# --------------------------------------------------------------------------- # +@pytest.fixture +async def exec_engine() -> AsyncIterator[SlayerQueryEngine]: + """Engine over the seeded NULL-bearing SQLite corpus.""" + with tempfile.TemporaryDirectory() as d: + db_path = os.path.join(d, "dev1746.db") + seed_dev1746_sqlite(db_path) + yield await make_sqlite_engine(os.path.join(d, "store"), db_path) + + +def _windowed_query() -> SlayerQuery: + """Windowed measure grouped by a NULLABLE dimension + a month bucket.""" + return SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + time_dimensions=[TimeDimension( + dimension=ColumnRef(name="created_at"), + granularity=TimeGranularity.MONTH, + )], + measures=[ModelMeasure(formula="amount:sum(window='90d')", name="rev_w")], + ) + + +def _windowed_chain_query() -> SlayerQuery: + """The same shape against the shared orders_x chain (postgres-shaped).""" + return SlayerQuery( + source_model="orders_x", + dimensions=[ColumnRef(name="status")], + time_dimensions=[TimeDimension( + dimension=ColumnRef(name="created_at"), + granularity=TimeGranularity.MONTH, + )], + measures=[ModelMeasure(formula="amount:sum(window='90d')", name="rev_w")], + ) + + +def _cm_shared_grain_query() -> SlayerQuery: + """Cross-model measure grouped by a grain shared with the target.""" + return SlayerQuery( + source_model="orders_x", + dimensions=[ColumnRef(name="customers_v2.status")], + measures=[ModelMeasure(formula="customers_v2.lifetime_value:sum")], + ) + + +# =========================================================================== # +# B1 — the ``_wm_`` inner grain goes null-safe. +# =========================================================================== # +class TestB1WindowedInnerGrainNullSafe: + """The inner ``_src`` join is the site that decides whether a NULL-dimension + group receives a real windowed value.""" + + @pytest.mark.parametrize( + "dialect,expected_op", + [ + ("postgres", "IS NOT DISTINCT FROM"), + ("duckdb", "IS NOT DISTINCT FROM"), + ("sqlite", " IS "), + ], + ) + async def test_inner_grain_equality_is_null_safe( + self, dialect: str, expected_op: str, + ) -> None: + """NEW (B1): the inner grain comparison uses the dialect's null-safe + equality, not a plain ``=``.""" + sql = await _gen(_windowed_chain_query(), dialect=dialect) + on = _norm(src_subquery_on_predicate(sql, dialect=dialect)) + assert expected_op in on, ( + f"[{dialect}] the _wm_ inner grain join is not null-safe.\n" + f"ON predicate: {on}\n\nfull SQL:\n{sql}" + ) + # The grain member specifically — the time-range bounds legitimately + # stay plain >= / <, so assert on the dimension comparison only. + assert "_src._w_dim_0 =" not in on, ( + f"[{dialect}] the grain member still compares with a plain `=`, " + f"so a NULL-dimension group cannot match.\nON predicate: {on}" + ) + + async def test_inner_grain_time_bounds_stay_plain_inequalities(self) -> None: + """The window's time-range bounds are NOT grain equality and must keep + their plain ``>=`` / ``<`` (a null-safe rewrite there would be wrong).""" + sql = await _gen(_windowed_chain_query(), dialect="postgres") + on = _norm(src_subquery_on_predicate(sql, dialect="postgres")) + assert "_src._w_time >=" in on, f"lower bound changed shape:\n{on}" + assert "_src._w_time <" in on, f"upper bound changed shape:\n{on}" + + async def test_outer_joinback_remains_null_safe(self) -> None: + """§5.7 requires BOTH ``_wm_`` comparison sites be asserted. The outer + join-back is already null-safe; this pins it so B1's inner-site change + cannot regress it.""" + sql = await _gen(_windowed_chain_query(), dialect="postgres") + on = _norm(joinback_on_predicate_for(sql, prefix="_wm_", dialect="postgres")) + assert "IS NOT DISTINCT FROM" in on, ( + f"the _wm_ outer join-back lost its null-safe equality:\n{on}" + ) + + async def test_null_dimension_group_gets_its_real_windowed_value( + self, exec_engine: SlayerQueryEngine, + ) -> None: + """NEW (B1), EXECUTED: the NULL-status group receives its real 90-day + windowed sum instead of NULL. + + February's window reaches back 90 days and so covers January too: + ``NULL`` → 5.0 + 7.0 = 12.0, ``paid`` → 10.0 + 20.0 = 30.0. The two + differ from their single-month sums, so this cannot pass by accident. + """ + resp = await exec_engine.execute(_windowed_query()) + by_group = { + (r["orders.status"], str(r["orders.created_at"])[:7]): r["orders.rev_w"] + for r in resp.data + } + feb_null = by_group.get((None, "2024-02")) + assert feb_null is not None, ( + "the NULL-status group still receives NULL — the inner grain join " + f"did not match on a NULL dimension.\nrows: {resp.data}" + ) + assert feb_null == pytest.approx(NULL_STATUS_FEB_WINDOW), ( + f"NULL-status February window: expected {NULL_STATUS_FEB_WINDOW}, " + f"got {feb_null}.\nrows: {resp.data}" + ) + assert by_group.get((None, "2024-01")) == pytest.approx(NULL_STATUS_JAN), ( + f"NULL-status January window: expected {NULL_STATUS_JAN}.\n" + f"rows: {resp.data}" + ) + # Control: the non-NULL group was already correct and must stay correct. + assert by_group.get(("paid", "2024-02")) == pytest.approx(PAID_FEB_WINDOW), ( + f"paid February window regressed: expected {PAID_FEB_WINDOW}.\n" + f"rows: {resp.data}" + ) + assert by_group.get(("paid", "2024-01")) == pytest.approx(PAID_JAN), ( + f"paid January window regressed: expected {PAID_JAN}.\nrows: {resp.data}" + ) + + async def test_null_group_count_unchanged_by_the_fix( + self, exec_engine: SlayerQueryEngine, + ) -> None: + """B1 changes VALUES, never cardinality — the core invariant.""" + resp = await exec_engine.execute(_windowed_query()) + assert len(resp.data) == 4, ( + "expected one row per (status, month) group — 2 statuses x 2 months; " + f"got {len(resp.data)}:\n{resp.data}" + ) + + +# =========================================================================== # +# B2 — join-backs built directly as AST (no string re-parse). +# =========================================================================== # +class TestB2JoinBackBuiltAsAst: + """The dotted-alias corruption is invisible on dialects that do not mangle + dots, so these assert on BigQuery and T-SQL specifically (§5.13).""" + + @pytest.mark.parametrize("dialect", ["bigquery", "tsql"]) + async def test_cm_joinback_references_only_bound_tables( + self, dialect: str, + ) -> None: + """NEW (B2): the join-back's qualifiers are the CTE aliases actually in + scope (``_base`` and the ``_cm_*`` CTE) — not a mangled composite.""" + sql = await _gen(_cm_shared_grain_query(), dialect=dialect) + on = joinback_on_predicate_for(sql, prefix="_cm_", dialect=dialect) + qualifiers = { + col.table for col in + sqlglot.parse_one(on, dialect=dialect).find_all(exp.Column) + if col.table + } + bound = {"_base"} | { + n for n in qualifiers if n.startswith("_cm_") and "___" not in n + } + unbound = qualifiers - bound + assert not unbound, ( + f"[{dialect}] the join-back references table(s) that do not exist: " + f"{sorted(unbound)}. The dotted public alias was re-parsed as a " + f"multi-part reference.\nON: {on}\n\nfull SQL:\n{sql}" + ) + + @pytest.mark.parametrize("dialect", ["bigquery", "tsql"]) + async def test_cm_joinback_shape_is_scope_closed(self, dialect: str) -> None: + """The scope validator is the belt that caught this in the golden + baseline; it must now pass for these shapes.""" + sql = await _gen(_cm_shared_grain_query(), dialect=dialect) + assert_scope_closed(sql, dialect=dialect) + + @pytest.mark.parametrize("dialect", ["bigquery", "tsql"]) + async def test_wm_joinback_references_only_bound_tables( + self, dialect: str, + ) -> None: + """B2 covers the ``_wm_`` join-back too — it shares the same builder.""" + sql = await _gen(_windowed_chain_query(), dialect=dialect) + on = joinback_on_predicate_for(sql, prefix="_wm_", dialect=dialect) + qualifiers = { + col.table for col in + sqlglot.parse_one(on, dialect=dialect).find_all(exp.Column) + if col.table + } + unbound = { + q for q in qualifiers + if not (q == "_base" or (q.startswith("_wm_") and "___" not in q)) + } + assert not unbound, ( + f"[{dialect}] the _wm_ join-back references non-existent table(s): " + f"{sorted(unbound)}.\nON: {on}\n\nfull SQL:\n{sql}" + ) + + async def test_join_backs_no_longer_route_through_the_string_round_trip( + self, monkeypatch: pytest.MonkeyPatch, + ) -> None: + """NEW (B2), production-path proof: ``_null_safe_join_pair_sql`` is + retained (P-J state 1) but must no longer be REACHED by either + join-back. Poisoning it proves the call sites migrated — a grep cannot, + because the function stays in the file. + """ + from slayer.sql.generator import SQLGenerator + + def _poisoned(*args, **kwargs): # noqa: ANN002, ANN003 + raise AssertionError( + "_null_safe_join_pair_sql was called — a grain join-back is " + "still using the string re-parse round-trip (B2 incomplete)." + ) + + monkeypatch.setattr( + SQLGenerator, "_null_safe_join_pair_sql", _poisoned, raising=True, + ) + cm_sql = await _gen(_cm_shared_grain_query(), dialect="postgres") + assert "LEFT JOIN _cm_" in cm_sql, cm_sql + wm_sql = await _gen(_windowed_chain_query(), dialect="postgres") + assert "LEFT JOIN _wm_" in wm_sql, wm_sql + + +# =========================================================================== # +# §5.7 explicit semantics — zero-column, composite, and type-coerced grains. +# =========================================================================== # +class TestGrainSemantics: + + async def test_zero_column_grain_emits_cross_join_and_no_on_clause( + self, + ) -> None: + """A scalar CMA has an EMPTY grain: no join predicate exists to be + null-safe, and the shape stays a CROSS JOIN. Unchanged by B1/B2 — pinned + because the shared builder returns ``None`` for empty pairs and the + caller must keep turning that into a CROSS JOIN.""" + query = SlayerQuery( + source_model="orders_x", + measures=[ModelMeasure(formula="customers_v2.lifetime_value:sum")], + ) + sql = await _gen(query, dialect="postgres") + assert "CROSS JOIN _cm_" in _norm(sql), ( + f"an empty-grain cross-model aggregate must CROSS JOIN:\n{sql}" + ) + with pytest.raises(AssertionError): + joinback_on_predicate_for(sql, prefix="_cm_", dialect="postgres") + + async def test_zero_column_grain_executes_to_the_scalar_value( + self, exec_engine: SlayerQueryEngine, + ) -> None: + """The CROSS JOIN must not multiply rows: one row, the scalar total.""" + query = SlayerQuery( + source_model="orders", + measures=[ModelMeasure(formula="customers.spend:sum", name="total")], + ) + resp = await exec_engine.execute(query) + assert len(resp.data) == 1, f"scalar CMA must yield ONE row:\n{resp.data}" + # 1000.0 + 250.0 + 75.0 over the three seeded customers. + assert resp.data[0]["orders.total"] == pytest.approx(1325.0), resp.data + + async def test_composite_grain_conjoins_one_null_safe_pair_per_member( + self, + ) -> None: + """A two-member grain yields two null-safe comparisons ANDed together — + one per member, none of them a plain ``=``. + + Both members must be reachable from the TARGET: a host-local dimension + is deliberately excluded from the shared grain (it cannot be re-derived + inside the target-rooted CTE), so pairing one with a target dimension + would yield a single-member grain and prove nothing. + """ + query = SlayerQuery( + source_model="orders_x", + dimensions=[ + ColumnRef(name="customers_v2.status"), + ColumnRef(name="customers_v2.ltv_x2"), + ], + measures=[ModelMeasure(formula="customers_v2.lifetime_value:sum")], + ) + sql = await _gen(query, dialect="postgres") + on = _norm(joinback_on_predicate_for(sql, prefix="_cm_", dialect="postgres")) + assert on.count("IS NOT DISTINCT FROM") >= 2, ( + "a composite grain must emit one null-safe comparison per shared " + f"member:\n{on}" + ) + + async def test_time_truncated_grain_member_is_null_safe(self) -> None: + """A type-coerced grain member (a DATE_TRUNC bucket) joins back + null-safely like any other.""" + query = SlayerQuery( + source_model="orders_x", + time_dimensions=[TimeDimension( + dimension=ColumnRef(name="customers_v2.signup_at"), + granularity=TimeGranularity.MONTH, + )], + measures=[ModelMeasure(formula="customers_v2.lifetime_value:sum")], + ) + sql = await _gen(query, dialect="postgres") + on = _norm(joinback_on_predicate_for(sql, prefix="_cm_", dialect="postgres")) + assert "IS NOT DISTINCT FROM" in on, ( + f"a time-truncated grain member lost its null-safe join-back:\n{on}" + ) + + async def test_null_grain_cross_model_value_is_not_lost( + self, exec_engine: SlayerQueryEngine, + ) -> None: + """EXECUTED regression: the ``_cm_`` NULL-grain group keeps its real + aggregate (this join-back is already null-safe; B2 rebuilds how the + predicate is constructed and must not change what it means).""" + query = SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="customers.tier")], + measures=[ModelMeasure(formula="customers.spend:sum", name="spend")], + ) + resp = await exec_engine.execute(query) + by_tier = {r["orders.customers.tier"]: r["orders.spend"] for r in resp.data} + assert None in by_tier, ( + f"the NULL-tier group vanished from the result:\n{resp.data}" + ) + # customers 101 (250.0) and 102 (75.0) both have a NULL tier. + assert by_tier[None] == pytest.approx(325.0), ( + f"NULL-tier group lost its aggregate: {by_tier}\nrows: {resp.data}" + ) + + +# =========================================================================== # +# The shared builder itself (Codex D2) — expression operands, quoting, dots. +# =========================================================================== # +class TestSharedGrainJoinBackBuilder: + """Direct unit coverage of the one mechanism ``_cm_``/``_wm_``/sjoin share. + + The builder takes EXPRESSION operands (not alias strings) so a caller can + hand it an already-resolved, already-cast reference; the alias-column helper + covers today's three callers, which all compare projected aliases. + """ + + @staticmethod + def _import(): + import importlib + + return importlib.import_module(_BUILDER_MODULE) + + def test_builder_returns_none_for_an_empty_grain(self) -> None: + """Zero-column grain → no predicate at all; the caller emits CROSS JOIN. + Returning a truthy ``TRUE`` instead would silently turn every scalar CMA + into an inner-join-shaped ON clause.""" + mod = self._import() + assert mod.build_grain_joinback_condition( + pairs=[], dialect=get_dialect("postgres"), + ) is None + + @pytest.mark.parametrize( + "dialect,expected", + [ + ("postgres", "IS NOT DISTINCT FROM"), + ("duckdb", "IS NOT DISTINCT FROM"), + ("sqlite", " IS "), + ("mysql", "<=>"), + ("tsql", " OR "), + ("snowflake", "IS NOT DISTINCT FROM"), + ], + ) + def test_builder_emits_the_dialect_null_safe_form( + self, dialect: str, expected: str, + ) -> None: + """One builder, every dialect's own null-safe spelling — including the + expanded ``a = b OR (a IS NULL AND b IS NULL)`` fallback on T-SQL.""" + mod = self._import() + strategy = get_dialect(dialect) + left = mod.grain_alias_column(alias="orders.status", table="_base") + right = mod.grain_alias_column(alias="orders.status", table="_cm_x") + cond = mod.build_grain_joinback_condition( + pairs=[(left, right)], dialect=strategy, + ) + assert cond is not None + rendered = cond.sql(dialect=strategy.sqlglot_name) + assert expected in rendered, ( + f"[{dialect}] expected {expected!r} in {rendered!r}" + ) + + @pytest.mark.parametrize("dialect", ["bigquery", "tsql", "postgres", "mysql"]) + def test_dotted_alias_stays_one_identifier(self, dialect: str) -> None: + """The B2 defect in miniature: a dotted PUBLIC ALIAS is one identifier, + never a ``table.column`` reference. Built as AST it cannot decompose.""" + mod = self._import() + strategy = get_dialect(dialect) + left = mod.grain_alias_column(alias="orders.customers.status", table="_base") + cond = mod.build_grain_joinback_condition( + pairs=[(left, mod.grain_alias_column( + alias="orders.customers.status", table="_cm_x"))], + dialect=strategy, + ) + assert cond is not None + for col in cond.find_all(exp.Column): + assert col.table in ("_base", "_cm_x"), ( + f"[{dialect}] qualifier {col.table!r} is not one of the two " + f"CTE aliases — the dotted alias decomposed into a reference." + ) + assert col.name == "orders.customers.status", ( + f"[{dialect}] the dotted alias was split: {col.name!r}" + ) + + def test_alias_containing_a_quote_is_not_injectable(self) -> None: + """An embedded quote must survive as data inside one identifier.""" + mod = self._import() + weird = 'orders."evil' + col = mod.grain_alias_column(alias=weird, table="_base") + assert col.name == weird, col.name + rendered = col.sql(dialect="postgres") + assert rendered.startswith('_base.'), rendered + # Re-parsing must give back exactly one column with the same name. + reparsed = sqlglot.parse_one(f"SELECT {rendered}", dialect="postgres") + cols = list(reparsed.find_all(exp.Column)) + assert len(cols) == 1 and cols[0].name == weird, ( + f"identifier did not survive a round trip: {rendered!r} -> " + f"{[c.name for c in cols]}" + ) + + def test_case_sensitive_alias_is_quoted(self) -> None: + """Mixed-case aliases must stay quoted, or a case-folding dialect + resolves them to a different column.""" + mod = self._import() + col = mod.grain_alias_column(alias="Orders.Status", table="_base") + rendered = col.sql(dialect="postgres") + assert '"Orders.Status"' in rendered, rendered + + def test_composite_grain_ands_every_pair(self) -> None: + mod = self._import() + strategy = get_dialect("postgres") + pairs = [ + (mod.grain_alias_column(alias="a", table="_base"), + mod.grain_alias_column(alias="a", table="_cm_x")), + (mod.grain_alias_column(alias="b", table="_base"), + mod.grain_alias_column(alias="b", table="_cm_x")), + ] + cond = mod.build_grain_joinback_condition(pairs=pairs, dialect=strategy) + assert cond is not None + rendered = cond.sql(dialect="postgres") + assert rendered.count("IS NOT DISTINCT FROM") == 2, rendered + assert " AND " in rendered, rendered + + def test_builder_accepts_arbitrary_expression_operands(self) -> None: + """Codex D2: the core API takes expressions, so a caller can compare a + CAST or any resolved reference — not only a projected alias.""" + mod = self._import() + strategy = get_dialect("postgres") + left = exp.cast(exp.column("x", table="_base"), "DATE") + right = exp.column("y", table="_cm_x") + cond = mod.build_grain_joinback_condition( + pairs=[(left, right)], dialect=strategy, + ) + assert cond is not None + rendered = cond.sql(dialect="postgres") + assert "CAST(" in rendered and "IS NOT DISTINCT FROM" in rendered, rendered + + +# =========================================================================== # +# Dialect-emission coverage (§5.13) for what execution cannot reach. +# =========================================================================== # +class TestDialectEmission: + + @pytest.mark.parametrize("dialect", ["snowflake", "bigquery", "tsql", "mysql"]) + async def test_joinback_sql_parses_under_its_own_dialect( + self, dialect: str, + ) -> None: + sql = await _gen(_cm_shared_grain_query(), dialect=dialect) + parsed = sqlglot.parse(sql, dialect=dialect) + assert len(parsed) == 1, f"[{dialect}] did not parse to one statement:\n{sql}" + + async def test_snowflake_uses_a_native_null_safe_equality(self) -> None: + """§5.13 names Snowflake null-safe equality specifically.""" + sql = await _gen(_cm_shared_grain_query(), dialect="snowflake") + on = _norm(joinback_on_predicate_for(sql, prefix="_cm_", dialect="snowflake")) + assert "IS NOT DISTINCT FROM" in on or "EQUAL_NULL" in on, ( + f"snowflake join-back is not null-safe:\n{on}" + ) + + async def test_tsql_uses_the_expanded_fallback(self) -> None: + """T-SQL has no native null-safe operator: the expanded + ``a = b OR (a IS NULL AND b IS NULL)`` must appear.""" + sql = await _gen(_cm_shared_grain_query(), dialect="tsql") + on = _norm(joinback_on_predicate_for(sql, prefix="_cm_", dialect="tsql")) + assert " OR " in on and "IS NULL" in on, ( + f"tsql join-back is missing the expanded null-safe form:\n{on}" + ) + + async def test_mysql_null_safe_operator_is_emitted(self) -> None: + """MySQL's ``<=>`` comes from sqlglot's transposition of ``NullSafeEQ`` + rather than a dialect override — pinned so a future sqlglot change or a + well-meaning 'fix' to the base docstring cannot silently drop it.""" + sql = await _gen(_cm_shared_grain_query(), dialect="mysql") + on = _norm(joinback_on_predicate_for(sql, prefix="_cm_", dialect="mysql")) + assert "<=>" in on, f"mysql join-back lost its null-safe operator:\n{on}" diff --git a/tests/test_dev1746_pagination.py b/tests/test_dev1746_pagination.py new file mode 100644 index 00000000..01c84927 --- /dev/null +++ b/tests/test_dev1746_pagination.py @@ -0,0 +1,393 @@ +"""DEV-1746 §5.9 — pagination through the dialect strategy (B3). + +Today SLayer paginates three different ways depending on which render path a +query happens to take: + +* the plain single-model path sets ``limit``/``offset`` on the sqlglot + ``Select`` (correct everywhere — sqlglot transposes T-SQL's ``TOP`` / + ``OFFSET … FETCH``), +* the transform-chain paths hand the detached nodes to ``emit_outer_wrap`` + (also correct — ``TsqlDialect`` re-attaches them to a ``Select``), +* and the **cross-model combined path appends raw text**:: + + sql += f"\\nLIMIT {planned_query.limit}" + + which emits a literal ``LIMIT 10 OFFSET 5`` on SQL Server. So the *same* + cross-model query is valid T-SQL when it carries a transform layer and + invalid when it does not. + +B3 routes every path through one dialect-strategy hook. The T-SQL rule is +specified rather than discovered: limit-only becomes ``TOP``; an offset without +an ``ORDER BY`` gets a deterministic ``ORDER BY (SELECT NULL)`` before +``OFFSET … ROWS``, because SQL Server rejects ``OFFSET`` without ordering. +sqlglot 30.11 happens to do this itself, but that is *its* behaviour, not our +contract — these tests pin the contract so a sqlglot upgrade that drops it +fails here instead of at a customer's database. + +The matrix is {limit-only, offset-only, both} x {with, without ORDER BY} x +{tsql, bigquery, postgres, snowflake, sqlite} x {plain, outer-trim, combined}, +plus SQLite execution for all three shapes (Codex D5 — pagination changes row +sets, so parse-only coverage is not enough). +""" + +from __future__ import annotations + +import os +import re +import tempfile +from typing import AsyncIterator, Optional + +import pytest +import sqlglot +from sqlglot import exp + +from slayer.core.models import ModelMeasure +from slayer.core.query import ColumnRef, OrderItem, SlayerQuery +from slayer.engine.query_engine import SlayerQueryEngine +from slayer.sql.dialects import get_dialect + +from tests._dev1746_fixtures import ( + make_sqlite_engine, + outer_clause_sql, + outer_statement, + seed_dev1746_sqlite, +) +from tests._engine_helpers import _engine_generate + +DIALECTS = ["tsql", "bigquery", "postgres", "snowflake", "sqlite"] + +#: (limit, offset) — the three pagination combinations §5.9 names. +PAGINATION_COMBOS = [ + pytest.param(10, None, id="limit-only"), + pytest.param(None, 5, id="offset-only"), + pytest.param(10, 5, id="both"), +] + +#: A standalone ``LIMIT`` keyword — not a substring of an identifier. +_BARE_LIMIT = re.compile(r"(? SlayerQuery: + """Single-model aggregate — no isolation CTE, no hidden slot.""" + return SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ModelMeasure(formula="amount:sum", name="revenue")], + order=[OrderItem(column="amount:sum", direction="desc")] if ordered else [], + limit=limit, + offset=offset, + ) + + +def _outer_trim_query( + *, limit: Optional[int], offset: Optional[int], ordered: bool, +) -> SlayerQuery: + """Ordering by an aggregate that is NOT projected materialises a hidden + slot, so the generator wraps the base SELECT to trim it — the outer-trim + shape, whose pagination lands on the wrapper rather than the base.""" + return SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ModelMeasure(formula="*:count", name="n")], + order=[OrderItem(column="amount:sum", direction="desc")] if ordered else [], + limit=limit, + offset=offset, + ) + + +def _combined_query( + *, limit: Optional[int], offset: Optional[int], ordered: bool, +) -> SlayerQuery: + """Cross-model measure — the combined-SELECT path that appends raw text.""" + return SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="customers.tier")], + measures=[ModelMeasure(formula="customers.spend:sum", name="spend")], + order=( + [OrderItem(column="customers.spend:sum", direction="desc")] + if ordered else [] + ), + limit=limit, + offset=offset, + ) + + +SHAPES = { + "plain": _plain_query, + "outer_trim": _outer_trim_query, + "combined": _combined_query, +} + + +async def _gen_sql(query: SlayerQuery, *, dialect: str) -> str: + from tests._dev1746_fixtures import dev1746_models + + models = dev1746_models() + return await _engine_generate( + query=query, model=models[0], dialect=dialect, extra_models=models[1:], + ) + + +@pytest.fixture +async def exec_engine() -> AsyncIterator[SlayerQueryEngine]: + with tempfile.TemporaryDirectory() as d: + db_path = os.path.join(d, "dev1746.db") + seed_dev1746_sqlite(db_path) + yield await make_sqlite_engine(os.path.join(d, "store"), db_path) + + +# =========================================================================== # +# The §5.9 matrix. +# =========================================================================== # +class TestPaginationMatrix: + + @pytest.mark.parametrize("shape", sorted(SHAPES)) + @pytest.mark.parametrize("dialect", DIALECTS) + @pytest.mark.parametrize("limit,offset", PAGINATION_COMBOS) + @pytest.mark.parametrize("ordered", [True, False], ids=["ordered", "unordered"]) + async def test_emitted_sql_parses_under_its_own_dialect( + self, shape: str, dialect: str, limit, offset, ordered: bool, + ) -> None: + """Every cell of the matrix must emit SQL its own dialect can parse.""" + query = SHAPES[shape](limit=limit, offset=offset, ordered=ordered) + sql = await _gen_sql(query, dialect=dialect) + parsed = sqlglot.parse(sql, dialect=dialect) + assert len(parsed) == 1, ( + f"[{dialect}/{shape}] did not parse to a single statement:\n{sql}" + ) + + @pytest.mark.parametrize("shape", sorted(SHAPES)) + @pytest.mark.parametrize("limit,offset", PAGINATION_COMBOS) + @pytest.mark.parametrize("ordered", [True, False], ids=["ordered", "unordered"]) + async def test_tsql_never_emits_a_bare_limit_keyword( + self, shape: str, limit, offset, ordered: bool, + ) -> None: + """NEW (B3): SQL Server has no ``LIMIT``. The combined shape emits one + today; after B3 no shape does.""" + query = SHAPES[shape](limit=limit, offset=offset, ordered=ordered) + sql = await _gen_sql(query, dialect="tsql") + found = _BARE_LIMIT.search(sql) + assert found is None, ( + f"[tsql/{shape}] emitted a literal {found.group(0)!r}, which SQL " + f"Server rejects:\n{sql}" + ) + + @pytest.mark.parametrize("shape", sorted(SHAPES)) + @pytest.mark.parametrize("ordered", [True, False], ids=["ordered", "unordered"]) + async def test_tsql_limit_only_uses_top( + self, shape: str, ordered: bool, + ) -> None: + """The specified T-SQL rule, part 1: a limit with no offset is ``TOP``. + + Asserted on the OUTER statement rendered without its CTEs — a global + search could be satisfied by a ``TOP`` inside an inner scope. + """ + query = SHAPES[shape](limit=10, offset=None, ordered=ordered) + sql = await _gen_sql(query, dialect="tsql") + outer = outer_clause_sql(sql, dialect="tsql") + assert re.search(r"\bTOP\b", outer, re.IGNORECASE), ( + f"[tsql/{shape}] limit-only did not transpose to TOP on the outer " + f"statement.\nouter: {outer}\n\nfull SQL:\n{sql}" + ) + + @pytest.mark.parametrize("shape", sorted(SHAPES)) + @pytest.mark.parametrize("limit,offset", [(None, 5), (10, 5)]) + async def test_tsql_offset_always_carries_an_order_by( + self, shape: str, limit, offset, + ) -> None: + """The specified T-SQL rule, part 2: ``OFFSET`` requires an ``ORDER BY`` + **on the SELECT that carries the OFFSET**. + + The ordering is asserted on the outer statement's own AST node, not by + searching the text: a window function's ``OVER (ORDER BY …)``, a hidden + order slot, or an inner CTE would satisfy a global search while leaving + the paginated SELECT unordered — precisely the SQL Server error this + rule exists to prevent. + """ + query = SHAPES[shape](limit=limit, offset=offset, ordered=False) + sql = await _gen_sql(query, dialect="tsql") + outer = outer_statement(sql, dialect="tsql") + assert outer.args.get("order") is not None, ( + f"[tsql/{shape}] the paginated SELECT has no ORDER BY of its own — " + f"SQL Server rejects OFFSET without ordering:\n{sql}" + ) + rendered = outer_clause_sql(sql, dialect="tsql") + assert re.search(r"\bOFFSET\s+\d+\s+ROWS?\b", rendered, re.IGNORECASE), ( + f"[tsql/{shape}] OFFSET did not transpose to `OFFSET n ROWS`:\n" + f"{rendered}" + ) + + @pytest.mark.parametrize("shape", sorted(SHAPES)) + async def test_tsql_limit_with_offset_uses_fetch(self, shape: str) -> None: + """With an offset present the limit becomes ``FETCH … ROWS ONLY`` + (``TOP`` cannot express a window).""" + query = SHAPES[shape](limit=10, offset=5, ordered=True) + sql = await _gen_sql(query, dialect="tsql") + outer = outer_clause_sql(sql, dialect="tsql") + assert re.search(r"\bFETCH\b.*\bROWS?\s+ONLY\b", outer, re.IGNORECASE | re.S), ( + f"[tsql/{shape}] limit+offset did not transpose to FETCH on the " + f"outer statement:\n{outer}" + ) + + @pytest.mark.parametrize( + "dialect", [d for d in DIALECTS if d != "tsql"], + ) + @pytest.mark.parametrize("shape", sorted(SHAPES)) + async def test_non_tsql_dialects_keep_limit_offset( + self, dialect: str, shape: str, + ) -> None: + """Regression guard: routing through the hook must not change the + dialects that were already correct — and the bounds must land on the + OUTER statement, not on some inner scope.""" + query = SHAPES[shape](limit=10, offset=5, ordered=True) + sql = await _gen_sql(query, dialect=dialect) + outer = outer_statement(sql, dialect=dialect) + assert outer.args.get("limit") is not None, ( + f"[{dialect}/{shape}] the outer statement carries no LIMIT:\n{sql}" + ) + assert outer.args.get("offset") is not None, ( + f"[{dialect}/{shape}] the outer statement carries no OFFSET:\n{sql}" + ) + + +# =========================================================================== # +# The hook itself. +# =========================================================================== # +class TestApplyPaginationHook: + """``SqlDialect.apply_pagination`` is the single place pagination is + expressed, so it gets direct coverage independent of any query shape.""" + + @staticmethod + def _select() -> exp.Select: + return exp.Select().select(exp.column("a")).from_("t") + + def test_hook_exists_on_the_dialect_strategy(self) -> None: + strategy = get_dialect("postgres") + assert hasattr(strategy, "apply_pagination"), ( + "the pagination hook is missing — pagination still lives in the " + "generator rather than the dialect strategy (P-H)." + ) + + def test_no_pagination_is_a_no_op(self) -> None: + strategy = get_dialect("postgres") + out = strategy.apply_pagination(self._select(), limit=None, offset=None) + assert "LIMIT" not in out.sql(dialect="postgres").upper() + assert "OFFSET" not in out.sql(dialect="postgres").upper() + + @pytest.mark.parametrize("dialect", DIALECTS) + def test_hook_returns_a_select_that_renders_both_bounds( + self, dialect: str, + ) -> None: + strategy = get_dialect(dialect) + out = strategy.apply_pagination(self._select(), limit=10, offset=5) + assert isinstance(out, exp.Select), ( + f"[{dialect}] the hook must return a Select — T-SQL's TOP/FETCH " + f"transposition only fires when the nodes sit on a Select, never " + f"on a free-standing Limit." + ) + rendered = out.sql(dialect=dialect) + assert "10" in rendered and "5" in rendered, ( + f"[{dialect}] pagination bounds missing from {rendered!r}" + ) + + def test_tsql_hook_injects_ordering_for_a_bare_offset(self) -> None: + """The rule this PR specifies, at the unit level.""" + strategy = get_dialect("tsql") + out = strategy.apply_pagination(self._select(), limit=None, offset=5) + rendered = out.sql(dialect="tsql") + assert re.search(r"\bORDER\s+BY\b", rendered, re.IGNORECASE), ( + f"tsql OFFSET emitted without ORDER BY: {rendered!r}" + ) + assert _BARE_LIMIT.search(rendered) is None, rendered + + def test_tsql_hook_preserves_a_user_order_by(self) -> None: + """The injected ordering is a fallback — never an override.""" + strategy = get_dialect("tsql") + select = self._select().order_by("a") + rendered = strategy.apply_pagination( + select, limit=None, offset=5, + ).sql(dialect="tsql") + assert "ORDER BY" in rendered.upper(), rendered + assert "SELECT NULL" not in rendered.upper(), ( + f"the user's ORDER BY was replaced by the fallback: {rendered!r}" + ) + + +# =========================================================================== # +# Execution — pagination changes row sets, so parse-only is not enough (D5). +# =========================================================================== # +class TestPaginationExecution: + """Seeded groups: ``paid`` sums to 30.0, the NULL-status group to 12.0 — + distinct, so ordering and slicing are unambiguous.""" + + async def test_plain_shape_limit_and_offset( + self, exec_engine: SlayerQueryEngine, + ) -> None: + top = await exec_engine.execute( + _plain_query(limit=1, offset=None, ordered=True), + ) + assert [r["orders.status"] for r in top.data] == ["paid"], top.data + assert top.data[0]["orders.revenue"] == pytest.approx(30.0), top.data + + second = await exec_engine.execute( + _plain_query(limit=1, offset=1, ordered=True), + ) + assert [r["orders.status"] for r in second.data] == [None], second.data + assert second.data[0]["orders.revenue"] == pytest.approx(12.0), second.data + + async def test_outer_trim_shape_limit_and_offset( + self, exec_engine: SlayerQueryEngine, + ) -> None: + """Pagination on the trim wrapper must slice the same ordering the + hidden aggregate defines, and must not resurrect the hidden column.""" + top = await exec_engine.execute( + _outer_trim_query(limit=1, offset=None, ordered=True), + ) + assert [r["orders.status"] for r in top.data] == ["paid"], top.data + assert all("amount_sum" not in k for k in top.data[0]), ( + f"the hidden order slot leaked into the response: {top.data[0]}" + ) + + second = await exec_engine.execute( + _outer_trim_query(limit=1, offset=1, ordered=True), + ) + assert [r["orders.status"] for r in second.data] == [None], second.data + + async def test_combined_shape_limit_and_offset( + self, exec_engine: SlayerQueryEngine, + ) -> None: + """NEW (B3), EXECUTED: the cross-model combined shape paginates.""" + top = await exec_engine.execute( + _combined_query(limit=1, offset=None, ordered=True), + ) + assert len(top.data) == 1, f"LIMIT 1 returned {len(top.data)} rows:\n{top.data}" + assert top.data[0]["orders.customers.tier"] == "gold", top.data + assert top.data[0]["orders.spend"] == pytest.approx(1000.0), top.data + + second = await exec_engine.execute( + _combined_query(limit=1, offset=1, ordered=True), + ) + assert len(second.data) == 1, second.data + assert second.data[0]["orders.customers.tier"] is None, second.data + assert second.data[0]["orders.spend"] == pytest.approx(325.0), second.data + + async def test_offset_past_the_end_returns_no_rows( + self, exec_engine: SlayerQueryEngine, + ) -> None: + resp = await exec_engine.execute( + _combined_query(limit=10, offset=50, ordered=True), + ) + assert resp.data == [], resp.data + + async def test_limit_without_offset_on_the_combined_shape( + self, exec_engine: SlayerQueryEngine, + ) -> None: + resp = await exec_engine.execute( + _combined_query(limit=2, offset=None, ordered=True), + ) + assert len(resp.data) == 2, resp.data diff --git a/tests/test_dev1746_projection_order.py b/tests/test_dev1746_projection_order.py new file mode 100644 index 00000000..ec189c65 --- /dev/null +++ b/tests/test_dev1746_projection_order.py @@ -0,0 +1,844 @@ +"""DEV-1746 §5.2 — one ordered public projection (B7, B8, B11). + +``PlannedQuery.projection`` already exists, is already in declaration order, and +already excludes hidden slots. What is missing is that **no renderer consumes +it**: the cross-model combined SELECT rebuilds an order out of four separate +grouped passes (host slots, then outer composites, then ``_cm_``, then ``_wm_``), +which is why a cross-model measure declared FIRST is emitted LAST. The generator +even admits it in a comment — "windowed columns are grouped after the ``_base`` +projection rather than woven into ``planned_query.projection`` order". B7 makes +every renderer walk the plan's list verbatim. + +Trimming hidden slots then stops being a mechanism at all: a hidden slot is +simply absent from ``projection``, which is what replaces the five bespoke +variants. One renderer-side assertion is kept as a belt because pydantic's +``model_copy(update=…)`` skips validation and rerooted plans use it (Codex D9). + +**B8** — eight inner-stage projection sites carry their aliases as +``sorted(aliases)``; one still carries the comment "matches legacy +``_generate_with_computed:1607``", i.e. it is byte-parity ballast. They become +plan-ordered. The tests name measures so that alphabetical order and declaration +order DISAGREE (``zz_`` declared before ``aa_``), because a test whose two +candidate orders coincide proves nothing. Each site also fails closed: the new +ordered list must contain exactly the aliases the old flattening did, so a +reorder can never silently drop one (Codex D8). + +**B11** — the base FROM's join order comes from a two-tier merge that exists to +reproduce legacy bytes ("→ byte-identical FROM"); it becomes the same +first-seen registration order the per-CTE path already uses. B11 is ratified as +**execution-identical**, so what is asserted here is the part that can regress: +the join SET is unchanged, every join still appears (including one discovered +late, from a filter rather than a projection), generation is deterministic, and +the results are right. The precise emitted ORDER is deliberately not pinned to a +literal here — both the old and new orders are deterministic and +execution-equivalent, so a literal pin would assert an implementation detail +rather than a contract; the order change itself is surfaced through the PR's +recompare churn list (Codex D6). +""" + +from __future__ import annotations + +import os +import tempfile +from typing import AsyncIterator, List + +import pytest + +from slayer.core.enums import TimeGranularity +from slayer.core.models import ModelMeasure +from slayer.core.query import ColumnRef, OrderItem, SlayerQuery, TimeDimension +from slayer.engine.planned import PlannedQuery, ValueSlot +from slayer.engine.query_engine import SlayerQueryEngine +from slayer.engine.source_bundle import ResolvedSourceBundle +from slayer.engine.stage_planner import plan_query +from slayer.sql.generator import SQLGenerator +from slayer.sql.scope import ScopeFrame + +from tests._cross_model_chain import ( + _countries, + _customers_v2, + _gen, + _orders_x, + _regions, +) +from tests._dev1746_fixtures import ( + base_cte_join_sequence, + carried_alias_drops, + carry_list_order_violations, + make_sqlite_engine, + outer_select_aliases, + seed_dev1746_sqlite, +) +from tests._engine_helpers import _engine_generate, _join_aliases + + +def _chain_bundle() -> ResolvedSourceBundle: + return ResolvedSourceBundle( + source_model=_orders_x(), + referenced_models=[_customers_v2(), _regions(), _countries()], + ) + + +@pytest.fixture +async def exec_engine() -> AsyncIterator[SlayerQueryEngine]: + with tempfile.TemporaryDirectory() as d: + db_path = os.path.join(d, "dev1746.db") + seed_dev1746_sqlite(db_path) + yield await make_sqlite_engine(os.path.join(d, "store"), db_path) + + +# --------------------------------------------------------------------------- # +# Shapes whose declaration order and current emitted order disagree. +# --------------------------------------------------------------------------- # +def _cm_declared_first_query() -> SlayerQuery: + """A cross-model measure declared BEFORE a local one. Emitted today as + ``status, local_second, cm_first``; B7 makes it ``status, cm_first, + local_second``.""" + return SlayerQuery( + source_model="orders_x", + dimensions=[ColumnRef(name="status")], + measures=[ + ModelMeasure(formula="customers_v2.lifetime_value:sum", name="cm_first"), + ModelMeasure(formula="amount:sum", name="local_second"), + ], + ) + + +def _windowed_declared_first_query() -> SlayerQuery: + """A windowed measure declared BEFORE a local one.""" + return SlayerQuery( + source_model="orders_x", + dimensions=[ColumnRef(name="status")], + time_dimensions=[TimeDimension( + dimension=ColumnRef(name="created_at"), + granularity=TimeGranularity.MONTH, + )], + measures=[ + ModelMeasure(formula="amount:sum(window='90d')", name="w_first"), + ModelMeasure(formula="amount:sum", name="local_second"), + ], + ) + + +def _interleaved_query() -> SlayerQuery: + """local, cross-model, local — an order no grouped-pass scheme can produce, + because any such scheme emits all host slots before all ``_cm_`` ones.""" + return SlayerQuery( + source_model="orders_x", + dimensions=[ColumnRef(name="status")], + measures=[ + ModelMeasure(formula="amount:sum", name="a_local"), + ModelMeasure(formula="customers_v2.lifetime_value:sum", name="b_cross"), + ModelMeasure(formula="amount:avg", name="c_local"), + ], + ) + + +def _all_slots(planned: PlannedQuery) -> List[ValueSlot]: + """Every slot a projection id can point at. + + A transform (``cumsum(...)``) is a ``combined_expression_slot``, not a row + or aggregate slot, so a lookup over only those two misses it. + """ + return ( + list(planned.row_slots) + + list(planned.aggregate_slots) + + list(planned.combined_expression_slots) + ) + + +def _expected_projection_aliases(query: SlayerQuery) -> List[str]: + """The public aliases the plan declares, in plan order. + + Derived from ``PlannedQuery.projection`` rather than hard-coded, so this + expresses the actual contract ("renderers consume the plan's list verbatim") + instead of a guess about how aliases are spelled. + """ + planned = plan_query(query=query, bundle=_chain_bundle()) + slots = {s.id: s for s in _all_slots(planned)} + out: List[str] = [] + for sid in planned.projection: + slot = slots[sid] + assert not slot.hidden, ( + f"hidden slot {sid} appeared in PlannedQuery.projection — the " + f"public projection must contain only public slots." + ) + out.append(slot.public_aliases[0] if slot.public_aliases else slot.public_name) + return out + + +def _alias_suffixes(aliases: List[str]) -> List[str]: + """Emitted aliases are host-rooted (``orders_x.status``); the plan names the + trailing public part. Compare on that.""" + return [a.split(".")[-1] for a in aliases] + + +# =========================================================================== # +# B7 — declaration-order projection. +# =========================================================================== # +class TestB7DeclarationOrderProjection: + + async def test_cross_model_measure_keeps_its_declared_position(self) -> None: + """NEW (B7): a cross-model measure declared first is emitted first.""" + sql = await _gen(_cm_declared_first_query(), dialect="postgres") + emitted = _alias_suffixes(outer_select_aliases(sql)) + assert emitted == ["status", "cm_first", "local_second"], ( + f"emitted projection order {emitted} is not declaration order — the " + f"cross-model measure was grouped after the host slots.\n\n{sql}" + ) + + async def test_windowed_measure_keeps_its_declared_position(self) -> None: + """NEW (B7): the same for a windowed measure.""" + sql = await _gen(_windowed_declared_first_query(), dialect="postgres") + emitted = _alias_suffixes(outer_select_aliases(sql)) + assert emitted == ["status", "created_at", "w_first", "local_second"], ( + f"emitted projection order {emitted} is not declaration order.\n\n{sql}" + ) + + async def test_interleaved_local_and_cross_model_measures(self) -> None: + """The decisive shape: local, cross-model, local. No grouped-pass scheme + can emit this order — only walking the plan's list can.""" + sql = await _gen(_interleaved_query(), dialect="postgres") + emitted = _alias_suffixes(outer_select_aliases(sql)) + assert emitted == ["status", "a_local", "b_cross", "c_local"], ( + f"emitted {emitted}; a cross-model measure cannot be woven between " + f"two host measures unless the renderer consumes the plan's " + f"ordered projection.\n\n{sql}" + ) + + @pytest.mark.parametrize( + "query_factory", + [_cm_declared_first_query, _windowed_declared_first_query, _interleaved_query], + ids=["cross_model", "windowed", "interleaved"], + ) + async def test_emitted_order_equals_the_plans_projection_order( + self, query_factory, + ) -> None: + """The contract itself, stated once: emitted order IS plan order.""" + query = query_factory() + sql = await _gen(query, dialect="postgres") + emitted = _alias_suffixes(outer_select_aliases(sql)) + expected = _alias_suffixes(_expected_projection_aliases(query)) + assert emitted == expected, ( + f"renderer did not consume PlannedQuery.projection verbatim:\n" + f" emitted: {emitted}\n plan: {expected}\n\n{sql}" + ) + + async def test_response_column_order_follows_declaration_order( + self, exec_engine: SlayerQueryEngine, + ) -> None: + """B7 is observable to callers: ``response.columns`` is read off the + emitted outer SELECT, so its order changes with the projection.""" + query = SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ + ModelMeasure(formula="customers.spend:sum", name="cm_first"), + ModelMeasure(formula="amount:sum", name="local_second"), + ], + ) + resp = await exec_engine.execute(query) + assert _alias_suffixes(list(resp.columns)) == [ + "status", "cm_first", "local_second", + ], f"response column order: {resp.columns}" + # Row keys follow the same order. + assert _alias_suffixes(list(resp.data[0].keys())) == [ + "status", "cm_first", "local_second", + ], f"row keys: {list(resp.data[0].keys())}" + + async def test_hidden_slots_are_absent_from_the_projection(self) -> None: + """Unified trimming: an order-only aggregate never reaches the public + projection because it is not in ``projection`` at all.""" + query = SlayerQuery( + source_model="orders_x", + dimensions=[ColumnRef(name="status")], + measures=[ModelMeasure(formula="*:count", name="n")], + order=[OrderItem(column="amount:sum", direction="desc")], + ) + sql = await _gen(query, dialect="postgres") + emitted = _alias_suffixes(outer_select_aliases(sql)) + assert emitted == ["status", "n"], ( + f"the hidden order-only aggregate leaked into the public " + f"projection: {emitted}\n\n{sql}" + ) + + async def test_hidden_cross_model_order_slot_is_trimmed_but_orderable( + self, exec_engine: SlayerQueryEngine, + ) -> None: + """A hidden CROSS-MODEL aggregate must still drive ORDER BY while being + absent from the projection — the case the ``trim_hidden`` flag handled + and which unified trimming must preserve.""" + query = SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="customers.tier")], + measures=[ModelMeasure(formula="*:count", name="n")], + order=[OrderItem(column="customers.spend:sum", direction="desc")], + ) + resp = await exec_engine.execute(query) + assert _alias_suffixes(list(resp.columns)) == ["tier", "n"], resp.columns + # gold (1000.0) outranks the NULL tier group (325.0). + assert [r["orders.customers.tier"] for r in resp.data] == ["gold", None], ( + f"hidden cross-model order slot did not drive the ordering: {resp.data}" + ) + + +# =========================================================================== # +# The projection invariant + its belt. +# =========================================================================== # +class TestProjectionInvariant: + + def test_a_hidden_slot_in_the_projection_is_rejected(self) -> None: + """Validated at construction: the public projection is public-only.""" + planned = plan_query( + query=_cm_declared_first_query(), bundle=_chain_bundle(), + ) + hidden = [ + s for s in list(planned.row_slots) + list(planned.aggregate_slots) + if s.hidden + ] + if not hidden: + # Synthesise one so the invariant is tested even when this shape + # happens to have no hidden slot. + slot = planned.aggregate_slots[0].model_copy( + update={"id": "hidden_x", "hidden": True, + "public_name": None, "public_aliases": []}, + ) + fields = dict(planned.__dict__) + fields["aggregate_slots"] = list(planned.aggregate_slots) + [slot] + fields["projection"] = list(planned.projection) + ["hidden_x"] + else: + fields = dict(planned.__dict__) + fields["projection"] = list(planned.projection) + [hidden[0].id] + with pytest.raises(ValueError, match="(?i)hidden"): + PlannedQuery(**fields) + + def test_a_duplicated_slot_in_the_projection_is_rejected(self) -> None: + """A duplicate would emit the same column twice and corrupt the + positional contract callers read ``columns`` by.""" + planned = plan_query( + query=_cm_declared_first_query(), bundle=_chain_bundle(), + ) + fields = dict(planned.__dict__) + fields["projection"] = list(planned.projection) + [planned.projection[0]] + with pytest.raises(ValueError, match="(?i)duplicate"): + PlannedQuery(**fields) + + def test_renderer_belt_catches_a_model_copy_that_skips_validation( + self, + ) -> None: + """Codex D9: ``model_copy(update=…)`` bypasses validators, and rerooting + uses it — so exactly one renderer-side assertion is kept. It must RAISE, + never silently skip the offending slot.""" + planned = plan_query( + query=_cm_declared_first_query(), bundle=_chain_bundle(), + ) + slots = { + s.id: s for s in list(planned.row_slots) + list(planned.aggregate_slots) + } + victim = planned.projection[0] + broken_slot = slots[victim].model_copy( + update={"hidden": True, "public_name": None, "public_aliases": []}, + ) + # Replace the slot ONLY in the collection that owns it. Putting it in + # both would additionally corrupt slot classification, so the belt + # could then fire for an unrelated reason and the test would pass + # without proving anything about hidden-slot detection. + update = {} + if any(s.id == victim for s in planned.row_slots): + update["row_slots"] = [ + broken_slot if s.id == victim else s for s in planned.row_slots + ] + else: + update["aggregate_slots"] = [ + broken_slot if s.id == victim else s + for s in planned.aggregate_slots + ] + corrupted = planned.model_copy(update=update) + gen = SQLGenerator(dialect="postgres") + with pytest.raises((AssertionError, ValueError)) as excinfo: + gen.generate_from_planned( + planned_query=corrupted, bundle=_chain_bundle(), + ) + message = str(excinfo.value).lower() + assert "hidden" in message, ( + "the belt fired, but not for the hidden slot — the message does not " + f"mention it, so this may be an unrelated failure: {excinfo.value!r}" + ) + + +# =========================================================================== # +# The ``_wm_`` grain invariant (Codex D1). +# =========================================================================== # +class TestWindowedGrainInvariant: + + def test_windowed_plan_grain_always_contains_the_window_time_dimension( + self, + ) -> None: + """Why the ``_wm_`` join-back can never be an empty-grain CROSS JOIN: + the planner always includes the window's time dimension in the grain. + Pinned at plan level so the render path does not need a special case.""" + planned = plan_query( + query=_windowed_declared_first_query(), bundle=_chain_bundle(), + ) + assert planned.windowed_aggregate_plans, "expected a windowed plan" + for wm in planned.windowed_aggregate_plans: + assert wm.grain_slot_ids, ( + "a windowed plan has an EMPTY grain — the outer join-back would " + "degenerate to a CROSS JOIN and multiply rows." + ) + assert wm.window_time_dimension_slot_id in wm.grain_slot_ids, ( + f"the window time dimension {wm.window_time_dimension_slot_id} " + f"is not part of the grain {wm.grain_slot_ids}." + ) + + +# =========================================================================== # +# B8 — inner-stage projections in plan order. +# =========================================================================== # +class TestB8InnerStagePlanOrder: + """Measures are named so alphabetical and declaration order DISAGREE: + ``zz_running`` is declared before ``aa_total``.""" + + def _transform_query(self) -> SlayerQuery: + return SlayerQuery( + source_model="orders_x", + time_dimensions=[TimeDimension( + dimension=ColumnRef(name="created_at"), + granularity=TimeGranularity.MONTH, + )], + measures=[ + ModelMeasure(formula="cumsum(amount:sum)", name="zz_running"), + ModelMeasure(formula="amount:avg", name="aa_total"), + ], + ) + + def _cross_model_transform_query(self) -> SlayerQuery: + return SlayerQuery( + source_model="orders_x", + time_dimensions=[TimeDimension( + dimension=ColumnRef(name="created_at"), + granularity=TimeGranularity.MONTH, + )], + measures=[ + ModelMeasure(formula="cumsum(amount:sum)", name="zz_running"), + ModelMeasure( + formula="customers_v2.lifetime_value:sum", name="aa_cross", + ), + ], + ) + + def _time_shift_query(self) -> SlayerQuery: + return SlayerQuery( + source_model="orders_x", + time_dimensions=[TimeDimension( + dimension=ColumnRef(name="created_at"), + granularity=TimeGranularity.MONTH, + )], + measures=[ + ModelMeasure(formula="time_shift(amount:sum, -1)", name="zz_prev"), + ModelMeasure(formula="amount:avg", name="aa_total"), + ], + ) + + def _consecutive_periods_query(self) -> SlayerQuery: + return SlayerQuery( + source_model="orders_x", + time_dimensions=[TimeDimension( + dimension=ColumnRef(name="created_at"), + granularity=TimeGranularity.MONTH, + )], + measures=[ + ModelMeasure( + formula="consecutive_periods(amount:sum > 0)", name="zz_streak", + ), + ModelMeasure(formula="amount:avg", name="aa_total"), + ], + ) + + @pytest.mark.parametrize( + "factory_name", + [ + "_transform_query", + "_cross_model_transform_query", + "_time_shift_query", + "_consecutive_periods_query", + ], + ) + async def test_carried_aliases_follow_base_order_not_alphabetical( + self, factory_name: str, + ) -> None: + """NEW (B8): every downstream stage carries its aliases in the order the + base stage projects them (plan order), not ``sorted()``. + + The measures are named so the two orders disagree: the base projects + ``created_at, aa_total, amount_sum`` while ``sorted()`` yields + ``aa_total, amount_sum, created_at``. + """ + query = getattr(self, factory_name)() + sql = await _gen(query, dialect="postgres") + violations = carry_list_order_violations(sql) + assert not violations, ( + f"[{factory_name}] inner stage(s) carry aliases alphabetically " + f"instead of in plan order:\n " + "\n ".join(violations) + + f"\n\n{sql}" + ) + + @pytest.mark.parametrize( + "factory_name", + [ + "_transform_query", + "_cross_model_transform_query", + "_time_shift_query", + "_consecutive_periods_query", + ], + ) + async def test_no_carried_alias_is_dropped(self, factory_name: str) -> None: + """Codex D8 — fail closed, at BOTH ends. + + Comparing only the plan against the outermost SELECT would miss the + failure mode that matters: an intermediate stage dropping a hidden + input, a time key, or an aggregate operand that a LATER stage still + references. So this asserts (a) every public alias survives to the + projection, and (b) no stage omits an alias the next stage reads. + """ + query = getattr(self, factory_name)() + sql = await _gen(query, dialect="postgres") + expected = set(_alias_suffixes(_expected_projection_aliases(query))) + emitted = set(_alias_suffixes(outer_select_aliases(sql))) + assert expected <= emitted, ( + f"[{factory_name}] aliases lost between plan and projection: " + f"{sorted(expected - emitted)}\n\n{sql}" + ) + drops = carried_alias_drops(sql) + assert not drops, ( + f"[{factory_name}] an inner stage dropped an alias a later stage " + f"still references:\n " + "\n ".join(drops) + f"\n\n{sql}" + ) + + async def test_local_transform_chain_executes_with_plan_ordered_stages( + self, exec_engine: SlayerQueryEngine, + ) -> None: + """EXECUTED (Codex D5): reordering inner-stage projections must not + change any value. Monthly totals are 15.0 (10+5) and 27.0 (20+7), so + the running total is 15.0 then 42.0.""" + query = SlayerQuery( + source_model="orders", + time_dimensions=[TimeDimension( + dimension=ColumnRef(name="created_at"), + granularity=TimeGranularity.MONTH, + )], + measures=[ + ModelMeasure(formula="cumsum(amount:sum)", name="zz_running"), + ModelMeasure(formula="amount:sum", name="aa_total"), + ], + ) + resp = await exec_engine.execute(query) + assert [r["orders.zz_running"] for r in resp.data] == pytest.approx( + [15.0, 42.0], + ), f"cumulative sum changed under the reorder: {resp.data}" + assert [r["orders.aa_total"] for r in resp.data] == pytest.approx( + [15.0, 27.0], + ), resp.data + + async def test_cross_model_transform_chain_executes( + self, exec_engine: SlayerQueryEngine, + ) -> None: + """EXECUTED (Codex D5), family 2 of 4: a transform chain that also + carries a cross-model measure — the chain whose WITH assembly this PR + rebuilds.""" + query = SlayerQuery( + source_model="orders", + time_dimensions=[TimeDimension( + dimension=ColumnRef(name="created_at"), + granularity=TimeGranularity.MONTH, + )], + measures=[ + ModelMeasure(formula="cumsum(amount:sum)", name="zz_running"), + ModelMeasure(formula="customers.spend:sum", name="aa_cross"), + ], + ) + resp = await exec_engine.execute(query) + assert [r["orders.zz_running"] for r in resp.data] == pytest.approx( + [15.0, 42.0], + ), resp.data + # The scalar cross-model total is the same on every row (no shared grain). + assert [r["orders.aa_cross"] for r in resp.data] == pytest.approx( + [1325.0, 1325.0], + ), resp.data + + async def test_time_shift_chain_executes( + self, exec_engine: SlayerQueryEngine, + ) -> None: + """EXECUTED (Codex D5), family 3 of 4: the ``sjoin_`` carry list.""" + query = SlayerQuery( + source_model="orders", + time_dimensions=[TimeDimension( + dimension=ColumnRef(name="created_at"), + granularity=TimeGranularity.MONTH, + )], + measures=[ + ModelMeasure(formula="time_shift(amount:sum, -1)", name="zz_prev"), + ModelMeasure(formula="amount:sum", name="aa_total"), + ], + ) + resp = await exec_engine.execute(query) + by_month = { + str(r["orders.created_at"])[:7]: ( + r["orders.aa_total"], r["orders.zz_prev"], + ) + for r in resp.data + } + assert by_month["2024-01"][0] == pytest.approx(15.0), resp.data + assert by_month["2024-02"][0] == pytest.approx(27.0), resp.data + # February's previous month is January's total; January has no + # predecessor in the seeded data. + assert by_month["2024-02"][1] == pytest.approx(15.0), ( + f"time_shift did not read the previous month: {resp.data}" + ) + assert by_month["2024-01"][1] is None, resp.data + + async def test_consecutive_periods_chain_executes( + self, exec_engine: SlayerQueryEngine, + ) -> None: + """EXECUTED (Codex D5), family 4 of 4: the ``cp_reset_`` carry list. + Both seeded months are positive, so the streak runs 1 then 2.""" + query = SlayerQuery( + source_model="orders", + time_dimensions=[TimeDimension( + dimension=ColumnRef(name="created_at"), + granularity=TimeGranularity.MONTH, + )], + measures=[ + ModelMeasure( + formula="consecutive_periods(amount:sum > 0)", name="zz_streak", + ), + ModelMeasure(formula="amount:sum", name="aa_total"), + ], + ) + resp = await exec_engine.execute(query) + assert [r["orders.zz_streak"] for r in resp.data] == [1, 2], ( + f"consecutive-period streak changed under the reorder: {resp.data}" + ) + + +# =========================================================================== # +# B11 — one deterministic FROM-join ordering mechanism. +# =========================================================================== # +class TestB11JoinOrdering: + + async def test_join_set_is_unchanged_for_a_projected_join(self) -> None: + sql = await _gen( + SlayerQuery( + source_model="orders_x", + dimensions=[ColumnRef(name="customers_v2.status")], + measures=[ModelMeasure(formula="amount:sum")], + ), + dialect="postgres", + ) + assert _join_aliases(sql) == {"customers_v2"}, _join_aliases(sql) + + async def test_join_discovered_only_by_a_filter_is_still_emitted(self) -> None: + """The late-registration case (Codex D6): a join that no projection + mentions — it is discovered while rendering the filter — must survive + the switch to registration-order collection.""" + sql = await _gen( + SlayerQuery( + source_model="orders_x", + dimensions=[ColumnRef(name="status")], + measures=[ModelMeasure(formula="amount:sum")], + filters=["customers_v2.status == 'active'"], + ), + dialect="postgres", + ) + assert "customers_v2" in _join_aliases(sql), ( + f"the filter-only join was dropped:\n{sql}" + ) + + async def test_filter_on_a_joined_derived_column_pulls_the_deeper_hop( + self, + ) -> None: + """Regression: filtering on a JOINED model's crossing derived column. + + ``customers_v2.deep_pop`` is declared on the joined model as + ``regions.population``. Both the filter renderer and the join scanner + expanded it as if ``customers_v2`` were the query root, so the ref came + out bare (``regions.population``) — which the scanner could not match to + a join path, leaving the hop unjoined and the filter pointing at a table + that is not in the FROM:: + + FROM orders AS orders_x LEFT JOIN customers AS customers_v2 ... + WHERE regions.population > 0 -- no such table + + Both sites now expand with ``is_root=False``, so the ref resolves to the + full path alias and the scanner registers the hop. The two must stay in + lockstep: discovery scans the SAME expansion the renderer emits. + """ + sql = await _gen( + SlayerQuery( + source_model="orders_x", + dimensions=[ColumnRef(name="status")], + measures=[ModelMeasure(formula="amount:sum")], + filters=["customers_v2.deep_pop > 0"], + ), + dialect="postgres", + ) + assert "customers_v2__regions" in _join_aliases(sql), ( + f"the deeper hop was never joined:\n{sql}" + ) + assert "customers_v2__regions.population" in sql, ( + f"the derived ref was not qualified to its path alias:\n{sql}" + ) + + async def test_multi_hop_joins_emit_parent_before_child(self) -> None: + """A three-hop path must emit every hop, each after its parent — the + structural requirement ANY ordering must satisfy, asserted over the + full emitted sequence rather than one adjacent pair.""" + sql = await _gen( + SlayerQuery( + source_model="orders_x", + dimensions=[ColumnRef(name="customers_v2.deep_gdp")], + measures=[ModelMeasure(formula="amount:sum")], + ), + dialect="postgres", + ) + expected = [ + "customers_v2", + "customers_v2__regions", + "customers_v2__regions__countries", + ] + assert set(expected) <= _join_aliases(sql), _join_aliases(sql) + sequence = base_cte_join_sequence(sql) + positions = [sequence.index(name) for name in expected] + assert positions == sorted(positions), ( + f"hops are not emitted parent-before-child: {sequence}\n\n{sql}" + ) + + async def test_generation_is_deterministic(self) -> None: + """Whatever the order is, it must be the SAME order every run — a set + would make emitted SQL vary between processes.""" + query = SlayerQuery( + source_model="orders_x", + dimensions=[ + ColumnRef(name="customers_v2.status"), + ColumnRef(name="customers_v2.deep_pop"), + ], + measures=[ModelMeasure(formula="amount:sum")], + filters=["customers_v2.status != 'x'"], + ) + first = await _gen(query, dialect="postgres") + second = await _gen(query, dialect="postgres") + assert first == second, "emitted SQL is not deterministic across runs" + + async def test_mixed_projection_and_filter_joins_all_present(self) -> None: + """The shape where the two tiers of the old merge differ: the two-hop + ``regions`` join arrives via a projected dimension while the one-hop + ``customers_v2`` join is also named by a filter. Both must be emitted + exactly once, whichever tier discovered them.""" + sql = await _gen( + SlayerQuery( + source_model="orders_x", + dimensions=[ColumnRef(name="customers_v2.deep_pop")], + measures=[ModelMeasure(formula="amount:sum")], + filters=["customers_v2.status != 'x'"], + ), + dialect="postgres", + ) + assert {"customers_v2", "customers_v2__regions"} <= _join_aliases(sql), sql + assert sql.count("LEFT JOIN regions AS customers_v2__regions") == 1, ( + f"the regions hop was emitted more than once:\n{sql}" + ) + + async def test_executed_multi_join_query_is_correct( + self, exec_engine: SlayerQueryEngine, + ) -> None: + """EXECUTED (Codex D5): join reordering is only safe if the rows are + unchanged. Region 2's name is NULL, so this also covers a nullable + two-hop dimension.""" + query = SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="customers.regions.name")], + measures=[ModelMeasure(formula="amount:sum", name="revenue")], + ) + resp = await exec_engine.execute(query) + by_region = { + r["orders.customers.regions.name"]: r["orders.revenue"] + for r in resp.data + } + # Orders 1,2 -> customer 100 -> region 1 ("West"): 10 + 20 = 30. + # Orders 3,4 -> customer 101 -> region 2 (NULL): 5 + 7 = 12. + assert by_region.get("West") == pytest.approx(30.0), resp.data + assert by_region.get(None) == pytest.approx(12.0), resp.data + + async def test_join_order_is_stable_across_dialects(self) -> None: + """One mechanism means the relative join order does not depend on the + dialect — only identifier quoting does. + + The sequence is read out of the parsed JOIN nodes. Building it by + testing known names for membership in the SQL string would return the + *caller's* ordering every time and assert nothing. + """ + query = SlayerQuery( + source_model="orders_x", + dimensions=[ColumnRef(name="customers_v2.deep_pop")], + measures=[ModelMeasure(formula="amount:sum")], + ) + sequences = {} + for dialect in ("postgres", "sqlite", "duckdb"): + sql = await _engine_generate( + query=query, model=_orders_x(), dialect=dialect, + extra_models=[_customers_v2(), _regions(), _countries()], + ) + sequences[dialect] = base_cte_join_sequence(sql, dialect=dialect) + assert all(seq for seq in sequences.values()), ( + f"no joins were found to compare: {sequences}" + ) + assert len(set(map(tuple, sequences.values()))) == 1, sequences + + async def test_every_registered_scope_path_is_emitted_as_a_join( + self, monkeypatch: pytest.MonkeyPatch, + ) -> None: + """B11's safety property, stated in terms of the mechanism it adopts. + + Whatever the host scope registers must end up in the FROM. This is the + half of B11 that can regress: switching the collector to + ``join_paths.as_list()`` changes which list is authoritative, and a path + present in one list but not the other becomes a missing join — invalid + SQL rather than a cosmetic reordering. + + NOTE ON WHY THE *ORDER* IS NOT PINNED HERE. Today the host scope is not + consulted at all for a plain joined dimension: ``ScopeFrame.resolve`` + is never called for that shape, and the dimension paths come from the + row-slot walk. So the new ordering cannot be asserted against the scope + before the implementation exists without presuming its internals — and + the ratified statement for B11 is *execution-identical*, with the order + change itself surfaced through the PR's recompare corpus. What is + pinned here instead are the properties that must hold either way: the + join SET, parent-before-child, determinism, and executed results. + """ + registered: List[str] = [] + original = ScopeFrame._register_join_paths + + def _wrapped(self_frame, parsed): + result = original(self_frame, parsed) + registered.extend( + "__".join(p) for p in self_frame.join_paths.as_list() + ) + return result + + monkeypatch.setattr( + ScopeFrame, "_register_join_paths", _wrapped, raising=True, + ) + query = SlayerQuery( + source_model="orders_x", + dimensions=[ColumnRef(name="customers_v2.status")], + measures=[ModelMeasure(formula="customers_v2.lifetime_value:sum")], + filters=["customers_v2.status != 'x'"], + ) + sql = await _gen(query, dialect="postgres") + emitted = set(_join_aliases(sql)) + missing = {p for p in registered if p not in emitted} + assert not missing, ( + f"path(s) {sorted(missing)} were registered on a scope but never " + f"emitted as joins — the FROM would reference an unbound table.\n\n" + f"{sql}" + ) diff --git a/tests/test_sql_generator.py b/tests/test_sql_generator.py index 4df63c8a..e683e38c 100644 --- a/tests/test_sql_generator.py +++ b/tests/test_sql_generator.py @@ -866,7 +866,12 @@ async def test_windowed_sum_uses_range_join_primitive(self, generator: SQLGenera # AST-based generation renders single-unit intervals via sqlglot's # per-dialect transpiler — Postgres caps the unit name. assert "INTERVAL '90 DAY'" in norm - assert '_src._w_dim_0 = _base."orders.status"' in norm + # The inner grain comparison is NULL-SAFE: a group whose dimension is + # NULL must receive its real windowed value, not NULL. The time-range + # bounds above stay plain inequalities — they are a range, not grain. + assert ( + '_src._w_dim_0 IS NOT DISTINCT FROM _base."orders.status"' in norm + ), norm async def test_windowed_sum_preserves_other_time_dim_grain( self, generator: SQLGenerator, orders_model: SlayerModel, From 142133d4d7efb6fa93c3d2b681379551966daeba Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Thu, 6 Aug 2026 13:07:02 +0200 Subject: [PATCH 39/98] =?UTF-8?q?DEV-1745:=20review=20nitpicks=20=E2=80=94?= =?UTF-8?q?=20imports,=20async-without-await,=20typing,=20docs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `import inspect` moves to module scope per CLAUDE.md. TestLowerLayersStaySilent carried @pytest.mark.asyncio but neither of its tests awaits anything, so they are plain `def` now — an async test that never awaits buys nothing and trips Sonar's S7503. _plan_outer_where_filters' three parameters were bare `list`; typed as List[FilterPhase] / List[CrossModelAggregatePlan] / List[ValueSlot]. The render package docstring lists every module it contains; `.parse` was missing. Co-Authored-By: Claude Opus 5 (1M context) --- slayer/engine/stage_planner.py | 6 +++++- slayer/mcp/server.py | 2 -- slayer/sql/render/__init__.py | 2 ++ tests/test_dev1745_mode_a_door.py | 4 ++-- tests/test_dev1745_warning_contract.py | 5 ++--- 5 files changed, 11 insertions(+), 8 deletions(-) diff --git a/slayer/engine/stage_planner.py b/slayer/engine/stage_planner.py index 27396b46..2eafcfd5 100644 --- a/slayer/engine/stage_planner.py +++ b/slayer/engine/stage_planner.py @@ -88,6 +88,7 @@ from slayer.engine.planned import ( BoundExpr as PlannedBoundExpr, BoundFilterId, + CrossModelAggregatePlan, FilterPhase, FilterReachability, OrderEntry, @@ -1557,7 +1558,10 @@ def _windowed_phase(bf: BoundFilter) -> Phase: def _plan_outer_where_filters( - *, filters_by_phase: list, cross_model_plans: list, slots: list, + *, + filters_by_phase: List[FilterPhase], + cross_model_plans: List[CrossModelAggregatePlan], + slots: List[ValueSlot], ) -> List[BoundFilterId]: """AGGREGATE-phase filters that must be applied on the OUTER combined SELECT instead of as HAVING inside a ``_cm_*`` CTE (DEV-1503). diff --git a/slayer/mcp/server.py b/slayer/mcp/server.py index dd601eab..ccc49d4a 100644 --- a/slayer/mcp/server.py +++ b/slayer/mcp/server.py @@ -1967,8 +1967,6 @@ def _format_json( would break ``json.loads`` on exactly the queries a caller most needs to inspect (DEV-1745 W5). """ - import json - if not warnings: return json.dumps(data, default=str) return json.dumps({"data": data, "warnings": warnings}, default=str) diff --git a/slayer/sql/render/__init__.py b/slayer/sql/render/__init__.py index 9b8a9644..5c5a2195 100644 --- a/slayer/sql/render/__init__.py +++ b/slayer/sql/render/__init__.py @@ -6,6 +6,8 @@ * :mod:`.value_expr` — one ``ValueKey`` → sqlglot-AST renderer, so a given key renders identically wherever it appears. * :mod:`.aggregates` — one registry for aggregation rendering. +* :mod:`.parse` — one parse of free SQL text into a SLayer-normalised sqlglot + AST, shared by the generator and the Mode-A door on ``ScopeFrame``. Nothing here imports ``generator``; the dependency runs one way. """ diff --git a/tests/test_dev1745_mode_a_door.py b/tests/test_dev1745_mode_a_door.py index ed03020f..d11f3b8a 100644 --- a/tests/test_dev1745_mode_a_door.py +++ b/tests/test_dev1745_mode_a_door.py @@ -31,6 +31,8 @@ from __future__ import annotations +import inspect + import pytest from sqlglot import exp @@ -121,8 +123,6 @@ def test_enter_expression_exists(self) -> None: def test_no_include_dotted_derived_flag_anywhere(self) -> None: """The flag is deleted, not threaded through the new door.""" - import inspect - for name in ("enter_predicate", "enter_expression"): fn = getattr(ScopeFrame, name, None) if fn is None: diff --git a/tests/test_dev1745_warning_contract.py b/tests/test_dev1745_warning_contract.py index e6c56edf..e82dd968 100644 --- a/tests/test_dev1745_warning_contract.py +++ b/tests/test_dev1745_warning_contract.py @@ -416,7 +416,6 @@ def test_each_subclass_declares_a_distinct_kind(self) -> None: assert kinds == {"normalization", "unreachable_filter_dropped"}, kinds -@pytest.mark.asyncio class TestLowerLayersStaySilent: """The emission is at the BOUNDARY. Planning and rendering must not warn on their own — otherwise 'exactly once' holds only by luck of deduplication.""" @@ -432,7 +431,7 @@ def _filter_warnings(caught) -> list: """ return [w for w in caught if "warehouses.code" in str(w.message)] - async def test_planning_emits_no_python_warning(self) -> None: + def test_planning_emits_no_python_warning(self) -> None: from slayer.engine.source_bundle import ResolvedSourceBundle from slayer.engine.stage_planner import plan_query @@ -448,7 +447,7 @@ async def test_planning_emits_no_python_warning(self) -> None: "at the engine boundary" ) - async def test_rendering_emits_no_python_warning(self) -> None: + def test_rendering_emits_no_python_warning(self) -> None: from slayer.engine.source_bundle import ResolvedSourceBundle from slayer.engine.stage_planner import plan_query from slayer.sql.generator import SQLGenerator From 9207d7a52740649153ec42dd10510df5e6e8caea Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Thu, 6 Aug 2026 13:21:07 +0200 Subject: [PATCH 40/98] =?UTF-8?q?DEV-1746=20stage=203:=20the=20combined=20?= =?UTF-8?q?layer=20is=20one=20sqlglot=20AST=20(B3,=20B7,=20=C2=A75.6)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The cross-model combined statement was assembled as text: a projection built by string concatenation, a FROM/JOIN chain appended with += , a WHERE glued on with a hand-rolled "\nAND " / "\nWHERE " connector, a WITH chain spliced from f-strings, and LIMIT/OFFSET appended raw. It is now built as a single exp.Select and rendered once. §5.6 — WITH assembly. slayer/sql/render/cte_assembly.py takes explicit (name, query, depends_on) entries and emits a stable topological order with declaration order as the tiebreak. Dependencies are DECLARED by the caller (_wm_ reads _base; _cm_ reads nothing), never discovered by scanning the rendered statement: a scan cannot tell a CTE reference from a same-named real table, is defeated by quoting and case folding, and would silently mis-order rather than fail. assert_unique_cte_names remains the belt for name collisions. A production-path test proves the assembler is what runs and that it receives real dependency metadata. CTE bodies now stay AST from renderer to assembler. The first attempt kept the renderers returning text and re-parsed it at the seam, which re-introduced exactly the corruption B2 had just removed — a dotted public alias round-trips through text as a multi-part reference on BigQuery, so `_base."orders_x.status"` came back as `_base___orders_x`.`status`. One parse seam remains, documented: the re-rooted cross-model CTE arrives as a complete nested WITH…SELECT from generate_from_planned. B7 — projection order. The combined projection was assembled in four grouped passes (host, outer composites, _cm_, _wm_), so a cross-model measure declared first was emitted last; the generator said so in a comment. Every column is now rendered per slot and emitted in planned_query.projection order. Hidden slots are absent from that list, so trimming them stops being a step. One subtlety the tests caught: a slot can appear in the projection more than once (C13 — one key under several declared names), so each occurrence consumes the NEXT of that slot's rendered columns; emitting the whole list per occurrence projected every alias once per name. B3 completed. The combined statement paginates through apply_pagination like every other path, so a cross-model query on SQL Server no longer emits a literal LIMIT. The combined ORDER BY is built as AST too. Rendering its terms to text and re-parsing them corrupted dotted aliases the same way the join-back did (`orders___customers`.`spend_sum`), so _build_combined_order_by_sql and _resolve_combined_order_term now return exp.Ordered nodes, and the hidden-CTE order refs and order-only composite expressions stay AST end to end. Recompare churn, all execution-identical and reviewed: - 25 golden entries (5 cross-model cases x 5 dialects) reformat: the statement is emitted by sqlglot's printer in one pass, so CTE bodies indent and clauses break where the printer breaks them. Regenerated through the ALLOWED_DELTAS protocol; manifest cleared. - tests/_engine_helpers._extract_src_body anchored on the exact text "\n) AS _src" and broke on the new indentation — 19 frame-bound tests failed on one helper. It is now whitespace-tolerant. - Three assertions pinned layout rather than structure (an indented HAVING, a single-line WHERE, a single-line ORDER BY regex) and now normalise whitespace. - One assertion pinned the pre-B7 projection order; both keys remain present and distinct, only their order changed. Stages 1-3 green: 10924 passing, no non-DEV-1746 failures. 18 TDD tests remain red for stages 4-6 (B8 carry lists, the projection invariant, the empty-base plan node, and the consumer= materialiser migration). Co-Authored-By: Claude Opus 5 (1M context) --- slayer/sql/generator.py | 266 +++++++++++++++-------- slayer/sql/render/cte_assembly.py | 109 ++++++++++ tests/_engine_helpers.py | 11 +- tests/golden/dev1745_sql_baseline.json | 50 ++--- tests/test_cross_model_rename_dev1448.py | 8 +- tests/test_dev1745_plan_time_routing.py | 9 +- tests/test_dev1746_cte_assembly.py | 13 +- tests/test_sql_generator.py | 2 +- 8 files changed, 339 insertions(+), 129 deletions(-) create mode 100644 slayer/sql/render/cte_assembly.py diff --git a/slayer/sql/generator.py b/slayer/sql/generator.py index 836495a0..ff03af6a 100644 --- a/slayer/sql/generator.py +++ b/slayer/sql/generator.py @@ -56,6 +56,7 @@ result_key_from_alias, ) from slayer.sql.render.aggregates import window_agg_class +from slayer.sql.render.cte_assembly import CteEntry, assemble_with_chain from slayer.sql.render.joins import ( build_grain_joinback_condition, grain_alias_column, @@ -916,6 +917,23 @@ def _quote_ident(self, name: str) -> str: """ return exp.to_identifier(name, quoted=True).sql(dialect=self.dialect) + def _parse_cte_body(self, sql: str) -> exp.Expression: + """Parse a rendered CTE body back into AST for the WITH assembler. + + The deliberate seam. The CTE renderers still return SQL text, and one of + them (the re-rooted cross-model CTE) returns a COMPLETE ``WITH … SELECT`` + statement produced by a nested ``generate_from_planned`` — threading AST + out through that whole pipeline is a larger change than this PR takes on. + Parsing once here keeps the assembly between scopes on AST, which is + what the doctrine is about; the alternative was splicing statement text + into an f-string, which is what it replaced. + + ``sqlglot.parse_one`` rather than :meth:`_parse`: this text is our own + freshly-emitted output, so it needs no prequoting or derived-ref + expansion — only structure. + """ + return sqlglot.parse_one(sql, dialect=self.dialect) + def _null_safe_join_pair_sql(self, *, left_sql: str, right_sql: str) -> str: """Render one dialect-aware null-safe equality (DEV-1708 / Codex F2) for a grain join-back ``ON`` clause. ``left_sql`` / ``right_sql`` are the @@ -4148,7 +4166,11 @@ def _alias_of(sid: str) -> str: for ga in grain_aliases: outer = outer.group_by(_base_col(ga)) - return outer.sql(dialect=self.dialect, pretty=True), grain_aliases + # Returned as AST: the caller assembles the WITH chain structurally. + # Rendering here and re-parsing later would re-introduce the very + # corruption B2 removed — a dotted public alias round-trips through + # text as a multi-part reference on BigQuery. + return outer, grain_aliases def _render_with_cross_model_plans( # NOSONAR(S3776) — orchestration of host ``_base`` CTE + per-plan ``_cm_*`` CTEs + combined SELECT + transform-chain step CTEs + outer ORDER BY/LIMIT wrap. Each block is a coherent compilation stage sharing planned_query / slots_by_id / cma_slot_ids / seen_base_ids state; extracting per-stage helpers would scatter the cross-cutting state. self, @@ -4560,7 +4582,8 @@ def _add_local_aux_slots( if base_having is not None: base_select = base_select.having(base_having) - base_cte_sql = base_select.sql(dialect=self.dialect, pretty=True) + # ``base_select`` stays AST: the WITH assembler takes the query + # structurally, and the transform-chain branch renders it on demand. # Per-plan ``_cm_*`` CTEs. The CTE name and projection use the # CANONICAL aggregate alias (path + canonical_agg_name); user- @@ -4633,7 +4656,7 @@ def _add_local_aux_slots( if plan.rerooted_plan is not None: # C1: nested re-rooted PlannedQuery rooted at the target, # preserving host dimension grain. - cte_sql, joinback_pairs, agg_col_alias = ( + rerooted_sql, joinback_pairs, agg_col_alias = ( self._render_rerooted_cross_model_cte( plan=plan, bundle=bundle, @@ -4641,8 +4664,12 @@ def _add_local_aux_slots( host_source_relation=source_relation, ) ) + # The ONE parse seam: this branch renders a complete nested + # ``WITH … SELECT`` through ``generate_from_planned``, so it + # arrives as text. Everything else in the chain is already AST. + cte_query = self._parse_cte_body(rerooted_sql) else: - cte_sql, shared_grain_aliases = self._render_cross_model_cte( + cte_query, shared_grain_aliases = self._render_cross_model_cte( plan=plan, agg_slot=agg_slot, full_agg_alias=canonical_alias, @@ -4654,7 +4681,7 @@ def _add_local_aux_slots( # Forward path: host alias == cte alias; agg under canonical. joinback_pairs = [(a, a) for a in shared_grain_aliases] agg_col_alias = canonical_alias - cm_ctes.append((cte_name, cte_sql)) + cm_ctes.append((cte_name, cte_query)) joinback_pairs_for_plan[plan.aggregate_slot_id] = joinback_pairs agg_col_alias_for_plan[plan.aggregate_slot_id] = agg_col_alias joinback_pairs_for_identity[identity] = joinback_pairs @@ -4687,13 +4714,13 @@ def _add_local_aux_slots( cte_name = cte_name_from_alias( "_wm_", full_agg_alias, allocator=wm_allocator, ) - cte_sql, grain_aliases = self._render_window_measure_cte_from_planned( + cte_query, grain_aliases = self._render_window_measure_cte_from_planned( plan=plan, agg_slot=agg_slot, source_model=source_model, source_relation=source_relation, bundle=bundle, planned_query=planned_query, slots_by_id=slots_by_id, aliases_by_slot_id=aliases_by_slot_id, full_agg_alias=full_agg_alias, ) - wm_ctes.append((cte_name, cte_sql)) + wm_ctes.append((cte_name, cte_query)) wm_cte_name_for_plan[plan.aggregate_slot_id] = cte_name wm_agg_col_for_plan[plan.aggregate_slot_id] = full_agg_alias wm_joinback_pairs_for_plan[plan.aggregate_slot_id] = [ @@ -4710,11 +4737,19 @@ def _add_local_aux_slots( # Build the combined SELECT: SELECT _base., # _cm_*. [AS ""] FROM _base [LEFT JOIN | # CROSS JOIN] _cm_* [ON ...]. - combined_parts: List[str] = [] + # Projection expressions per slot, emitted below in the PLAN's declared + # order (B7). Collecting per slot first is what allows one ordered pass: + # the host, outer-composite, cross-model and windowed sides each know how + # to render their own columns, but none of them knows where those columns + # belong relative to the others — only ``planned_query.projection`` does. + proj_exprs: Dict[str, List[exp.Expression]] = {} # ``combined_aliases_by_slot_id`` records the output column alias each # slot surfaces in the combined SELECT — the input the transform chain # (when present) binds against (the combined result is its base CTE). combined_aliases_by_slot_id: Dict[str, List[str]] = {} + + def _emit(sid: str, expr: exp.Expression) -> None: + proj_exprs.setdefault(sid, []).append(expr) # Host-side projection: every slot in base_projection surfaces # its picked alias(es). Multi-alias slots emit one entry per # alias (C13). With a transform chain on top, the combined SELECT is @@ -4729,7 +4764,7 @@ def _add_local_aux_slots( for sid in host_combined_ids: aliases = aliases_by_slot_id.get(sid, []) for full_alias in aliases: - combined_parts.append(f'_base.{self._quote_ident(full_alias)}') + _emit(sid, grain_alias_column(alias=full_alias, table="_base")) if aliases: combined_aliases_by_slot_id[sid] = list(aliases) # DEV-1503 (Codex round 2 #1) — composite slots routed to the outer @@ -4739,7 +4774,7 @@ def _add_local_aux_slots( # promoted as aux above). Wrap with ``AS ""`` so the # composite surfaces under the user-declared name. outer_composite_order_alias_by_sid: Dict[str, str] = {} - outer_composite_order_expressions: Dict[str, str] = {} + outer_composite_order_expressions: Dict[str, exp.Expression] = {} if outer_composite_slot_ids: outer_composite_cm_map: Dict[str, Tuple[str, str]] = {} for plan in planned_query.cross_model_aggregate_plans: @@ -4760,7 +4795,7 @@ def _add_local_aux_slots( wm_agg_col_for_plan[plan.aggregate_slot_id], ) - def _render_outer_composite(cslot) -> str: + def _render_outer_composite(cslot) -> exp.Expression: rendered = self._render_filter_for_outer_wrapper( key=cslot.key, slot_by_key=slot_by_key, @@ -4769,7 +4804,7 @@ def _render_outer_composite(cslot) -> str: ) if cslot.type is not None: rendered = _wrap_cast_for_type(rendered, cslot.type) - return rendered.sql(dialect=self.dialect) + return rendered # Projected outer composites: cycle through ``public_aliases`` # for each occurrence in ``planned_query.projection``. C13 lets @@ -4795,8 +4830,9 @@ def _render_outer_composite(cslot) -> str: ) outer_emission_count[sid] = idx + 1 full_alias = f"{source_relation}.{public_alias}" - combined_parts.append( - f'{_render_outer_composite(cslot)} AS {self._quote_ident(full_alias)}', + _emit( + sid, + _render_outer_composite(cslot).as_(full_alias, quoted=True), ) combined_aliases_by_slot_id.setdefault(sid, []).append( full_alias, @@ -4861,12 +4897,11 @@ def _render_outer_composite(cslot) -> str: ) ) for pub in public_aliases: - if pub == agg_col_alias: - combined_parts.append(f'{cte_name}.{self._quote_ident(agg_col_alias)}') - else: - combined_parts.append( - f'{cte_name}.{self._quote_ident(agg_col_alias)} AS {self._quote_ident(pub)}', - ) + col = grain_alias_column(alias=agg_col_alias, table=cte_name) + _emit( + plan.aggregate_slot_id, + col if pub == agg_col_alias else col.as_(pub, quoted=True), + ) combined_aliases_by_slot_id[plan.aggregate_slot_id] = list( public_aliases, ) @@ -4900,12 +4935,11 @@ def _render_outer_composite(cslot) -> str: ) full_aliases = [f"{source_relation}.{p}" for p in public_names] or [agg_col] for full in full_aliases: - if full == agg_col: - combined_parts.append(f'{cte_name}.{self._quote_ident(agg_col)}') - else: - combined_parts.append( - f'{cte_name}.{self._quote_ident(agg_col)} AS {self._quote_ident(full)}', - ) + col = grain_alias_column(alias=agg_col, table=cte_name) + _emit( + plan.aggregate_slot_id, + col if full == agg_col else col.as_(full, quoted=True), + ) combined_aliases_by_slot_id[plan.aggregate_slot_id] = list(full_aliases) # Grain join-backs (P-I). Both plan kinds join back identically — on the @@ -4913,7 +4947,37 @@ def _render_outer_composite(cslot) -> str: # truncated time bucket keeps its aggregate instead of dropping it. An # EMPTY grain (a scalar aggregate) has nothing to join on and becomes a # CROSS JOIN; the builder signals that by returning ``None``. - from_clause_str = "FROM _base" + # The public projection, in the plan's declared order (B7). Every + # renderer consumes ``planned_query.projection`` verbatim rather than + # reconstructing an order from the separate host / composite / cross- + # model / windowed lists — which is why a cross-model measure declared + # first used to be emitted last. Hidden slots are simply absent from the + # plan's list, so trimming them is not a step: it is the absence of one. + # One slot can appear in the projection more than once: C13 lets the + # same key be selected under several user-declared names, and the plan + # lists it once per name. Each occurrence therefore consumes the NEXT of + # that slot's rendered columns — emitting the whole list per occurrence + # would project every alias once per name. + combined_select_exprs: List[exp.Expression] = [] + consumed: Dict[str, int] = {} + for sid in planned_query.projection: + exprs = proj_exprs.get(sid) + if not exprs: + continue + idx = consumed.get(sid, 0) + if idx < len(exprs): + combined_select_exprs.append(exprs[idx]) + consumed[sid] = idx + 1 + # Columns the plan does not publish but the statement still needs: with + # a transform chain the combined SELECT is that chain's base CTE, so it + # must also carry hidden inputs (transform operands, order-only slots) + # for the step CTEs to read. The outer wrap trims them back afterwards. + for sid, exprs in proj_exprs.items(): + combined_select_exprs.extend(exprs[consumed.get(sid, 0):]) + + combined_select = exp.Select().select(*combined_select_exprs) + combined_select = combined_select.from_("_base") + joined_cte_names: set = set() joinback_specs = [ ( @@ -4943,17 +5007,14 @@ def _render_outer_composite(cslot) -> str: dialect=self._dialect, ) if on_condition is None: - from_clause_str += f"\nCROSS JOIN {cte_name}" + combined_select = combined_select.join( + cte_name, join_type="CROSS", + ) else: - from_clause_str += ( - f"\nLEFT JOIN {cte_name} ON " - + on_condition.sql(dialect=self.dialect) + combined_select = combined_select.join( + cte_name, on=on_condition, join_type="LEFT", ) - combined_select_sql = ( - f"SELECT {', '.join(combined_parts)}\n{from_clause_str}" - ) - # DEV-1503 — outer combined-SELECT WHERE wrapper. AGGREGATE-phase # host filters routed here in the classification pass above # (``outer_where_filters``) render now against the joined-back @@ -4977,7 +5038,6 @@ def _render_outer_composite(cslot) -> str: cross_model_agg_slot_to_cm[plan.aggregate_slot_id] = ( cte_name, agg_col_alias, ) - outer_where_parts: List[str] = [] for fp in outer_where_filters: rendered = self._render_filter_for_outer_wrapper( key=fp.expression.value_key, @@ -4987,10 +5047,7 @@ def _render_outer_composite(cslot) -> str: ) if isinstance(rendered, (exp.And, exp.Or)): rendered = exp.Paren(this=rendered) - outer_where_parts.append(rendered.sql(dialect=self.dialect)) - combined_select_sql += ( - "\nWHERE " + _SQL_AND_JOINER.join(outer_where_parts) - ) + combined_select = combined_select.where(rendered) # DEV-1714 Stage 10 — POST-phase filters referencing a windowed measure # render as an outer WHERE on the combined SELECT (never HAVING on the @@ -5004,7 +5061,11 @@ def _render_outer_composite(cslot) -> str: ) for p in planned_query.windowed_aggregate_plans } - wm_post_parts: List[str] = [] + # ``Select.where`` conjoins, so a POST-phase windowed filter composes + # with any outer-WHERE filter above without the caller choosing + # between ``WHERE`` and ``AND`` — the hand-rolled connector this + # replaces glued an ``AND`` onto a predicate built elsewhere, with no + # parenthesisation of the union. for fp in planned_query.filters_by_phase: if fp.phase != Phase.POST or fp.expression is None: continue @@ -5016,10 +5077,7 @@ def _render_outer_composite(cslot) -> str: ) if isinstance(rendered, (exp.And, exp.Or)): rendered = exp.Paren(this=rendered) - wm_post_parts.append(rendered.sql(dialect=self.dialect)) - if wm_post_parts: - connector = "\nAND " if outer_where_filters else "\nWHERE " - combined_select_sql += connector + _SQL_AND_JOINER.join(wm_post_parts) + combined_select = combined_select.where(rendered) # DEV-1450 stage 7b.15e (C2): a transform layer over a cross-model # aggregate (``cumsum(customers.avg_score:avg)``) runs on TOP of the @@ -5038,21 +5096,41 @@ def _render_outer_composite(cslot) -> str: "G4); the cross-model transform chain does not carry `_wm_` " "CTEs.", ) + # The transform chain is still string-assembled (it adopts the + # shared assembler in PR 4, with the local chain), so render its + # prelude CTEs here rather than threading AST into it. return self._render_cross_model_transform_chain( - prelude_ctes=[("_base", base_cte_sql)] + cm_ctes, - combined_select_sql=combined_select_sql, + prelude_ctes=[ + ("_base", base_select.sql(dialect=self.dialect, pretty=True)), + ] + [ + (name, query.sql(dialect=self.dialect, pretty=True)) + for name, query in cm_ctes + ], + combined_select_sql=combined_select.sql( + dialect=self.dialect, pretty=True, + ), planned_query=planned_query, slots_by_id=slots_by_id, combined_aliases_by_slot_id=combined_aliases_by_slot_id, source_relation=source_relation, ) - all_ctes = [("_base", base_cte_sql)] + cm_ctes + wm_ctes + [("_combined", combined_select_sql)] - - # Stitch the WITH chain together. Inner CTEs first; the final - # ``_combined`` is the outermost FROM target. - cte_strs = [f"{name} AS (\n{sql}\n)" for name, sql in all_ctes[:-1]] - sql = f"WITH {', '.join(cte_strs)}\n{combined_select_sql}" + # Assemble the WITH chain (§5.6). Dependencies are DECLARED, not + # discovered by scanning the rendered statement: ``_wm_`` CTEs select + # FROM ``_base``, the cross-model CTEs are rooted at their own targets + # and depend on nothing. The assembler emits a stable topological order + # with declaration order as the tiebreak. + cte_entries = [CteEntry(name="_base", query=base_select)] + cte_entries += [ + CteEntry(name=name, query=query) for name, query in cm_ctes + ] + cte_entries += [ + CteEntry(name=name, query=query, depends_on=["_base"]) + for name, query in wm_ctes + ] + combined_statement = assemble_with_chain( + entries=cte_entries, final=combined_select, + ) # ORDER BY / LIMIT / OFFSET: emitted at the combined SELECT # level. ORDER BY columns must be qualified — ``_base`` columns @@ -5062,7 +5140,7 @@ def _render_outer_composite(cslot) -> str: # are trimmed from the projection above, so their ORDER BY term must be # CTE-qualified (``_cm_*.""``) rather than the bare # combined-SELECT alias. - hidden_cte_order_refs: Dict[str, str] = {} + hidden_cte_order_refs: Dict[str, exp.Expression] = {} for plan in planned_query.cross_model_aggregate_plans: # Only CMAs actually trimmed from the projection (hidden + no # transform chain) need the CTE-qualified ORDER BY reference. @@ -5071,7 +5149,7 @@ def _render_outer_composite(cslot) -> str: _agg_col = agg_col_alias_for_plan[plan.aggregate_slot_id] _cte = cm_cte_name_for_plan[plan.aggregate_slot_id] hidden_cte_order_refs[plan.aggregate_slot_id] = ( - f'{_cte}.{self._quote_ident(_agg_col)}' + grain_alias_column(alias=_agg_col, table=_cte) ) # DEV-1733: same treatment for a hidden (order-only) WINDOWED aggregate # trimmed from the combined projection above — reference its ``_wm_`` @@ -5079,11 +5157,11 @@ def _render_outer_composite(cslot) -> str: for plan in planned_query.windowed_aggregate_plans: if not (plan.hidden and not planned_query.transform_layers): continue - hidden_cte_order_refs[plan.aggregate_slot_id] = ( - f'{wm_cte_name_for_plan[plan.aggregate_slot_id]}.' - f'{self._quote_ident(wm_agg_col_for_plan[plan.aggregate_slot_id])}' + hidden_cte_order_refs[plan.aggregate_slot_id] = grain_alias_column( + alias=wm_agg_col_for_plan[plan.aggregate_slot_id], + table=wm_cte_name_for_plan[plan.aggregate_slot_id], ) - order_sql = self._build_combined_order_by_sql( + order_terms = self._build_combined_order_by_sql( planned_query=planned_query, slots_by_id=slots_by_id, cma_slot_ids=cma_slot_ids, @@ -5096,19 +5174,25 @@ def _render_outer_composite(cslot) -> str: outer_composite_expressions=outer_composite_order_expressions, hidden_cte_order_refs=hidden_cte_order_refs, ) - if order_sql: - sql += "\n" + order_sql - if planned_query.limit is not None: - sql += f"\nLIMIT {planned_query.limit}" - if planned_query.offset is not None: - sql += f"\nOFFSET {planned_query.offset}" + if order_terms: + combined_statement.set("order", exp.Order(expressions=order_terms)) + + # Pagination through the dialect strategy (B3). This path used to append + # raw ``LIMIT``/``OFFSET`` text, which emitted a literal ``LIMIT`` on + # SQL Server — while the same query carrying a transform layer went + # through the outer wrap and came out correct. + combined_statement = self._dialect.apply_pagination( + combined_statement, + limit=planned_query.limit, + offset=planned_query.offset, + ) # Outer projection trim — the inner already projects the public # list in declared order, so the trim is normally a no-op. Skip # the trim machinery here because the legacy path goes through # an EnrichedQuery-driven ``_apply_outer_projection_trim`` that # we don't have on the new side. Future slices may re-enable. - return sql + return combined_statement.sql(dialect=self.dialect, pretty=True) def _render_cross_model_transform_chain( # NOSONAR(S3776) — pre-existing complexity in the window-layer chain; this PR only threaded the CTE-name allocator through it, which re-attributed the function as new code. The chain is rebuilt as sqlglot AST in the scope-assembly PR, where the layering is what gets simplified. self, @@ -6036,8 +6120,7 @@ def _register_filter_join_paths(sql_text: Optional[str]) -> None: if cte_having is not None: cte_select = cte_select.having(cte_having) - cte_sql = cte_select.sql(dialect=self.dialect, pretty=True) - return cte_sql, shared_grain_aliases + return cte_select, shared_grain_aliases def _register_routed_filter_joins( # NOSONAR(S3776) — a cohesive recursive ValueKey tree-walk dispatcher (the heavy AggregateKey arm is already extracted to _register_agg_key_joins); the remaining branches are the closed-union dispatch contract, mirroring the sibling walkers _value_key_join_paths / _collect_base_aux_slot_ids in this file. self, @@ -6457,10 +6540,15 @@ def _build_combined_order_by_sql( cm_alias_for_plan: Dict[str, str], bare_order_slot_ids: Optional[Set[str]] = None, outer_composite_aliases: Optional[Dict[str, str]] = None, - outer_composite_expressions: Optional[Dict[str, str]] = None, - hidden_cte_order_refs: Optional[Dict[str, str]] = None, - ) -> Optional[str]: - """Build the ORDER BY clause for the combined SELECT. + outer_composite_expressions: Optional[Dict[str, exp.Expression]] = None, + hidden_cte_order_refs: Optional[Dict[str, exp.Expression]] = None, + ) -> List[exp.Ordered]: + """Build the combined SELECT's ORDER BY terms, as AST. + + Terms are AST rather than text because they reference DOTTED public + aliases, and rendering them to a string only to re-parse it re-reads + such an alias as a multi-part reference on BigQuery — the same + corruption the grain join-back suffered. PROJECTED local slots are referenced as ``_base.""`` (legacy parity); cross-model slots are referenced as bare @@ -6482,12 +6570,12 @@ def _build_combined_order_by_sql( would dangle). """ if not planned_query.order: - return None + return [] bare_ids = bare_order_slot_ids or set() outer_aliases = outer_composite_aliases or {} outer_expressions = outer_composite_expressions or {} hidden_cte_refs = hidden_cte_order_refs or {} - parts: List[str] = [] + parts: List[exp.Ordered] = [] for entry in planned_query.order: slot = slots_by_id.get(entry.slot_id) if slot is None: @@ -6505,9 +6593,7 @@ def _build_combined_order_by_sql( ) if term is not None: parts.append(term) - if not parts: - return None - return "ORDER BY " + ", ".join(parts) + return parts def _resolve_combined_order_term( self, @@ -6519,10 +6605,10 @@ def _resolve_combined_order_term( cm_alias_for_plan: Dict[str, str], bare_ids: Set[str], outer_aliases: Dict[str, str], - outer_expressions: Optional[Dict[str, str]] = None, - hidden_cte_refs: Optional[Dict[str, str]] = None, - ) -> Optional[str]: - """Resolve one ``OrderEntry`` to its ``"alias" `` term. + outer_expressions: Optional[Dict[str, exp.Expression]] = None, + hidden_cte_refs: Optional[Dict[str, exp.Expression]] = None, + ) -> Optional[exp.Ordered]: + """Resolve one ``OrderEntry`` to an ``exp.Ordered`` term. Cross-model agg slot → bare CTE alias; projected outer-composite slot → bare combined-SELECT alias; order-only outer composite @@ -6534,7 +6620,11 @@ def _resolve_combined_order_term( cross-model alias map has no entry (the order slot can't be rendered). """ - direction = "ASC" if entry.direction == "asc" else "DESC" + descending = entry.direction != "asc" + + def _ordered(col: exp.Expression) -> exp.Ordered: + return exp.Ordered(this=col, desc=descending) + # DEV-1712 / DEV-1733: a HIDDEN (order-only) aggregate that lives in its # own CTE — cross-model (``_cm_``) or windowed (``_wm_``) — is trimmed # from the combined projection, so the bare alias no longer names a @@ -6544,24 +6634,24 @@ def _resolve_combined_order_term( # branch below and dangle. hidden_ref = (hidden_cte_refs or {}).get(entry.slot_id) if hidden_ref is not None: - return f'{hidden_ref} {direction}' + return _ordered(hidden_ref.copy()) if entry.slot_id in cma_slot_ids: alias = cm_alias_for_plan.get(entry.slot_id) if alias is None: return None - return f'{self._quote_ident(alias)} {direction}' + return _ordered(exp.column(alias, quoted=True)) if entry.slot_id in outer_aliases: - return f'{self._quote_ident(outer_aliases[entry.slot_id])} {direction}' + return _ordered(exp.column(outer_aliases[entry.slot_id], quoted=True)) if outer_expressions and entry.slot_id in outer_expressions: - return f'{outer_expressions[entry.slot_id]} {direction}' + return _ordered(outer_expressions[entry.slot_id].copy()) full_alias = self._full_alias_for_slot( slot=slot, source_relation=source_relation, alias_index={}, ) if entry.slot_id in bare_ids: - return f'{self._quote_ident(full_alias)} {direction}' - return f'_base.{self._quote_ident(full_alias)} {direction}' + return _ordered(exp.column(full_alias, quoted=True)) + return _ordered(grain_alias_column(alias=full_alias, table="_base")) def _full_alias_for_slot( self, diff --git a/slayer/sql/render/cte_assembly.py b/slayer/sql/render/cte_assembly.py new file mode 100644 index 00000000..a80ae6ce --- /dev/null +++ b/slayer/sql/render/cte_assembly.py @@ -0,0 +1,109 @@ +"""WITH-chain assembly in topological order (§5.6). + +The cross-model paths used to splice their WITH chain out of f-strings:: + + cte_strs = [f"{name} AS (\\n{sql}\\n)" for name, sql in all_ctes[:-1]] + sql = f"WITH {', '.join(cte_strs)}\\n{combined_select_sql}" + +so the emitted order was whatever order the python list happened to be built +in, and the transform chain read its predecessor positionally +(``prev_cte = ctes[-1][0]``). That works only while one hard-coded sequence +stays correct; it carries no statement of what actually depends on what. + +Here a caller DECLARES each CTE's dependencies and the assembler emits a stable +topological order, with insertion order as the tiebreak so independent CTEs +keep declaration order and the SQL is byte-stable across runs. + +Dependencies are declared, never discovered by scanning the rendered AST. A +scan cannot tell a CTE reference from a same-named real table, is defeated by +quoting and case folding, and — worst — would silently mis-order rather than +fail. The caller already knows the answer structurally (``_wm_`` reads +``_base``; transform step N reads step N-1; ``_cm_`` reads nothing), so it says +so. + +Ordering is the only thing this module owns. Name collisions remain +``assert_unique_cte_names``' job, which validates the emitted statement per +WITH scope and case-folds on dialects that fold. +""" + +from __future__ import annotations + +from typing import Dict, List, Sequence + +from pydantic import BaseModel, ConfigDict +from sqlglot import exp + +__all__ = ["CteEntry", "assemble_with_chain"] + + +class CteEntry(BaseModel): + """One CTE: its allocator-minted name, its query, and what it reads.""" + + model_config = ConfigDict(arbitrary_types_allowed=True) + + name: str + query: exp.Expression + #: Names of CTEs this one references. Must all be present in the same + #: assembly; a dangling name is a wiring bug, not a no-op. + depends_on: List[str] = [] + + +def assemble_with_chain( + *, entries: Sequence[CteEntry], final: exp.Select, +) -> exp.Select: + """Attach ``entries`` to ``final`` as a WITH clause in dependency order. + + Returns ``final`` unchanged when there are no entries — an empty ``WITH`` is + not valid SQL. + + Raises ``ValueError`` on a duplicate name, a dependency naming a CTE that + was not supplied, or a cycle. All three are wiring bugs whose SQL would be + invalid or silently wrong, so they fail here rather than at the database. + """ + if not entries: + return final + + by_name: Dict[str, CteEntry] = {} + for entry in entries: + if entry.name in by_name: + raise ValueError( + f"duplicate CTE name {entry.name!r} in one WITH chain", + ) + by_name[entry.name] = entry + + for entry in entries: + unknown = [d for d in entry.depends_on if d not in by_name] + if unknown: + raise ValueError( + f"CTE {entry.name!r} declares unknown dependencies " + f"{unknown!r}; known CTEs are {sorted(by_name)}", + ) + + # Depth-first emit in declaration order: the first entry that is ready goes + # first, and a dependency is emitted immediately before the entry needing + # it. Declaration order is preserved wherever dependencies permit. + ordered: List[CteEntry] = [] + emitted: set[str] = set() + visiting: List[str] = [] + + def _visit(entry: CteEntry) -> None: + if entry.name in emitted: + return + if entry.name in visiting: + cycle = " -> ".join([*visiting[visiting.index(entry.name):], entry.name]) + raise ValueError(f"dependency cycle between CTEs: {cycle}") + visiting.append(entry.name) + for dep in entry.depends_on: + _visit(by_name[dep]) + visiting.pop() + emitted.add(entry.name) + ordered.append(entry) + + for entry in entries: + _visit(entry) + + out = final.copy() + out.set("with_", None) + for entry in ordered: + out = out.with_(entry.name, as_=entry.query.copy(), copy=False) + return out diff --git a/tests/_engine_helpers.py b/tests/_engine_helpers.py index ab062c03..1a9c0ca9 100644 --- a/tests/_engine_helpers.py +++ b/tests/_engine_helpers.py @@ -156,11 +156,12 @@ def _extract_src_body(sql: str) -> str: keyword or its formatting would surface as a confusing assertion against the wrong text rather than a clear failure here. """ - end = sql.index("\n) AS _src") - open_token = "LEFT JOIN (\n" - open_at = sql.rfind(open_token, 0, end) - assert open_at != -1, f"No {open_token!r} opening the _src subquery in:\n{sql}" - return sql[open_at + len(open_token):end] + close = re.search(r"\n[ \t]*\) AS _src", sql) + assert close is not None, f"No `) AS _src` closing the _src subquery in:\n{sql}" + end = close.start() + opens = list(re.finditer(r"LEFT JOIN \(\n", sql[:end])) + assert opens, f"No `LEFT JOIN (` opening the _src subquery in:\n{sql}" + return sql[opens[-1].end():end] def _extract_cte_body(sql: str, cte_name_pattern: str) -> str: diff --git a/tests/golden/dev1745_sql_baseline.json b/tests/golden/dev1745_sql_baseline.json index 25f1fb79..a943799b 100644 --- a/tests/golden/dev1745_sql_baseline.json +++ b/tests/golden/dev1745_sql_baseline.json @@ -1,19 +1,19 @@ { - "cm/fragment_default_crossing::bigquery": "WITH _base AS (\nSELECT\n orders.status AS `orders___status`\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__customers__spend_wscaled_sum AS (\nSELECT\n SUM(customers.spend * regions.weight) AS `orders___customers___spend_wscaled_sum`\nFROM customers AS customers\nLEFT JOIN regions AS regions\n ON customers.region_id = regions.id\n)\nSELECT _base.`orders___status`, _cm_orders__customers__spend_wscaled_sum.`orders___customers___spend_wscaled_sum`\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_wscaled_sum", - "cm/fragment_default_crossing::duckdb": "WITH _base AS (\nSELECT\n orders.status AS \"orders.status\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__customers__spend_wscaled_sum AS (\nSELECT\n SUM(customers.spend * regions.weight) AS \"orders.customers.spend_wscaled_sum\"\nFROM customers AS customers\nLEFT JOIN regions AS regions\n ON customers.region_id = regions.id\n)\nSELECT _base.\"orders.status\", _cm_orders__customers__spend_wscaled_sum.\"orders.customers.spend_wscaled_sum\"\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_wscaled_sum", - "cm/fragment_default_crossing::postgres": "WITH _base AS (\nSELECT\n orders.status AS \"orders.status\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__customers__spend_wscaled_sum AS (\nSELECT\n SUM(customers.spend * regions.weight) AS \"orders.customers.spend_wscaled_sum\"\nFROM customers AS customers\nLEFT JOIN regions AS regions\n ON customers.region_id = regions.id\n)\nSELECT _base.\"orders.status\", _cm_orders__customers__spend_wscaled_sum.\"orders.customers.spend_wscaled_sum\"\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_wscaled_sum", - "cm/fragment_default_crossing::sqlite": "WITH _base AS (\nSELECT\n orders.status AS \"orders.status\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__customers__spend_wscaled_sum AS (\nSELECT\n SUM(customers.spend * regions.weight) AS \"orders.customers.spend_wscaled_sum\"\nFROM customers AS customers\nLEFT JOIN regions AS regions\n ON customers.region_id = regions.id\n)\nSELECT _base.\"orders.status\", _cm_orders__customers__spend_wscaled_sum.\"orders.customers.spend_wscaled_sum\"\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_wscaled_sum", - "cm/fragment_default_crossing::tsql": "WITH _base AS (\nSELECT\n orders.status AS [orders___status]\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__customers__spend_wscaled_sum AS (\nSELECT\n SUM(customers.spend * regions.weight) AS [orders___customers___spend_wscaled_sum]\nFROM customers AS customers\nLEFT JOIN regions AS regions\n ON customers.region_id = regions.id\n)\nSELECT _base.[orders___status], _cm_orders__customers__spend_wscaled_sum.[orders___customers___spend_wscaled_sum]\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_wscaled_sum", - "cm/joined_measure::bigquery": "WITH _base AS (\nSELECT\n orders.status AS `orders___status`\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__customers__spend_sum AS (\nSELECT\n SUM(customers.spend) AS `orders___customers___spend_sum`\nFROM customers AS customers\n)\nSELECT _base.`orders___status`, _cm_orders__customers__spend_sum.`orders___customers___spend_sum`\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_sum", - "cm/joined_measure::duckdb": "WITH _base AS (\nSELECT\n orders.status AS \"orders.status\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__customers__spend_sum AS (\nSELECT\n SUM(customers.spend) AS \"orders.customers.spend_sum\"\nFROM customers AS customers\n)\nSELECT _base.\"orders.status\", _cm_orders__customers__spend_sum.\"orders.customers.spend_sum\"\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_sum", - "cm/joined_measure::postgres": "WITH _base AS (\nSELECT\n orders.status AS \"orders.status\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__customers__spend_sum AS (\nSELECT\n SUM(customers.spend) AS \"orders.customers.spend_sum\"\nFROM customers AS customers\n)\nSELECT _base.\"orders.status\", _cm_orders__customers__spend_sum.\"orders.customers.spend_sum\"\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_sum", - "cm/joined_measure::sqlite": "WITH _base AS (\nSELECT\n orders.status AS \"orders.status\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__customers__spend_sum AS (\nSELECT\n SUM(customers.spend) AS \"orders.customers.spend_sum\"\nFROM customers AS customers\n)\nSELECT _base.\"orders.status\", _cm_orders__customers__spend_sum.\"orders.customers.spend_sum\"\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_sum", - "cm/joined_measure::tsql": "WITH _base AS (\nSELECT\n orders.status AS [orders___status]\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__customers__spend_sum AS (\nSELECT\n SUM(customers.spend) AS [orders___customers___spend_sum]\nFROM customers AS customers\n)\nSELECT _base.[orders___status], _cm_orders__customers__spend_sum.[orders___customers___spend_sum]\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_sum", - "cm/outer_where_wrapper::bigquery": "WITH _base AS (\nSELECT\n orders.status AS `orders___status`\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__eu_amount_sum AS (\nSELECT\n orders.status AS `orders___status`,\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS FLOAT64) AS `orders___eu`\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n)\nSELECT _base.`orders___status`, _cm_orders__eu_amount_sum.`orders___eu`\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum ON _base.`orders___status` IS NOT DISTINCT FROM _cm_orders__eu_amount_sum.`orders___status`\nWHERE _cm_orders__eu_amount_sum.`orders___eu` > 100", - "cm/outer_where_wrapper::duckdb": "WITH _base AS (\nSELECT\n orders.status AS \"orders.status\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__eu_amount_sum AS (\nSELECT\n orders.status AS \"orders.status\",\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS DOUBLE) AS \"orders.eu\"\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n)\nSELECT _base.\"orders.status\", _cm_orders__eu_amount_sum.\"orders.eu\"\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum ON _base.\"orders.status\" IS NOT DISTINCT FROM _cm_orders__eu_amount_sum.\"orders.status\"\nWHERE _cm_orders__eu_amount_sum.\"orders.eu\" > 100", - "cm/outer_where_wrapper::postgres": "WITH _base AS (\nSELECT\n orders.status AS \"orders.status\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__eu_amount_sum AS (\nSELECT\n orders.status AS \"orders.status\",\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS DOUBLE PRECISION) AS \"orders.eu\"\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n)\nSELECT _base.\"orders.status\", _cm_orders__eu_amount_sum.\"orders.eu\"\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum ON _base.\"orders.status\" IS NOT DISTINCT FROM _cm_orders__eu_amount_sum.\"orders.status\"\nWHERE _cm_orders__eu_amount_sum.\"orders.eu\" > 100", - "cm/outer_where_wrapper::sqlite": "WITH _base AS (\nSELECT\n orders.status AS \"orders.status\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__eu_amount_sum AS (\nSELECT\n orders.status AS \"orders.status\",\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS REAL) AS \"orders.eu\"\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n)\nSELECT _base.\"orders.status\", _cm_orders__eu_amount_sum.\"orders.eu\"\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum ON _base.\"orders.status\" IS _cm_orders__eu_amount_sum.\"orders.status\"\nWHERE _cm_orders__eu_amount_sum.\"orders.eu\" > 100", - "cm/outer_where_wrapper::tsql": "WITH _base AS (\nSELECT\n orders.status AS [orders___status]\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__eu_amount_sum AS (\nSELECT\n orders.status AS [orders___status],\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS FLOAT) AS [orders___eu]\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n)\nSELECT _base.[orders___status], _cm_orders__eu_amount_sum.[orders___eu]\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum ON (_base.[orders___status] = _cm_orders__eu_amount_sum.[orders___status] OR (_base.[orders___status] IS NULL AND _cm_orders__eu_amount_sum.[orders___status] IS NULL))\nWHERE _cm_orders__eu_amount_sum.[orders___eu] > 100", + "cm/fragment_default_crossing::bigquery": "WITH _base AS (\n SELECT\n orders.status AS `orders___status`\n FROM orders AS orders\n WHERE\n orders.amount >= 0\n GROUP BY\n orders.status\n), _cm_orders__customers__spend_wscaled_sum AS (\n SELECT\n SUM(customers.spend * regions.weight) AS `orders___customers___spend_wscaled_sum`\n FROM customers AS customers\n LEFT JOIN regions AS regions\n ON customers.region_id = regions.id\n)\nSELECT\n _base.`orders___status`,\n _cm_orders__customers__spend_wscaled_sum.`orders___customers___spend_wscaled_sum`\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_wscaled_sum", + "cm/fragment_default_crossing::duckdb": "WITH _base AS (\n SELECT\n orders.status AS \"orders.status\"\n FROM orders AS orders\n WHERE\n orders.amount >= 0\n GROUP BY\n orders.status\n), _cm_orders__customers__spend_wscaled_sum AS (\n SELECT\n SUM(customers.spend * regions.weight) AS \"orders.customers.spend_wscaled_sum\"\n FROM customers AS customers\n LEFT JOIN regions AS regions\n ON customers.region_id = regions.id\n)\nSELECT\n _base.\"orders.status\",\n _cm_orders__customers__spend_wscaled_sum.\"orders.customers.spend_wscaled_sum\"\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_wscaled_sum", + "cm/fragment_default_crossing::postgres": "WITH _base AS (\n SELECT\n orders.status AS \"orders.status\"\n FROM orders AS orders\n WHERE\n orders.amount >= 0\n GROUP BY\n orders.status\n), _cm_orders__customers__spend_wscaled_sum AS (\n SELECT\n SUM(customers.spend * regions.weight) AS \"orders.customers.spend_wscaled_sum\"\n FROM customers AS customers\n LEFT JOIN regions AS regions\n ON customers.region_id = regions.id\n)\nSELECT\n _base.\"orders.status\",\n _cm_orders__customers__spend_wscaled_sum.\"orders.customers.spend_wscaled_sum\"\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_wscaled_sum", + "cm/fragment_default_crossing::sqlite": "WITH _base AS (\n SELECT\n orders.status AS \"orders.status\"\n FROM orders AS orders\n WHERE\n orders.amount >= 0\n GROUP BY\n orders.status\n), _cm_orders__customers__spend_wscaled_sum AS (\n SELECT\n SUM(customers.spend * regions.weight) AS \"orders.customers.spend_wscaled_sum\"\n FROM customers AS customers\n LEFT JOIN regions AS regions\n ON customers.region_id = regions.id\n)\nSELECT\n _base.\"orders.status\",\n _cm_orders__customers__spend_wscaled_sum.\"orders.customers.spend_wscaled_sum\"\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_wscaled_sum", + "cm/fragment_default_crossing::tsql": "WITH _base AS (\n SELECT\n orders.status AS [orders___status]\n FROM orders AS orders\n WHERE\n orders.amount >= 0\n GROUP BY\n orders.status\n), _cm_orders__customers__spend_wscaled_sum AS (\n SELECT\n SUM(customers.spend * regions.weight) AS [orders___customers___spend_wscaled_sum]\n FROM customers AS customers\n LEFT JOIN regions AS regions\n ON customers.region_id = regions.id\n)\nSELECT\n _base.[orders___status],\n _cm_orders__customers__spend_wscaled_sum.[orders___customers___spend_wscaled_sum]\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_wscaled_sum", + "cm/joined_measure::bigquery": "WITH _base AS (\n SELECT\n orders.status AS `orders___status`\n FROM orders AS orders\n WHERE\n orders.amount >= 0\n GROUP BY\n orders.status\n), _cm_orders__customers__spend_sum AS (\n SELECT\n SUM(customers.spend) AS `orders___customers___spend_sum`\n FROM customers AS customers\n)\nSELECT\n _base.`orders___status`,\n _cm_orders__customers__spend_sum.`orders___customers___spend_sum`\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_sum", + "cm/joined_measure::duckdb": "WITH _base AS (\n SELECT\n orders.status AS \"orders.status\"\n FROM orders AS orders\n WHERE\n orders.amount >= 0\n GROUP BY\n orders.status\n), _cm_orders__customers__spend_sum AS (\n SELECT\n SUM(customers.spend) AS \"orders.customers.spend_sum\"\n FROM customers AS customers\n)\nSELECT\n _base.\"orders.status\",\n _cm_orders__customers__spend_sum.\"orders.customers.spend_sum\"\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_sum", + "cm/joined_measure::postgres": "WITH _base AS (\n SELECT\n orders.status AS \"orders.status\"\n FROM orders AS orders\n WHERE\n orders.amount >= 0\n GROUP BY\n orders.status\n), _cm_orders__customers__spend_sum AS (\n SELECT\n SUM(customers.spend) AS \"orders.customers.spend_sum\"\n FROM customers AS customers\n)\nSELECT\n _base.\"orders.status\",\n _cm_orders__customers__spend_sum.\"orders.customers.spend_sum\"\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_sum", + "cm/joined_measure::sqlite": "WITH _base AS (\n SELECT\n orders.status AS \"orders.status\"\n FROM orders AS orders\n WHERE\n orders.amount >= 0\n GROUP BY\n orders.status\n), _cm_orders__customers__spend_sum AS (\n SELECT\n SUM(customers.spend) AS \"orders.customers.spend_sum\"\n FROM customers AS customers\n)\nSELECT\n _base.\"orders.status\",\n _cm_orders__customers__spend_sum.\"orders.customers.spend_sum\"\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_sum", + "cm/joined_measure::tsql": "WITH _base AS (\n SELECT\n orders.status AS [orders___status]\n FROM orders AS orders\n WHERE\n orders.amount >= 0\n GROUP BY\n orders.status\n), _cm_orders__customers__spend_sum AS (\n SELECT\n SUM(customers.spend) AS [orders___customers___spend_sum]\n FROM customers AS customers\n)\nSELECT\n _base.[orders___status],\n _cm_orders__customers__spend_sum.[orders___customers___spend_sum]\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_sum", + "cm/outer_where_wrapper::bigquery": "WITH _base AS (\n SELECT\n orders.status AS `orders___status`\n FROM orders AS orders\n WHERE\n orders.amount >= 0\n GROUP BY\n orders.status\n), _cm_orders__eu_amount_sum AS (\n SELECT\n orders.status AS `orders___status`,\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS FLOAT64) AS `orders___eu`\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n WHERE\n orders.amount >= 0\n GROUP BY\n orders.status\n)\nSELECT\n _base.`orders___status`,\n _cm_orders__eu_amount_sum.`orders___eu`\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum\n ON _base.`orders___status` IS NOT DISTINCT FROM _cm_orders__eu_amount_sum.`orders___status`\nWHERE\n _cm_orders__eu_amount_sum.`orders___eu` > 100", + "cm/outer_where_wrapper::duckdb": "WITH _base AS (\n SELECT\n orders.status AS \"orders.status\"\n FROM orders AS orders\n WHERE\n orders.amount >= 0\n GROUP BY\n orders.status\n), _cm_orders__eu_amount_sum AS (\n SELECT\n orders.status AS \"orders.status\",\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS DOUBLE) AS \"orders.eu\"\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n WHERE\n orders.amount >= 0\n GROUP BY\n orders.status\n)\nSELECT\n _base.\"orders.status\",\n _cm_orders__eu_amount_sum.\"orders.eu\"\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum\n ON _base.\"orders.status\" IS NOT DISTINCT FROM _cm_orders__eu_amount_sum.\"orders.status\"\nWHERE\n _cm_orders__eu_amount_sum.\"orders.eu\" > 100", + "cm/outer_where_wrapper::postgres": "WITH _base AS (\n SELECT\n orders.status AS \"orders.status\"\n FROM orders AS orders\n WHERE\n orders.amount >= 0\n GROUP BY\n orders.status\n), _cm_orders__eu_amount_sum AS (\n SELECT\n orders.status AS \"orders.status\",\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS DOUBLE PRECISION) AS \"orders.eu\"\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n WHERE\n orders.amount >= 0\n GROUP BY\n orders.status\n)\nSELECT\n _base.\"orders.status\",\n _cm_orders__eu_amount_sum.\"orders.eu\"\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum\n ON _base.\"orders.status\" IS NOT DISTINCT FROM _cm_orders__eu_amount_sum.\"orders.status\"\nWHERE\n _cm_orders__eu_amount_sum.\"orders.eu\" > 100", + "cm/outer_where_wrapper::sqlite": "WITH _base AS (\n SELECT\n orders.status AS \"orders.status\"\n FROM orders AS orders\n WHERE\n orders.amount >= 0\n GROUP BY\n orders.status\n), _cm_orders__eu_amount_sum AS (\n SELECT\n orders.status AS \"orders.status\",\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS REAL) AS \"orders.eu\"\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n WHERE\n orders.amount >= 0\n GROUP BY\n orders.status\n)\nSELECT\n _base.\"orders.status\",\n _cm_orders__eu_amount_sum.\"orders.eu\"\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum\n ON _base.\"orders.status\" IS _cm_orders__eu_amount_sum.\"orders.status\"\nWHERE\n _cm_orders__eu_amount_sum.\"orders.eu\" > 100", + "cm/outer_where_wrapper::tsql": "WITH _base AS (\n SELECT\n orders.status AS [orders___status]\n FROM orders AS orders\n WHERE\n orders.amount >= 0\n GROUP BY\n orders.status\n), _cm_orders__eu_amount_sum AS (\n SELECT\n orders.status AS [orders___status],\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS FLOAT) AS [orders___eu]\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n WHERE\n orders.amount >= 0\n GROUP BY\n orders.status\n)\nSELECT\n _base.[orders___status],\n _cm_orders__eu_amount_sum.[orders___eu]\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum\n ON (\n _base.[orders___status] = _cm_orders__eu_amount_sum.[orders___status]\n OR (\n _base.[orders___status] IS NULL\n AND _cm_orders__eu_amount_sum.[orders___status] IS NULL\n )\n )\nWHERE\n _cm_orders__eu_amount_sum.[orders___eu] > 100", "expand/derived_of_derived::bigquery": "SELECT\n CAST((\n customers__regions.population * 2\n ) AS FLOAT64) AS `orders___deep_pop`,\n CAST(SUM(orders.amount) AS FLOAT64) AS `orders___m`\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nLEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\nWHERE\n orders.amount >= 0\nGROUP BY\n CAST((\n customers__regions.population * 2\n ) AS FLOAT64)", "expand/derived_of_derived::duckdb": "SELECT\n CAST((\n customers__regions.population * 2\n ) AS DOUBLE) AS \"orders.deep_pop\",\n CAST(SUM(orders.amount) AS DOUBLE) AS \"orders.m\"\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nLEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\nWHERE\n orders.amount >= 0\nGROUP BY\n CAST((\n customers__regions.population * 2\n ) AS DOUBLE)", "expand/derived_of_derived::postgres": "SELECT\n CAST((\n customers__regions.population * 2\n ) AS DOUBLE PRECISION) AS \"orders.deep_pop\",\n CAST(SUM(orders.amount) AS DOUBLE PRECISION) AS \"orders.m\"\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nLEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\nWHERE\n orders.amount >= 0\nGROUP BY\n CAST((\n customers__regions.population * 2\n ) AS DOUBLE PRECISION)", @@ -24,11 +24,11 @@ "expand/multi_model_derived::postgres": "SELECT\n CAST(customers.spend + customers__regions.population AS DOUBLE PRECISION) AS \"orders.multi_model\",\n CAST(SUM(orders.amount) AS DOUBLE PRECISION) AS \"orders.m\"\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nLEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\nWHERE\n orders.amount >= 0\nGROUP BY\n CAST(customers.spend + customers__regions.population AS DOUBLE PRECISION)", "expand/multi_model_derived::sqlite": "SELECT\n CAST(customers.spend + customers__regions.population AS REAL) AS \"orders.multi_model\",\n CAST(SUM(orders.amount) AS REAL) AS \"orders.m\"\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nLEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\nWHERE\n orders.amount >= 0\nGROUP BY\n CAST(customers.spend + customers__regions.population AS REAL)", "expand/multi_model_derived::tsql": "SELECT\n CAST(customers.spend + customers__regions.population AS FLOAT) AS [orders___multi_model],\n CAST(SUM(orders.amount) AS FLOAT) AS [orders___m]\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nLEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\nWHERE\n orders.amount >= 0\nGROUP BY\n CAST(customers.spend + customers__regions.population AS FLOAT)", - "host/column_filter_crossing::bigquery": "WITH _base AS (\nSELECT\n orders.status AS `orders___status`\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__eu_amount_sum AS (\nSELECT\n orders.status AS `orders___status`,\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS FLOAT64) AS `orders___m`\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n)\nSELECT _base.`orders___status`, _cm_orders__eu_amount_sum.`orders___m`\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum ON _base.`orders___status` IS NOT DISTINCT FROM _cm_orders__eu_amount_sum.`orders___status`", - "host/column_filter_crossing::duckdb": "WITH _base AS (\nSELECT\n orders.status AS \"orders.status\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__eu_amount_sum AS (\nSELECT\n orders.status AS \"orders.status\",\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS DOUBLE) AS \"orders.m\"\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n)\nSELECT _base.\"orders.status\", _cm_orders__eu_amount_sum.\"orders.m\"\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum ON _base.\"orders.status\" IS NOT DISTINCT FROM _cm_orders__eu_amount_sum.\"orders.status\"", - "host/column_filter_crossing::postgres": "WITH _base AS (\nSELECT\n orders.status AS \"orders.status\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__eu_amount_sum AS (\nSELECT\n orders.status AS \"orders.status\",\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS DOUBLE PRECISION) AS \"orders.m\"\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n)\nSELECT _base.\"orders.status\", _cm_orders__eu_amount_sum.\"orders.m\"\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum ON _base.\"orders.status\" IS NOT DISTINCT FROM _cm_orders__eu_amount_sum.\"orders.status\"", - "host/column_filter_crossing::sqlite": "WITH _base AS (\nSELECT\n orders.status AS \"orders.status\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__eu_amount_sum AS (\nSELECT\n orders.status AS \"orders.status\",\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS REAL) AS \"orders.m\"\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n)\nSELECT _base.\"orders.status\", _cm_orders__eu_amount_sum.\"orders.m\"\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum ON _base.\"orders.status\" IS _cm_orders__eu_amount_sum.\"orders.status\"", - "host/column_filter_crossing::tsql": "WITH _base AS (\nSELECT\n orders.status AS [orders___status]\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n), _cm_orders__eu_amount_sum AS (\nSELECT\n orders.status AS [orders___status],\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS FLOAT) AS [orders___m]\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.amount >= 0\nGROUP BY\n orders.status\n)\nSELECT _base.[orders___status], _cm_orders__eu_amount_sum.[orders___m]\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum ON (_base.[orders___status] = _cm_orders__eu_amount_sum.[orders___status] OR (_base.[orders___status] IS NULL AND _cm_orders__eu_amount_sum.[orders___status] IS NULL))", + "host/column_filter_crossing::bigquery": "WITH _base AS (\n SELECT\n orders.status AS `orders___status`\n FROM orders AS orders\n WHERE\n orders.amount >= 0\n GROUP BY\n orders.status\n), _cm_orders__eu_amount_sum AS (\n SELECT\n orders.status AS `orders___status`,\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS FLOAT64) AS `orders___m`\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n WHERE\n orders.amount >= 0\n GROUP BY\n orders.status\n)\nSELECT\n _base.`orders___status`,\n _cm_orders__eu_amount_sum.`orders___m`\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum\n ON _base.`orders___status` IS NOT DISTINCT FROM _cm_orders__eu_amount_sum.`orders___status`", + "host/column_filter_crossing::duckdb": "WITH _base AS (\n SELECT\n orders.status AS \"orders.status\"\n FROM orders AS orders\n WHERE\n orders.amount >= 0\n GROUP BY\n orders.status\n), _cm_orders__eu_amount_sum AS (\n SELECT\n orders.status AS \"orders.status\",\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS DOUBLE) AS \"orders.m\"\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n WHERE\n orders.amount >= 0\n GROUP BY\n orders.status\n)\nSELECT\n _base.\"orders.status\",\n _cm_orders__eu_amount_sum.\"orders.m\"\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum\n ON _base.\"orders.status\" IS NOT DISTINCT FROM _cm_orders__eu_amount_sum.\"orders.status\"", + "host/column_filter_crossing::postgres": "WITH _base AS (\n SELECT\n orders.status AS \"orders.status\"\n FROM orders AS orders\n WHERE\n orders.amount >= 0\n GROUP BY\n orders.status\n), _cm_orders__eu_amount_sum AS (\n SELECT\n orders.status AS \"orders.status\",\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS DOUBLE PRECISION) AS \"orders.m\"\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n WHERE\n orders.amount >= 0\n GROUP BY\n orders.status\n)\nSELECT\n _base.\"orders.status\",\n _cm_orders__eu_amount_sum.\"orders.m\"\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum\n ON _base.\"orders.status\" IS NOT DISTINCT FROM _cm_orders__eu_amount_sum.\"orders.status\"", + "host/column_filter_crossing::sqlite": "WITH _base AS (\n SELECT\n orders.status AS \"orders.status\"\n FROM orders AS orders\n WHERE\n orders.amount >= 0\n GROUP BY\n orders.status\n), _cm_orders__eu_amount_sum AS (\n SELECT\n orders.status AS \"orders.status\",\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS REAL) AS \"orders.m\"\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n WHERE\n orders.amount >= 0\n GROUP BY\n orders.status\n)\nSELECT\n _base.\"orders.status\",\n _cm_orders__eu_amount_sum.\"orders.m\"\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum\n ON _base.\"orders.status\" IS _cm_orders__eu_amount_sum.\"orders.status\"", + "host/column_filter_crossing::tsql": "WITH _base AS (\n SELECT\n orders.status AS [orders___status]\n FROM orders AS orders\n WHERE\n orders.amount >= 0\n GROUP BY\n orders.status\n), _cm_orders__eu_amount_sum AS (\n SELECT\n orders.status AS [orders___status],\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS FLOAT) AS [orders___m]\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n WHERE\n orders.amount >= 0\n GROUP BY\n orders.status\n)\nSELECT\n _base.[orders___status],\n _cm_orders__eu_amount_sum.[orders___m]\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum\n ON (\n _base.[orders___status] = _cm_orders__eu_amount_sum.[orders___status]\n OR (\n _base.[orders___status] IS NULL\n AND _cm_orders__eu_amount_sum.[orders___status] IS NULL\n )\n )", "host/column_sql_derived::bigquery": "SELECT\n CAST(orders.amount * 2 AS FLOAT64) AS `orders___doubled`,\n CAST(SUM(orders.amount) AS FLOAT64) AS `orders___m`\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n CAST(orders.amount * 2 AS FLOAT64)", "host/column_sql_derived::duckdb": "SELECT\n CAST(orders.amount * 2 AS DOUBLE) AS \"orders.doubled\",\n CAST(SUM(orders.amount) AS DOUBLE) AS \"orders.m\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n CAST(orders.amount * 2 AS DOUBLE)", "host/column_sql_derived::postgres": "SELECT\n CAST(orders.amount * 2 AS DOUBLE PRECISION) AS \"orders.doubled\",\n CAST(SUM(orders.amount) AS DOUBLE PRECISION) AS \"orders.m\"\nFROM orders AS orders\nWHERE\n orders.amount >= 0\nGROUP BY\n CAST(orders.amount * 2 AS DOUBLE PRECISION)", @@ -64,9 +64,9 @@ "windowed/date_range_filter::postgres": "SELECT\n DATE_TRUNC('MONTH', orders.created_at) AS \"orders.created_at\",\n CAST(SUM(orders.amount) AS DOUBLE PRECISION) AS \"orders.m\"\nFROM orders AS orders\nWHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\nGROUP BY\n DATE_TRUNC('MONTH', orders.created_at)", "windowed/date_range_filter::sqlite": "SELECT\n STRFTIME('%Y-%m-01', orders.created_at) AS \"orders.created_at\",\n CAST(SUM(orders.amount) AS REAL) AS \"orders.m\"\nFROM orders AS orders\nWHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\nGROUP BY\n STRFTIME('%Y-%m-01', orders.created_at)", "windowed/date_range_filter::tsql": "SELECT\n DATETRUNC(month, orders.created_at) AS [orders___created_at],\n CAST(SUM(orders.amount) AS FLOAT) AS [orders___m]\nFROM orders AS orders\nWHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\nGROUP BY\n DATETRUNC(month, orders.created_at)", - "windowed/src_scope::bigquery": "WITH _base AS (\nSELECT\n DATE_TRUNC(orders.created_at, MONTH) AS `orders___created_at`\nFROM orders AS orders\nWHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\nGROUP BY\n DATE_TRUNC(orders.created_at, MONTH)\n), _cm_orders__eu_amount_sum AS (\nSELECT\n DATE_TRUNC(orders.created_at, MONTH) AS `orders___created_at`,\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS FLOAT64) AS `orders___m`\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\nGROUP BY\n DATE_TRUNC(orders.created_at, MONTH)\n)\nSELECT _base.`orders___created_at`, _cm_orders__eu_amount_sum.`orders___m`\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum ON _base.`orders___created_at` IS NOT DISTINCT FROM _cm_orders__eu_amount_sum.`orders___created_at`", - "windowed/src_scope::duckdb": "WITH _base AS (\nSELECT\n DATE_TRUNC('MONTH', orders.created_at) AS \"orders.created_at\"\nFROM orders AS orders\nWHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\nGROUP BY\n DATE_TRUNC('MONTH', orders.created_at)\n), _cm_orders__eu_amount_sum AS (\nSELECT\n DATE_TRUNC('MONTH', orders.created_at) AS \"orders.created_at\",\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS DOUBLE) AS \"orders.m\"\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\nGROUP BY\n DATE_TRUNC('MONTH', orders.created_at)\n)\nSELECT _base.\"orders.created_at\", _cm_orders__eu_amount_sum.\"orders.m\"\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum ON _base.\"orders.created_at\" IS NOT DISTINCT FROM _cm_orders__eu_amount_sum.\"orders.created_at\"", - "windowed/src_scope::postgres": "WITH _base AS (\nSELECT\n DATE_TRUNC('MONTH', orders.created_at) AS \"orders.created_at\"\nFROM orders AS orders\nWHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\nGROUP BY\n DATE_TRUNC('MONTH', orders.created_at)\n), _cm_orders__eu_amount_sum AS (\nSELECT\n DATE_TRUNC('MONTH', orders.created_at) AS \"orders.created_at\",\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS DOUBLE PRECISION) AS \"orders.m\"\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\nGROUP BY\n DATE_TRUNC('MONTH', orders.created_at)\n)\nSELECT _base.\"orders.created_at\", _cm_orders__eu_amount_sum.\"orders.m\"\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum ON _base.\"orders.created_at\" IS NOT DISTINCT FROM _cm_orders__eu_amount_sum.\"orders.created_at\"", - "windowed/src_scope::sqlite": "WITH _base AS (\nSELECT\n STRFTIME('%Y-%m-01', orders.created_at) AS \"orders.created_at\"\nFROM orders AS orders\nWHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\nGROUP BY\n STRFTIME('%Y-%m-01', orders.created_at)\n), _cm_orders__eu_amount_sum AS (\nSELECT\n STRFTIME('%Y-%m-01', orders.created_at) AS \"orders.created_at\",\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS REAL) AS \"orders.m\"\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\nGROUP BY\n STRFTIME('%Y-%m-01', orders.created_at)\n)\nSELECT _base.\"orders.created_at\", _cm_orders__eu_amount_sum.\"orders.m\"\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum ON _base.\"orders.created_at\" IS _cm_orders__eu_amount_sum.\"orders.created_at\"", - "windowed/src_scope::tsql": "WITH _base AS (\nSELECT\n DATETRUNC(month, orders.created_at) AS [orders___created_at]\nFROM orders AS orders\nWHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\nGROUP BY\n DATETRUNC(month, orders.created_at)\n), _cm_orders__eu_amount_sum AS (\nSELECT\n DATETRUNC(month, orders.created_at) AS [orders___created_at],\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS FLOAT) AS [orders___m]\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nWHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\nGROUP BY\n DATETRUNC(month, orders.created_at)\n)\nSELECT _base.[orders___created_at], _cm_orders__eu_amount_sum.[orders___m]\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum ON (_base.[orders___created_at] = _cm_orders__eu_amount_sum.[orders___created_at] OR (_base.[orders___created_at] IS NULL AND _cm_orders__eu_amount_sum.[orders___created_at] IS NULL))" + "windowed/src_scope::bigquery": "WITH _base AS (\n SELECT\n DATE_TRUNC(orders.created_at, MONTH) AS `orders___created_at`\n FROM orders AS orders\n WHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\n GROUP BY\n DATE_TRUNC(orders.created_at, MONTH)\n), _cm_orders__eu_amount_sum AS (\n SELECT\n DATE_TRUNC(orders.created_at, MONTH) AS `orders___created_at`,\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS FLOAT64) AS `orders___m`\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n WHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\n GROUP BY\n DATE_TRUNC(orders.created_at, MONTH)\n)\nSELECT\n _base.`orders___created_at`,\n _cm_orders__eu_amount_sum.`orders___m`\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum\n ON _base.`orders___created_at` IS NOT DISTINCT FROM _cm_orders__eu_amount_sum.`orders___created_at`", + "windowed/src_scope::duckdb": "WITH _base AS (\n SELECT\n DATE_TRUNC('MONTH', orders.created_at) AS \"orders.created_at\"\n FROM orders AS orders\n WHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\n GROUP BY\n DATE_TRUNC('MONTH', orders.created_at)\n), _cm_orders__eu_amount_sum AS (\n SELECT\n DATE_TRUNC('MONTH', orders.created_at) AS \"orders.created_at\",\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS DOUBLE) AS \"orders.m\"\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n WHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\n GROUP BY\n DATE_TRUNC('MONTH', orders.created_at)\n)\nSELECT\n _base.\"orders.created_at\",\n _cm_orders__eu_amount_sum.\"orders.m\"\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum\n ON _base.\"orders.created_at\" IS NOT DISTINCT FROM _cm_orders__eu_amount_sum.\"orders.created_at\"", + "windowed/src_scope::postgres": "WITH _base AS (\n SELECT\n DATE_TRUNC('MONTH', orders.created_at) AS \"orders.created_at\"\n FROM orders AS orders\n WHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\n GROUP BY\n DATE_TRUNC('MONTH', orders.created_at)\n), _cm_orders__eu_amount_sum AS (\n SELECT\n DATE_TRUNC('MONTH', orders.created_at) AS \"orders.created_at\",\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS DOUBLE PRECISION) AS \"orders.m\"\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n WHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\n GROUP BY\n DATE_TRUNC('MONTH', orders.created_at)\n)\nSELECT\n _base.\"orders.created_at\",\n _cm_orders__eu_amount_sum.\"orders.m\"\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum\n ON _base.\"orders.created_at\" IS NOT DISTINCT FROM _cm_orders__eu_amount_sum.\"orders.created_at\"", + "windowed/src_scope::sqlite": "WITH _base AS (\n SELECT\n STRFTIME('%Y-%m-01', orders.created_at) AS \"orders.created_at\"\n FROM orders AS orders\n WHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\n GROUP BY\n STRFTIME('%Y-%m-01', orders.created_at)\n), _cm_orders__eu_amount_sum AS (\n SELECT\n STRFTIME('%Y-%m-01', orders.created_at) AS \"orders.created_at\",\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS REAL) AS \"orders.m\"\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n WHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\n GROUP BY\n STRFTIME('%Y-%m-01', orders.created_at)\n)\nSELECT\n _base.\"orders.created_at\",\n _cm_orders__eu_amount_sum.\"orders.m\"\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum\n ON _base.\"orders.created_at\" IS _cm_orders__eu_amount_sum.\"orders.created_at\"", + "windowed/src_scope::tsql": "WITH _base AS (\n SELECT\n DATETRUNC(month, orders.created_at) AS [orders___created_at]\n FROM orders AS orders\n WHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\n GROUP BY\n DATETRUNC(month, orders.created_at)\n), _cm_orders__eu_amount_sum AS (\n SELECT\n DATETRUNC(MONTH, orders.created_at) AS [orders___created_at],\n CAST(SUM(CASE WHEN customers.tier = 'eu' THEN orders.amount END) AS FLOAT) AS [orders___m]\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n WHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-12-31' AND orders.amount >= 0\n GROUP BY\n DATETRUNC(MONTH, orders.created_at)\n)\nSELECT\n _base.[orders___created_at],\n _cm_orders__eu_amount_sum.[orders___m]\nFROM _base\nLEFT JOIN _cm_orders__eu_amount_sum\n ON (\n _base.[orders___created_at] = _cm_orders__eu_amount_sum.[orders___created_at]\n OR (\n _base.[orders___created_at] IS NULL\n AND _cm_orders__eu_amount_sum.[orders___created_at] IS NULL\n )\n )" } diff --git a/tests/test_cross_model_rename_dev1448.py b/tests/test_cross_model_rename_dev1448.py index 091970a3..8418175a 100644 --- a/tests/test_cross_model_rename_dev1448.py +++ b/tests/test_cross_model_rename_dev1448.py @@ -61,7 +61,7 @@ from slayer.engine.query_engine import SlayerQueryEngine from slayer.storage.yaml_storage import YAMLStorage -from tests._engine_helpers import _outer_select +from tests._engine_helpers import _norm, _outer_select # --------------------------------------------------------------------------- # Rendered-SQL helpers. @@ -905,8 +905,8 @@ async def test_cross_model_rename_vs_arithmetic_mangled_name_stay_distinct( aliases = _public_projection_aliases(sql) assert aliases == [ "orders.status", - "orders.revenue_sum / 100", "orders.revenue_sum__div__100", + "orders.revenue_sum / 100", ], ( f"the arithmetic measure and the renamed cross-model measure must " f"stay distinct public keys; got {aliases!r}\nSQL:\n{sql}" @@ -1177,7 +1177,7 @@ async def test_filter_via_user_alias_resolves_to_cross_model_having( assert _public_projection_aliases(sql) == [ "orders.status", "orders.cust_rev", ], sql - assert "HAVING\n SUM(customers.lifetime_revenue) > 100" in sql, ( + assert "HAVING SUM(customers.lifetime_revenue) > 100" in _norm(sql), ( f"the bare user alias must resolve to a HAVING on the cross-model " f"aggregate:\n{sql}" ) @@ -1221,7 +1221,7 @@ async def test_cross_model_filter_colon_form_with_rename_deferred( assert _public_projection_aliases(sql) == [ "orders.status", "orders.cust_rev", ], sql - assert "HAVING\n SUM(customers.lifetime_revenue) > 100" in sql, ( + assert "HAVING SUM(customers.lifetime_revenue) > 100" in _norm(sql), ( f"colon-form cross-model filter must land as HAVING on the " f"cross-model aggregate:\n{sql}" ) diff --git a/tests/test_dev1745_plan_time_routing.py b/tests/test_dev1745_plan_time_routing.py index f7024430..8ddc8700 100644 --- a/tests/test_dev1745_plan_time_routing.py +++ b/tests/test_dev1745_plan_time_routing.py @@ -29,7 +29,7 @@ from slayer.engine.source_bundle import ResolvedSourceBundle from slayer.engine.stage_planner import plan_query -from tests._engine_helpers import _engine_generate +from tests._engine_helpers import _norm, _engine_generate # --------------------------------------------------------------------------- # @@ -134,11 +134,14 @@ async def _sql(self, query: SlayerQuery) -> str: # The predicate applied to the JOINED-BACK ``_cm_`` column on the outer, # non-aggregating SELECT — the shape this routing exists to produce, and # one nothing else in the query emits. + # Whitespace-normalised: the combined statement is emitted by sqlglot's + # printer, which breaks after ``WHERE``. The claim is about the predicate + # landing on the outer SELECT against the joined-back column, not layout. OUTER_WHERE = 'WHERE _cm_orders__eu_amount_sum."orders.eu" > 100' async def test_outer_where_is_emitted_for_the_isolated_shape(self) -> None: sql = await self._sql(_outer_where_query()) - assert self.OUTER_WHERE in sql, sql + assert self.OUTER_WHERE in _norm(sql), sql async def test_clearing_the_plan_field_removes_the_outer_where(self) -> None: """P-D: the plan is authoritative. A generator that re-walks the @@ -161,7 +164,7 @@ async def test_clearing_the_plan_field_removes_the_outer_where(self) -> None: cleared = planned.model_copy(update={"outer_where_filter_ids": []}) gen = SQLGenerator(dialect="postgres") sql = gen.generate_from_planned(planned_query=cleared, bundle=_bundle()) - assert self.OUTER_WHERE not in sql, ( + assert self.OUTER_WHERE not in _norm(sql), ( "the generator re-derived the outer-WHERE routing instead of " f"consuming the plan:\n{sql}" ) diff --git a/tests/test_dev1746_cte_assembly.py b/tests/test_dev1746_cte_assembly.py index e03751a8..0956174c 100644 --- a/tests/test_dev1746_cte_assembly.py +++ b/tests/test_dev1746_cte_assembly.py @@ -301,15 +301,22 @@ async def test_the_assembler_is_what_builds_the_cross_model_chain( it receives EXPLICIT dependency metadata (Codex D3) rather than being handed a bare list to sort by itself. """ - mod = TestWithChainAssembler._mod() + from slayer.sql import generator as generator_mod + calls: list = [] - original = mod.assemble_with_chain + original = generator_mod.assemble_with_chain def _wrapped(*, entries, final, **kwargs): calls.append(list(entries)) return original(entries=entries, final=final, **kwargs) - monkeypatch.setattr(mod, "assemble_with_chain", _wrapped, raising=True) + # Patch the GENERATOR's binding, not the defining module's: the + # generator imports the symbol directly (imports live at the top of the + # file), so rebinding the source module would leave production calling + # the original and the spy would record nothing. + monkeypatch.setattr( + generator_mod, "assemble_with_chain", _wrapped, raising=True, + ) sql = await _gen(_mixed_query(), dialect="postgres") assert calls, ( "the cross-model WITH chain was assembled without the shared " diff --git a/tests/test_sql_generator.py b/tests/test_sql_generator.py index e683e38c..1e2e0468 100644 --- a/tests/test_sql_generator.py +++ b/tests/test_sql_generator.py @@ -9816,7 +9816,7 @@ async def test_order_by_projected_composite_over_isolated_resolves_at_combined( assert "Loss_Payment" not in base_body assert "Loss_Reserve" not in base_body # ORDER BY must use the bare combined alias, NOT _base."". - order_match = _re.search(r"ORDER BY[^\n]+", sql) + order_match = _re.search(r"ORDER BY\s+[^\n]+", sql) assert order_match, f"Expected ORDER BY in:\n{sql}" order_clause = order_match.group(0) assert "total_loss" in order_clause, ( From 901eecf68bc33d4aa34a1e4f98dea518600bd0a8 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Thu, 6 Aug 2026 13:24:41 +0200 Subject: [PATCH 41/98] DEV-1745: hoist the remaining test-local imports to module scope MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The last CodeRabbit nitpick: AggregateKey / ColumnKey / SQLGenerator in test_dev1745_fragment_joins.py, and Decimal / ScalarCallKey / SlayerQuery / plan_query in test_dev1745_reachability.py were imported inside test bodies. None of them need to be — CLAUDE.md wants imports at the top. Co-Authored-By: Claude Opus 5 (1M context) --- tests/test_dev1745_fragment_joins.py | 5 ++--- tests/test_dev1745_reachability.py | 12 +++++------- 2 files changed, 7 insertions(+), 10 deletions(-) diff --git a/tests/test_dev1745_fragment_joins.py b/tests/test_dev1745_fragment_joins.py index b1695471..c4a4d554 100644 --- a/tests/test_dev1745_fragment_joins.py +++ b/tests/test_dev1745_fragment_joins.py @@ -23,6 +23,7 @@ import pytest from slayer.core.enums import DataType +from slayer.core.keys import AggregateKey, ColumnKey from slayer.core.models import ( Aggregation, AggregationParam, @@ -31,6 +32,7 @@ SlayerModel, ) from slayer.core.query import SlayerQuery +from slayer.sql.generator import SQLGenerator from tests._engine_helpers import _engine_generate @@ -131,9 +133,6 @@ class TestOnlySubstitutedKwargsAreSql: @staticmethod def _entered_fragments(*, kwargs, agg="sum") -> list: - from slayer.core.keys import AggregateKey, ColumnKey - from slayer.sql.generator import SQLGenerator - gen = SQLGenerator(dialect="postgres") seen: list = [] gen._enter_mode_a_expression = ( # type: ignore[method-assign] diff --git a/tests/test_dev1745_reachability.py b/tests/test_dev1745_reachability.py index f9e170b8..b0d80dc4 100644 --- a/tests/test_dev1745_reachability.py +++ b/tests/test_dev1745_reachability.py @@ -25,6 +25,8 @@ from __future__ import annotations +from decimal import Decimal + import pytest from slayer.core.enums import DataType @@ -37,9 +39,12 @@ InKey, LiteralKey, Phase, + ScalarCallKey, ) from slayer.core.models import Column, ModelJoin, SlayerModel +from slayer.core.query import SlayerQuery from slayer.engine.source_bundle import ResolvedSourceBundle +from slayer.engine.stage_planner import plan_query # --------------------------------------------------------------------------- # @@ -435,10 +440,6 @@ def test_decimal_aggregate_kwarg_is_scalar(self) -> None: assert ("customers",) in _paths_for(key) def test_decimal_scalar_call_arg_is_scalar(self) -> None: - from decimal import Decimal - - from slayer.core.keys import ScalarCallKey - key = ScalarCallKey( name="round", args=(ColumnKey(path=("customers",), leaf="balance"), Decimal("2")), @@ -456,9 +457,6 @@ def test_string_and_bool_args_are_scalars(self) -> None: def test_parametric_aggregate_filter_plans(self) -> None: """End-to-end: the shape that crashed. A filter over a parametric aggregate must plan, not raise.""" - from slayer.core.query import SlayerQuery - from slayer.engine.stage_planner import plan_query - planned = plan_query( query=SlayerQuery( source_model="orders", From 6f2ae85310dc59ebde1016c1da10e17b6ecbd2c8 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Thu, 6 Aug 2026 13:42:50 +0200 Subject: [PATCH 42/98] DEV-1745: hoist PlannedQuery import and name the expected frame-bound column MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both from CodeRabbit's latest pass, both introduced by my previous commit. The PlannedQuery import landed inside the test when I replaced the vacuous hasattr check; it belongs at module scope. test_frame_bound_columns_covers_the_time_dimension asserted only that the list was non-empty. The query has exactly one time dimension, so a plan carrying some OTHER column would have satisfied that while getting the frame-bound set wrong — the same weakness as its sibling, which I had just fixed. It now names created_at. Co-Authored-By: Claude Opus 5 (1M context) --- tests/test_dev1745_plan_time_routing.py | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/tests/test_dev1745_plan_time_routing.py b/tests/test_dev1745_plan_time_routing.py index 36101d8b..0417218c 100644 --- a/tests/test_dev1745_plan_time_routing.py +++ b/tests/test_dev1745_plan_time_routing.py @@ -26,6 +26,7 @@ from slayer.core.enums import DataType, TimeGranularity from slayer.core.models import Column, ModelJoin, SlayerModel from slayer.core.query import SlayerQuery +from slayer.engine.planned import PlannedQuery from slayer.engine.source_bundle import ResolvedSourceBundle from slayer.engine.stage_planner import plan_query @@ -186,13 +187,16 @@ def test_plan_carries_frame_bound_columns(self) -> None: """A DECLARED field, checked the same way as outer_where_filter_ids. ``hasattr`` is always true for a field with a default_factory, so it could not fail regardless of planner behaviour.""" - from slayer.engine.planned import PlannedQuery - assert "frame_bound_columns" in PlannedQuery.model_fields def test_frame_bound_columns_covers_the_time_dimension(self) -> None: + """Names the expected column, not just "non-empty" — the query has one + time dimension, so a plan carrying some OTHER column would satisfy a + truthiness check while getting the frame-bound set wrong.""" planned = plan_query(query=self._windowed_query(), bundle=_bundle()) - assert planned.frame_bound_columns, ( - "the time dimension's raw column must be carried on the plan so " - "both strip_frame_bounds call sites read the SAME set" + leaves = {getattr(k, "leaf", None) for k in planned.frame_bound_columns} + assert "created_at" in leaves, ( + f"the time dimension's raw column must be carried on the plan so " + f"both strip_frame_bounds call sites read the SAME set; got " + f"{planned.frame_bound_columns!r}" ) From 8de252219c9025c10e24428a8b62c115392d4934 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Thu, 6 Aug 2026 13:53:21 +0200 Subject: [PATCH 43/98] =?UTF-8?q?DEV-1746=20stages=204-5=20+=20=C2=A75.1:?= =?UTF-8?q?=20plan-order=20carry=20lists,=20the=20projection=20invariant,?= =?UTF-8?q?=20the=20empty-base=20node,=20and=20one=20materialiser?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit B8 — inner-stage carry lists follow plan order. Eight sites projected their carried aliases as sorted(...); one still carried the comment "matches legacy _generate_with_computed:1607", i.e. it was byte-parity ballast. Alphabetical order is unrelated to anything the query means and made a step CTE project its columns in a different order from the base it selects them from. They now share one helper. aliases_by_slot_id is populated as slots are rendered, so its insertion order IS plan order; iterating it directly is what plan order means here. §5.2 — the projection invariant, with one belt. PlannedQuery now validates that its projection contains no hidden slot and no slot more often than it has declared names. The second half is subtler than it looks: a slot may legitimately repeat, because C13 lets one key be selected under several names and the plan lists it once per name — the tests caught an assumption that duplicates were always wrong. Because pydantic's model_copy(update=...) skips validators and rerooting uses exactly that, ONE renderer-side assertion is kept at the single entry point every render path passes through. It raises rather than skipping the slot: silently dropping a column the plan asked for is how a wrong answer reaches a user. §5.12 — the empty-base grain is a typed plan node. EmptyBaseGrainPlan carries the host-local filter ids; its presence is the discriminator. The generator consumed its own render order and re-walked filters_by_phase to rediscover both; it now reads the node and applies exactly the filters marked host-local. No grain_slot_ids field: in this shape the grain is empty by definition. The invariant that would have encoded is asserted instead — and it had to be restated during implementation, because CrossModelAggregatePlan.shared_grain_slots can name a HIDDEN row slot a filter created, which never becomes a projected grain column. Emitted SQL is byte-identical, pinned verbatim. §5.1 — one materialiser on the cross-model path. The two direct sites in _render_cross_model_cte (a crossing grain, a crossing value) now materialise through the producing scope instead of an ad-hoc allocate_val() plus a local alias map. ScopeFrame gains materialize_for(expr, consumer=...): resolve() anchors a ref and closes it, this is the second half alone, for a producer that built its template itself. The cross-model CTE needs it — its templates are a date-truncated grain and a value carrying its column's declared CAST, and re-deriving those by anchoring a ref would project a different expression than the aggregate consumes. Same table, same dedup key, same aliases as resolve. That unification fixes a defect the tests surfaced: grouping a first/last cross-model aggregate by the same crossing expression it aggregates projected that expression TWICE (`regions.weight AS _val_0` and `AS _val_1`), because the grain site and the value site kept separate dedup maps. One table per scope, one projection. Newly surfaced divergence, on the approval list. _build_first_last_base_select keeps its own flow until PR 5 replaces it with RankedAggregatePlan; the boundary is pinned by a test that fails when it moves. Integration churn (executed, both engines): the windowed NULL-dimension test asserted the OLD behavior in its NAME — a NULL group's windowed value came back NULL — and now asserts the real value B1 produces (Feb 300, Mar 700). test_order_only_windowed_region_grain_discriminates lost its discriminator as a side effect: it relied on NULL groups having NULL rolling values, and with real values the rolling and plain-sum orderings coincide on that seed. The discrimination is now structural too — the ORDER BY must reference the windowed CTE, which a dropped window cannot satisfy however the rows tie. 10943 unit + 508 integration passing; ruff clean. Co-Authored-By: Claude Opus 5 (1M context) --- slayer/engine/planned.py | 83 +++++++- slayer/engine/stage_planner.py | 61 ++++++ slayer/sql/generator.py | 191 ++++++++++++------ slayer/sql/scope.py | 17 ++ .../test_integration_windowed_measures.py | 83 +++++--- .../test_dev1746_consumer_materialization.py | 85 +++++--- tests/test_dev1746_empty_base_plan.py | 34 +++- tests/test_dev1746_projection_order.py | 36 +++- 8 files changed, 458 insertions(+), 132 deletions(-) diff --git a/slayer/engine/planned.py b/slayer/engine/planned.py index 4e6e6981..a706a757 100644 --- a/slayer/engine/planned.py +++ b/slayer/engine/planned.py @@ -23,7 +23,7 @@ from __future__ import annotations -from typing import List, Optional, Tuple +from typing import Dict, List, Optional, Tuple from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator @@ -61,6 +61,7 @@ "BoundExpr", "BoundFilterId", "CrossModelAggregatePlan", + "EmptyBaseGrainPlan", "FilterPhase", "JoinRequirement", "OrderEntry", @@ -417,6 +418,32 @@ class FilterReachability(BaseModel): has_host_local_ref: bool = False +class EmptyBaseGrainPlan(BaseModel): + """The host base has no columns of its own (DEV-1503, §5.12). + + Set when every projected value is an isolated aggregate — no host row + slots, no host-local aggregates — so ``_base`` has nothing to project and + becomes a one-row spine for the combined ``CROSS JOIN`` to hang off. Its + PRESENCE is the discriminator; there is no ``grain_slot_ids`` field because + in this shape the grain is empty by definition, which is precisely why the + join-back degenerates to a CROSS JOIN. + + ``host_filter_ids`` are the ROW-phase filters that stay host-local (not + routed into a ``_cm_*`` CTE or the outer WHERE). When any exist the spine is + emitted as ``SELECT 1 AS _placeholder FROM WHERE ... LIMIT 1``; + otherwise as a bare ``SELECT 1 AS _placeholder`` with no FROM at all. + + The ``LIMIT 1`` is load-bearing rather than an optimisation: the filtered + form keeps the host FROM so the WHERE can gate the result, but a host FROM + yields N rows and CROSS JOINing N rows to a one-row scalar aggregate would + repeat the answer N times. ``LIMIT 1`` collapses the spine to a single row + while an empty match still yields zero rows overall. The unfiltered form + drops the FROM entirely for the same reason. + """ + + host_filter_ids: List[BoundFilterId] = Field(default_factory=list) + + class PlannedQuery(BaseModel): """The fully typed plan for one query stage (P7). @@ -497,6 +524,60 @@ class PlannedQuery(BaseModel): # parent: the paths only mean anything relative to the root they were # anchored at. Read via ``filter_reachability_for``. filter_reachability: List[FilterReachability] = Field(default_factory=list) + # DEV-1746 (§5.12) — set when the host base has no columns of its own and + # is emitted as a one-row placeholder spine. Decided at plan time; the + # generator consumes it and never re-derives the shape. + empty_base_plan: Optional[EmptyBaseGrainPlan] = None + + @model_validator(mode="after") + def _projection_is_public_and_well_formed(self) -> "PlannedQuery": + """``projection`` is the ONE authoritative public column list (§5.2). + + Every renderer consumes it verbatim, which is what makes hidden-slot + trimming the absence of a step rather than a step. Two ways that could + break, both checked here rather than discovered as wrong SQL: + + * a HIDDEN slot appearing in it — hidden slots carry no public name, so + the renderer would have nothing to alias the column as; + * a slot appearing MORE times than it has declared names. A slot may + legitimately repeat: C13 lets one key be selected under several user + names, and the plan lists it once per name, each occurrence consuming + the next alias. One occurrence too many means a column emitted twice + under the same name. + """ + by_id = { + slot.id: slot + for slot in ( + list(self.row_slots) + + list(self.aggregate_slots) + + list(self.combined_expression_slots) + ) + } + counts: Dict[SlotId, int] = {} + for sid in self.projection: + counts[sid] = counts.get(sid, 0) + 1 + for sid, count in counts.items(): + slot = by_id.get(sid) + if slot is None: + # Slot tables can legitimately be partial in nested plans; the + # renderer resolves what it needs. Only slots we can SEE are + # checked, so this validator never rejects a plan for a reason + # it cannot substantiate. + continue + if slot.hidden: + raise ValueError( + f"hidden slot {sid!r} appears in the public projection; " + f"hidden slots carry no public name and must be absent", + ) + declared = len(slot.public_aliases) or (1 if slot.public_name else 0) + if declared and count > declared: + raise ValueError( + f"slot {sid!r} appears {count} times in the public " + f"projection but declares only {declared} public name(s) " + f"{list(slot.public_aliases) or [slot.public_name]!r} — " + f"the extra occurrence would emit a duplicate column", + ) + return self # ``CrossModelAggregatePlan.rerooted_plan`` is a forward reference to diff --git a/slayer/engine/stage_planner.py b/slayer/engine/stage_planner.py index a59ba322..97d889bb 100644 --- a/slayer/engine/stage_planner.py +++ b/slayer/engine/stage_planner.py @@ -88,6 +88,8 @@ from slayer.engine.planned import ( BoundExpr as PlannedBoundExpr, BoundFilterId, + EmptyBaseGrainPlan, + SlotId, FilterPhase, FilterReachability, OrderEntry, @@ -1528,6 +1530,16 @@ def _windowed_phase(bf: BoundFilter) -> Phase: slots=[*row_slots, *agg_slots, *combined_slots], ) + empty_base_plan = _plan_empty_base_grain( + projection=projection.public_projection, + agg_slots=agg_slots, + cross_model_plans=cross_model_plans, + windowed_plans=windowed_plans, + order_entries=order_entries, + filters_by_phase=filters_by_phase, + outer_where_filter_ids=outer_where_filter_ids, + ) + return PlannedQuery( source_relation=source_relation, row_slots=row_slots, @@ -1548,9 +1560,58 @@ def _windowed_phase(bf: BoundFilter) -> Phase: frame_bound_columns=frame_bound_columns, outer_where_filter_ids=outer_where_filter_ids, filter_reachability=filter_reachability, + empty_base_plan=empty_base_plan, ) +def _plan_empty_base_grain( + *, + projection: List[SlotId], + agg_slots: list, + cross_model_plans: list, + windowed_plans: list, + order_entries: list, + filters_by_phase: list, + outer_where_filter_ids: List[BoundFilterId], +) -> "EmptyBaseGrainPlan | None": + """Decide the DEV-1503 empty-base spine at plan time (§5.12). + + The host base has nothing of its own exactly when every value the query + asks for is an isolated aggregate: no row slots, no host-LOCAL aggregates, + no combined expressions, and nothing ordered that would have to be + materialised there. The generator used to re-derive this from its own + render order; deciding it here keeps the policy on the plan (P-D). + + ``host_filter_ids`` are the ROW-phase filters that remain host-local — not + routed into a ``_cm_*`` CTE and not lifted to the outer WHERE. Without them + the spine would aggregate across host rows the user filtered out. + """ + isolated = {p.aggregate_slot_id for p in cross_model_plans} + isolated |= {p.aggregate_slot_id for p in windowed_plans} + if not projection or any(sid not in isolated for sid in projection): + return None + # A host-LOCAL aggregate would have to be computed in ``_base``, which then + # has a column of its own and is not a placeholder spine. + if any(slot.id not in isolated for slot in agg_slots): + return None + # An order target that is not itself isolated must be materialised in + # ``_base`` too, for the same reason. + if any(entry.slot_id not in isolated for entry in order_entries): + return None + routed: set = set(outer_where_filter_ids) + for plan in cross_model_plans: + routed.update(plan.where_filter_ids) + routed.update(plan.having_filter_ids) + host_filter_ids = [ + fp.id + for fp in filters_by_phase + if fp.phase == Phase.ROW + and fp.id not in routed + and (fp.expression is not None or fp.text is not None) + ] + return EmptyBaseGrainPlan(host_filter_ids=host_filter_ids) + + def _plan_outer_where_filters( *, filters_by_phase: list, cross_model_plans: list, slots: list, ) -> List[BoundFilterId]: diff --git a/slayer/sql/generator.py b/slayer/sql/generator.py index ff03af6a..0c78c117 100644 --- a/slayer/sql/generator.py +++ b/slayer/sql/generator.py @@ -934,6 +934,32 @@ def _parse_cte_body(self, sql: str) -> exp.Expression: """ return sqlglot.parse_one(sql, dialect=self.dialect) + @staticmethod + def _carry_aliases_in_plan_order( + aliases_by_slot_id: Dict[str, List[str]], + ) -> List[str]: + """Aliases an inner stage carries forward, in PLAN order (B8). + + These lists used to be ``sorted(...)`` — one site still carried the + comment "matches legacy ``_generate_with_computed:1607``", i.e. it was + byte-parity ballast rather than a requirement. Alphabetical order is + unrelated to anything the query means, and it made a step CTE project + its columns in a different order from the base it selects them from. + + ``aliases_by_slot_id`` is populated as slots are rendered, so its + insertion order IS the plan's render order; iterating it directly is + what "plan order" means here. Duplicates are dropped (two slots can + share an alias) while preserving first appearance. + """ + out: List[str] = [] + seen: Set[str] = set() + for aliases in aliases_by_slot_id.values(): + for alias in aliases: + if alias not in seen: + seen.add(alias) + out.append(alias) + return out + def _null_safe_join_pair_sql(self, *, left_sql: str, right_sql: str) -> str: """Render one dialect-aware null-safe equality (DEV-1708 / Codex F2) for a grain join-back ``ON`` clause. ``left_sql`` / ``right_sql`` are the @@ -1576,6 +1602,7 @@ def generate_from_planned(self, planned_query, *, bundle) -> str: ``generate_from_planned``) is a self-contained statement and gets its own allocator, with the parent's restored afterwards. """ + self._assert_projection_is_public(planned_query) prev_allocator = getattr(self, "_gen_allocator", None) self._gen_allocator = self._new_allocator() try: @@ -1585,6 +1612,38 @@ def generate_from_planned(self, planned_query, *, bundle) -> str: finally: self._gen_allocator = prev_allocator + @staticmethod + def _assert_projection_is_public(planned_query) -> None: + """The renderer-side belt for the public-projection invariant (§5.2). + + ``PlannedQuery`` validates this at construction, but pydantic's + ``model_copy(update=...)`` skips validators — and rerooting a plan uses + exactly that. So the ONE place every render path passes through checks + it again. It RAISES rather than skipping the offending slot: silently + dropping a column the plan asked for is how a wrong answer reaches a + user, whereas a raise names the slot. + + This is the only such check left; the defensive ``if slot.hidden: + continue`` guards the renderers used to carry are redundant now that + the projection is authoritative. + """ + slots = { + slot.id: slot + for slot in ( + list(planned_query.row_slots) + + list(planned_query.aggregate_slots) + + list(planned_query.combined_expression_slots) + ) + } + for sid in planned_query.projection: + slot = slots.get(sid) + if slot is not None and slot.hidden: + raise ValueError( + f"hidden slot {sid!r} reached the public projection; the " + f"plan's projection must contain only public slots " + f"(a model_copy that skips validation is the usual cause)", + ) + def _generate_from_planned_impl( # NOSONAR(S3776) — top-level dispatch over cross-model / transform-chain / plain branches plus the conditional outer-trim wrap. Each branch is a coherent compilation strategy; extracting would scatter the shared planned_query / slots_by_id / aliases_by_slot_id state across helpers without simplifying anything. self, planned_query, @@ -1854,8 +1913,8 @@ def _generate_from_planned_impl( # NOSONAR(S3776) — top-level dispatch over c step_num += 1 step_name = cte_allocator.allocate_cte(f"step{step_num}") prev_cte = ctes[-1][0] - carry_aliases_sorted = sorted( - a for aliases in aliases_by_slot_id.values() for a in aliases + carry_aliases_sorted = self._carry_aliases_in_plan_order( + aliases_by_slot_id, ) step_parts = [self._quote_ident(a) for a in carry_aliases_sorted] for layer in ready_window: @@ -1954,8 +2013,8 @@ def _generate_from_planned_impl( # NOSONAR(S3776) — top-level dispatch over c step_num += 1 step_name = cte_allocator.allocate_cte(f"step{step_num}") prev_cte = ctes[-1][0] - carry_aliases_sorted = sorted( - a for aliases in aliases_by_slot_id.values() for a in aliases + carry_aliases_sorted = self._carry_aliases_in_plan_order( + aliases_by_slot_id, ) step_parts = [self._quote_ident(a) for a in carry_aliases_sorted] for cslot in unmaterialised: @@ -1991,11 +2050,10 @@ def _generate_from_planned_impl( # NOSONAR(S3776) — top-level dispatch over c ctes.append((step_name, step_sql)) # Inner SELECT inside _outer wrap: ALL carried aliases sorted - # (matches legacy _generate_with_computed:1607). + # in PLAN order (B8 — this list used to be sorted alphabetically to + # match the legacy renderer byte-for-byte). final_cte = ctes[-1][0] - inner_sorted = sorted( - a for aliases in aliases_by_slot_id.values() for a in aliases - ) + inner_sorted = self._carry_aliases_in_plan_order(aliases_by_slot_id) inner_sql = ( "SELECT\n " + _SQL_COL_SEP.join(self._quote_ident(a) for a in inner_sorted) @@ -4434,27 +4492,22 @@ def _add_local_aux_slots( # here (every projected slot is a cross-model aggregate that # owns its own ``FROM `` inside the ``_cm_*`` CTE), so # dropping the host FROM is safe. - empty_base = not base_render_order - if empty_base: - # Routed filter ids — filters applied inside per-plan ``_cm_*`` - # CTEs (where/having) or at the outer combined WHERE wrapper. - # Host-local ROW filters NOT in this set fall through to the - # placeholder via ``DROP_HOST_LOCAL`` routing (forward cross- - # model classifier) or simply by being unrouted; they must - # apply HERE or the query silently aggregates across host - # rows the user filtered out (Codex round 4 / CodeRabbit). - routed_ids: Set[str] = set(outer_where_filter_ids) - for plan in planned_query.cross_model_aggregate_plans: - routed_ids.update(plan.where_filter_ids) - routed_ids.update(plan.having_filter_ids) - from slayer.core.keys import Phase as _Phase - has_host_local_filter = any( - fp.phase == _Phase.ROW - and fp.id not in routed_ids - and (fp.expression is not None or fp.text is not None) + # DEV-1746 (§5.12): the shape, and which filters stay host-local, are + # decided at plan time and consumed here (P-D). The generator used to + # re-derive both from its own render order and a re-walk of + # ``filters_by_phase``. + empty_base_plan = planned_query.empty_base_plan + if empty_base_plan is not None: + # Apply exactly the filters the plan marked host-local; every + # other filter is applied somewhere else (a ``_cm_*`` CTE or the + # outer WHERE) and must be skipped here. + host_filter_ids = set(empty_base_plan.host_filter_ids) + placeholder_skip_ids = { + fp.id for fp in planned_query.filters_by_phase - ) - if has_host_local_filter: + if fp.id not in host_filter_ids + } + if host_filter_ids: # Build the placeholder over the host with WHERE + LIMIT 1. # LIMIT 1 collapses the host rowset to a single row so the # combined CROSS JOIN to the scalar ``_cm_*`` does not @@ -4482,7 +4535,7 @@ def _add_local_aux_slots( self._resolve_where_filter_joins_via_scope( planned_query=planned_query, scope=placeholder_scope, - skip_filter_ids=routed_ids, + skip_filter_ids=placeholder_skip_ids, ) placeholder_from, placeholder_joins = self._build_from_and_joins( source_model=source_model, @@ -4505,7 +4558,7 @@ def _add_local_aux_slots( source_relation=source_relation, source_model=source_model, bundle=bundle, - skip_filter_ids=routed_ids, + skip_filter_ids=placeholder_skip_ids, ) if base_where is not None: base_select = base_select.where(base_where) @@ -5273,8 +5326,8 @@ def _render_cross_model_transform_chain( # NOSONAR(S3776) — pre-existing comp step_num += 1 step_name = cte_allocator.allocate_cte(f"step{step_num}") prev_cte = ctes[-1][0] - carry_aliases_sorted = sorted( - a for aliases in aliases_by_slot_id.values() for a in aliases + carry_aliases_sorted = self._carry_aliases_in_plan_order( + aliases_by_slot_id, ) step_parts = [self._quote_ident(a) for a in carry_aliases_sorted] for layer in ready: @@ -5327,8 +5380,8 @@ def _render_cross_model_transform_chain( # NOSONAR(S3776) — pre-existing comp step_num += 1 step_name = cte_allocator.allocate_cte(f"step{step_num}") prev_cte = ctes[-1][0] - carry_aliases_sorted = sorted( - a for aliases in aliases_by_slot_id.values() for a in aliases + carry_aliases_sorted = self._carry_aliases_in_plan_order( + aliases_by_slot_id, ) step_parts = [self._quote_ident(a) for a in carry_aliases_sorted] for cslot in unmaterialised: @@ -5359,9 +5412,7 @@ def _render_cross_model_transform_chain( # NOSONAR(S3776) — pre-existing comp ctes.append((step_name, step_sql)) final_cte = ctes[-1][0] - inner_sorted = sorted( - a for aliases in aliases_by_slot_id.values() for a in aliases - ) + inner_sorted = self._carry_aliases_in_plan_order(aliases_by_slot_id) inner_sql = ( "SELECT\n " + _SQL_COL_SEP.join(self._quote_ident(a) for a in inner_sorted) @@ -5636,7 +5687,26 @@ def _render_cross_model_cte( # NOSONAR(S3776) — single conceptual unit: share cte_allocator = self._gen_allocator or self._new_allocator() self._reserve_model_column_names(cte_allocator, target_model) is_first_or_last = agg_slot.key.agg in ("first", "last") - grain_extra_projections: List[Tuple[str, exp.Expression]] = [] + # The CTE's own scope, and the scope of the ranked subquery it may + # select FROM. Both are created here, before the grain loop, because a + # crossing grain is materialised in the RANKED scope for the CTE scope + # to consume — Law 2's producer/consumer pair. + cte_scope = ScopeFrame( + scope_id=cte_allocator.next_scope_id(target_relation), + root_model=target_model, + root_relation=target_relation, + bundle=bundle, + dialect=self._dialect, + allocator=cte_allocator, + ) + ranked_scope = ScopeFrame( + scope_id=cte_allocator.next_scope_id(target_relation), + root_model=target_model, + root_relation=target_relation, + bundle=bundle, + dialect=self._dialect, + allocator=cte_allocator, + ) for sid in plan.shared_grain_slots: if sid not in base_projection_ids: continue @@ -5753,9 +5823,13 @@ def _render_cross_model_cte( # NOSONAR(S3776) — single conceptual unit: share ) ) if grain_crosses: - val_alias = cte_allocator.allocate_val() - grain_extra_projections.append((val_alias, col_expr.copy())) - alias_ref = exp.column(val_alias) + # Law 2 through the SCOPE (§5.1): the ranked subquery projects + # the crossing expression and the CTE consumes the alias. The + # scope owns the dedup table, so a value materialised once for + # the grain is not materialised again for the aggregate. + alias_ref = ranked_scope.materialize_for( + col_expr.copy(), consumer=cte_scope, + ) cte_select_columns.append(alias_ref.copy().as_(host_alias)) cte_group_by.append(alias_ref.copy()) cte_partition_exprs.append(col_expr.copy()) @@ -5807,14 +5881,6 @@ def _render_cross_model_cte( # NOSONAR(S3776) — single conceptual unit: share # shares the generation-wide allocator (``cte_allocator``, hoisted above # the grain loop) so ``_val_`` materialisation names (Law 2) are # unique across the host base, the grain projections, and every CTE. - cte_scope = ScopeFrame( - scope_id=cte_allocator.next_scope_id(target_relation), - root_model=target_model, - root_relation=target_relation, - bundle=bundle, - dialect=self._dialect, - allocator=cte_allocator, - ) # DEV-1701: merge the shared-grain derived-TIME-dim crossed joins # collected in the loop above. for _p in shared_grain_join_paths: @@ -6018,9 +6084,9 @@ def _register_filter_join_paths(sql_text: Optional[str]) -> None: # collected in the grain loop, so a first/last aggregate grouped by a # crossing derived grain materialises that grain INSIDE the subquery # too (the outer SELECT / GROUP BY reference the alias). - extra_projections: List[Tuple[str, exp.Expression]] = list( - grain_extra_projections, - ) + extra_projections: List[Tuple[str, exp.Expression]] = [ + (m.alias, m.expr) for m in ranked_scope.materializations + ] if synth.sql: # DEV-1709: materialise the RESOLVED value (qualified + # ``Column.type`` inner CAST for non-bare expressions) and @@ -6037,8 +6103,17 @@ def _register_filter_join_paths(sql_text: Optional[str]) -> None: sql_expr=value_expr, source_relation=target_relation, source_model=target_model, bundle=bundle, ): - val_alias = cte_allocator.allocate_val() - extra_projections.append((val_alias, value_expr)) + # Law 2 through the SCOPE, same as the crossing grain above. + # Because both go through ONE dedup table, grouping a + # first/last aggregate by the very expression it aggregates + # now projects that expression once instead of twice — the + # two sites used to keep separate alias maps. + val_alias = ranked_scope.materialize_for( + value_expr, consumer=cte_scope, + ).name + extra_projections = [ + (m.alias, m.expr) for m in ranked_scope.materializations + ] outer_synth = synth.model_copy(update={"sql": val_alias}) # A HAVING on this same aggregate must reference the alias, # not the raw crossing ref (out of scope in the outer SELECT). @@ -7864,8 +7939,8 @@ def _add_partition(pk_obj, *, where: str) -> None: # then add the shifted measure under EACH of the slot's public # aliases (DEV-1450 C13). prev_cte = ctes[-2][0] # the CTE just before the shifted CTE - carry_aliases_sorted = sorted( - a for aliases in aliases_by_slot_id.values() for a in aliases + carry_aliases_sorted = self._carry_aliases_in_plan_order( + aliases_by_slot_id, ) sjoin_select_parts = [ f'{prev_cte}.{self._quote_ident(a)}' for a in carry_aliases_sorted @@ -8047,8 +8122,8 @@ def _emit_consecutive_periods_ctes_for_planned( # NOSONAR(S3776) — one cohesi # Build the reset CTE. prev_cte = ctes[-1][0] - carry_aliases_sorted = sorted( - a for aliases in aliases_by_slot_id.values() for a in aliases + carry_aliases_sorted = self._carry_aliases_in_plan_order( + aliases_by_slot_id, ) carry_select = ",\n ".join(self._quote_ident(a) for a in carry_aliases_sorted) partition_clause = ( diff --git a/slayer/sql/scope.py b/slayer/sql/scope.py index bf482586..d69bd2a7 100644 --- a/slayer/sql/scope.py +++ b/slayer/sql/scope.py @@ -247,6 +247,23 @@ def _register_join_paths(self, parsed: exp.Expression) -> None: ): self.join_paths.add(path) + def materialize_for( + self, template: exp.Expression, *, consumer: "ScopeFrame", + ) -> exp.Expression: + """Law 2 for an expression the caller has ALREADY anchored. + + :meth:`resolve` anchors a ref and then closes it; this is the second + half on its own, for a producer that built its template itself — a + date-truncated grain, a value carrying its column's declared CAST. Those + cannot be re-derived by anchoring a ref without changing the expression + that gets projected, so they arrive here as AST. + + Same table, same dedup key, same aliases as :meth:`resolve`: there is + one materialisation mechanism per scope, which is the point. + """ + self._register_join_paths(template) + return self._close(template, consumer=consumer) + def _close( self, template: exp.Expression, *, consumer: "ScopeFrame | None", ) -> exp.Expression: diff --git a/tests/integration/test_integration_windowed_measures.py b/tests/integration/test_integration_windowed_measures.py index 645a157c..4cd4028b 100644 --- a/tests/integration/test_integration_windowed_measures.py +++ b/tests/integration/test_integration_windowed_measures.py @@ -220,13 +220,23 @@ async def test_rolling_90d_window_reaches_before_date_range_start( assert result.row_count == 1, result.data assert float(result.data[0][_key(result.data[0], "rev_90d")]) == 900.0, result.data - async def test_null_dimension_group_gets_null_windowed_value( + async def test_null_dimension_group_gets_its_real_windowed_value( self, windowed_engine: SlayerQueryEngine, ) -> None: - """A base group whose dimension value is NULL never matches ``_src`` rows - — the CTE join-back uses plain ``=`` and ``NULL = NULL`` is not TRUE — so - its windowed value comes back NULL. Documented consequence of the - equality join-back (the F1-style pinned semantic), not a bug.""" + """A base group whose dimension is NULL receives its REAL windowed value. + + This used to come back NULL, and was pinned as a documented consequence: + the ``_src`` join-back inside the ``_wm_`` CTE compared the grain with a + plain ``=``, and ``NULL = NULL`` is not TRUE, so the group matched no + rows. The outer join-back and the cross-model join-back were already + null-safe, so the answer depended on which isolation shape a measure + landed in. The inner comparison is now null-safe too, which is what makes + the value appear. + + Executed against real engines because this is a VALUE change, not a + spelling one — and the two dialects spell null-safe equality differently + (``IS NOT DISTINCT FROM`` on DuckDB, bare ``IS`` on SQLite). + """ query = SlayerQuery( source_model="orders", dimensions=[ColumnRef(name="region")], @@ -238,8 +248,16 @@ async def test_null_dimension_group_gets_null_windowed_value( result = await windowed_engine.execute(query=query) null_rows = [r for r in result.data if r[_key(r, "region")] is None] assert null_rows, result.data - for r in null_rows: - assert r[_key(r, "rev_90d")] is None, r + by_month = { + str(r[_key(r, "created_at")])[:7]: float(r[_key(r, "rev_90d")]) + for r in null_rows + } + # The NULL-region rows are 300 (2024-02-15) and 400 (2024-03-15). + # February's 90-day window reaches back over 300 only; March's reaches + # back over both, so the two months differ and neither matches the + # single-month sum by accident. + assert by_month.get("2024-02") == 300.0, result.data + assert by_month.get("2024-03") == 700.0, result.data # Sanity: a non-NULL dimension group still computes a real rolling value # (the US rows, both in January, roll up to 300). us_rows = [r for r in result.data if r[_key(r, "region")] == "US"] @@ -408,20 +426,20 @@ async def test_hidden_and_public_windowed_together( async def test_order_only_windowed_region_grain_discriminates( self, windowed_engine: SlayerQueryEngine, ) -> None: - """The assertion a plain-``SUM`` regression cannot satisfy under ANY - tie-break, and the NULL join-back semantic in one. - - On the ``region`` + month grain the groups are (US, Jan), (NULL, Feb), - (NULL, Mar). A NULL-dimension group never matches ``_src`` (the - join-back is an equality and ``NULL = NULL`` is not TRUE), so its - ROLLING value is NULL — pinned by - ``test_null_dimension_group_gets_null_windowed_value``. The rolling - values are therefore ``US/Jan = 300`` and NULL for both others, so - ``DESC`` puts the US row first on every engine that sorts NULLs last. - - The PLAIN sums are US/Jan 300, NULL/Feb 300, NULL/Mar 400 — all - non-NULL — so a plain-sum regression puts the NULL/Mar row (400) first. - The two orderings cannot coincide. + """The ROLLING value drives the sort on a grain with a NULL dimension. + + This test used to discriminate through the NULL group's value being + NULL: rolling gave ``US/Jan = 300`` and NULL elsewhere, while a plain + ``SUM`` gave ``NULL/Mar = 400`` first, so the two orderings could not + coincide. Now that the ``_src`` join-back is null-safe the NULL groups + carry real rolling values (Feb 300, Mar 700), and on THIS seed the + rolling and plain-sum orderings agree — so the ordering alone no longer + proves the window survived. + + The discrimination is therefore structural as well: the ORDER BY must + reference the windowed CTE's column. A plain-``SUM`` regression (the + DEV-1733 defect, where the window was silently dropped) emits no ``_wm_`` + CTE at all, so it cannot satisfy that no matter how the rows tie. """ query = SlayerQuery( source_model="orders", @@ -432,16 +450,27 @@ async def test_order_only_windowed_region_grain_discriminates( measures=[{"formula": "id:count", "name": "n"}], order=[{"column": "revenue:sum(window='90d')", "direction": "desc"}], ) + sql = (await windowed_engine.execute(query=query, dry_run=True)).sql or "" + assert "_wm_" in sql, ( + f"no windowed CTE was emitted — the window was dropped and the " + f"ORDER BY is over a plain SUM:\n{sql}" + ) + order_at = sql.rfind("ORDER BY") + assert order_at != -1, sql + assert "_wm_" in sql[order_at:], ( + f"the ORDER BY does not reference the windowed CTE:\n{sql}" + ) + result = await windowed_engine.execute(query=query) regions = [r[_key(r, "region")] for r in result.data] assert len(regions) == 3, result.data - assert regions[0] == "US", ( - f"the US/Jan group is the only one with a non-NULL ROLLING value, " - f"so it must sort first; a plain SUM would lead with the NULL/Mar " - f"group (400). got: {regions}\nrows: {result.data}" + # Rolling values: US/Jan 300, NULL/Feb 300, NULL/Mar 700 -> Mar leads. + assert regions[0] is None, ( + f"the NULL/Mar group has the largest ROLLING value (700, its own " + f"400 plus February's 300 inside the 90-day window), so it must " + f"sort first. got: {regions}\nrows: {result.data}" ) - assert regions[1] is None, result.data - assert regions[2] is None, result.data + assert set(regions[1:]) == {"US", None}, result.data assert all("90d" not in c for c in result.columns), result.columns diff --git a/tests/test_dev1746_consumer_materialization.py b/tests/test_dev1746_consumer_materialization.py index bfd46e8c..79b3a71f 100644 --- a/tests/test_dev1746_consumer_materialization.py +++ b/tests/test_dev1746_consumer_materialization.py @@ -127,24 +127,33 @@ def _two_distinct_crossing_values_query() -> SlayerQuery: class _ResolveSpy: - """Records every ``ScopeFrame.resolve`` call and whether it named a consumer. - - A spy rather than a grep: the point of §5.1 is that the branch runs on the - PRODUCTION path, which only an observed call can establish. + """Records every close of a scoped value and whether it named a consumer. + + Spies on ``ScopeFrame._close`` — the Law-2 branch itself — rather than on + ``resolve``. Both public entry points funnel through it: ``resolve(ref, + consumer=…)`` for a value the scope anchors from a key, and + ``materialize_for(expr, consumer=…)`` for one the producer anchored itself. + + The cross-model CTE uses the second. It must: the expressions it + materialises are a date-truncated grain and a value carrying its column's + declared CAST, and re-deriving those by anchoring a ref would project a + different expression than the one the aggregate consumes. What §5.1 asks + for is that the materialisation branch runs on the production path with a + named consumer, which is exactly what this observes. """ def __init__(self) -> None: self.calls: List[Tuple[str, bool]] = [] def install(self, monkeypatch: pytest.MonkeyPatch) -> None: - original = ScopeFrame.resolve + original = ScopeFrame._close spy = self - def _wrapped(self_frame, ref, *, consumer: Optional[ScopeFrame] = None): - spy.calls.append((type(ref).__name__, consumer is not None)) - return original(self_frame, ref, consumer=consumer) + def _wrapped(self_frame, template, *, consumer: Optional[ScopeFrame] = None): + spy.calls.append((type(template).__name__, consumer is not None)) + return original(self_frame, template, consumer=consumer) - monkeypatch.setattr(ScopeFrame, "resolve", _wrapped, raising=True) + monkeypatch.setattr(ScopeFrame, "_close", _wrapped, raising=True) @property def consumer_calls(self) -> int: @@ -177,19 +186,19 @@ class TestConsumerMaterializationOnTheProductionPath: [_crossing_grain_query, _crossing_value_query], ids=["crossing_grain", "crossing_value"], ) - async def test_resolve_is_called_with_a_consumer( + async def test_a_value_is_closed_for_a_named_consumer( self, query_factory, monkeypatch: pytest.MonkeyPatch, ) -> None: - """NEW (§5.1): generating a real query exercises the materialisation - branch of ``ScopeFrame.resolve``.""" + """NEW (§5.1): generating a real query exercises the projection-boundary + branch with a named consumer.""" spy = _ResolveSpy() spy.install(monkeypatch) await _gen(query_factory(), dialect="postgres") - assert spy.calls, "ScopeFrame.resolve was never called at all" + assert spy.calls, "no value was closed through a scope at all" assert spy.consumer_calls > 0, ( - "no production call passed `consumer=` — the cross-model CTE is " + "no production call named a `consumer=` — the cross-model CTE is " "still minting `_val_` aliases through the generator's own " - "materialiser, so `resolve`'s projection-boundary branch remains " + "materialiser, so the projection-boundary branch remains " f"unexercised in production ({len(spy.calls)} consumer-less calls)." ) @@ -216,28 +225,38 @@ async def test_scope_frame_mints_the_val_alias( f"the scope materialised a value nobody projected:\n{sql}" ) - async def test_apply_materializations_has_a_production_caller( + async def test_what_the_scope_materialises_is_what_gets_projected( self, monkeypatch: pytest.MonkeyPatch, ) -> None: - """The other half of the contract: what the scope materialises must be - projected into the producing SELECT by ``apply_materializations``.""" - seen: List[int] = [] - original = ScopeFrame.apply_materializations + """The other half of the contract: the scope's materialisation table is + the SOURCE of the producing SELECT's extra projections. + + Asserted on the table's contents reaching the SQL rather than on + ``apply_materializations`` being the call used. The producing scope here + is a ranked subquery the generator assembles from parts, so it reads the + table directly; what must hold either way is that every materialisation + is projected under its own alias and that nothing else invents one. + """ + captured: List[List[Tuple[str, str]]] = [] + original = ScopeFrame._materialize - def _wrapped(self_frame, select): - seen.append(len(self_frame.materializations)) - return original(self_frame, select) + def _wrapped(self_frame, template): + alias = original(self_frame, template) + captured.append([ + (m.alias, m.expr.sql(dialect="postgres")) + for m in self_frame.materializations + ]) + return alias - monkeypatch.setattr( - ScopeFrame, "apply_materializations", _wrapped, raising=True, - ) - await _gen(_crossing_grain_query(), dialect="postgres") - assert seen, ( - "apply_materializations was never called on the production path" - ) - assert any(count > 0 for count in seen), ( - "apply_materializations ran but the scope held no materialisations" - ) + monkeypatch.setattr(ScopeFrame, "_materialize", _wrapped, raising=True) + sql = await _gen(_crossing_grain_query(), dialect="postgres") + assert captured, "no scope materialised anything on the production path" + final = captured[-1] + for alias, expr_sql in final: + assert f"{expr_sql} AS {alias}" in sql, ( + f"materialisation {alias}={expr_sql!r} is not projected in the " + f"producing SELECT:\n{sql}" + ) # =========================================================================== # diff --git a/tests/test_dev1746_empty_base_plan.py b/tests/test_dev1746_empty_base_plan.py index c2bbe350..71ed1ca4 100644 --- a/tests/test_dev1746_empty_base_plan.py +++ b/tests/test_dev1746_empty_base_plan.py @@ -153,20 +153,36 @@ def test_node_is_absent_when_the_base_is_not_empty(self) -> None: "the empty-base node was set for a query that HAS host row slots" ) - def test_node_presence_implies_every_join_back_is_empty(self) -> None: + def test_node_presence_implies_the_projection_is_isolated_only(self) -> None: """Codex D7: the grain semantics the node would have carried as a field, - asserted as the invariant it actually is. An empty grain is precisely - why the combined SELECT CROSS JOINs instead of joining on a predicate. + asserted as the invariant it actually is. + + The node is present exactly when every projected value is an isolated + aggregate — which is what leaves ``_base`` with no grain columns, and so + why the combined SELECT CROSS JOINs rather than joining on a predicate. + + Deliberately NOT asserted against ``CrossModelAggregatePlan.shared_grain_slots``: + that list can name a HIDDEN row slot a filter created (``status`` in the + filtered shape), which never becomes a projected grain column. The + emitted CROSS JOIN is asserted directly in + ``TestEmittedSqlIsUnchanged.test_combined_select_cross_joins_the_scalar_cte``, + which is the operative guarantee. """ for factory in (_unfiltered_query, _filtered_query): planned = plan_query(query=factory(), bundle=_bundle()) assert planned.empty_base_plan is not None - for plan in planned.cross_model_aggregate_plans: - assert not plan.join_back_pairs, ( - f"empty-base plan present but cross-model plan " - f"{plan.aggregate_slot_id} declares join-back pairs " - f"{plan.join_back_pairs} — the base has no grain to join on." - ) + isolated = { + p.aggregate_slot_id for p in planned.cross_model_aggregate_plans + } | { + p.aggregate_slot_id for p in planned.windowed_aggregate_plans + } + assert planned.projection, "expected a non-empty projection" + assert all(sid in isolated for sid in planned.projection), ( + f"empty-base plan present but the projection {planned.projection} " + f"contains a slot that is not an isolated aggregate " + f"(isolated: {sorted(isolated)}) — such a slot would have to be " + f"materialised in _base, which then is not a placeholder spine." + ) def test_generator_consumes_the_node_rather_than_re_deriving(self) -> None: """P-D: clearing the plan field must change the emitted SQL. If it does diff --git a/tests/test_dev1746_projection_order.py b/tests/test_dev1746_projection_order.py index ec189c65..a6566210 100644 --- a/tests/test_dev1746_projection_order.py +++ b/tests/test_dev1746_projection_order.py @@ -312,15 +312,43 @@ def test_a_hidden_slot_in_the_projection_is_rejected(self) -> None: with pytest.raises(ValueError, match="(?i)hidden"): PlannedQuery(**fields) - def test_a_duplicated_slot_in_the_projection_is_rejected(self) -> None: - """A duplicate would emit the same column twice and corrupt the - positional contract callers read ``columns`` by.""" + def test_a_slot_may_repeat_once_per_declared_public_name(self) -> None: + """C13: one key selected under two names IS listed twice, and each + occurrence consumes the next alias. Pinned so the duplicate check below + cannot be tightened into rejecting a legitimate plan.""" + query = SlayerQuery( + source_model="orders_x", + time_dimensions=[TimeDimension( + dimension=ColumnRef(name="created_at"), + granularity=TimeGranularity.MONTH, + )], + measures=[ + ModelMeasure(formula="amount:sum(window='90d')", name="wa"), + ModelMeasure(formula="amount:sum(window='90d')", name="wb"), + ], + ) + planned = plan_query(query=query, bundle=_chain_bundle()) + repeated = [ + sid for sid in set(planned.projection) + if planned.projection.count(sid) > 1 + ] + assert repeated, ( + f"expected one slot listed twice for the two names: " + f"{planned.projection}" + ) + slot = {s.id: s for s in _all_slots(planned)}[repeated[0]] + assert len(slot.public_aliases) == planned.projection.count(repeated[0]) + + def test_more_occurrences_than_declared_names_is_rejected(self) -> None: + """One occurrence too many would emit the same column twice under the + same name — the duplication that the per-occurrence alias cursor in the + combined projection exists to prevent.""" planned = plan_query( query=_cm_declared_first_query(), bundle=_chain_bundle(), ) fields = dict(planned.__dict__) fields["projection"] = list(planned.projection) + [planned.projection[0]] - with pytest.raises(ValueError, match="(?i)duplicate"): + with pytest.raises(ValueError, match="(?i)duplicate|public name"): PlannedQuery(**fields) def test_renderer_belt_catches_a_model_copy_that_skips_validation( From 4a9dd91c168f72317e30d80636727a0b91ec7a1e Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Thu, 6 Aug 2026 13:57:26 +0200 Subject: [PATCH 44/98] DEV-1746 stage 7: docs + design-decision record MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit docs/concepts/queries.md — `columns` (and each row's key order) follows the order fields were declared in the query. Worth stating explicitly because B7 changed it: a measure on a joined model now appears where it was declared rather than after the local ones. docs/concepts/formulas.md — a windowed measure's NULL-dimension group gets its real value. The old behaviour (NULL, because the rolling aggregate was matched to its group with a plain equality) was observable, so the note says what changed rather than only what is true now. DECISIONS.md — one dated entry covering B1/B2/B3/B7/B8, §5.1/§5.6/§5.12, the rulings behind them, and the three things implementation contradicted: that re-parsing a rendered CTE body is safe (it re-introduces the dotted-alias corruption B2 removes), that a slot appears at most once in the projection (C13 lists it once per declared name), and that `shared_grain_slots` is empty in the empty-base shape (it can name a hidden slot a filter created). Co-Authored-By: Claude Opus 5 (1M context) --- DECISIONS.md | 1 + docs/concepts/formulas.md | 6 ++++++ docs/concepts/queries.md | 8 +++++++- 3 files changed, 14 insertions(+), 1 deletion(-) diff --git a/DECISIONS.md b/DECISIONS.md index 2d3f7609..1ce0084c 100644 --- a/DECISIONS.md +++ b/DECISIONS.md @@ -101,3 +101,4 @@ implementation detail. Include issue refs when known. - 2026-08-05 — Renderer call-site migration deferred to the scope-assembly PR (DEV-1744 → DEV-1746). PR 1 lands the complete, tested `render_value_key` API but does NOT reroute the generator's own render paths through it; only the ScalarCall policy is genuinely shared (all six paths call `render_scalar_call`). Deferred because the filter and composite paths carry state the context does not yet consume — the local-aggregate HAVING branch with its slot lookup and inline-aggregate emission, the first/last ranked state, the filter-side CAST policy, and the rn-suffix / filtered-rank / composite-alias / resolved-kwarg maps — and because the cross-scope paths are the ones that should supply `resolve(consumer=...)` its first production caller, which is scope-assembly work by nature. Splitting the reroute across two PRs would move that state twice. Full detail, per call site, in the `slayer/sql/render/value_expr.py` module docstring. - 2026-08-05 — One Mode-A door, plan-time filter routing, structural reachability (DEV-1745, PR 2 of the DEV-1742 series). **P-A:** every fragment of free (Mode-A) SQL now enters a SELECT scope through ONE door — `ScopeFrame.enter_predicate` / `enter_expression`, two surfaces over one implementation differing only in the parse helper (the `SELECT 1 WHERE …` statement-keyword guard for predicates). The door prequotes, parses, scans, expands, re-parses, re-scans, unions both scans into `join_paths`, then applies Law 2. Join discovery is a side effect of entering, so a caller cannot forget it; the DEV-1494 dual-scan contract is preserved because a dotted ref whose derived column inlines to a constant is only visible to the PRE-expansion scan. The surface's grammar is a static property of the field being read (`Column.filter` and model `filters` are predicates, `Column.sql` is an expression) — deliberately no content sniffing and no "try one grammar then the other" retry, either of which would put classification back into render time. **The door adds NO qualification pass:** `expand_derived_refs_sync` already qualifies, against the OWNING model's canonical alias, and deliberately leaves a node alone when its alias path does not resolve because that is an opaque CTE / subquery reference — a blanket pass against `root_relation` would corrupt exactly those. `SQLGenerator._parse` and `_parse_predicate` were byte-identical apart from one line and now delegate to a shared `slayer/sql/render/parse.py`, which is what lets the door take over a call site without changing the SQL it emits. **Failure is loud:** an unparseable Mode-A fragment raises `ModeASqlParseError` carrying the fragment and its location. Three swallow-all `except Exception` lanes leave the production path, including `_filter_join_paths._scan`, which converted a parse failure into ZERO join paths — emitting a query missing its joins rather than reporting the problem. Two consequences on emitted SQL were accepted explicitly: undeclared bare identifiers in Mode-A text are now qualified against the scope root (the old fallbacks qualified only DECLARED columns, leaving the rest to bind to whatever was in scope after re-rendering inside a rerooted CTE), and the shifted CTE renders its model filter from the door's AST rather than passing regex-substituted text through, so sqlglot's canonical form appears. **P-D:** the outer combined-SELECT WHERE routing is decided by the planner (`PlannedQuery.outer_where_filter_ids`) and consumed verbatim; the generator's render-time re-walk of `filters_by_phase` is deleted. **Reachability is structural (5.3):** `classify_host_filter` routed derived-column references by asking whether the declaring model's NAME appeared anywhere in `target_path`. That flat membership test called a SIBLING branch reachable whenever it shared a name with the target path, and called a host-model derived column whose `Column.sql` crossed INTO the target host-local. It is replaced by one rule for every key kind — a dependency is reachable iff its anchored join path is a PREFIX of `target_path` — computed per filter at plan time by `slayer/engine/filter_reachability.py`, recursively over the whole key tree, failing CLOSED on an unhandled kind. The summary lives on `PlannedQuery`, NOT on `ColumnSqlKey` (interned, and `_reroot_path_ref` copies unknown fields through rerooting stale) and NOT on `ValueSlot` (`filter_referenced_slot_ids` silently skips keys with no interned slot — a filter-only derived column is exactly such a key — and slots are copied wholesale into nested plans); it is recomputed per plan so the paths always mean what they say relative to the root they were anchored at. **Warnings (5.5):** dropped-filter emission moves from mid-render (once per cross-model plan, so nested subplans double-fired, and silent on any path that never reached that step) to the engine boundary — collected across every plan, deduped per user filter on `(location, original text)`, with drop reasons required to AGREE (disagreement raises rather than keeping the first, which is why the reason is now target-independent). `SlayerResponse.warnings` widens to a `kind`-discriminated union and is surfaced by REST, MCP and the CLI (stderr, so stdout stays pipeable). **Two live bugs fixed in passing:** a `Column.sql` that is exactly one bare reference to another derived column was silently not expanded, because `col.replace()` is a no-op when that column IS the parsed root; and a `_cm_` CTE never registered the joins its aggregation template fragments crossed, emitting `SUM(customers.spend * regions.weight) FROM customers` — SQL no database accepts. **P-J:** the superseded helpers stay callable and test-pinned rather than deleted (state 1); newly unreferenced by production are `_render_model_filter_sql`, `_filter_join_paths`, `_render_mode_a_predicate`, `_expand_degenerate_derived_root`, `_column_ref_is_derived`, `_predicate_references_derived` and `_qualify_mode_a_sql_filter`. `_qualify_column_filter_sql` remains live via `_expand_column_filter_sql`'s bundle-less branch, and `_BARE_IDENT_RE` remains in use outside the Mode-A surfaces. +- 2026-08-06 — Scope assembly: null-safe grain doctrine, pagination through the dialect, and the combined layer as one AST (DEV-1746, PR 3 of the DEV-1742 consolidation). Six consequences worth recording. (1) **One grain join-back builder** (`slayer/sql/render/joins.py`), taking EXPRESSION operand pairs and returning `None` for an empty grain so the caller emits `CROSS JOIN` — a scalar aggregate has no grain to join on, and a truthy `TRUE` would erase that distinction. It replaced a string round-trip that re-parsed pre-quoted operands: SLayer's public aliases are dotted, and on BigQuery the round-trip re-read `_base."orders.customers.status"` as a multi-part *reference*, emitting a qualifier for a table that does not exist (``_base___orders___customers`.`status``). Three golden entries recorded `ScopeLeakError` for exactly that and now record real SQL. The `_wm_` INNER grain went null-safe in the same move: it compared with a plain `=`, so a NULL-dimension group matched nothing and silently received NULL — while the outer join-back and the `_cm_` join-back were already null-safe, so the answer depended on which isolation shape a measure landed in. Executed on DuckDB and SQLite; the integration test that asserted the old behaviour said so in its NAME (`test_null_dimension_group_gets_null_windowed_value`). (2) **Pagination is a dialect-strategy method.** `apply_pagination` is the one place LIMIT/OFFSET is expressed; the T-SQL override owns its rule explicitly (TOP for a bare limit; a deterministic `ORDER BY (SELECT NULL)` before OFFSET/FETCH when the query is unordered, because SQL Server rejects OFFSET without ordering) rather than relying on sqlglot happening to apply it — so the rule survives a sqlglot upgrade and lands in the AST where callers can see it. The cross-model combined path used to append raw text and emitted a literal `LIMIT` on SQL Server, while the SAME query carrying a transform layer went through the outer wrap and came out correct. (3) **The combined statement is one `exp.Select`.** Projection, FROM/JOIN chain, WHERE, ORDER BY and pagination were all string-assembled; the WHERE in particular was glued on with a hand-rolled `"\nAND "` / `"\nWHERE "` connector that chose between them by asking whether an earlier filter existed. `Select.where` conjoins, so that choice disappears. The WITH chain is assembled by `slayer/sql/render/cte_assembly.py` from **declared** `(name, query, depends_on)` entries, never dependencies discovered by scanning the rendered statement: a scan cannot tell a CTE reference from a same-named real table, is defeated by quoting and case folding, and would silently mis-order rather than fail. **CTE bodies stay AST end to end** — the first attempt kept the renderers returning text and parsed at the seam, which re-introduced exactly the dotted-alias corruption the same PR had just removed. One documented parse seam remains: the re-rooted cross-model CTE arrives as a complete nested `WITH … SELECT` from `generate_from_planned`. The combined ORDER BY had to become AST for the same reason — rendering its terms and re-parsing corrupted them into `` `orders___customers`.`spend_sum` ``. (4) **`PlannedQuery.projection` is consumed verbatim** (B7). The combined projection was assembled in four grouped passes, so a cross-model measure declared first was emitted last; the generator admitted it in a comment. Hidden-slot trimming stops being a mechanism at all — a hidden slot is absent from the list. Two subtleties: a slot may legitimately appear MORE THAN ONCE (C13 lets one key be selected under several names, and the plan lists it once per name), so each occurrence consumes the NEXT of that slot's rendered columns — emitting the whole list per occurrence projected every alias once per name; and because pydantic's `model_copy(update=…)` skips validators and rerooting uses exactly that, one renderer-side assertion is kept at the single entry point, raising rather than skipping, since silently dropping a column the plan asked for is how a wrong answer reaches a user. (5) **One materialiser on the cross-model path.** `ScopeFrame` gained `materialize_for(expr, consumer=…)` — `resolve` anchors a ref and closes it, this is the second half alone, for a producer that anchored its own template (a date-truncated grain, a value carrying its column's declared CAST; re-deriving those from a ref would project a different expression than the aggregate consumes). Unifying the dedup tables fixed a defect: grouping a first/last cross-model aggregate by the same crossing expression it aggregates projected that expression TWICE (`regions.weight AS _val_0` and `AS _val_1`), because the grain site and the value site kept separate maps. `_build_first_last_base_select` keeps its own flow until PR 5's `RankedAggregatePlan`; the boundary is pinned by a test that fails when it moves, rather than left as an assumption. (6) **The empty-base grain is a typed plan node** (`EmptyBaseGrainPlan`, presence as the discriminator, carrying the host-local filter ids). No `grain_slot_ids` field — in that shape the grain is empty by definition — and the invariant that field would have encoded had to be restated during implementation, because `CrossModelAggregatePlan.shared_grain_slots` can name a HIDDEN row slot a filter created, which never becomes a projected grain column. Emitted SQL byte-identical. Reformatting churn was accepted and reviewed: emitting the statement in one pass means sqlglot's printer indents CTE bodies, which moved 25 golden entries and broke assertions that pinned layout rather than structure — one test helper (`_extract_src_body`, anchored on the exact text `"\n) AS _src"`) accounted for 19 failures by itself and is now whitespace-tolerant. Also fixed in passing, because it was two lines and not a refactor: a filter on a JOINED model's crossing derived column emitted invalid SQL — both the join scanner and the filter renderer expanded the column without `is_root=False`, so a further-joined ref came out bare, the scanner could not match it to a join path, the hop was never joined, and the filter referenced a table absent from the FROM. The two must stay in lockstep, since discovery scans the same expansion the renderer emits. diff --git a/docs/concepts/formulas.md b/docs/concepts/formulas.md index d59c3708..796b22d2 100644 --- a/docs/concepts/formulas.md +++ b/docs/concepts/formulas.md @@ -67,6 +67,12 @@ Windowed measures need exactly one resolvable time dimension (a single a windowed measure (`{"formula": "revenue:sum(window='90d') > 100"}`) applies after aggregation, and the windowed measure must also be selected. +A group whose dimension value is NULL gets its real windowed value, like any +other group. (Earlier versions returned NULL for such groups: the rolling +aggregate was matched to its group with a plain equality, and `NULL = NULL` is +not true. Grouping keys now compare null-safely everywhere, so a NULL region or +an unmatched outer join no longer silently blanks the measure.) + A windowed measure may also be used purely as an **order** target without being selected — `{"order": [{"column": "revenue:sum(window='90d')", "direction": "desc"}]}` ranks by the rolling value and keeps it out of the result. That works both for a diff --git a/docs/concepts/queries.md b/docs/concepts/queries.md index d0b095d2..426298ab 100644 --- a/docs/concepts/queries.md +++ b/docs/concepts/queries.md @@ -152,11 +152,17 @@ Query results are returned as a `SlayerResponse`: | Field | Type | Description | |-------|------|-------------| | `data` | list[dict] | Rows as dictionaries | -| `columns` | list[str] | Column names in `model_name.column_name` format (e.g., `"orders._count"`, `"orders.customers.regions.name"` for multi-hop) | +| `columns` | list[str] | Column names in `model_name.column_name` format (e.g., `"orders._count"`, `"orders.customers.regions.name"` for multi-hop), **in the order you declared them** | | `row_count` | int | Number of rows | | `sql` | string | The generated SQL (useful for debugging) | | `attributes` | ResponseAttributes | Field metadata split by type: `attributes.dimensions` and `attributes.measures`, each a dict of column alias → FieldMetadata (label, format) | +`columns` — and the key order of each row in `data` — follows the order you +declared fields in the query: dimensions, then time dimensions, then measures, +each in the order given. This holds regardless of how a measure is computed, so +a measure on a joined model appears where you declared it rather than after the +local ones. Fields used only for ordering are computed but not returned. + ```json { "data": [ From 1a908df8d17e3482cc61f9f25aec12c60107b7fd Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Thu, 6 Aug 2026 14:37:58 +0200 Subject: [PATCH 45/98] DEV-1746 stage 6: one isolation decision, and the DEV-1688 seam MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Whether an aggregate gets its own CTE, and where that CTE is rooted, was three predicates inlined in the planner's aggregate loop. They ran in a fixed order and each knew about the others by omission: the windowed skip existed because a windowed measure would otherwise trip the crossing-input trigger, and the crossing-input trigger excluded path-bearing sources because the target-rooted branch had already claimed them. Reading one meant reading all three. slayer/engine/isolation.py makes it one function returning one value (IsolationKind: NONE / WINDOWED / TARGET_ROOTED / HOST_ROOTED). Nothing about the decision changed — the same inputs produce the same kind, which the whole suite passing unchanged is the evidence for, and which the new tests assert directly by comparing the classifier's verdict against what the planner BUILT rather than by restating the predicate. may_inline_crossing_inputs is the DEV-1688 seam: hardcoded False, so every crossing aggregate isolates, which is today's behaviour. It is pinned both ways — the constant, and that flipping it actually changes the verdict. A hook that reads nothing and decides nothing is exactly what DEV-1688 must not inherit, so the test that flips it is the one that matters. Its render-time counterpart, ScopeFrame.may_inline, guards individual values at the projection boundary rather than whole aggregates; both are pinned together so neither is mistaken for the other when cardinality metadata arrives. The two shapes that exist only because the old predicates knew about each other now have tests of their own: a windowed measure whose Column.filter crosses a join must stay WINDOWED rather than being isolated twice, and a sub-plan suppresses host-rooted isolation to stop the recursion. Scope boundary, stated rather than implied: this consolidates the TRIGGER decision. cross_model_planner's forward-vs-filtered-local dispatch chooses the CTE shape once isolation is decided, and is downstream of it. 11025 unit tests passing; ruff clean. Co-Authored-By: Claude Opus 5 (1M context) --- slayer/engine/isolation.py | 134 ++++++++ slayer/engine/stage_planner.py | 56 +--- tests/test_dev1746_isolation_classifier.py | 371 +++++++++++++++++++++ 3 files changed, 518 insertions(+), 43 deletions(-) create mode 100644 slayer/engine/isolation.py create mode 100644 tests/test_dev1746_isolation_classifier.py diff --git a/slayer/engine/isolation.py b/slayer/engine/isolation.py new file mode 100644 index 00000000..67bec4ea --- /dev/null +++ b/slayer/engine/isolation.py @@ -0,0 +1,134 @@ +"""One plan-time decision about how an aggregate is isolated (P-C). + +An aggregate that needs its own rows — because it crosses inputs, orders its +own rows (first/last), or carries its own frame (windowed) — is compiled as a +plan-shaped CTE rooted where its rows live and joined back on the query grain. +The host base contains only purely-local aggregates, so host cardinality never +changes. + +*Whether* an aggregate needs that, and *where* its CTE is rooted, used to be +decided by three predicates inlined in the planner's aggregate loop. They ran in +a fixed order and each knew about the others by omission — the windowed skip +existed because a windowed measure would otherwise trip the crossing-input +trigger, and the crossing-input trigger excluded path-bearing sources because +the target-rooted branch had already claimed them. Reading any one of them meant +reading all three. + +They are one function here, returning one value. Nothing about the decision +changed: the same inputs produce the same kind, which is what +``tests/test_dev1746_isolation_classifier.py`` pins. + +This is also where cardinality-aware inlining will land (DEV-1688). +:func:`may_inline_crossing_inputs` is the seam — hardcoded ``False``, so every +crossing aggregate isolates, which is today's behaviour. When a future change +lets a provably 1:1 crossing input stay inline, it changes there and every +isolation kind sees it at once, rather than in the four places that used to +decide independently. The render-time counterpart is ``ScopeFrame.may_inline``, +which guards the projection boundary for individual values; this one guards +whole aggregates. +""" + +from __future__ import annotations + +from enum import Enum +from typing import Any, Sequence, Set, Tuple + +from slayer.core.keys import AggregateKey +from slayer.engine.aggregate_input_paths import compute_aggregate_input_join_paths + +__all__ = [ + "IsolationKind", + "classify_isolation", + "may_inline_crossing_inputs", +] + + +class IsolationKind(str, Enum): + """How one aggregate slot is compiled.""" + + #: Purely local — renders inline in the host base SELECT. + NONE = "none" + #: Its own ``_wm_`` CTE: a host-rooted range join carrying its own frame. + WINDOWED = "windowed" + #: Its own ``_cm_`` CTE rooted at the TARGET the aggregate's source names. + TARGET_ROOTED = "target_rooted" + #: Its own ``_cm_`` CTE rooted at the HOST, because a LOCAL aggregate's + #: inputs (a ``Column.filter``, the source's ``Column.sql``, an arg or a + #: kwarg) cross a join and so need their own rows. + HOST_ROOTED = "host_rooted" + + @property + def needs_own_cte(self) -> bool: + return self is not IsolationKind.NONE + + +def may_inline_crossing_inputs(crossed_paths: Sequence[Tuple[str, ...]]) -> bool: # NOSONAR(S1172) — crossed_paths is the documented DEV-1688 seam; the cardinality-aware decision reads it, hardcoded False until then. + """Whether an aggregate whose inputs cross ``crossed_paths`` may stay in the + host base instead of being isolated into its own CTE. + + Hardcoded ``False``: a crossing input is isolated, always. Inlining one is + only safe when the crossed join is provably 1:N-free for this aggregate, + which needs the cardinality metadata DEV-1688 is about. Until then this is + the single place that answer is given, so the future change has one site + rather than one per isolation kind. + """ + return False + + +def classify_isolation( + *, + slot: Any, + windowed_slot_ids: Set[str], + bundle: Any, + disable_host_rooted_isolation: bool = False, +) -> IsolationKind: + """How ``slot`` is compiled. The single trigger decision. + + ``disable_host_rooted_isolation`` is set when planning a nested sub-plan: + that sub-plan contains the same crossing measure and would otherwise recurse + forever, and inside a CTE the crossing input renders inline legally, because + the CTE is the aggregate's own scope. + """ + if slot.id in windowed_slot_ids: + # A windowed measure always compiles to its own ``_wm_`` CTE, even when + # its ``Column.filter`` crosses a join — that crossing is carried inside + # the windowed CTE, not by a second isolation on top of it. + return IsolationKind.WINDOWED + + key = slot.key + if not isinstance(key, AggregateKey): + return IsolationKind.NONE + + if getattr(key.source, "path", ()): + # The source names another model: the aggregate's rows live there. + return IsolationKind.TARGET_ROOTED + + if disable_host_rooted_isolation: + return IsolationKind.NONE + + crossed = _crossing_input_paths(key=key, bundle=bundle) + if not crossed: + return IsolationKind.NONE + if may_inline_crossing_inputs(crossed): + return IsolationKind.NONE + return IsolationKind.HOST_ROOTED + + +def _crossing_input_paths(*, key: AggregateKey, bundle: Any) -> list: + """Join paths a LOCAL aggregate's inputs cross. + + A ``Column.filter`` carries its crossings as typed + ``referenced_join_paths`` from binder time; everything else (the source's + ``Column.sql``, positional args including an explicit first/last time arg, + and kwargs — column refs, user template fragments, and non-overridden + model-default aggregation params) is computed structurally. + """ + if key.column_filter_key is not None and key.column_filter_key.referenced_join_paths: + return list(key.column_filter_key.referenced_join_paths) + source_model = getattr(bundle, "source_model", None) + return list(compute_aggregate_input_join_paths( + key=key, + anchor_model=source_model, + anchor_relation=source_model.name if source_model is not None else "", + bundle=bundle, + )) diff --git a/slayer/engine/stage_planner.py b/slayer/engine/stage_planner.py index 678d37ea..85e5fe99 100644 --- a/slayer/engine/stage_planner.py +++ b/slayer/engine/stage_planner.py @@ -63,9 +63,7 @@ from slayer.core.time_bounds import strip_frame_bounds from slayer.core.window_duration import parse_window_duration from slayer.core.scope import ModelScope, StageColumn, StageSchema -from slayer.engine.aggregate_input_paths import ( - compute_aggregate_input_join_paths, -) +from slayer.engine.isolation import IsolationKind, classify_isolation from slayer.engine.binding import ( BoundExpr as BinderBoundExpr, BoundFilter, @@ -1388,48 +1386,20 @@ def _windowed_phase(bf: BoundFilter) -> Phase: cross_model_plans = [] host_slots_for_classifier = projection.registry.slots for slot in agg_slots: - # DEV-1714 Stage 10 — a windowed slot renders via its own ``_wm_`` CTE - # (host-rooted range join), never a cross-model ``_cm_`` CTE, even when - # its ``Column.filter`` crosses a join (which would otherwise trip the - # host-rooted isolation trigger below). - if slot.id in windowed_slot_ids: - continue - key = slot.key - if not isinstance(key, AggregateKey): - continue - agg_path = getattr(key.source, "path", ()) - # DEV-1503 / DEV-1709 — Law-3 trigger predicate. Invoke the - # cross-model planner when the aggregate's source carries a - # non-empty join path (target-rooted, existing behaviour) OR when - # ANY other input of a LOCAL aggregate crosses a join (host-rooted - # isolation): ``Column.filter`` (typed ``referenced_join_paths`` - # from binder time — DEV-1503), source ``Column.sql``, positional - # args incl. the explicit first/last time arg, kwargs (column - # refs, user template fragments, and non-overridden model-default - # ``AggregationParam`` fragments) — DEV-1709's widened trigger, - # computed plan-time by ``compute_aggregate_input_join_paths``. - has_crossing_filter = ( - key.column_filter_key is not None - and bool(key.column_filter_key.referenced_join_paths) - ) - has_crossing_input = ( - not disable_host_rooted_isolation - and not agg_path - and ( - has_crossing_filter - or bool(compute_aggregate_input_join_paths( - key=key, - anchor_model=bundle.source_model, - anchor_relation=( - bundle.source_model.name - if bundle.source_model is not None else "" - ), - bundle=bundle, - )) - ) + # ONE trigger decision (P-C / DEV-1688 seam). The windowed skip, the + # target-rooted branch and the host-rooted crossing trigger were three + # predicates here that each knew about the others by omission; they are + # one classifier now, and the cardinality-aware inlining decision has a + # single place to land. + kind = classify_isolation( + slot=slot, + windowed_slot_ids=windowed_slot_ids, + bundle=bundle, + disable_host_rooted_isolation=disable_host_rooted_isolation, ) - if not agg_path and not has_crossing_input: + if kind in (IsolationKind.NONE, IsolationKind.WINDOWED): continue + key = slot.key # DEV-1450 #2: re-rooting (C1) is owned by the strategy. We hand it # the host query, the public projection, and a sub-plan builder so it # can compile a nested re-rooted PlannedQuery when the host carries diff --git a/tests/test_dev1746_isolation_classifier.py b/tests/test_dev1746_isolation_classifier.py new file mode 100644 index 00000000..13c05173 --- /dev/null +++ b/tests/test_dev1746_isolation_classifier.py @@ -0,0 +1,371 @@ +"""DEV-1746 stage 6 — one isolation decision, and the DEV-1688 seam. + +Whether an aggregate gets its own CTE, and where that CTE is rooted, was decided +by three predicates inlined in the planner's aggregate loop. They ran in a fixed +order and each knew about the others by omission: the windowed skip existed +because a windowed measure would otherwise trip the crossing-input trigger, and +the crossing-input trigger excluded path-bearing sources because the +target-rooted branch had already claimed them. Reading one meant reading all +three, and a future cardinality-aware change would have had to land in each. + +They are one classifier now. This module pins two things: + +* **Every isolation kind is reachable and correctly classified** — including the + two that exist only because of the omissions above (a windowed measure whose + ``Column.filter`` crosses a join, which must stay WINDOWED rather than + becoming host-rooted; and a sub-plan, where host-rooted isolation is + suppressed to stop the recursion). +* **The decision did not change.** This is a refactor, so the classifier's + verdict must agree with what the planner actually built — asserted by + comparing the classification against the plans, not by restating the + predicate. + +``may_inline_crossing_inputs`` is the DEV-1688 seam: hardcoded ``False`` so +every crossing aggregate isolates, which is today's behaviour. It is pinned both +ways — the constant, and that flipping it actually changes the verdict, so the +seam is load-bearing rather than decorative. +""" + +from __future__ import annotations + +import pytest + +from slayer.core.enums import TimeGranularity +from slayer.core.models import ModelMeasure +from slayer.core.query import ColumnRef, SlayerQuery, TimeDimension +from slayer.engine import isolation as isolation_mod +from slayer.engine.isolation import ( + IsolationKind, + classify_isolation, + may_inline_crossing_inputs, +) +from slayer.engine.planned import PlannedQuery +from slayer.engine.source_bundle import ResolvedSourceBundle +from slayer.engine.stage_planner import plan_query + +from tests._cross_model_chain import ( + _countries, + _customers_v2, + _orders_x, + _regions, +) + + +def _bundle() -> ResolvedSourceBundle: + return ResolvedSourceBundle( + source_model=_orders_x(), + referenced_models=[_customers_v2(), _regions(), _countries()], + ) + + +def _classify_all(query: SlayerQuery) -> "tuple[dict, PlannedQuery]": + """Every aggregate slot's isolation kind keyed by slot id, and the plan.""" + planned = plan_query(query=query, bundle=_bundle()) + windowed_ids = { + p.aggregate_slot_id for p in planned.windowed_aggregate_plans + } + return { + slot.id: classify_isolation( + slot=slot, windowed_slot_ids=windowed_ids, bundle=_bundle(), + ) + for slot in planned.aggregate_slots + }, planned + + +_MONTH = [TimeDimension( + dimension=ColumnRef(name="created_at"), granularity=TimeGranularity.MONTH, +)] + + +# =========================================================================== # +# Each kind is reachable and correctly named. +# =========================================================================== # +class TestEveryIsolationKindIsClassified: + + def test_a_purely_local_aggregate_is_not_isolated(self) -> None: + kinds, planned = _classify_all(SlayerQuery( + source_model="orders_x", + dimensions=[ColumnRef(name="status")], + measures=[ModelMeasure(formula="amount:sum")], + )) + assert set(kinds.values()) == {IsolationKind.NONE}, kinds + assert not planned.cross_model_aggregate_plans + assert not planned.windowed_aggregate_plans + + def test_a_target_rooted_aggregate_is_classified(self) -> None: + kinds, planned = _classify_all(SlayerQuery( + source_model="orders_x", + dimensions=[ColumnRef(name="status")], + measures=[ModelMeasure(formula="customers_v2.lifetime_value:sum")], + )) + assert IsolationKind.TARGET_ROOTED in kinds.values(), kinds + assert len(planned.cross_model_aggregate_plans) == 1 + + def test_a_windowed_aggregate_is_classified(self) -> None: + kinds, planned = _classify_all(SlayerQuery( + source_model="orders_x", + dimensions=[ColumnRef(name="status")], + time_dimensions=list(_MONTH), + measures=[ModelMeasure( + formula="amount:sum(window='90d')", name="w", + )], + )) + assert IsolationKind.WINDOWED in kinds.values(), kinds + assert len(planned.windowed_aggregate_plans) == 1 + + def test_a_local_aggregate_with_a_crossing_filter_is_host_rooted( + self, + ) -> None: + """The trigger that exists so a LOCAL aggregate whose ``Column.filter`` + reaches another model still gets its own rows.""" + crossing = [ + _orders_x().columns[0].model_copy(update={ + "name": "eu_amount", "sql": "amount", + "filter": "customers_v2.status = 'eu'", "primary_key": False, + }), + ] + planned = plan_query( + query=SlayerQuery( + source_model="orders_x", + dimensions=[ColumnRef(name="status")], + measures=[ModelMeasure(formula="eu_amount:sum", name="eu")], + ), + bundle=ResolvedSourceBundle( + source_model=_orders_x(extra_columns=crossing), + referenced_models=[_customers_v2(), _regions(), _countries()], + ), + ) + bundle = ResolvedSourceBundle( + source_model=_orders_x(extra_columns=crossing), + referenced_models=[_customers_v2(), _regions(), _countries()], + ) + kinds = { + slot.id: classify_isolation( + slot=slot, windowed_slot_ids=set(), bundle=bundle, + ) + for slot in planned.aggregate_slots + } + assert IsolationKind.HOST_ROOTED in kinds.values(), kinds + assert planned.cross_model_aggregate_plans, ( + "a crossing Column.filter must produce an isolated CTE" + ) + + +# =========================================================================== # +# The two cases that only exist because the old predicates knew about each other. +# =========================================================================== # +class TestOrderingBetweenTheOldPredicates: + + def test_a_windowed_measure_with_a_crossing_filter_stays_windowed( + self, + ) -> None: + """The windowed skip came FIRST for a reason: this shape would trip the + crossing-input trigger and be isolated twice.""" + crossing = [ + _orders_x().columns[0].model_copy(update={ + "name": "eu_amount", "sql": "amount", + "filter": "customers_v2.status = 'eu'", "primary_key": False, + }), + ] + bundle = ResolvedSourceBundle( + source_model=_orders_x(extra_columns=crossing), + referenced_models=[_customers_v2(), _regions(), _countries()], + ) + planned = plan_query( + query=SlayerQuery( + source_model="orders_x", + time_dimensions=list(_MONTH), + measures=[ModelMeasure( + formula="eu_amount:sum(window='90d')", name="w", + )], + ), + bundle=bundle, + ) + windowed_ids = { + p.aggregate_slot_id for p in planned.windowed_aggregate_plans + } + assert windowed_ids, "expected a windowed plan" + for slot in planned.aggregate_slots: + if slot.id not in windowed_ids: + continue + assert classify_isolation( + slot=slot, windowed_slot_ids=windowed_ids, bundle=bundle, + ) is IsolationKind.WINDOWED + assert not planned.cross_model_aggregate_plans, ( + "the windowed measure was ALSO isolated into a _cm_ CTE" + ) + + def test_suppressing_host_rooted_isolation_yields_none(self) -> None: + """Inside a sub-plan the crossing input renders inline — legal there, + because the CTE is the aggregate's own scope, and required because the + sub-plan holds the same measure and would otherwise recurse.""" + crossing = [ + _orders_x().columns[0].model_copy(update={ + "name": "eu_amount", "sql": "amount", + "filter": "customers_v2.status = 'eu'", "primary_key": False, + }), + ] + bundle = ResolvedSourceBundle( + source_model=_orders_x(extra_columns=crossing), + referenced_models=[_customers_v2(), _regions(), _countries()], + ) + planned = plan_query( + query=SlayerQuery( + source_model="orders_x", + dimensions=[ColumnRef(name="status")], + measures=[ModelMeasure(formula="eu_amount:sum", name="eu")], + ), + bundle=bundle, + ) + for slot in planned.aggregate_slots: + assert classify_isolation( + slot=slot, + windowed_slot_ids=set(), + bundle=bundle, + disable_host_rooted_isolation=True, + ) is IsolationKind.NONE + + def test_a_target_rooted_source_is_not_reclassified_as_host_rooted( + self, + ) -> None: + """The crossing trigger only applies to a LOCAL aggregate; a source that + already names another model is target-rooted, whatever else it crosses.""" + kinds, _ = _classify_all(SlayerQuery( + source_model="orders_x", + dimensions=[ColumnRef(name="status")], + measures=[ModelMeasure(formula="customers_v2.deep_pop:sum")], + )) + assert IsolationKind.TARGET_ROOTED in kinds.values(), kinds + assert IsolationKind.HOST_ROOTED not in kinds.values(), kinds + + +# =========================================================================== # +# The decision did not change. +# =========================================================================== # +class TestClassificationAgreesWithThePlan: + + @pytest.mark.parametrize( + "query", + [ + SlayerQuery( + source_model="orders_x", + dimensions=[ColumnRef(name="status")], + measures=[ModelMeasure(formula="amount:sum")], + ), + SlayerQuery( + source_model="orders_x", + dimensions=[ColumnRef(name="status")], + measures=[ + ModelMeasure(formula="customers_v2.lifetime_value:sum"), + ModelMeasure(formula="amount:sum"), + ], + ), + SlayerQuery( + source_model="orders_x", + dimensions=[ColumnRef(name="status")], + time_dimensions=_MONTH, + measures=[ + ModelMeasure(formula="amount:sum(window='90d')", name="w"), + ModelMeasure(formula="customers_v2.lifetime_value:sum"), + ModelMeasure(formula="amount:avg"), + ], + ), + SlayerQuery( + source_model="orders_x", + measures=[ModelMeasure(formula="customers_v2.lifetime_value:sum")], + ), + ], + ids=["local", "local+cm", "mixed", "scalar_cm"], + ) + def test_every_isolated_slot_is_classified_isolated_and_no_other( + self, query: SlayerQuery, + ) -> None: + """The refactor's real assertion: the classifier's verdict matches what + the planner BUILT, for every aggregate slot in the query.""" + kinds, planned = _classify_all(query) + cm_ids = { + p.aggregate_slot_id for p in planned.cross_model_aggregate_plans + } + wm_ids = { + p.aggregate_slot_id for p in planned.windowed_aggregate_plans + } + for sid, kind in kinds.items(): + if sid in wm_ids: + assert kind is IsolationKind.WINDOWED, (sid, kind) + elif sid in cm_ids: + assert kind.needs_own_cte and kind is not IsolationKind.WINDOWED, ( + sid, kind, + ) + else: + assert kind is IsolationKind.NONE, ( + f"slot {sid} classified {kind} but the planner built no CTE " + f"for it" + ) + + +# =========================================================================== # +# The DEV-1688 seam. +# =========================================================================== # +class TestMayInlineSeam: + + def test_inlining_a_crossing_input_is_refused(self) -> None: + """Hardcoded ``False``: a crossing input is isolated, always. Inlining + one is only safe when the crossed join is provably 1:N-free, which needs + cardinality metadata SLayer does not carry yet.""" + assert may_inline_crossing_inputs([("customers_v2",)]) is False + assert may_inline_crossing_inputs([]) is False + + def test_the_seam_is_load_bearing( + self, monkeypatch: pytest.MonkeyPatch, + ) -> None: + """Flipping the seam must change the verdict. + + Otherwise it is decorative — a hook that reads nothing and decides + nothing, which is exactly what DEV-1688 must not inherit. With it + returning ``True`` a crossing-input aggregate stops being isolated, + which is the behaviour a cardinality-aware version would enable. + """ + crossing = [ + _orders_x().columns[0].model_copy(update={ + "name": "eu_amount", "sql": "amount", + "filter": "customers_v2.status = 'eu'", "primary_key": False, + }), + ] + bundle = ResolvedSourceBundle( + source_model=_orders_x(extra_columns=crossing), + referenced_models=[_customers_v2(), _regions(), _countries()], + ) + planned = plan_query( + query=SlayerQuery( + source_model="orders_x", + dimensions=[ColumnRef(name="status")], + measures=[ModelMeasure(formula="eu_amount:sum", name="eu")], + ), + bundle=bundle, + ) + slot = next( + s for s in planned.aggregate_slots + if classify_isolation( + slot=s, windowed_slot_ids=set(), bundle=bundle, + ) is IsolationKind.HOST_ROOTED + ) + monkeypatch.setattr( + isolation_mod, "may_inline_crossing_inputs", lambda paths: True, + ) + assert classify_isolation( + slot=slot, windowed_slot_ids=set(), bundle=bundle, + ) is IsolationKind.NONE, ( + "flipping may_inline_crossing_inputs did not change the verdict — " + "the seam is not consulted" + ) + + def test_the_render_time_seam_is_separate_and_still_false(self) -> None: + """``ScopeFrame.may_inline`` guards individual values at the projection + boundary; this module's seam guards whole aggregates at plan time. Both + are ``False``; they are pinned together so neither is mistaken for the + other when DEV-1688 lands.""" + from slayer.sql.scope import ScopeFrame + + assert ScopeFrame.may_inline( + ScopeFrame.__new__(ScopeFrame), [("customers_v2",)], + ) is False From 222f412097858b2790b21992232f318c388266a2 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Thu, 6 Aug 2026 14:39:27 +0200 Subject: [PATCH 46/98] =?UTF-8?q?DEV-1746:=20fix=20SonarCloud=20S5863=20?= =?UTF-8?q?=E2=80=94=20assert=20the=20assembly=20order,=20not=20run-vs-run?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `assert build() == build()` compared two calls of the same function, which Sonar flags as an assertion with the same actual and expected expression. The rule is right about more than style here: comparing two runs to each other proves they agree but not that they agree on the RIGHT order, and a set-based implementation could be stable within one process while varying across them. Now asserts both runs against the expected order — base pulled ahead of the wm that depends on it, cm keeping its declared position — which is what the test was trying to say. Co-Authored-By: Claude Opus 5 (1M context) --- tests/test_dev1746_cte_assembly.py | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/tests/test_dev1746_cte_assembly.py b/tests/test_dev1746_cte_assembly.py index 0956174c..2bb01526 100644 --- a/tests/test_dev1746_cte_assembly.py +++ b/tests/test_dev1746_cte_assembly.py @@ -141,6 +141,13 @@ def test_independent_entries_keep_insertion_order(self) -> None: assert names == ["first", "second", "third"], names def test_ordering_is_deterministic_across_repeated_assembly(self) -> None: + """The same entries must assemble the same way every time. + + Asserted against the expected order rather than only against a second + run: comparing two runs to each other proves they agree but not that + they agree on the RIGHT thing, and a set-based implementation could + still be stable within one process while varying across them. + """ mod = self._mod() def build() -> list[str]: @@ -152,7 +159,13 @@ def build() -> list[str]: out = mod.assemble_with_chain(entries=entries, final=self._sel("base")) return [cte.alias_or_name for cte in out.args["with_"].expressions] - assert build() == build(), "assembly order is not deterministic" + # ``wm`` depends on ``base``, so ``base`` is pulled ahead of it; ``cm`` + # depends on nothing and keeps its declared position between them. + expected = ["base", "wm", "cm"] + first_run = build() + second_run = build() + assert first_run == expected, first_run + assert second_run == expected, second_run def test_a_dependency_cycle_raises(self) -> None: """A cycle cannot be emitted as a WITH chain at all. Failing loudly From fc9731d0ea25f88f42a52eece42a2008340df360 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Thu, 6 Aug 2026 14:58:52 +0200 Subject: [PATCH 47/98] DEV-1746: two fixes from the Codex implementation review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both are about a leak being silent rather than loud. 1. The combined projection's leftover pass appended ANY unconsumed rendered column, not just those of slots the projection never mentions. A slot the plan DID publish has exactly as many occurrences as declared names, so a leftover means the two disagree — and appending it emitted an extra public column, at the end, under a name the caller never asked for. The planner makes that unreachable today (one occurrence per alias), but the loop should not depend on an invariant it does not state. It now carries only never-mentioned slots and raises on the disagreement. 2. assemble_with_chain discarded a WITH clause already on `final`, leaving its references dangling. Production passes a fresh SELECT so nothing hit it, but PR 4 adopts this assembler for the transform chains, which build a statement that already has CTEs before wrapping it — exactly the caller that would have hit it. It now rejects that input and says to merge those CTEs into `entries` with their dependencies declared. Codex also flagged the B8 alias dedupe as a multiplicity change. Kept deliberately: two slots sharing a rendered alias made the old code emit the same column twice in a step CTE, which leaves the downstream reference ambiguous. The dedupe makes that shape valid rather than broken, and the allocator should prevent it arising at all. 11026 unit tests passing; ruff clean. Co-Authored-By: Claude Opus 5 (1M context) --- slayer/sql/generator.py | 15 ++++++++++++++- slayer/sql/render/cte_assembly.py | 11 +++++++++++ tests/test_dev1746_cte_assembly.py | 10 ++++++++++ 3 files changed, 35 insertions(+), 1 deletion(-) diff --git a/slayer/sql/generator.py b/slayer/sql/generator.py index 6a227414..4b92e8f1 100644 --- a/slayer/sql/generator.py +++ b/slayer/sql/generator.py @@ -5025,8 +5025,21 @@ def _render_outer_composite(cslot) -> exp.Expression: # a transform chain the combined SELECT is that chain's base CTE, so it # must also carry hidden inputs (transform operands, order-only slots) # for the step CTEs to read. The outer wrap trims them back afterwards. + # + # Only slots the projection never mentions are carried. A slot the plan + # DID publish has exactly as many occurrences as it has declared names, + # so a leftover would mean the two disagree — and appending it would + # emit an extra public column, at the end, under a name the caller did + # not ask for. Fail instead: a silent extra column is the harder bug. for sid, exprs in proj_exprs.items(): - combined_select_exprs.extend(exprs[consumed.get(sid, 0):]) + if sid not in consumed: + combined_select_exprs.extend(exprs) + elif consumed[sid] < len(exprs): + raise ValueError( + f"slot {sid!r} rendered {len(exprs)} column(s) but the " + f"projection consumed only {consumed[sid]}; the plan's " + f"declared names and the rendered columns disagree", + ) combined_select = exp.Select().select(*combined_select_exprs) combined_select = combined_select.from_("_base") diff --git a/slayer/sql/render/cte_assembly.py b/slayer/sql/render/cte_assembly.py index a80ae6ce..f35bdae3 100644 --- a/slayer/sql/render/cte_assembly.py +++ b/slayer/sql/render/cte_assembly.py @@ -56,10 +56,21 @@ def assemble_with_chain( Returns ``final`` unchanged when there are no entries — an empty ``WITH`` is not valid SQL. + ``final`` must not already carry a WITH clause. The assembler owns the + statement's CTE list, and silently discarding one the caller had attached + would leave its references dangling — a live hazard for the transform + chains, which build a statement that already has CTEs before wrapping it. + Raises ``ValueError`` on a duplicate name, a dependency naming a CTE that was not supplied, or a cycle. All three are wiring bugs whose SQL would be invalid or silently wrong, so they fail here rather than at the database. """ + if final.args.get("with_") is not None: + raise ValueError( + "assemble_with_chain owns the WITH clause, but `final` already " + "carries one; merge those CTEs into `entries` (with their " + "dependencies declared) rather than attaching them beforehand", + ) if not entries: return final diff --git a/tests/test_dev1746_cte_assembly.py b/tests/test_dev1746_cte_assembly.py index 2bb01526..c139f328 100644 --- a/tests/test_dev1746_cte_assembly.py +++ b/tests/test_dev1746_cte_assembly.py @@ -189,6 +189,16 @@ def test_duplicate_names_raise(self) -> None: with pytest.raises(ValueError, match="(?i)duplicate|dup"): mod.assemble_with_chain(entries=entries, final=self._sel("dup")) + def test_a_final_select_that_already_has_ctes_is_rejected(self) -> None: + """The assembler owns the WITH clause. Silently discarding one the + caller had attached would leave its references dangling — a live hazard + for the transform chains, which build a statement that already has CTEs + before wrapping it, and which adopt this assembler next.""" + mod = self._mod() + final = self._sel("seed").with_("seed", as_=self._sel()) + with pytest.raises(ValueError, match="(?i)already carries"): + mod.assemble_with_chain(entries=[self._entry("a", [])], final=final) + def test_no_entries_yields_the_final_select_unwrapped(self) -> None: """No CTEs means no WITH clause — not an empty one, which is invalid.""" mod = self._mod() From c3b5d9233481262d3bd27243fae3d3dd8a3f68be Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Thu, 6 Aug 2026 15:33:13 +0200 Subject: [PATCH 48/98] DEV-1746: fail loudly on a duplicate carried alias, and pin the projected-and- consumed slot MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit From the Codex follow-up review. _carry_aliases_in_plan_order silently deduped where the old sorted(...) did not. Silence was the wrong half to keep: two slots rendering the same alias is an allocator invariant violation, and collapsing it changes the stage's arity while hiding the cause. It now names both slots and raises. The whole suite passes unchanged, so nothing in the corpus produces a duplicate — the raise documents an invariant rather than guessing at one. Codex could not verify from the diff whether a slot can be BOTH publicly projected and carried as a transform input, which is the shape where the projection and the rendered columns could disagree and now trip the leftover guard. It can: `amount:sum` selected as `total` and also operated on by `cumsum` is one key, so one slot, reached by both. Pinned — one column per declared name, in declaration order, no leftover. 11027 unit tests passing; ruff clean. Co-Authored-By: Claude Opus 5 (1M context) --- slayer/sql/generator.py | 24 ++++++++++++++------ tests/test_dev1746_projection_order.py | 31 ++++++++++++++++++++++++++ 2 files changed, 48 insertions(+), 7 deletions(-) diff --git a/slayer/sql/generator.py b/slayer/sql/generator.py index 4b92e8f1..1a5f8416 100644 --- a/slayer/sql/generator.py +++ b/slayer/sql/generator.py @@ -948,16 +948,26 @@ def _carry_aliases_in_plan_order( ``aliases_by_slot_id`` is populated as slots are rendered, so its insertion order IS the plan's render order; iterating it directly is - what "plan order" means here. Duplicates are dropped (two slots can - share an alias) while preserving first appearance. + what "plan order" means here. + + A duplicate alias RAISES. Two slots sharing a rendered alias is an + allocator invariant violation: the old ``sorted(...)`` emitted the + column twice, which leaves the downstream ``SELECT "x" FROM step1`` + ambiguous, and silently collapsing it instead would change the stage's + arity while hiding the violation that caused it. """ out: List[str] = [] - seen: Set[str] = set() - for aliases in aliases_by_slot_id.values(): + owner_of: Dict[str, str] = {} + for sid, aliases in aliases_by_slot_id.items(): for alias in aliases: - if alias not in seen: - seen.add(alias) - out.append(alias) + if alias in owner_of: + raise ValueError( + f"slots {owner_of[alias]!r} and {sid!r} both render the " + f"alias {alias!r}; an inner stage cannot carry the same " + f"output name twice", + ) + owner_of[alias] = sid + out.append(alias) return out def _null_safe_join_pair_sql(self, *, left_sql: str, right_sql: str) -> str: diff --git a/tests/test_dev1746_projection_order.py b/tests/test_dev1746_projection_order.py index a6566210..0918432b 100644 --- a/tests/test_dev1746_projection_order.py +++ b/tests/test_dev1746_projection_order.py @@ -246,6 +246,37 @@ async def test_response_column_order_follows_declaration_order( "status", "cm_first", "local_second", ], f"row keys: {list(resp.data[0].keys())}" + async def test_a_slot_that_is_both_projected_and_a_transform_operand( + self, + ) -> None: + """A slot can be publicly projected AND consumed as a transform input. + + ``amount:sum`` is selected as ``total`` and is also what ``cumsum`` + operates on — one key, so one slot, reached by both. The combined SELECT + carries hidden transform inputs as well as public columns, so this is the + shape where the two could disagree about how many columns the slot + renders. It must emit the slot ONCE for its one declared name, and must + not trip the leftover-column guard. + """ + query = SlayerQuery( + source_model="orders_x", + time_dimensions=[TimeDimension( + dimension=ColumnRef(name="created_at"), + granularity=TimeGranularity.MONTH, + )], + measures=[ + ModelMeasure(formula="amount:sum", name="total"), + ModelMeasure(formula="cumsum(amount:sum)", name="running"), + ], + ) + sql = await _gen(query, dialect="postgres") + emitted = _alias_suffixes(outer_select_aliases(sql)) + assert emitted == ["created_at", "total", "running"], ( + f"expected each declared name once, in declaration order: " + f"{emitted}\n\n{sql}" + ) + assert emitted.count("total") == 1, emitted + async def test_hidden_slots_are_absent_from_the_projection(self) -> None: """Unified trimming: an order-only aggregate never reaches the public projection because it is not in ``projection`` at all.""" From b3ea0ebe391a786fa826ad7706e0dd123027cef5 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Thu, 6 Aug 2026 15:42:31 +0200 Subject: [PATCH 49/98] DEV-1746: make the shared-slot regression test actually test the guard MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Third Codex pass. Two findings, both on the test I added in the previous commit, and both fair. It was not self-verifying that the projected measure and the transform's operand are ONE slot — which is the entire premise. If the planner made two slots the test would pass while covering nothing. It now resolves the transform's `input` key back to a slot id and asserts that equals the projected measure's slot, and asserts the query takes the cross-model path and carries a transform chain, since both are preconditions for reaching the guard at all. More seriously, it asserted on the OUTERMOST select. The guard lives in the combined-SELECT builder, and with a transform chain the combined select becomes the `base` CTE — a leftover column appended there is carried through step1 and then trimmed by the outer wrap, so the outermost select cannot see it. The assertion moved to the `base` CTE's own alias list. (Matching that CTE needs `\bbase`: a bare `base` also matches the host `_base` CTE, which is a different scope and carries none of these columns — the first version of this test silently asserted against the wrong CTE and failed for the right reason.) The query also had to change to a cross-model shape: a purely local transform never enters the combined-SELECT builder, so the original `amount:sum` version did not reach the guard either. Also from that pass: the duplicate-alias raise reported "slots X and Y both render" even when X IS Y — a single slot listing the same alias twice now gets a message that says so. 11027 unit tests passing; ruff clean. Co-Authored-By: Claude Opus 5 (1M context) --- slayer/sql/generator.py | 13 +++- tests/test_dev1746_projection_order.py | 84 ++++++++++++++++++++++---- 2 files changed, 83 insertions(+), 14 deletions(-) diff --git a/slayer/sql/generator.py b/slayer/sql/generator.py index 1a5f8416..1b2f072a 100644 --- a/slayer/sql/generator.py +++ b/slayer/sql/generator.py @@ -960,10 +960,17 @@ def _carry_aliases_in_plan_order( owner_of: Dict[str, str] = {} for sid, aliases in aliases_by_slot_id.items(): for alias in aliases: - if alias in owner_of: + owner = owner_of.get(alias) + if owner == sid: raise ValueError( - f"slots {owner_of[alias]!r} and {sid!r} both render the " - f"alias {alias!r}; an inner stage cannot carry the same " + f"slot {sid!r} renders the alias {alias!r} more than " + f"once; an inner stage cannot carry the same output " + f"name twice", + ) + if owner is not None: + raise ValueError( + f"slots {owner!r} and {sid!r} both render the alias " + f"{alias!r}; an inner stage cannot carry the same " f"output name twice", ) owner_of[alias] = sid diff --git a/tests/test_dev1746_projection_order.py b/tests/test_dev1746_projection_order.py index 0918432b..b9c1bb53 100644 --- a/tests/test_dev1746_projection_order.py +++ b/tests/test_dev1746_projection_order.py @@ -39,6 +39,7 @@ from __future__ import annotations import os +import re import tempfile from typing import AsyncIterator, List @@ -69,7 +70,11 @@ outer_select_aliases, seed_dev1746_sqlite, ) -from tests._engine_helpers import _engine_generate, _join_aliases +from tests._engine_helpers import ( + _engine_generate, + _extract_cte_body, + _join_aliases, +) def _chain_bundle() -> ResolvedSourceBundle: @@ -251,12 +256,22 @@ async def test_a_slot_that_is_both_projected_and_a_transform_operand( ) -> None: """A slot can be publicly projected AND consumed as a transform input. - ``amount:sum`` is selected as ``total`` and is also what ``cumsum`` - operates on — one key, so one slot, reached by both. The combined SELECT - carries hidden transform inputs as well as public columns, so this is the - shape where the two could disagree about how many columns the slot - renders. It must emit the slot ONCE for its one declared name, and must - not trip the leftover-column guard. + This is the shape where the projection's occurrence count and the slot's + rendered columns could disagree, which is what the combined SELECT's + leftover guard exists to catch. + + Three things have to line up for the test to mean anything, so each is + asserted rather than assumed: + + * the measure and the transform's operand must be ONE slot (same key → + same slot), otherwise there is no slot reached by both paths; + * the query must take the CROSS-MODEL path, because the guard lives in + the combined-SELECT builder — a purely local transform never reaches + it; + * the assertion must be on the COMBINED select (which becomes the + transform chain's ``base`` CTE), because a leftover column appended + there would be trimmed by the outer wrap and never show up in the + outermost SELECT. """ query = SlayerQuery( source_model="orders_x", @@ -265,17 +280,64 @@ async def test_a_slot_that_is_both_projected_and_a_transform_operand( granularity=TimeGranularity.MONTH, )], measures=[ - ModelMeasure(formula="amount:sum", name="total"), - ModelMeasure(formula="cumsum(amount:sum)", name="running"), + ModelMeasure( + formula="customers_v2.lifetime_value:sum", name="ltv", + ), + ModelMeasure( + formula="cumsum(customers_v2.lifetime_value:sum)", + name="running", + ), ], ) + planned = plan_query(query=query, bundle=_chain_bundle()) + + # (1) One slot, reached by both. + slots = {s.id: s for s in _all_slots(planned)} + ltv_sid = next( + sid for sid in planned.projection + if slots[sid].public_name == "ltv" + ) + running_sid = next( + sid for sid in planned.projection + if slots[sid].public_name == "running" + ) + operand_key = slots[running_sid].key.input + operand_sid = next( + (sid for sid, s in slots.items() if s.key == operand_key), None, + ) + assert operand_sid == ltv_sid, ( + f"precondition: the transform's operand must be the SAME slot as " + f"the projected measure, or this shape does not exercise the " + f"guard. operand={operand_sid} projected={ltv_sid}" + ) + + # (2) The cross-model path — the guard lives in that builder. + assert planned.cross_model_aggregate_plans, ( + "precondition: this query must take the cross-model path" + ) + assert planned.transform_layers, ( + "precondition: this query must carry a transform chain" + ) + sql = await _gen(query, dialect="postgres") + + # (3) The combined SELECT — the transform chain's ``base`` CTE. + # ``\bbase`` and not ``base``: the latter also matches the host ``_base`` + # CTE, which is a different scope and carries none of these columns. + base_body = _extract_cte_body(sql, r"\bbase") + base_aliases = _alias_suffixes(re.findall(r'AS "([^"]+)"', base_body)) + assert base_aliases.count("ltv") == 1, ( + f"the combined SELECT carries {base_aliases.count('ltv')} columns " + f"for the shared slot; a leftover would be appended here and then " + f"trimmed by the outer wrap, invisible from the outermost SELECT.\n" + f"{base_body}" + ) + emitted = _alias_suffixes(outer_select_aliases(sql)) - assert emitted == ["created_at", "total", "running"], ( + assert emitted == ["created_at", "ltv", "running"], ( f"expected each declared name once, in declaration order: " f"{emitted}\n\n{sql}" ) - assert emitted.count("total") == 1, emitted async def test_hidden_slots_are_absent_from_the_projection(self) -> None: """Unified trimming: an order-only aggregate never reaches the public From 6f2cfd9c5fd5e8f2274b0f361396c69da252ba9d Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Thu, 6 Aug 2026 15:46:28 +0200 Subject: [PATCH 50/98] DEV-1746: parse the combined CTE's SELECT list instead of regexing it Fourth Codex pass, last finding. `re.findall(r'AS "([^"]+)"', base_body)` is not SELECT-list-aware: it missed the `created_at` column entirely (projected without an `AS`), so the list it built was half the CTE's output, and it would also match a CAST's quoted type name or an alias inside a nested subquery. Now parsed via the same `named_selects` path the response's column order is derived from, and the assertion tightened to the exact expected list rather than a count of one name. 11027 unit tests passing; ruff clean. Co-Authored-By: Claude Opus 5 (1M context) --- tests/test_dev1746_projection_order.py | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/tests/test_dev1746_projection_order.py b/tests/test_dev1746_projection_order.py index b9c1bb53..d2b25260 100644 --- a/tests/test_dev1746_projection_order.py +++ b/tests/test_dev1746_projection_order.py @@ -39,7 +39,6 @@ from __future__ import annotations import os -import re import tempfile from typing import AsyncIterator, List @@ -325,7 +324,14 @@ async def test_a_slot_that_is_both_projected_and_a_transform_operand( # ``\bbase`` and not ``base``: the latter also matches the host ``_base`` # CTE, which is a different scope and carries none of these columns. base_body = _extract_cte_body(sql, r"\bbase") - base_aliases = _alias_suffixes(re.findall(r'AS "([^"]+)"', base_body)) + # Parsed, not regexed: an ``AS "..."`` scan misses a column projected + # without an alias and would also match a CAST's type name or an alias + # inside a nested subquery. + base_aliases = _alias_suffixes(outer_select_aliases(base_body)) + assert base_aliases == ["created_at", "ltv"], ( + f"the combined SELECT should carry exactly the created_at grain and " + f"one column for the shared slot: {base_aliases}\n{base_body}" + ) assert base_aliases.count("ltv") == 1, ( f"the combined SELECT carries {base_aliases.count('ltv')} columns " f"for the shared slot; a leftover would be appended here and then " From a69866522923d51ef5519aa793a57abddb129b2d Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Thu, 6 Aug 2026 15:57:30 +0200 Subject: [PATCH 51/98] DEV-1746: drop the assertion the exact-list check subsumes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The exact-list assertion added in the previous commit already verifies cardinality, contents and order, so the `count("ltv") == 1` check beneath it was unreachable — it can only fail in a case the list comparison has already rejected. Its failure message was the useful part, so that moved into the surviving assertion. Final state of the Codex review chain: no open findings across the grain join-back builder, the WITH assembler and its ownership guard, the pagination hook, the combined-projection ordering and its leftover guard, the duplicate-alias raise, the isolation classifier, the empty-base plan node, and the is_root filter fix. 11027 unit + 508 integration passing; ruff clean. Co-Authored-By: Claude Opus 5 (1M context) --- tests/test_dev1746_projection_order.py | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/tests/test_dev1746_projection_order.py b/tests/test_dev1746_projection_order.py index d2b25260..e1f41083 100644 --- a/tests/test_dev1746_projection_order.py +++ b/tests/test_dev1746_projection_order.py @@ -329,14 +329,10 @@ async def test_a_slot_that_is_both_projected_and_a_transform_operand( # inside a nested subquery. base_aliases = _alias_suffixes(outer_select_aliases(base_body)) assert base_aliases == ["created_at", "ltv"], ( - f"the combined SELECT should carry exactly the created_at grain and " - f"one column for the shared slot: {base_aliases}\n{base_body}" - ) - assert base_aliases.count("ltv") == 1, ( - f"the combined SELECT carries {base_aliases.count('ltv')} columns " - f"for the shared slot; a leftover would be appended here and then " - f"trimmed by the outer wrap, invisible from the outermost SELECT.\n" - f"{base_body}" + f"the combined SELECT must carry the grain and exactly ONE column " + f"for the shared slot: {base_aliases}. A leftover would be appended " + f"here and then trimmed by the outer wrap, invisible from the " + f"outermost SELECT.\n{base_body}" ) emitted = _alias_suffixes(outer_select_aliases(sql)) From ad94281058bfbbc1c48a445223b7d7e65bb77cc4 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Thu, 6 Aug 2026 16:08:31 +0200 Subject: [PATCH 52/98] DEV-1746: fail on an under-rendered projection occurrence too (CodeRabbit) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The leftover guard caught a slot rendering MORE columns than the projection consumed, but the opposite direction was silent: when `idx >= len(exprs)` the loop simply skipped the occurrence, dropping a column the plan had asked for. A missing column is at least as bad as an extra one, and the comment right above claimed that failure mode must not stay silent. It now raises. NOT made symmetric in the other direction, deliberately: a slot with NO rendered columns is skipped and must stay skipped. A transform slot is in the plan's projection but is computed by a later step CTE and projected there, so its absence from the combined SELECT is correct — raising there would reject every transform query. Verified against a real plan: for `cumsum(customers_v2.lifetime_value:sum)` the projection is [created_at, ltv, running] where `running` is a TransformKey with no combined column. The regression test now asserts that shape explicitly, so the distinction is pinned rather than living in a comment. The new raise is unreachable in the corpus (11027 unit tests pass unchanged), so it documents an invariant rather than changing behaviour. Co-Authored-By: Claude Opus 5 (1M context) --- slayer/sql/generator.py | 20 ++++++++++++++++---- tests/test_dev1746_projection_order.py | 11 +++++++++++ 2 files changed, 27 insertions(+), 4 deletions(-) diff --git a/slayer/sql/generator.py b/slayer/sql/generator.py index 1b2f072a..afe1c3fd 100644 --- a/slayer/sql/generator.py +++ b/slayer/sql/generator.py @@ -5033,11 +5033,21 @@ def _render_outer_composite(cslot) -> exp.Expression: for sid in planned_query.projection: exprs = proj_exprs.get(sid) if not exprs: + # Not rendered by THIS scope at all. A transform slot is in the + # plan's projection but is computed by a later step CTE and + # projected there, so its absence here is correct — unlike a + # slot that renders some columns but fewer than the projection + # asks for, which is caught below. continue idx = consumed.get(sid, 0) - if idx < len(exprs): - combined_select_exprs.append(exprs[idx]) - consumed[sid] = idx + 1 + if idx >= len(exprs): + raise ValueError( + f"slot {sid!r} appears {idx + 1} times in the public " + f"projection but rendered only {len(exprs)} column(s); the " + f"occurrence would be dropped from the result", + ) + combined_select_exprs.append(exprs[idx]) + consumed[sid] = idx + 1 # Columns the plan does not publish but the statement still needs: with # a transform chain the combined SELECT is that chain's base CTE, so it # must also carry hidden inputs (transform operands, order-only slots) @@ -5047,7 +5057,9 @@ def _render_outer_composite(cslot) -> exp.Expression: # DID publish has exactly as many occurrences as it has declared names, # so a leftover would mean the two disagree — and appending it would # emit an extra public column, at the end, under a name the caller did - # not ask for. Fail instead: a silent extra column is the harder bug. + # not ask for. Fail instead. Both directions of that disagreement fail: + # too FEW rendered columns is caught in the loop above, where the + # occurrence would otherwise be dropped from the result. for sid, exprs in proj_exprs.items(): if sid not in consumed: combined_select_exprs.extend(exprs) diff --git a/tests/test_dev1746_projection_order.py b/tests/test_dev1746_projection_order.py index e1f41083..d52087e3 100644 --- a/tests/test_dev1746_projection_order.py +++ b/tests/test_dev1746_projection_order.py @@ -45,6 +45,7 @@ import pytest from slayer.core.enums import TimeGranularity +from slayer.core.keys import TransformKey from slayer.core.models import ModelMeasure from slayer.core.query import ColumnRef, OrderItem, SlayerQuery, TimeDimension from slayer.engine.planned import PlannedQuery, ValueSlot @@ -318,6 +319,16 @@ async def test_a_slot_that_is_both_projected_and_a_transform_operand( "precondition: this query must carry a transform chain" ) + # The transform slot is IN the projection but is rendered by a later + # step CTE, not by the combined SELECT. That is why the combined + # projection loop skips a slot with no rendered columns instead of + # treating it as a dropped column: making that case raise — the + # symmetric-looking guard — would reject every transform query. + assert running_sid in planned.projection, planned.projection + assert isinstance(slots[running_sid].key, TransformKey), ( + f"expected a transform slot, got {type(slots[running_sid].key).__name__}" + ) + sql = await _gen(query, dialect="postgres") # (3) The combined SELECT — the transform chain's ``base`` CTE. From 69a1de2023edec3ca8f8c961fabd2a4352bd2a4b Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Thu, 6 Aug 2026 16:22:24 +0200 Subject: [PATCH 53/98] DEV-1746: address the CodeRabbit review, including a real T-SQL ORDER BY bug MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The important one first. **Combined ORDER BY bypassed the T-SQL NULLS-emulation guard.** Converting the combined ORDER BY from text to AST introduced a local `exp.Ordered(...)` that leaves `nulls_first` unset. The generator has `_ordered` precisely to pin it: on T-SQL an unset `nulls_first` makes sqlglot emit ORDER BY CASE WHEN [alias] IS NULL THEN 1 ELSE 0 END, [alias] ASC and the bracketed alias inside that CASE does not resolve at the ORDER BY scope — SQL Server rejects it with `Invalid column name`. Reproduced, fixed by routing through `self._ordered`, and pinned for both directions. The whole PR's ordering tests used `direction="desc"`, and the wrapper only appears on ASC, which is exactly why 11k tests missed it. The old text-built ORDER BY never constructed an `Ordered` node, so this path was newly exposed by the AST migration. **Type annotations still described the pre-migration contract.** The CTE renderers now return sqlglot ASTs and the CTE containers hold ASTs, but `_render_cross_model_cte` and `_render_window_measure_cte_from_planned` still declared `Tuple[str, ...]`, and `cm_ctes`/`wm_ctes` still declared `List[Tuple[str, str]]`. A signature that promises `str` misdescribes the boundary this PR made structural. Test quality, all from the same review: - Lazy `importlib` indirection in two test modules protected against a missing implementation that this PR now provides; it only hid the dependency and violated the repo's imports-at-top rule. Same for a function-local `dev1746_models` import. - The B11 scope spy recorded paths from throwaway frames (agg-kwarg resolution, explicit time columns, Mode-A entry) whose `join_paths` are deliberately discarded, so a path registered there that the base FROM legitimately never emits would have failed the assertion for a CORRECT generator. Now filtered to the host scope. - The pagination hook test asserted `"10" in rendered and "5" in rendered`, which matches any digits anywhere and would pass if a bound were dropped and the other rendered as `105`. Now asserts the Select's `limit`/`offset` args. - `assemble_with_chain` was at cognitive complexity 16 against Sonar's limit of 15; the validation and ordering extracted into two helpers. - `carry_aliases_sorted` / `inner_sorted` no longer hold sorted lists — renamed at all 16 references. - Split a composite `and` assertion so the failure names which materialisation is missing; hoisted a bundle construction out of `pytest.raises`. 11029 unit + 508 integration passing; ruff clean. Co-Authored-By: Claude Opus 5 (1M context) --- slayer/sql/generator.py | 56 ++++++++------ slayer/sql/render/cte_assembly.py | 73 +++++++++++-------- .../test_dev1746_consumer_materialization.py | 5 +- tests/test_dev1746_cte_assembly.py | 52 ++++--------- tests/test_dev1746_null_safe_grain.py | 55 +++++--------- tests/test_dev1746_pagination.py | 13 +++- tests/test_dev1746_projection_order.py | 52 +++++++++++-- 7 files changed, 167 insertions(+), 139 deletions(-) diff --git a/slayer/sql/generator.py b/slayer/sql/generator.py index afe1c3fd..78996dae 100644 --- a/slayer/sql/generator.py +++ b/slayer/sql/generator.py @@ -1930,10 +1930,10 @@ def _generate_from_planned_impl( # NOSONAR(S3776) — top-level dispatch over c step_num += 1 step_name = cte_allocator.allocate_cte(f"step{step_num}") prev_cte = ctes[-1][0] - carry_aliases_sorted = self._carry_aliases_in_plan_order( + carry_aliases = self._carry_aliases_in_plan_order( aliases_by_slot_id, ) - step_parts = [self._quote_ident(a) for a in carry_aliases_sorted] + step_parts = [self._quote_ident(a) for a in carry_aliases] for layer in ready_window: for slot_id in layer.slot_ids: slot = slots_by_id[slot_id] @@ -2030,10 +2030,10 @@ def _generate_from_planned_impl( # NOSONAR(S3776) — top-level dispatch over c step_num += 1 step_name = cte_allocator.allocate_cte(f"step{step_num}") prev_cte = ctes[-1][0] - carry_aliases_sorted = self._carry_aliases_in_plan_order( + carry_aliases = self._carry_aliases_in_plan_order( aliases_by_slot_id, ) - step_parts = [self._quote_ident(a) for a in carry_aliases_sorted] + step_parts = [self._quote_ident(a) for a in carry_aliases] for cslot in unmaterialised: alias = ( cslot.public_aliases[0] @@ -2070,10 +2070,10 @@ def _generate_from_planned_impl( # NOSONAR(S3776) — top-level dispatch over c # in PLAN order (B8 — this list used to be sorted alphabetically to # match the legacy renderer byte-for-byte). final_cte = ctes[-1][0] - inner_sorted = self._carry_aliases_in_plan_order(aliases_by_slot_id) + inner_aliases = self._carry_aliases_in_plan_order(aliases_by_slot_id) inner_sql = ( "SELECT\n " - + _SQL_COL_SEP.join(self._quote_ident(a) for a in inner_sorted) + + _SQL_COL_SEP.join(self._quote_ident(a) for a in inner_aliases) + f"\nFROM {final_cte}" ) @@ -4059,7 +4059,7 @@ def _render_window_measure_cte_from_planned( # NOSONAR(S3776) — one cohesive slots_by_id: Dict[str, Any], aliases_by_slot_id: Dict[str, List[str]], full_agg_alias: str, - ) -> Tuple[str, List[str]]: + ) -> Tuple[exp.Select, List[str]]: """Render one ``_wm_`` duration-windowed-measure CTE (DEV-1714 Stage 10). The CTE is host-rooted: ``FROM _base LEFT JOIN (<_src>) AS _src`` where @@ -4667,7 +4667,7 @@ def _add_local_aux_slots( # canonical alias plus an ``AS`` remap at the combined # SELECT — matches the result-key contract while keeping # legacy parity for the unaliased shape. - cm_ctes: List[Tuple[str, str]] = [] + cm_ctes: List[Tuple[str, exp.Expression]] = [] # Dedup identity is the STRUCTURAL key (the typed AggregateKey plus the # source relation), never the sanitised CTE-name string. The canonical # alias omits the aggregate's column filter, so a filtered and an @@ -4761,7 +4761,7 @@ def _add_local_aux_slots( # is host-rooted (``FROM _base LEFT JOIN _src``), grouped at the query # grain, and joined back to ``_base`` on that grain (host alias == cte # column alias, since the CTE projects the grain under the same alias). - wm_ctes: List[Tuple[str, str]] = [] + wm_ctes: List[Tuple[str, exp.Expression]] = [] wm_cte_name_for_plan: Dict[str, str] = {} wm_agg_col_for_plan: Dict[str, str] = {} wm_joinback_pairs_for_plan: Dict[str, List[Tuple[str, str]]] = {} @@ -5368,10 +5368,10 @@ def _render_cross_model_transform_chain( # NOSONAR(S3776) — pre-existing comp step_num += 1 step_name = cte_allocator.allocate_cte(f"step{step_num}") prev_cte = ctes[-1][0] - carry_aliases_sorted = self._carry_aliases_in_plan_order( + carry_aliases = self._carry_aliases_in_plan_order( aliases_by_slot_id, ) - step_parts = [self._quote_ident(a) for a in carry_aliases_sorted] + step_parts = [self._quote_ident(a) for a in carry_aliases] for layer in ready: for slot_id in layer.slot_ids: slot = slots_by_id[slot_id] @@ -5422,10 +5422,10 @@ def _render_cross_model_transform_chain( # NOSONAR(S3776) — pre-existing comp step_num += 1 step_name = cte_allocator.allocate_cte(f"step{step_num}") prev_cte = ctes[-1][0] - carry_aliases_sorted = self._carry_aliases_in_plan_order( + carry_aliases = self._carry_aliases_in_plan_order( aliases_by_slot_id, ) - step_parts = [self._quote_ident(a) for a in carry_aliases_sorted] + step_parts = [self._quote_ident(a) for a in carry_aliases] for cslot in unmaterialised: alias = ( cslot.public_aliases[0] @@ -5454,10 +5454,10 @@ def _render_cross_model_transform_chain( # NOSONAR(S3776) — pre-existing comp ctes.append((step_name, step_sql)) final_cte = ctes[-1][0] - inner_sorted = self._carry_aliases_in_plan_order(aliases_by_slot_id) + inner_aliases = self._carry_aliases_in_plan_order(aliases_by_slot_id) inner_sql = ( "SELECT\n " - + _SQL_COL_SEP.join(self._quote_ident(a) for a in inner_sorted) + + _SQL_COL_SEP.join(self._quote_ident(a) for a in inner_aliases) + f"\nFROM {final_cte}" ) cte_clause = ( @@ -5657,10 +5657,14 @@ def _render_cross_model_cte( # NOSONAR(S3776) — single conceptual unit: share planned_query, slots_by_id: Dict[str, Any], base_projection_ids: Set[str], - ) -> Tuple[str, List[str]]: - """Render one ``_cm_<...>`` CTE body and return its SQL + + ) -> Tuple[exp.Select, List[str]]: + """Render one ``_cm_<...>`` CTE body and return it as AST, plus the shared-grain alias list (for the outer ``LEFT JOIN ON`` clause). + AST rather than SQL text: the caller assembles the WITH chain + structurally, and rendering here only to re-parse there would re-read a + dotted public alias as a multi-part reference on BigQuery. + The CTE is rooted at the terminal target model (legacy rerooted shape). Shared-grain slots whose key path is a prefix of the target_path participate as both projection and GROUP BY @@ -6731,10 +6735,16 @@ def _resolve_combined_order_term( cross-model alias map has no entry (the order slot can't be rendered). """ - descending = entry.direction != "asc" + ascending = entry.direction == "asc" def _ordered(col: exp.Expression) -> exp.Ordered: - return exp.Ordered(this=col, desc=descending) + # Through ``self._ordered``, NOT a bare ``exp.Ordered``: on T-SQL an + # unset ``nulls_first`` makes sqlglot emit a ``CASE WHEN IS + # NULL ...`` NULLS-emulation wrapper, and the bracketed alias inside + # it mis-resolves against the FROM scope (``Invalid column name``). + # The old text-built ORDER BY never constructed an ``Ordered`` node, + # so building one here newly exposed this path. + return self._ordered(col, ascending=ascending) # DEV-1712 / DEV-1733: a HIDDEN (order-only) aggregate that lives in its # own CTE — cross-model (``_cm_``) or windowed (``_wm_``) — is trimmed @@ -7975,11 +7985,11 @@ def _add_partition(pk_obj, *, where: str) -> None: # then add the shifted measure under EACH of the slot's public # aliases (DEV-1450 C13). prev_cte = ctes[-2][0] # the CTE just before the shifted CTE - carry_aliases_sorted = self._carry_aliases_in_plan_order( + carry_aliases = self._carry_aliases_in_plan_order( aliases_by_slot_id, ) sjoin_select_parts = [ - f'{prev_cte}.{self._quote_ident(a)}' for a in carry_aliases_sorted + f'{prev_cte}.{self._quote_ident(a)}' for a in carry_aliases ] slot_full_aliases: List[str] = [] for slot_alias in slot_aliases: @@ -8158,10 +8168,10 @@ def _emit_consecutive_periods_ctes_for_planned( # NOSONAR(S3776) — one cohesi # Build the reset CTE. prev_cte = ctes[-1][0] - carry_aliases_sorted = self._carry_aliases_in_plan_order( + carry_aliases = self._carry_aliases_in_plan_order( aliases_by_slot_id, ) - carry_select = ",\n ".join(self._quote_ident(a) for a in carry_aliases_sorted) + carry_select = ",\n ".join(self._quote_ident(a) for a in carry_aliases) partition_clause = ( _SQL_PARTITION_BY + ", ".join(self._quote_ident(a) for a in partition_aliases) if partition_aliases diff --git a/slayer/sql/render/cte_assembly.py b/slayer/sql/render/cte_assembly.py index f35bdae3..9abfe560 100644 --- a/slayer/sql/render/cte_assembly.py +++ b/slayer/sql/render/cte_assembly.py @@ -48,32 +48,12 @@ class CteEntry(BaseModel): depends_on: List[str] = [] -def assemble_with_chain( - *, entries: Sequence[CteEntry], final: exp.Select, -) -> exp.Select: - """Attach ``entries`` to ``final`` as a WITH clause in dependency order. - - Returns ``final`` unchanged when there are no entries — an empty ``WITH`` is - not valid SQL. +def _index_entries(entries: Sequence[CteEntry]) -> Dict[str, CteEntry]: + """Name → entry, rejecting a duplicate name or a dangling dependency. - ``final`` must not already carry a WITH clause. The assembler owns the - statement's CTE list, and silently discarding one the caller had attached - would leave its references dangling — a live hazard for the transform - chains, which build a statement that already has CTEs before wrapping it. - - Raises ``ValueError`` on a duplicate name, a dependency naming a CTE that - was not supplied, or a cycle. All three are wiring bugs whose SQL would be - invalid or silently wrong, so they fail here rather than at the database. + Both are wiring bugs whose SQL would be invalid or silently wrong, so they + fail here rather than at the database. """ - if final.args.get("with_") is not None: - raise ValueError( - "assemble_with_chain owns the WITH clause, but `final` already " - "carries one; merge those CTEs into `entries` (with their " - "dependencies declared) rather than attaching them beforehand", - ) - if not entries: - return final - by_name: Dict[str, CteEntry] = {} for entry in entries: if entry.name in by_name: @@ -81,7 +61,6 @@ def assemble_with_chain( f"duplicate CTE name {entry.name!r} in one WITH chain", ) by_name[entry.name] = entry - for entry in entries: unknown = [d for d in entry.depends_on if d not in by_name] if unknown: @@ -89,10 +68,16 @@ def assemble_with_chain( f"CTE {entry.name!r} declares unknown dependencies " f"{unknown!r}; known CTEs are {sorted(by_name)}", ) + return by_name - # Depth-first emit in declaration order: the first entry that is ready goes - # first, and a dependency is emitted immediately before the entry needing - # it. Declaration order is preserved wherever dependencies permit. + +def _topological_order( + *, entries: Sequence[CteEntry], by_name: Dict[str, CteEntry], +) -> List[CteEntry]: + """Depth-first emit in declaration order: the first entry that is ready goes + first, and a dependency is emitted immediately before the entry needing it. + Declaration order is preserved wherever dependencies permit. + """ ordered: List[CteEntry] = [] emitted: set[str] = set() visiting: List[str] = [] @@ -101,7 +86,9 @@ def _visit(entry: CteEntry) -> None: if entry.name in emitted: return if entry.name in visiting: - cycle = " -> ".join([*visiting[visiting.index(entry.name):], entry.name]) + cycle = " -> ".join( + [*visiting[visiting.index(entry.name):], entry.name], + ) raise ValueError(f"dependency cycle between CTEs: {cycle}") visiting.append(entry.name) for dep in entry.depends_on: @@ -112,9 +99,35 @@ def _visit(entry: CteEntry) -> None: for entry in entries: _visit(entry) + return ordered + + +def assemble_with_chain( + *, entries: Sequence[CteEntry], final: exp.Select, +) -> exp.Select: + """Attach ``entries`` to ``final`` as a WITH clause in dependency order. + + Returns ``final`` unchanged when there are no entries — an empty ``WITH`` is + not valid SQL. + + ``final`` must not already carry a WITH clause. The assembler owns the + statement's CTE list, and silently discarding one the caller had attached + would leave its references dangling — a live hazard for the transform + chains, which build a statement that already has CTEs before wrapping it. + """ + if final.args.get("with_") is not None: + raise ValueError( + "assemble_with_chain owns the WITH clause, but `final` already " + "carries one; merge those CTEs into `entries` (with their " + "dependencies declared) rather than attaching them beforehand", + ) + if not entries: + return final + + by_name = _index_entries(entries) + ordered = _topological_order(entries=entries, by_name=by_name) out = final.copy() - out.set("with_", None) for entry in ordered: out = out.with_(entry.name, as_=entry.query.copy(), copy=False) return out diff --git a/tests/test_dev1746_consumer_materialization.py b/tests/test_dev1746_consumer_materialization.py index 79b3a71f..e7dc5368 100644 --- a/tests/test_dev1746_consumer_materialization.py +++ b/tests/test_dev1746_consumer_materialization.py @@ -294,9 +294,8 @@ async def test_two_distinct_crossing_values_get_distinct_aliases( """Different values must never collapse onto one alias — that would silently aggregate the wrong column.""" sql = await _gen(_two_distinct_crossing_values_query(), dialect="postgres") - assert "_val_0" in sql and "_val_1" in sql, ( - f"expected two distinct materialisations:\n{sql}" - ) + assert "_val_0" in sql, f"first materialisation missing:\n{sql}" + assert "_val_1" in sql, f"second materialisation missing:\n{sql}" async def test_one_expression_is_materialised_once_per_scope(self) -> None: """NEWLY SURFACED DIVERGENCE — grouping a first/last cross-model diff --git a/tests/test_dev1746_cte_assembly.py b/tests/test_dev1746_cte_assembly.py index c139f328..39e9848e 100644 --- a/tests/test_dev1746_cte_assembly.py +++ b/tests/test_dev1746_cte_assembly.py @@ -37,14 +37,11 @@ from slayer.core.models import ModelMeasure from slayer.core.query import ColumnRef, SlayerQuery, TimeDimension from slayer.sql.naming import assert_unique_cte_names +from slayer.sql.render.cte_assembly import CteEntry, assemble_with_chain from tests._cross_model_chain import _gen from tests._dev1746_fixtures import cte_names_in_order -#: Imported lazily so a missing implementation fails these tests rather than -#: erroring collection for the whole module. -_ASSEMBLER_MODULE = "slayer.sql.render.cte_assembly" - DIALECTS = ["postgres", "sqlite", "duckdb", "tsql", "bigquery"] @@ -102,31 +99,23 @@ def _mixed_query() -> SlayerQuery: # =========================================================================== # class TestWithChainAssembler: - @staticmethod - def _mod(): - import importlib - - return importlib.import_module(_ASSEMBLER_MODULE) - @staticmethod def _sel(from_: str = "t") -> exp.Select: return exp.Select().select(exp.column("a")).from_(from_) - def _entry(self, name: str, deps: list[str]): - mod = self._mod() - return mod.CteEntry(name=name, query=self._sel(), depends_on=deps) + def _entry(self, name: str, deps: list[str]) -> CteEntry: + return CteEntry(name=name, query=self._sel(), depends_on=deps) def test_dependencies_precede_their_dependents(self) -> None: """The one hard ordering rule: a CTE is emitted after everything it declares a dependency on. SQL requires it — a CTE cannot reference a later sibling.""" - mod = self._mod() entries = [ self._entry("c", ["b"]), self._entry("b", ["a"]), self._entry("a", []), ] - out = mod.assemble_with_chain(entries=entries, final=self._sel("c")) + out = assemble_with_chain(entries=entries, final=self._sel("c")) names = [cte.alias_or_name for cte in out.args["with_"].expressions] assert names.index("a") < names.index("b") < names.index("c"), names @@ -134,9 +123,8 @@ def test_independent_entries_keep_insertion_order(self) -> None: """The tiebreak. Two CTEs with no dependency between them must come out in the order the caller declared them — otherwise emitted SQL would vary run to run for the same plan.""" - mod = self._mod() entries = [self._entry(n, []) for n in ("first", "second", "third")] - out = mod.assemble_with_chain(entries=entries, final=self._sel("first")) + out = assemble_with_chain(entries=entries, final=self._sel("first")) names = [cte.alias_or_name for cte in out.args["with_"].expressions] assert names == ["first", "second", "third"], names @@ -148,7 +136,6 @@ def test_ordering_is_deterministic_across_repeated_assembly(self) -> None: they agree on the RIGHT thing, and a set-based implementation could still be stable within one process while varying across them. """ - mod = self._mod() def build() -> list[str]: entries = [ @@ -156,7 +143,7 @@ def build() -> list[str]: self._entry("cm", []), self._entry("base", []), ] - out = mod.assemble_with_chain(entries=entries, final=self._sel("base")) + out = assemble_with_chain(entries=entries, final=self._sel("base")) return [cte.alias_or_name for cte in out.args["with_"].expressions] # ``wm`` depends on ``base``, so ``base`` is pulled ahead of it; ``cm`` @@ -170,46 +157,40 @@ def build() -> list[str]: def test_a_dependency_cycle_raises(self) -> None: """A cycle cannot be emitted as a WITH chain at all. Failing loudly beats emitting a plausible-looking order that references forward.""" - mod = self._mod() entries = [self._entry("a", ["b"]), self._entry("b", ["a"])] with pytest.raises(ValueError, match="(?i)cycle"): - mod.assemble_with_chain(entries=entries, final=self._sel("a")) + assemble_with_chain(entries=entries, final=self._sel("a")) def test_an_unknown_dependency_raises(self) -> None: """Declaring a dependency on a CTE that was never supplied is a wiring bug; silently ignoring it would emit SQL referencing a missing table.""" - mod = self._mod() entries = [self._entry("a", ["nope"])] with pytest.raises(ValueError, match="(?i)unknown|missing|nope"): - mod.assemble_with_chain(entries=entries, final=self._sel("a")) + assemble_with_chain(entries=entries, final=self._sel("a")) def test_duplicate_names_raise(self) -> None: - mod = self._mod() entries = [self._entry("dup", []), self._entry("dup", [])] with pytest.raises(ValueError, match="(?i)duplicate|dup"): - mod.assemble_with_chain(entries=entries, final=self._sel("dup")) + assemble_with_chain(entries=entries, final=self._sel("dup")) def test_a_final_select_that_already_has_ctes_is_rejected(self) -> None: """The assembler owns the WITH clause. Silently discarding one the caller had attached would leave its references dangling — a live hazard for the transform chains, which build a statement that already has CTEs before wrapping it, and which adopt this assembler next.""" - mod = self._mod() final = self._sel("seed").with_("seed", as_=self._sel()) with pytest.raises(ValueError, match="(?i)already carries"): - mod.assemble_with_chain(entries=[self._entry("a", [])], final=final) + assemble_with_chain(entries=[self._entry("a", [])], final=final) def test_no_entries_yields_the_final_select_unwrapped(self) -> None: """No CTEs means no WITH clause — not an empty one, which is invalid.""" - mod = self._mod() - out = mod.assemble_with_chain(entries=[], final=self._sel()) + out = assemble_with_chain(entries=[], final=self._sel()) assert out.args.get("with_") is None, out.sql() def test_assembled_statement_is_a_select_not_a_string(self) -> None: """The point of §5.6: the chain is AST all the way, so a caller can keep transforming it (pagination, outer wraps) without re-parsing.""" - mod = self._mod() - out = mod.assemble_with_chain( + out = assemble_with_chain( entries=[self._entry("a", [])], final=self._sel("a"), ) assert isinstance(out, exp.Select), type(out) @@ -218,9 +199,8 @@ def test_assembled_statement_is_a_select_not_a_string(self) -> None: def test_assembled_statement_round_trips_through_every_dialect( self, dialect: str, ) -> None: - mod = self._mod() entries = [self._entry("base", []), self._entry("wm", ["base"])] - out = mod.assemble_with_chain(entries=entries, final=self._sel("wm")) + out = assemble_with_chain(entries=entries, final=self._sel("wm")) rendered = out.sql(dialect=dialect) parsed = sqlglot.parse(rendered, dialect=dialect) assert len(parsed) == 1, f"[{dialect}] did not round-trip:\n{rendered}" @@ -228,9 +208,8 @@ def test_assembled_statement_round_trips_through_every_dialect( def test_quoted_and_mixed_case_names_survive_assembly(self) -> None: """A quoted alias must stay one identifier through assembly.""" - mod = self._mod() entries = [self._entry("MixedCase", []), self._entry("other", ["MixedCase"])] - out = mod.assemble_with_chain(entries=entries, final=self._sel("other")) + out = assemble_with_chain(entries=entries, final=self._sel("other")) names = [cte.alias_or_name for cte in out.args["with_"].expressions] assert names == ["MixedCase", "other"], names @@ -238,9 +217,8 @@ def test_case_folding_duplicates_are_rejected(self) -> None: """DEV-1726: two names differing only in case collide on a folding dialect. The belt catches it in emitted SQL; the assembler must not be the thing that introduces it.""" - mod = self._mod() entries = [self._entry("dup", []), self._entry("DUP", [])] - out = mod.assemble_with_chain(entries=entries, final=self._sel("dup")) + out = assemble_with_chain(entries=entries, final=self._sel("dup")) with pytest.raises(ValueError): assert_unique_cte_names(out.sql(dialect="snowflake"), dialect="snowflake") diff --git a/tests/test_dev1746_null_safe_grain.py b/tests/test_dev1746_null_safe_grain.py index b6bf0327..b1393657 100644 --- a/tests/test_dev1746_null_safe_grain.py +++ b/tests/test_dev1746_null_safe_grain.py @@ -31,9 +31,6 @@ cover — the dotted-alias mangling itself — is asserted as emission per §5.13, because neither SQLite nor DuckDB mangles dots. -The new shared builder is imported inside the tests that exercise it directly, -so a missing implementation fails those tests rather than erroring collection -for the whole module. """ from __future__ import annotations @@ -51,6 +48,10 @@ from slayer.core.query import ColumnRef, SlayerQuery, TimeDimension from slayer.engine.query_engine import SlayerQueryEngine from slayer.sql.dialects import get_dialect +from slayer.sql.render.joins import ( + build_grain_joinback_condition, + grain_alias_column, +) from slayer.sql.scope_check import assert_scope_closed from tests._cross_model_chain import _gen @@ -66,11 +67,6 @@ ) from tests._engine_helpers import _norm -#: The module under construction. Imported lazily inside tests so a missing -#: implementation fails the builder tests, not module collection. -_BUILDER_MODULE = "slayer.sql.render.joins" - - # --------------------------------------------------------------------------- # # Fixtures # --------------------------------------------------------------------------- # @@ -412,18 +408,11 @@ class TestSharedGrainJoinBackBuilder: covers today's three callers, which all compare projected aliases. """ - @staticmethod - def _import(): - import importlib - - return importlib.import_module(_BUILDER_MODULE) - def test_builder_returns_none_for_an_empty_grain(self) -> None: """Zero-column grain → no predicate at all; the caller emits CROSS JOIN. Returning a truthy ``TRUE`` instead would silently turn every scalar CMA into an inner-join-shaped ON clause.""" - mod = self._import() - assert mod.build_grain_joinback_condition( + assert build_grain_joinback_condition( pairs=[], dialect=get_dialect("postgres"), ) is None @@ -443,11 +432,10 @@ def test_builder_emits_the_dialect_null_safe_form( ) -> None: """One builder, every dialect's own null-safe spelling — including the expanded ``a = b OR (a IS NULL AND b IS NULL)`` fallback on T-SQL.""" - mod = self._import() strategy = get_dialect(dialect) - left = mod.grain_alias_column(alias="orders.status", table="_base") - right = mod.grain_alias_column(alias="orders.status", table="_cm_x") - cond = mod.build_grain_joinback_condition( + left = grain_alias_column(alias="orders.status", table="_base") + right = grain_alias_column(alias="orders.status", table="_cm_x") + cond = build_grain_joinback_condition( pairs=[(left, right)], dialect=strategy, ) assert cond is not None @@ -460,11 +448,10 @@ def test_builder_emits_the_dialect_null_safe_form( def test_dotted_alias_stays_one_identifier(self, dialect: str) -> None: """The B2 defect in miniature: a dotted PUBLIC ALIAS is one identifier, never a ``table.column`` reference. Built as AST it cannot decompose.""" - mod = self._import() strategy = get_dialect(dialect) - left = mod.grain_alias_column(alias="orders.customers.status", table="_base") - cond = mod.build_grain_joinback_condition( - pairs=[(left, mod.grain_alias_column( + left = grain_alias_column(alias="orders.customers.status", table="_base") + cond = build_grain_joinback_condition( + pairs=[(left, grain_alias_column( alias="orders.customers.status", table="_cm_x"))], dialect=strategy, ) @@ -480,9 +467,8 @@ def test_dotted_alias_stays_one_identifier(self, dialect: str) -> None: def test_alias_containing_a_quote_is_not_injectable(self) -> None: """An embedded quote must survive as data inside one identifier.""" - mod = self._import() weird = 'orders."evil' - col = mod.grain_alias_column(alias=weird, table="_base") + col = grain_alias_column(alias=weird, table="_base") assert col.name == weird, col.name rendered = col.sql(dialect="postgres") assert rendered.startswith('_base.'), rendered @@ -497,21 +483,19 @@ def test_alias_containing_a_quote_is_not_injectable(self) -> None: def test_case_sensitive_alias_is_quoted(self) -> None: """Mixed-case aliases must stay quoted, or a case-folding dialect resolves them to a different column.""" - mod = self._import() - col = mod.grain_alias_column(alias="Orders.Status", table="_base") + col = grain_alias_column(alias="Orders.Status", table="_base") rendered = col.sql(dialect="postgres") assert '"Orders.Status"' in rendered, rendered def test_composite_grain_ands_every_pair(self) -> None: - mod = self._import() strategy = get_dialect("postgres") pairs = [ - (mod.grain_alias_column(alias="a", table="_base"), - mod.grain_alias_column(alias="a", table="_cm_x")), - (mod.grain_alias_column(alias="b", table="_base"), - mod.grain_alias_column(alias="b", table="_cm_x")), + (grain_alias_column(alias="a", table="_base"), + grain_alias_column(alias="a", table="_cm_x")), + (grain_alias_column(alias="b", table="_base"), + grain_alias_column(alias="b", table="_cm_x")), ] - cond = mod.build_grain_joinback_condition(pairs=pairs, dialect=strategy) + cond = build_grain_joinback_condition(pairs=pairs, dialect=strategy) assert cond is not None rendered = cond.sql(dialect="postgres") assert rendered.count("IS NOT DISTINCT FROM") == 2, rendered @@ -520,11 +504,10 @@ def test_composite_grain_ands_every_pair(self) -> None: def test_builder_accepts_arbitrary_expression_operands(self) -> None: """Codex D2: the core API takes expressions, so a caller can compare a CAST or any resolved reference — not only a projected alias.""" - mod = self._import() strategy = get_dialect("postgres") left = exp.cast(exp.column("x", table="_base"), "DATE") right = exp.column("y", table="_cm_x") - cond = mod.build_grain_joinback_condition( + cond = build_grain_joinback_condition( pairs=[(left, right)], dialect=strategy, ) assert cond is not None diff --git a/tests/test_dev1746_pagination.py b/tests/test_dev1746_pagination.py index 01c84927..aa57a6bf 100644 --- a/tests/test_dev1746_pagination.py +++ b/tests/test_dev1746_pagination.py @@ -47,6 +47,7 @@ from slayer.sql.dialects import get_dialect from tests._dev1746_fixtures import ( + dev1746_models, make_sqlite_engine, outer_clause_sql, outer_statement, @@ -125,8 +126,6 @@ def _combined_query( async def _gen_sql(query: SlayerQuery, *, dialect: str) -> str: - from tests._dev1746_fixtures import dev1746_models - models = dev1746_models() return await _engine_generate( query=query, model=models[0], dialect=dialect, extra_models=models[1:], @@ -291,8 +290,14 @@ def test_hook_returns_a_select_that_renders_both_bounds( f"on a free-standing Limit." ) rendered = out.sql(dialect=dialect) - assert "10" in rendered and "5" in rendered, ( - f"[{dialect}] pagination bounds missing from {rendered!r}" + # Structural, not substring: ``"10" in rendered`` matches any digits + # anywhere — a column name, the other bound's digits, or a stray + # literal — so it would pass even if one bound were dropped. + assert out.args.get("limit") is not None, ( + f"[{dialect}] no LIMIT bound on the Select: {rendered!r}" + ) + assert out.args.get("offset") is not None, ( + f"[{dialect}] no OFFSET bound on the Select: {rendered!r}" ) def test_tsql_hook_injects_ordering_for_a_bare_offset(self) -> None: diff --git a/tests/test_dev1746_projection_order.py b/tests/test_dev1746_projection_order.py index d52087e3..3c4a7914 100644 --- a/tests/test_dev1746_projection_order.py +++ b/tests/test_dev1746_projection_order.py @@ -352,6 +352,41 @@ async def test_a_slot_that_is_both_projected_and_a_transform_operand( f"{emitted}\n\n{sql}" ) + @pytest.mark.parametrize("direction", ["asc", "desc"]) + async def test_combined_order_by_suppresses_the_tsql_nulls_emulation( + self, direction: str, + ) -> None: + """The combined ORDER BY must go through the generator's ``_ordered``. + + On T-SQL an unset ``nulls_first`` makes sqlglot emulate NULLS ordering + with ``ORDER BY CASE WHEN IS NULL THEN 1 ELSE 0 END, ``, + and the bracketed alias inside that CASE mis-resolves against the FROM + scope — SQL Server rejects it with ``Invalid column name``. The + generator has ``_ordered`` precisely to pin ``nulls_first`` and suppress + the wrapper. + + This became reachable only when the combined ORDER BY moved from text to + AST: the string form never built an ``Ordered`` node, so there was + nothing for sqlglot to emulate around. Both directions are covered + because the wrapper appears on ASC only — every other ordering test in + this PR used ``desc`` and would not have caught it. + """ + query = SlayerQuery( + source_model="orders_x", + dimensions=[ColumnRef(name="customers_v2.status")], + measures=[ModelMeasure( + formula="customers_v2.lifetime_value:sum", name="ltv", + )], + order=[OrderItem( + column="customers_v2.lifetime_value:sum", direction=direction, + )], + ) + sql = await _gen(query, dialect="tsql") + assert "CASE WHEN" not in sql.upper(), ( + f"[{direction}] the T-SQL NULLS-emulation wrapper is back; its " + f"bracketed alias does not resolve at the ORDER BY scope:\n{sql}" + ) + async def test_hidden_slots_are_absent_from_the_projection(self) -> None: """Unified trimming: an order-only aggregate never reaches the public projection because it is not in ``projection`` at all.""" @@ -488,11 +523,10 @@ def test_renderer_belt_catches_a_model_copy_that_skips_validation( for s in planned.aggregate_slots ] corrupted = planned.model_copy(update=update) + bundle = _chain_bundle() gen = SQLGenerator(dialect="postgres") with pytest.raises((AssertionError, ValueError)) as excinfo: - gen.generate_from_planned( - planned_query=corrupted, bundle=_chain_bundle(), - ) + gen.generate_from_planned(planned_query=corrupted, bundle=bundle) message = str(excinfo.value).lower() assert "hidden" in message, ( "the belt fired, but not for the hidden slot — the message does not " @@ -954,9 +988,15 @@ async def test_every_registered_scope_path_is_emitted_as_a_join( def _wrapped(self_frame, parsed): result = original(self_frame, parsed) - registered.extend( - "__".join(p) for p in self_frame.join_paths.as_list() - ) + # Only the HOST scope. The generator also builds throwaway frames + # (agg-kwarg resolution, explicit time columns, Mode-A entry) whose + # join_paths are documented as intentionally discarded — a path + # registered there need not appear in the base FROM, so recording + # them would make this property false for a correct generator. + if self_frame.root_relation == "orders_x": + registered.extend( + "__".join(p) for p in self_frame.join_paths.as_list() + ) return result monkeypatch.setattr( From 79b8a0ceca1e526fa27635e1302e0b67bbca2d6b Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Thu, 6 Aug 2026 16:46:31 +0200 Subject: [PATCH 54/98] DEV-1746: clear the 10 SonarCloud issues and route the _wm_ inner grain through the shared builder Sonar S5778 (5x, cte_assembly tests): hoist the setup call out of the pytest.raises block so only the call under test can throw. Sonar S9073 (5x, three test modules): split composite 'and' assertions so a failure names which half broke. Codex (minor): the windowed CTE's inner _base<->_src grain correlation built its null-safe equalities pair-by-pair instead of going through build_grain_joinback_condition, diverging from the single-builder doctrine this PR establishes. An empty grain yields None there, which is NOT the builder's CROSS JOIN case: the range predicates still correlate the two sides, so it stays a LEFT JOIN. Co-Authored-By: Claude Opus 5 (1M context) --- slayer/sql/generator.py | 28 +++++++++++++++------- tests/test_dev1746_cte_assembly.py | 22 +++++++++++------ tests/test_dev1746_isolation_classifier.py | 5 ++-- tests/test_dev1746_null_safe_grain.py | 16 +++++++++---- 4 files changed, 49 insertions(+), 22 deletions(-) diff --git a/slayer/sql/generator.py b/slayer/sql/generator.py index 78996dae..3700340d 100644 --- a/slayer/sql/generator.py +++ b/slayer/sql/generator.py @@ -4102,7 +4102,12 @@ def _alias_of(sid: str) -> str: return al[0] if al else sid src_cols: List[exp.Expression] = [] - join_eqs: List[exp.Expression] = [] + # Grain operand pairs for the inner ``_base``↔``_src`` correlation. They + # go through the same builder as the outer join-back (P-I) rather than + # calling ``build_null_safe_eq`` per pair here, so both sites share one + # answer to "how is a grain compared" — including whatever a dialect + # later needs that a bare per-pair equality could not express. + grain_pairs: List[Tuple[exp.Expression, exp.Expression]] = [] grain_aliases: List[str] = [] # Query dimensions → ``_w_dim_`` (Law-1 resolve registers crossed @@ -4112,9 +4117,9 @@ def _alias_of(sid: str) -> str: base_alias = _alias_of(sid) expr = src_scope.resolve(dslot.key) src_cols.append(expr.as_(f"_w_dim_{idx}")) - join_eqs.append(self._dialect.build_null_safe_eq( - _src_col(f"_w_dim_{idx}"), _base_col(base_alias), - )) + grain_pairs.append( + (_src_col(f"_w_dim_{idx}"), _base_col(base_alias)), + ) grain_aliases.append(base_alias) # Non-window time dimensions → ``_w_td_`` (date-trunc'd), equality- @@ -4135,9 +4140,9 @@ def _alias_of(sid: str) -> str: col_expr=raw, granularity=TimeGranularity(tslot.key.granularity), ) src_cols.append(trunc.as_(f"_w_td_{idx}")) - join_eqs.append(self._dialect.build_null_safe_eq( - _src_col(f"_w_td_{idx}"), _base_col(base_alias), - )) + grain_pairs.append( + (_src_col(f"_w_td_{idx}"), _base_col(base_alias)), + ) grain_aliases.append(base_alias) # The window time dimension's RAW column → ``_w_time`` (the range axis). @@ -4216,8 +4221,15 @@ def _alias_of(sid: str) -> str: sign=-1, ) src_w_time = _src_col("_w_time") + # An empty grain yields ``None`` here — the windowed measure is scalar + # over the whole host, so the ON carries only the range bounds. That is + # NOT the builder's CROSS-JOIN case: the range predicates still + # correlate the two sides, so this stays a LEFT JOIN either way. + grain_condition = build_grain_joinback_condition( + pairs=grain_pairs, dialect=self._dialect, + ) on_range = exp.and_( - *join_eqs, + *([grain_condition] if grain_condition is not None else []), exp.GTE(this=src_w_time, expression=lower_bound), exp.LT(this=src_w_time.copy(), expression=bucket_end.copy()), ) diff --git a/tests/test_dev1746_cte_assembly.py b/tests/test_dev1746_cte_assembly.py index 39e9848e..8fcd420b 100644 --- a/tests/test_dev1746_cte_assembly.py +++ b/tests/test_dev1746_cte_assembly.py @@ -158,20 +158,23 @@ def test_a_dependency_cycle_raises(self) -> None: """A cycle cannot be emitted as a WITH chain at all. Failing loudly beats emitting a plausible-looking order that references forward.""" entries = [self._entry("a", ["b"]), self._entry("b", ["a"])] + final = self._sel("a") with pytest.raises(ValueError, match="(?i)cycle"): - assemble_with_chain(entries=entries, final=self._sel("a")) + assemble_with_chain(entries=entries, final=final) def test_an_unknown_dependency_raises(self) -> None: """Declaring a dependency on a CTE that was never supplied is a wiring bug; silently ignoring it would emit SQL referencing a missing table.""" entries = [self._entry("a", ["nope"])] + final = self._sel("a") with pytest.raises(ValueError, match="(?i)unknown|missing|nope"): - assemble_with_chain(entries=entries, final=self._sel("a")) + assemble_with_chain(entries=entries, final=final) def test_duplicate_names_raise(self) -> None: entries = [self._entry("dup", []), self._entry("dup", [])] + final = self._sel("dup") with pytest.raises(ValueError, match="(?i)duplicate|dup"): - assemble_with_chain(entries=entries, final=self._sel("dup")) + assemble_with_chain(entries=entries, final=final) def test_a_final_select_that_already_has_ctes_is_rejected(self) -> None: """The assembler owns the WITH clause. Silently discarding one the @@ -179,8 +182,9 @@ def test_a_final_select_that_already_has_ctes_is_rejected(self) -> None: for the transform chains, which build a statement that already has CTEs before wrapping it, and which adopt this assembler next.""" final = self._sel("seed").with_("seed", as_=self._sel()) + entries = [self._entry("a", [])] with pytest.raises(ValueError, match="(?i)already carries"): - assemble_with_chain(entries=[self._entry("a", [])], final=final) + assemble_with_chain(entries=entries, final=final) def test_no_entries_yields_the_final_select_unwrapped(self) -> None: """No CTEs means no WITH clause — not an empty one, which is invalid.""" @@ -219,8 +223,9 @@ def test_case_folding_duplicates_are_rejected(self) -> None: the thing that introduces it.""" entries = [self._entry("dup", []), self._entry("DUP", [])] out = assemble_with_chain(entries=entries, final=self._sel("dup")) + rendered = out.sql(dialect="snowflake") with pytest.raises(ValueError): - assert_unique_cte_names(out.sql(dialect="snowflake"), dialect="snowflake") + assert_unique_cte_names(rendered, dialect="snowflake") # =========================================================================== # @@ -260,8 +265,11 @@ async def test_two_independent_cross_model_ctes_follow_declaration_order( sql = await _gen(_two_cross_model_measures_query(), dialect="postgres") names = [n for n in cte_names_in_order(sql) if n.startswith("_cm_")] assert len(names) == 2, f"expected two _cm_ CTEs, got {names}" - assert "sum" in names[0] and "avg" in names[1], ( - f"_cm_ CTEs are not in measure-declaration order: {names}" + assert "sum" in names[0], ( + f"the first _cm_ CTE is not the sum measure: {names}" + ) + assert "avg" in names[1], ( + f"the second _cm_ CTE is not the avg measure: {names}" ) @pytest.mark.parametrize("dialect", DIALECTS) diff --git a/tests/test_dev1746_isolation_classifier.py b/tests/test_dev1746_isolation_classifier.py index 13c05173..1aca3276 100644 --- a/tests/test_dev1746_isolation_classifier.py +++ b/tests/test_dev1746_isolation_classifier.py @@ -293,9 +293,8 @@ def test_every_isolated_slot_is_classified_isolated_and_no_other( if sid in wm_ids: assert kind is IsolationKind.WINDOWED, (sid, kind) elif sid in cm_ids: - assert kind.needs_own_cte and kind is not IsolationKind.WINDOWED, ( - sid, kind, - ) + assert kind.needs_own_cte, (sid, kind) + assert kind is not IsolationKind.WINDOWED, (sid, kind) else: assert kind is IsolationKind.NONE, ( f"slot {sid} classified {kind} but the planner built no CTE " diff --git a/tests/test_dev1746_null_safe_grain.py b/tests/test_dev1746_null_safe_grain.py index b1393657..e0e49aad 100644 --- a/tests/test_dev1746_null_safe_grain.py +++ b/tests/test_dev1746_null_safe_grain.py @@ -475,7 +475,11 @@ def test_alias_containing_a_quote_is_not_injectable(self) -> None: # Re-parsing must give back exactly one column with the same name. reparsed = sqlglot.parse_one(f"SELECT {rendered}", dialect="postgres") cols = list(reparsed.find_all(exp.Column)) - assert len(cols) == 1 and cols[0].name == weird, ( + assert len(cols) == 1, ( + f"identifier did not survive a round trip as ONE column: " + f"{rendered!r} -> {[c.name for c in cols]}" + ) + assert cols[0].name == weird, ( f"identifier did not survive a round trip: {rendered!r} -> " f"{[c.name for c in cols]}" ) @@ -512,7 +516,8 @@ def test_builder_accepts_arbitrary_expression_operands(self) -> None: ) assert cond is not None rendered = cond.sql(dialect="postgres") - assert "CAST(" in rendered and "IS NOT DISTINCT FROM" in rendered, rendered + assert "CAST(" in rendered, rendered + assert "IS NOT DISTINCT FROM" in rendered, rendered # =========================================================================== # @@ -541,8 +546,11 @@ async def test_tsql_uses_the_expanded_fallback(self) -> None: ``a = b OR (a IS NULL AND b IS NULL)`` must appear.""" sql = await _gen(_cm_shared_grain_query(), dialect="tsql") on = _norm(joinback_on_predicate_for(sql, prefix="_cm_", dialect="tsql")) - assert " OR " in on and "IS NULL" in on, ( - f"tsql join-back is missing the expanded null-safe form:\n{on}" + assert " OR " in on, ( + f"tsql join-back is missing the expanded null-safe disjunction:\n{on}" + ) + assert "IS NULL" in on, ( + f"tsql join-back is missing the expanded null-safe NULL tests:\n{on}" ) async def test_mysql_null_safe_operator_is_emitted(self) -> None: From a224cb20847ca4012b050f2393718b6754b6786f Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Thu, 6 Aug 2026 17:10:38 +0200 Subject: [PATCH 55/98] DEV-1747: the test suite, written first 194 tests across eight modules, plus a shared fixture corpus. Every one of them fails right now, and each fails because the feature is absent rather than because the setup is wrong -- the distinction the TDD step exists to protect, and the reason the suite lands before a line of implementation. What they pin: * reroot_visitor -- one re-rooting rule over the whole ValueKey union, including the asymmetry that matters: an AggregateKey's column_filter_key is owner-anchored and must NOT strip, while a standalone SqlExprKey must. * prebound_planner -- the seam. Equivalence across seven query shapes, plus runtime sentinels (not source greps) proving the parser is never entered inside the reroot subtree. * grouped_joined_order / order_entry -- the host-grain marker and the direction-aware wrap. * reroot_filter_routing -- the B6 defect, reproduced: reachable, host-local and unreachable filters all report `where=[] having=[] applied=[] dropped=[]`, indistinguishable from one another. * order_resolver -- the D4 defect, reproduced per render path: four of the five silently drop an unresolvable ORDER BY term. * local_with_chain / derived_crossing_order -- D8 and D9. Sentinels are runtime wherever a claim is about the production path. A grep for `_reroot_ref(` also matches a docstring or a dead branch, and stops matching the moment someone renames the symbol; a function that raises when called is exactly as strong as the claim being made. Co-Authored-By: Claude Opus 5 (1M context) --- slayer/engine/prebound.py | 262 ++++++++++ tests/_dev1747_fixtures.py | 504 ++++++++++++++++++ tests/test_dev1747_derived_crossing_order.py | 233 +++++++++ tests/test_dev1747_grouped_joined_order.py | 420 +++++++++++++++ tests/test_dev1747_local_with_chain.py | 280 ++++++++++ tests/test_dev1747_order_entry.py | 447 ++++++++++++++++ tests/test_dev1747_order_resolver.py | 508 ++++++++++++++++++ tests/test_dev1747_prebound_planner.py | 345 +++++++++++++ tests/test_dev1747_reroot_filter_routing.py | 510 +++++++++++++++++++ tests/test_dev1747_reroot_visitor.py | 503 ++++++++++++++++++ 10 files changed, 4012 insertions(+) create mode 100644 slayer/engine/prebound.py create mode 100644 tests/_dev1747_fixtures.py create mode 100644 tests/test_dev1747_derived_crossing_order.py create mode 100644 tests/test_dev1747_grouped_joined_order.py create mode 100644 tests/test_dev1747_local_with_chain.py create mode 100644 tests/test_dev1747_order_entry.py create mode 100644 tests/test_dev1747_order_resolver.py create mode 100644 tests/test_dev1747_prebound_planner.py create mode 100644 tests/test_dev1747_reroot_filter_routing.py create mode 100644 tests/test_dev1747_reroot_visitor.py diff --git a/slayer/engine/prebound.py b/slayer/engine/prebound.py new file mode 100644 index 00000000..5a51c5d3 --- /dev/null +++ b/slayer/engine/prebound.py @@ -0,0 +1,262 @@ +"""The pre-bound planner seam (DEV-1742 §5.4, P-E). + +``plan_query`` used to be the only door into binding: hand it a +``SlayerQuery`` and it parsed every measure / filter / order string, bound +each against a scope, and planned the result in one pass. Re-rooting needed a +nested plan built from keys it already held, so it SERIALIZED them back to +formula text and let ``plan_query`` re-derive the very identities it had just +thrown away. + +``PreboundQuery`` is that bind product made explicit. ``bind_query_inputs`` +produces it; ``plan_query(prebound=…)`` consumes it and skips binding +entirely. A caller holding typed keys re-roots them structurally and hands +them straight back — no text in the loop. + +``StrictQueryCarrier`` closes the second half. ``plan_query`` reads a handful +of query-level scalars that are not bind products (``source_model``, +``name``); a pre-bound caller that forgot one would silently inherit a Pydantic +default and plan the wrong thing. The carrier approves exactly those two and +raises on everything else, so a new post-bind ``query.*`` read fails loudly +instead of quietly. + +Also home to the key → slot-metadata lifts (``type`` / ``format`` / +``description``). They live here rather than in ``stage_planner`` because both +the planner and the re-rooting strategy need them, and the strategy cannot +import the planner (the recursion is injected as ``subplan_builder`` precisely +to avoid that cycle). +""" + +from __future__ import annotations + +from typing import FrozenSet, List, Optional, Tuple + +from pydantic import BaseModel, ConfigDict, Field + +from slayer.core.enums import DataType +from slayer.core.format import NumberFormat +from slayer.core.keys import ( + AggregateKey, + ColumnKey, + ColumnSqlKey, + StarKey, + TimeTruncKey, + ValueKey, +) +from slayer.core.models import SlayerModel +from slayer.engine.binding import BoundFilter +from slayer.engine.planning import DeclaredMeasure, OrderSpec +from slayer.engine.response_meta import _infer_aggregated_format + + +__all__ = [ + "PreboundQuery", + "StrictQueryCarrier", + "aggregated_type", + "dimension_key_metadata", + "measure_key_format_description", + "measure_key_type", + "walk_key_path", +] + + +# --------------------------------------------------------------------------- +# The seam types +# --------------------------------------------------------------------------- + + +class PreboundQuery(BaseModel): + """The typed product of ``plan_query``'s bind block. + + Everything downstream of binding reads from here, so a caller that already + holds bound keys can plan without a parser. The ``n_*`` counts are the + dimension / time-dimension prefix lengths of ``declared_measures``, which + the projection and partition-key passes slice by. + """ + + model_config = ConfigDict(arbitrary_types_allowed=True) + + declared_measures: List[DeclaredMeasure] = Field(default_factory=list) + bound_filters: List[BoundFilter] = Field(default_factory=list) + # Parallel to ``bound_filters``: the originating user-filter text, or + # ``None`` for a synthesized date-range bound. Carried so the cross-model + # routing can report a filter the way the caller wrote it. + bound_filter_texts: List[Optional[str]] = Field(default_factory=list) + n_date_range: int = 0 + order_specs: List[OrderSpec] = Field(default_factory=list) + main_time_key: Optional[TimeTruncKey] = None + n_dims: int = 0 + n_time_dimensions: int = 0 + limit: Optional[int] = None + offset: Optional[int] = None + distinct_dimension_values: bool = True + + +class StrictQueryCarrier(BaseModel): + """The post-bind ``query.*`` surface the §5.4 seam approves. + + Anything not declared here raises rather than returning a default, so a + new post-bind read in ``plan_query`` cannot silently plan a re-rooted + sub-query against the wrong value. + """ + + model_config = ConfigDict(extra="forbid") + + source_model: Optional[str] = None + name: Optional[str] = None + prebound: Optional[PreboundQuery] = None + + def __getattr__(self, item: str): + if item.startswith("_"): + return super().__getattr__(item) + raise AttributeError( + f"{type(self).__name__} does not carry {item!r}. The pre-bound " + f"seam approves only " + f"{sorted(type(self).model_fields)}; add the field here (and " + f"populate it at every construction site) rather than letting the " + f"planner read a default." + ) + + +# --------------------------------------------------------------------------- +# Key -> slot metadata +# --------------------------------------------------------------------------- + +_COUNT_AGGREGATIONS: FrozenSet[str] = frozenset( + {"count", "count_distinct", "count_distinct_approx"} +) +_FLOAT_AGGREGATIONS: FrozenSet[str] = frozenset({ + "avg", "weighted_avg", "median", + "stddev_samp", "stddev_pop", "var_samp", "var_pop", + "corr", "covar_samp", "covar_pop", "percentile", +}) + + +def aggregated_type( + *, + model: SlayerModel, + measure_name: Optional[str], + aggregation: str, +) -> Optional[DataType]: + """Type for an aggregated measure slot. Mirrors + ``_infer_aggregated_format`` (decision #2 of the Stage B plan): + + * ``*:count`` (measure_name=``"*"``) → ``INT`` + * ``count`` / ``count_distinct`` / ``count_distinct_approx`` → ``INT`` + * ``avg`` / ``weighted_avg`` / ``median`` / parametric / stat aggs → + ``DOUBLE`` + * ``sum`` / ``min`` / ``max`` / ``first`` / ``last`` → inherit from + source column type (DOUBLE if absent). + """ + if measure_name == "*": + return DataType.INT + if aggregation in _COUNT_AGGREGATIONS: + return DataType.INT + if aggregation in _FLOAT_AGGREGATIONS: + return DataType.DOUBLE + # sum / min / max / first / last — preserve source column type. + if measure_name is None: + return None + col = model.get_column(measure_name) + if col is not None and col.type is not None: + return col.type + return None + + +def _local_aggregate_source_name(key: ValueKey) -> Optional[str]: + """The source column name of a LOCAL aggregate, or ``None``. + + ``None`` for anything that isn't a bare local aggregate — a non-aggregate + key, an unsupported source shape, or a cross-model source (whose metadata + is lifted by ``response_meta`` against the target model instead). + """ + if not isinstance(key, AggregateKey): + return None + src = key.source + if isinstance(src, StarKey): + return "*" + if not isinstance(src, (ColumnKey, ColumnSqlKey)): + return None + if getattr(src, "path", ()): + return None + return getattr(src, "leaf", None) or getattr(src, "column_name", None) + + +def measure_key_type( + *, model: SlayerModel, key: ValueKey, +) -> Optional[DataType]: + """``type`` for a measure slot, from its bound key alone.""" + name = _local_aggregate_source_name(key) + if name is None: + return None + return aggregated_type( + model=model, measure_name=name, aggregation=key.agg, + ) + + +def measure_key_format_description( + *, model: SlayerModel, key: ValueKey, +) -> Tuple[Optional[NumberFormat], Optional[str]]: + """``format`` / ``description`` for a measure slot, from its bound key. + + ``*:count`` has an inferred INTEGER format but no description — there is + no source column to document it. + """ + name = _local_aggregate_source_name(key) + if name is None: + return None, None + fmt = _infer_aggregated_format( + model=model, measure_name=name, aggregation=key.agg, + ) + if name == "*": + return fmt, None + col = model.get_column(name) + return fmt, (col.description if col is not None else None) + + +def walk_key_path( + *, model: SlayerModel, path: Tuple[str, ...], bundle, +) -> Optional[SlayerModel]: + """Walk ``path`` as join hops from ``model``; ``None`` on any miss. + + The structural counterpart to binding a dotted reference: it answers + "is this join path traversable?" without a parser and without raising, + which is what re-rooting needs to decide reachability. + """ + current = model + visited = {current.name} + for hop in path: + if not any(j.target_model == hop for j in current.joins): + return None + nxt = bundle.get_referenced_model(hop) + if nxt is None or nxt.name in visited: + return None + visited.add(nxt.name) + current = nxt + return current + + +def dimension_key_metadata( + *, model: SlayerModel, key: ValueKey, bundle, +) -> Tuple[Optional[DataType], Optional[NumberFormat], Optional[str]]: + """``(type, format, description)`` for a dimension slot, from its key. + + A LOCAL dimension carries the source column's full display contract; a + JOINED one carries only its type. That asymmetry is deliberate and + pre-existing: joined refs surface format / description through + ``response_meta``, which resolves them against the owning model. + """ + inner = key.column if isinstance(key, TimeTruncKey) else key + path = tuple(getattr(inner, "path", ()) or ()) + leaf = getattr(inner, "leaf", None) or getattr(inner, "column_name", None) + if leaf is None: + return None, None, None + if not path: + col = model.get_column(leaf) + if col is None: + return None, None, None + return col.type, col.format, col.description + terminal = walk_key_path(model=model, path=path, bundle=bundle) + if terminal is None: + return None, None, None + col = terminal.get_column(leaf) + return (col.type if col is not None else None), None, None diff --git a/tests/_dev1747_fixtures.py b/tests/_dev1747_fixtures.py new file mode 100644 index 00000000..82797c78 --- /dev/null +++ b/tests/_dev1747_fixtures.py @@ -0,0 +1,504 @@ +"""Shared fixtures + helpers for the DEV-1747 rerooting / ORDER BY modules. + +Underscore-prefixed (like ``tests/_dev1746_fixtures.py``) so pytest skips it +during collection while ``from tests._dev1747_fixtures import ...`` works. + +What lives here: + +* **A sort-key corpus** (:func:`seed_dev1747_sqlite`) built so the DIRECTION of + the order-only aggregate wrap is observable. Group ``A`` spans regions + ``Alpha`` and ``Zulu``; group ``B`` sits alone on ``Bravo``. Ordering ASC by + ``MIN`` therefore yields ``[A, B]`` while ordering ASC by ``MAX`` would yield + ``[B, A]`` — so a test cannot pass under the old unconditional-``MAX`` rule by + accident. The names are deliberately not the group names, so a test reading + the wrong column fails rather than coincidentally matching. +* **A 1:N fan-out table** (``order_tags``) so the DEV-1735 containment claim is + testable: ordering by a tag name must not multiply a sibling ``amount:sum``. + Order 1 carries three tags; every other order carries one. +* **A NULL joined name** (region 4) so null-ordering has real data to sort. +* **Model builders** — :func:`dev1747_models` (SQLite-shaped, bare + ``sql_table``) and :func:`dev1747_pg_models` (same graph, for dry-run SQL + shape assertions), including the derived crossing column ``cust_region`` + whose ``Column.sql`` reaches through a join. +* :func:`make_sqlite_engine` — storage + engine wiring bound to the seeded file. +* ORDER BY shape helpers the four render sites all have to satisfy. +""" + +from __future__ import annotations + +import sqlite3 +from typing import Dict, List, Optional, Set, Tuple + +import sqlglot +from sqlglot import exp + +from slayer.core.enums import DataType +from slayer.core.models import Column, DatasourceConfig, ModelJoin, SlayerModel +from slayer.engine.query_engine import SlayerQueryEngine +from slayer.storage.yaml_storage import YAMLStorage + +# --------------------------------------------------------------------------- # +# The corpus +# --------------------------------------------------------------------------- # +#: Group A spans two regions; group B sits on one BETWEEN them alphabetically. +#: MIN(A)="Alpha" < MIN(B)="Bravo" -> ASC by MIN == [A, B] +#: MAX(A)="Zulu" > MAX(B)="Bravo" -> ASC by MAX == [B, A] +#: The two orderings disagree, which is exactly what makes D10 observable. +REGION_A_LOW = "Alpha" +REGION_A_HIGH = "Zulu" +REGION_B_ONLY = "Bravo" + +#: Sibling-measure totals. Distinct per group, and neither is a multiple of the +#: other, so a fan-out that doubled or tripled one would be unmistakable. +GROUP_A_AMOUNT = 11.0 + 13.0 # 24.0 — two orders +GROUP_B_AMOUNT = 17.0 # 17.0 — one order +GROUP_NULL_AMOUNT = 19.0 # the NULL-region group + +#: Order 1 carries THREE tags; if its join were pulled into the host base, +#: GROUP_A_AMOUNT would read 11.0 * 3 + 13.0 = 46.0 instead of 24.0. +ORDER_1_TAG_COUNT = 3 + + +def seed_dev1747_sqlite(db_path: str) -> None: + """Create + seed the DEV-1747 SQLite corpus at ``db_path``.""" + con = sqlite3.connect(db_path) + con.executescript( + """ + CREATE TABLE regions ( + id INTEGER PRIMARY KEY, + name TEXT, + population REAL + ); + CREATE TABLE customers ( + id INTEGER PRIMARY KEY, + region_id INTEGER, + tier TEXT, + spend REAL + ); + CREATE TABLE orders ( + id INTEGER PRIMARY KEY, + customer_id INTEGER, + status TEXT, + created_at TEXT, + amount REAL + ); + CREATE TABLE order_tags ( + id INTEGER PRIMARY KEY, + order_id INTEGER, + name TEXT + ); + """ + ) + con.executemany( + "INSERT INTO regions VALUES (?,?,?)", + [ + (1, REGION_A_LOW, 100.0), + (2, REGION_A_HIGH, 200.0), + (3, REGION_B_ONLY, 300.0), + # Region 4's name is NULL — the null-ordering group. + (4, None, 400.0), + ], + ) + con.executemany( + "INSERT INTO customers VALUES (?,?,?,?)", + [ + (100, 1, "gold", 1000.0), + (101, 2, "gold", 250.0), + (102, 3, "silver", 75.0), + (103, 4, "silver", 50.0), + ], + ) + con.executemany( + "INSERT INTO orders VALUES (?,?,?,?,?)", + [ + # Group A: two orders, two DIFFERENT regions (Alpha and Zulu). + (1, 100, "A", "2024-01-15", 11.0), + (2, 101, "A", "2024-02-15", 13.0), + # Group B: one order, region Bravo. + (3, 102, "B", "2024-01-20", 17.0), + # Group NULL-region: one order whose region name is NULL. + (4, 103, "N", "2024-02-20", 19.0), + ], + ) + con.executemany( + "INSERT INTO order_tags VALUES (?,?,?)", + [ + # Order 1 fans out 3:1 — the containment probe. + (1, 1, "rush"), + (2, 1, "gift"), + (3, 1, "fragile"), + (4, 2, "rush"), + # Distinct per group so the tag sort key never ties across groups + # (a tie makes the row order unstable and the assertion flaky). + (5, 3, "sale"), + (6, 4, "trial"), + ], + ) + con.commit() + con.close() + + +def _regions_model(*, data_source: str = "test") -> SlayerModel: + return SlayerModel( + name="regions", sql_table="regions", data_source=data_source, + columns=[ + Column(name="id", type=DataType.INT, primary_key=True), + Column(name="name", type=DataType.TEXT), + Column(name="population", type=DataType.DOUBLE), + ], + ) + + +def _customers_model(*, data_source: str = "test") -> SlayerModel: + return SlayerModel( + name="customers", sql_table="customers", data_source=data_source, + columns=[ + Column(name="id", type=DataType.INT, primary_key=True), + Column(name="region_id", type=DataType.INT), + Column(name="tier", type=DataType.TEXT), + Column(name="spend", type=DataType.DOUBLE), + ], + joins=[ModelJoin(target_model="regions", join_pairs=[["region_id", "id"]])], + ) + + +def _order_tags_model(*, data_source: str = "test") -> SlayerModel: + return SlayerModel( + name="order_tags", sql_table="order_tags", data_source=data_source, + columns=[ + Column(name="id", type=DataType.INT, primary_key=True), + Column(name="order_id", type=DataType.INT), + Column(name="name", type=DataType.TEXT), + ], + ) + + +def _orders_model(*, data_source: str = "test") -> SlayerModel: + return SlayerModel( + name="orders", sql_table="orders", data_source=data_source, + default_time_dimension="created_at", + columns=[ + Column(name="id", type=DataType.INT, primary_key=True), + Column(name="customer_id", type=DataType.INT), + Column(name="status", type=DataType.TEXT), + Column(name="created_at", type=DataType.TIMESTAMP), + Column(name="amount", type=DataType.DOUBLE), + # The DEV-1735 "also in scope" shape: a LOCAL derived column whose + # ``Column.sql`` reaches THROUGH a join. Ordering by it is rejected + # today in both grouped and ungrouped queries even though the bare + # ``customers.regions.name`` resolves ungrouped (DEV-1703 Phase 1). + Column( + name="cust_region", type=DataType.TEXT, + sql="customers__regions.name", + ), + # A NON-crossing derived column — the control. Ordering by it must + # keep working exactly as it does today. + Column(name="amount_x2", type=DataType.DOUBLE, sql="amount * 2"), + ], + joins=[ + ModelJoin(target_model="customers", join_pairs=[["customer_id", "id"]]), + ModelJoin(target_model="order_tags", join_pairs=[["id", "order_id"]]), + ], + ) + + +def dev1747_models(*, data_source: str = "test") -> List[SlayerModel]: + """``orders -> customers -> regions`` plus the 1:N ``order_tags``. + + Returned host-first; ``[0]`` is the host and the rest are ``extra_models``. + """ + return [ + _orders_model(data_source=data_source), + _customers_model(data_source=data_source), + _regions_model(data_source=data_source), + _order_tags_model(data_source=data_source), + ] + + +def dev1747_bundle(): + """A ``ResolvedSourceBundle`` over the corpus models, host-first. + + Lets the plan-level modules call ``plan_query`` directly instead of going + through the engine — §5.10's contract is that the PLAN carries the order + scope/phase/nulls, so it has to be assertable without rendering. + """ + from slayer.engine.source_bundle import ResolvedSourceBundle + + models = dev1747_models() + return ResolvedSourceBundle( + source_model=models[0], referenced_models=models[1:], + ) + + +async def make_sqlite_engine(base_dir: str, db_path: str) -> SlayerQueryEngine: + """Storage + engine bound to the seeded SQLite file at ``db_path``.""" + storage = YAMLStorage(base_dir=base_dir) + await storage.save_datasource( + DatasourceConfig(name="test", type="sqlite", database=db_path), + ) + for model in dev1747_models(): + await storage.save_model(model) + return SlayerQueryEngine(storage=storage) + + +# --------------------------------------------------------------------------- # +# ORDER BY shape helpers +# --------------------------------------------------------------------------- # +def outermost_select(sql: str, *, dialect: str = "postgres") -> exp.Select: + """The OUTERMOST SELECT — the one ORDER BY and pagination land on. + + Searching the whole statement is unsound here: a window function's + ``OVER (ORDER BY …)``, an inner CTE, or a ranked subquery all carry an + ``Order`` node, so a global ``find`` would happily assert against a clause + the user's ORDER BY never reaches. + """ + parsed = sqlglot.parse_one(sql, dialect=dialect) + assert parsed is not None, f"SQL failed to parse:\n{sql}" + if isinstance(parsed, exp.Select): + return parsed + select = parsed.find(exp.Select) + assert select is not None, f"no SELECT found in SQL:\n{sql}" + return select + + +def order_terms(sql: str, *, dialect: str = "postgres") -> List[str]: + """Rendered ORDER BY terms of the OUTERMOST select, in emitted order. + + Empty list when the statement has no outer ORDER BY — distinguishable from + a term list, so a silently-dropped sort key fails loudly instead of + matching a substring somewhere else in the statement. + """ + order = outermost_select(sql, dialect=dialect).args.get("order") + if order is None: + return [] + return [t.sql(dialect=dialect) for t in order.expressions] + + +def order_by_text(sql: str, *, dialect: str = "postgres") -> str: + """The outermost ORDER BY as one comma-joined string ('' when absent).""" + return ", ".join(order_terms(sql, dialect=dialect)) + + +def aggregate_funcs_over(sql: str, column: str, *, dialect: str = "postgres") -> List[str]: + """Names of aggregate functions applied to ``column`` anywhere in ``sql``. + + D10 asserts the order wrap is ``MIN`` on ASC and ``MAX`` on DESC; this + reads the emitted function rather than grepping, so a ``MIN`` appearing in + an unrelated measure cannot satisfy the assertion. + """ + tree = sqlglot.parse_one(sql, dialect=dialect) + found: List[str] = [] + for node in tree.find_all(exp.Min, exp.Max): + target = node.this + name = getattr(target, "name", None) or ( + target.sql(dialect=dialect) if target is not None else "" + ) + if name == column or (target is not None and column in target.sql(dialect=dialect)): + found.append(type(node).__name__.upper()) + return found + + +def with_node_of(sql: str, *, dialect: str = "postgres"): + """The statement's WITH node, wherever it sits. + + The local transform chain emits ``SELECT … FROM (WITH … SELECT …) AS _outer`` + on every dialect except T-SQL, so the WITH is NOT on the top-level + statement. Looking only at ``tree.args["with_"]`` would report "no CTEs" + for exactly the shape this PR rewrites. + """ + tree = sqlglot.parse_one(sql, dialect=dialect) + top = tree.args.get("with_") + if top is not None: + return top + return tree.find(exp.With) + + +def cte_body_names(sql: str, *, dialect: str = "postgres") -> List[str]: + """Names of the statement's CTEs, in emitted order.""" + with_node = with_node_of(sql, dialect=dialect) + if with_node is None: + return [] + return [cte.alias_or_name for cte in with_node.expressions] + + +def base_from_join_aliases(sql: str, *, dialect: str = "postgres") -> set: + """Aliases joined into the *host base* relation. + + The DEV-1735 containment claim is that a grouped joined sort key does NOT + pull its join into the host base — it lives inside the isolated CTE. That + is only checkable by looking at the base's own JOIN list, not the + statement's. + """ + tree = sqlglot.parse_one(sql, dialect=dialect) + with_node = tree.args.get("with_") + aliases: set = set() + candidates: List[exp.Expression] = [] + if with_node is not None: + for cte in with_node.expressions: + if cte.alias_or_name in ("_base", "base"): + candidates.append(cte.this) + if not candidates: + candidates.append(tree) + for candidate in candidates: + for join in candidate.find_all(exp.Join): + target = join.this + if isinstance(target, exp.Table): + aliases.add(target.alias_or_name) + return aliases + + +def cte_map(sql: str, *, dialect: str = "postgres") -> Dict[str, exp.Expression]: + """``{cte_name: cte_body}`` for every CTE in the statement. + + Finds WITH clauses wherever they sit, including the one the local transform + chain nests inside a derived table, so a caller can reach an isolated CTE + without knowing which render shape produced it. + """ + tree = sqlglot.parse_one(sql, dialect=dialect) + out: Dict[str, exp.Expression] = {} + for with_node in tree.find_all(exp.With): + for cte in with_node.expressions: + out[cte.alias_or_name] = cte.this + return out + + +def isolated_cte_bodies( + sql: str, *, dialect: str = "postgres", +) -> Dict[str, exp.Expression]: + """The CTEs the isolation machinery mints — everything but ``_base``. + + The DEV-1735 containment claim is two-sided: the crossed join must be ABSENT + from the host base AND PRESENT in the isolated CTE. Asserting only the first + half would also pass if the join vanished entirely and the sort key silently + resolved to nothing. + """ + return { + name: body for name, body in cte_map(sql, dialect=dialect).items() + if name not in ("_base", "base") + } + + +def relation_names(node: exp.Expression) -> Set[str]: + """Every table name AND alias ``node`` reads from. + + Both, because a joined relation appears under its alias + (``regions AS customers__regions``) while a CTE reference appears under its + own name — and a caller asserting containment should not have to know which + of the two it is looking at. + """ + names: Set[str] = set() + for table in node.find_all(exp.Table): + names.add(table.name) + names.add(table.alias_or_name) + names.discard("") + return names + + +def is_null_safe_eq(predicate: exp.Expression) -> bool: + """True when ``predicate`` compares two values NULL-safely. + + Accepts all three spellings ``SqlDialect.build_null_safe_eq`` emits — + sqlglot's ``NullSafeEQ`` (``IS NOT DISTINCT FROM`` / MySQL ``<=>``), SQLite's + bare ``IS`` between two non-NULL operands, and the expanded + ``(a = b OR (a IS NULL AND b IS NULL))`` — and nothing else. A substring test + for ``" IS "`` would also match ``IS NULL`` in an unrelated WHERE clause, + which is how a plain ``=`` join-back could pass a null-safety assertion. + """ + node = predicate.unnest() + if isinstance(node, exp.NullSafeEQ): + return True + if isinstance(node, exp.Is): + return not isinstance(node.expression, exp.Null) + if isinstance(node, exp.Or): + arms = [node.this.unnest(), node.expression.unnest()] + has_eq = any(isinstance(arm, exp.EQ) for arm in arms) + has_both_null = any( + isinstance(arm, exp.And) + and all( + isinstance(side.unnest(), exp.Is) + and isinstance(side.unnest().expression, exp.Null) + for side in (arm.this, arm.expression) + ) + for arm in arms + ) + return has_eq and has_both_null + return False + + +def all_conjuncts_null_safe(predicate: exp.Expression) -> bool: + """Every top-level ``AND`` conjunct of ``predicate`` is null-safe. + + A multi-member grain joins back on an ``AND`` chain; one plain ``=`` among + null-safe siblings still drops the NULL group, so the check has to be per + conjunct rather than "contains a null-safe comparison". + """ + parts: List[exp.Expression] = [] + + def _split(node: exp.Expression) -> None: + node = node.unnest() + if isinstance(node, exp.And): + _split(node.this) + _split(node.expression) + else: + parts.append(node) + + _split(predicate) + return bool(parts) and all(is_null_safe_eq(part) for part in parts) + + +def grain_join_back_predicates( + sql: str, *, dialect: str = "postgres", +) -> List[exp.Expression]: + """``ON`` predicates of the joins that attach an isolated CTE. + + Only those: the model joins inside a CTE are plain equalities by design, so + including them would make a null-safety assertion fail for the wrong reason. + """ + tree = sqlglot.parse_one(sql, dialect=dialect) + minted = set(isolated_cte_bodies(sql, dialect=dialect)) + out: List[exp.Expression] = [] + for join in tree.find_all(exp.Join): + target = join.this + if not isinstance(target, exp.Table): + continue + if target.name not in minted and target.alias_or_name not in minted: + continue + on = join.args.get("on") + if on is not None: + out.append(on) + return out + + +def aggregate_calls_in( + node: exp.Expression, *, dialect: str = "postgres", +) -> List[Tuple[str, str]]: + """``(FUNC, rendered-argument)`` for every aggregate call under ``node``. + + Lets a test say "the sort key is wrapped in MIN" about the SORT KEY rather + than about the statement — ``"MIN(" in sql`` is satisfied by a MIN over any + column at all, including a sibling measure. + + Identifier quoting is stripped from the argument so one assertion reads the + same across dialects (``"a"."b"``, ```a`.`b```, ``[a].[b]``). + """ + out: List[Tuple[str, str]] = [] + for call in node.find_all(exp.Min, exp.Max, exp.Sum, exp.Count, exp.Avg): + arg = call.this + rendered = arg.sql(dialect=dialect) if arg is not None else "" + for quote in ('"', "`", "[", "]"): + rendered = rendered.replace(quote, "") + out.append((type(call).__name__.upper(), rendered)) + return out + + +def response_column_values(rows: List[dict], key: str) -> List[Optional[object]]: + """``rows[i][key]`` for every row, preserving result order. + + Raises rather than defaulting on a missing key so a renamed result key + surfaces as a clear failure instead of a list of ``None``. + """ + out: List[Optional[object]] = [] + for i, row in enumerate(rows): + assert key in row, f"row {i} has no key {key!r}; keys are {sorted(row)}" + out.append(row[key]) + return out diff --git a/tests/test_dev1747_derived_crossing_order.py b/tests/test_dev1747_derived_crossing_order.py new file mode 100644 index 00000000..a282ad03 --- /dev/null +++ b/tests/test_dev1747_derived_crossing_order.py @@ -0,0 +1,233 @@ +"""DEV-1747 / DEV-1735 — ORDER BY on a LOCAL DERIVED column whose SQL crosses. + +``orders.cust_region`` is a local derived column (``path == ()``) whose +``Column.sql`` is ``customers__regions.name`` — it reaches THROUGH two joins. + +DEV-1735 recorded this as "rejected in BOTH grouped and ungrouped queries". +That is **stale for the grouped half**: DEV-1709 widened the Law-3 host-rooted +isolation trigger to any crossing INPUT, and a derived source whose SQL crosses +is such an input, so a grouped query already routes it to a host-rooted CTE +today. Group 1 pins that shape so the rerooting/ORDER BY consolidation cannot +regress it — it is the exact reference shape the bare joined column adopts in +``test_dev1747_grouped_joined_order.py``. + +The UNGROUPED half is still rejected, and that is the real inconsistency +DEV-1735 names: ``ORDER BY customers.regions.name`` resolves ungrouped (Law 1 +pulls the join, DEV-1703 Phase 1) but ``ORDER BY cust_region`` — the same +column reached through a derived definition — raises. Group 2 closes it. + +The fix is plan-side: the crossed paths are already structural (§5.3), so the +planner registers them and Law 1 pulls the join into the base FROM, exactly as +it does for the bare joined ref. That is also what lets the render-time +throwaway-``ScopeFrame`` probe in ``_apply_order_limit_from_planned`` go — it +exists solely to DETECT this crossing at render time. + +Refs: DEV-1747 (D9), DEV-1735 ("Also in scope"), DEV-1709, DEV-1703 Phase 1. +""" +from __future__ import annotations + +import os +import tempfile + +from slayer.core.errors import UnresolvableOrderColumnError +from slayer.core.query import ColumnRef, OrderItem, SlayerQuery +from tests._dev1747_fixtures import ( + GROUP_A_AMOUNT, + GROUP_B_AMOUNT, + GROUP_NULL_AMOUNT, + base_from_join_aliases, + dev1747_bundle, + dev1747_models, + make_sqlite_engine, + order_by_text, + response_column_values, + seed_dev1747_sqlite, +) +from tests._engine_helpers import _engine_generate + +_MEASURE = [{"formula": "amount:sum", "name": "rev"}] + + +async def _sql(query: SlayerQuery, *, dialect: str = "postgres") -> str: + models = dev1747_models() + return await _engine_generate( + query=query, model=models[0], extra_models=models[1:], dialect=dialect, + ) + + +async def _execute(query: SlayerQuery): + with tempfile.TemporaryDirectory() as d: + db = os.path.join(d, "dev1747.db") + seed_dev1747_sqlite(db) + engine = await make_sqlite_engine(d, db) + return await engine.execute(query) + + +def _grouped(direction: str, column: str = "cust_region") -> SlayerQuery: + return SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=_MEASURE, + order=[OrderItem(column=ColumnRef(name=column), direction=direction)], + ) + + +def _ungrouped(direction: str, column: str = "cust_region") -> SlayerQuery: + """Raw rows — ``distinct_dimension_values=False`` means no GROUP BY, so the + row IS the grain and a bare reference to the sort key is legal.""" + return SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + distinct_dimension_values=False, + order=[OrderItem(column=ColumnRef(name=column), direction=direction)], + ) + + +# --------------------------------------------------------------------------- +# Group 1 — grouped: already works; pin the shape and the D10 direction +# --------------------------------------------------------------------------- +class TestGroupedDerivedCrossing: + async def test_grouped_derived_crossing_resolves(self) -> None: + sql = await _sql(_grouped("asc")) + assert order_by_text(sql), f"no ORDER BY emitted:\n{sql}" + + async def test_crossed_join_lives_in_the_isolated_cte_not_the_base(self) -> None: + sql = await _sql(_grouped("asc")) + assert "customers__regions" not in base_from_join_aliases(sql), ( + f"the derived column's join leaked into the host base:\n{sql}" + ) + assert "customers__regions" in sql + + async def test_ascending_orders_by_each_group_minimum(self) -> None: + """D10 changes this from today's MAX. Under MAX the ASC order is + [B, A, N] (Bravo < Zulu); under MIN it is [A, B, N] (Alpha < Bravo). + The corpus makes the two orderings disagree on purpose.""" + response = await _execute(_grouped("asc")) + assert response_column_values(response.data, "orders.status") == ["A", "B", "N"] + + async def test_descending_orders_by_each_group_maximum(self) -> None: + response = await _execute(_grouped("desc")) + assert response_column_values(response.data, "orders.status") == ["A", "B", "N"] + + async def test_sibling_measure_untouched(self) -> None: + response = await _execute(_grouped("asc")) + by_status = {r["orders.status"]: r["orders.rev"] for r in response.data} + assert by_status == { + "A": GROUP_A_AMOUNT, "B": GROUP_B_AMOUNT, "N": GROUP_NULL_AMOUNT, + } + + async def test_derived_sort_key_is_not_projected(self) -> None: + response = await _execute(_grouped("asc")) + for row in response.data: + assert set(row) == {"orders.status", "orders.rev"} + + +# --------------------------------------------------------------------------- +# Group 2 — ungrouped: the DEV-1735 remainder +# --------------------------------------------------------------------------- +class TestUngroupedDerivedCrossing: + async def test_ungrouped_derived_crossing_no_longer_rejects(self) -> None: + try: + sql = await _sql(_ungrouped("asc")) + except UnresolvableOrderColumnError as exc: # pragma: no cover + raise AssertionError( + f"ungrouped derived crossing still rejects: {exc}" + ) from exc + assert order_by_text(sql), f"no ORDER BY emitted:\n{sql}" + + async def test_ungrouped_pulls_the_join_into_the_base_from(self) -> None: + """Ungrouped is the Law-1 case, NOT the isolation case: the row is the + grain, so the join belongs in the base FROM exactly as it does for the + bare joined ref that already works.""" + sql = await _sql(_ungrouped("asc")) + assert "customers__regions" in base_from_join_aliases(sql), ( + f"Law 1 did not pull the derived column's join:\n{sql}" + ) + + async def test_ungrouped_emits_no_aggregate_wrap(self) -> None: + """No GROUP BY means no wrap — an aggregate here would both be wrong + and force grouping the query never asked for.""" + sql = await _sql(_ungrouped("asc")) + upper = sql.upper() + assert "MIN(" not in upper and "MAX(" not in upper, ( + f"ungrouped sort key must not be aggregate-wrapped:\n{sql}" + ) + + async def test_ungrouped_derived_matches_the_bare_joined_shape(self) -> None: + """The consistency DEV-1735 asks for: ``ORDER BY cust_region`` and + ``ORDER BY customers.regions.name`` denote the same column, so their + emitted sort terms must agree.""" + derived_sql = await _sql(_ungrouped("asc")) + bare_sql = await _sql(SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + distinct_dimension_values=False, + order=[OrderItem( + column=ColumnRef(name="name", model="customers.regions"), + direction="asc", + )], + )) + assert order_by_text(derived_sql) == order_by_text(bare_sql) + + async def test_ungrouped_executes_in_the_right_order(self) -> None: + response = await _execute(_ungrouped("asc")) + statuses = response_column_values(response.data, "orders.status") + # Alpha(A) < Bravo(B) < Zulu(A), NULL last → A, B, A, N. + assert statuses == ["A", "B", "A", "N"] + + async def test_non_crossing_derived_column_still_works_ungrouped(self) -> None: + """The control: ``amount_x2`` is derived but LOCAL. It resolved before + this change and must keep resolving — the fix must not be "stop + checking whether the SQL crosses".""" + sql = await _sql(_ungrouped("asc", column="amount_x2")) + assert order_by_text(sql), f"non-crossing derived sort key broke:\n{sql}" + assert "customers__regions" not in base_from_join_aliases(sql) + + +# --------------------------------------------------------------------------- +# Group 3 — the render-time probe is gone +# --------------------------------------------------------------------------- +class TestRenderTimeProbeRemoved: + async def test_the_probe_host_method_is_never_called( + self, monkeypatch, + ) -> None: + """The probe builds a throwaway ``ScopeFrame`` inside + ``_apply_order_limit_from_planned`` purely to DETECT this crossing at + render time, then raises when it finds one. §5.10 makes the decision at + plan time, so the method must leave the production path entirely. + + A sentinel on the method rather than on ``ScopeFrame.__init__``: + legitimate scopes are constructed constantly, so counting them cannot + isolate the probe, whereas "this method is not called" is exactly the + claim. + """ + from slayer.sql.generator import SQLGenerator + + assert hasattr(SQLGenerator, "_apply_order_limit_from_planned"), ( + "the method was deleted; P-J defers deletion to PR 6" + ) + + def _boom(*_a, **_kw): + raise AssertionError( + "the render-time crossing probe is still on the production " + "path — §5.10 requires the decision to be planned" + ) + + monkeypatch.setattr( + SQLGenerator, "_apply_order_limit_from_planned", _boom, + ) + sql = await _sql(_ungrouped("desc")) + assert order_by_text(sql) + + async def test_plan_carries_the_crossing_decision(self) -> None: + """Plan-level: the order entry's scope is decided before rendering + (P-D). A HOST_BASE_HIDDEN scope on the ungrouped derived entry is what + tells the renderer to split-emit rather than probe.""" + from slayer.engine.planned import OrderScope + from slayer.engine.stage_planner import plan_query + + plan = plan_query(query=_ungrouped("asc"), bundle=dev1747_bundle()) + assert plan.order, "plan carries no order entries" + assert plan.order[0].scope in ( + OrderScope.HOST_BASE, OrderScope.HOST_BASE_HIDDEN, + ) diff --git a/tests/test_dev1747_grouped_joined_order.py b/tests/test_dev1747_grouped_joined_order.py new file mode 100644 index 00000000..c73b0e94 --- /dev/null +++ b/tests/test_dev1747_grouped_joined_order.py @@ -0,0 +1,420 @@ +"""DEV-1747 / DEV-1735 — grouped ORDER BY on a JOINED row column. + +Today a grouped query whose sort key is an unprojected JOINED column raises +``UnresolvableOrderColumnError`` (``stage_planner.py``), because an +``AggregateKey`` with a non-empty ``source.path`` always routes to a +TARGET-rooted CTE, which for a host-grain sort key degenerates to a scalar +CROSS JOIN — every group would get the same global value and the sort would +silently do nothing. + +DEV-1747 gives the wrap a structural marker (``AggregateKey.grain == "host"``) +that routes it to the DEV-1709 HOST-ROOTED isolated CTE instead: the crossed +join is pulled inside that CTE, the wrap is computed there, the CTE is grouped +on the query grain, and it joins back null-safe. The reference shape already +exists in production for a crossing DERIVED column (see +``test_dev1747_derived_crossing_order.py``); this module holds the bare joined +column to the same contract. + +Two things are asserted that SQL-shape alone cannot establish: + +* **Per-group, not global.** The corpus gives group ``A`` two different regions + and group ``B`` one, so a scalar CROSS JOIN would order every group by one + constant. Executed row order proves it did not. +* **Containment.** ``order_tags`` is 1:N against ``orders`` (order 1 carries + three tags). If the sort key's join were pulled into the host base, the + sibling ``amount:sum`` would multiply. Executed values prove it did not, and + the base's own JOIN list proves why. + +D10: the wrap is DIRECTION-AWARE — ``MIN`` on ASC, ``MAX`` on DESC — replacing +today's unconditional ``MAX``. The corpus is built so the two disagree: group +``A`` spans ``Alpha``..``Zulu`` and group ``B`` sits on ``Bravo`` between them, +so ASC-by-MIN yields ``[A, B]`` while ASC-by-MAX yields ``[B, A]``. + +Refs: DEV-1747 (D2, D9, D10), DEV-1735, DEV-1709 (the host-rooted vehicle), +DEV-1742 P-C. +""" +from __future__ import annotations + +import os +import tempfile + +import pytest +import sqlglot +from sqlglot import exp + +from slayer.core.query import ColumnRef, OrderItem, SlayerQuery +from tests._dev1747_fixtures import ( + GROUP_A_AMOUNT, + GROUP_B_AMOUNT, + GROUP_NULL_AMOUNT, + aggregate_calls_in, + all_conjuncts_null_safe, + base_from_join_aliases, + cte_map, + dev1747_models, + grain_join_back_predicates, + isolated_cte_bodies, + make_sqlite_engine, + order_by_text, + order_terms, + relation_names, + response_column_values, + seed_dev1747_sqlite, +) +from tests._engine_helpers import _engine_generate + +_MEASURE = [{"formula": "amount:sum", "name": "rev"}] + + +def _wrap_functions(sql: str, column: str, *, dialect: str = "postgres") -> list[str]: + """Aggregate functions applied to ``column`` — by AST, not by substring. + + ``"MIN(" in sql`` is satisfied by a MIN over ANY column, so it cannot + distinguish "the sort key is wrapped in MIN" from "some unrelated measure + is". D10 is precisely a claim about which function wraps which column. + """ + tree = sqlglot.parse_one(sql, dialect=dialect) + return [ + func for func, arg in aggregate_calls_in(tree, dialect=dialect) + if column in arg + ] + + +def _grouped_order_query(*, model: str, name: str, direction: str) -> SlayerQuery: + """Grouped by ``orders.status``, ordered by an UNPROJECTED joined column.""" + return SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=_MEASURE, + order=[OrderItem( + column=ColumnRef(name=name, model=model), direction=direction, + )], + ) + + +async def _execute(query: SlayerQuery): + with tempfile.TemporaryDirectory() as d: + db = os.path.join(d, "dev1747.db") + seed_dev1747_sqlite(db) + engine = await make_sqlite_engine(d, db) + return await engine.execute(query) + + +async def _sql(query: SlayerQuery, *, dialect: str = "postgres") -> str: + models = dev1747_models() + return await _engine_generate( + query=query, model=models[0], extra_models=models[1:], dialect=dialect, + ) + + +# --------------------------------------------------------------------------- +# Group 1 — it resolves at all (the DEV-1735 headline) +# --------------------------------------------------------------------------- +class TestGroupedJoinedOrderResolves: + async def test_one_hop_joined_sort_key_resolves(self) -> None: + sql = await _sql(_grouped_order_query( + model="customers", name="tier", direction="asc", + )) + assert order_by_text(sql), "grouped joined sort key produced no ORDER BY" + + async def test_multi_hop_joined_sort_key_resolves(self) -> None: + sql = await _sql(_grouped_order_query( + model="customers.regions", name="name", direction="asc", + )) + assert order_by_text(sql), "multi-hop joined sort key produced no ORDER BY" + + async def test_sort_key_is_not_projected(self) -> None: + """The wrap is HIDDEN — it must not leak into the public projection, + or adding a sort silently adds a result column.""" + response = await _execute(_grouped_order_query( + model="customers.regions", name="name", direction="asc", + )) + assert response.data + for row in response.data: + assert set(row) == {"orders.status", "orders.rev"}, ( + f"hidden order wrap leaked into the result row: {sorted(row)}" + ) + + +# --------------------------------------------------------------------------- +# Group 2 — per-group, not a global constant (DEV-1735 acceptance) +# --------------------------------------------------------------------------- +class TestPerGroupSortKey: + async def test_ascending_uses_each_group_own_minimum(self) -> None: + """Group A spans Alpha..Zulu, group B sits on Bravo, group N is NULL. + + ASC by each group's MIN → Alpha < Bravo < NULL(last) → A, B, N. + A global scalar would leave the groups in their unsorted order, and + the old unconditional MAX would give B, A, N — both distinguishable. + """ + response = await _execute(_grouped_order_query( + model="customers.regions", name="name", direction="asc", + )) + assert response_column_values(response.data, "orders.status") == ["A", "B", "N"] + + async def test_descending_uses_each_group_own_maximum(self) -> None: + """DESC by each group's MAX → Zulu > Bravo > NULL(last) → A, B, N.""" + response = await _execute(_grouped_order_query( + model="customers.regions", name="name", direction="desc", + )) + assert response_column_values(response.data, "orders.status") == ["A", "B", "N"] + + async def test_sibling_measure_values_are_untouched(self) -> None: + """Adding a sort key must not change any other field's value — the + core principle. Distinct per group so a wrong column cannot match.""" + response = await _execute(_grouped_order_query( + model="customers.regions", name="name", direction="asc", + )) + by_status = {r["orders.status"]: r["orders.rev"] for r in response.data} + assert by_status == { + "A": GROUP_A_AMOUNT, "B": GROUP_B_AMOUNT, "N": GROUP_NULL_AMOUNT, + } + + async def test_host_cardinality_is_unchanged(self) -> None: + response = await _execute(_grouped_order_query( + model="customers.regions", name="name", direction="asc", + )) + assert len(response.data) == 3 + + +# --------------------------------------------------------------------------- +# Group 3 — D10 direction-aware MIN/MAX +# --------------------------------------------------------------------------- +class TestDirectionAwareWrap: + async def test_ascending_emits_min_not_max(self) -> None: + sql = await _sql(_grouped_order_query( + model="customers.regions", name="name", direction="asc", + )) + assert _wrap_functions(sql, "customers__regions.name") == ["MIN"], ( + f"ASC must wrap THE SORT KEY in MIN and nothing else (D10); got:\n{sql}" + ) + + async def test_descending_emits_max(self) -> None: + sql = await _sql(_grouped_order_query( + model="customers.regions", name="name", direction="desc", + )) + assert _wrap_functions(sql, "customers__regions.name") == ["MAX"], ( + f"DESC must wrap THE SORT KEY in MAX (D10); got:\n{sql}" + ) + + async def test_local_row_column_wrap_is_direction_aware_too(self) -> None: + """D10 also changes the EXISTING local grouped wrap, which has emitted + an unconditional ``MAX`` since DEV-1703 Phase 1. Both shapes must agree + or the two order paths mean different things by ``ASC``.""" + sql = await _sql(SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=_MEASURE, + order=[OrderItem( + column=ColumnRef(name="created_at"), direction="asc", + )], + )) + assert _wrap_functions(sql, "created_at") == ["MIN"], ( + f"local grouped ASC wrap must be MIN over created_at (D10); got:\n{sql}" + ) + + async def test_two_directions_on_one_column_are_distinct_slots(self) -> None: + """``ORDER BY a ASC, a DESC`` needs MIN(a) and MAX(a) — two different + aggregates over one column. The order-key remap must therefore be + keyed by (key, direction), not by key alone, or the second entry + silently reuses the first's slot. + + Asserted per COLUMN: a statement-wide "contains MIN and MAX" would also + pass if both wraps landed on the same column, which is the bug.""" + sql = await _sql(SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=_MEASURE, + order=[ + OrderItem(column=ColumnRef(name="created_at"), direction="asc"), + OrderItem(column=ColumnRef(name="amount"), direction="desc"), + ], + )) + assert _wrap_functions(sql, "created_at") == ["MIN"], f"\n{sql}" + assert "MAX" in _wrap_functions(sql, "amount"), f"\n{sql}" + + +# --------------------------------------------------------------------------- +# Group 4 — fan-out containment (P-C) +# --------------------------------------------------------------------------- +class TestFanoutContainment: + """``order_tags`` is 1:N — order 1 carries three tags. Ordering by a tag + name must not multiply the sibling ``amount:sum``.""" + + def _tag_ordered(self, direction: str = "asc") -> SlayerQuery: + return _grouped_order_query( + model="order_tags", name="name", direction=direction, + ) + + async def test_sibling_sum_is_not_multiplied_by_the_fanout(self) -> None: + response = await _execute(self._tag_ordered()) + by_status = {r["orders.status"]: r["orders.rev"] for r in response.data} + assert by_status == { + "A": GROUP_A_AMOUNT, "B": GROUP_B_AMOUNT, "N": GROUP_NULL_AMOUNT, + }, ( + "sibling measure changed when a 1:N sort key was added — the " + "crossed join leaked into the host base (P-C violation)." + ) + + async def test_row_count_is_unchanged_by_the_fanout(self) -> None: + response = await _execute(self._tag_ordered()) + assert len(response.data) == 3 + + async def test_host_base_does_not_join_the_sort_key_table(self) -> None: + """The structural reason the values above hold: the crossed join lives + inside the isolated CTE, never in the base.""" + sql = await _sql(self._tag_ordered()) + assert "order_tags" not in base_from_join_aliases(sql), ( + f"order_tags joined into the host base — fan-out is not contained:\n{sql}" + ) + + async def test_the_isolated_cte_is_where_the_fanout_join_lives(self) -> None: + """The other half of containment. "Absent from the base" alone is also + satisfied by the join disappearing altogether — which would leave the + sort key resolving to nothing rather than being contained.""" + sql = await _sql(self._tag_ordered()) + bodies = isolated_cte_bodies(sql) + assert bodies, f"no isolated CTE was emitted at all:\n{sql}" + holders = [ + name for name, body in bodies.items() + if "order_tags" in relation_names(body) + ] + assert holders, ( + f"no isolated CTE joins order_tags — the sort key's rows are " + f"nowhere:\n{sql}" + ) + + async def test_sort_key_still_orders_correctly_under_fanout(self) -> None: + """Containment must not cost correctness: MIN tag per group is + fragile(A) < sale(B) < trial(N).""" + response = await _execute(self._tag_ordered()) + assert response_column_values(response.data, "orders.status") == ["A", "B", "N"] + + +# --------------------------------------------------------------------------- +# Group 5 — the isolated-CTE shape +# --------------------------------------------------------------------------- +class TestIsolatedCteShape: + async def test_crossed_join_lives_in_a_cte_not_the_base(self) -> None: + sql = await _sql(_grouped_order_query( + model="customers.regions", name="name", direction="asc", + )) + base_joins = base_from_join_aliases(sql) + assert "customers__regions" not in base_joins, ( + f"the sort key's join was pulled into the host base:\n{sql}" + ) + assert "customers__regions" in sql, ( + f"the crossed join is missing entirely — the sort key cannot " + f"resolve:\n{sql}" + ) + + @pytest.mark.parametrize("dialect", ["postgres", "sqlite", "tsql"]) + async def test_join_back_is_null_safe(self, dialect: str) -> None: + """A NULL grain member must still join back (P-I). The corpus has no + NULL status, so this asserts the emitted PREDICATE rather than a row — + the executed NULL-grain case is owned by the DEV-1746 suite. + + Asserted on the AST of the isolated CTE's own ``ON`` clause: a text + search for ``" IS "`` also matches an ``IS NULL`` anywhere else in the + statement, so it would pass on a plain ``=`` join-back. The three + dialects cover the three spellings ``build_null_safe_eq`` emits. + """ + sql = await _sql( + _grouped_order_query( + model="customers.regions", name="name", direction="asc", + ), + dialect=dialect, + ) + predicates = grain_join_back_predicates(sql, dialect=dialect) + assert predicates, ( + f"no isolated CTE is joined back at all on {dialect}:\n{sql}" + ) + for predicate in predicates: + assert all_conjuncts_null_safe(predicate), ( + f"grain join-back is not null-safe on {dialect}: " + f"{predicate.sql(dialect=dialect)}\n{sql}" + ) + + @pytest.mark.parametrize("dialect", ["tsql", "bigquery"]) + async def test_grouped_joined_sort_key_survives_a_dialect_round_trip( + self, dialect: str, + ) -> None: + """T-SQL and BigQuery mangle dotted aliases; the wrap's internal alias + must survive both (§5.13). + + ``assert sql`` would pass on any non-empty string, including one whose + ORDER BY was dropped — so this parses the emitted SQL back and requires + a sort term that names a relation the statement actually defines.""" + sql = await _sql( + _grouped_order_query( + model="customers.regions", name="name", direction="asc", + ), + dialect=dialect, + ) + terms = order_terms(sql, dialect=dialect) + assert terms, f"{dialect} lost the ORDER BY:\n{sql}" + tree = sqlglot.parse_one(sql, dialect=dialect) + order = tree.find(exp.Order) + assert order is not None + qualifiers = { + column.table for column in order.find_all(exp.Column) if column.table + } + known = set(cte_map(sql, dialect=dialect)) | relation_names(tree) + assert qualifiers <= known, ( + f"the sort term is qualified by {qualifiers - known}, which is not " + f"a relation in scope — a dotted alias was re-read as a multi-part " + f"reference:\n{sql}" + ) + + async def test_the_wrap_is_computed_inside_the_cte_over_the_joined_relation( + self, + ) -> None: + """The generator's local-aggregate walkers skip path-bearing sources + today, which is what would leave a ``grain="host"`` wrap unrendered. + + The observable contract: the aggregate is evaluated INSIDE the isolated + CTE against the pulled join, and the host base computes no such wrap. + """ + sql = await _sql(_grouped_order_query( + model="customers.regions", name="name", direction="asc", + )) + bodies = isolated_cte_bodies(sql) + assert bodies, f"no isolated CTE was emitted:\n{sql}" + in_cte = [ + (func, arg) + for body in bodies.values() + for func, arg in aggregate_calls_in(body) + if "customers__regions.name" in arg + ] + assert [func for func, _ in in_cte] == ["MIN"], ( + f"the host-grain wrap is not computed inside the isolated CTE:\n{sql}" + ) + base = cte_map(sql).get("_base") + if base is not None: + assert not [ + func for func, arg in aggregate_calls_in(base) + if "customers__regions.name" in arg + ], f"the wrap was ALSO computed in the host base:\n{sql}" + + +# --------------------------------------------------------------------------- +# Group 6 — the grouped reject is gone +# --------------------------------------------------------------------------- +class TestRejectRemoved: + @pytest.mark.parametrize( + ("model", "name"), + [("customers", "tier"), ("customers.regions", "name"), ("order_tags", "name")], + ) + async def test_no_unresolvable_order_column_error( + self, model: str, name: str, + ) -> None: + from slayer.core.errors import UnresolvableOrderColumnError + + try: + await _sql(_grouped_order_query( + model=model, name=name, direction="asc", + )) + except UnresolvableOrderColumnError as exc: # pragma: no cover - failure path + pytest.fail( + f"grouped joined ORDER BY on {model}.{name} still rejects: {exc}" + ) diff --git a/tests/test_dev1747_local_with_chain.py b/tests/test_dev1747_local_with_chain.py new file mode 100644 index 00000000..1fb3286c --- /dev/null +++ b/tests/test_dev1747_local_with_chain.py @@ -0,0 +1,280 @@ +"""DEV-1747 D8 — the LOCAL transform chain's WITH clause, assembled as AST. + +PR 3 (DEV-1746) built one WITH assembler (``slayer/sql/render/cte_assembly.py``) +and adopted it on the cross-model sites, deferring the LOCAL single-model +transform chain to this PR — its two near-identical sites still splice out of +f-strings:: + + cte_clause = "WITH " + ",\\n".join(f"{name} AS (\\n{sql}\\n)" for ...) + chain_sql = f"{cte_clause}\\n{inner_sql}" + +so the emitted order is whatever order the python list happened to be built in, +and each step reads its predecessor POSITIONALLY (``prev_cte = ctes[-1][0]``). +D8 moves both onto ``assemble_with_chain`` with DECLARED dependencies (step N +depends on step N-1), and — per PR 3's first hard-won lesson — keeps the bodies +as ``exp.Select`` from renderer to assembler rather than rendering to text and +re-parsing. + +That lesson is the reason for :class:`TestNoTextRoundTrip` below. A dotted +public alias round-trips through text as a MULTI-PART reference on BigQuery +(``_base."orders_x.status"`` came back as ``` `_base___orders_x`.`status` ```), +and this path is full of dotted ``.`` names — so a parse seam +here would silently corrupt exactly the identifiers it carries. + +Refs: DEV-1747 (D8), DEV-1746 handoff, DEV-1742 §5.6. +""" +from __future__ import annotations + +import pytest +from sqlglot import exp + +from slayer.core.enums import TimeGranularity +from slayer.core.query import ColumnRef, OrderItem, SlayerQuery, TimeDimension +from tests._dev1747_fixtures import ( + cte_body_names, + dev1747_models, + order_by_text, + with_node_of, +) +from tests._engine_helpers import _engine_generate + +#: A LOCAL (single-model) transform chain — no cross-model measure, so it takes +#: the f-string splice path rather than the cross-model one PR 3 already fixed. +_CHAIN_QUERY = SlayerQuery( + source_model="orders", + time_dimensions=[TimeDimension( + dimension=ColumnRef(name="created_at"), + granularity=TimeGranularity.MONTH, + )], + measures=[ + {"formula": "amount:sum", "name": "rev"}, + {"formula": "cumsum(amount:sum)", "name": "cs"}, + ], +) + +#: Two chained transforms, so the chain has more than one step and dependency +#: ORDER is actually observable. +_MULTI_STEP_QUERY = SlayerQuery( + source_model="orders", + time_dimensions=[TimeDimension( + dimension=ColumnRef(name="created_at"), + granularity=TimeGranularity.MONTH, + )], + measures=[ + {"formula": "amount:sum", "name": "rev"}, + {"formula": "cumsum(amount:sum)", "name": "cs"}, + {"formula": "change(amount:sum)", "name": "ch"}, + ], +) + + +async def _sql(query: SlayerQuery, *, dialect: str = "postgres") -> str: + models = dev1747_models() + return await _engine_generate( + query=query, model=models[0], extra_models=models[1:], dialect=dialect, + ) + + +def _assembler_spy(monkeypatch) -> list: + """Record the ``CteEntry`` list handed to ``assemble_with_chain``. + + Patched as BOUND in ``slayer.sql.generator`` (which does ``from … import + assemble_with_chain``), so patching the defining module would record + nothing and every assertion downstream would pass vacuously. The vacuity + guard is the ``assert entries`` in each caller. + """ + from slayer.sql import generator + from slayer.sql.render import cte_assembly + + seen: list = [] + original = cte_assembly.assemble_with_chain + + def _recording(*, entries, final): + seen.extend(entries) + return original(entries=entries, final=final) + + monkeypatch.setattr(generator, "assemble_with_chain", _recording) + return seen + + +# --------------------------------------------------------------------------- +# Group 1 — assembled, and in dependency order +# --------------------------------------------------------------------------- +class TestWithChainAssembly: + async def test_chain_emits_a_single_top_level_with(self) -> None: + sql = await _sql(_CHAIN_QUERY) + assert cte_body_names(sql), f"no WITH clause emitted:\n{sql}" + + async def test_each_step_is_emitted_after_the_step_it_reads(self) -> None: + """The invariant the positional ``ctes[-1][0]`` encodes implicitly and + the assembler makes explicit: a CTE must never be referenced before it + is defined.""" + sql = await _sql(_MULTI_STEP_QUERY) + with_node = with_node_of(sql, dialect="postgres") + assert with_node is not None, f"no WITH clause:\n{sql}" + defined: set[str] = set() + for cte in with_node.expressions: + for table in cte.this.find_all(exp.Table): + name = table.name + if name in {c.alias_or_name for c in with_node.expressions}: + assert name in defined, ( + f"CTE {cte.alias_or_name!r} reads {name!r} before it is " + f"defined:\n{sql}" + ) + defined.add(cte.alias_or_name) + + async def test_cte_names_are_unique(self) -> None: + sql = await _sql(_MULTI_STEP_QUERY) + names = cte_body_names(sql) + assert len(names) == len(set(names)), f"duplicate CTE names {names}:\n{sql}" + + async def test_chain_still_orders_and_paginates(self) -> None: + """The chain's outer wrap is where ORDER BY / LIMIT land; D8 must not + disturb that while replacing the splice.""" + sql = await _sql(SlayerQuery( + source_model="orders", + time_dimensions=[TimeDimension( + dimension=ColumnRef(name="created_at"), + granularity=TimeGranularity.MONTH, + )], + measures=[{"formula": "cumsum(amount:sum)", "name": "cs"}], + order=[OrderItem(column=ColumnRef(name="cs"), direction="desc")], + limit=2, + )) + assert order_by_text(sql), f"chain lost its ORDER BY:\n{sql}" + assert "LIMIT" in sql.upper() or "TOP" in sql.upper() + + +# --------------------------------------------------------------------------- +# Group 2 — no render-to-text-and-re-parse (PR 3's lesson 1) +# --------------------------------------------------------------------------- +class TestNoTextRoundTrip: + async def test_window_transform_renderer_returns_ast(self) -> None: + """``_render_window_transform_sql`` returns a string today, which is + what forces a parse at the assembler seam.""" + import inspect + + from slayer.sql.generator import SQLGenerator + + signature = inspect.signature(SQLGenerator._render_window_transform_sql) + assert signature.return_annotation is not str, ( + "the window transform renderer must hand back AST so the local " + "chain never round-trips through text (D8)" + ) + + async def test_local_chain_does_not_call_the_parse_seam( + self, monkeypatch, + ) -> None: + """``_parse_cte_body`` is the documented seam PR 3 kept for the ONE + input that arrives as a complete nested statement. The local chain + builds its own bodies, so for THIS query it must not be called at all. + + A source-level ``count(...) <= 2`` would pass while permitting two live + round-trips — the very thing D8 removes. A raising sentinel scoped to + this render is the exact claim: zero. + """ + from slayer.sql.generator import SQLGenerator + + def _boom(self, sql): # noqa: ANN001 - signature mirrors the seam + raise AssertionError( + "the local transform chain routed a CTE body through the parse " + "seam; keep bodies as exp.Select from renderer to assembler (D8)" + ) + + monkeypatch.setattr(SQLGenerator, "_parse_cte_body", _boom) + await _sql(_MULTI_STEP_QUERY) + + async def test_chain_bodies_reach_the_assembler_as_ast( + self, monkeypatch, + ) -> None: + """The positive half. Every entry the chain hands the assembler must + already be a ``exp.Select`` — a string body would mean the renderer + still emits text and something downstream re-parses it.""" + entries = _assembler_spy(monkeypatch) + await _sql(_MULTI_STEP_QUERY) + assert entries, ( + "the local chain never called assemble_with_chain — it is still " + "splicing its WITH clause out of f-strings (D8)" + ) + for entry in entries: + assert isinstance(entry.query, exp.Select), ( + f"CTE {entry.name!r} reached the assembler as " + f"{type(entry.query).__name__}, not exp.Select" + ) + + async def test_chain_declares_its_dependencies(self, monkeypatch) -> None: + """D8's substantive change. Emitting in the right order is not the + contract — DECLARING the dependency is, because the current code gets + the order right by reading ``ctes[-1][0]`` positionally, which is + correct only for as long as nothing ever inserts a step. + + A correct emission order therefore cannot distinguish "fixed" from + "still positional"; the declared ``depends_on`` can. + """ + entries = _assembler_spy(monkeypatch) + await _sql(_MULTI_STEP_QUERY) + assert entries, "assemble_with_chain was never called (D8)" + declared = {entry.name: set(entry.depends_on) for entry in entries} + chained = [name for name, deps in declared.items() if deps] + assert chained, ( + f"no CTE declares a dependency, so the chain is still ordered " + f"positionally: {declared}" + ) + names = set(declared) + for name, deps in declared.items(): + assert deps <= names, ( + f"CTE {name!r} declares dependencies outside the chain: " + f"{deps - names}" + ) + + @pytest.mark.parametrize("dialect", ["bigquery", "tsql"]) + async def test_dotted_aliases_survive_on_mangling_dialects( + self, dialect: str, + ) -> None: + """The concrete corruption PR 3 hit: a dotted public alias re-parsed as + a multi-part reference. This path carries dotted + ``.`` names throughout, so it is the exposed one.""" + sql = await _sql(_MULTI_STEP_QUERY, dialect=dialect) + assert "_base___" not in sql, ( + f"a dotted alias was re-read as a multi-part reference:\n{sql}" + ) + + +# --------------------------------------------------------------------------- +# Group 3 — semantics preserved across dialects +# --------------------------------------------------------------------------- +class TestChainSemanticsPreserved: + @pytest.mark.parametrize( + "dialect", ["postgres", "sqlite", "duckdb", "bigquery", "tsql"], + ) + async def test_chain_parses_on_every_tier_one_dialect( + self, dialect: str, + ) -> None: + sql = await _sql(_MULTI_STEP_QUERY, dialect=dialect) + assert sql + + async def test_hidden_order_slot_survives_the_chain(self) -> None: + """A hidden order-only slot has to be carried through every step; the + carry lists are plan-ordered (B8, PR 3) and must stay that way.""" + sql = await _sql(SlayerQuery( + source_model="orders", + time_dimensions=[TimeDimension( + dimension=ColumnRef(name="created_at"), + granularity=TimeGranularity.MONTH, + )], + measures=[{"formula": "cumsum(amount:sum)", "name": "cs"}], + order=[OrderItem(column=ColumnRef(name="status"), direction="desc")], + )) + assert order_by_text(sql) + + async def test_post_phase_filter_still_wraps_the_chain(self) -> None: + sql = await _sql(SlayerQuery( + source_model="orders", + time_dimensions=[TimeDimension( + dimension=ColumnRef(name="created_at"), + granularity=TimeGranularity.MONTH, + )], + measures=[{"formula": "cumsum(amount:sum)", "name": "cs"}], + filters=["cs > 5"], + )) + assert "WHERE" in sql.upper() diff --git a/tests/test_dev1747_order_entry.py b/tests/test_dev1747_order_entry.py new file mode 100644 index 00000000..10557df9 --- /dev/null +++ b/tests/test_dev1747_order_entry.py @@ -0,0 +1,447 @@ +"""DEV-1747 §5.10 — ``OrderEntry`` enrichment at PLAN time. + +Today ``OrderEntry`` carries only ``(slot_id, direction)``, so each of the +three renderers re-derives everything else at render time — and disagrees: + +* ``_apply_order_limit_from_planned`` dispatches on the SLOT KIND; +* ``_resolve_combined_order_term`` runs a 5-way precedence chain + (hidden-CTE ref → cross-model alias → outer-composite alias → outer-composite + expression → bare → ``_base.``-qualified); +* ``_planned_order_by_sql`` builds text and knows about none of it. + +§5.10 moves the classification into the plan: ``scope`` names WHERE the ordered +value lives, ``phase`` its phase, ``nulls`` the null-ordering policy. This +module asserts the PLAN — no SQL — because "plan decides, render emits" (P-D) +is only true if the decision is observable without rendering. + +``scope`` and ``phase`` are REQUIRED with no default. A shape the planner +forgets to classify must fail loudly rather than fall through to the +``_base.``-qualified branch, which is how an order term silently attaches to +the wrong scope today. + +Refs: DEV-1747 (D3, D5), DEV-1742 §5.10 / P-D. +""" +from __future__ import annotations + +import pytest + +from slayer.core.query import ColumnRef, OrderItem, SlayerQuery, TimeDimension +from slayer.core.enums import TimeGranularity +from slayer.engine.planned import OrderEntry, OrderScope +from slayer.engine.stage_planner import plan_query +from tests._dev1747_fixtures import dev1747_bundle + +_MEASURE = [{"formula": "amount:sum", "name": "rev"}] + + +def _plan(query: SlayerQuery): + return plan_query(query=query, bundle=dev1747_bundle()) + + +def _sole_entry(query: SlayerQuery) -> OrderEntry: + plan = _plan(query) + assert len(plan.order) == 1, ( + f"expected one order entry, got {len(plan.order)}" + ) + return plan.order[0] + + +# --------------------------------------------------------------------------- +# Group 1 — the field contract +# --------------------------------------------------------------------------- +class TestOrderEntryShape: + def test_scope_is_required(self) -> None: + """No default. A planner path that forgets to classify must fail at + construction, not silently order against ``_base``.""" + with pytest.raises(Exception): + OrderEntry(slot_id="s1", direction="asc") # type: ignore[call-arg] + + def test_nulls_defaults_to_dialect_default(self) -> None: + from slayer.core.keys import Phase + + entry = OrderEntry( + slot_id="s1", direction="asc", + scope=OrderScope.HOST_BASE, phase=Phase.ROW, + ) + assert entry.nulls == "default" + + def test_nulls_rejects_an_unknown_policy(self) -> None: + from slayer.core.keys import Phase + + with pytest.raises(Exception): + OrderEntry( + slot_id="s1", direction="asc", scope=OrderScope.HOST_BASE, + phase=Phase.ROW, nulls="sometimes", # type: ignore[arg-type] + ) + + def test_direction_validation_still_applies(self) -> None: + """The existing contract must survive the enrichment.""" + from slayer.core.keys import Phase + + with pytest.raises(Exception): + OrderEntry( + slot_id="s1", direction="ASC", # type: ignore[arg-type] + scope=OrderScope.HOST_BASE, phase=Phase.ROW, + ) + + +# --------------------------------------------------------------------------- +# Group 2 — scope classification per shape +# --------------------------------------------------------------------------- +class TestScopeClassification: + def test_projected_dimension_is_host_base(self) -> None: + entry = _sole_entry(SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=_MEASURE, + order=[OrderItem(column=ColumnRef(name="status"), direction="asc")], + )) + assert entry.scope is OrderScope.HOST_BASE + + def test_projected_measure_is_host_base(self) -> None: + entry = _sole_entry(SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=_MEASURE, + order=[OrderItem(column=ColumnRef(name="rev"), direction="desc")], + )) + assert entry.scope is OrderScope.HOST_BASE + + def test_hidden_local_aggregate_is_host_base_hidden(self) -> None: + """Materialised in the base but trimmed from the public projection — + the distinction the ``bare_ids`` set encodes at render time today.""" + entry = _sole_entry(SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=_MEASURE, + order=[OrderItem(column=ColumnRef(name="created_at"), direction="asc")], + )) + assert entry.scope is OrderScope.HOST_BASE_HIDDEN + + def test_cross_model_aggregate_is_cross_model_cte(self) -> None: + entry = _sole_entry(SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ + {"formula": "amount:sum", "name": "rev"}, + {"formula": "customers.spend:sum", "name": "cs"}, + ], + order=[OrderItem(column=ColumnRef(name="cs"), direction="desc")], + )) + assert entry.scope is OrderScope.CROSS_MODEL_CTE + + def test_windowed_measure_is_windowed_cte(self) -> None: + entry = _sole_entry(SlayerQuery( + source_model="orders", + time_dimensions=[TimeDimension( + dimension=ColumnRef(name="created_at"), + granularity=TimeGranularity.MONTH, + )], + measures=[{"formula": "amount:sum(window='90d')", "name": "w"}], + order=[OrderItem(column=ColumnRef(name="w"), direction="desc")], + )) + assert entry.scope is OrderScope.WINDOWED_CTE + + def test_transform_measure_is_transform_step(self) -> None: + entry = _sole_entry(SlayerQuery( + source_model="orders", + time_dimensions=[TimeDimension( + dimension=ColumnRef(name="created_at"), + granularity=TimeGranularity.MONTH, + )], + measures=[{"formula": "cumsum(amount:sum)", "name": "cs"}], + order=[OrderItem(column=ColumnRef(name="cs"), direction="asc")], + )) + assert entry.scope is OrderScope.TRANSFORM_STEP + + def test_composite_over_a_cross_model_operand_is_outer_composite(self) -> None: + """``customers.spend:sum + amount:sum`` cannot be evaluated in ``_base`` + — one operand lives in a ``_cm_`` CTE — so it is rendered in the outer + combined SELECT and ordered there. + + This is its own scope, not a variant of ``CROSS_MODEL_CTE``: the value + has no CTE column to name, only an outer alias or a re-rendered + expression, which is exactly the branch + ``_resolve_combined_order_term``'s precedence chain gets wrong.""" + entry = _sole_entry(SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[{"formula": "customers.spend:sum + amount:sum", "name": "mix"}], + order=[OrderItem(column=ColumnRef(name="mix"), direction="desc")], + )) + assert entry.scope is OrderScope.OUTER_COMPOSITE + + def test_hidden_composite_order_is_outer_composite_too(self) -> None: + """The hidden variant: nothing projects the composite, so a scope that + fell back to ``HOST_BASE`` would render it inline in ``_base`` and + silently substitute a plain aggregate for the cross-model one.""" + entry = _sole_entry(SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=_MEASURE, + order=[OrderItem( + column="customers.spend:sum + amount:sum", direction="desc", + )], + )) + assert entry.scope is OrderScope.OUTER_COMPOSITE + + def test_grouped_joined_wrap_is_cross_model_cte(self) -> None: + """The DEV-1735 wrap lives in its own host-rooted CTE, so it resolves + CTE-qualified — not as a bare ``_base`` alias.""" + entry = _sole_entry(SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=_MEASURE, + order=[OrderItem( + column=ColumnRef(name="name", model="customers.regions"), + direction="asc", + )], + )) + assert entry.scope is OrderScope.CROSS_MODEL_CTE + + +# --------------------------------------------------------------------------- +# Group 3 — phase and direction ride through +# --------------------------------------------------------------------------- +class TestPhaseAndDirection: + def test_row_target_carries_row_phase(self) -> None: + from slayer.core.keys import Phase + + entry = _sole_entry(SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=_MEASURE, + order=[OrderItem(column=ColumnRef(name="status"), direction="asc")], + )) + assert entry.phase is Phase.ROW + + def test_aggregate_target_carries_aggregate_phase(self) -> None: + from slayer.core.keys import Phase + + entry = _sole_entry(SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=_MEASURE, + order=[OrderItem(column=ColumnRef(name="rev"), direction="desc")], + )) + assert entry.phase is Phase.AGGREGATE + + def test_multiple_entries_keep_their_own_scope_and_direction(self) -> None: + plan = _plan(SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=_MEASURE, + order=[ + OrderItem(column=ColumnRef(name="status"), direction="asc"), + OrderItem(column=ColumnRef(name="rev"), direction="desc"), + OrderItem(column=ColumnRef(name="created_at"), direction="asc"), + ], + )) + assert [e.direction for e in plan.order] == ["asc", "desc", "asc"] + assert [e.scope for e in plan.order] == [ + OrderScope.HOST_BASE, OrderScope.HOST_BASE, OrderScope.HOST_BASE_HIDDEN, + ] + + +# --------------------------------------------------------------------------- +# Group 4 — the direction-aware wrap is decided at plan time (D10) +# --------------------------------------------------------------------------- +class TestDirectionAwareWrapIsPlanned: + def _wrap_aggs(self, direction: str) -> list[str]: + from slayer.core.keys import AggregateKey + + plan = _plan(SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=_MEASURE, + order=[OrderItem( + column=ColumnRef(name="created_at"), direction=direction, + )], + )) + return [ + s.key.agg for s in plan.aggregate_slots + if isinstance(s.key, AggregateKey) and s.hidden + ] + + def test_ascending_plans_a_min_wrap(self) -> None: + assert self._wrap_aggs("asc") == ["min"] + + def test_descending_plans_a_max_wrap(self) -> None: + assert self._wrap_aggs("desc") == ["max"] + + def test_same_column_both_directions_plans_two_slots(self) -> None: + """MIN(a) and MAX(a) are different values, so they must be different + slots. Keying the order remap by key alone collapses them.""" + from slayer.core.keys import AggregateKey + + plan = _plan(SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=_MEASURE, + order=[ + OrderItem(column=ColumnRef(name="created_at"), direction="asc"), + OrderItem(column=ColumnRef(name="created_at"), direction="desc"), + ], + )) + aggs = sorted( + s.key.agg for s in plan.aggregate_slots + if isinstance(s.key, AggregateKey) and s.hidden + ) + assert aggs == ["max", "min"] + assert len({e.slot_id for e in plan.order}) == 2 + + +# --------------------------------------------------------------------------- +# Group 5 — the host-grain marker (D2) +# --------------------------------------------------------------------------- +class TestHostGrainMarker: + def test_joined_order_wrap_is_marked_host_grain(self) -> None: + from slayer.core.keys import AggregateKey + + plan = _plan(SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=_MEASURE, + order=[OrderItem( + column=ColumnRef(name="name", model="customers.regions"), + direction="asc", + )], + )) + wraps = [ + s.key for s in plan.aggregate_slots + if isinstance(s.key, AggregateKey) and s.hidden + ] + assert wraps, "no hidden order wrap was planned" + assert wraps[0].grain == "host" + assert wraps[0].source.path == ("customers", "regions") + + def test_local_wrap_keeps_the_default_grain(self) -> None: + from slayer.core.keys import AggregateKey + + plan = _plan(SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=_MEASURE, + order=[OrderItem(column=ColumnRef(name="created_at"), direction="asc")], + )) + wraps = [ + s.key for s in plan.aggregate_slots + if isinstance(s.key, AggregateKey) and s.hidden + ] + assert wraps and wraps[0].grain == "target" + + def test_host_grain_and_target_grain_are_distinct_identities(self) -> None: + """A user-declared ``customers.regions.name:max`` measure and the + synthetic host-grain wrap mean different things (global vs per-group), + so they must not intern onto one slot.""" + from slayer.core.keys import AggregateKey, ColumnKey + + source = ColumnKey(path=("customers", "regions"), leaf="name") + target_rooted = AggregateKey(source=source, agg="max") + host_rooted = AggregateKey(source=source, agg="max", grain="host") + assert target_rooted != host_rooted + assert hash(target_rooted) != hash(host_rooted) + assert len({target_rooted, host_rooted}) == 2 + + +# --------------------------------------------------------------------------- +# Group 6 — the revised Law-3 trigger, branch by branch (D2) +# --------------------------------------------------------------------------- +_GROUPED_HEAD = { + "source_model": "orders", + "dimensions": [ColumnRef(name="status")], +} + +#: ``grain="host"`` + non-empty ``source.path`` — the NEW branch. Today the +#: trigger reads ``if not agg_path and not has_crossing_input: continue`` and +#: then routes any path-bearing key to a TARGET-rooted CTE. +_HOST_GRAIN_WITH_PATH = SlayerQuery( + **_GROUPED_HEAD, measures=_MEASURE, + order=[OrderItem( + column=ColumnRef(name="name", model="customers.regions"), direction="asc", + )], +) + +#: Target-grain + path — the pre-existing cross-model case, unchanged. +_TARGET_GRAIN_WITH_PATH = SlayerQuery( + **_GROUPED_HEAD, + measures=[ + {"formula": "amount:sum", "name": "rev"}, + {"formula": "customers.spend:sum", "name": "cs"}, + ], +) + +#: Host-grain, NO path, crossing input — the DEV-1709 branch that already +#: works. Included so the matrix proves D2 does not disturb it. +_HOST_GRAIN_CROSSING_INPUT = SlayerQuery( + **_GROUPED_HEAD, measures=_MEASURE, + order=[OrderItem(column=ColumnRef(name="cust_region"), direction="asc")], +) + + +class TestLaw3TriggerMatrix: + """Each branch of the widened trigger, and where it routes. + + ``cte_root_model`` is the discriminator the planner already carries: set to + the HOST model name for a host-rooted CTE, ``None`` for a target-rooted one. + A host-grain wrap landing target-rooted is the scalar-CROSS-JOIN + degeneration DEV-1735 describes, and it is invisible in a per-case test that + only asserts "a plan exists". + """ + + @pytest.mark.parametrize( + ("query", "expect_host_rooted"), + [ + pytest.param(_HOST_GRAIN_WITH_PATH, True, id="host-grain+path"), + pytest.param(_TARGET_GRAIN_WITH_PATH, False, id="target-grain+path"), + pytest.param( + _HOST_GRAIN_CROSSING_INPUT, True, id="host-grain+crossing-input", + ), + ], + ) + def test_trigger_routes_each_branch( + self, query: SlayerQuery, expect_host_rooted: bool, + ) -> None: + plan = _plan(query) + assert plan.cross_model_aggregate_plans, ( + "the Law-3 trigger did not fire for this branch" + ) + cma = plan.cross_model_aggregate_plans[0] + assert (cma.cte_root_model is not None) is expect_host_rooted, ( + f"cte_root_model={cma.cte_root_model!r} — expected " + f"{'host' if expect_host_rooted else 'target'}-rooted" + ) + + def test_isolation_disabled_renders_inline_instead_of_recursing(self) -> None: + """The recursion guard. The host-rooted sub-plan contains the SAME + crossing key, so if the trigger fired again inside it the planner would + recurse without bound — which is why ``subplan_builder`` always passes + ``disable_host_rooted_isolation=True``. + + Under that flag the key must render INLINE (base-pull), which is legal + there because the CTE is the aggregate's own scope. Observable as: no + cross-model plan at all. + """ + plan = plan_query( + query=_HOST_GRAIN_WITH_PATH, + bundle=dev1747_bundle(), + disable_host_rooted_isolation=True, + ) + assert not plan.cross_model_aggregate_plans, ( + "a host-grain wrap still isolated under " + "disable_host_rooted_isolation — the sub-plan would recurse" + ) + + def test_the_disabled_flag_does_not_suppress_target_rooted_plans(self) -> None: + """The flag is scoped to HOST-rooted isolation. Suppressing genuine + cross-model aggregates too would silently inline a joined SUM into the + host base and multiply it by the join's fan-out.""" + plan = plan_query( + query=_TARGET_GRAIN_WITH_PATH, + bundle=dev1747_bundle(), + disable_host_rooted_isolation=True, + ) + assert plan.cross_model_aggregate_plans, ( + "disable_host_rooted_isolation wrongly suppressed a target-rooted " + "cross-model plan" + ) diff --git a/tests/test_dev1747_order_resolver.py b/tests/test_dev1747_order_resolver.py new file mode 100644 index 00000000..a8f54d46 --- /dev/null +++ b/tests/test_dev1747_order_resolver.py @@ -0,0 +1,508 @@ +"""DEV-1747 §5.10 — the single order-term resolver. + +Four renderers currently build ORDER BY terms independently +(``_apply_order_limit_from_planned``, ``_resolve_combined_order_term``, +``_planned_order_by_sql``, and the ``emit_outer_wrap`` qualifier-strip fix-up). +They disagree on null ordering, on what happens when a term cannot be resolved, +and on how the reference is qualified. §5.10 replaces all four with +``slayer.sql.render.order_terms.resolve_order_term(entry, env)``: one dict +dispatch on ``entry.scope``, zero precedence. + +Two behaviour changes fall out of the consolidation and are pinned here: + +* **D4** — the combined path currently returns ``None`` when a cross-model + order slot has no alias, which SILENTLY drops the sort term and returns + unsorted rows. ``_planned_order_by_sql`` already raises in the same + situation. The resolver raises everywhere. +* **D5** — the T-SQL ``nulls_first`` pin that suppresses sqlglot's mis-resolving + ``CASE WHEN … IS NULL`` emulation lives in ``SQLGenerator._ordered`` and is + therefore MISSING on the combined and transform-chain paths. It moves into + the dialect strategy (``SqlDialect.build_ordered``), so every path gets it + (P-H). + +Refs: DEV-1747 (D3, D4, D5), DEV-1742 §5.10 / P-G / P-H, DEV-1571 Bug 2. +""" +from __future__ import annotations + +import pytest +from sqlglot import exp + +from slayer.core.enums import TimeGranularity +from slayer.core.query import ColumnRef, OrderItem, SlayerQuery, TimeDimension +from tests._dev1747_fixtures import ( + cte_map, + dev1747_bundle, + dev1747_models, + order_by_text, + order_terms, + outermost_select, +) +from tests._engine_helpers import _engine_generate + +_MEASURE = [{"formula": "amount:sum", "name": "rev"}] + +_MONTH = TimeDimension( + dimension=ColumnRef(name="created_at"), granularity=TimeGranularity.MONTH, +) + +#: One query per render path that builds its own ORDER BY term. D4's claim is +#: that ALL of them raise on an unresolvable slot; today only the transform +#: chain does, and the rest return unsorted rows. +_D4_SHAPES = { + "host_base": SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=_MEASURE, + order=[OrderItem(column=ColumnRef(name="rev"), direction="desc")], + ), + "cross_model": SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ + {"formula": "amount:sum", "name": "rev"}, + {"formula": "customers.spend:sum", "name": "cs"}, + ], + order=[OrderItem(column=ColumnRef(name="cs"), direction="desc")], + ), + "outer_composite": SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[{"formula": "customers.spend:sum + amount:sum", "name": "mix"}], + order=[OrderItem(column=ColumnRef(name="mix"), direction="desc")], + ), + "windowed": SlayerQuery( + source_model="orders", + time_dimensions=[_MONTH], + measures=[{"formula": "amount:sum(window='90d')", "name": "w"}], + order=[OrderItem(column=ColumnRef(name="w"), direction="desc")], + ), + "transform_chain": SlayerQuery( + source_model="orders", + time_dimensions=[_MONTH], + measures=[{"formula": "cumsum(amount:sum)", "name": "cs"}], + order=[OrderItem(column=ColumnRef(name="cs"), direction="asc")], + ), +} + + +def _outer_projection(sql: str, *, dialect: str = "postgres"): + """The outermost SELECT's projection expressions.""" + return outermost_select(sql, dialect=dialect).expressions + + +def _order_columns(sql: str, *, dialect: str = "postgres"): + """Every ``exp.Column`` appearing in the outermost ORDER BY.""" + order = outermost_select(sql, dialect=dialect).args.get("order") + return list(order.find_all(exp.Column)) if order is not None else [] + + +async def _sql(query: SlayerQuery, *, dialect: str = "postgres") -> str: + models = dev1747_models() + return await _engine_generate( + query=query, model=models[0], extra_models=models[1:], dialect=dialect, + ) + + +def _assert_orders_by(sql: str, alias: str, *, dialect: str = "postgres") -> None: + """The outermost ORDER BY names ``alias``. + + ``assert order_by_text(sql)`` only says a sort term exists — it is equally + satisfied by a term pointing at the wrong column, which is the failure mode + four independent resolvers actually produce. + """ + terms = order_terms(sql, dialect=dialect) + assert terms, f"no ORDER BY emitted:\n{sql}" + referenced = {c.name for c in _order_columns(sql, dialect=dialect)} + assert alias in referenced, ( + f"ORDER BY names {referenced or set(terms)}, not {alias!r}:\n{sql}" + ) + + +def _assert_order_reference_resolves(sql: str, *, dialect: str = "postgres") -> None: + """Every column in the outermost ORDER BY is something the outer SELECT can + actually name: one of its own output aliases, or a reference qualified by a + relation in scope. + + An UNQUALIFIED name that is neither is resolvable only by falling through to + an input column of the FROM — which Postgres allows and other engines do + not, and which silently picks a different column the moment two scopes + project the same name. That is the shape ``_resolve_combined_order_term`` + emits for a cross-model measure today (``ORDER BY + "orders.customers.spend_sum"`` while the SELECT projects it ``AS + "orders.cs"``), and the drift §5.10's single resolver removes. + """ + select = outermost_select(sql, dialect=dialect) + projected = {s.alias_or_name for s in select.expressions} + in_scope = set(cte_map(sql, dialect=dialect)) + for table in select.find_all(exp.Table): + in_scope.add(table.alias_or_name) + for column in _order_columns(sql, dialect=dialect): + if column.table: + assert column.table in in_scope, ( + f"ORDER BY is qualified by {column.table!r}, which is not a " + f"relation in scope ({sorted(in_scope)}):\n{sql}" + ) + continue + assert column.name in projected, ( + f"ORDER BY names {column.name!r} unqualified, but the outer SELECT " + f"projects {sorted(projected)} — the term resolves only by falling " + f"through to an input column:\n{sql}" + ) + + +# --------------------------------------------------------------------------- +# Group 1 — the §5.10 target matrix +# --------------------------------------------------------------------------- +class TestOrderTargetMatrix: + async def test_duplicate_display_aliases_order_on_one_column(self) -> None: + """One structural slot carrying TWO public aliases (C13): the same + formula declared under two names interns once and projects twice. The + sort must name a column the SELECT actually projects — the resolver + takes the FIRST alias, since both hold the same value.""" + sql = await _sql(SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ + {"formula": "amount:sum", "name": "rev"}, + {"formula": "amount:sum", "name": "rev_again"}, + ], + order=[OrderItem(column=ColumnRef(name="rev_again"), direction="desc")], + )) + terms = order_by_text(sql) + assert terms + projected = {s.alias_or_name for s in _outer_projection(sql)} + referenced = { + c.name for c in _order_columns(sql) + } + assert referenced <= projected, ( + f"ORDER BY names {referenced - projected}, which the SELECT does " + f"not project:\n{sql}" + ) + + async def test_dotted_joined_dimension_alias(self) -> None: + """A joined dimension projects under its DOTTED result key, so the + ORDER BY must match that, not the flat ``__`` declared name.""" + sql = await _sql(SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="name", model="customers.regions")], + measures=_MEASURE, + order=[OrderItem( + column=ColumnRef(name="name", model="customers.regions"), + direction="asc", + )], + )) + terms = order_by_text(sql) + assert "customers.regions.name" in terms or "customers__regions" in terms, ( + f"dotted joined sort key did not resolve to a projected alias:\n{sql}" + ) + + async def test_hidden_measure_orders_without_being_projected(self) -> None: + """Both halves: the sort happens, and the hidden slot stays out of the + public projection. Asserting only the first would pass if the slot were + quietly projected, which changes the result shape.""" + sql = await _sql(SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=_MEASURE, + order=[OrderItem(column=ColumnRef(name="id"), direction="desc")], + )) + assert order_terms(sql), f"hidden sort key produced no ORDER BY:\n{sql}" + projected = {s.alias_or_name for s in _outer_projection(sql)} + assert projected == {"orders.status", "orders.rev"}, ( + f"the hidden order slot leaked into the projection: {projected}" + ) + + async def test_ordinal_looking_alias_is_not_read_as_a_position(self) -> None: + """A digit-led alias would make ``ORDER BY `` ambiguous with + SQL's positional form, which silently sorts by whatever is first. + + ``ColumnRef`` forbids a leading digit outright, so the truly ordinal + case is unreachable — pinned here so a future relaxation of that + validator cannot open the hole silently. The nearest REACHABLE shape + (``_1``) must still emit a quoted identifier, not a bare token.""" + import pydantic + + with pytest.raises(pydantic.ValidationError): + ColumnRef(name="1") + + sql = await _sql(SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[{"formula": "amount:sum", "name": "_1"}], + order=[OrderItem(column=ColumnRef(name="_1"), direction="asc")], + )) + for term in order_terms(sql): + stripped = term.strip() + assert not stripped[0].isdigit(), ( + f"sort term reads as a positional reference:\n{sql}" + ) + + async def test_transformed_measure_orders_at_the_chain_outer_wrap(self) -> None: + sql = await _sql(SlayerQuery( + source_model="orders", + time_dimensions=[TimeDimension( + dimension=ColumnRef(name="created_at"), + granularity=TimeGranularity.MONTH, + )], + measures=[{"formula": "cumsum(amount:sum)", "name": "cs"}], + order=[OrderItem(column=ColumnRef(name="cs"), direction="desc")], + )) + _assert_orders_by(sql, "orders.cs") + + async def test_cross_model_aggregate_orders_on_its_cte_column(self) -> None: + sql = await _sql(SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ + {"formula": "amount:sum", "name": "rev"}, + {"formula": "customers.spend:sum", "name": "cs"}, + ], + order=[OrderItem(column=ColumnRef(name="cs"), direction="desc")], + )) + _assert_order_reference_resolves(sql) + + async def test_rerooted_aggregate_orders_correctly(self) -> None: + """A cross-model aggregate whose CTE is RE-ROOTED at the target still + exposes one column to order on — the reroot must not change the + order-term resolution.""" + sql = await _sql(SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="name", model="customers.regions")], + measures=[ + {"formula": "amount:sum", "name": "rev"}, + {"formula": "customers.spend:sum", "name": "cs"}, + ], + order=[OrderItem(column=ColumnRef(name="cs"), direction="desc")], + )) + _assert_order_reference_resolves(sql) + + async def test_windowed_measure_orders_on_its_cte_column(self) -> None: + sql = await _sql(SlayerQuery( + source_model="orders", + time_dimensions=[TimeDimension( + dimension=ColumnRef(name="created_at"), + granularity=TimeGranularity.MONTH, + )], + measures=[{"formula": "amount:sum(window='90d')", "name": "w"}], + order=[OrderItem(column=ColumnRef(name="w"), direction="desc")], + )) + _assert_orders_by(sql, "orders.w") + + +# --------------------------------------------------------------------------- +# Group 2 — D4: unresolvable is an error, never a silent drop +# --------------------------------------------------------------------------- +class TestUnresolvableRaises: + def test_resolver_raises_when_the_scope_lookup_misses(self) -> None: + from slayer.core.keys import Phase + from slayer.engine.planned import OrderEntry, OrderScope + from slayer.sql.render.order_terms import OrderEnv, resolve_order_term + + entry = OrderEntry( + slot_id="missing", direction="asc", + scope=OrderScope.CROSS_MODEL_CTE, phase=Phase.AGGREGATE, + ) + with pytest.raises(Exception) as exc: + resolve_order_term(entry=entry, env=OrderEnv()) + assert "missing" in str(exc.value), ( + "the error must name the unresolvable slot so the wiring bug is " + "findable; a bare exception is barely better than the silent drop" + ) + + def test_every_scope_raises_on_a_missing_slot(self) -> None: + """Totality of the failure mode, over the dispatch table rather than + over its source text. A ``return None`` in ONE arm is enough to + reintroduce the silent drop, and a per-scope loop is what catches an + arm added later without one.""" + from slayer.core.keys import Phase + from slayer.engine.planned import OrderEntry, OrderScope + from slayer.sql.render.order_terms import OrderEnv, resolve_order_term + + for scope in OrderScope: + entry = OrderEntry( + slot_id="missing", direction="asc", + scope=scope, phase=Phase.AGGREGATE, + ) + with pytest.raises(Exception): + resolve_order_term(entry=entry, env=OrderEnv()) + + @pytest.mark.parametrize("shape", sorted(_D4_SHAPES)) + def test_no_render_path_silently_drops_an_unresolvable_term( + self, shape: str, + ) -> None: + """D4 "everywhere", proven per render path. + + Today only the transform chain raises. The other four rebuild the term + independently and return unsorted rows with no error — verified by + rewriting the planned order entry to name a slot that does not exist + and rendering. Injecting at the PLAN is what makes the injection + path-independent; every renderer reads the same field. + """ + from slayer.engine.stage_planner import plan_query + from slayer.sql.generator import generate_from_planned + + plan = plan_query(query=_D4_SHAPES[shape], bundle=dev1747_bundle()) + assert plan.order, f"{shape} planned no order entry — test is vacuous" + broken = plan.model_copy(update={ + "order": [ + entry.model_copy(update={"slot_id": "no_such_slot"}) + for entry in plan.order + ], + }) + with pytest.raises(Exception) as exc: + generate_from_planned(broken, bundle=dev1747_bundle()) + assert "no_such_slot" in str(exc.value), ( + f"{shape} raised without naming the slot: {exc.value}" + ) + + +# --------------------------------------------------------------------------- +# Group 3 — D5: null ordering through the dialect strategy +# --------------------------------------------------------------------------- +class TestNullOrdering: + @pytest.mark.parametrize("direction", ["asc", "desc"]) + @pytest.mark.parametrize("policy", ["default", "first", "last"]) + def test_dialect_hook_covers_every_direction_and_policy( + self, direction: str, policy: str, + ) -> None: + from slayer.sql.dialects.base import SqlDialect + + ordered = SqlDialect().build_ordered( + exp.column("a", quoted=True), + descending=(direction == "desc"), + nulls=policy, + ) + assert isinstance(ordered, exp.Ordered) + assert ordered.args.get("desc") is (direction == "desc") + + @pytest.mark.parametrize("direction", ["asc", "desc"]) + def test_tsql_pins_nulls_first_to_its_native_default( + self, direction: str, + ) -> None: + """T-SQL's ORDER BY resolver mis-resolves the bracketed alias INSIDE + sqlglot's CASE-WHEN nulls emulation, so the pin suppresses it.""" + from slayer.sql.dialects.tsql import TsqlDialect + + ordered = TsqlDialect().build_ordered( + exp.column("a", quoted=True), + descending=(direction == "desc"), + nulls="default", + ) + assert ordered.args.get("nulls_first") is (direction == "asc") + + async def test_tsql_combined_path_has_no_case_when_emulation(self) -> None: + """The gap D5 closes: the combined (cross-model) path builds its own + ``exp.Ordered`` today and skips the pin entirely.""" + sql = await _sql( + SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ + {"formula": "amount:sum", "name": "rev"}, + {"formula": "customers.spend:sum", "name": "cs"}, + ], + order=[OrderItem(column=ColumnRef(name="cs"), direction="desc")], + ), + dialect="tsql", + ) + order_clause = order_by_text(sql, dialect="tsql") + assert "CASE" not in order_clause.upper(), ( + f"T-SQL combined ORDER BY still emits the CASE-WHEN nulls " + f"emulation:\n{sql}" + ) + + async def test_tsql_transform_chain_has_no_case_when_emulation(self) -> None: + sql = await _sql( + SlayerQuery( + source_model="orders", + time_dimensions=[TimeDimension( + dimension=ColumnRef(name="created_at"), + granularity=TimeGranularity.MONTH, + )], + measures=[{"formula": "cumsum(amount:sum)", "name": "cs"}], + order=[OrderItem(column=ColumnRef(name="cs"), direction="asc")], + ), + dialect="tsql", + ) + assert "CASE" not in order_by_text(sql, dialect="tsql").upper() + + +# --------------------------------------------------------------------------- +# Group 4 — one resolver, not four +# --------------------------------------------------------------------------- +class TestSingleResolver: + """The superseded resolvers stay in the file (P-J state 1) but must lose + every production caller. + + Proven with raising sentinels over every render shape rather than by + grepping the module: a source scan cannot tell a live call from one inside + a docstring or an unreachable branch, and it silently stops meaning + anything the moment the method is renamed. + """ + + @pytest.mark.parametrize("shape", sorted(_D4_SHAPES)) + @pytest.mark.parametrize( + "method", + ["_resolve_combined_order_term", "_apply_order_limit_from_planned"], + ) + async def test_superseded_resolver_is_never_called( + self, method: str, shape: str, monkeypatch, + ) -> None: + from slayer.sql.generator import SQLGenerator + + assert hasattr(SQLGenerator, method), ( + f"{method} has been deleted; P-J defers deletion to PR 6, so " + f"update this test deliberately rather than losing the guard" + ) + + def _boom(*_a, **_kw): + raise AssertionError( + f"{method} is still on the production render path — §5.10 " + f"replaces all four resolvers with resolve_order_term" + ) + + monkeypatch.setattr(SQLGenerator, method, _boom) + await _sql(_D4_SHAPES[shape]) + + @pytest.mark.parametrize("dialect", ["postgres", "sqlite", "duckdb", "tsql"]) + async def test_same_construct_same_sort_term_across_paths( + self, dialect: str, + ) -> None: + """P-G: ordering by ``rev`` means the same thing whether the query is + single-model (base path) or cross-model (combined path). + + "Both emit an ORDER BY" is not that claim — two paths can both emit a + term and order by different things. What is pinned is that the two + paths emit the SAME term, and that each term resolves against the outer + SELECT. Compared to EACH OTHER rather than to a literal, because the + public alias is dotted and the mangling dialects legitimately spell it + differently (``orders___rev`` on T-SQL).""" + rendered = {} + for label, query in ( + ("base", SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=_MEASURE, + order=[OrderItem(column=ColumnRef(name="rev"), direction="desc")], + )), + ("combined", SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ + {"formula": "amount:sum", "name": "rev"}, + {"formula": "customers.spend:sum", "name": "cs"}, + ], + order=[OrderItem(column=ColumnRef(name="rev"), direction="desc")], + )), + ): + sql = await _sql(query, dialect=dialect) + terms = order_terms(sql, dialect=dialect) + assert terms, f"no ORDER BY on {label}/{dialect}:\n{sql}" + _assert_order_reference_resolves(sql, dialect=dialect) + rendered[label] = { + c.name for c in _order_columns(sql, dialect=dialect) + } + assert rendered["base"] == rendered["combined"], ( + f"on {dialect} the same construct sorts by {rendered['base']} on " + f"the base path and {rendered['combined']} on the combined path" + ) diff --git a/tests/test_dev1747_prebound_planner.py b/tests/test_dev1747_prebound_planner.py new file mode 100644 index 00000000..a959c303 --- /dev/null +++ b/tests/test_dev1747_prebound_planner.py @@ -0,0 +1,345 @@ +"""DEV-1747 §5.4 — the pre-bound planner seam, and zero text round-trips. + +Rerooting today builds its nested plan by SERIALIZING typed keys back to +formula text and re-parsing them:: + + formula = _local_agg_formula(agg_key) # AggregateKey -> "spend:sum" + rr = _reroot_ref(...) # ColumnRef -> "regions.name" + SlayerQuery(measures=[ModelMeasure(formula=formula)], filters=[]) + sub_plan = subplan_builder(rerooted_query, rerooted_bundle) # re-parses all of it + +That is the P-E violation: structural identity is laundered through a string +and re-derived. §5.4 replaces it with ``PreboundQuery`` — the typed product of +``plan_query``'s bind block — handed straight back to ``plan_query(prebound=…)``, +which then skips binding entirely. + +Two properties are asserted: + +* **Equivalence** — a prebound plan is structurally identical to the plan the + text path produces, across a battery of shapes. Without this the seam is + just a second planner. +* **No round-trips** — the parser is never invoked inside the reroot subtree. + The spy is scoped to the subplan-builder boundary rather than being a global + counter, because the HOST query legitimately parses. + +The carrier guard is the third leg: ``plan_query`` reads ``query.*`` in several +places AFTER the bind block (``distinct_dimension_values``, ``source_model``, +``dimensions``, ``time_dimensions``, ``limit``, ``offset``). A prebound call +that forgets one would silently inherit a default, so the reroot passes a +STRICT carrier that raises on any attribute the seam has not approved. + +Refs: DEV-1747 (D1), DEV-1742 §5.4 / P-D / P-E. +""" +from __future__ import annotations + +import pytest + +from slayer.core.enums import TimeGranularity +from slayer.core.query import ColumnRef, OrderItem, SlayerQuery, TimeDimension +from slayer.engine.stage_planner import plan_query +from tests._dev1747_fixtures import dev1747_bundle + +_SHAPES = { + "plain": SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[{"formula": "amount:sum", "name": "rev"}], + ), + "cross_model": SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ + {"formula": "amount:sum", "name": "rev"}, + {"formula": "customers.spend:sum", "name": "cs"}, + ], + ), + "rerooted": SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="name", model="customers.regions")], + measures=[ + {"formula": "amount:sum", "name": "rev"}, + {"formula": "customers.spend:sum", "name": "cs"}, + ], + ), + "filtered": SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[{"formula": "amount:sum", "name": "rev"}], + filters=["customers.tier == 'gold'"], + ), + "time_dimension": SlayerQuery( + source_model="orders", + time_dimensions=[TimeDimension( + dimension=ColumnRef(name="created_at"), + granularity=TimeGranularity.MONTH, + date_range=["2024-01-01", "2024-12-31"], + )], + measures=[{"formula": "amount:sum", "name": "rev"}], + ), + "ordered_paginated": SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[{"formula": "amount:sum", "name": "rev"}], + order=[OrderItem(column=ColumnRef(name="rev"), direction="desc")], + limit=5, + offset=2, + ), + "raw_rows": SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + distinct_dimension_values=False, + limit=3, + ), + # The HOST-rooted nested plan (``cte_root_model == "orders"``): a derived + # column whose ``Column.sql`` crosses a join. A different set of helpers + # builds this one, so the equivalence battery and the P-J sentinels both + # need it alongside the target-rooted "rerooted" shape. + "host_rooted": SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[{"formula": "amount:sum", "name": "rev"}], + order=[OrderItem(column=ColumnRef(name="cust_region"), direction="asc")], + ), +} + + +def _plan(query: SlayerQuery): + return plan_query(query=query, bundle=dev1747_bundle()) + + +# --------------------------------------------------------------------------- +# Group 1 — the seam exists and is faithful +# --------------------------------------------------------------------------- +class TestPreboundEquivalence: + @pytest.mark.parametrize("shape", sorted(_SHAPES)) + def test_prebound_plan_matches_the_text_plan(self, shape: str) -> None: + """Extract the bind product from a normal plan, feed it back through + ``prebound=``, and require an identical plan. Any divergence means the + seam is a second planner rather than the same one.""" + from slayer.engine.stage_planner import bind_query_inputs + + query = _SHAPES[shape] + expected = _plan(query) + prebound = bind_query_inputs(query=query, bundle=dev1747_bundle()) + actual = plan_query( + query=query, bundle=dev1747_bundle(), prebound=prebound, + ) + assert actual.model_dump() == expected.model_dump() + + def test_prebound_is_optional(self) -> None: + """Every existing caller passes no ``prebound`` and must be unaffected.""" + assert _plan(_SHAPES["plain"]).order == [] + + def test_bind_query_inputs_carries_the_post_bind_scalars(self) -> None: + """The fields ``plan_query`` reads off ``query`` AFTER binding. A + missing one silently inherits a default — which is exactly the class of + bug the strict carrier below is meant to make impossible.""" + from slayer.engine.stage_planner import bind_query_inputs + + prebound = bind_query_inputs( + query=_SHAPES["ordered_paginated"], bundle=dev1747_bundle(), + ) + assert prebound.limit == 5 + assert prebound.offset == 2 + assert prebound.n_dims == 1 + assert prebound.n_time_dimensions == 0 + + def test_prebound_carries_the_resolved_main_time_key(self) -> None: + """``_resolve_main_time_dimension`` takes the whole ``query``; the + prebound path must supply the resolved key instead of re-deriving it + from a text carrier.""" + from slayer.engine.stage_planner import bind_query_inputs + + prebound = bind_query_inputs( + query=_SHAPES["time_dimension"], bundle=dev1747_bundle(), + ) + assert prebound.main_time_key is not None + + +# --------------------------------------------------------------------------- +# Group 2 — no formula-text round-trips in the reroot path +# --------------------------------------------------------------------------- +class TestNoTextRoundTrip: + def _reroot_query(self) -> SlayerQuery: + return _SHAPES["rerooted"] + + def test_reroot_does_not_parse_inside_the_subplan_boundary( + self, monkeypatch, + ) -> None: + """Scoped spy: the HOST query parses legitimately, so a global counter + would be meaningless. The sentinel raises only once the reroot has + begun building its nested plan.""" + from slayer.engine import cross_model_planner + + real_builder_calls: list[int] = [] + original = cross_model_planner._maybe_reroot_cross_model_plan + + def _wrapped(**kwargs): + real_builder_calls.append(1) + from slayer.engine import stage_planner + + def _boom(*_a, **_kw): + raise AssertionError( + "the reroot path parsed formula text — §5.4 requires the " + "nested PlannedQuery to be built from typed keys" + ) + + # Patch the names as BOUND in each module (both do + # ``from … import parse_expr``), so patching the defining module + # would miss them entirely and the test would pass vacuously. + for module in (cross_model_planner, stage_planner): + for symbol in ( + "parse_expr", "parse_filter_expr", + "bind_expr", "bind_filter", "bind_time_dimension", + ): + if hasattr(module, symbol): + monkeypatch.setattr(module, symbol, _boom) + try: + return original(**kwargs) + finally: + monkeypatch.undo() + + monkeypatch.setattr( + cross_model_planner, "_maybe_reroot_cross_model_plan", _wrapped, + ) + _plan(self._reroot_query()) + assert real_builder_calls, "the reroot path never ran — test is vacuous" + + def test_the_text_serializers_are_never_called(self, monkeypatch) -> None: + """The text serializers stay (P-J state 1) but must lose every + production caller — otherwise the round-trip is still live. + + Sentinels rather than a source scan: a grep for ``_reroot_ref(`` also + matches a call inside a docstring or a dead branch, and stops matching + the moment the symbol is renamed. A function that raises when invoked + is exactly as strong as the claim being made. + + Both nested-plan routes are planned, because the serializers are split + between them — ``_reroot_ref`` on the TARGET-rooted reroot, + ``_local_agg_formula`` on the HOST-rooted one. Exercising a single + shape would leave the other's serializer free to stay live. + """ + from slayer.engine import cross_model_planner + + patched = [] + for symbol in ("_local_agg_formula", "_reroot_ref", "_render_ref_formula"): + if not hasattr(cross_model_planner, symbol): + continue + + def _boom(*_a, _symbol=symbol, **_kw): + raise AssertionError( + f"{_symbol} is still on the production reroot path — §5.4 " + f"replaces formula-text round-trips with typed keys" + ) + + monkeypatch.setattr(cross_model_planner, symbol, _boom) + patched.append(symbol) + assert patched, ( + "none of the text serializers exist any more; P-J state 1 keeps " + "them until PR 6, so update this test deliberately rather than " + "letting it pass vacuously" + ) + target_rooted = _plan(_SHAPES["rerooted"]) + assert target_rooted.cross_model_aggregate_plans[0].rerooted_plan is not None + host_rooted = _plan(_SHAPES["host_rooted"]) + assert host_rooted.cross_model_aggregate_plans[0].cte_root_model == "orders" + + def test_nested_plan_measure_is_a_typed_key_not_a_formula(self) -> None: + """The observable end state: the nested plan's aggregate slot carries + the RE-ROOTED typed key, byte-identical to what the visitor produces — + not something re-derived from a string.""" + from slayer.core.keys import AggregateKey, ColumnKey, reroot_value_key + + plan = _plan(self._reroot_query()) + cma = plan.cross_model_aggregate_plans[0] + assert cma.rerooted_plan is not None + sub_agg = next( + s for s in cma.rerooted_plan.aggregate_slots + if isinstance(s.key, AggregateKey) + ) + expected = reroot_value_key( + AggregateKey( + source=ColumnKey(path=("customers",), leaf="spend"), agg="sum", + ), + target_path=("customers",), + ) + assert sub_agg.key == expected + + +# --------------------------------------------------------------------------- +# Group 3 — the strict carrier guard +# --------------------------------------------------------------------------- +class TestStrictCarrier: + def test_unapproved_attribute_access_raises(self) -> None: + """The guard Codex asked for: if ``plan_query`` grows a new post-bind + ``query.*`` read and the seam does not carry it, the reroot must FAIL + rather than silently plan against a default.""" + from slayer.engine.stage_planner import StrictQueryCarrier + + carrier = StrictQueryCarrier(source_model="orders") + with pytest.raises(AttributeError): + _ = carrier.some_field_the_seam_never_approved + + def test_approved_attributes_pass_through(self) -> None: + from slayer.engine.stage_planner import StrictQueryCarrier + + carrier = StrictQueryCarrier(source_model="orders") + assert carrier.source_model == "orders" + + def test_reroot_actually_constructs_the_strict_carrier( + self, monkeypatch, + ) -> None: + """Wiring check — the guard only guards if the reroot actually uses it. + + Recorded at runtime: the name appearing in the module source proves + nothing about the live path (an import, a comment, or a branch that is + never taken all satisfy a grep). + """ + from slayer.engine import cross_model_planner + from slayer.engine.stage_planner import StrictQueryCarrier + + built: list = [] + + class _Recording(StrictQueryCarrier): + def __init__(self, **kwargs): + super().__init__(**kwargs) + built.append(self) + + monkeypatch.setattr(cross_model_planner, "StrictQueryCarrier", _Recording) + _plan(_SHAPES["rerooted"]) + assert built, ( + "the reroot built no StrictQueryCarrier — it is still handing the " + "nested planner a full SlayerQuery, so an unapproved post-bind " + "``query.*`` read would silently inherit a default" + ) + + def test_the_nested_planner_receives_the_carrier_not_a_text_query( + self, monkeypatch, + ) -> None: + """The seam's boundary condition. A nested ``plan_query`` given a real + ``SlayerQuery`` would re-bind formula text no matter how the keys were + built upstream.""" + from slayer.engine import cross_model_planner + + seen: list = [] + original = cross_model_planner.IsolatedCteCrossModelPlanner.plan + + def _recording(self, **kwargs): + builder = kwargs.get("subplan_builder") + if builder is not None: + def _wrapped(query, bundle, _builder=builder): + seen.append(query) + return _builder(query, bundle) + + kwargs["subplan_builder"] = _wrapped + return original(self, **kwargs) + + monkeypatch.setattr( + cross_model_planner.IsolatedCteCrossModelPlanner, "plan", _recording, + ) + _plan(_SHAPES["rerooted"]) + assert seen, "the subplan builder was never invoked — test is vacuous" + assert not any(isinstance(q, SlayerQuery) for q in seen), ( + f"the nested planner was handed a SlayerQuery ({seen!r}); §5.4 " + f"requires the typed pre-bound carrier" + ) diff --git a/tests/test_dev1747_reroot_filter_routing.py b/tests/test_dev1747_reroot_filter_routing.py new file mode 100644 index 00000000..25192092 --- /dev/null +++ b/tests/test_dev1747_reroot_filter_routing.py @@ -0,0 +1,510 @@ +"""DEV-1747 B6 — no silent filter drops in rerooting. + +The defect, reproducible today on this corpus: a cross-model aggregate whose +CTE is RE-ROOTED at its target ends up with EMPTY routing lists and ZERO +warnings, whatever the filter is:: + + reachable customers.regions.name == 'Alpha' -> where [] having [] dropped [] + host-local status == 'A' -> where [] having [] dropped [] + unreachable order_tags.name == 'rush' -> where [] having [] dropped [] + +Two mechanisms conspire. ``_maybe_reroot_cross_model_plan`` decides which +filters ride into the re-rooted CTE by TRY-BINDING each one against the target +scope and swallowing ``_REROOT_BIND_ERRORS`` — a tuple that includes bare +``ValueError``, so a planner bug is indistinguishable from "not reachable". It +then CLEARS ``dropped_filter_warnings`` / ``where_filter_ids`` / +``having_filter_ids`` / ``applied_filter_ids``, throwing away the routing the +decision table had already produced. + +B6 (with D6/D7) fixes both: decide reroot-vs-forward FIRST, then run +``classify_host_filter`` exactly once per host filter in the coordinate system +of the CTE that will actually exist, and keep the result. Unreachable filters +warn per §5.5. Binder/planner failures RAISE — they never masquerade as +expected drops. + +D7 scopes this to FILTERS. An unreachable rerooted DIMENSION still drops (the +documented reroot contract), but structurally — not via a swallowed exception. + +Refs: DEV-1747 (B6, D6, D7), DEV-1742 §5.4 / §5.5. +""" +from __future__ import annotations + +import os +import tempfile +import warnings + +import pytest + +from slayer.core.errors import UnreachableFilterDroppedWarning +from slayer.core.query import ColumnRef, OrderItem, SlayerQuery +from slayer.engine.stage_planner import plan_query +from tests._dev1747_fixtures import ( + dev1747_bundle, + make_sqlite_engine, + seed_dev1747_sqlite, +) + +#: A cross-model aggregate PLUS a dimension one hop PAST the target, which is +#: what makes the planner re-root the CTE at ``customers`` instead of using the +#: forward-path shape. +_CROSS_MODEL_MEASURE = {"formula": "customers.spend:sum", "name": "cs"} + +#: Reachable from the re-rooted target (``customers -> regions``). +FILTER_REACHABLE = "customers.regions.name == 'Alpha'" +#: Purely host-local — filters host rows, stays at the host base, never warns. +FILTER_HOST_LOCAL = "status == 'A'" +#: Off the target's graph entirely (``orders -> order_tags``) — unreachable +#: from a CTE rooted at ``customers``. +FILTER_UNREACHABLE = "order_tags.name == 'rush'" + +#: A HOST-ROOTED shape (``cte_root_model == "orders"``): ordering by a derived +#: column whose ``Column.sql`` crosses. The DEV-1503/DEV-1709 helpers live only +#: on this route, so a sentinel aimed at them has to plan THIS query — a +#: target-rooted one leaves them untouched and asserts nothing. +_HOST_ROOTED_QUERY = SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[{"formula": "amount:sum", "name": "rev"}], + order=[OrderItem(column=ColumnRef(name="cust_region"), direction="asc")], +) + + +def _query(*filters: str) -> SlayerQuery: + return SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="name", model="customers.regions")], + measures=[{"formula": "amount:sum", "name": "rev"}, _CROSS_MODEL_MEASURE], + filters=list(filters) or None, + ) + + +def _plans(*filters: str): + plan = plan_query(query=_query(*filters), bundle=dev1747_bundle()) + assert plan.cross_model_aggregate_plans, "no cross-model plan was produced" + return plan.cross_model_aggregate_plans + + +def _sole_plan(*filters: str): + plans = _plans(*filters) + assert len(plans) == 1, f"expected one cross-model plan, got {len(plans)}" + return plans[0] + + +# --------------------------------------------------------------------------- +# Group 1 — routing survives the reroot (D6) +# --------------------------------------------------------------------------- +class TestRoutingSurvivesReroot: + def test_the_plan_is_actually_rerooted(self) -> None: + """Guard for the rest of the module: if the shape stops re-rooting, + every assertion below becomes vacuous.""" + assert _sole_plan(FILTER_REACHABLE).rerooted_plan is not None + + def test_reachable_filter_is_routed_not_blanked(self) -> None: + plan = _sole_plan(FILTER_REACHABLE) + assert plan.applied_filter_ids, ( + "the re-rooted CTE applies this filter, so the plan must SAY so — " + "today the routing lists are cleared wholesale" + ) + assert plan.where_filter_ids or plan.having_filter_ids + + def test_host_local_filter_is_neither_propagated_nor_warned(self) -> None: + """``DROP_HOST_LOCAL``: the host base applies it and the join-back + propagates the cardinality reduction, so pushing it into the CTE would + risk binding a bare name to a same-named TARGET column.""" + plan = _sole_plan(FILTER_HOST_LOCAL) + assert not plan.where_filter_ids + assert not plan.having_filter_ids + assert not plan.dropped_filter_warnings + + def test_routing_lists_are_not_cleared_wholesale(self) -> None: + """The clear-and-redecide block sets all four lists to ``[]`` at once. + With a reachable filter present, that is observably wrong.""" + plan = _sole_plan(FILTER_REACHABLE, FILTER_HOST_LOCAL) + assert plan.applied_filter_ids, ( + "all routing was cleared even though a reachable filter exists" + ) + + def test_mixed_filters_route_independently(self) -> None: + plan = _sole_plan(FILTER_REACHABLE, FILTER_HOST_LOCAL, FILTER_UNREACHABLE) + assert plan.applied_filter_ids, "the reachable filter was not applied" + assert plan.dropped_filter_warnings, "the unreachable filter did not warn" + + +# --------------------------------------------------------------------------- +# Group 1b — classified EXACTLY once, in the CTE's coordinate system (D6) +# --------------------------------------------------------------------------- +def _classifier_spy(monkeypatch) -> list: + """Record every ``classify_host_filter`` call and its arguments. + + Patched as BOUND in ``cross_model_planner`` — the module both defines and + calls it, and a future move of the call site into another module would make + this spy silently record nothing, which the vacuity assertions below catch. + """ + from slayer.engine import cross_model_planner + + calls: list = [] + original = cross_model_planner.classify_host_filter + + def _recording(**kwargs): + route = original(**kwargs) + calls.append({**kwargs, "route": route}) + return route + + monkeypatch.setattr(cross_model_planner, "classify_host_filter", _recording) + return calls + + +class TestClassifiedExactlyOnce: + def test_each_host_filter_is_classified_once_per_cte(self, monkeypatch) -> None: + """D6's actual claim. Warning DEDUP at the engine boundary hides a + double classification, so counting warnings cannot establish this — the + count has to come from the classifier itself. + + One cross-model aggregate ⇒ one CTE ⇒ exactly one classification per + host filter. + """ + calls = _classifier_spy(monkeypatch) + _sole_plan(FILTER_REACHABLE, FILTER_HOST_LOCAL, FILTER_UNREACHABLE) + assert calls, "classify_host_filter was never called — spy is vacuous" + per_filter: dict = {} + for call in calls: + fid = call["host_filter"].filter_id + per_filter[fid] = per_filter.get(fid, 0) + 1 + assert set(per_filter.values()) == {1}, ( + f"host filters were classified more than once: {per_filter} — the " + f"clear-and-redecide block re-runs the decision (D6)" + ) + + def test_the_classifier_sees_the_cte_actual_root(self, monkeypatch) -> None: + """The coordinate-system half of D6. The re-rooted CTE is rooted at + ``customers``, so the classifier must be asked about ``target_path == + ("customers",)``. Classifying against the FORWARD path and then + re-rooting the CTE is how a reachable filter ends up judged unreachable + (and vice versa).""" + calls = _classifier_spy(monkeypatch) + plan = _sole_plan(FILTER_REACHABLE) + assert plan.rerooted_plan is not None, "shape stopped re-rooting" + assert calls, "classify_host_filter was never called — spy is vacuous" + target_paths = {call["target_path"] for call in calls} + assert target_paths == {("customers",)}, ( + f"the classifier was asked about {target_paths}, not the CTE's " + f"actual root ('customers',)" + ) + + def test_the_classifier_receives_the_structural_summary( + self, monkeypatch, + ) -> None: + """§5.3's structural crossing metadata is the input the decision is + made from. A classifier called with an EMPTY summary would judge every + filter host-local and drop nothing — passing the warning tests below by + accident.""" + calls = _classifier_spy(monkeypatch) + _sole_plan(FILTER_UNREACHABLE) + crossed = { + path + for call in calls + for path in call["host_filter"].crossed_join_paths + } + assert ("order_tags",) in crossed, ( + f"the unreachable filter reached the classifier without its " + f"crossed-path summary; saw {crossed}" + ) + + +# --------------------------------------------------------------------------- +# Group 2 — unreachable warns instead of vanishing (B6 / §5.5) +# --------------------------------------------------------------------------- +class TestUnreachableWarns: + def test_unreachable_filter_produces_a_warning_on_the_plan(self) -> None: + plan = _sole_plan(FILTER_UNREACHABLE) + assert plan.dropped_filter_warnings, ( + "an unreachable filter was dropped from the re-rooted CTE with no " + "warning — the B6 defect" + ) + + def test_warning_carries_the_original_filter_text(self) -> None: + """§5.5 payload fidelity — a warning that does not name the user's own + filter text cannot be acted on.""" + warning = _sole_plan(FILTER_UNREACHABLE).dropped_filter_warnings[0] + assert FILTER_UNREACHABLE in warning.filter_text + + def test_warning_carries_a_reason(self) -> None: + warning = _sole_plan(FILTER_UNREACHABLE).dropped_filter_warnings[0] + assert warning.reason and "reach" in warning.reason.lower() + + async def test_exactly_one_warning_per_filter_per_execute(self) -> None: + """The boundary dedups per filter identity. Two cross-model measures + classify the same filter twice; the user must still see it once.""" + query = SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="name", model="customers.regions")], + measures=[ + {"formula": "amount:sum", "name": "rev"}, + {"formula": "customers.spend:sum", "name": "cs"}, + {"formula": "customers.regions.population:sum", "name": "pop"}, + ], + filters=[FILTER_UNREACHABLE], + ) + with tempfile.TemporaryDirectory() as d: + db = os.path.join(d, "dev1747.db") + seed_dev1747_sqlite(db) + engine = await make_sqlite_engine(d, db) + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + await engine.execute(query) + dropped = [ + w for w in caught + if isinstance(w.message, UnreachableFilterDroppedWarning) + ] + assert len(dropped) == 1, ( + f"expected exactly one UnreachableFilterDroppedWarning, got " + f"{len(dropped)}: {[str(w.message) for w in dropped]}" + ) + + async def test_warnings_as_errors_mode_surfaces_the_drop(self) -> None: + """Someone running with ``-W error`` must be stopped, not silently + given fewer rows than they asked for.""" + with tempfile.TemporaryDirectory() as d: + db = os.path.join(d, "dev1747.db") + seed_dev1747_sqlite(db) + engine = await make_sqlite_engine(d, db) + with warnings.catch_warnings(): + warnings.simplefilter("error", UnreachableFilterDroppedWarning) + with pytest.raises(UnreachableFilterDroppedWarning): + await engine.execute(_query(FILTER_UNREACHABLE)) + + async def test_two_textually_distinct_filters_warn_separately(self) -> None: + """Identity is per FILTER, not per text-dedup bucket — two different + unreachable filters must not collapse into one warning.""" + query = SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="name", model="customers.regions")], + measures=[{"formula": "amount:sum", "name": "rev"}, _CROSS_MODEL_MEASURE], + filters=["order_tags.name == 'rush'", "order_tags.name == 'gift'"], + ) + with tempfile.TemporaryDirectory() as d: + db = os.path.join(d, "dev1747.db") + seed_dev1747_sqlite(db) + engine = await make_sqlite_engine(d, db) + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + await engine.execute(query) + dropped = [ + w for w in caught + if isinstance(w.message, UnreachableFilterDroppedWarning) + ] + assert len(dropped) == 2 + + +# --------------------------------------------------------------------------- +# Group 3 — internal failures RAISE (§5.5) +# --------------------------------------------------------------------------- +class TestInternalFailuresRaise: + def test_the_swallow_and_drop_path_is_never_taken(self, monkeypatch) -> None: + """``_REROOT_BIND_ERRORS`` includes bare ``ValueError``, so any planner + bug inside the reroot path currently reads as "filter unreachable". + + A RUNTIME sentinel rather than a source scan: emptying the tuple makes + every ``except _REROOT_BIND_ERRORS`` catch nothing, so if production + still swallows there, the swallowed exception escapes here. A grep for + the symbol would instead pass the moment someone renamed it. + """ + from slayer.engine import cross_model_planner + + monkeypatch.setattr(cross_model_planner, "_REROOT_BIND_ERRORS", ()) + plan = _sole_plan(FILTER_REACHABLE, FILTER_HOST_LOCAL, FILTER_UNREACHABLE) + assert plan.rerooted_plan is not None + assert plan.applied_filter_ids + assert plan.dropped_filter_warnings + + def test_the_text_filter_classifier_is_never_called(self, monkeypatch) -> None: + """P-J state 1: ``_classify_subplan_filters`` — which re-derives sub-plan + filters from ``routing.text`` — stays in the file but must lose every + production caller. D6 routes from the typed classification instead. + + Exercised on the HOST-ROOTED route: the helper is called only from + ``_plan_filtered_local``, so a target-rooted query would leave this + sentinel untripped and the test would assert nothing. + """ + from slayer.engine import cross_model_planner + + assert hasattr(cross_model_planner, "_classify_subplan_filters"), ( + "the helper was deleted; P-J defers deletion to PR 6" + ) + + def _boom(*_a, **_kw): + raise AssertionError( + "_classify_subplan_filters is still on the production path — " + "D6 classifies once, structurally, against the CTE's own root" + ) + + monkeypatch.setattr( + cross_model_planner, "_classify_subplan_filters", _boom, + ) + plan = plan_query( + query=_HOST_ROOTED_QUERY, bundle=dev1747_bundle(), + ) + assert plan.cross_model_aggregate_plans, "no host-rooted CTE was planned" + assert plan.cross_model_aggregate_plans[0].cte_root_model == "orders", ( + "the shape stopped being host-rooted — the sentinel would be " + "untripped for the wrong reason" + ) + + def test_no_bare_except_in_the_reroot_path(self) -> None: + """Scoped to the reroot functions rather than the whole module, so an + unrelated ``except Exception`` elsewhere in the file cannot fail this + (or, worse, be deleted to make it pass).""" + import inspect + + from slayer.engine import cross_model_planner + + for name in ( + "_maybe_reroot_cross_model_plan", + "_plan_filtered_local", + "_route_host_filters", + ): + target = getattr(cross_model_planner, name, None) or getattr( + cross_model_planner.IsolatedCteCrossModelPlanner, name, None, + ) + if target is None: + continue + source = inspect.getsource(target) + assert "except Exception" not in source, ( + f"{name} swallows all exceptions — that re-creates the B6 defect" + ) + + def test_planner_failure_propagates_rather_than_warning(self, monkeypatch) -> None: + """A genuine internal error must not be reported as an expected drop.""" + from slayer.engine import cross_model_planner + + boom = RuntimeError("planner exploded") + + def _explode(**_kwargs): + raise boom + + monkeypatch.setattr( + cross_model_planner, "classify_host_filter", _explode, raising=True, + ) + with pytest.raises(RuntimeError, match="planner exploded"): + _plans(FILTER_REACHABLE) + + +# --------------------------------------------------------------------------- +# Group 4 — D7: dims still drop, but structurally +# --------------------------------------------------------------------------- +class TestUnreachableDimensionsStillDrop: + def test_unreachable_dimension_is_dropped_without_a_warning(self) -> None: + """D7 keeps the documented reroot contract for dims — the change is + that the decision is structural, not a swallowed bind failure.""" + query = SlayerQuery( + source_model="orders", + dimensions=[ + ColumnRef(name="name", model="customers.regions"), + ColumnRef(name="name", model="order_tags"), + ], + measures=[{"formula": "amount:sum", "name": "rev"}, _CROSS_MODEL_MEASURE], + ) + plan = plan_query(query=query, bundle=dev1747_bundle()) + cma = plan.cross_model_aggregate_plans[0] + assert cma.rerooted_plan is not None + assert not cma.dropped_filter_warnings, ( + "a dropped DIMENSION must not raise a dropped-FILTER warning (D7)" + ) + + def test_reachable_dimensions_still_form_the_grain(self) -> None: + plan = _sole_plan() + assert plan.rerooted_grain_pairs, ( + "the re-rooted CTE lost its grain — the join-back would broadcast" + ) + + +# --------------------------------------------------------------------------- +# Group 5 — D2: the host-rooted dispatch accepts a path-bearing key +# --------------------------------------------------------------------------- +class TestFilteredLocalDispatchAccounting: + """``_dispatch_filtered_local`` is reached only when ``source.path`` is + EMPTY (``if not path:`` in ``IsolatedCteCrossModelPlanner.plan``), and it + raises "this is a plain local aggregate" unless it finds a crossing input. + + A ``grain="host"`` wrap has a NON-empty path and no crossing *input* — its + crossing IS the path. Both halves therefore have to change together, and + each fails in a different way: the first sends the wrap to a target-rooted + CTE (silent scalar CROSS JOIN), the second raises. Neither is visible in a + test that only checks the emitted SQL has an ORDER BY. + """ + + def _grouped_joined_order(self) -> SlayerQuery: + return SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[{"formula": "amount:sum", "name": "rev"}], + order=[OrderItem( + column=ColumnRef(name="name", model="customers.regions"), + direction="asc", + )], + ) + + def _dispatch_spy(self, monkeypatch) -> list: + from slayer.engine.cross_model_planner import IsolatedCteCrossModelPlanner + + calls: list = [] + original = IsolatedCteCrossModelPlanner._dispatch_filtered_local + + def _recording(self, **kwargs): + calls.append(kwargs["aggregate_key"]) + return original(self, **kwargs) + + monkeypatch.setattr( + IsolatedCteCrossModelPlanner, "_dispatch_filtered_local", _recording, + ) + return calls + + def test_host_grain_wrap_dispatches_to_the_host_rooted_route( + self, monkeypatch, + ) -> None: + calls = self._dispatch_spy(monkeypatch) + plan_query(query=self._grouped_joined_order(), bundle=dev1747_bundle()) + assert calls, ( + "the host-grain wrap never reached _dispatch_filtered_local — it " + "was routed to a TARGET-rooted CTE, which degenerates to a scalar " + "CROSS JOIN (D2)" + ) + assert calls[0].grain == "host" + assert calls[0].source.path == ("customers", "regions"), ( + "the wrap lost its path on the way to the host-rooted route" + ) + + def test_a_target_grain_aggregate_does_not_take_that_route( + self, monkeypatch, + ) -> None: + """The contrast: a genuine cross-model measure must keep going to the + target-rooted CTE. Widening the dispatch to every path-bearing key + would move it, and its value would silently become per-host-group.""" + calls = self._dispatch_spy(monkeypatch) + plan_query( + query=SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ + {"formula": "amount:sum", "name": "rev"}, + _CROSS_MODEL_MEASURE, + ], + ), + bundle=dev1747_bundle(), + ) + assert not calls, ( + f"a target-grain aggregate was routed host-rooted: {calls}" + ) + + def test_the_path_counts_as_the_crossing_input(self) -> None: + """The accounting change itself. Without it the dispatch raises + ``ValueError('… this is a plain local aggregate')`` — a path-bearing + key has no ``column_filter_key`` and no crossing arg to find.""" + plan = plan_query( + query=self._grouped_joined_order(), bundle=dev1747_bundle(), + ) + assert plan.cross_model_aggregate_plans, "no host-rooted CTE was planned" + cma = plan.cross_model_aggregate_plans[0] + assert cma.cte_root_model == "orders", ( + f"the wrap's CTE is rooted at {cma.cte_root_model!r}, not the host" + ) diff --git a/tests/test_dev1747_reroot_visitor.py b/tests/test_dev1747_reroot_visitor.py new file mode 100644 index 00000000..b227bb0d --- /dev/null +++ b/tests/test_dev1747_reroot_visitor.py @@ -0,0 +1,503 @@ +"""DEV-1747 §5.4 — the total reroot visitor over the ValueKey union. + +Rerooting today is done by SERIALIZING typed keys back to formula text +(``_local_agg_formula`` / ``_reroot_ref`` in ``slayer/engine/cross_model_planner.py``) +and re-parsing them into a nested ``SlayerQuery``. §5.4 replaces that with +``reroot_value_key(key, *, target_path)`` — a visitor that is: + +* **total** over the union — every member has an explicit case, so a key kind + added later cannot silently ride through unrerooted; and +* **fail-closed** — an unhandled kind RAISES rather than being returned as-is, + because "returned unchanged" is indistinguishable from "correctly identity" + and is exactly how a mis-anchored ref reaches the SQL generator. + +The reroot rule is prefix-strip-with-residual, identical to the one +``reroot_aggregate_key`` already applies to ``AggregateKey``: a ``path`` +starting with ``target_path`` drops that prefix and keeps the residual hops; +any other ``path``, and any scalar, is returned unchanged. + +``AggregateKey.column_filter_key`` is deliberately copied UNCHANGED — see +``TestColumnFilterKeyInvariance`` for why that is not an oversight. + +Refs: DEV-1747 (§5.4), DEV-1707 (the symmetric ``reroot_aggregate_key`` this +generalises), DEV-1742 P-E. +""" +from __future__ import annotations + +from decimal import Decimal + +import pytest + +from slayer.core.keys import ( + AggregateKey, + ArithmeticKey, + BetweenKey, + ColumnKey, + ColumnSqlKey, + InKey, + LiteralKey, + ScalarCallKey, + SqlExprKey, + StarKey, + TimeTruncKey, + TransformKey, +) + +# The visitor under construction. Imported at module scope (not inside each +# test) so the whole module reports ONE clear collection error while it does +# not exist yet, rather than N identical failures. +from slayer.core.keys import reroot_value_key # noqa: E402 + +TARGET = ("customers",) +DEEP_TARGET = ("customers", "regions") + + +# --------------------------------------------------------------------------- +# Group 1 — leaf kinds +# --------------------------------------------------------------------------- +class TestLeafKinds: + """The three path-bearing leaves plus the two path-free ones.""" + + def test_column_key_exact_match_becomes_local(self) -> None: + out = reroot_value_key( + ColumnKey(path=("customers",), leaf="tier"), target_path=TARGET, + ) + assert out == ColumnKey(path=(), leaf="tier") + + def test_column_key_deeper_hop_keeps_residual(self) -> None: + out = reroot_value_key( + ColumnKey(path=("customers", "regions"), leaf="name"), target_path=TARGET, + ) + assert out == ColumnKey(path=("regions",), leaf="name") + + def test_column_key_off_path_is_unchanged(self) -> None: + """A path that does not START with target_path is left alone — the + function never invents an anchoring it cannot justify.""" + key = ColumnKey(path=("suppliers",), leaf="name") + assert reroot_value_key(key, target_path=TARGET) == key + + def test_column_key_local_is_unchanged(self) -> None: + key = ColumnKey(path=(), leaf="amount") + assert reroot_value_key(key, target_path=TARGET) == key + + def test_column_sql_key_strips_prefix(self) -> None: + out = reroot_value_key( + ColumnSqlKey(path=("customers",), model="customers", column_name="cr"), + target_path=TARGET, + ) + assert out == ColumnSqlKey(path=(), model="customers", column_name="cr") + + def test_star_key_strips_prefix(self) -> None: + out = reroot_value_key(StarKey(path=("customers",)), target_path=TARGET) + assert out == StarKey(path=()) + + def test_literal_key_is_identity(self) -> None: + key = LiteralKey(value=Decimal("1")) + assert reroot_value_key(key, target_path=TARGET) == key + + def test_time_trunc_key_reroots_its_column(self) -> None: + """``TimeTruncKey`` carries its path on the WRAPPED column, which is + why ``walk_value_keys`` needs a special case for it — the visitor must + not inherit that blind spot.""" + out = reroot_value_key( + TimeTruncKey( + column=ColumnKey(path=("customers",), leaf="signup_at"), + granularity="month", + ), + target_path=TARGET, + ) + assert out == TimeTruncKey( + column=ColumnKey(path=(), leaf="signup_at"), granularity="month", + ) + + def test_time_trunc_key_over_derived_column(self) -> None: + out = reroot_value_key( + TimeTruncKey( + column=ColumnSqlKey( + path=("customers",), model="customers", column_name="signup_d", + ), + granularity="day", + ), + target_path=TARGET, + ) + assert out.column == ColumnSqlKey( + path=(), model="customers", column_name="signup_d", + ) + + def test_sql_expr_key_strips_referenced_join_paths(self) -> None: + """§5.4 lists ``SqlExprKey`` paths explicitly. A standalone fragment + anchored at the query root must be re-anchored at the target.""" + out = reroot_value_key( + SqlExprKey( + canonical_sql="customers__regions.name = 'US'", + referenced_join_paths=(("customers",), ("customers", "regions")), + ), + target_path=TARGET, + ) + assert out.canonical_sql == "customers__regions.name = 'US'" + assert out.referenced_join_paths == ((), ("regions",)) + + +# --------------------------------------------------------------------------- +# Group 2 — composite kinds +# --------------------------------------------------------------------------- +class TestCompositeKinds: + def test_aggregate_source_args_and_kwargs(self) -> None: + out = reroot_value_key( + AggregateKey( + source=ColumnKey(path=("customers",), leaf="spend"), + agg="last", + args=(ColumnKey(path=("customers",), leaf="signup_at"),), + kwargs=(("weight", ColumnKey(path=("customers", "regions"), leaf="pop")),), + ), + target_path=TARGET, + ) + assert out.source == ColumnKey(path=(), leaf="spend") + assert out.args == (ColumnKey(path=(), leaf="signup_at"),) + assert out.kwargs == (("weight", ColumnKey(path=("regions",), leaf="pop")),) + + def test_aggregate_scalar_kwarg_passes_through(self) -> None: + out = reroot_value_key( + AggregateKey( + source=ColumnKey(path=("customers",), leaf="spend"), + agg="percentile", + kwargs=(("p", Decimal("0.5")),), + ), + target_path=TARGET, + ) + assert out.kwargs == (("p", Decimal("0.5")),) + + def test_transform_input_partition_and_time_keys(self) -> None: + out = reroot_value_key( + TransformKey( + op="cumsum", + input=AggregateKey( + source=ColumnKey(path=("customers",), leaf="spend"), agg="sum", + ), + partition_keys=frozenset({ColumnKey(path=("customers",), leaf="tier")}), + time_key=TimeTruncKey( + column=ColumnKey(path=("customers",), leaf="signup_at"), + granularity="month", + ), + ), + target_path=TARGET, + ) + assert out.input.source == ColumnKey(path=(), leaf="spend") + assert out.partition_keys == frozenset({ColumnKey(path=(), leaf="tier")}) + assert out.time_key.column == ColumnKey(path=(), leaf="signup_at") + + def test_transform_scalar_args_are_type_prohibited_from_holding_keys(self) -> None: + """``TransformKey.args``/``kwargs`` are ``Tuple[Scalar, ...]`` where + ``Scalar = Union[Decimal, str, bool, None]`` — no ValueKey can hide + there. Pinned so that widening the annotation later trips this test + and forces the visitor to grow the matching traversal.""" + anno = TransformKey.model_fields["args"].annotation + assert "ValueKey" not in str(anno), ( + f"TransformKey.args now admits {anno!r}; reroot_value_key must " + f"traverse it (§5.4 totality)." + ) + + def test_arithmetic_operands(self) -> None: + out = reroot_value_key( + ArithmeticKey( + op="+", + operands=( + ColumnKey(path=("customers",), leaf="spend"), + LiteralKey(value=Decimal("1")), + ), + ), + target_path=TARGET, + ) + assert out.operands[0] == ColumnKey(path=(), leaf="spend") + assert out.operands[1] == LiteralKey(value=Decimal("1")) + + def test_scalar_call_args(self) -> None: + out = reroot_value_key( + ScalarCallKey( + name="coalesce", + args=(ColumnKey(path=("customers",), leaf="tier"), "unknown"), + ), + target_path=TARGET, + ) + assert out.args[0] == ColumnKey(path=(), leaf="tier") + assert out.args[1] == "unknown" + + def test_between_members(self) -> None: + out = reroot_value_key( + BetweenKey( + column=ColumnKey(path=("customers",), leaf="signup_at"), + low=LiteralKey(value="2024-01-01"), + high=LiteralKey(value="2024-12-31"), + ), + target_path=TARGET, + ) + assert out.column == ColumnKey(path=(), leaf="signup_at") + assert out.low == LiteralKey(value="2024-01-01") + + def test_in_members(self) -> None: + out = reroot_value_key( + InKey( + column=ColumnKey(path=("customers",), leaf="tier"), + values=(LiteralKey(value="gold"),), + ), + target_path=TARGET, + ) + assert out.column == ColumnKey(path=(), leaf="tier") + assert out.values == (LiteralKey(value="gold"),) + assert out.negated is False + + def test_deeply_nested_mixed_composition(self) -> None: + """One tree that exercises every traversal edge at once — per-member + tests can each pass while a combination still drops a branch.""" + key = ArithmeticKey( + op="/", + operands=( + TransformKey( + op="cumsum", + input=AggregateKey( + source=ColumnKey(path=("customers", "regions"), leaf="pop"), + agg="sum", + ), + partition_keys=frozenset({ + ColumnKey(path=("customers",), leaf="tier"), + }), + ), + ScalarCallKey( + name="coalesce", + args=( + InKey( + column=ColumnKey(path=("customers",), leaf="tier"), + values=(LiteralKey(value="gold"),), + ), + BetweenKey( + column=TimeTruncKey( + column=ColumnKey( + path=("customers",), leaf="signup_at", + ), + granularity="day", + ), + low=LiteralKey(value="a"), + high=LiteralKey(value="b"), + ), + ), + ), + ), + ) + out = reroot_value_key(key, target_path=TARGET) + transform, call = out.operands + assert transform.input.source == ColumnKey(path=("regions",), leaf="pop") + assert transform.partition_keys == frozenset({ColumnKey(path=(), leaf="tier")}) + in_key, between_key = call.args + assert in_key.column == ColumnKey(path=(), leaf="tier") + assert between_key.column.column == ColumnKey(path=(), leaf="signup_at") + + +# --------------------------------------------------------------------------- +# Group 3 — totality and fail-closed +# --------------------------------------------------------------------------- +class TestTotalityAndFailClosed: + def test_every_union_member_is_handled(self) -> None: + """Enumerate the union and assert the visitor accepts each member. + + Reading the union rather than hard-coding the list means a NEW key kind + fails this test the day it is added, which is the whole point of a + total visitor. + """ + from typing import get_args + + from slayer.core.keys import ValueKey + + samples = { + ColumnKey: ColumnKey(path=("customers",), leaf="tier"), + ColumnSqlKey: ColumnSqlKey( + path=("customers",), model="customers", column_name="cr", + ), + TimeTruncKey: TimeTruncKey( + column=ColumnKey(path=("customers",), leaf="signup_at"), + granularity="day", + ), + StarKey: StarKey(path=("customers",)), + LiteralKey: LiteralKey(value="x"), + AggregateKey: AggregateKey( + source=ColumnKey(path=("customers",), leaf="spend"), agg="sum", + ), + TransformKey: TransformKey( + op="cumsum", + input=AggregateKey( + source=ColumnKey(path=("customers",), leaf="spend"), agg="sum", + ), + ), + ArithmeticKey: ArithmeticKey( + op="+", + operands=( + ColumnKey(path=("customers",), leaf="spend"), + LiteralKey(value=Decimal("1")), + ), + ), + ScalarCallKey: ScalarCallKey( + name="coalesce", args=(ColumnKey(path=("customers",), leaf="tier"),), + ), + BetweenKey: BetweenKey( + column=ColumnKey(path=("customers",), leaf="signup_at"), + low=LiteralKey(value="a"), + high=LiteralKey(value="b"), + ), + InKey: InKey( + column=ColumnKey(path=("customers",), leaf="tier"), + values=(LiteralKey(value="gold"),), + ), + } + members = set(get_args(ValueKey)) + missing = members - set(samples) + assert not missing, ( + f"ValueKey grew {sorted(m.__name__ for m in missing)}; add a sample " + f"here and a case to reroot_value_key (§5.4 totality)." + ) + for member in members: + reroot_value_key(samples[member], target_path=TARGET) + + def test_unknown_kind_raises(self) -> None: + """Fail closed. Returning an unhandled kind unchanged is what lets a + mis-anchored ref reach the generator looking correct.""" + + class NotAKey: + path = ("customers",) + + with pytest.raises(TypeError): + reroot_value_key(NotAKey(), target_path=TARGET) + + def test_empty_target_path_is_identity(self) -> None: + """``target_path == ()`` is the filtered-local case — the empty prefix + strips zero hops, so every key comes back equal.""" + key = AggregateKey( + source=ColumnKey(path=("customers",), leaf="spend"), agg="sum", + ) + assert reroot_value_key(key, target_path=()) == key + + def test_reroot_is_idempotent_at_the_fixed_point(self) -> None: + """Rerooting an ALREADY-local key again must not strip a second time — + otherwise a double-dispatch anywhere in the planner corrupts the ref.""" + once = reroot_value_key( + ColumnKey(path=("customers", "customers"), leaf="x"), target_path=TARGET, + ) + assert once == ColumnKey(path=("customers",), leaf="x") + twice = reroot_value_key(once, target_path=TARGET) + assert twice == ColumnKey(path=(), leaf="x") + + +# --------------------------------------------------------------------------- +# Group 4 — column_filter_key invariance +# --------------------------------------------------------------------------- +class TestColumnFilterKeyInvariance: + """``AggregateKey.column_filter_key`` is copied unchanged, and that is + correct rather than an oversight. + + ``binding._resolve_column_filter_key`` walks ``source.path`` FIRST and then + stamps ``anchor_model = ``, so the fragment's + ``referenced_join_paths`` are expressed relative to the model that OWNS the + filtered column. Rerooting only changes how that owner is reached from the + query root; it never moves the owner. Hence the paths are invariant. + + A standalone ``SqlExprKey`` (not attached to an aggregate) is a different + animal — it can be anchored at the query root, so it DOES strip. Both + directions are pinned so a future "simplification" that reroutes + ``column_filter_key`` through the stripping case fails here. + """ + + def _filtered_agg(self) -> AggregateKey: + return AggregateKey( + source=ColumnKey(path=("customers",), leaf="spend"), + agg="sum", + column_filter_key=SqlExprKey( + canonical_sql="regions.name = 'US'", + referenced_join_paths=(("regions",),), + ), + ) + + def test_column_filter_key_survives_reroot_unchanged(self) -> None: + key = self._filtered_agg() + out = reroot_value_key(key, target_path=TARGET) + assert out.source == ColumnKey(path=(), leaf="spend") + assert out.column_filter_key == key.column_filter_key + + def test_column_filter_key_unchanged_even_when_paths_share_the_prefix(self) -> None: + """The adversarial case: the fragment's own paths LOOK strippable. + They must still not be stripped — they are owner-relative, and a strip + here would silently re-anchor the filter one hop too shallow.""" + key = AggregateKey( + source=ColumnKey(path=("customers",), leaf="spend"), + agg="sum", + column_filter_key=SqlExprKey( + canonical_sql="customers.tier = 'gold'", + referenced_join_paths=(("customers",),), + ), + ) + out = reroot_value_key(key, target_path=TARGET) + assert out.column_filter_key.referenced_join_paths == (("customers",),) + + def test_standalone_sql_expr_key_does_strip(self) -> None: + """The contrasting direction — proves the invariance above is a + deliberate per-position rule, not a missing traversal.""" + out = reroot_value_key( + SqlExprKey( + canonical_sql="x", referenced_join_paths=(("customers", "regions"),), + ), + target_path=TARGET, + ) + assert out.referenced_join_paths == (("regions",),) + + def test_multi_hop_target_keeps_owner_relative_filter(self) -> None: + key = AggregateKey( + source=ColumnKey(path=("customers", "regions"), leaf="pop"), + agg="sum", + column_filter_key=SqlExprKey( + canonical_sql="active = 1", referenced_join_paths=(), + ), + ) + out = reroot_value_key(key, target_path=DEEP_TARGET) + assert out.source == ColumnKey(path=(), leaf="pop") + assert out.column_filter_key == key.column_filter_key + + +# --------------------------------------------------------------------------- +# Group 5 — the public-identity invariant §5.4 names explicitly +# --------------------------------------------------------------------------- +class TestPublicResultKeysUnchanged: + def test_reroot_aggregate_key_delegates_to_the_visitor(self) -> None: + """``reroot_aggregate_key`` stays (P-J state 1) but must become a thin + wrapper, so the two cannot drift into two reroot semantics — which is + precisely the drift §5.4 exists to end.""" + from slayer.core.keys import reroot_aggregate_key + + key = AggregateKey( + source=ColumnKey(path=("customers",), leaf="spend"), + agg="last", + args=(ColumnKey(path=("customers", "regions"), leaf="opened_at"),), + ) + assert reroot_aggregate_key(key, target_path=TARGET) == reroot_value_key( + key, target_path=TARGET, + ) + + def test_agg_and_column_filter_fields_ride_through(self) -> None: + """Fields the visitor does not own must survive — a rebuild that + enumerated only the rerootable fields would silently drop them.""" + key = AggregateKey( + source=ColumnKey(path=("customers",), leaf="spend"), + agg="approx_count_distinct", + column_filter_key=SqlExprKey(canonical_sql="a = 1"), + ) + out = reroot_value_key(key, target_path=TARGET) + assert out.agg == "approx_count_distinct" + assert out.column_filter_key is not None + + def test_kwargs_stay_canonically_sorted_after_reroot(self) -> None: + key = AggregateKey( + source=ColumnKey(path=("customers",), leaf="spend"), + agg="corr", + kwargs=( + ("other", ColumnKey(path=("customers",), leaf="x")), + ("alpha", Decimal("1")), + ), + ) + out = reroot_value_key(key, target_path=TARGET) + assert [k for k, _ in out.kwargs] == sorted(k for k, _ in out.kwargs) From e5eaa63d758ee34261f4dfc0913dd6f028b34d89 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Thu, 6 Aug 2026 17:11:05 +0200 Subject: [PATCH 56/98] DEV-1747: the pre-bound seam, host-grain ordering, and B6 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four pieces, in dependency order. The first is load-bearing for the rest. D1 -- the pre-bound planner seam (§5.4, P-E) plan_query was the only door into binding, so re-rooting -- which already held the typed keys it needed -- serialized them back to formula text and let the planner re-derive the identities it had just discarded. That is the P-E violation, and it was not merely inelegant: formula text cannot express a path-bearing source or a grain marker, so shapes that were expressible in keys were unreachable through the string. bind_query_inputs now produces a PreboundQuery; plan_query(prebound=) consumes one and skips the parser. Re-rooting re-anchors keys structurally via reroot_value_key and hands them straight back. The proof is negative and mechanical: cross_model_planner no longer imports bind_expr, bind_filter, bind_time_dimension, parse_expr, parse_filter_expr, ModelMeasure, SlayerQuery, ColumnRef, TimeDimension or ModelScope. Ruff removed them because nothing referenced them any more. StrictQueryCarrier closes the other half. plan_query reads a few query-level scalars after binding; a pre-bound caller that forgot one would silently inherit a Pydantic default. The carrier approves exactly source_model and name, and raises on everything else. D2 -- host-grain ordering AggregateKey gains grain: "target" | "host", in hash and equality, with a _host alias suffix in the naming authority (without it the synthetic wrap and a declared `customers.regions.name:min` intern as distinct slots but collide on one output column name). The marker separates where a value is READ from where it is GROUPED. A path-bearing source used to route unconditionally to a target-rooted CTE, which for a sort key degenerates to a scalar CROSS JOIN -- every group gets the same value and the sort silently does nothing. That case was rejected outright rather than sorted wrongly. It now routes host-rooted: the crossed join is pulled INSIDE the CTE, which groups on the query grain, and the host base joins the per-group extreme back. The base itself never joins the sort key's table, so a sibling SUM is not multiplied by the fan-out. This is why the DEV-1645 and DEV-1712 rejection tests are rewritten rather than deleted. Their three xfail(strict) companions aspired to a bare `ORDER BY customers__regions.name`, which in a GROUPED query is invalid SQL on every Tier-1 dialect -- the column is not in GROUP BY. The aspiration was wrong; the shape here is what the value actually requires. D10 -- the wrap is direction-aware ASC takes each group's MIN, DESC its MAX. An unconditional MAX sorts ASC by each group's LARGEST member, which is not what was asked for whenever groups overlap in range. The remap is re-keyed by (value_key, direction), so `ORDER BY a ASC, a DESC` mints two slots instead of collapsing onto whichever interned first. B6 / D6 -- no silent filter drops The reroot classified filters against the forward path, then re-decided and BLANKED all four routing lists. Reachable, host-local and unreachable filters therefore looked identical on the plan, and an unreachable one narrowed the user's result with no warning at all. The decision now precedes the classification, so the one classification runs in the coordinate system of the CTE that will actually exist, and its result is consumed rather than overwritten. Rerooted reachability is decided by walking the target's join graph, not by a prefix test: a host-side sibling branch is reachable whenever the target joins to it too, which is ordinary in a star schema. Getting that wrong in either direction is a correctness bug -- too narrow drops a filter the user wrote, too wide emits SQL naming an unbound table. The swallow-and-drop path is gone with it. _REROOT_BIND_ERRORS included bare ValueError, so any planner bug inside the reroot read as "filter unreachable". Superseded but retained (P-J state 1): _reroot_ref, _host_ref_path, _render_ref_formula, _scalar_formula_literal, _local_agg_formula, _classify_subplan_filters and _REROOT_BIND_ERRORS are now production-unreferenced. Their tests stay green so the two mechanisms can be compared; deletion is one sweep in PR 6, not smeared across the series. 11,188 pre-existing tests pass, zero non-DEV-1747 failures, ruff clean. DEV-1747 is at 173/194 -- §5.10, D8 and D9 are still to come. Co-Authored-By: Claude Opus 5 (1M context) --- slayer/core/keys.py | 224 +++-- slayer/engine/cross_model_planner.py | 836 +++++++++++++----- slayer/engine/filter_reachability.py | 28 +- slayer/engine/planned.py | 44 +- slayer/engine/stage_planner.py | 500 ++++++----- slayer/sql/dialects/base.py | 29 + slayer/sql/dialects/tsql.py | 27 + slayer/sql/generator.py | 112 ++- slayer/sql/naming.py | 8 + tests/test_dev1645_invalid_postgres_sql.py | 152 ++-- tests/test_dev1712_order_only_hidden_slots.py | 76 +- ..._dev1733_order_only_transform_composite.py | 26 +- tests/test_nested_dag_cross_stage_refs.py | 10 +- tests/test_planned.py | 38 +- tests/test_sql_generator.py | 8 +- 15 files changed, 1481 insertions(+), 637 deletions(-) diff --git a/slayer/core/keys.py b/slayer/core/keys.py index 91269d09..f611154b 100644 --- a/slayer/core/keys.py +++ b/slayer/core/keys.py @@ -24,7 +24,7 @@ from decimal import Decimal from enum import IntEnum -from typing import Optional, Tuple, Union +from typing import Literal, Optional, Tuple, TypeVar, Union, cast from pydantic import BaseModel, ConfigDict, field_validator @@ -470,6 +470,20 @@ class AggregateKey(_FrozenKey): ``column_filter_key`` is the ``Column.filter`` attached to the aggregated column, if any — pulled into the structural key so two aggregates with different attached filters do not collide. + + ``grain`` names WHERE the aggregate is evaluated when ``source.path`` is + non-empty (DEV-1747 D2). ``"target"`` — the default and the meaning of + every user-declared cross-model aggregate — evaluates it in a CTE rooted at + the target, one value per target row-group. ``"host"`` evaluates it in a + CTE rooted at the HOST with the path's joins pulled in, grouped on the + query grain: one value per HOST group. The DEV-1735 order wrap needs the + latter, because a target-rooted CTE for a host-grain sort key degenerates + to a scalar CROSS JOIN and sorts every group by one constant. + + It participates in identity deliberately: a declared + ``customers.regions.name:max`` and the synthetic host-grain wrap over the + same column are different values (global vs per-group), so interning them + onto one slot would silently give the user the wrong one. """ source: _AggregateSource @@ -477,6 +491,7 @@ class AggregateKey(_FrozenKey): args: Tuple[_AggregateArgValue, ...] = () kwargs: Tuple[Tuple[str, _AggregateKwargValue], ...] = () column_filter_key: Optional[SqlExprKey] = None + grain: Literal["target", "host"] = "target" @field_validator("kwargs", mode="before") @classmethod @@ -495,6 +510,7 @@ def __hash__(self) -> int: _typed_args(self.args), _typed_kwargs(self.kwargs), self.column_filter_key, + self.grain, )) def __eq__(self, other: object) -> bool: @@ -506,9 +522,16 @@ def __eq__(self, other: object) -> bool: and _typed_args(self.args) == _typed_args(other.args) and _typed_kwargs(self.kwargs) == _typed_kwargs(other.kwargs) and self.column_filter_key == other.column_filter_key + and self.grain == other.grain ) +#: Rerooting is type-preserving — a ``ColumnKey`` in, a ``ColumnKey`` out. +#: Expressing that keeps call sites precisely typed rather than collapsing +#: every rerooted key to the union. +_RerootableT = TypeVar("_RerootableT") + + def _reroot_path_ref(ref, *, target_path: Tuple[str, ...]): """Re-anchor a single embedded reference from the host coordinate system into the target's local scope (DEV-1707). @@ -538,68 +561,20 @@ def _reroot_path_ref(ref, *, target_path: Tuple[str, ...]): def reroot_aggregate_key( key: "AggregateKey", *, target_path: Tuple[str, ...], ) -> "AggregateKey": - """Re-anchor EVERY embedded reference of a cross-model ``AggregateKey`` - from the host's coordinate system into its target's local scope + """Re-anchor a cross-model ``AggregateKey`` into its target's local scope (DEV-1707 / DEV-1703 Stage 3). - When a cross-model aggregate (``customers.revenue:sum``, - ``customers.amount:last(customers.signup_at)``) is rendered inside its - target-rooted CTE, its ``source``, positional ``args``, keyword ``kwargs`` - values, and — invariantly — its ``column_filter_key`` must all be - expressed relative to the target rather than the query root. This is the - single, symmetric replacement for the per-field strip logic that used to - live in ``slayer/engine/cross_model_planner.py`` (``_local_agg_formula`` / - ``_reroot_col_kwarg``) and ``slayer/sql/generator.py`` (the inline - ``_reroot_kwarg`` / ``local_args`` / ``_reroot_having`` blocks) with two - divergent semantics. - - ``target_path`` is the join path of the aggregate's source (the hops from - the query root to the target model). Semantics — prefix-strip with - residual, applied uniformly (see ``_reroot_path_ref``): - - * ``source``, each positional arg, and each kwarg VALUE whose ``path`` - starts with ``target_path`` drops that prefix (exact match → local); - * a ref whose ``path`` does not start with ``target_path`` is left - unchanged — the function is TOTAL and never raises; a genuinely - mis-pathed ref surfaces at the downstream binder / kwarg-path validator - exactly as before; - * scalar args / kwargs pass through untouched; kwarg NAMES are preserved - (and the ``AggregateKey`` validator keeps them canonically sorted); - * ``target_path == ()`` is the identity (the filtered-local case, where - the source is already host-local — the empty prefix strips zero hops). - - ``column_filter_key`` is copied UNCHANGED. Its ``canonical_sql`` and - ``referenced_join_paths`` are anchored at the OWNING MODEL of the source - column (stamped by ``slayer.engine.binding._resolve_column_filter_key`` - via ``compute_column_filter_join_paths`` with ``anchor_model`` = that - owning model). Rerooting only changes how that owner is REACHED from the - query root; it never moves the owner, so the filter's owner-relative SQL - and paths are invariant under reroot. After rerooting a filtered - cross-model aggregate the source reads local (``path == ()``) while - ``referenced_join_paths`` stays non-empty — precisely the DEV-1503 - filtered-local isolation trigger shape. + A thin alias for :func:`reroot_value_key`, which applies the same + prefix-strip rule over the whole ``ValueKey`` union. Two implementations + would be free to drift into two reroot semantics — the drift §5.4 removes. + + ``column_filter_key`` rides through unchanged: its paths are anchored at + the OWNING model of the source column, and rerooting changes only how that + owner is reached. After rerooting a filtered cross-model aggregate the + source reads local while ``referenced_join_paths`` stays non-empty — + exactly the filtered-local isolation trigger shape. """ - target_path = tuple(target_path) - if not target_path: - return key - # Rebuild from the existing key so fields reroot does NOT own (``agg``, - # ``column_filter_key``, and any field added to ``AggregateKey`` later) - # ride through automatically rather than being silently dropped. Only - # ``source`` / ``args`` / ``kwargs`` carry rerootable paths. ``model_copy`` - # skips the ``_canonicalize_kwargs`` validator, which is a no-op here: - # reroot preserves kwarg names and order, so the input's already-canonical - # sort is unchanged (pinned by - # ``test_kwargs_canonical_sort_preserved_after_reroot``). - return key.model_copy(update={ - "source": _reroot_path_ref(key.source, target_path=target_path), - "args": tuple( - _reroot_path_ref(a, target_path=target_path) for a in key.args - ), - "kwargs": tuple( - (k, _reroot_path_ref(v, target_path=target_path)) - for k, v in key.kwargs - ), - }) + return reroot_value_key(key, target_path=target_path) class TransformKey(_FrozenKey): @@ -820,3 +795,132 @@ def phase(self) -> Phase: InKey.model_rebuild() # TimeTruncKey.column is a Union[ColumnKey, ColumnSqlKey] (DEV-1450 #4a). TimeTruncKey.model_rebuild() + + +# --------------------------------------------------------------------------- +# The total reroot visitor +# --------------------------------------------------------------------------- + + +def _reroot_sql_expr_key( + key: SqlExprKey, *, target_path: Tuple[str, ...], +) -> SqlExprKey: + """Re-anchor a STANDALONE Mode-A fragment's referenced join paths. + + Only correct when the fragment is anchored at the QUERY ROOT. A fragment + reached as ``AggregateKey.column_filter_key`` is anchored at the owning + model instead and must NOT come through here — see the note in + :func:`reroot_value_key`. + """ + return key.model_copy(update={ + "referenced_join_paths": tuple( + path[len(target_path):] + if tuple(path[: len(target_path)]) == target_path else path + for path in key.referenced_join_paths + ), + }) + + +def reroot_value_key( + key: _RerootableT, *, target_path: Tuple[str, ...], +) -> _RerootableT: + """Re-anchor every embedded reference in ``key`` from the query root into + ``target_path``'s local scope. + + The generalisation of :func:`reroot_aggregate_key` over the whole + ``ValueKey`` union (plus the standalone ``SqlExprKey``). Two properties + make it safe to reroot a plan structurally instead of via formula text: + + **Total.** Every union member has an explicit case. A kind added to + ``ValueKey`` later has none, so it lands in the fail-closed arm rather than + riding through unrerooted. + + **Fail-closed.** An unhandled kind raises ``TypeError``. Returning it + unchanged would be indistinguishable from "correctly identity", which is + how a mis-anchored ref reaches the SQL generator looking well-formed. + + The rule is prefix-strip-with-residual, applied per position: a ``path`` + starting with ``target_path`` drops that prefix and keeps the residual + hops; any other ``path``, and any scalar, is returned unchanged. + ``target_path == ()`` is the identity — the empty prefix strips zero hops. + + ``AggregateKey.column_filter_key`` is deliberately copied UNCHANGED. + ``binding._resolve_column_filter_key`` walks ``source.path`` first and only + then stamps the anchor, so the fragment's paths are expressed relative to + the model that OWNS the filtered column. Rerooting changes how that owner + is reached from the query root; it never moves the owner, so those paths + are invariant. A standalone ``SqlExprKey`` is anchored at the query root + and therefore does strip — the asymmetry is per position, not per type. + """ + target_path = tuple(target_path) + if not target_path: + return key + + def _recurse(value): + return reroot_value_key(value, target_path=target_path) + + # Scalars ride through untouched — they appear as ScalarCallKey args and + # as AggregateKey kwarg values. + if key is None or isinstance(key, (Decimal, str, bool, int, float)): + return key + + # --- leaves --------------------------------------------------------- + if isinstance(key, (ColumnKey, ColumnSqlKey, StarKey)): + # ``_reroot_path_ref`` also accepts bare scalars, so it cannot carry the + # type-preserving annotation; the isinstance guard above establishes it. + return cast(_RerootableT, _reroot_path_ref(key, target_path=target_path)) + if isinstance(key, LiteralKey): + return key + if isinstance(key, TimeTruncKey): + # The path lives on the WRAPPED column, which is why walk_value_keys + # needs a special case here; the visitor must not inherit that blind + # spot. + return key.model_copy(update={"column": _recurse(key.column)}) + if isinstance(key, SqlExprKey): + return _reroot_sql_expr_key(key, target_path=target_path) + + # --- composites ----------------------------------------------------- + if isinstance(key, AggregateKey): + return key.model_copy(update={ + "source": _recurse(key.source), + "args": tuple(_recurse(a) for a in key.args), + "kwargs": tuple((n, _recurse(v)) for n, v in key.kwargs), + }) + if isinstance(key, TransformKey): + # ``args`` / ``kwargs`` are Tuple[Scalar, ...] — type-prohibited from + # holding a ValueKey, so there is nothing to traverse there. + return key.model_copy(update={ + "input": _recurse(key.input), + "partition_keys": frozenset( + _recurse(p) for p in key.partition_keys + ), + "time_key": ( + None if key.time_key is None else _recurse(key.time_key) + ), + }) + if isinstance(key, ArithmeticKey): + return key.model_copy(update={ + "operands": tuple(_recurse(o) for o in key.operands), + }) + if isinstance(key, ScalarCallKey): + return key.model_copy(update={ + "args": tuple(_recurse(a) for a in key.args), + }) + if isinstance(key, BetweenKey): + return key.model_copy(update={ + "column": _recurse(key.column), + "low": _recurse(key.low), + "high": _recurse(key.high), + }) + if isinstance(key, InKey): + return key.model_copy(update={ + "column": _recurse(key.column), + "values": tuple(_recurse(v) for v in key.values), + }) + + raise TypeError( + f"reroot_value_key has no case for {type(key).__name__!r}. The visitor " + f"is total over ValueKey by design: add an explicit case rather than " + f"letting an unrerooted key through, which the SQL generator cannot " + f"distinguish from a correctly-local one." + ) diff --git a/slayer/engine/cross_model_planner.py b/slayer/engine/cross_model_planner.py index b4e557e7..cf65b816 100644 --- a/slayer/engine/cross_model_planner.py +++ b/slayer/engine/cross_model_planner.py @@ -64,18 +64,17 @@ ValueKey, column_path, reroot_aggregate_key, + reroot_value_key, ) -from slayer.core.models import ModelMeasure, SlayerModel -from slayer.core.query import ColumnRef, SlayerQuery, TimeDimension +from slayer.core.models import SlayerModel from slayer.sql.naming import canonical_aggregate_alias -from slayer.core.scope import ModelScope, StageColumn, StageSchema +from slayer.core.scope import StageColumn, StageSchema from slayer.engine.aggregate_input_paths import ( compute_aggregate_input_join_paths, ) from slayer.engine.binding import ( - bind_expr, - bind_filter, - bind_time_dimension, + BoundExpr, + BoundFilter, walk_value_keys, ) from slayer.engine.filter_reachability import path_is_reachable @@ -87,8 +86,16 @@ SlotId, ValueSlot, ) +from slayer.engine.planning import DeclaredMeasure, _canonical_name +from slayer.engine.prebound import ( + PreboundQuery, + StrictQueryCarrier, + dimension_key_metadata, + measure_key_format_description, + measure_key_type, + walk_key_path, +) from slayer.engine.source_bundle import ResolvedSourceBundle -from slayer.engine.syntax import parse_expr, parse_filter_expr # --------------------------------------------------------------------------- @@ -120,6 +127,11 @@ class HostFilterRouting(BaseModel): phase: Phase referenced_slot_ids: List[SlotId] = Field(default_factory=list) text: Optional[str] = None + # §5.4 — the typed predicate behind ``text``. A sub-plan that inherits a + # host filter re-roots THIS rather than re-parsing the string, so the + # inherited predicate keeps its structural identity. Optional because + # direct callers and test doubles build routings without one. + bound: Optional[BoundFilter] = None # DEV-1745 (W4 / D9) — the filter's structural reachability summary, in the # host plan's coordinate system. ``crossed_join_paths`` is every join path # its dependency tree is anchored at; ``has_host_local_ref`` marks a @@ -171,6 +183,7 @@ def classify_host_filter( host_slots: List[ValueSlot], target_path: Tuple[str, ...], host_model_name: Optional[str] = None, + reachable_paths: "Optional[frozenset]" = None, ) -> FilterRoute: """Classify one host filter for cross-model CTE propagation. @@ -208,7 +221,9 @@ def classify_host_filter( # be a second copy free to drift from it — the exact failure this PR removes. unreachable_paths = [ p for p in crossed - if not path_is_reachable(path=p, target_path=target_path) + if not path_is_reachable( + path=p, target_path=target_path, reachable_paths=reachable_paths, + ) ] if unknown or aggregate_other or unreachable_paths: @@ -253,10 +268,10 @@ def plan( host_filters: List[HostFilterRouting], public_alias: Optional[str] = None, hidden: bool = False, - host_query: Optional[SlayerQuery] = None, + host_query: Optional[StrictQueryCarrier] = None, public_projection: Optional[List[SlotId]] = None, subplan_builder: Optional[ - Callable[[SlayerQuery, ResolvedSourceBundle], PlannedQuery] + Callable[[StrictQueryCarrier, ResolvedSourceBundle], PlannedQuery] ] = None, ) -> CrossModelAggregatePlan: ... @@ -430,15 +445,20 @@ def _match_filtered_local_grain_pairs( def _find_filtered_local_sub_agg_slot( *, sub_plan: PlannedQuery, - formula: str, + aggregate_key: AggregateKey, host_model: SlayerModel, ) -> SlotId: - """Locate the sub-plan's single local aggregate slot. + """Locate the sub-plan's slot for the isolated aggregate. - Recursion suppression guarantees no nested cross-model plans so the - sub-plan has exactly one local aggregate — the filtered measure being - isolated. + The exact key match comes first (§5.4: the sub-plan is planned FROM this + key, so it interns under the same identity). The path-less fallback covers + a sub-plan whose own pass rewrote the key — recursion suppression + guarantees no nested cross-model plans, so at most one local aggregate is + present to match. """ + for s in sub_plan.aggregate_slots: + if s.key == aggregate_key: + return s.id for s in sub_plan.aggregate_slots: if isinstance(s.key, AggregateKey) and not getattr( s.key.source, "path", (), @@ -446,7 +466,7 @@ def _find_filtered_local_sub_agg_slot( return s.id raise ValueError( "DEV-1503 sub-plan produced no local aggregate slot for " - f"{formula!r} on {host_model.name!r} — planner bug." + f"{aggregate_key!r} on {host_model.name!r} — planner bug." ) @@ -484,10 +504,44 @@ def _build_filtered_local_cte_schema( ) +def _route_host_rooted_filters( + *, host_filters: List[HostFilterRouting], +) -> _FilterRoutes: + """Route host filters for a HOST-ROOTED CTE (DEV-1503 / DEV-1747 D6). + + The re-rooted table's question is reachability — can a CTE rooted at the + TARGET evaluate this predicate? A host-rooted CTE is rooted at the host, so + reachability is never in doubt and the question is PHASE instead: + + * ROW — propagate. The sub-plan applies it to the aggregate's rowset; + without it a predicate like ``status = 'active'`` would not affect the + value that joins back. + * AGGREGATE — do NOT propagate. As a HAVING inside the CTE it would drop + CTE rows where the aggregate fails, and the outer LEFT JOIN would then + surface the host row with a NULL aggregate instead of dropping it. The + generator's outer-WHERE wrapper applies it on the joined-back column, so + the row is actually dropped. + * POST — do NOT propagate; it stays at the host's post-transform wrapper. + + Nothing is ever DROPPED here, so no warning can arise: every filter is + applied somewhere, either in the CTE or at the host. + """ + where_ids: List[BoundFilterId] = [] + for routing in host_filters: + if routing.text is None: + continue # date_range bound — re-attached by the caller, in order + if routing.phase in (Phase.POST, Phase.AGGREGATE): + continue + if routing.bound is None: + continue + where_ids.append(routing.filter_id) + return _FilterRoutes(applied=list(where_ids), where_ids=where_ids) + + def _classify_subplan_filters( *, host_filters: List[HostFilterRouting], -) -> Optional[List[str]]: +) -> List[BoundFilter]: """Decide which host-query filters propagate into the DEV-1503 sub-plan. ROW: pass through — the sub-plan applies them to the aggregate's rowset @@ -510,17 +564,38 @@ def _classify_subplan_filters( ``host_query.filters`` here would mis-pair phases when a ``date_range``- bearing time_dimension is present (Codex review) OR when dedup drops user-filter entries. + + §5.4 — the TYPED ``routing.bound`` rides into the sub-plan; ``routing.text`` + now only distinguishes a user filter from a synthesized date-range bound + (which the caller re-attaches itself, in date-range-first order). """ - sub_filter_texts: List[str] = [] + inherited: List[BoundFilter] = [] for routing in host_filters: if routing.text is None: # date_range bound — not a user filter, do not propagate. continue if routing.phase in (Phase.POST, Phase.AGGREGATE): continue + if routing.bound is None: + continue # ROW phase — propagate. - sub_filter_texts.append(routing.text) - return sub_filter_texts or None + inherited.append(routing.bound) + return inherited + + +class _FilterRoutes(BaseModel): + """The routing decision for one CTE's whole host-filter set. + + Grouped into a record so it can be produced once and threaded to the plan + constructor without four positional lists (DEV-1747 D6). + """ + + model_config = ConfigDict(arbitrary_types_allowed=True) + + applied: List[BoundFilterId] = Field(default_factory=list) + where_ids: List[BoundFilterId] = Field(default_factory=list) + having_ids: List[BoundFilterId] = Field(default_factory=list) + dropped: List[UnreachableFilterDroppedWarning] = Field(default_factory=list) def _route_host_filters( @@ -530,15 +605,18 @@ def _route_host_filters( target_path: Tuple[str, ...], host_model: SlayerModel, terminal_model: SlayerModel, -) -> Tuple[ - List[BoundFilterId], List[BoundFilterId], List[BoundFilterId], - List[UnreachableFilterDroppedWarning], -]: + reachable_paths: "Optional[frozenset]" = None, +) -> _FilterRoutes: """Classify each host filter via the ``inherited_filter_policy`` decision - table (``classify_host_filter``) into ``(applied, where_ids, having_ids, - dropped)`` — extracted from ``IsolatedCteCrossModelPlanner.plan`` (DEV-1708) - to keep that method focused. ``DROP_HOST_LOCAL`` / ``STAY_AT_HOST_POST`` are - neither propagated nor warned.""" + table (``classify_host_filter``) — extracted from + ``IsolatedCteCrossModelPlanner.plan`` (DEV-1708) to keep that method + focused. ``DROP_HOST_LOCAL`` / ``STAY_AT_HOST_POST`` are neither propagated + nor warned. + + ``reachable_paths``, when supplied, is the set of host-coordinate join + paths a RE-ROOTED CTE can actually evaluate — walked from the target's own + join graph by the caller. It replaces the forward-path prefix test, so a + filter the re-rooted CTE will genuinely apply is not reported as dropped.""" applied: List[BoundFilterId] = [] where_ids: List[BoundFilterId] = [] having_ids: List[BoundFilterId] = [] @@ -549,6 +627,7 @@ def _route_host_filters( host_slots=host_slots, target_path=target_path, host_model_name=host_model.name, + reachable_paths=reachable_paths, ) if route is FilterRoute.PROPAGATE_WHERE: where_ids.append(hf.filter_id) @@ -570,7 +649,10 @@ def _route_host_filters( f"it still applies at the host, and is dropped from the CTE." ), )) - return applied, where_ids, having_ids, dropped + return _FilterRoutes( + applied=applied, where_ids=where_ids, + having_ids=having_ids, dropped=dropped, + ) def _compute_shared_grain_slots( @@ -624,10 +706,10 @@ def plan( host_filters: List[HostFilterRouting], public_alias: Optional[str] = None, hidden: bool = False, - host_query: Optional[SlayerQuery] = None, + host_query: Optional[StrictQueryCarrier] = None, public_projection: Optional[List[SlotId]] = None, subplan_builder: Optional[ - Callable[[SlayerQuery, ResolvedSourceBundle], PlannedQuery] + Callable[[StrictQueryCarrier, ResolvedSourceBundle], PlannedQuery] ] = None, ) -> CrossModelAggregatePlan: host_model = bundle.source_model @@ -640,7 +722,12 @@ def plan( agg_source = aggregate_key.source path = getattr(agg_source, "path", ()) - if not path: + # DEV-1747 D2 — ``grain="host"`` routes to the HOST-rooted CTE even + # though the source carries a path. The path says WHERE the value is + # read from; the grain says WHERE it is grouped. A joined ORDER BY wrap + # reads through the join but must be grouped per HOST row-group, so it + # belongs on the same route as a crossing-input local aggregate. + if not path or getattr(aggregate_key, "grain", "target") == "host": return self._dispatch_filtered_local( aggregate_slot_id=aggregate_slot_id, aggregate_key=aggregate_key, @@ -673,14 +760,6 @@ def plan( )) target_path = path - applied, where_ids, having_ids, dropped = _route_host_filters( - host_filters=host_filters, - host_slots=host_slots, - target_path=target_path, - host_model=host_model, - terminal_model=terminal_model, - ) - target_model_filters = list(terminal_model.filters or []) # Shared grain: host ROW dimensions / time-dimensions on the target's @@ -702,38 +781,55 @@ def plan( join_back_pairs=join_back_pairs, ) - forward_plan = CrossModelAggregatePlan( - aggregate_slot_id=aggregate_slot_id, - target_model=terminal_model.name, - datasource=host_model.data_source, - join_chain=join_chain, - join_back_pairs=join_back_pairs, - cte_stage_schema=cte_schema, - shared_grain_slots=shared_grain, - applied_filter_ids=applied, - where_filter_ids=where_ids, - having_filter_ids=having_ids, - target_model_filters=target_model_filters, - dropped_filter_warnings=dropped, - hidden=hidden, - public_alias=public_alias, - ) + def _make_plan(routes: "_FilterRoutes") -> CrossModelAggregatePlan: + return CrossModelAggregatePlan( + aggregate_slot_id=aggregate_slot_id, + target_model=terminal_model.name, + datasource=host_model.data_source, + join_chain=join_chain, + join_back_pairs=join_back_pairs, + cte_stage_schema=cte_schema, + shared_grain_slots=shared_grain, + applied_filter_ids=routes.applied, + where_filter_ids=routes.where_ids, + having_filter_ids=routes.having_ids, + target_model_filters=target_model_filters, + dropped_filter_warnings=routes.dropped, + hidden=hidden, + public_alias=public_alias, + ) # DEV-1450 #2: re-rooting is the strategy's call. When the caller # supplies the host query + a sub-plan builder, decide forward-plan # vs re-rooted-plan here; without them (direct ``plan(...)`` callers / # test doubles) return the forward plan unchanged. + # + # DEV-1747 D6 — the reroot decision is made BEFORE the filters are + # classified, so the one classification runs in the coordinate system + # of the CTE that will actually exist. Classifying against the forward + # path and then re-rooting is how a reachable filter ended up judged + # unreachable, and then had that judgement blanked. if subplan_builder is not None and host_query is not None: return _maybe_reroot_cross_model_plan( - plan=forward_plan, + make_plan=_make_plan, query=host_query, agg_key=aggregate_key, bundle=bundle, host_model=host_model, + host_slots=host_slots, + host_filters=host_filters, public_projection=public_projection or [], subplan_builder=subplan_builder, + target_model_name=terminal_model.name, + target_path=target_path, ) - return forward_plan + return _make_plan(_route_host_filters( + host_filters=host_filters, + host_slots=host_slots, + target_path=target_path, + host_model=host_model, + terminal_model=terminal_model, + )) # ---------------------------------------------------------------------- # DEV-1503 — filtered-local isolation @@ -750,10 +846,10 @@ def _dispatch_filtered_local( host_filters: List[HostFilterRouting], public_alias: Optional[str], hidden: bool, - host_query: Optional[SlayerQuery], + host_query: Optional[StrictQueryCarrier], public_projection: Optional[List[SlotId]], subplan_builder: Optional[ - Callable[[SlayerQuery, ResolvedSourceBundle], PlannedQuery] + Callable[[StrictQueryCarrier, ResolvedSourceBundle], PlannedQuery] ], ) -> CrossModelAggregatePlan: """Validate the host-rooted trigger preconditions and dispatch @@ -768,7 +864,16 @@ def _dispatch_filtered_local( has_crossing_filter = cfk is not None and bool( cfk.referenced_join_paths, ) - has_crossing_input = has_crossing_filter or bool( + # DEV-1747 D2 — for a ``grain="host"`` aggregate the SOURCE PATH is + # itself the crossing input: the value is read through a join that the + # CTE has to pull in. Omitting it here would make the check below + # reject the wrap as "a plain local aggregate", since a path-bearing + # source carries no ``column_filter_key`` and no crossing arg. + has_crossing_source = ( + getattr(aggregate_key, "grain", "target") == "host" + and bool(getattr(agg_source, "path", ())) + ) + has_crossing_input = has_crossing_filter or has_crossing_source or bool( compute_aggregate_input_join_paths( key=aggregate_key, anchor_model=host_model, @@ -817,49 +922,67 @@ def _plan_filtered_local( host_model: SlayerModel, host_slots: List[ValueSlot], host_filters: List[HostFilterRouting], - host_query: SlayerQuery, + host_query: StrictQueryCarrier, public_alias: Optional[str], public_projection: List[SlotId], hidden: bool, subplan_builder: Callable[ - [SlayerQuery, ResolvedSourceBundle], PlannedQuery, + [StrictQueryCarrier, ResolvedSourceBundle], PlannedQuery, ], ) -> CrossModelAggregatePlan: """Build a host-rooted nested sub-plan for a cross-model-FILTERED local measure (DEV-1503). - The sub-plan is a ``SlayerQuery`` rooted at the SAME host model with - ``measures=[]`` and the host's dimensions / - time_dimensions. The sub-plan's ``plan_query`` recursion handles - the filter-target join (its ``Column.filter`` will pull in the - joined table at the generator's inline path), the host model's own + The sub-plan is a ``PreboundQuery`` rooted at the SAME host model, + carrying the filtered measure and the host's bound dimensions / + time dimensions. The sub-plan's ``plan_query`` recursion handles the + filter-target join (its ``Column.filter`` pulls in the joined table at + the generator's inline path), the host model's own ``SlayerModel.filters``, and the per-dimension GROUP BY — producing a per-grain aggregate that the host base LEFT JOINs back. - Host query filters are NOT propagated into the sub-plan here — the - host base CTE applies them. The generator's outer-WHERE wrapper - handles aggregate-referencing filters separately (DEV-1503 spec). + Only ROW-phase host filters propagate (see + ``_classify_subplan_filters``); the rest stay at the host base or at + the generator's outer-WHERE wrapper (DEV-1503 spec). """ - # Reconstruct the local measure formula from the AggregateKey. The - # source.path is empty so ``_local_agg_formula`` emits a bare - # ``leaf:agg`` shape (plus any args / kwargs). Carry the user- - # supplied alias through so a host filter referencing the rename - # (``latest_pmt > 500`` for a measure named ``latest_pmt``) binds - # against the same alias in the sub-plan rather than the canonical - # ``latest_payment_last_updated_at`` form. - formula = _local_agg_formula(aggregate_key) - measure_name_for_subplan = public_alias - sub_filters = _classify_subplan_filters(host_filters=host_filters) - rerooted_query = SlayerQuery( - source_model=host_model.name, - measures=[ModelMeasure( - formula=formula, name=measure_name_for_subplan, - )], - dimensions=list(host_query.dimensions or []) or None, - time_dimensions=list(host_query.time_dimensions or []) or None, - filters=sub_filters, + # §5.4 — the sub-plan is rooted at the SAME host model, so the + # aggregate key and the host's bound dimensions carry over VERBATIM; + # there is nothing to re-root and so nothing to serialize. The + # user-supplied alias rides along so a host filter referencing the + # rename (``latest_pmt > 500`` for a measure named ``latest_pmt``) + # resolves against the same alias in the sub-plan rather than the + # canonical ``latest_payment_last_updated_at`` form. + host_prebound = host_query.prebound + if host_prebound is None: + raise ValueError( + "DEV-1503 filtered-local isolation needs the host's typed " + "bind product; the carrier arrived without one. The " + "stage_planner must pass the PreboundQuery it planned from." + ) + host_rooted_routes = _route_host_rooted_filters( + host_filters=host_filters, + ) + routing_by_id = {hf.filter_id: hf for hf in host_filters} + sub_prebound = _nested_prebound( + host_prebound=host_prebound, + aggregate_measure=_aggregate_declared_measure( + key=aggregate_key, + model=host_model, + public_alias=public_alias, + ), + grain_measures=list(_grain_declared_measures(host_prebound)), + inherited_filters=[ + routing_by_id[fid].bound + for fid in host_rooted_routes.applied + if routing_by_id[fid].bound is not None + ], + ) + sub_plan = subplan_builder( + StrictQueryCarrier( + source_model=host_model.name, prebound=sub_prebound, + ), + bundle, ) - sub_plan = subplan_builder(rerooted_query, bundle) grain_pairs = _match_filtered_local_grain_pairs( host_slots=host_slots, @@ -867,7 +990,8 @@ def _plan_filtered_local( sub_plan=sub_plan, ) sub_agg_sid = _find_filtered_local_sub_agg_slot( - sub_plan=sub_plan, formula=formula, host_model=host_model, + sub_plan=sub_plan, aggregate_key=aggregate_key, + host_model=host_model, ) cte_schema = _build_filtered_local_cte_schema( aggregate_key=aggregate_key, host_model=host_model, @@ -885,7 +1009,13 @@ def _plan_filtered_local( join_back_pairs=[], cte_stage_schema=cte_schema, shared_grain_slots=[host_sid for host_sid, _ in grain_pairs], - applied_filter_ids=[], + # DEV-1747 D6 — the plan states which host filters the CTE applies + # rather than reporting an empty routing while the sub-plan quietly + # carries them. ``where_filter_ids`` stays empty: the sub-plan + # already holds these predicates as its OWN filters (they were + # handed to it typed), so listing them again would render them + # twice. Nothing is dropped on this route, so no warning can arise. + applied_filter_ids=host_rooted_routes.applied, where_filter_ids=[], having_filter_ids=[], target_model_filters=[], @@ -909,13 +1039,18 @@ def _plan_filtered_local( # collapses the host dimension to a scalar CROSS JOIN -- every host row gets # the global aggregate. # -# The fix mirrors legacy ``_build_rerooted_enriched``: build a full nested -# ``SlayerQuery`` rooted at the target (so all of the target's joins are in -# scope for dimensions AND filters), compile it via ``subplan_builder``, and -# attach the sub-plan to the ``CrossModelAggregatePlan``. The generator -# renders the sub-plan as the ``_cm_*`` CTE and joins it back to the host base -# on the (re-rooted) dimension. Dimensions / filters that don't resolve from -# the target are dropped -- matching legacy's drop-unreachable behaviour. +# The fix: re-anchor the host's bound keys into the target's coordinate +# system (so all of the target's joins are in scope for dimensions AND +# filters), plan them via ``subplan_builder``, and attach the sub-plan to the +# ``CrossModelAggregatePlan``. The generator renders the sub-plan as the +# ``_cm_*`` CTE and joins it back to the host base on the re-rooted dimension. +# Dimensions / filters that don't resolve from the target are dropped. +# +# DEV-1742 §5.4: the re-anchoring is STRUCTURAL. Until then this pass +# regenerated formula text for every ref and let the planner re-bind it, so a +# key's identity survived only as far as the string could carry it — which is +# why a path-bearing source or a host-grain marker could not be expressed at +# all. ``_reroot_host_key`` transforms the key; nothing is re-parsed. # # DEV-1450 #2: this used to be a post-hoc pass in ``stage_planner.plan_query``; # it now lives behind ``IsolatedCteCrossModelPlanner.plan`` so the @@ -924,6 +1059,241 @@ def _plan_filtered_local( # module does not import ``stage_planner`` (no cycle). +def _grain_declared_measures(prebound: PreboundQuery) -> List[DeclaredMeasure]: + """The host's dimension + time-dimension declarations — the grain prefix of + ``declared_measures``, which the nested plan groups by unchanged.""" + n = prebound.n_dims + prebound.n_time_dimensions + return list(prebound.declared_measures[:n]) + + +def _aggregate_declared_measure( + *, + key: AggregateKey, + model: SlayerModel, + public_alias: Optional[str], +) -> DeclaredMeasure: + """The nested plan's single measure declaration, built from the key. + + Reproduces what ``_declared_measures_from_query`` derives for a measure — + canonical alias, type, format, description — without a formula string to + re-parse. An explicit ``public_alias`` surfaces as the name while the + canonical form is retained as ``canonical_alias``, so a colon-form filter + or ORDER BY still resolves onto the same slot (DEV-1443). + """ + canonical = canonical_aggregate_alias(key, profile="stage_formula") + if canonical is None: # pragma: no cover — binder restricts source shapes + canonical = _canonical_name(key) + fmt, desc = measure_key_format_description(model=model, key=key) + return DeclaredMeasure( + bound=BoundExpr(value_key=key), + declared_name=public_alias or canonical, + public_name=public_alias or canonical, + canonical_alias=canonical if public_alias else None, + type=measure_key_type(model=model, key=key), + format=fmt, + description=desc, + ) + + +def _nested_prebound( + *, + host_prebound: PreboundQuery, + aggregate_measure: DeclaredMeasure, + grain_measures: List[DeclaredMeasure], + inherited_filters: List[BoundFilter], + date_range_filters: Optional[List[BoundFilter]] = None, + n_dims: Optional[int] = None, + n_time_dimensions: Optional[int] = None, + main_time_key: Optional[TimeTruncKey] = None, +) -> PreboundQuery: + """Assemble the nested plan's bind product from typed pieces (§5.4). + + Filter order mirrors ``bind_query_inputs``: date-range bounds first (so + ``n_date_range`` still slices them off), then inherited user filters. The + nested plan is never ordered or paginated — the host owns both. + """ + bounds = list( + host_prebound.bound_filters[: host_prebound.n_date_range] + if date_range_filters is None else date_range_filters + ) + return PreboundQuery( + declared_measures=[*grain_measures, aggregate_measure], + bound_filters=[*bounds, *inherited_filters], + bound_filter_texts=( + [None] * len(bounds) + [None] * len(inherited_filters) + ), + n_date_range=len(bounds), + order_specs=[], + main_time_key=( + host_prebound.main_time_key if main_time_key is None + else main_time_key + ), + n_dims=host_prebound.n_dims if n_dims is None else n_dims, + n_time_dimensions=( + host_prebound.n_time_dimensions if n_time_dimensions is None + else n_time_dimensions + ), + distinct_dimension_values=True, + ) + + +def _reroot_host_path( + path: Tuple[str, ...], *, target_path: Tuple[str, ...], + host_model_name: str, +) -> Tuple[str, ...]: + """The path-level half of :func:`_reroot_host_key` — the same three rules, + applied to a bare join path so reachability can be decided without a key.""" + path = tuple(path) + if not path: + return (host_model_name,) + if path[: len(target_path)] == tuple(target_path): + return path[len(target_path):] + return path + + +def _rerooted_reachable_paths( + *, + host_filters: List[HostFilterRouting], + target_path: Tuple[str, ...], + target_model: SlayerModel, + host_model_name: str, + bundle: ResolvedSourceBundle, +) -> frozenset: + """Which host-coordinate join paths a RE-ROOTED CTE can evaluate. + + A prefix test cannot answer this: the CTE is rooted at the target with the + target's whole join graph in scope, so a host-side SIBLING branch is + reachable whenever the target happens to join to it too — common in star + schemas, where several fact-adjacent tables share a dimension. Walking the + graph is the only honest test, and getting it wrong in either direction is + a correctness bug: too narrow drops a filter the user wrote, too wide + emits SQL referencing an unbound table. + """ + reachable = set() + for hf in host_filters: + for p in hf.crossed_join_paths: + if not p or p in reachable: + continue + rr = _reroot_host_path( + p, target_path=target_path, host_model_name=host_model_name, + ) + if walk_key_path( + model=target_model, path=rr, bundle=bundle, + ) is not None: + reachable.add(tuple(p)) + return frozenset(reachable) + + +def _reroot_host_key( + key: ValueKey, *, target_path: Tuple[str, ...], host_model_name: str, +) -> ValueKey: + """Re-anchor one host-coordinate key into the target's coordinate system. + + The typed counterpart of :func:`_reroot_ref`, and the same three rules: + + * host-local (empty path) → reached FROM the target by naming the host as + the first hop, so ``status`` becomes ``orders.status``; + * on or through the target → the target prefix is stripped, which is + exactly ``reroot_value_key``; + * anywhere else → unchanged, to be resolved through the target's own joins. + + The host-local prepend is per-key rather than per-leaf: a composite whose + leaves sit at different depths would need each leaf re-anchored + separately, and no such shape reaches re-rooting today (dimensions and + time dimensions are single references). ``_key_reaches_from`` rejects + anything that does not resolve, so a future composite is dropped rather + than mis-anchored. + """ + inner = key.column if isinstance(key, TimeTruncKey) else key + path = tuple(getattr(inner, "path", ()) or ()) + if not path: + if not hasattr(inner, "path"): + return key + rerooted = inner.model_copy(update={"path": (host_model_name,)}) + if isinstance(key, TimeTruncKey): + return key.model_copy(update={"column": rerooted}) + return rerooted + return reroot_value_key(key, target_path=target_path) + + +def _key_reaches_from( + *, key: ValueKey, model: SlayerModel, bundle: ResolvedSourceBundle, +) -> bool: + """Whether every column-like leaf of ``key`` resolves from ``model``. + + The structural stand-in for "does this bind against the target scope?" — + re-rooting must not call the binder (§5.4), so reachability is decided by + walking the join graph and checking the terminal model owns the leaf. + """ + saw_column = False + for k in walk_value_keys(key): + if isinstance(k, TimeTruncKey): + continue # its wrapped column is walked in its own right + if not isinstance(k, (ColumnKey, ColumnSqlKey, StarKey)): + continue + saw_column = True + terminal = walk_key_path( + model=model, path=tuple(k.path), bundle=bundle, + ) + if terminal is None: + return False + if isinstance(k, StarKey): + continue + leaf = getattr(k, "leaf", None) or getattr(k, "column_name", None) + if leaf is None or terminal.get_column(leaf) is None: + return False + return saw_column + + +def _rerooted_dimension_measure( + *, + key: ValueKey, + label: Optional[str], + target_model: SlayerModel, + bundle: ResolvedSourceBundle, +) -> DeclaredMeasure: + """One re-rooted dimension / time-dimension declaration for the sub-plan. + + ``_canonical_name`` produces the same ``__``-flattened alias the text path + derived from the re-rooted dotted reference, so the CTE's column names and + the host's join-back are unchanged by the switch to typed re-rooting. + """ + if isinstance(key, TimeTruncKey): + return DeclaredMeasure( + bound=BoundExpr(value_key=key), + declared_name=_canonical_name(key), + public_name=_canonical_name(key), + label=label, + type=DataType.TIMESTAMP, + ) + dim_type, fmt, desc = dimension_key_metadata( + model=target_model, key=key, bundle=bundle, + ) + return DeclaredMeasure( + bound=BoundExpr(value_key=key), + declared_name=_canonical_name(key), + public_name=_canonical_name(key), + label=label, + type=dim_type, + format=fmt, + description=desc, + ) + + +# --------------------------------------------------------------------------- +# Superseded by the typed re-rooting above (DEV-1742 §5.4 / P-J state 1) +# --------------------------------------------------------------------------- +# +# ``_reroot_ref``, ``_host_ref_path``, ``_render_ref_formula``, +# ``_scalar_formula_literal``, ``_local_agg_formula`` and +# ``_REROOT_BIND_ERRORS`` are the formula-text round-trip these functions +# replaced. They are PRODUCTION-UNREFERENCED as of this change; their tests +# stay green so the two mechanisms can be compared, and deletion happens in +# one sweep (PR 6) rather than being smeared across the series. +# +# ``_filter_ref_paths`` is NOT in this group — the typed path still uses it. + + def _reroot_ref( *, model_prefix: Optional[str], name: str, host_model_name: str, target_model_name: str, @@ -1031,141 +1401,192 @@ def _local_agg_formula(key: AggregateKey) -> str: ) -def _maybe_reroot_cross_model_plan( +def _maybe_reroot_cross_model_plan( # NOSONAR(S3776) — one re-rooting decision over three parallel input kinds (dimensions, time dimensions, filters). Each loop re-anchors, tests target-reachability, and votes on `needs_reroot`; the vote is the shared state that makes them one pass rather than three functions. *, - plan, - query: SlayerQuery, + make_plan: Callable[["_FilterRoutes"], CrossModelAggregatePlan], + query: StrictQueryCarrier, agg_key: AggregateKey, bundle: ResolvedSourceBundle, host_model: SlayerModel, + host_slots: List[ValueSlot], + host_filters: List[HostFilterRouting], public_projection: List[str], - subplan_builder: Callable[[SlayerQuery, ResolvedSourceBundle], PlannedQuery], + subplan_builder: Callable[ + [StrictQueryCarrier, ResolvedSourceBundle], PlannedQuery, + ], + target_model_name: str, + target_path: Tuple[str, ...], ): - """Attach a re-rooted sub-``PlannedQuery`` to ``plan`` when the host - query carries dimensions reachable from the target by re-rooting through - the target's join graph. Returns ``plan`` unchanged when re-rooting is - unnecessary (only forward-path or genuinely unreachable dims).""" - target_model_name = plan.target_model + """Decide forward-vs-re-rooted, classify the host filters ONCE for whichever + shape won, and build the plan. + + Re-rooting applies when the host carries dimensions or filters reachable + from the target only by walking the TARGET's own join graph — off the + host→target forward path, which the forward CTE cannot evaluate. + + §5.4 — every input arrives already bound, on the carrier's + ``PreboundQuery``. Re-rooting re-anchors those keys structurally and hands + them straight back to the planner, so the sub-plan's slot identities are + the host's, transformed — never re-derived from a regenerated string. + + D6 — the decision precedes the classification. The two used to run in the + other order, with the reroot then BLANKING what the classifier had decided + against a coordinate system that no longer applied. + """ target_model = bundle.get_referenced_model(target_model_name) - if target_model is None: - return plan - target_path = tuple(getattr(agg_key.source, "path", ())) + host_prebound = query.prebound + + def _forward_only(): + return make_plan(_route_host_filters( + host_filters=host_filters, host_slots=host_slots, + target_path=target_path, host_model=host_model, + terminal_model=target_model or host_model, + )) + + if target_model is None or host_prebound is None: + return _forward_only() rerooted_bundle = bundle.model_copy(update={"source_model": target_model}) - target_scope = ModelScope(source_model=target_model) - - def _resolvable_ref(ref_str: str) -> Optional[ValueKey]: - try: - return bind_expr( - parse_expr(ref_str), - scope=target_scope, - bundle=rerooted_bundle, - ).value_key - except _REROOT_BIND_ERRORS: - return None + + def _reroot(key: ValueKey) -> ValueKey: + return _reroot_host_key( + key, target_path=target_path, host_model_name=host_model.name, + ) + + def _reaches(key: ValueKey) -> bool: + return _key_reaches_from( + key=key, model=target_model, bundle=rerooted_bundle, + ) def _is_forward(path: Tuple[str, ...]) -> bool: # On the host->target path (handled by the forward-path CTE already). return bool(path) and path == target_path[: len(path)] - n_dims = len(query.dimensions or []) - rerooted_dims: List[ColumnRef] = [] - rerooted_tds: List[TimeDimension] = [] + n_dims = host_prebound.n_dims + n_tds = host_prebound.n_time_dimensions + grain_declared: List[DeclaredMeasure] = [] grain_host_sids: List[str] = [] grain_rerooted_keys: List[ValueKey] = [] needs_reroot = False - for i, dim in enumerate(query.dimensions or []): + for i, dm in enumerate(host_prebound.declared_measures[: n_dims + n_tds]): host_sid = public_projection[i] if i < len(public_projection) else None - host_path = _host_ref_path(dim.model) - rr = _reroot_ref( - model_prefix=dim.model, name=dim.name, - host_model_name=host_model.name, target_model_name=target_model_name, + host_key = dm.bound.value_key + inner = ( + host_key.column if isinstance(host_key, TimeTruncKey) else host_key ) - rr_key = _resolvable_ref(rr) - if rr_key is None: + host_path = tuple(getattr(inner, "path", ()) or ()) + rr_key = _reroot(host_key) + if not _reaches(rr_key): continue # unreachable from target -> drop if not _is_forward(host_path): needs_reroot = True if host_sid is None: continue - rerooted_dims.append(ColumnRef(name=rr, label=dim.label)) - grain_host_sids.append(host_sid) - grain_rerooted_keys.append(rr_key) - - for j, td in enumerate(query.time_dimensions or []): - idx = n_dims + j - host_sid = public_projection[idx] if idx < len(public_projection) else None - host_path = _host_ref_path(td.dimension.model) - rr = _reroot_ref( - model_prefix=td.dimension.model, name=td.dimension.name, - host_model_name=host_model.name, target_model_name=target_model_name, - ) - rr_td = TimeDimension( - dimension=ColumnRef(name=rr), - granularity=td.granularity, - date_range=td.date_range, - label=td.label, - ) - try: - rr_key = bind_time_dimension( - rr_td, scope=target_scope, bundle=rerooted_bundle, - ).value_key - except _REROOT_BIND_ERRORS: - continue - if not _is_forward(host_path): - needs_reroot = True - if host_sid is None: - continue - rerooted_tds.append(rr_td) + grain_declared.append(_rerooted_dimension_measure( + key=rr_key, label=dm.label, target_model=target_model, + bundle=rerooted_bundle, + )) grain_host_sids.append(host_sid) grain_rerooted_keys.append(rr_key) - # Filters. A purely host-local filter (every ref on the host's own - # columns) filters host rows -- it stays at the host base; the join-back - # propagates the cardinality reduction, so adding it to the CTE would risk - # binding a bare name to a same-named TARGET column. A join-traversing - # filter affects the aggregate value and rides into the re-rooted CTE; one - # that reaches OFF the host->target forward path is exactly what the - # forward-path classifier drops, so it also triggers re-rooting (covers a - # cross-model agg filtered through the target's graph with no dimensions). - host_scope = ModelScope(source_model=host_model) - rerooted_filters: List[str] = [] - for f in (query.filters or []): - try: - host_bound = bind_filter( - parse_filter_expr(f), scope=host_scope, bundle=bundle, - ) - except _REROOT_BIND_ERRORS: - continue - host_paths = _filter_ref_paths(host_bound.value_key) - if all(p == () for p in host_paths): + # Filters vote structurally, before any classification. A filter that + # reaches OFF the host→target forward path is exactly what the forward CTE + # cannot evaluate, so wanting it is a reason to re-root — the same vote a + # non-forward dimension casts. Only a filter the RE-ROOTED CTE could + # actually evaluate votes: an ``order_tags`` predicate is unreachable + # either way and must not drag the plan into a shape that does not help it. + reachable_paths = _rerooted_reachable_paths( + host_filters=host_filters, + target_path=target_path, + target_model=target_model, + host_model_name=host_model.name, + bundle=rerooted_bundle, + ) + for hf in host_filters: + crossed = [p for p in hf.crossed_join_paths if p] + if not crossed: continue # host-local -> applied at the host base only - # The binder strips a same-model self-prefix (C14), so a - # ``.col`` ref binds locally against the target scope without - # any string surgery -- pass the filter through verbatim. - try: - bind_filter( - parse_filter_expr(f), scope=target_scope, bundle=rerooted_bundle, - ) - except _REROOT_BIND_ERRORS: + if not all(p in reachable_paths for p in crossed): continue - rerooted_filters.append(f) - if any(p != target_path[: len(p)] for p in host_paths if p): + if any(not _is_forward(p) for p in crossed): needs_reroot = True + break - if not needs_reroot or not ( - rerooted_dims or rerooted_tds or rerooted_filters - ): + routes = _route_host_filters( + host_filters=host_filters, + host_slots=host_slots, + target_path=target_path, + host_model=host_model, + terminal_model=target_model, + reachable_paths=reachable_paths if needs_reroot else None, + ) + plan = make_plan(routes) + if not needs_reroot or not (grain_declared or routes.applied): return plan - rerooted_query = SlayerQuery( - source_model=target_model_name, - measures=[ModelMeasure(formula=_local_agg_formula(agg_key))], - dimensions=rerooted_dims or None, - time_dimensions=rerooted_tds or None, - filters=rerooted_filters or None, + # The CTE applies exactly what the routing says it applies — one decision, + # consumed, never re-derived. ``routing_by_id`` maps those ids back to the + # typed predicates so each rides in re-anchored rather than re-parsed. + # + # Date-range bounds (the ``[:n_date_range]`` prefix of ``bound_filters``) + # are not user filters and carry no routing id; they are re-anchored below + # alongside their time dimension. + routing_by_id = {hf.filter_id: hf for hf in host_filters} + rerooted_filters: List[BoundFilter] = [] + for fid in routes.applied: + hf = routing_by_id.get(fid) + if hf is None or hf.bound is None: + continue + rr_key = reroot_value_key(hf.bound.value_key, target_path=target_path) + rerooted_filters.append(BoundFilter( + value_key=rr_key, + phase=hf.bound.phase, + referenced_keys=tuple(walk_value_keys(rr_key)), + )) + + # Date-range bounds ride into the CTE alongside their re-rooted time + # dimension, so the sub-plan applies the same window the host does. + rerooted_bounds = [ + BoundFilter( + value_key=reroot_value_key(bf.value_key, target_path=target_path), + phase=bf.phase, + referenced_keys=tuple(walk_value_keys( + reroot_value_key(bf.value_key, target_path=target_path), + )), + ) + for bf in host_prebound.bound_filters[: host_prebound.n_date_range] + if _reaches(reroot_value_key(bf.value_key, target_path=target_path)) + ] + n_rerooted_tds = sum( + 1 for dm in grain_declared + if isinstance(dm.bound.value_key, TimeTruncKey) + ) + sub_prebound = _nested_prebound( + host_prebound=host_prebound, + aggregate_measure=_aggregate_declared_measure( + key=reroot_value_key(agg_key, target_path=target_path), + model=target_model, + public_alias=None, + ), + grain_measures=grain_declared, + inherited_filters=rerooted_filters, + date_range_filters=rerooted_bounds, + n_dims=len(grain_declared) - n_rerooted_tds, + n_time_dimensions=n_rerooted_tds, + main_time_key=next( + ( + dm.bound.value_key for dm in grain_declared + if isinstance(dm.bound.value_key, TimeTruncKey) + ), + None, + ), + ) + sub_plan = subplan_builder( + StrictQueryCarrier( + source_model=target_model_name, prebound=sub_prebound, + ), + rerooted_bundle, ) - sub_plan = subplan_builder(rerooted_query, rerooted_bundle) sub_row_by_key = {s.key: s.id for s in sub_plan.row_slots} grain_pairs: List[Tuple[str, str]] = [] @@ -1184,17 +1605,14 @@ def _is_forward(path: Tuple[str, ...]) -> bool: if sub_agg_sid is None: return plan + # DEV-1747 B6/D6 — the routing is NOT cleared. It was decided once, in the + # coordinate system of the CTE that now exists, and the sub-plan applies + # exactly the filters it records as applied. Blanking it here is what made + # a reachable filter, a host-local one, and a genuinely unreachable one all + # report ``where=[] having=[] applied=[] dropped=[]`` — indistinguishable, + # and in the unreachable case a silent narrowing of the user's result. return plan.model_copy(update={ "rerooted_plan": sub_plan, "rerooted_grain_pairs": grain_pairs, "rerooted_agg_slot_id": sub_agg_sid, - # The forward-path classifier marked these host filters - # DROP_UNREACHABLE, but the re-rooted CTE re-applies every - # target-reachable filter (and the host base keeps the rest for - # cardinality), so nothing is silently dropped -- clear the now-stale - # warnings and forward-only routing ids. - "dropped_filter_warnings": [], - "where_filter_ids": [], - "having_filter_ids": [], - "applied_filter_ids": [], }) diff --git a/slayer/engine/filter_reachability.py b/slayer/engine/filter_reachability.py index d3b9de40..fa090444 100644 --- a/slayer/engine/filter_reachability.py +++ b/slayer/engine/filter_reachability.py @@ -354,15 +354,31 @@ def _walk(node) -> bool: return _walk(key) -def path_is_reachable(*, path: Path, target_path: Path) -> bool: +def path_is_reachable( + *, + path: Path, + target_path: Path, + reachable_paths: "Optional[frozenset]" = None, +) -> bool: """The ONE reachability rule, for every key kind. - ``path`` is reachable from a CTE rooted at ``target_path`` iff it is a - prefix of it. A path DEEPER than the target is not available (the target's - scope stops there); a SIBLING branch that happens to share a model name is - not available either, which is precisely what the old flat membership test - got wrong. + A FORWARD-path CTE selects from the bare target and carries only the + host→target hops, so ``path`` is reachable iff it is a PREFIX of + ``target_path``. A path deeper than the target is not available (the + target's scope stops there); a sibling branch that happens to share a model + name is not available either, which is precisely what the old flat + membership test got wrong. + + A RE-ROOTED CTE (DEV-1747 D6) is planned against the TARGET as its own + root, so the target's WHOLE join graph is in scope and the prefix test no + longer describes it: a host-side sibling branch can be reachable from the + target by a different route entirely. That question is about the model + graph, not about string prefixes, so the caller — which holds the bundle — + walks it and passes the answer in as ``reachable_paths``. Membership then + replaces the prefix test outright. """ + if reachable_paths is not None: + return tuple(path) in reachable_paths return tuple(path) == tuple(target_path[: len(path)]) diff --git a/slayer/engine/planned.py b/slayer/engine/planned.py index a706a757..f954e8d3 100644 --- a/slayer/engine/planned.py +++ b/slayer/engine/planned.py @@ -23,7 +23,8 @@ from __future__ import annotations -from typing import Dict, List, Optional, Tuple +from enum import Enum +from typing import Dict, List, Literal, Optional, Tuple from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator @@ -65,6 +66,7 @@ "FilterPhase", "JoinRequirement", "OrderEntry", + "OrderScope", "PlannedQuery", "SlotId", "TransformLayer", @@ -376,11 +378,49 @@ class FilterPhase(BaseModel): # --------------------------------------------------------------------------- +class OrderScope(str, Enum): + """WHERE the ordered value lives — the one thing a renderer needs to know + to build a sort term (DEV-1747 §5.10). + + Every render site used to re-derive this, and they disagreed: one + dispatched on the slot KIND, one ran a five-way precedence chain over + alias maps, and one knew about neither. Naming the producing scope in the + plan is what lets a single resolver replace all of them (P-D). + """ + + #: Materialised in ``_base`` and projected publicly. + HOST_BASE = "host_base" + #: Materialised in ``_base`` but trimmed from the public projection — + #: an order-only aggregate or an unprojected host dimension. + HOST_BASE_HIDDEN = "host_base_hidden" + #: Lives in a cross-model / host-rooted isolated (``_cm_``) CTE. + CROSS_MODEL_CTE = "cross_model_cte" + #: Lives in a windowed (``_wm_``) CTE. + WINDOWED_CTE = "windowed_cte" + #: Produced by a step of the transform chain. + TRANSFORM_STEP = "transform_step" + #: A composite whose operands span scopes, so it can only be evaluated in + #: the outer combined SELECT — never inside ``_base``. + OUTER_COMPOSITE = "outer_composite" + + class OrderEntry(BaseModel): - """One entry in the ORDER BY of a planned query.""" + """One entry in the ORDER BY of a planned query. + + ``scope`` and ``phase`` are REQUIRED and have no default: a planner path + that forgets to classify must fail at construction rather than fall through + to the ``_base.``-qualified branch, which is how an order term silently + attached to the wrong scope. + """ slot_id: SlotId direction: str # "asc" or "desc" + scope: OrderScope + phase: Phase + #: Null-ordering policy. ``"default"`` defers to the dialect's native + #: ordering for the direction; the dialect strategy owns the spelling + #: (P-H), so no render site emits a NULLS clause of its own. + nulls: Literal["default", "first", "last"] = "default" @field_validator("direction") @classmethod diff --git a/slayer/engine/stage_planner.py b/slayer/engine/stage_planner.py index 678d37ea..ae00479f 100644 --- a/slayer/engine/stage_planner.py +++ b/slayer/engine/stage_planner.py @@ -25,7 +25,7 @@ from __future__ import annotations -from typing import Dict, FrozenSet, List, Optional, Tuple, Union +from typing import Dict, FrozenSet, List, Optional, Set, Tuple, Union from slayer.core.enums import DataType from slayer.core.format import NumberFormat @@ -33,7 +33,6 @@ AmbiguousReferenceError, DistinctDimensionValuesError, UnknownReferenceError, - UnresolvableOrderColumnError, ) from slayer.core.keys import ( AggregateKey, @@ -45,7 +44,6 @@ LiteralKey, Phase, ScalarCallKey, - StarKey, TimeTruncKey, TransformKey, ValueKey, @@ -80,7 +78,6 @@ IsolatedCteCrossModelPlanner, ) from slayer.engine.measure_expansion import expand_model_measures -from slayer.engine.response_meta import _infer_aggregated_format from slayer.engine.filter_reachability import ( compute_key_join_paths, key_has_host_local_ref, @@ -93,6 +90,7 @@ FilterPhase, FilterReachability, OrderEntry, + OrderScope, PlannedQuery, SlotId, SrcFilterRewrite, @@ -110,6 +108,12 @@ lower_sugar_transforms, rewrite_rank_partition_keys, ) +from slayer.engine.prebound import ( + PreboundQuery, + StrictQueryCarrier, + measure_key_format_description, + measure_key_type, +) from slayer.engine.source_bundle import ( ResolvedSourceBundle, _apply_extension_overlay, @@ -124,7 +128,13 @@ from slayer.sql.sql_predicate import parse_sql_predicate -__all__ = ["plan_query", "plan_stages"] +__all__ = [ + "PreboundQuery", + "StrictQueryCarrier", + "bind_query_inputs", + "plan_query", + "plan_stages", +] # Stage 7b.10 — TIME_NEEDING transform ops that require a resolvable @@ -645,52 +655,48 @@ def _reject_measure_refs_in_order( ) -def plan_query( # NOSONAR(S3776) — planner entry-point dispatcher. The DEV-1503 addition is a small trigger-predicate branch + a kwarg pass-through; the function's pre-existing complexity is owned by the multi-stage scope / bundle / projection / filter-routing wiring it orchestrates and is tracked as a separate refactor. +def _resolve_scope( + *, + query: SlayerQuery, + bundle: ResolvedSourceBundle, + stage_schemas: Optional[Dict[str, StageSchema]], +) -> Union[ModelScope, StageSchema]: + """Default binding scope: an upstream ``StageSchema`` when the query's + ``source_model`` names a sibling stage, else a ``ModelScope`` over the + bundle's host model.""" + source = query.source_model + if isinstance(source, str) and source in (stage_schemas or {}): + return (stage_schemas or {})[source] + return ModelScope(source_model=bundle.source_model) + + +def bind_query_inputs( # NOSONAR(S3776) — one cohesive bind pass. The stages are strictly sequential and share the growing `declared_measures` / `bound_filters` / `order_specs` triple: parse+bind, time-key attachment, sugar lowering, rank-partition validation. Splitting them would thread the same three lists through four signatures without removing a branch. *, query: SlayerQuery, bundle: ResolvedSourceBundle, scope: Optional[Union[ModelScope, StageSchema]] = None, - cross_model_planner: Optional[CrossModelPlanner] = None, stage_schemas: Optional[Dict[str, StageSchema]] = None, - disable_host_rooted_isolation: bool = False, -) -> PlannedQuery: - """Compile one ``SlayerQuery`` into a typed ``PlannedQuery``. - - ``scope`` defaults to a ``ModelScope`` over ``bundle.source_model``; - pass an explicit ``StageSchema`` to bind against an upstream stage. - ``stage_schemas`` is a name → StageSchema map used by - ``plan_stages`` to wire multi-stage references. - - ``disable_host_rooted_isolation`` (DEV-1503; renamed and widened by - DEV-1709) suppresses the HOST-ROOTED half of the Law-3 trigger — the - isolation of a LOCAL aggregate whose ``Column.filter`` or any other - input (source ``Column.sql``, positional args, kwargs) crosses a join. - It never affects target-rooted isolation (``source.path`` non-empty). - Threaded through ``subplan_builder`` whenever a cross-model strategy - recurses for a host-rooted (or target-rooted) nested sub-plan, so the - sub-plan's same crossing measure is rendered inline (not infinitely - re-isolated) and so re-rooting of a genuine cross-model aggregate - doesn't redundantly isolate the target's own filter joins. +) -> PreboundQuery: + """Parse and bind every text surface of a ``SlayerQuery`` (DEV-1742 §5.4). + + This is the ONLY door into the parser. ``plan_query`` calls it when no + ``prebound`` is supplied; a caller that already holds typed keys builds a + ``PreboundQuery`` structurally and skips it entirely, which is what makes + re-rooting free of formula-text round-trips (P-E). + + The returned keys are fully normalized — time keys attached, ``change`` / + ``change_pct`` sugar lowered, rank ``partition_by`` columns validated and + rewritten to their time buckets — so planning sees exactly one key shape. + + Model filters are deliberately NOT included: they are a property of the + SCOPE, not the query, and ``plan_query`` lifts them from ``scope`` + directly so a pre-bound caller inherits its own model's filters rather + than the host's. """ - stage_schemas = stage_schemas or {} - cross_model_planner = ( - cross_model_planner or IsolatedCteCrossModelPlanner() - ) - if scope is None: - source = query.source_model - if isinstance(source, str) and source in stage_schemas: - scope = stage_schemas[source] - else: - scope = ModelScope(source_model=bundle.source_model) - - # The generator must render this stage's FROM / joins against the SAME - # model the binder used. For a ModelScope that's the (possibly overlaid / - # synthetic) host; for a StageSchema chain stage it's None (the generator - # builds a synthetic model from the upstream schema). - render_source_model = ( - scope.source_model if isinstance(scope, ModelScope) else None - ) + scope = _resolve_scope( + query=query, bundle=bundle, stage_schemas=stage_schemas, + ) # DEV-1543: raw-rows mode rejects measure references in filters / order. # Runs BEFORE binding so the targeted, actionable error wins over the @@ -766,7 +772,6 @@ def plan_query( # NOSONAR(S3776) — planner entry-point dispatcher. The DEV-15 # (which has not been deduped, unlike bound_filters) — CR PR #153 # thread r3350000254. bound_filter_texts: List[Optional[str]] = [] - text_filter_entries: List[FilterPhase] = [] # 1. date_range filters (one per TD with a 2-element date_range) for td in (query.time_dimensions or []): @@ -779,12 +784,11 @@ def plan_query( # NOSONAR(S3776) — planner entry-point dispatcher. The DEV-15 bound_filter_texts.append(None) n_date_range = len(bound_filters) - # 2. SlayerModel.filters — Mode-A SQL, always-applied WHERE. - if isinstance(scope, ModelScope) and scope.source_model is not None: - for j, mf in enumerate(scope.source_model.filters or []): - text_filter_entries.append(_validate_model_filter( - mf=mf, idx=j, model=scope.source_model, - )) + # 2. SlayerModel.filters — Mode-A SQL, always-applied WHERE. Lifted from + # the SCOPE in ``plan_query``, not here (§5.4): they belong to the model + # being planned against, so a re-rooted sub-plan must pick up the + # TARGET's model filters rather than inherit the host's through the + # carrier. # 3. user query filters (Mode-B DSL). # @@ -1103,6 +1107,97 @@ def _rw(vk: ValueKey) -> ValueKey: for spec in order_specs ] + return PreboundQuery( + declared_measures=declared_measures, + bound_filters=bound_filters, + bound_filter_texts=bound_filter_texts, + n_date_range=n_date_range, + order_specs=order_specs, + main_time_key=active_td_key, + n_dims=n_dims, + n_time_dimensions=n_tds, + limit=query.limit, + offset=query.offset, + distinct_dimension_values=query.distinct_dimension_values, + ) + + +def plan_query( # NOSONAR(S3776) — planner entry-point dispatcher. The DEV-1503 addition is a small trigger-predicate branch + a kwarg pass-through; the function's pre-existing complexity is owned by the multi-stage scope / bundle / projection / filter-routing wiring it orchestrates and is tracked as a separate refactor. + *, + query: Union[SlayerQuery, StrictQueryCarrier], + bundle: ResolvedSourceBundle, + scope: Optional[Union[ModelScope, StageSchema]] = None, + cross_model_planner: Optional[CrossModelPlanner] = None, + stage_schemas: Optional[Dict[str, StageSchema]] = None, + disable_host_rooted_isolation: bool = False, + prebound: Optional[PreboundQuery] = None, +) -> PlannedQuery: + """Compile one query into a typed ``PlannedQuery``. + + ``scope`` defaults to a ``ModelScope`` over ``bundle.source_model``; + pass an explicit ``StageSchema`` to bind against an upstream stage. + ``stage_schemas`` is a name → StageSchema map used by + ``plan_stages`` to wire multi-stage references. + + ``prebound`` (DEV-1742 §5.4) supplies the bind product directly, skipping + the parser. ``query`` is then a ``StrictQueryCarrier`` holding only the + post-bind scalars the planner still needs — anything else it is asked for + raises, so a new read cannot silently fall back to a default. + + ``disable_host_rooted_isolation`` (DEV-1503; renamed and widened by + DEV-1709) suppresses the HOST-ROOTED half of the Law-3 trigger — the + isolation of a LOCAL aggregate whose ``Column.filter`` or any other + input (source ``Column.sql``, positional args, kwargs) crosses a join. + It never affects target-rooted isolation (``source.path`` non-empty). + Threaded through ``subplan_builder`` whenever a cross-model strategy + recurses for a host-rooted (or target-rooted) nested sub-plan, so the + sub-plan's same crossing measure is rendered inline (not infinitely + re-isolated) and so re-rooting of a genuine cross-model aggregate + doesn't redundantly isolate the target's own filter joins. + """ + stage_schemas = stage_schemas or {} + cross_model_planner = ( + cross_model_planner or IsolatedCteCrossModelPlanner() + ) + + if scope is None: + scope = _resolve_scope( + query=query, bundle=bundle, stage_schemas=stage_schemas, + ) + + # The generator must render this stage's FROM / joins against the SAME + # model the binder used. For a ModelScope that's the (possibly overlaid / + # synthetic) host; for a StageSchema chain stage it's None (the generator + # builds a synthetic model from the upstream schema). + render_source_model = ( + scope.source_model if isinstance(scope, ModelScope) else None + ) + + if prebound is None: + prebound = bind_query_inputs( + query=query, bundle=bundle, scope=scope, + stage_schemas=stage_schemas, + ) + declared_measures = list(prebound.declared_measures) + bound_filters = list(prebound.bound_filters) + bound_filter_texts = list(prebound.bound_filter_texts) + n_date_range = prebound.n_date_range + order_specs = list(prebound.order_specs) + active_td_key = prebound.main_time_key + n_dims = prebound.n_dims + n_tds = prebound.n_time_dimensions + distinct_dimension_values = prebound.distinct_dimension_values + + # SlayerModel.filters — Mode-A SQL, always-applied WHERE. Scope-derived + # (see ``bind_query_inputs``), so a pre-bound sub-plan picks up its OWN + # model's filters. + text_filter_entries: List[FilterPhase] = [] + if isinstance(scope, ModelScope) and scope.source_model is not None: + for j, mf in enumerate(scope.source_model.filters or []): + text_filter_entries.append(_validate_model_filter( + mf=mf, idx=j, model=scope.source_model, + )) + source_col_names = _source_column_names(scope) host_model_name = _host_model_name(scope) @@ -1144,7 +1239,7 @@ def _rw(vk: ValueKey) -> ValueKey: # raw-rows query materialised a hidden aggregate and emitted # ``GROUP BY ... HAVING ...``, the exact auto-aggregation the flag exists # to turn off. - if query.distinct_dimension_values is False and agg_slots: + if distinct_dimension_values is False and agg_slots: offender = _canonical_name(agg_slots[0].key) raise DistinctDimensionValuesError( f"distinct_dimension_values=False rejects measure references, but " @@ -1204,17 +1299,17 @@ def _rw(vk: ValueKey) -> ValueKey: # bare reference is legal); # * LOCAL row column, GROUPED query -> the bare reference is NOT legal # (the column is not in GROUP BY), so it materialises as a hidden - # ``:max`` aggregate slot and the order entry is repointed at it. - # MAX is order-preserving per group and portable across every Tier-1 - # dialect; - # * JOINED row column, GROUPED query -> still - # ``UnresolvableOrderColumnError``. A host-rooted MAX over a joined - # column is not expressible today: an ``AggregateKey`` with a - # non-empty ``source.path`` always routes to a TARGET-rooted CTE - # (Law 3), which for a host-grain sort key degenerates to a scalar - # CROSS JOIN — every group would get the same global value and the - # sort would silently do nothing. Failing loudly beats sorting by a - # constant. Full support (host-rooted crossing MAX) is DEV-1735; + # direction-aware aggregate slot (``:min`` for ASC, ``:max`` for DESC) + # and the order entry is repointed at it. Both are order-preserving + # per group and portable across every Tier-1 dialect; + # * JOINED row column, GROUPED query -> the same wrap, marked + # ``grain="host"`` (DEV-1747 D2). The marker is what separates WHERE + # the value is READ (through the join, per ``source.path``) from WHERE + # it is GROUPED (per host row-group). Without it a path-bearing source + # always routed to a TARGET-rooted CTE, which for a host-grain sort key + # degenerates to a scalar CROSS JOIN — every group gets the same global + # value and the sort silently does nothing. That case used to be + # rejected outright rather than sorted wrongly; # * transform / composite -> materialised as a hidden slot and ordered at # the outer wrap (DEV-1733), same Law-2 discipline as aggregates. # @@ -1224,11 +1319,14 @@ def _rw(vk: ValueKey) -> ValueKey: # it, and a sort must not fail because ``max`` is not whitelisted on the # column being sorted. _has_grouping = bool(agg_slots) or ( - bool(query.dimensions or query.time_dimensions) - and query.distinct_dimension_values + bool(n_dims or n_tds) and distinct_dimension_values ) - # ORDER BY targets rewritten to a hidden aggregate: original key -> MAX key. - order_key_remap: Dict[ValueKey, ValueKey] = {} + # ORDER BY targets rewritten to a hidden aggregate wrap, keyed by + # (original key, DIRECTION). DEV-1747 D10 makes the wrap direction-aware, + # so ``ORDER BY a ASC, a DESC`` needs MIN(a) and MAX(a) — two different + # values over one column. Keying by the value key alone would collapse them + # onto whichever slot was interned first. + order_key_remap: Dict[Tuple[ValueKey, str], ValueKey] = {} for spec in order_specs: okey = spec.bound.value_key osid = projection.registry.find_by_key(okey) @@ -1240,29 +1338,35 @@ def _rw(vk: ValueKey) -> ValueKey: if not _has_grouping: continue # raw-rows query -> split emission, no wrap needed path = _row_key_path(okey) - if path: - # ``UnresolvableOrderColumnError`` formats ``qualifier.column``; - # pass the bare leaf as ``column`` and the joined path as the - # qualifier so the message reads ``customers.regions.name`` and - # not a duplicated ``customers.customers.regions.name``. - disp = _partition_key_display(okey) - raise UnresolvableOrderColumnError( - column=disp.rsplit(".", 1)[-1], qualifier=".".join(path), - ) # A TimeTruncKey is not a legal aggregate source; wrap its # underlying column instead. DATE_TRUNC is monotonic - # non-decreasing, so MAX(col) and MAX(trunc(col)) sort a group - # identically. + # non-decreasing, so the wrap over the trunc and over the raw + # column sort a group identically. src = okey.column if isinstance(okey, TimeTruncKey) else okey - max_key = AggregateKey(source=src, agg="max") - if projection.registry.find_by_key(max_key) is None: + # DEV-1747 D10 — ASC orders each group by its MINIMUM and DESC by + # its MAXIMUM: the extreme the direction actually puts first. + # An unconditional MAX (the pre-DEV-1747 behaviour) sorts ASC by + # each group's LARGEST member, which is not what the user asked + # for whenever groups overlap in range. + wrap_key = AggregateKey( + source=src, + agg="min" if spec.direction == "asc" else "max", + # DEV-1747 D2 — a JOINED sort key is host-grain: it must be + # computed in a CTE rooted at the HOST with the crossed join + # pulled inside, grouped on the query grain. Routing it to a + # target-rooted CTE (which a bare path-bearing source does) + # degenerates to a scalar CROSS JOIN, giving every group the + # same global value. + grain="host" if path else "target", + ) + if projection.registry.find_by_key(wrap_key) is None: projection.registry.intern( - key=max_key, - declared_name=_canonical_name(max_key), + key=wrap_key, + declared_name=_canonical_name(wrap_key), hidden=True, - phase=max_key.phase, + phase=wrap_key.phase, ) - order_key_remap[okey] = max_key + order_key_remap[(okey, spec.direction)] = wrap_key # DEV-1733: TransformKey / ArithmeticKey / ScalarCallKey — an inline # transform or composite expression referenced only in ORDER BY. These # materialise as hidden slots (a step CTE on the transform path, a @@ -1339,6 +1443,16 @@ def _windowed_phase(bf: BoundFilter) -> Phase: if isinstance(query.source_model, str) else host_model_name ) + # The post-bind ``query.*`` surface a re-rooted sub-plan is allowed to see. + # Built here so the SAME object carries the typed bind product into the + # cross-model strategy and back out through ``subplan_builder`` (§5.4). + host_carrier = StrictQueryCarrier( + source_model=( + query.source_model if isinstance(query.source_model, str) else None + ), + name=query.name, + prebound=prebound, + ) filter_reachability: List[FilterReachability] = [] # One expansion cache for the whole plan — the two visitors ask for the # same derived column's expansion, and so does every filter that mentions it. @@ -1377,6 +1491,9 @@ def _windowed_phase(bf: BoundFilter) -> Phase: bf, projection.registry, )), text=ftext, + # §5.4 — the typed predicate, so a sub-plan that inherits this + # filter re-roots the KEY rather than re-parsing ``text``. + bound=bf, crossed_join_paths=( summary.crossed_join_paths if summary is not None else () ), @@ -1428,8 +1545,25 @@ def _windowed_phase(bf: BoundFilter) -> Phase: )) ) ) + # DEV-1747 D2 — the third branch: a HOST-grain aggregate over a joined + # source. Its crossing IS the path, so it isolates HOST-rooted (the + # crossed join is pulled INTO the CTE, which is grouped on the query + # grain) rather than target-rooted, which for a host-grain sort key + # would degenerate to a scalar CROSS JOIN. + is_host_grain = ( + bool(agg_path) and getattr(key, "grain", "target") == "host" + ) if not agg_path and not has_crossing_input: continue + if is_host_grain and disable_host_rooted_isolation: + # The recursion guard, for the same reason it applies to the + # crossing-input branch: inside the nested sub-plan the CTE is + # already this aggregate's own scope, so it renders inline + # (base-pull). Isolating again would nest the same key forever. + # Target-rooted cross-model aggregates are deliberately NOT + # suppressed — inlining a joined SUM into the host base would + # multiply it by the join's fan-out. + continue # DEV-1450 #2: re-rooting (C1) is owned by the strategy. We hand it # the host query, the public projection, and a sub-plan builder so it # can compile a nested re-rooted PlannedQuery when the host carries @@ -1457,7 +1591,7 @@ def _windowed_phase(bf: BoundFilter) -> Phase: host_filters=host_filter_routings, public_alias=slot.public_name, hidden=slot.hidden, - host_query=query if reroot_enabled else None, + host_query=host_carrier if reroot_enabled else None, public_projection=( projection.public_projection if reroot_enabled else None ), @@ -1465,6 +1599,9 @@ def _windowed_phase(bf: BoundFilter) -> Phase: (lambda q, b: plan_query( query=q, bundle=b, cross_model_planner=cross_model_planner, disable_host_rooted_isolation=True, + # The carrier IS the hand-off: its ``prebound`` is the + # typed sub-query the strategy re-rooted structurally. + prebound=q.prebound, )) if reroot_enabled else None ), @@ -1473,9 +1610,11 @@ def _windowed_phase(bf: BoundFilter) -> Phase: order_entries = [] for spec in order_specs: - # A grouped row-column sort key was rewritten to a hidden ``:max`` - # aggregate above; order on that slot, not the bare row key. - okey = order_key_remap.get(spec.bound.value_key, spec.bound.value_key) + # A grouped row-column sort key was rewritten to a hidden direction- + # aware aggregate wrap above; order on that slot, not the bare row key. + okey = order_key_remap.get( + (spec.bound.value_key, spec.direction), spec.bound.value_key, + ) sid = projection.registry.find_by_key(okey) if sid is None: # DEV-1733: an order target that reached here without a slot would @@ -1493,13 +1632,27 @@ def _windowed_phase(bf: BoundFilter) -> Phase: f"arithmetic / scalar expression, a dimension, or declare the " f"expression as a measure and order by its name." ) - order_entries.append( - OrderEntry(slot_id=sid, direction=spec.direction), - ) + order_slot = projection.registry.get(sid) + order_entries.append(OrderEntry( + slot_id=sid, + direction=spec.direction, + scope=_classify_order_scope( + slot=order_slot, + cross_model_slot_ids={ + p.aggregate_slot_id for p in cross_model_plans + }, + windowed_slot_ids=set(windowed_slot_ids), + public_projection=projection.public_projection, + slot_by_key={ + s.key: s.id for s in projection.registry.slots + }, + ), + phase=order_slot.key.phase, + )) transform_layers = _emit_transform_layers(slots=projection.registry.slots) stage_schema = _emit_stage_schema( - query=query, projection=projection, + stage_name=query.name, projection=projection, ) # Stage 7b.10 — the active TD's slot id (``active_td_slot_id``) is resolved # right after projection above so the windowed-plan builder can use it. @@ -1557,12 +1710,12 @@ def _windowed_phase(bf: BoundFilter) -> Phase: filters_by_phase=filters_by_phase, projection=projection.public_projection, order=order_entries, - limit=query.limit, - offset=query.offset, + limit=prebound.limit, + offset=prebound.offset, stage_schema=stage_schema, active_time_dimension_slot_id=active_td_slot_id, render_source_model=render_source_model, - distinct_dimension_values=query.distinct_dimension_values, + distinct_dimension_values=distinct_dimension_values, frame_bound_columns=frame_bound_columns, outer_where_filter_ids=outer_where_filter_ids, filter_reachability=filter_reachability, @@ -1877,123 +2030,30 @@ def _format_description_for_dimension( return col.format, col.description -_COUNT_AGGREGATIONS: FrozenSet[str] = frozenset( - {"count", "count_distinct", "count_distinct_approx"} -) -_FLOAT_AGGREGATIONS: FrozenSet[str] = frozenset({ - "avg", "weighted_avg", "median", - "stddev_samp", "stddev_pop", "var_samp", "var_pop", - "corr", "covar_samp", "covar_pop", "percentile", -}) - - -def _infer_aggregated_type( - *, - model: SlayerModel, - measure_name: Optional[str], - aggregation: str, -) -> Optional[DataType]: - """Type for an aggregated measure slot. Mirrors - ``_infer_aggregated_format`` (decision #2 of the Stage B plan): - - * ``*:count`` (measure_name=``"*"``) → ``INT`` - * ``count`` / ``count_distinct`` / ``count_distinct_approx`` → ``INT`` - * ``avg`` / ``weighted_avg`` / ``median`` / parametric / stat aggs → - ``DOUBLE`` - * ``sum`` / ``min`` / ``max`` / ``first`` / ``last`` → inherit from - source column type (DOUBLE if absent). - """ - if measure_name == "*": - return DataType.INT - if aggregation in _COUNT_AGGREGATIONS: - return DataType.INT - if aggregation in _FLOAT_AGGREGATIONS: - return DataType.DOUBLE - # sum / min / max / first / last — preserve source column type. - if measure_name is None: - return None - col = model.get_column(measure_name) - if col is not None and col.type is not None: - return col.type - return None - - def _format_description_for_measure_formula( *, scope: Union[ModelScope, StageSchema], bound, ) -> Tuple[Optional[NumberFormat], Optional[str]]: - """Lift ``format`` / ``description`` for a measure formula. The - aggregation-aware format comes from ``_infer_aggregated_format`` when - the bound expression is a bare local aggregate; description follows - the source ``Column`` (sum / min / max preserve documentation). - """ + """Scope adapter over ``measure_key_format_description`` — a StageSchema + scope has no source model to lift a column's display contract from.""" if not isinstance(scope, ModelScope) or scope.source_model is None: return None, None - if not isinstance(bound.value_key, AggregateKey): - return None, None - src = bound.value_key.source - if isinstance(src, StarKey): - # ``*:count`` — INTEGER format inferred by helper; no description. - return ( - _infer_aggregated_format( - model=scope.source_model, - measure_name="*", - aggregation=bound.value_key.agg, - ), - None, - ) - if not isinstance(src, (ColumnKey, ColumnSqlKey)): - return None, None - if getattr(src, "path", ()): # cross-model — handled by response_meta - return None, None - bare = getattr(src, "leaf", None) or getattr(src, "column_name", None) - if bare is None: - return None, None - fmt = _infer_aggregated_format( - model=scope.source_model, - measure_name=bare, - aggregation=bound.value_key.agg, + return measure_key_format_description( + model=scope.source_model, key=bound.value_key, ) - col = scope.source_model.get_column(bare) - desc = col.description if col is not None else None - return fmt, desc def _type_for_measure_formula( *, scope: Union[ModelScope, StageSchema], bound, ) -> Optional[DataType]: - """Lift ``type`` for a measure-formula slot. - - Mirrors ``_format_description_for_measure_formula`` — sources the - type from ``_infer_aggregated_type`` for local aggregates so the - migrated query-backed virtual model carries ``*:count → INT``, - ``avg → DOUBLE``, ``sum → source column type``. Declared - ``ModelMeasure.type`` overrides — that flow runs through - ``expand_model_measures`` so the bound key already carries the - declared type via the source column lookup. + """Scope adapter over ``measure_key_type``. + + Declared ``ModelMeasure.type`` overrides — that flow runs through + ``expand_model_measures`` so the bound key already carries the declared + type via the source column lookup. """ if not isinstance(scope, ModelScope) or scope.source_model is None: return None - if not isinstance(bound.value_key, AggregateKey): - return None - src = bound.value_key.source - if isinstance(src, StarKey): - return _infer_aggregated_type( - model=scope.source_model, - measure_name="*", - aggregation=bound.value_key.agg, - ) - if not isinstance(src, (ColumnKey, ColumnSqlKey)): - return None - if getattr(src, "path", ()): # cross-model — handled elsewhere - return None - bare = getattr(src, "leaf", None) or getattr(src, "column_name", None) - if bare is None: - return None - return _infer_aggregated_type( - model=scope.source_model, - measure_name=bare, - aggregation=bound.value_key.agg, - ) + return measure_key_type(model=scope.source_model, key=bound.value_key) def _joined_column_type( @@ -2407,6 +2467,43 @@ def _host_model_name( return "(stage)" +def _classify_order_scope( + *, + slot: ValueSlot, + cross_model_slot_ids: Set[SlotId], + windowed_slot_ids: Set[SlotId], + public_projection: List[SlotId], + slot_by_key: Dict[ValueKey, SlotId], +) -> OrderScope: + """Name the scope that PRODUCES ``slot``'s value (DEV-1747 §5.10). + + Order matters. A slot can satisfy more than one test — the DEV-1735 order + wrap is both hidden and cross-model — and the producing scope is the + narrower fact, so isolated scopes are checked before the host base. + + A composite is classified OUTER_COMPOSITE when any operand lives in an + isolated CTE: it cannot be evaluated inside ``_base`` at all, and falling + back to a host-base scope would silently substitute a plain aggregate for + the cross-model or rolling one. + """ + if slot.id in cross_model_slot_ids: + return OrderScope.CROSS_MODEL_CTE + if slot.id in windowed_slot_ids: + return OrderScope.WINDOWED_CTE + if isinstance(slot.key, TransformKey): + return OrderScope.TRANSFORM_STEP + if isinstance(slot.key, (ArithmeticKey, ScalarCallKey)): + for dep in walk_value_keys(slot.key): + if not isinstance(dep, AggregateKey): + continue + dep_sid = slot_by_key.get(dep) + if dep_sid in cross_model_slot_ids or dep_sid in windowed_slot_ids: + return OrderScope.OUTER_COMPOSITE + if slot.hidden or slot.id not in public_projection: + return OrderScope.HOST_BASE_HIDDEN + return OrderScope.HOST_BASE + + def _bucket_slots(slots: List[ValueSlot]): row: List[ValueSlot] = [] agg: List[ValueSlot] = [] @@ -2423,7 +2520,7 @@ def _bucket_slots(slots: List[ValueSlot]): def _emit_stage_schema( *, - query: SlayerQuery, + stage_name: Optional[str], projection, ) -> StageSchema: """Build the StageSchema from the projection plan. @@ -2474,8 +2571,9 @@ def _emit_stage_schema( format=slot.format, description=slot.description, )) - relation_name = query.name or "(unnamed_stage)" - return StageSchema(relation_name=relation_name, columns=columns) + return StageSchema( + relation_name=stage_name or "(unnamed_stage)", columns=columns, + ) def _emit_transform_layers(*, slots: List[ValueSlot]) -> List[TransformLayer]: diff --git a/slayer/sql/dialects/base.py b/slayer/sql/dialects/base.py index 7d18da52..20ed4ed7 100644 --- a/slayer/sql/dialects/base.py +++ b/slayer/sql/dialects/base.py @@ -219,6 +219,35 @@ def build_null_safe_eq( """ return exp.NullSafeEQ(this=left, expression=right) + # ------------------------------------------------------------------ + # ORDER BY term construction (DEV-1747 D5 / P-H) + # ------------------------------------------------------------------ + + def build_ordered( + self, + order_col: exp.Expression, + *, + descending: bool, + nulls: str = "default", + ) -> exp.Ordered: + """Build one ``ORDER BY`` term with its null-ordering policy applied. + + The single place any render site turns a resolved column plus a + direction into an ``exp.Ordered`` (P-H). It previously lived on the + generator as ``_ordered``, which meant the combined and transform-chain + paths — which built their own ``exp.Ordered`` — silently skipped the + T-SQL pin below. + + ``nulls="default"`` defers to the dialect's own ordering and emits no + NULLS clause; ``"first"`` / ``"last"`` are explicit. + """ + kwargs: dict = {"this": order_col, "desc": descending} + if nulls == "first": + kwargs["nulls_first"] = True + elif nulls == "last": + kwargs["nulls_first"] = False + return exp.Ordered(**kwargs) + @staticmethod def _expanded_null_safe_eq( left: exp.Expression, right: exp.Expression, diff --git a/slayer/sql/dialects/tsql.py b/slayer/sql/dialects/tsql.py index 24907cf5..d998bbed 100644 --- a/slayer/sql/dialects/tsql.py +++ b/slayer/sql/dialects/tsql.py @@ -78,6 +78,33 @@ def build_null_safe_eq( portable expanded ``a = b OR (a IS NULL AND b IS NULL)``.""" return self._expanded_null_safe_eq(left, right) + def build_ordered( + self, + order_col: exp.Expression, + *, + descending: bool, + nulls: str = "default", + ) -> exp.Ordered: + """DEV-1571 Bug 2 / DEV-1716 — pin ``nulls_first`` to T-SQL's native + default for the direction (FIRST on ASC, LAST on DESC). + + Left unset, sqlglot emits ``CASE WHEN IS NULL THEN 1 ELSE 0 + END, `` to emulate NULLS ordering; the bracketed alias INSIDE + the CASE WHEN mis-resolves against the FROM scope (``Invalid column + name``). Pinning the native default suppresses the wrapper without + changing the ordering. + + An EXPLICIT ``first`` / ``last`` policy is honoured as asked — the pin + exists to avoid the emulation, not to override a stated intent. + """ + if nulls == "default": + return exp.Ordered( + this=order_col, desc=descending, nulls_first=not descending, + ) + return super().build_ordered( + order_col, descending=descending, nulls=nulls, + ) + def build_approx_count_distinct( self, col_sql: str, diff --git a/slayer/sql/generator.py b/slayer/sql/generator.py index 6a227414..93b874b5 100644 --- a/slayer/sql/generator.py +++ b/slayer/sql/generator.py @@ -459,6 +459,27 @@ def _wrap_filter(sql_str: str, filter_sql: Optional[str]) -> str: return f"(CASE WHEN {filter_sql} THEN {sql_str} END)" +def _is_host_grain(key) -> bool: + """True for an ``AggregateKey`` marked ``grain="host"`` (DEV-1747 D2). + + The marker separates WHERE a value is READ from WHERE it is GROUPED: the + source ``path`` says the value comes through a join, ``grain="host"`` says + the aggregate is nonetheless computed per HOST row-group. Such a key + renders INLINE over the joined relation inside its own scope, rather than + in a target-rooted CTE that would collapse it to one global value. + """ + return getattr(key, "grain", "target") == "host" + + +def _host_grain_join_alias(path) -> str: + """The FROM alias a join ``path`` is emitted under. + + Mirrors ``_build_from_and_joins``: the first hop uses the target's bare + name, later hops the ``__``-delimited path alias. + """ + return "__".join(path) + + def _first_bare_column_name(key) -> Optional[str]: """Return the leaf name of the first bare column reference inside a ROW-phase composite key (DEV-1576 / DEV-1717 error messages). @@ -971,21 +992,20 @@ def _null_safe_join_pair_sql(self, *, left_sql: str, right_sql: str) -> str: right = self._parse(right_sql) return self._dialect.build_null_safe_eq(left, right).sql(dialect=self.dialect) - def _ordered(self, order_col: exp.Expression, *, ascending: bool) -> exp.Ordered: - """Build an ``exp.Ordered`` node, suppressing sqlglot's NULLS-emulation - ``CASE WHEN`` on T-SQL (DEV-1571 Bug 2 / DEV-1716). + def _ordered( + self, order_col: exp.Expression, *, ascending: bool, + nulls: str = "default", + ) -> exp.Ordered: + """Build an ``exp.Ordered`` node via the dialect strategy. - On T-SQL, sqlglot emits ``CASE WHEN IS NULL THEN 1 ELSE 0 END, - `` to emulate NULLS ordering whenever ``nulls_first`` is unset; - the bracketed alias INSIDE the CASE WHEN mis-resolves against the FROM - scope (``Invalid column name``). Pinning ``nulls_first`` to T-SQL's - native default for the direction (FIRST on ASC, LAST on DESC) - suppresses the wrapper. No-op on every other dialect. + DEV-1747 D5 — the T-SQL ``nulls_first`` pin used to live here, which + left the combined and transform-chain paths (which build their own + ``exp.Ordered``) without it. It now lives in ``SqlDialect.build_ordered`` + so every render site gets identical null ordering (P-H). """ - kwargs: dict = {"this": order_col, "desc": not ascending} - if self.dialect == "tsql": - kwargs["nulls_first"] = ascending - return exp.Ordered(**kwargs) + return self._dialect.build_ordered( + order_col, descending=not ascending, nulls=nulls, + ) @@ -2523,6 +2543,7 @@ def _ready(key) -> bool: def _resolve_agg_inputs_via_scope( # NOSONAR(S3776) — one cohesive Law-1 discovery pass: three ordered sub-passes (Column.filter → source → kwargs) over the local aggregates via small closures sharing scope/resolved. Extracting them would scatter the ordered-registration contract that keeps the base FROM byte-identical. self, *, base_render_order, slots_by_id, scope: ScopeFrame, + skip_cross_model_aggs: bool = False, ) -> "Dict[Any, Dict[str, ResolvedAggKwarg]]": """Resolve every LOCAL aggregate's join-crossing inputs through the host ``scope`` (Law 1) — ``scope.resolve`` anchors each ref and registers the @@ -2573,7 +2594,16 @@ def _resolve_agg_inputs_via_scope( # NOSONAR(S3776) — one cohesive Law-1 disc def _walk(key, fn) -> None: if isinstance(key, AggregateKey): - if not getattr(key.source, "path", ()): + # DEV-1747 D2 — a HOST-GRAIN aggregate reads through a join but + # is grouped at the host grain, so it renders INLINE here and + # its source join has to register like any other crossing input + # (Law 1). When the caller owns it in a ``_cm_*`` CTE + # (``skip_cross_model_aggs``) the join belongs to that CTE, not + # to this base — registering it here would add an unused, and + # for a one-to-many join cardinality-changing, LEFT JOIN. + if not getattr(key.source, "path", ()) or ( + _is_host_grain(key) and not skip_cross_model_aggs + ): fn(key) elif isinstance(key, ArithmeticKey): for o in key.operands: @@ -2603,6 +2633,9 @@ def _resolve_column_filter(key) -> None: def _resolve_source(key) -> None: if isinstance(key.source, ColumnSqlKey): scope.resolve(key.source) # register-only; render re-expands + elif getattr(key.source, "path", ()): + # DEV-1747 D2 — the host-grain source IS the crossing input. + scope.resolve(key.source) # register-only def _resolve_kwargs(key) -> None: kw: Dict[str, ResolvedAggKwarg] = {} @@ -2789,6 +2822,7 @@ def _build_base_select_for_planned( # NOSONAR(S3776) — join-path collection a base_render_order=base_render_order, slots_by_id=slots_by_id, scope=host_scope, + skip_cross_model_aggs=skip_cross_model_aggs, ) # Merge the scope's registered paths (positions 2-7, in first-seen order) # after the dimension paths (position 1) → byte-identical FROM. @@ -2948,16 +2982,22 @@ def _record_alias(sid: str, full_alias: str) -> None: agg_path = getattr(key.source, "path", ()) if agg_path: if skip_cross_model_aggs: - # Cross-model aggregate; rendered by the per-plan - # ``_cm_*`` CTE. Skip in the host base. + # Owned by a per-plan ``_cm_*`` CTE — target-rooted or + # (DEV-1747 D2) host-rooted. Skip in the host base. continue - raise NotImplementedError( - f"DEV-1450 stage 7b.12: cross-model aggregate " - f"(source.path={agg_path!r}) reached the local " - f"base SELECT path. The cross-model orchestrator " - f"should have routed this through `_render_with_" - f"cross_model_plans`." - ) + if not _is_host_grain(key): + raise NotImplementedError( + f"DEV-1450 stage 7b.12: cross-model aggregate " + f"(source.path={agg_path!r}) reached the local " + f"base SELECT path. The cross-model orchestrator " + f"should have routed this through `_render_with_" + f"cross_model_plans`." + ) + # DEV-1747 D2 — a HOST-GRAIN aggregate inside its own CTE: + # the crossed join is already in this scope's FROM, so the + # aggregate renders inline over the joined relation and + # GROUPs at the query grain. This is the base-pull the + # recursion guard exists to reach. # DEV-1450 stage 7b.12: ``column_filter_key`` is now # propagated into the synthetic EnrichedMeasure's # ``filter_sql`` field so ``_build_agg`` wraps the @@ -9040,6 +9080,20 @@ def _validate_aggregate_kwarg_paths( ), ) + def _walk_join_path_model(self, *, source_model, path, bundle): + """The terminal model of a join ``path`` walked from ``source_model``, + or ``None`` if any hop is missing. Non-raising: callers use it to + re-anchor a reference that the planner has already validated.""" + current = source_model + for hop in path: + if not any(j.target_model == hop for j in current.joins): + return None + nxt = bundle.get_referenced_model(hop) + if nxt is None: + return None + current = nxt + return current + def _build_agg_render_spec_from_planned( # NOSONAR(S3776) — sequential isinstance dispatch over StarKey / ColumnKey / ColumnSqlKey with helper extractions for aggregation-def lookup, kwarg path validation, and explicit-time-arg resolution. Further splitting would scatter the per-source-kind contract. self, *, @@ -9094,6 +9148,18 @@ def _build_agg_render_spec_from_planned( # NOSONAR(S3776) — sequential isinst type=slot_type, ) if isinstance(source, (ColumnKey, ColumnSqlKey)): + # DEV-1747 D2 — a HOST-GRAIN aggregate reads its source THROUGH a + # join, so the column lives on the terminal model and qualifies to + # that join's FROM alias. Re-anchor before the lookup below; the + # join itself is already in this scope's FROM, registered by the + # aggregate-input scope pass. + if source.path and _is_host_grain(key) and bundle is not None: + terminal = self._walk_join_path_model( + source_model=source_model, path=source.path, bundle=bundle, + ) + if terminal is not None: + source_model = terminal + source_relation = _host_grain_join_alias(source.path) # ColumnKey is a bare / trivial column (``sql`` None or a bare # identifier remap); ColumnSqlKey is a derived column (``Column.sql`` # set to a non-trivial expression — ``amount * 2``). Both resolve diff --git a/slayer/sql/naming.py b/slayer/sql/naming.py index ce9e555a..d41e22f6 100644 --- a/slayer/sql/naming.py +++ b/slayer/sql/naming.py @@ -380,6 +380,14 @@ def canonical_aggregate_alias( # NOSONAR(S3776) — sequential dispatch over th } or None, ) + # DEV-1747 D2 — a host-grain aggregate is a DIFFERENT value from the + # target-grain one over the same column (per host group vs global), and + # ``grain`` is part of the key's identity, so the two intern as separate + # slots. Without a distinct alias those slots would collide on one output + # column name and the renderer would emit whichever it wrote last. + if getattr(key, "grain", "target") == "host": + canonical = f"{canonical}_host" + # --- prefix, per profile --- if profile in ("cte_schema", "declared_name"): return canonical diff --git a/tests/test_dev1645_invalid_postgres_sql.py b/tests/test_dev1645_invalid_postgres_sql.py index 92d36505..50ad7bf8 100644 --- a/tests/test_dev1645_invalid_postgres_sql.py +++ b/tests/test_dev1645_invalid_postgres_sql.py @@ -28,7 +28,6 @@ import sqlglot from slayer.core.enums import DataType, TimeGranularity -from slayer.core.errors import UnresolvableOrderColumnError from slayer.core.models import Column, ModelJoin, SlayerModel from slayer.core.query import ColumnRef, OrderItem, SlayerQuery, TimeDimension from slayer.sql.generator import SQLGenerator @@ -229,26 +228,35 @@ async def test_orderby_projected_alias_combined_cte_path_unchanged(self) -> None assert 'ORDER BY "accts.clusters.score_sum" DESC' in sql, sql assert "ORDER BY accts." not in sql, sql - async def test_orderby_unresolvable_joined_column_rejected(self) -> None: - """Ordering by an unprojected multi-hop joined column whose join was - never pulled into scope cannot bind to any FROM table — reject at - compile time rather than emit invalid SQL (DEV-1645, user decision on - Codex review of PR #224).""" + async def test_orderby_unprojected_joined_column_resolves_host_rooted( + self, + ) -> None: + """DEV-1645 rejected an unprojected multi-hop joined ORDER BY because it + could not bind to any FROM table. DEV-1747 D2 gives it a binding: a + HOST-rooted CTE computes the per-group extreme over the crossed join and + the outer query orders on that CTE's column, so nothing is unbound.""" orders, joined = _orders_customers_regions() query = SlayerQuery( source_model="orders", measures=[{"formula": "amount:sum", "name": "rev"}], order=[OrderItem(column=ColumnRef(name="name", model="customers.regions"), direction="desc")], ) - with pytest.raises(UnresolvableOrderColumnError): - await _engine_generate(query=query, model=orders, extra_models=joined) - - async def test_orderby_joined_column_rejected_even_when_filter_pulls_join_in(self) -> None: - """An unprojected joined ORDER BY is rejected even when a filter pulls - the join into the base FROM: the compiler's outer-wrapping layers - (measure CTEs, pagination, first/last ranked, projection trim) relocate - the ORDER BY into a scope where the joined table is unbound, so resolving - it is unsafe. Project the column or order by a projected field instead.""" + sql = _norm(await _engine_generate( + query=query, model=orders, extra_models=joined, + )) + # The sort key's join lives in the CTE, never in the host base — that + # containment is what keeps ``SUM(orders.amount)`` unmultiplied. + base = sql.split("_cm_")[0] + assert "JOIN" not in base.upper(), sql + assert "MAX(customers__regions.name)" in sql, sql + assert "ORDER BY _cm_" in sql, sql + + async def test_orderby_joined_column_resolves_when_filter_pulls_join_in( + self, + ) -> None: + """The same resolution when a filter already pulled the join into the + host base. The CTE re-applies the filter rather than depending on the + base's copy, so the two scopes agree on which rows the extreme is over.""" orders, joined = _orders_customers_regions() query = SlayerQuery( source_model="orders", @@ -256,15 +264,18 @@ async def test_orderby_joined_column_rejected_even_when_filter_pulls_join_in(sel filters=["customers.regions.name == 'US'"], order=[OrderItem(column=ColumnRef(name="name", model="customers.regions"), direction="desc")], ) - with pytest.raises(UnresolvableOrderColumnError): - await _engine_generate(query=query, model=orders, extra_models=joined) + sql = _norm(await _engine_generate( + query=query, model=orders, extra_models=joined, + )) + assert sql.count("customers__regions.name = 'US'") == 2, sql + assert "ORDER BY _cm_" in sql, sql - async def test_orderby_joined_column_rejected_in_cte_wrapped_scope(self) -> None: - """A joined column can be resolved in the base SELECT (joins in FROM), - but NOT in a CTE-wrapped scope (measure CTEs / windowed measures order - from `_base`, where the join alias is unbound). Even when a filter pulls - the join in, ordering by that joined column in the combined-CTE path must - reject rather than emit an unbound reference (Codex review of PR #224).""" + async def test_orderby_joined_column_resolves_in_cte_wrapped_scope( + self, + ) -> None: + """The CTE-wrapped (windowed-measure) path. The sort key's CTE groups on + the query grain and joins back NULL-safely, so the outer ORDER BY names a + column the wrapper actually projects.""" orders, joined = _orders_customers_regions() query = SlayerQuery( source_model="orders", @@ -273,14 +284,18 @@ async def test_orderby_joined_column_rejected_in_cte_wrapped_scope(self) -> None filters=["customers.regions.name == 'US'"], # pulls the join into resolved_joins order=[OrderItem(column=ColumnRef(name="name", model="customers.regions"), direction="desc")], ) - with pytest.raises(UnresolvableOrderColumnError): - await _engine_generate(query=query, model=orders, extra_models=joined) + sql = _norm(await _engine_generate( + query=query, model=orders, extra_models=joined, + )) + assert "IS NOT DISTINCT FROM" in sql, sql + assert 'ORDER BY "orders.customers.regions.name_max_host" DESC' in sql, sql - async def test_orderby_joined_column_rejected_in_first_last_ranked_scope(self) -> None: - """first/last measures wrap the FROM (with its joins) in a ranked - subquery that only re-exposes model.* — so the outer ORDER BY can't see - a joined column even when a filter pulled the join in. Must reject, not - emit an unbound reference (Codex review of PR #224).""" + async def test_orderby_joined_column_resolves_in_first_last_ranked_scope( + self, + ) -> None: + """The first/last ranked-subquery path. The ranked wrap only re-exposes + ``model.*``, which is precisely why the sort key cannot be a bare joined + reference — it is computed in its own CTE instead.""" orders, joined = _orders_customers_regions() query = SlayerQuery( source_model="orders", @@ -288,47 +303,31 @@ async def test_orderby_joined_column_rejected_in_first_last_ranked_scope(self) - filters=["customers.regions.name == 'US'"], # pulls the join into resolved_joins order=[OrderItem(column=ColumnRef(name="name", model="customers.regions"), direction="desc")], ) - with pytest.raises(UnresolvableOrderColumnError): - await _engine_generate(query=query, model=orders, extra_models=joined) + sql = _norm(await _engine_generate( + query=query, model=orders, extra_models=joined, + )) + assert "_last_rn" in sql, sql + assert "ORDER BY _cm_" in sql, sql # ============================================================================ -# Flavor A — DEFERRED capability: joined / cross-model ORDER BY resolution +# Flavor A — joined / cross-model ORDER BY resolution (was DEFERRED) # ============================================================================ -class TestFlavorAJoinedOrderByDeferred: +class TestFlavorAJoinedOrderByResolves: """Ordering by an unprojected JOINED column in a GROUPED query. - DEV-1645 originally rejected every unprojected joined/cross-model ORDER BY - (``UnresolvableOrderColumnError``). DEV-1703 Phase 1 narrowed that: an - order-only ref now resolves like a filter ref, so a joined sort key in a - RAW-ROWS query pulls its join (Law 1) and split-emits, and a LOCAL row - column in a grouped query materialises a hidden ``:max`` wrap. - - All three queries below are GROUPED with a JOINED sort key — the one shape - still rejected, because it has no host-rooted representation today (see - ``_REASON``). They stay ``strict=True`` so they flip to XPASS the moment - DEV-1735 lands, prompting removal of the xfail and the reject. The - companion tests in ``TestFlavorAOrderByUnprojected`` pin the reject - contract; ``tests/test_dev1712_order_only_hidden_slots.py`` pins the - resolved shapes.""" - - _REASON = ( - "DEV-1735: joined ORDER BY resolution in a GROUPED query is deferred. " - "DEV-1703 Phase 1 resolved the raw-rows case (Law-1 join pull + split " - "emission) and the grouped LOCAL case (hidden ``:max`` wrap), but a " - "grouped JOINED sort key has no host-rooted representation — an " - "AggregateKey with a non-empty source.path always routes to a " - "target-rooted CTE, which degenerates to a scalar CROSS JOIN whose " - "value is constant per group, so the sort would silently do nothing. " - "Rejecting loudly is preferred until DEV-1735 lands host-rooted " - "crossing MAX." - ) + DEV-1645 rejected every unprojected joined / cross-model ORDER BY. DEV-1703 + Phase 1 narrowed that (raw-rows split emission; a grouped LOCAL column's + hidden wrap), and DEV-1747 D2 closed the remainder. - @pytest.mark.xfail(strict=True, reason=_REASON) - async def test_joined_orderby_with_filter_in_scope_should_resolve(self) -> None: - """A filter pulls the join into the base FROM; ordering by that joined - column should resolve to the canonical ``__`` alias.""" + These three cases were ``xfail(strict=True)`` aspiring to a BARE split + reference, ``ORDER BY customers__regions.name``. That aspiration was wrong + for a grouped query — the column is not in GROUP BY, so a bare reference is + invalid SQL on every Tier-1 dialect. The shape below is what the value + actually requires: computed per group in its own scope, then joined back.""" + + async def test_joined_orderby_with_filter_in_scope_resolves(self) -> None: orders, joined = _orders_customers_regions() query = SlayerQuery( source_model="orders", @@ -339,12 +338,13 @@ async def test_joined_orderby_with_filter_in_scope_should_resolve(self) -> None: sql = _norm(await _engine_generate( query=query, model=orders, extra_models=joined, )) - assert "ORDER BY customers__regions.name" in sql + assert "MAX(customers__regions.name)" in sql, sql - @pytest.mark.xfail(strict=True, reason=_REASON) - async def test_joined_orderby_without_filter_should_pull_join_and_resolve(self) -> None: - """Ordering by a joined column with no other reference should pull the - join in (like filters do) and resolve, rather than reject.""" + async def test_joined_orderby_without_filter_pulls_join_into_the_cte( + self, + ) -> None: + """With nothing else referencing the join, it is pulled into the sort + key's CTE — and only there.""" orders, joined = _orders_customers_regions() query = SlayerQuery( source_model="orders", @@ -354,12 +354,10 @@ async def test_joined_orderby_without_filter_should_pull_join_and_resolve(self) sql = _norm(await _engine_generate( query=query, model=orders, extra_models=joined, )) - assert "ORDER BY customers__regions.name" in sql + assert "MAX(customers__regions.name)" in sql, sql + assert "JOIN" not in sql.split("_cm_")[0].upper(), sql - @pytest.mark.xfail(strict=True, reason=_REASON) - async def test_joined_orderby_in_cte_wrapped_scope_should_resolve(self) -> None: - """A windowed-measure (combined-CTE) query that filters on and orders by - a joined column should eventually resolve it in the outer scope.""" + async def test_joined_orderby_in_cte_wrapped_scope_resolves(self) -> None: orders, joined = _orders_customers_regions() query = SlayerQuery( source_model="orders", @@ -371,13 +369,7 @@ async def test_joined_orderby_in_cte_wrapped_scope_should_resolve(self) -> None: sql = _norm(await _engine_generate( query=query, model=orders, extra_models=joined, )) - # Assert on the ORDER BY clause specifically (not a bare substring that - # the ``== 'US'`` filter's WHERE would satisfy): the aspiration is that - # the joined sort key resolves to the ``__`` path alias. On the typed - # pipeline the ORDER BY still emits the unprojected dotted alias - # ``"customers.regions.name"`` (the DEV-1645 gap → Stages 8-9), so this - # correctly xfails until joined ORDER BY resolution lands. - assert "ORDER BY customers__regions.name" in sql + assert 'ORDER BY "orders.customers.regions.name_max_host" DESC' in sql, sql # ============================================================================ diff --git a/tests/test_dev1712_order_only_hidden_slots.py b/tests/test_dev1712_order_only_hidden_slots.py index f8feb15c..f499d947 100644 --- a/tests/test_dev1712_order_only_hidden_slots.py +++ b/tests/test_dev1712_order_only_hidden_slots.py @@ -284,10 +284,13 @@ async def test_split_key_preserves_asc_and_limit(self, engine) -> None: # =========================================================================== class TestGroupedLocalRowColumnMaxWrapped: """DEV-1703 Phase 1 supersedes Stage 8's rejection: a LOCAL row column - ordered in a GROUPED query materialises as a hidden ``:max`` - aggregate and orders on that alias. MAX is order-preserving per group and - portable across every Tier-1 dialect. The wrap is a hidden slot, so it is - trimmed from the public projection.""" + ordered in a GROUPED query materialises as a hidden aggregate wrap and + orders on that alias. The wrap is a hidden slot, so it is trimmed from the + public projection. + + DEV-1747 D10 made the wrap DIRECTION-AWARE — ``MIN`` on ASC, ``MAX`` on + DESC — so each group is ordered by the extreme the direction actually puts + first. The tests below are DESC unless stated otherwise, hence MAX.""" async def test_dedup_on_row_column_order_max_wraps(self, engine) -> None: query = SlayerQuery( @@ -378,7 +381,10 @@ async def test_max_wrap_bypasses_the_aggregation_gate(self, engine) -> None: order=[OrderItem(column=ColumnRef(name="id"), direction="asc")], ) sql = await _sql(engine, query) - assert re.search(r"MAX\(\s*orders\.id\s*\)", sql), sql + # DEV-1747 D10 — the wrap is direction-aware: ASC sorts each group by + # its MINIMUM. The point of this test is the aggregation GATE, which + # the wrap bypasses regardless of which extreme it takes. + assert re.search(r"MIN\(\s*orders\.id\s*\)", sql), sql # =========================================================================== @@ -408,34 +414,27 @@ async def test_joined_row_column_ungrouped_pulls_join_and_splits( # The sort key is not projected — ordering must not change the shape. assert _outer_select_columns(sql) == ["orders.status"], sql - async def test_joined_row_column_grouped_raises(self, engine) -> None: - query = SlayerQuery( - source_model="orders", - dimensions=[ColumnRef(name="status")], - measures=[ModelMeasure(formula="*:count")], - order=[OrderItem(column=ColumnRef(name="customers.region"), direction="desc")], - ) - with pytest.raises(UnresolvableOrderColumnError): - await _sql(engine, query) - - async def test_joined_reject_message_does_not_duplicate_qualifier(self, engine) -> None: - """The rejection message names the column once (``customers.region``), - not a duplicated ``customers.customers.region`` (CodeRabbit). + async def test_joined_row_column_grouped_resolves_host_rooted( + self, engine, + ) -> None: + """DEV-1747 D2: the GROUPED shape used to be rejected outright, because + a path-bearing aggregate source always routed to a TARGET-rooted CTE — + a scalar CROSS JOIN that gives every group the same global value. - Uses the GROUPED shape — the ungrouped one now resolves (DEV-1703 - Phase 1), so the message contract is pinned where the reject survives. - """ + It now resolves via a HOST-rooted CTE: the crossed join is pulled + INSIDE, the CTE groups on the query grain, and the host base joins the + per-group extreme back. The sort key is still not projected.""" query = SlayerQuery( source_model="orders", dimensions=[ColumnRef(name="status")], measures=[ModelMeasure(formula="*:count")], order=[OrderItem(column=ColumnRef(name="customers.region"), direction="desc")], ) - with pytest.raises(UnresolvableOrderColumnError) as ei: - await _sql(engine, query) - msg = str(ei.value) - assert "customers.region" in msg - assert "customers.customers" not in msg + sql = await _sql(engine, query) + assert _outer_select_columns(sql) == ["orders.status", "orders._count"], sql + assert "WITH" in sql.upper(), sql + # DESC takes each group's MAXIMUM (D10). + assert re.search(r"(?i)\bMAX\s*\(", sql), sql async def test_ungrouped_order_by_derived_crossing_column_rejected(self, engine) -> None: """A hidden order-only LOCAL DERIVED column whose ``Column.sql`` crosses @@ -451,11 +450,17 @@ async def test_ungrouped_order_by_derived_crossing_column_rejected(self, engine) with pytest.raises(UnresolvableOrderColumnError): await _sql(engine, query) - async def test_joined_order_ref_colliding_local_leaf_raises(self, tmp_path) -> None: + async def test_joined_order_ref_colliding_local_leaf_stays_joined( + self, tmp_path, + ) -> None: """A joined order ref whose LEAF collides with a local declared dimension (``owners.status`` vs a local ``status``) must NOT silently - bind to the local column and sort by the wrong field — it is a joined - ref and is rejected (Codex / DEV-1712).""" + bind to the local column and sort by the wrong field (Codex / + DEV-1712). + + DEV-1747 D2 turned the rejection into a resolution, so the guarantee is + now pinned positively: the emitted SQL must actually reach ``owners``. + A silent rebind to the local column would leave it absent.""" storage = YAMLStorage(base_dir=str(tmp_path)) await storage.save_datasource( DatasourceConfig(name="test", type="sqlite", database=":memory:") @@ -484,14 +489,11 @@ async def test_joined_order_ref_colliding_local_leaf_raises(self, tmp_path) -> N measures=[ModelMeasure(formula="*:count")], order=[OrderItem(column="owners.status", direction="desc")], # joined ) - with pytest.raises(UnresolvableOrderColumnError): - resp = await engine.execute(query, dry_run=True) - # If it did not raise, it must at least NOT have silently sorted by - # the local column (which would be the bug). - assert "owners" in (resp.sql or ""), ( - f"joined order ref silently bound to the local column.\n" - f"SQL:\n{resp.sql}" - ) + resp = await engine.execute(query, dry_run=True) + assert "owners" in (resp.sql or ""), ( + f"joined order ref silently bound to the local column.\n" + f"SQL:\n{resp.sql}" + ) # =========================================================================== diff --git a/tests/test_dev1733_order_only_transform_composite.py b/tests/test_dev1733_order_only_transform_composite.py index 04e3588a..894370c2 100644 --- a/tests/test_dev1733_order_only_transform_composite.py +++ b/tests/test_dev1733_order_only_transform_composite.py @@ -52,7 +52,6 @@ from slayer.core.errors import ( DistinctDimensionValuesError, UnknownReferenceError, - UnresolvableOrderColumnError, ) from slayer.core.models import Column, DatasourceConfig, ModelJoin, ModelMeasure, SlayerModel from slayer.core.query import ColumnRef, OrderItem, SlayerQuery, TimeDimension @@ -1006,15 +1005,24 @@ async def test_non_sum_avg_windowed_order_target_still_raises(self, engine) -> N # must not swallow the Stage-8 rejections. # =========================================================================== class TestStillRejected: - async def test_joined_row_column_order_still_raises(self, engine) -> None: + async def test_joined_row_column_order_resolves_host_rooted( + self, engine, + ) -> None: + """DEV-1747 D2 replaced this rejection with a host-rooted CTE. What + Group 8 still guards is that widening the hidden-order branch did not + change the GRAIN: the sort key must not join the base or reach the + projection.""" query = SlayerQuery( source_model="orders", dimensions=[ColumnRef(name="status")], measures=[ModelMeasure(formula="*:count")], order=[OrderItem(column="customers.region", direction="desc")], ) - with pytest.raises(UnresolvableOrderColumnError): - await _sql(engine, query) + sql = await _sql(engine, query) + parsed = _outermost_select(sql, dialect="sqlite") + assert [e.alias_or_name for e in parsed.expressions] == [ + "orders.status", "orders._count", + ], sql async def test_ungrouped_row_column_still_splits(self, engine) -> None: """The DEV-1712 split-emission path must be untouched.""" @@ -1126,7 +1134,9 @@ def test_hidden_order_branch_rejects_unlisted_slot_kinds(self) -> None: is not in the GROUP BY. """ from slayer.core.keys import ColumnKey, Phase - from slayer.engine.planned import OrderEntry, PlannedQuery, ValueSlot + from slayer.engine.planned import ( + OrderEntry, OrderScope, PlannedQuery, ValueSlot, + ) from slayer.sql.generator import SQLGenerator slot = ValueSlot( @@ -1139,7 +1149,11 @@ def test_hidden_order_branch_rejects_unlisted_slot_kinds(self) -> None: planned = PlannedQuery( source_relation="orders", row_slots=[slot], - order=[OrderEntry(slot_id="s0", direction="asc")], + order=[OrderEntry( + slot_id="s0", direction="asc", + # DEV-1747 §5.10 — classification is required on every entry. + scope=OrderScope.HOST_BASE_HIDDEN, phase=Phase.ROW, + )], ) # Everything that can throw is built OUTSIDE the raises block, so the # only invocation under test is the call itself. diff --git a/tests/test_nested_dag_cross_stage_refs.py b/tests/test_nested_dag_cross_stage_refs.py index a9d810e7..0112bec6 100644 --- a/tests/test_nested_dag_cross_stage_refs.py +++ b/tests/test_nested_dag_cross_stage_refs.py @@ -1299,7 +1299,9 @@ async def test_intercepted_cmm_order_only_bare_grouped_max_wraps( hidden ``:max`` over the stage column and orders on that alias. The stage boundary is the point of this test: the wrap must resolve - against the ``s1`` rowset (``MAX(s1.customers__revenue_sum)``), never + against the ``s1`` rowset (``MIN(s1.customers__revenue_sum)`` — the + order defaults to ASC and DEV-1747 D10 made the wrap direction-aware), + never reach back into s1's own source tables, and must not widen the outer grain or leak into the public projection.""" engine, tmp = await _engine_with_join_chain() @@ -1319,12 +1321,12 @@ async def test_intercepted_cmm_order_only_bare_grouped_max_wraps( resp = await engine.execute(query=[inner, outer], dry_run=True) sql = " ".join(resp.sql.split()) # Wrapped against the STAGE rowset, not s1's underlying tables. - assert "MAX(s1.customers__revenue_sum)" in sql, sql + assert "MIN(s1.customers__revenue_sum)" in sql, sql # Outer grain unchanged, wrap trimmed from the public projection. assert "GROUP BY s1.customers__regions__name" in sql, sql - assert 'ORDER BY "s1.customers__revenue_sum_max"' in sql, sql + assert 'ORDER BY "s1.customers__revenue_sum_min"' in sql, sql outer_select = sql.rsplit(") AS _outer", 1)[0].rsplit("SELECT", 1)[0] - assert "customers__revenue_sum_max" not in outer_select.split( + assert "customers__revenue_sum_min" not in outer_select.split( "FROM (" )[0], sql finally: diff --git a/tests/test_planned.py b/tests/test_planned.py index 07b93325..b5cfd4d5 100644 --- a/tests/test_planned.py +++ b/tests/test_planned.py @@ -29,6 +29,7 @@ FilterPhase, JoinRequirement, OrderEntry, + OrderScope, PlannedQuery, TransformLayer, ValueSlot, @@ -312,25 +313,47 @@ def test_post_phase(self): class TestOrderEntry: + # DEV-1747 §5.10 — ``scope`` and ``phase`` are required with no default, so + # a planner path that forgets to classify fails at construction instead of + # falling through to the ``_base.``-qualified render branch. + _CLASSIFIED = {"scope": OrderScope.HOST_BASE, "phase": Phase.ROW} + def test_asc(self): - o = OrderEntry(slot_id="s1", direction="asc") + o = OrderEntry(slot_id="s1", direction="asc", **self._CLASSIFIED) assert o.direction == "asc" def test_desc(self): - o = OrderEntry(slot_id="s1", direction="desc") + o = OrderEntry(slot_id="s1", direction="desc", **self._CLASSIFIED) assert o.direction == "desc" + def test_scope_and_phase_are_required(self): + with pytest.raises(ValueError): + OrderEntry(slot_id="s1", direction="asc") # type: ignore[call-arg] + + def test_nulls_defaults_to_the_dialect_default(self): + o = OrderEntry(slot_id="s1", direction="asc", **self._CLASSIFIED) + assert o.nulls == "default" + def test_invalid_direction_rejected(self): with pytest.raises(ValueError): - OrderEntry(slot_id="s1", direction="random") # type: ignore[arg-type] + OrderEntry( + slot_id="s1", direction="random", # type: ignore[arg-type] + **self._CLASSIFIED, + ) def test_uppercase_direction_rejected(self): # OrderEntry is planner-produced — strict lowercase is intentional. # If user input ever feeds it directly, the caller must lowercase. with pytest.raises(ValueError): - OrderEntry(slot_id="s1", direction="ASC") # type: ignore[arg-type] + OrderEntry( + slot_id="s1", direction="ASC", # type: ignore[arg-type] + **self._CLASSIFIED, + ) with pytest.raises(ValueError): - OrderEntry(slot_id="s1", direction="DESC") # type: ignore[arg-type] + OrderEntry( + slot_id="s1", direction="DESC", # type: ignore[arg-type] + **self._CLASSIFIED, + ) # --------------------------------------------------------------------------- @@ -460,6 +483,9 @@ def test_transform_layer_in_planned(self): assert pq.transform_layers == [layer] def test_order_in_planned(self): - oe = OrderEntry(slot_id="s1", direction="desc") + oe = OrderEntry( + slot_id="s1", direction="desc", + scope=OrderScope.HOST_BASE, phase=Phase.AGGREGATE, + ) pq = PlannedQuery(source_relation="orders", order=[oe]) assert pq.order == [oe] diff --git a/tests/test_sql_generator.py b/tests/test_sql_generator.py index 1e2e0468..dc012a90 100644 --- a/tests/test_sql_generator.py +++ b/tests/test_sql_generator.py @@ -7517,8 +7517,10 @@ async def test_hidden_row_order_target_max_wraps_without_widening_grain( History: this raised ``NotImplementedError``, then (DEV-1712 Stage 8) a plan-time ``ValueError``. DEV-1703 Phase 1 resolves it instead — the - column materialises as a hidden ``customer_id:max`` aggregate and the - ORDER BY names that alias. The invariant this test has always really + column materialises as a hidden aggregate wrap and the ORDER BY names + that alias — ``customer_id:min`` here, since DEV-1747 D10 made the wrap + direction-aware and this order is ASC. The invariant this test has always + really been about is preserved and pinned explicitly below: the sort key must NEVER reach GROUP BY, because widening the grain would change both the row count and every other measure's value. @@ -7541,7 +7543,7 @@ async def test_hidden_row_order_target_max_wraps_without_widening_grain( ) resp = await engine.execute(query, dry_run=True) sql = resp.sql - assert _re.search(r"MAX\(\s*orders\.customer_id\s*\)", sql), sql + assert _re.search(r"MIN\(\s*orders\.customer_id\s*\)", sql), sql # The sort key must not widen the grain: GROUP BY stays on status. inner = sqlglot.parse_one(sql, dialect="postgres").find(sqlglot.exp.Group) assert inner is not None, sql From c3822dd203f784ec7a518c93d24726a6e5c0c970 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Thu, 6 Aug 2026 18:00:01 +0200 Subject: [PATCH 57/98] =?UTF-8?q?DEV-1747=20=C2=A75.10:=20one=20ORDER=20BY?= =?UTF-8?q?=20resolver,=20and=20D4=20stops=20the=20silent=20drops?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four render sites turned an OrderEntry into a sort term, each its own way. They disagreed on three things, and each disagreement was a bug: * An unresolvable slot. Only the transform chain raised; the host base, the combined SELECT and the outer trim wrap `continue`d or returned None, which drops the term and returns UNSORTED ROWS with no error anywhere. That is D4, and it is now reproduced per render path. * Null ordering. Only the base path went through the dialect's `build_ordered`, so the same query sorted NULLs differently depending on whether it carried a transform — SQLite's native NULLs-first on the chain, SLayer's NULLs-last everywhere else. Now every path builds its term through the strategy, so the policy is one policy (T-SQL still pins native: its NULLS emulation puts a bracketed alias inside a CASE that re-resolves against the FROM and fails). * How the reference is qualified. A five-way precedence chain over four alias maps named a PROJECTED cross-model aggregate by its CTE column while the SELECT projected it under the user's alias, so the term resolved only by falling through to an input column of the FROM. Postgres allows that; other engines do not, and it picks the wrong column as soon as two scopes project the same name. The producing scope is not something a renderer should re-derive — the planner already knows it and names it on the entry. So `resolve_order_term(entry, env)` is a dict lookup keyed by `OrderScope`, no precedence, no fallback: a render site declares the scopes it produces, and a slot missing from its own scope is an error. P-J: `_resolve_combined_order_term`, `_apply_order_limit_from_planned` and `_planned_order_by_sql` keep their tests and lose every production caller. Pinned with raising sentinels over all five render shapes rather than by grepping, since a source scan cannot tell a live call from a docstring. Three existing tests asserted a superseded shape and were rewritten to their own stated subject: * dev1645 combined-CTE ORDER BY pinned the bare alias; its docstring says it guards the whole-quoted composite form, which the CTE-qualified reference also has. * the MySQL backtick test pinned text immediately after `ORDER BY`; MySQL has no NULLS syntax so the emulation now precedes the term. Asserted over the clause instead. * dev1733's change() ordering leaned on SQLite's native NULLs-first to tell the two directions apart. Under one policy the NULL bucket is last either way, so the discrimination comes from a control query instead. Co-Authored-By: Claude Opus 5 (1M context) --- slayer/engine/planned.py | 9 +- slayer/sql/dialects/base.py | 21 +- slayer/sql/dialects/tsql.py | 9 +- slayer/sql/generator.py | 351 +++++++++++++++--- slayer/sql/render/order_terms.py | 134 +++++++ tests/dialects/test_mysql.py | 11 +- tests/test_dev1645_invalid_postgres_sql.py | 19 +- ..._dev1733_order_only_transform_composite.py | 48 ++- 8 files changed, 517 insertions(+), 85 deletions(-) create mode 100644 slayer/sql/render/order_terms.py diff --git a/slayer/engine/planned.py b/slayer/engine/planned.py index f954e8d3..96158be2 100644 --- a/slayer/engine/planned.py +++ b/slayer/engine/planned.py @@ -417,9 +417,12 @@ class OrderEntry(BaseModel): direction: str # "asc" or "desc" scope: OrderScope phase: Phase - #: Null-ordering policy. ``"default"`` defers to the dialect's native - #: ordering for the direction; the dialect strategy owns the spelling - #: (P-H), so no render site emits a NULLS clause of its own. + #: Null-ordering policy. ``"default"`` is NULLs last, which is what SLayer + #: means by unstated on every dialect — a semantic layer whose NULLs sort + #: first on SQLite and last on Postgres answers the same question two ways. + #: The dialect strategy owns the SPELLING (P-H) — an explicit clause, an + #: emulation, or T-SQL's native pin where the emulation does not run — so + #: no render site emits a NULLS clause of its own. nulls: Literal["default", "first", "last"] = "default" @field_validator("direction") diff --git a/slayer/sql/dialects/base.py b/slayer/sql/dialects/base.py index 20ed4ed7..7a13b229 100644 --- a/slayer/sql/dialects/base.py +++ b/slayer/sql/dialects/base.py @@ -235,11 +235,22 @@ def build_ordered( The single place any render site turns a resolved column plus a direction into an ``exp.Ordered`` (P-H). It previously lived on the generator as ``_ordered``, which meant the combined and transform-chain - paths — which built their own ``exp.Ordered`` — silently skipped the - T-SQL pin below. - - ``nulls="default"`` defers to the dialect's own ordering and emits no - NULLS clause; ``"first"`` / ``"last"`` are explicit. + paths — which built their own ``exp.Ordered`` — silently skipped it. + + ``nulls="default"`` leaves ``nulls_first`` unset, which sqlglot renders + as **nulls last on every dialect** — an explicit ``NULLS LAST`` where + the native default differs and the syntax exists, a ``CASE WHEN + IS NULL …`` emulation where it does not (MySQL / SQLite). That + uniformity is the point: a semantic layer whose NULLs sort first on + SQLite and last on Postgres answers the same question two ways. + + T-SQL is the one exception and overrides this, because its emulation + does not merely look different — the bracketed alias inside the CASE + re-resolves against the FROM scope and the statement fails. + + ``"first"`` / ``"last"`` are an explicit intent and are honoured as + asked, emulation included — that is the only way to express them on a + dialect with no NULLS syntax. """ kwargs: dict = {"this": order_col, "desc": descending} if nulls == "first": diff --git a/slayer/sql/dialects/tsql.py b/slayer/sql/dialects/tsql.py index d998bbed..b3efae95 100644 --- a/slayer/sql/dialects/tsql.py +++ b/slayer/sql/dialects/tsql.py @@ -89,10 +89,11 @@ def build_ordered( default for the direction (FIRST on ASC, LAST on DESC). Left unset, sqlglot emits ``CASE WHEN IS NULL THEN 1 ELSE 0 - END, `` to emulate NULLS ordering; the bracketed alias INSIDE - the CASE WHEN mis-resolves against the FROM scope (``Invalid column - name``). Pinning the native default suppresses the wrapper without - changing the ordering. + END, `` to emulate the nulls-last ordering every other dialect + gets; the bracketed alias INSIDE the CASE WHEN mis-resolves against the + FROM scope (``Invalid column name``). So T-SQL trades null-ordering + parity for a statement that runs — the one place SLayer's null ordering + is dialect-specific, and only because the portable form is unavailable. An EXPLICIT ``first`` / ``last`` policy is honoured as asked — the pin exists to avoid the emulation, not to override a stated intent. diff --git a/slayer/sql/generator.py b/slayer/sql/generator.py index df24da0d..9f6cc2f6 100644 --- a/slayer/sql/generator.py +++ b/slayer/sql/generator.py @@ -61,6 +61,11 @@ build_grain_joinback_condition, grain_alias_column, ) +from slayer.sql.render.order_terms import ( + HOST_BASE_SCOPES, + OrderEnv, + resolve_order_term, +) from slayer.sql.render.value_expr import ( render_arithmetic, render_scalar_call, @@ -1863,7 +1868,7 @@ def _generate_from_planned_impl( # NOSONAR(S3776) — top-level dispatch over c slots_by_id=slots_by_id, bundle=bundle, ) - base_select = self._apply_order_limit_from_planned( + base_select = self._apply_planned_order_limit( select=base_select, planned_query=planned_query, source_relation=source_relation, @@ -5279,48 +5284,60 @@ def _render_outer_composite(cslot) -> exp.Expression: entries=cte_entries, final=combined_select, ) - # ORDER BY / LIMIT / OFFSET: emitted at the combined SELECT - # level. ORDER BY columns must be qualified — ``_base`` columns - # use ``_base."..."``, cross-model columns use the bare alias - # (only present on one side). - # DEV-1712 / DEV-1495 bug 2: hidden (order-only) cross-model aggregates - # are trimmed from the projection above, so their ORDER BY term must be - # CTE-qualified (``_cm_*.""``) rather than the bare - # combined-SELECT alias. - hidden_cte_order_refs: Dict[str, exp.Expression] = {} + # ORDER BY / LIMIT / OFFSET: emitted at the combined SELECT level, + # through the one resolver (§5.10). Each scope names its own value and + # nothing else does — the superseded chain ran a five-way precedence + # over four alias maps and put a projected cross-model aggregate under + # its CTE COLUMN name while the SELECT projected it under the user's + # alias, which resolves only by falling through to an input column of + # the FROM (Postgres permits that; other engines do not, and it picks + # the wrong column the moment two scopes project the same name). + order_env = OrderEnv(dialect=self._dialect) + # An isolated aggregate is CTE-qualified whether or not it is ALSO + # projected: hidden it has no combined-SELECT alias to name, projected + # its alias is the user's, not the CTE column's. One form, both cases. for plan in planned_query.cross_model_aggregate_plans: - # Only CMAs actually trimmed from the projection (hidden + no - # transform chain) need the CTE-qualified ORDER BY reference. - if not (plan.hidden and not planned_query.transform_layers): - continue - _agg_col = agg_col_alias_for_plan[plan.aggregate_slot_id] - _cte = cm_cte_name_for_plan[plan.aggregate_slot_id] - hidden_cte_order_refs[plan.aggregate_slot_id] = ( - grain_alias_column(alias=_agg_col, table=_cte) - ) - # DEV-1733: same treatment for a hidden (order-only) WINDOWED aggregate - # trimmed from the combined projection above — reference its ``_wm_`` - # CTE column rather than a bare alias the SELECT no longer emits. + order_env.cross_model_cte[plan.aggregate_slot_id] = grain_alias_column( + alias=agg_col_alias_for_plan[plan.aggregate_slot_id], + table=cm_cte_name_for_plan[plan.aggregate_slot_id], + ) for plan in planned_query.windowed_aggregate_plans: - if not (plan.hidden and not planned_query.transform_layers): - continue - hidden_cte_order_refs[plan.aggregate_slot_id] = grain_alias_column( + order_env.windowed_cte[plan.aggregate_slot_id] = grain_alias_column( alias=wm_agg_col_for_plan[plan.aggregate_slot_id], table=wm_cte_name_for_plan[plan.aggregate_slot_id], ) - order_terms = self._build_combined_order_by_sql( - planned_query=planned_query, - slots_by_id=slots_by_id, - cma_slot_ids=cma_slot_ids, - cm_alias_for_plan=canonical_alias_for_plan, - # DEV-1714: windowed slots are referenced bare in the combined ORDER - # BY — they surface as a projected combined-SELECT column (from their - # ``_wm_`` CTE), so a ``_base.`` qualifier would dangle. - bare_order_slot_ids=set(order_only_local_ids) | windowed_slot_ids, - outer_composite_aliases=outer_composite_order_alias_by_sid, - outer_composite_expressions=outer_composite_order_expressions, - hidden_cte_order_refs=hidden_cte_order_refs, - ) + # A PROJECTED outer composite orders on its combined-SELECT alias; an + # order-only one has no alias and renders INLINE, so no synthetic + # column leaks into the public projection. + for _sid, _alias in outer_composite_order_alias_by_sid.items(): + order_env.outer_composite[_sid] = exp.column(_alias, quoted=True) + for _sid, _expr in outer_composite_order_expressions.items(): + order_env.outer_composite.setdefault(_sid, _expr) + # Local slots live in ``_base``. One trimmed from the combined + # projection (order-only) is named BARE — a ``_base.`` qualifier would + # dangle under an outer projection-trim wrapper, which exposes only the + # public aliases — and the bare name still resolves unambiguously + # against ``_base`` in the combined FROM. A projected one keeps the + # qualifier. + _local_bare_ids = set(order_only_local_ids) + for entry in planned_query.order: + if entry.scope not in HOST_BASE_SCOPES: + continue + slot = slots_by_id.get(entry.slot_id) + if slot is None: + continue + _full_alias = self._full_alias_for_slot( + slot=slot, source_relation=source_relation, alias_index={}, + ) + getattr(order_env, entry.scope.value)[entry.slot_id] = ( + exp.column(_full_alias, quoted=True) + if entry.slot_id in _local_bare_ids + else grain_alias_column(alias=_full_alias, table="_base") + ) + order_terms = [ + resolve_order_term(entry=entry, env=order_env) + for entry in planned_query.order + ] if order_terms: combined_statement.set("order", exp.Order(expressions=order_terms)) @@ -7468,16 +7485,12 @@ def _emit_planned_outer_wrap( T-SQL override also transposes pagination to ``TOP`` / ``FETCH NEXT n ROWS ONLY``. """ - order_sql = self._planned_order_by_sql( + order_terms = self._planned_order_terms( planned_query=planned_query, slots_by_id=slots_by_id, available_alias_by_slot_id=available_alias_by_slot_id, ) - order_expr = ( - self._parse(f"SELECT 1 ORDER BY {order_sql}").args.get("order") - if order_sql - else None - ) + order_expr = exp.Order(expressions=order_terms) if order_terms else None limit_expr = ( exp.Limit(expression=exp.Literal.number(planned_query.limit)) if planned_query.limit is not None @@ -7497,6 +7510,37 @@ def _emit_planned_outer_wrap( parse=self._parse, ) + def _planned_order_terms( + self, + *, + planned_query, + slots_by_id: Dict[str, Any], + available_alias_by_slot_id: Dict[str, str], + ) -> List[exp.Ordered]: + """ORDER BY terms for a plan whose sort keys resolve to CTE-chain + aliases (§5.10). + + Every value the chain materialised is one column of the wrapped + subquery by the time the outer wrap is emitted, so the producing scope + no longer distinguishes anything here — hence + :meth:`OrderEnv.uniform`. Built as AST rather than rendered to text and + re-parsed: SLayer's aliases are dotted, and a re-parse re-reads + ``"orders.cs"`` as a multi-part reference on a dialect that mangles + dots at emission. + """ + env = OrderEnv.uniform( + { + sid: exp.column(alias, quoted=True) + for sid, alias in available_alias_by_slot_id.items() + if sid in slots_by_id + }, + dialect=self._dialect, + ) + return [ + resolve_order_term(entry=entry, env=env) + for entry in planned_query.order + ] + def _planned_order_by_sql( self, *, @@ -10167,9 +10211,9 @@ def _build_outer_trim_wrap_sql( # Outer ORDER BY references each order entry's materialised alias # — the first alias per slot is canonical (C13-duplicate aliases # of a single slot share the same column value). Reuse - # ``_apply_order_limit_from_planned`` to apply ORDER BY / LIMIT / + # ``_apply_planned_order_limit`` to apply ORDER BY / LIMIT / # OFFSET so the dialect-aware sqlglot emission path is shared. - return self._apply_order_limit_from_planned( + return self._apply_planned_order_limit( select=outer_select, planned_query=planned_query, source_relation=source_relation, @@ -10357,6 +10401,221 @@ def _apply_order_limit_from_planned( # NOSONAR(S3776) — per-order-entry slot- select, limit=planned_query.limit, offset=planned_query.offset, ) + # ----------------------------------------------------------------- + # §5.10 — the host-base render path's ORDER BY, through the one + # resolver. Supersedes ``_apply_order_limit_from_planned`` above. + # ----------------------------------------------------------------- + + def _apply_planned_order_limit( + self, + *, + select: exp.Select, + planned_query, + source_relation: str, + slots_by_id: dict, + source_model=None, + bundle=None, + aliases_by_slot_id: Optional[Dict[str, List[str]]] = None, + ) -> exp.Select: + """ORDER BY / LIMIT / OFFSET for a base SELECT with no CTE chain. + + The per-entry resolution is :func:`resolve_order_term`; this method + only builds the environment it reads. An entry whose slot the base + never materialised raises there rather than being skipped — the + superseded method ``continue``d past it and returned unsorted rows. + """ + env = self._host_base_order_env( + planned_query=planned_query, + source_relation=source_relation, + slots_by_id=slots_by_id, + source_model=source_model, + bundle=bundle, + aliases_by_slot_id=aliases_by_slot_id, + ) + for order_entry in planned_query.order: + select = select.order_by( + resolve_order_term(entry=order_entry, env=env), + ) + return self._dialect.apply_pagination( + select, limit=planned_query.limit, offset=planned_query.offset, + ) + + def _host_base_order_env( + self, + *, + planned_query, + source_relation: str, + slots_by_id: dict, + source_model, + bundle, + aliases_by_slot_id: Optional[Dict[str, List[str]]], + ) -> OrderEnv: + """Name every order slot the base SELECT produces, under the scope the + PLANNER assigned it (P-D). + + Both host-base scopes reference a column of the same SELECT, so the + reference form is the same; what differs is only whether the alias + survives an outer projection trim, which is the planner's + ``HOST_BASE`` / ``HOST_BASE_HIDDEN`` distinction and not something + re-derived here. + """ + env = OrderEnv(dialect=self._dialect) + for order_entry in planned_query.order: + slot = slots_by_id.get(order_entry.slot_id) + if slot is None: + # Deliberately not an early raise: leaving the slot absent is + # what makes the resolver report it, so every path reports it + # the same way. + continue + getattr(env, order_entry.scope.value)[order_entry.slot_id] = ( + self._host_base_order_ref( + slot=slot, + source_relation=source_relation, + source_model=source_model, + bundle=bundle, + aliases_by_slot_id=aliases_by_slot_id, + ) + ) + return env + + def _host_base_order_ref( # NOSONAR(S3776) — per-key-kind resolution of ONE hidden slot to a base-SELECT reference (materialised alias vs split row emission vs local derived expansion). Each branch is a distinct contract with its own invariant; splitting them scatters the chain that makes their order meaningful. + self, + *, + slot, + source_relation: str, + source_model, + bundle, + aliases_by_slot_id: Optional[Dict[str, List[str]]], + ) -> exp.Expression: + """How one slot's value is NAMED in the base SELECT. + + A public slot is its projected alias. A hidden slot is one of three + shapes: an aggregate materialised for ordering only (its materialised + alias), a bare ROW column in an ungrouped query (split + ``.`` emission, Law 2), or a local derived column + (its expansion, provided it crosses no join — a hidden derived column + never had its join pulled into the base FROM). + """ + from slayer.core.keys import ( + AggregateKey, + ArithmeticKey, + ColumnKey, + ColumnSqlKey, + ScalarCallKey, + TimeTruncKey, + TransformKey, + ) + + # DEV-1733: the EXACT set of hidden key kinds that resolve to a + # materialised alias. Deliberately enumerated rather than "any hidden + # slot that happens to carry an alias" — a hidden ROW slot with an + # alias must still hit the split-emission / invariant branches below, + # never be ordered on as a bare column that is not in the GROUP BY. + _MATERIALISED_ORDER_KINDS = ( + AggregateKey, ArithmeticKey, ScalarCallKey, TransformKey, + ) + + if not slot.hidden: + # DEV-1713: resolve to the SAME full alias the projection emits — + # a joined ROW dimension projects under the DOTTED result key + # (``orders.customers.regions.name``), so the ORDER BY must match + # it, not the flat ``declared_name`` (``customers__regions__name``), + # which would name a column the SELECT never projects. + return exp.Column( + this=exp.to_identifier( + self._full_alias_for_slot( + slot=slot, source_relation=source_relation, + alias_index={}, + ), + quoted=True, + ), + ) + + # DEV-1501: hidden AGGREGATE slots are materialised in the base SELECT. + # Resolve to the materialised full alias — identical shape to the + # public-alias branch above; the inner subquery exposes it as a column + # the outer wrap can reference by quoted identifier. + aliases = ( + aliases_by_slot_id.get(slot.id, []) + if aliases_by_slot_id is not None + else [] + ) + if aliases and isinstance(slot.key, _MATERIALISED_ORDER_KINDS): + return exp.Column(this=exp.to_identifier(aliases[0], quoted=True)) + + # DEV-1712 (Law 2, split emission): a hidden ROW column ordered in an + # UNGROUPED query. Plan-time order validation guarantees the only + # hidden ROW slot that reaches here is a bare column in a query with no + # GROUP BY — grouped row columns are rejected or wrapped up front, and + # aggregates took the branch above. Emit a SPLIT + # ``.`` reference (mixed-case-aware) against the base + # FROM scope, identical to how the column would render if it were a + # projected dimension. + # + # DEV-1703 Phase 1: a JOINED column is emitted the same way, under its + # ``__`` path alias (``customers__regions.name``). The row IS the grain + # in an ungrouped query, so the bare reference is legal; Law 1 pulls + # the crossed join into the base FROM. + key = slot.key + row_key = key.column if isinstance(key, TimeTruncKey) else key + if source_model is not None and isinstance(row_key, ColumnKey): + return self._joined_or_local_dim_expr( + path=row_key.path, leaf=row_key.leaf, + source_model=source_model, + source_relation=source_relation, bundle=bundle, + ) + + # A LOCAL DERIVED column (``ColumnSqlKey``, path empty): resolve its + # ``Column.sql`` through a throwaway host scope. That both anchors the + # expansion AND surfaces whether the SQL crosses a join. A hidden + # order-only derived column is NOT projected, so its join was never + # pulled into the base FROM — ordering on it would reference an unbound + # table. Reject that (project it), rather than emit invalid SQL; a + # non-crossing derived column (e.g. a bare mixed-case identifier) + # orders on its expression. + if ( + source_model is not None + and bundle is not None + and isinstance(row_key, ColumnSqlKey) + and not row_key.path + ): + # Detect join crossing via a throwaway scope (register-only); the + # resolved expr is discarded — its expansion lacks the DEV-1645 + # mixed-case quoting the planned-dim helper applies. + allocator = self._new_allocator() + scope = ScopeFrame( + scope_id=allocator.next_scope_id(source_relation), + root_model=source_model, + root_relation=source_relation, + bundle=bundle, + dialect=self._dialect, + allocator=allocator, + ) + scope.resolve(row_key) + if scope.join_paths: + # The derived column IS local (``orders.cust_region``); it + # merely depends on an unpulled join. Report its own qualified + # name, not a fabricated ``customers.cust_region``. + raise UnresolvableOrderColumnError( + column=row_key.column_name, qualifier=source_relation, + ) + # Non-crossing local derived column — emit through the planned-dim + # helper so the expansion is quoted identically to a projected + # dimension (mixed-case-safe). + return self._joined_or_local_dim_expr( + path=(), leaf=row_key.column_name, + source_model=source_model, + source_relation=source_relation, bundle=bundle, + ) + + # Defensive: any other hidden shape should have been rejected at plan + # time (transform / composite / joined / grouped-row). + raise NotImplementedError( + f"ORDER BY references a hidden slot (id={slot.id!r}, key=" + f"{type(slot.key).__name__}) that was not resolved at plan " + f"time — this is an internal invariant violation." + ) + # =========================================================================== # DEV-1450 stage 7b.8 — module-level shim entry point. diff --git a/slayer/sql/render/order_terms.py b/slayer/sql/render/order_terms.py new file mode 100644 index 00000000..1a9579b2 --- /dev/null +++ b/slayer/sql/render/order_terms.py @@ -0,0 +1,134 @@ +"""The one ORDER BY term resolver (P-D / P-G). + +Four render sites used to turn an ``OrderEntry`` into a sort term, each with +its own idea of what a slot id resolves to. They disagreed on three things: + +* **What happens when the slot cannot be resolved.** The transform-chain path + raised; the other three returned ``None`` or ``continue``d, which drops the + sort term and returns *unsorted rows* under a wiring bug that produces no + error anywhere. +* **Null ordering.** Only the base path went through the dialect's + ``build_ordered``, so the T-SQL pin that suppresses sqlglot's mis-resolving + ``CASE WHEN … IS NULL`` emulation was missing on the combined and chain + paths. +* **How the reference is qualified.** A five-way precedence chain over four + alias maps decided between a bare alias, a ``_base.``-qualified one, a + CTE-qualified one, and an inline expression — and a projected cross-model + aggregate came out named by its CTE column while the SELECT projected it + under the user's alias, so the term resolved only by falling through to an + input column of the FROM. + +The producing scope is not something a renderer should re-derive: the planner +already knows it and names it on the entry (``OrderScope``). So the resolution +is a dict lookup keyed by that scope, with no precedence and no fallback — +each render site fills the scopes it can produce, and a slot that is missing +from its own scope's environment is an error rather than a silent drop. +""" + +from __future__ import annotations + +from typing import Dict, Optional + +from pydantic import BaseModel, ConfigDict, Field +from sqlglot import exp + +from slayer.engine.planned import OrderEntry, OrderScope +from slayer.sql.dialects.base import SqlDialect + +__all__ = [ + "HOST_BASE_SCOPES", + "OrderEnv", + "OrderSlotNotMaterialisedError", + "resolve_order_term", +] + +#: The scopes whose value is a column of the host ``_base`` SELECT. A render +#: site that names ``_base`` columns one way and CTE columns another asks this +#: rather than testing the two members by hand, so adding a third host-base +#: scope cannot leave one site behind. +HOST_BASE_SCOPES = frozenset( + {OrderScope.HOST_BASE, OrderScope.HOST_BASE_HIDDEN}, +) + + +class OrderSlotNotMaterialisedError(RuntimeError): + """An ORDER BY entry names a slot its producing scope never materialised. + + Always an internal wiring bug — the plan validated the order term long + before rendering — so it names the slot and what the scope did carry, + which is the difference between a five-minute fix and a silent + wrong-results report. + """ + + +class OrderEnv(BaseModel): + """Where each order slot's value can be NAMED, per producing scope. + + One mapping per :class:`OrderScope`, so a render site declares only the + scopes it actually produces and cannot accidentally satisfy a lookup from + a neighbouring scope's aliases. Values are sqlglot expressions rather than + alias strings because an order-only outer composite has no alias at all — + it renders inline. + """ + + model_config = ConfigDict(arbitrary_types_allowed=True) + + host_base: Dict[str, exp.Expression] = Field(default_factory=dict) + host_base_hidden: Dict[str, exp.Expression] = Field(default_factory=dict) + cross_model_cte: Dict[str, exp.Expression] = Field(default_factory=dict) + windowed_cte: Dict[str, exp.Expression] = Field(default_factory=dict) + transform_step: Dict[str, exp.Expression] = Field(default_factory=dict) + outer_composite: Dict[str, exp.Expression] = Field(default_factory=dict) + #: Owns the null-ordering spelling (P-H). Defaults to the portable base. + dialect: Optional[SqlDialect] = None + + @classmethod + def uniform( + cls, + refs: Dict[str, exp.Expression], + *, + dialect: Optional[SqlDialect] = None, + ) -> "OrderEnv": + """An environment where every scope names a value the same way. + + The transform chain's outer wrap is that case: whatever produced a + value, by the time the chain is wrapped it is one column of the + wrapped subquery, reachable under one alias. The producing scope is + spent, so refusing to answer for it would be a lie about the SQL. + """ + return cls(**{scope.value: dict(refs) for scope in OrderScope}, + dialect=dialect) + + +_MISSING_ARMS = sorted( + scope.value for scope in OrderScope if scope.value not in OrderEnv.model_fields +) +if _MISSING_ARMS: # pragma: no cover - import-time structural guard + raise RuntimeError( + f"OrderEnv has no environment for OrderScope {_MISSING_ARMS} — a scope " + f"added without its arm would make resolve_order_term raise on every " + f"query that uses it.", + ) + +_DEFAULT_DIALECT = SqlDialect() + + +def resolve_order_term(*, entry: OrderEntry, env: OrderEnv) -> exp.Ordered: + """One ``OrderEntry`` → one ``ORDER BY`` term. + + Raises :class:`OrderSlotNotMaterialisedError` when the entry's scope did + not materialise the slot. Returning an unsorted result instead is the one + outcome a caller can neither detect nor recover from. + """ + refs: Dict[str, exp.Expression] = getattr(env, entry.scope.value) + ref = refs.get(entry.slot_id) + if ref is None: + raise OrderSlotNotMaterialisedError( + f"ORDER BY references slot id={entry.slot_id!r}, which the " + f"{entry.scope.value} scope did not materialise " + f"(it carries {sorted(refs)}).", + ) + dialect = env.dialect or _DEFAULT_DIALECT + return dialect.build_ordered( + ref.copy(), descending=entry.direction == "desc", nulls=entry.nulls, + ) diff --git a/tests/dialects/test_mysql.py b/tests/dialects/test_mysql.py index 6e5b8dff..25a8ec49 100644 --- a/tests/dialects/test_mysql.py +++ b/tests/dialects/test_mysql.py @@ -318,9 +318,16 @@ async def test_mysql_time_shift_inner_cte_uses_backticks_not_ansi_quotes() -> No # The outer ORDER BY must reference a backticked identifier, not a # single-quoted string literal (which is what sqlglot emits when it # re-parses an ANSI-quoted alias under MySQL dialect). - assert "ORDER BY\n `orders.created_at`" in sql or "ORDER BY `orders.created_at`" in sql, ( + # + # Asserted over the ORDER BY *clause* rather than the text immediately + # after the keyword: MySQL has no NULLS syntax, so the term is preceded by + # sqlglot's ``CASE WHEN IS NULL …`` emulation of the nulls-last + # ordering every dialect gets. Which term comes first is not this test's + # subject — how the alias is quoted is. + order_clause = sql[sql.rindex("ORDER BY"):] + assert "`orders.created_at`" in order_clause, ( f"ORDER BY must reference a backticked alias, not a string literal:\n{sql}" ) - assert "ORDER BY\n 'orders.created_at'" not in sql, ( + assert "'orders.created_at'" not in order_clause, ( f"sqlglot re-parsed an ANSI-quoted identifier as a string literal:\n{sql}" ) diff --git a/tests/test_dev1645_invalid_postgres_sql.py b/tests/test_dev1645_invalid_postgres_sql.py index 50ad7bf8..87babe4b 100644 --- a/tests/test_dev1645_invalid_postgres_sql.py +++ b/tests/test_dev1645_invalid_postgres_sql.py @@ -218,15 +218,18 @@ async def test_orderby_projected_alias_combined_cte_path_unchanged(self) -> None query=query, model=accts, extra_models=[clusters], )) # The typed pipeline sorts on the cross-model CTE's own canonical - # output column rather than the user's rename. Both name the same - # value (the outer SELECT projects that column AS "accts.sc"), and the - # CTE is CROSS JOINed into the combined SELECT, so Postgres resolves - # the reference against the FROM inputs. What this test guards is the - # WHOLE-QUOTED composite form at the combined-CTE ORDER BY site — a - # split ``_cm_x.accts.clusters.score_sum`` would be a nonexistent - # column — not which of the two equivalent names is chosen. - assert 'ORDER BY "accts.clusters.score_sum" DESC' in sql, sql + # output column, QUALIFIED by the CTE that emits it. It used to be the + # bare name, which resolved only by falling through to an input column + # of the FROM — legal on Postgres, not everywhere, and ambiguous the + # moment two scopes project the same name. What this test guards is + # the WHOLE-QUOTED composite form at the combined-CTE ORDER BY site — + # a split ``_cm_x.accts.clusters.score_sum`` names a column that does + # not exist. + assert ( + 'ORDER BY _cm_accts__clusters__score_sum."accts.clusters.score_sum" DESC' + ) in sql, sql assert "ORDER BY accts." not in sql, sql + assert '"accts.clusters.score_sum"' in sql, sql async def test_orderby_unprojected_joined_column_resolves_host_rooted( self, diff --git a/tests/test_dev1733_order_only_transform_composite.py b/tests/test_dev1733_order_only_transform_composite.py index 894370c2..35e007b1 100644 --- a/tests/test_dev1733_order_only_transform_composite.py +++ b/tests/test_dev1733_order_only_transform_composite.py @@ -1307,33 +1307,47 @@ async def test_order_only_change_orders_by_the_delta(self, exec_engine) -> None: """``change(amount:sum)``: January has no prior bucket so its delta is NULL; February's is 25 - 50 = -25. - SQLite sorts NULL below every value, so DESC puts February (-25) first - and January (NULL) last, and ASC reverses it. Asserting BOTH directions - is what makes this non-vacuous: an absent or dropped ORDER BY returns - the same (bucket) order twice, so it cannot satisfy both. + SLayer sorts NULLs LAST on every dialect (``OrderEntry.nulls`` default; + the dialect strategy owns the spelling), so the NULL bucket is last in + BOTH directions and February leads either way. That is a real claim + rather than an accident of SQLite's native ordering, which puts NULLs + first on ASC — the transform chain used to inherit that, so the same + query sorted differently depending on whether it carried a transform. + + Non-vacuous because of the CONTROL: the same shape ordered by + ``amount:sum`` DESC leads with January. A dropped ORDER BY cannot + produce both leaders, so the sort demonstrably runs and reads the term + the query asked for. """ def _months(rows) -> list[str]: return [str(r["orders.created_at"])[:7] for r in rows] - desc = SlayerQuery( - source_model="orders", - time_dimensions=_MONTH, - measures=[ModelMeasure(formula="amount:sum")], - order=[OrderItem(column="change(amount:sum)", direction="desc")], + def _query(order: OrderItem) -> SlayerQuery: + return SlayerQuery( + source_model="orders", + time_dimensions=_MONTH, + measures=[ModelMeasure(formula="amount:sum")], + order=[order], + ) + + resp_desc = await exec_engine.execute( + _query(OrderItem(column="change(amount:sum)", direction="desc")), ) - resp_desc = await exec_engine.execute(desc) assert _months(resp_desc.data) == ["2025-02", "2025-01"], resp_desc.data - asc = SlayerQuery( - source_model="orders", - time_dimensions=_MONTH, - measures=[ModelMeasure(formula="amount:sum")], - order=[OrderItem(column="change(amount:sum)", direction="asc")], + resp_asc = await exec_engine.execute( + _query(OrderItem(column="change(amount:sum)", direction="asc")), ) - resp_asc = await exec_engine.execute(asc) - assert _months(resp_asc.data) == ["2025-01", "2025-02"], resp_asc.data + assert _months(resp_asc.data) == ["2025-02", "2025-01"], resp_asc.data assert set(resp_asc.columns) == {"orders.created_at", "orders.amount_sum"} + resp_control = await exec_engine.execute( + _query(OrderItem(column="amount_sum", direction="desc")), + ) + assert _months(resp_control.data) == ["2025-01", "2025-02"], ( + resp_control.data + ) + async def test_hidden_order_slots_stripped_from_response(self, exec_engine) -> None: query = SlayerQuery( source_model="orders", From fc29af2b8fe12faace4c457357dc386085dd4a29 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Thu, 6 Aug 2026 18:07:51 +0200 Subject: [PATCH 58/98] DEV-1747 D9: an ungrouped derived sort key crosses like any other ref MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `ORDER BY customers.regions.name` resolves in a raw-rows query — Law 1 pulls the join and the term emits the split reference. `ORDER BY cust_region`, the SAME column reached through a derived definition, raised. That is the inconsistency DEV-1735 recorded, and the cause was where the crossing got noticed: the renderer built a throwaway ScopeFrame purely to DETECT it, found one, and rejected — because the join genuinely had not been pulled, since the join-registration pass only walked slots the base projects. An ORDER-BY-only target is deliberately not in `base_render_order` (materialising it there would project it and add it to GROUP BY, changing the grain), but Law 1 does not care that ORDER BY is the only thing referencing a ref: the join has to be in the base FROM either way. So the derived-dimension scope pass now walks order targets too, exactly as `_collect_joined_paths_for_base` already does for the bare joined case, and the probe is gone rather than merely relaxed. Only ROW slots reach that pass, so a GROUPED query contributes nothing — its sort key was already rewritten to a direction-aware aggregate wrap, which is isolated into a host-rooted CTE rather than pulled into the base. The two halves of DEV-1735 now differ only in the way they must. The sort term comes out as `customers__regions.name` — byte-identical to the bare joined form — which is asserted against that form rather than a literal, in both the D9 suite and the DEV-1712 test that pinned the old rejection. Co-Authored-By: Claude Opus 5 (1M context) --- slayer/sql/generator.py | 63 +++++++++---------- tests/test_dev1712_order_only_hidden_slots.py | 49 ++++++++++----- 2 files changed, 64 insertions(+), 48 deletions(-) diff --git a/slayer/sql/generator.py b/slayer/sql/generator.py index 9f6cc2f6..da8f1e4c 100644 --- a/slayer/sql/generator.py +++ b/slayer/sql/generator.py @@ -2821,6 +2821,7 @@ def _build_base_select_for_planned( # NOSONAR(S3776) — join-path collection a base_render_order=base_render_order, slots_by_id=slots_by_id, source_relation=source_relation, source_model=source_model, bundle=bundle, scope=host_scope, + order_slot_ids=[e.slot_id for e in planned_query.order], ) # WHERE-phase filters referencing joined columns (direct, derived, or # Mode-A ``__`` paths) register their joins into the scope too (position @@ -8697,6 +8698,7 @@ def _register_fragment_kwarg_joins( def _expand_derived_row_dims( # NOSONAR(S3776) — one cohesive per-slot pass expanding derived ROW/TIME dimensions and registering the joins they cross. self, *, base_render_order, slots_by_id, source_relation: str, source_model, bundle, scope: ScopeFrame, + order_slot_ids: Optional[List[str]] = None, ) -> Dict[str, exp.Expression]: """Pre-expand derived (``ColumnSqlKey``) ROW dimensions and derived TIME dimensions for the base SELECT: inline sibling/joined derived refs @@ -8704,6 +8706,16 @@ def _expand_derived_row_dims( # NOSONAR(S3776) — one cohesive per-slot pass e ``scope.join_paths`` (Law 1 — the join-discovery side effect), and return the expanded-expr-by-slot-id map the render branch reads from. Extracted from ``_build_base_select_for_planned``. + + ``order_slot_ids`` extends the pass to ORDER-BY-only targets, which are + deliberately NOT in ``base_render_order`` — materialising one there + would project it and add it to GROUP BY, changing the grain. A hidden + derived sort key still crosses whatever its ``Column.sql`` crosses, and + Law 1 does not care that ORDER BY is the only thing referencing it: the + join has to be in the base FROM or the sort term is unbound. Only ROW + slots reach here, so a GROUPED query contributes nothing — the planner + has already rewritten its sort key to an aggregate wrap, which is + isolated rather than pulled (DEV-1735 / D9). """ from slayer.core.keys import ColumnSqlKey, Phase, TimeTruncKey @@ -8712,7 +8724,11 @@ def _add(path: Tuple[str, ...]) -> None: scope.join_paths.add(path) derived_expr_by_sid: Dict[str, exp.Expression] = {} - for sid in base_render_order: + seen_sids: Set[str] = set() + for sid in [*base_render_order, *(order_slot_ids or ())]: + if sid in seen_sids: + continue + seen_sids.add(sid) slot = slots_by_id.get(sid) if slot is None or slot.phase != Phase.ROW: continue @@ -10565,43 +10581,24 @@ def _host_base_order_ref( # NOSONAR(S3776) — per-key-kind resolution of ONE h source_relation=source_relation, bundle=bundle, ) - # A LOCAL DERIVED column (``ColumnSqlKey``, path empty): resolve its - # ``Column.sql`` through a throwaway host scope. That both anchors the - # expansion AND surfaces whether the SQL crosses a join. A hidden - # order-only derived column is NOT projected, so its join was never - # pulled into the base FROM — ordering on it would reference an unbound - # table. Reject that (project it), rather than emit invalid SQL; a - # non-crossing derived column (e.g. a bare mixed-case identifier) - # orders on its expression. + # A LOCAL DERIVED column (``ColumnSqlKey``, path empty). Emitted + # through the planned-dim helper, so its expansion is quoted + # identically to a projected dimension (DEV-1645 mixed-case-safe) — + # and, when the ``Column.sql`` reaches through a join, comes out as the + # same ``customers__regions.name`` reference the bare joined sort key + # emits. That equality is the point of D9: the two spellings name one + # column and must sort the same way. + # + # This used to build a throwaway ``ScopeFrame`` here purely to DETECT + # the crossing at render time, and raised when it found one, because + # the join had not been pulled into the base FROM. It is pulled now — + # ``_expand_derived_row_dims`` walks ORDER BY targets, so Law 1 applies + # to a sort key exactly as it does to a projected dimension. if ( source_model is not None - and bundle is not None and isinstance(row_key, ColumnSqlKey) and not row_key.path ): - # Detect join crossing via a throwaway scope (register-only); the - # resolved expr is discarded — its expansion lacks the DEV-1645 - # mixed-case quoting the planned-dim helper applies. - allocator = self._new_allocator() - scope = ScopeFrame( - scope_id=allocator.next_scope_id(source_relation), - root_model=source_model, - root_relation=source_relation, - bundle=bundle, - dialect=self._dialect, - allocator=allocator, - ) - scope.resolve(row_key) - if scope.join_paths: - # The derived column IS local (``orders.cust_region``); it - # merely depends on an unpulled join. Report its own qualified - # name, not a fabricated ``customers.cust_region``. - raise UnresolvableOrderColumnError( - column=row_key.column_name, qualifier=source_relation, - ) - # Non-crossing local derived column — emit through the planned-dim - # helper so the expansion is quoted identically to a projected - # dimension (mixed-case-safe). return self._joined_or_local_dim_expr( path=(), leaf=row_key.column_name, source_model=source_model, diff --git a/tests/test_dev1712_order_only_hidden_slots.py b/tests/test_dev1712_order_only_hidden_slots.py index f499d947..530e8b6f 100644 --- a/tests/test_dev1712_order_only_hidden_slots.py +++ b/tests/test_dev1712_order_only_hidden_slots.py @@ -47,10 +47,7 @@ from sqlglot import exp from slayer.core.enums import DataType -from slayer.core.errors import ( - DistinctDimensionValuesError, - UnresolvableOrderColumnError, -) +from slayer.core.errors import DistinctDimensionValuesError from slayer.core.models import Column, DatasourceConfig, ModelJoin, ModelMeasure, SlayerModel from slayer.core.query import ColumnRef, OrderItem, SlayerQuery, TimeDimension from slayer.engine.query_engine import SlayerQueryEngine @@ -436,19 +433,41 @@ async def test_joined_row_column_grouped_resolves_host_rooted( # DESC takes each group's MAXIMUM (D10). assert re.search(r"(?i)\bMAX\s*\(", sql), sql - async def test_ungrouped_order_by_derived_crossing_column_rejected(self, engine) -> None: + async def test_ungrouped_order_by_derived_crossing_column_resolves( + self, engine, + ) -> None: """A hidden order-only LOCAL DERIVED column whose ``Column.sql`` crosses - a join (``cust_region`` = ``customers.region``) is not projected, so its - join is never pulled into the base FROM. Ordering on it must be rejected - rather than emit an unbound ``ORDER BY`` (CodeRabbit / T3).""" - query = SlayerQuery( - source_model="orders", - dimensions=[ColumnRef(name="status")], - distinct_dimension_values=False, - order=[OrderItem(column=ColumnRef(name="cust_region"), direction="desc")], + a join (``cust_region`` = ``customers.region``) used to be rejected: it + is not projected, so its join was never pulled into the base FROM and + the sort term would have been unbound. + + DEV-1747 D9 pulls it. Law 1 applies to a sort key exactly as it does to + a filter ref, so the join is bound and the term emits the same split + reference the BARE joined sort key emits — the two spellings name one + column, which is the consistency DEV-1735 asked for. Pinned positively + against the bare form rather than against a literal, so the two cannot + drift apart again.""" + def _q(column: ColumnRef) -> SlayerQuery: + return SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + distinct_dimension_values=False, + order=[OrderItem(column=column, direction="desc")], + ) + + derived_sql = await _sql(engine, _q(ColumnRef(name="cust_region"))) + bare_sql = await _sql(engine, _q(ColumnRef(name="customers.region"))) + + assert ("customers", "region") in _outer_order_by_columns(derived_sql), ( + derived_sql ) - with pytest.raises(UnresolvableOrderColumnError): - await _sql(engine, query) + assert ( + _outer_order_by_columns(derived_sql) + == _outer_order_by_columns(bare_sql) + ), f"derived:\n{derived_sql}\nbare:\n{bare_sql}" + # Law 1 bound the join, and the sort key is still not projected. + assert re.search(r"(?i)\bJOIN\b", derived_sql), derived_sql + assert _outer_select_columns(derived_sql) == ["orders.status"], derived_sql async def test_joined_order_ref_colliding_local_leaf_stays_joined( self, tmp_path, From 4cf22c1cc6392045e62f2dd8af60d23732b09aed Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Thu, 6 Aug 2026 18:29:42 +0200 Subject: [PATCH 59/98] DEV-1747 D8: the transform chains hand the assembler AST MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PR 3 built one WITH assembler and adopted it on the cross-model sites, leaving the two transform chains splicing their WITH clause out of f-strings: cte_clause = "WITH " + ",\n".join(f"{name} AS (\n{sql}\n)" for ...) so the emitted order was whatever order the python list happened to be built in, and each step read its predecessor POSITIONALLY (`prev_cte = ctes[-1][0]`). That is correct exactly as long as nothing ever inserts a step, and it carries no statement of what actually depends on what. Both chains now declare their dependencies and go through `assemble_with_chain`. The substantive part is that the bodies stay `exp.Select` the whole way, per PR 3's first hard-won lesson: this path carries dotted `.` names throughout, and a dotted alias round-trips through text as a MULTI-PART reference on a dialect that mangles dots at emission. So: * `_render_window_transform_sql` returns AST — real `exp.Window` nodes with `Sum` / `Lag` / `Rank` / `Ntile` / `FirstValue` and a built `WindowSpec` — rather than a formatted string the caller re-parsed to apply a CAST. * the shifted / sjoin pair and the consecutive-periods pair build their SELECTs, joins, GROUP BYs and CASE predicates as AST. Every part was already an expression; they were rendered to text only at the last step. * the cross-model chain takes its prelude as AST rather than pre-rendered strings, so `_base` and the `_cm_` CTEs are no longer serialised and spliced. A window's ORDER BY takes the emitter's NATIVE null ordering, not SLayer's nulls-last policy: an emulation term inside a frame changes which rows the frame covers. `SqlDialect.native_nulls_first` is that fact, read from the same sqlglot dialect class that generates the clause so it cannot drift. Two tests pinned emission shapes that only ever held because the old code hand-formatted them: the rank window is now compared as a re-emitted Window node, since sqlglot's pretty printer line-breaks an OVER clause that long. Co-Authored-By: Claude Opus 5 (1M context) --- slayer/sql/dialects/base.py | 30 ++ slayer/sql/generator.py | 531 ++++++++++++++++++++---------------- tests/test_sql_generator.py | 16 +- 3 files changed, 347 insertions(+), 230 deletions(-) diff --git a/slayer/sql/dialects/base.py b/slayer/sql/dialects/base.py index 7a13b229..013feb14 100644 --- a/slayer/sql/dialects/base.py +++ b/slayer/sql/dialects/base.py @@ -259,6 +259,36 @@ def build_ordered( kwargs["nulls_first"] = False return exp.Ordered(**kwargs) + def native_nulls_first(self, *, descending: bool) -> bool: + """Where NULLs sort in this dialect's OWN ordering for ``descending``. + + Setting ``nulls_first`` to this value is what makes sqlglot emit a bare + ``ORDER BY``: no NULLS clause, no ``CASE WHEN … IS NULL`` emulation. + That is wanted for orderings that are internal machinery rather than a + user-visible sort — a window frame's ``OVER (ORDER BY …)``, where an + emulation term would change which rows the frame covers. + + Read from the same dialect class that GENERATES the clause, for the + same reason :func:`_sqlglot_backslash_escapes` reads the tokenizer: a + hand-kept table would silently disagree with the emitter, and the + symptom is a wrong sort rather than an error. + """ + ordering = getattr( + _SqlglotDialect.get_or_raise(self.sqlglot_name), + "NULL_ORDERING", None, + ) + if ordering == "nulls_are_last": + return False + if ordering == "nulls_are_small": + return not descending + if ordering == "nulls_are_large": + return descending + raise RuntimeError( + f"Cannot derive the native null ordering for sqlglot dialect " + f"{self.sqlglot_name!r}: NULL_ORDERING is {ordering!r}. A sqlglot " + f"upgrade may have changed this API.", + ) + @staticmethod def _expanded_null_safe_eq( left: exp.Expression, right: exp.Expression, diff --git a/slayer/sql/generator.py b/slayer/sql/generator.py index da8f1e4c..323f389d 100644 --- a/slayer/sql/generator.py +++ b/slayer/sql/generator.py @@ -1879,9 +1879,12 @@ def _generate_from_planned_impl( # NOSONAR(S3776) — top-level dispatch over c ) return base_select.sql(dialect=self.dialect, pretty=True) - # 7b.10 — transform layers present. Build the CTE chain. - base_cte_sql = base_select.sql(dialect=self.dialect, pretty=True) - ctes: list[tuple[str, str]] = [("base", base_cte_sql)] + # 7b.10 — transform layers present. Build the CTE chain. Bodies stay + # ``exp.Select`` from renderer to assembler (D8): this chain carries + # dotted ``.`` names throughout, and a render-to-text- + # and-re-parse seam re-reads one as a multi-part reference on any + # dialect that mangles dots at emission. + ctes: List[CteEntry] = [CteEntry(name="base", query=base_select)] # DEV-1692: collision-safe CTE-name allocator for the whole transform # chain. The hoisted time_shift slot alias (``_time_shift_inner``) # repeats across arithmetic-wrapped shifts, so two ``shifted_`` / @@ -1889,7 +1892,7 @@ def _generate_from_planned_impl( # NOSONAR(S3776) — top-level dispatch over c # CTE name is reserved/allocated through this one allocator so the # ``step`` / ``shifted_`` / ``sjoin_`` / ``cp_`` families never collide. cte_allocator = self._new_allocator() - cte_allocator.reserve(*(name for name, _ in ctes)) + cte_allocator.reserve(*(entry.name for entry in ctes)) # Codex (PR #269): also reserve every already-projected column alias's # BARE form so a hidden transform alias minted below # (``_time_shift_inner`` / ``_consecutive_periods_inner``) can never @@ -1954,11 +1957,13 @@ def _generate_from_planned_impl( # NOSONAR(S3776) — top-level dispatch over c if ready_window: step_num += 1 step_name = cte_allocator.allocate_cte(f"step{step_num}") - prev_cte = ctes[-1][0] + prev_cte = ctes[-1].name carry_aliases = self._carry_aliases_in_plan_order( aliases_by_slot_id, ) - step_parts = [self._quote_ident(a) for a in carry_aliases] + step_parts = [ + exp.column(a, quoted=True) for a in carry_aliases + ] for layer in ready_window: for slot_id in layer.slot_ids: slot = slots_by_id[slot_id] @@ -1968,7 +1973,7 @@ def _generate_from_planned_impl( # NOSONAR(S3776) — top-level dispatch over c else slot.declared_name ) full_alias = f"{source_relation}.{alias}" - window_sql = self._render_window_transform_sql( + window_expr = self._render_window_transform_sql( slot=slot, slots_by_id=slots_by_id, slot_id_by_key=slot_id_by_key, @@ -1976,23 +1981,23 @@ def _generate_from_planned_impl( # NOSONAR(S3776) — top-level dispatch over c planned_query=planned_query, ) if slot.type is not None: - wrapped = _wrap_cast_for_type( - self._parse(window_sql), slot.type, + window_expr = _wrap_cast_for_type( + window_expr, slot.type, ) - window_sql = wrapped.sql(dialect=self.dialect) - step_parts.append(f'{window_sql} AS {self._quote_ident(full_alias)}') + step_parts.append( + window_expr.as_(full_alias, quoted=True), + ) aliases_by_slot_id.setdefault(slot_id, []).append( full_alias, ) available_alias_by_slot_id.setdefault( slot_id, full_alias, ) - step_sql = ( - "SELECT\n " - + _SQL_COL_SEP.join(step_parts) - + f"\nFROM {prev_cte}" - ) - ctes.append((step_name, step_sql)) + ctes.append(CteEntry( + name=step_name, + query=exp.Select().select(*step_parts).from_(prev_cte), + depends_on=[prev_cte], + )) # --- time_shift layers (each gets shifted_ + sjoin_ pair) - for layer in ready_time_shift: for slot_id in layer.slot_ids: @@ -2054,11 +2059,11 @@ def _generate_from_planned_impl( # NOSONAR(S3776) — top-level dispatch over c if unmaterialised: step_num += 1 step_name = cte_allocator.allocate_cte(f"step{step_num}") - prev_cte = ctes[-1][0] + prev_cte = ctes[-1].name carry_aliases = self._carry_aliases_in_plan_order( aliases_by_slot_id, ) - step_parts = [self._quote_ident(a) for a in carry_aliases] + step_parts = [exp.column(a, quoted=True) for a in carry_aliases] for cslot in unmaterialised: alias = ( cslot.public_aliases[0] @@ -2071,42 +2076,33 @@ def _generate_from_planned_impl( # NOSONAR(S3776) — top-level dispatch over c slot_id_by_key=slot_id_by_key, available_alias_by_slot_id=available_alias_by_slot_id, ) - expr_sql = rendered.sql(dialect=self.dialect) if cslot.type is not None: - wrapped = _wrap_cast_for_type( - self._parse(expr_sql), cslot.type, - ) - expr_sql = wrapped.sql(dialect=self.dialect) - step_parts.append(f'{expr_sql} AS {self._quote_ident(full_alias)}') + rendered = _wrap_cast_for_type(rendered, cslot.type) + step_parts.append(rendered.as_(full_alias, quoted=True)) aliases_by_slot_id.setdefault(cslot.id, []).append( full_alias, ) available_alias_by_slot_id.setdefault( cslot.id, full_alias, ) - step_sql = ( - "SELECT\n " - + _SQL_COL_SEP.join(step_parts) - + f"\nFROM {prev_cte}" - ) - ctes.append((step_name, step_sql)) + ctes.append(CteEntry( + name=step_name, + query=exp.Select().select(*step_parts).from_(prev_cte), + depends_on=[prev_cte], + )) # Inner SELECT inside _outer wrap: ALL carried aliases sorted # in PLAN order (B8 — this list used to be sorted alphabetically to # match the legacy renderer byte-for-byte). - final_cte = ctes[-1][0] + final_cte = ctes[-1].name inner_aliases = self._carry_aliases_in_plan_order(aliases_by_slot_id) - inner_sql = ( - "SELECT\n " - + _SQL_COL_SEP.join(self._quote_ident(a) for a in inner_aliases) - + f"\nFROM {final_cte}" - ) + inner_select = exp.Select().select( + *(exp.column(a, quoted=True) for a in inner_aliases), + ).from_(final_cte) - cte_clause = ( - _SQL_WITH - + ",\n".join(f"{name} AS (\n{sql}\n)" for name, sql in ctes) - ) - chain_sql = f"{cte_clause}\n{inner_sql}" + chain_sql = assemble_with_chain( + entries=ctes, final=inner_select, + ).sql(dialect=self.dialect, pretty=True) # POST-phase filter wrap (filters referencing transform / arith # slots). Mirrors legacy _generate_with_computed:1627-1648 — @@ -5249,19 +5245,9 @@ def _render_outer_composite(cslot) -> exp.Expression: "G4); the cross-model transform chain does not carry `_wm_` " "CTEs.", ) - # The transform chain is still string-assembled (it adopts the - # shared assembler in PR 4, with the local chain), so render its - # prelude CTEs here rather than threading AST into it. return self._render_cross_model_transform_chain( - prelude_ctes=[ - ("_base", base_select.sql(dialect=self.dialect, pretty=True)), - ] + [ - (name, query.sql(dialect=self.dialect, pretty=True)) - for name, query in cm_ctes - ], - combined_select_sql=combined_select.sql( - dialect=self.dialect, pretty=True, - ), + prelude_ctes=[("_base", base_select), *cm_ctes], + combined_select=combined_select, planned_query=planned_query, slots_by_id=slots_by_id, combined_aliases_by_slot_id=combined_aliases_by_slot_id, @@ -5362,8 +5348,8 @@ def _render_outer_composite(cslot) -> exp.Expression: def _render_cross_model_transform_chain( # NOSONAR(S3776) — pre-existing complexity in the window-layer chain; this PR only threaded the CTE-name allocator through it, which re-attributed the function as new code. The chain is rebuilt as sqlglot AST in the scope-assembly PR, where the layering is what gets simplified. self, *, - prelude_ctes: List[Tuple[str, str]], - combined_select_sql: str, + prelude_ctes: List[Tuple[str, exp.Expression]], + combined_select: exp.Select, planned_query, slots_by_id: Dict[str, Any], combined_aliases_by_slot_id: Dict[str, List[str]], @@ -5391,9 +5377,16 @@ def _render_cross_model_transform_chain( # NOSONAR(S3776) — pre-existing comp f"into an earlier stage.", ) - ctes: List[Tuple[str, str]] = list(prelude_ctes) + [ - ("base", combined_select_sql), - ] + ctes: List[CteEntry] = [ + CteEntry(name=name, query=query) for name, query in prelude_ctes + ] + [CteEntry( + name="base", + query=combined_select, + # The combined SELECT reads ``_base`` and every ``_cm_`` CTE the + # prelude carries; declaring that is what keeps the assembler from + # emitting it before them. + depends_on=[name for name, _ in prelude_ctes], + )] # P-F: this chain previously minted ``step`` names with a # bare f-string and held no allocator at all, so nothing connected its # names to the ``_cm_*`` CTEs already in ``prelude_ctes`` or to the @@ -5401,7 +5394,7 @@ def _render_cross_model_transform_chain( # NOSONAR(S3776) — pre-existing comp # instance that minted the ``_cm_`` names, so its used-set already # covers them) and reserve the inherited literals before allocating. cte_allocator = self._gen_allocator or self._new_allocator() - cte_allocator.reserve(*(name for name, _ in ctes)) + cte_allocator.reserve(*(entry.name for entry in ctes)) aliases_by_slot_id: Dict[str, List[str]] = { sid: list(a) for sid, a in combined_aliases_by_slot_id.items() } @@ -5437,11 +5430,11 @@ def _render_cross_model_transform_chain( # NOSONAR(S3776) — pre-existing comp ) step_num += 1 step_name = cte_allocator.allocate_cte(f"step{step_num}") - prev_cte = ctes[-1][0] + prev_cte = ctes[-1].name carry_aliases = self._carry_aliases_in_plan_order( aliases_by_slot_id, ) - step_parts = [self._quote_ident(a) for a in carry_aliases] + step_parts = [exp.column(a, quoted=True) for a in carry_aliases] for layer in ready: for slot_id in layer.slot_ids: slot = slots_by_id[slot_id] @@ -5451,7 +5444,7 @@ def _render_cross_model_transform_chain( # NOSONAR(S3776) — pre-existing comp else slot.declared_name ) full_alias = f"{source_relation}.{alias}" - window_sql = self._render_window_transform_sql( + window_expr = self._render_window_transform_sql( slot=slot, slots_by_id=slots_by_id, slot_id_by_key=slot_id_by_key, @@ -5459,18 +5452,19 @@ def _render_cross_model_transform_chain( # NOSONAR(S3776) — pre-existing comp planned_query=planned_query, ) if slot.type is not None: - window_sql = _wrap_cast_for_type( - self._parse(window_sql), slot.type, - ).sql(dialect=self.dialect) - step_parts.append(f'{window_sql} AS {self._quote_ident(full_alias)}') + window_expr = _wrap_cast_for_type( + window_expr, slot.type, + ) + step_parts.append( + window_expr.as_(full_alias, quoted=True), + ) aliases_by_slot_id.setdefault(slot_id, []).append(full_alias) available_alias_by_slot_id.setdefault(slot_id, full_alias) - step_sql = ( - "SELECT\n " - + _SQL_COL_SEP.join(step_parts) - + f"\nFROM {prev_cte}" - ) - ctes.append((step_name, step_sql)) + ctes.append(CteEntry( + name=step_name, + query=exp.Select().select(*step_parts).from_(prev_cte), + depends_on=[prev_cte], + )) pending_layers = not_ready # Materialise any projected POST-phase ArithmeticKey / ScalarCallKey @@ -5491,11 +5485,11 @@ def _render_cross_model_transform_chain( # NOSONAR(S3776) — pre-existing comp if unmaterialised: step_num += 1 step_name = cte_allocator.allocate_cte(f"step{step_num}") - prev_cte = ctes[-1][0] + prev_cte = ctes[-1].name carry_aliases = self._carry_aliases_in_plan_order( aliases_by_slot_id, ) - step_parts = [self._quote_ident(a) for a in carry_aliases] + step_parts = [exp.column(a, quoted=True) for a in carry_aliases] for cslot in unmaterialised: alias = ( cslot.public_aliases[0] @@ -5508,33 +5502,25 @@ def _render_cross_model_transform_chain( # NOSONAR(S3776) — pre-existing comp slot_id_by_key=slot_id_by_key, available_alias_by_slot_id=available_alias_by_slot_id, ) - expr_sql = rendered.sql(dialect=self.dialect) if cslot.type is not None: - expr_sql = _wrap_cast_for_type( - self._parse(expr_sql), cslot.type, - ).sql(dialect=self.dialect) - step_parts.append(f'{expr_sql} AS {self._quote_ident(full_alias)}') + rendered = _wrap_cast_for_type(rendered, cslot.type) + step_parts.append(rendered.as_(full_alias, quoted=True)) aliases_by_slot_id.setdefault(cslot.id, []).append(full_alias) available_alias_by_slot_id.setdefault(cslot.id, full_alias) - step_sql = ( - "SELECT\n " - + _SQL_COL_SEP.join(step_parts) - + f"\nFROM {prev_cte}" - ) - ctes.append((step_name, step_sql)) + ctes.append(CteEntry( + name=step_name, + query=exp.Select().select(*step_parts).from_(prev_cte), + depends_on=[prev_cte], + )) - final_cte = ctes[-1][0] + final_cte = ctes[-1].name inner_aliases = self._carry_aliases_in_plan_order(aliases_by_slot_id) - inner_sql = ( - "SELECT\n " - + _SQL_COL_SEP.join(self._quote_ident(a) for a in inner_aliases) - + f"\nFROM {final_cte}" - ) - cte_clause = ( - _SQL_WITH - + ",\n".join(f"{name} AS (\n{sql}\n)" for name, sql in ctes) - ) - chain_sql = f"{cte_clause}\n{inner_sql}" + inner_select = exp.Select().select( + *(exp.column(a, quoted=True) for a in inner_aliases), + ).from_(final_cte) + chain_sql = assemble_with_chain( + entries=ctes, final=inner_select, + ).sql(dialect=self.dialect, pretty=True) post_filter_conditions = self._render_post_phase_filter_conditions( planned_query=planned_query, @@ -7115,7 +7101,28 @@ def _joined_or_local_dim_expr( table=exp.to_identifier(current_alias), ) - def _render_window_transform_sql( + def _window_ordered(self, col: exp.Expression, *, descending: bool = False) -> exp.Ordered: + """One ``ORDER BY`` term INSIDE an ``OVER (…)`` clause. + + Not :meth:`SqlDialect.build_ordered`: a window's frame ordering is + internal machinery, not a user-visible sort, so it takes the emitter's + own null ordering rather than SLayer's nulls-last policy — which on a + dialect without NULLS syntax would expand into a ``CASE WHEN … IS + NULL`` term inside the frame and change which rows the frame covers. + """ + args: Dict[str, Any] = { + "this": col, + "nulls_first": self._dialect.native_nulls_first( + descending=descending, + ), + } + if descending: + # ``desc=False`` would emit an explicit ``ASC``; leaving the key + # off emits the bare column, which is what ascending means. + args["desc"] = True + return exp.Ordered(**args) + + def _render_window_transform_sql( # NOSONAR(S3776) — one per-op dispatch over the window-transform vocabulary, sharing the resolved measure / frame / partition state every arm reads. Each arm is one line; splitting the dispatch scatters that state without simplifying it. self, *, slot, @@ -7123,14 +7130,18 @@ def _render_window_transform_sql( slot_id_by_key: Dict[Any, str], available_alias_by_slot_id: Dict[str, str], planned_query, - ) -> str: - """Render one window-transform slot as an OVER() expression. + ) -> exp.Expression: + """Render one window-transform slot as an ``OVER()`` expression. - Direct port of ``_build_transform_sql:1794`` but reads from the - typed ``TransformKey`` instead of legacy ``EnrichedTransform``. - Auto-partition matches legacy: ``partition_aliases = query - dimensions only`` (NOT time dimensions) for non-rank ops; - rank-family defaults to no PARTITION BY. + Returns AST (DEV-1747 D8). It used to return a SQL string, which the + caller then spliced into an f-string CTE body — so every dotted public + alias this path carries (``orders.rev``) made a round trip through text + before reaching the assembler, and on a dialect that mangles dots at + emission a re-parse reads such an alias as a multi-part reference. + + Auto-partition matches legacy: ``partition_aliases = query dimensions + only`` (NOT time dimensions) for non-rank ops; rank-family defaults to + no PARTITION BY. """ from slayer.core.keys import ( ColumnKey, @@ -7163,7 +7174,7 @@ def _render_window_transform_sql( key=key.input, slot_id_by_key=slot_id_by_key, available_alias_by_slot_id=available_alias_by_slot_id, - ).sql(dialect=self.dialect) + ) else: # Resolve input alias (slotted leaf). input_sid = slot_id_by_key.get(key.input) @@ -7172,11 +7183,12 @@ def _render_window_transform_sql( f"transform input not materialised: slot id={slot.id!r}, " f"op={key.op!r}, input_key={key.input!r}.", ) - input_alias = available_alias_by_slot_id[input_sid] - measure = self._quote_ident(input_alias) + measure = exp.column( + available_alias_by_slot_id[input_sid], quoted=True, + ) # Resolve time-key alias (None for rank-family without time). - time_alias: Optional[str] = None + time_col: Optional[exp.Expression] = None if key.time_key is not None: tk_sid = slot_id_by_key.get(key.time_key) if tk_sid is None or tk_sid not in available_alias_by_slot_id: @@ -7185,7 +7197,9 @@ def _render_window_transform_sql( f"slot id={slot.id!r}, op={key.op!r}, " f"time_key={key.time_key!r}.", ) - time_alias = self._quote_ident(available_alias_by_slot_id[tk_sid]) + time_col = exp.column( + available_alias_by_slot_id[tk_sid], quoted=True, + ) # Resolve partition aliases. Explicit partition_keys take # precedence; otherwise auto-partition by query dimension slots @@ -7224,17 +7238,44 @@ def _render_window_transform_sql( if alias is not None: partition_aliases.append(alias) - partition_clause = ( - _SQL_PARTITION_BY + ", ".join(self._quote_ident(a) for a in partition_aliases) - if partition_aliases - else "" + partition_by = [exp.column(a, quoted=True) for a in partition_aliases] + + def _over( + fn: exp.Expression, + *, + order: Optional[exp.Order] = None, + spec: Optional[exp.WindowSpec] = None, + ) -> exp.Window: + """``fn OVER (PARTITION BY … ORDER BY … )``. + + Built rather than formatted so the partition and order columns stay + single quoted identifiers all the way to emission — the dotted + public aliases here (``orders.rev``) are exactly the shape a text + round trip re-reads as a multi-part reference. + """ + args: Dict[str, Any] = {"this": fn} + if partition_by: + args["partition_by"] = [c.copy() for c in partition_by] + if order is not None: + args["order"] = order + if spec is not None: + args["spec"] = spec + return exp.Window(**args) + + time_order = ( + exp.Order(expressions=[self._window_ordered(time_col.copy())]) + if time_col is not None + else None ) - order_clause = ( - f"ORDER BY {time_alias}" if time_alias else "" + #: The rank family orders by the MEASURE descending, not by time. + rank_order = exp.Order( + expressions=[self._window_ordered(measure.copy(), descending=True)], + ) + unbounded_frame = exp.WindowSpec( + kind="ROWS", + start="UNBOUNDED", start_side="PRECEDING", + end="UNBOUNDED", end_side="FOLLOWING", ) - over_parts = " ".join(p for p in (partition_clause, order_clause) if p) - rank_order = f"ORDER BY {measure} DESC" - rank_over = " ".join(p for p in (partition_clause, rank_order) if p) kwarg_map = dict(key.kwargs) op = key.op @@ -7264,19 +7305,25 @@ def _normalise_periods(raw: Any, *, kw: str = "periods") -> int: ) if op == "cumsum": - return f"SUM({measure}) OVER ({over_parts})" + return _over(exp.Sum(this=measure), order=time_order) if op == "lag": n = abs(_normalise_periods(kwarg_map.get("periods", 1))) - return f"LAG({measure}, {n}) OVER ({over_parts})" + return _over( + exp.Lag(this=measure, offset=exp.Literal.number(n)), + order=time_order, + ) if op == "lead": n = abs(_normalise_periods(kwarg_map.get("periods", 1))) - return f"LEAD({measure}, {n}) OVER ({over_parts})" + return _over( + exp.Lead(this=measure, offset=exp.Literal.number(n)), + order=time_order, + ) if op == "rank": - return f"RANK() OVER ({rank_over})" + return _over(exp.Rank(), order=rank_order) if op == "percent_rank": - return f"PERCENT_RANK() OVER ({rank_over})" + return _over(exp.PercentRank(), order=rank_order) if op == "dense_rank": - return f"DENSE_RANK() OVER ({rank_over})" + return _over(exp.DenseRank(), order=rank_order) if op == "ntile": n = kwarg_map.get("n") if not isinstance(n, int): @@ -7292,22 +7339,28 @@ def _normalise_periods(raw: Any, *, kw: str = "periods") -> int: raise ValueError( f"ntile requires a positive integer n, got {n!r}", ) - return f"NTILE({n}) OVER ({rank_over})" + return _over( + exp.Ntile(this=exp.Literal.number(n)), order=rank_order, + ) if op == "first": - return ( - f"FIRST_VALUE({measure}) OVER ({over_parts} " - f"ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING)" + return _over( + exp.FirstValue(this=measure), + order=time_order, spec=unbounded_frame, ) if op == "last": - if time_alias is None: + if time_col is None: raise ValueError( f"Transform 'last' requires an unambiguous time " f"dimension (binder/planner gap; slot id={slot.id!r}).", ) - return ( - f"FIRST_VALUE({measure}) OVER " - f"({partition_clause} ORDER BY {time_alias} DESC " - f"ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING)" + # ``last`` is ``first`` over the REVERSED time axis, so it takes + # the descending order rather than ``time_order``. + return _over( + exp.FirstValue(this=measure), + order=exp.Order(expressions=[ + self._window_ordered(time_col.copy(), descending=True), + ]), + spec=unbounded_frame, ) raise NotImplementedError( f"DEV-1450 stage 7b.10: transform op {op!r} not in the " @@ -7961,22 +8014,21 @@ def _add_partition(pk_obj, *, where: str) -> None: granularity=TimeGranularity(time_key.granularity), ) - # Build the shifted CTE. - shifted_select_parts: list[str] = [] - shifted_group_by: list[str] = [] + # Build the shifted CTE — as AST, so the dotted base aliases it + # projects reach the assembler as single identifiers (D8). + shifted_select_parts: List[exp.Expression] = [] + shifted_group_by: List[exp.Expression] = [] # Projected: time-trunc shifted under the base time alias. - shifted_trunc_sql = shifted_trunc_expr.sql(dialect=self.dialect) shifted_select_parts.append( - f'{shifted_trunc_sql} AS {self._quote_ident(time_alias)}', + shifted_trunc_expr.as_(time_alias, quoted=True), ) - shifted_group_by.append(shifted_trunc_sql) + shifted_group_by.append(shifted_trunc_expr.copy()) # partition_keys: SELECT + GROUP BY under their base aliases. for _, pk_alias, pk_expr in partition_specs: - pk_sql = pk_expr.sql(dialect=self.dialect) - shifted_select_parts.append(f'{pk_sql} AS {self._quote_ident(pk_alias)}') - shifted_group_by.append(pk_sql) + shifted_select_parts.append(pk_expr.as_(pk_alias, quoted=True)) + shifted_group_by.append(pk_expr.copy()) # Aggregate: re-emit the AggregateKey using the same synth / # _build_agg dance the base CTE uses. @@ -8002,7 +8054,7 @@ def _add_partition(pk_obj, *, where: str) -> None: agg_expr, _ = self._build_agg(synth) agg_expr = _wrap_cast_for_type(agg_expr, inner_slot.type) shifted_select_parts.append( - f'{agg_expr.sql(dialect=self.dialect)} AS {self._quote_ident(input_alias)}', + agg_expr.as_(input_alias, quoted=True), ) else: # Row-level column input (not aggregated). Resolve through the scope @@ -8010,9 +8062,9 @@ def _add_partition(pk_obj, *, where: str) -> None: # correctly (Law 1), same as every other ref in this CTE. col_expr = shifted_scope.resolve(inner_key) shifted_select_parts.append( - f'{col_expr.sql(dialect=self.dialect)} AS {self._quote_ident(input_alias)}', + col_expr.as_(input_alias, quoted=True), ) - shifted_group_by.append(col_expr.sql(dialect=self.dialect)) + shifted_group_by.append(col_expr.copy()) # DEV-1711: register the join paths the shifted WHERE filters cross # (computed once by ``_build_shifted_cte_where_parts``) so a joined-column @@ -8036,24 +8088,23 @@ def _add_partition(pk_obj, *, where: str) -> None: ) shifted_joins = [] - from_parts = [f"FROM {from_clause.sql(dialect=self.dialect)}"] + shifted_select = exp.Select().select(*shifted_select_parts).from_( + from_clause, + ) for join_expr, on_expr, join_type in shifted_joins: - from_parts.append( - f"{join_type} JOIN {join_expr.sql(dialect=self.dialect)} " - f"ON {on_expr.sql(dialect=self.dialect)}" - ) - - shifted_sql_parts = [_SQL_SELECT_HEAD + ",\n ".join(shifted_select_parts)] - shifted_sql_parts.extend(from_parts) - if shifted_where_parts: - shifted_sql_parts.append( - "WHERE " + _SQL_AND_JOINER.join(shifted_where_parts), + shifted_select = shifted_select.join( + join_expr, on=on_expr, join_type=join_type, ) - if shifted_group_by: - shifted_sql_parts.append( - "GROUP BY\n " + ",\n ".join(shifted_group_by), + # ``shifted_where_parts`` is the one text input left on this path: the + # WHERE builder renders Mode-A predicates to SQL. Parsed once here + # rather than concatenated into a body string, so the surrounding CTE + # stays AST. + for _where_part in shifted_where_parts: + shifted_select = shifted_select.where( + self._parse_predicate(_where_part), ) - shifted_sql = "\n".join(shifted_sql_parts) + for _gb in shifted_group_by: + shifted_select = shifted_select.group_by(_gb) # Pick the slot's user-facing alias(es). DEV-1450 C13: two # declared measures sharing a structural key intern to ONE @@ -8075,25 +8126,29 @@ def _add_partition(pk_obj, *, where: str) -> None: shifted_cte_name = cte_allocator.allocate_cte(f"shifted_{cte_name_alias}") sjoin_cte_name = cte_allocator.allocate_cte(f"sjoin_{cte_name_alias}") - ctes.append((shifted_cte_name, shifted_sql)) + # The shifted CTE reads the SOURCE table, not the chain, so it declares + # no dependency; the assembler keeps it in declaration order. + ctes.append(CteEntry(name=shifted_cte_name, query=shifted_select)) # Build the sjoin CTE: LEFT JOIN prev_cte + shifted on time + # partition equalities. Carry every prev_cte alias forward, # then add the shifted measure under EACH of the slot's public # aliases (DEV-1450 C13). - prev_cte = ctes[-2][0] # the CTE just before the shifted CTE + prev_cte = ctes[-2].name # the CTE just before the shifted CTE carry_aliases = self._carry_aliases_in_plan_order( aliases_by_slot_id, ) - sjoin_select_parts = [ - f'{prev_cte}.{self._quote_ident(a)}' for a in carry_aliases + sjoin_select_parts: List[exp.Expression] = [ + grain_alias_column(alias=a, table=prev_cte) for a in carry_aliases ] slot_full_aliases: List[str] = [] for slot_alias in slot_aliases: full_slot_alias = f"{source_relation}.{slot_alias}" slot_full_aliases.append(full_slot_alias) sjoin_select_parts.append( - f'{shifted_cte_name}.{self._quote_ident(input_alias)} AS {self._quote_ident(full_slot_alias)}', + grain_alias_column( + alias=input_alias, table=shifted_cte_name, + ).as_(full_slot_alias, quoted=True), ) # JOIN conditions: time equality + every partition equality. The sjoin is @@ -8114,13 +8169,14 @@ def _add_partition(pk_obj, *, where: str) -> None: ], dialect=self._dialect, ) - sjoin_sql = ( - "SELECT " + ", ".join(sjoin_select_parts) - + f"\nFROM {prev_cte}" - + f"\nLEFT JOIN {shifted_cte_name}" - + "\n ON " + sjoin_on.sql(dialect=self.dialect) - ) - ctes.append((sjoin_cte_name, sjoin_sql)) + sjoin_select = exp.Select().select(*sjoin_select_parts).from_( + prev_cte, + ).join(shifted_cte_name, on=sjoin_on, join_type="LEFT") + ctes.append(CteEntry( + name=sjoin_cte_name, + query=sjoin_select, + depends_on=[prev_cte, shifted_cte_name], + )) # Record EACH alias in both the per-slot list (C13 carry-forward # in the outer SELECT) and the "pick one" map (transform input / @@ -8204,9 +8260,14 @@ def _emit_consecutive_periods_ctes_for_planned( # NOSONAR(S3776) — one cohesi f"consecutive_periods input not materialised: " f"slot id={slot.id!r}, input={inner_key!r}.", ) - input_alias = available_alias_by_slot_id[input_sid] - predicate_sql = ( - f'{self._quote_ident(input_alias)} IS NOT NULL AND {self._quote_ident(input_alias)} <> 0' + input_col = exp.column( + available_alias_by_slot_id[input_sid], quoted=True, + ) + predicate = exp.And( + this=exp.Is(this=input_col.copy(), expression=exp.Null()).not_(), + expression=exp.NEQ( + this=input_col.copy(), expression=exp.Literal.number(0), + ), ) predicate_is_boolean = False elif isinstance(inner_key, ArithmeticKey): @@ -8218,12 +8279,11 @@ def _emit_consecutive_periods_ctes_for_planned( # NOSONAR(S3776) — one cohesi f"ArithmeticKey op={inner_key.op!r}) are deferred to " f"a follow-up slice (slot id={slot.id!r}).", ) - rendered = self._render_value_key_against_aliases( + predicate = self._render_value_key_against_aliases( key=inner_key, slot_id_by_key=slot_id_by_key, available_alias_by_slot_id=available_alias_by_slot_id, ) - predicate_sql = rendered.sql(dialect=self.dialect) predicate_is_boolean = True else: raise NotImplementedError( @@ -8233,9 +8293,11 @@ def _emit_consecutive_periods_ctes_for_planned( # NOSONAR(S3776) — one cohesi # COALESCE / numeric wrap. if predicate_is_boolean: - pred_in_case = f"COALESCE({predicate_sql}, FALSE)" + pred_in_case: exp.Expression = exp.Coalesce( + this=predicate, expressions=[exp.false()], + ) else: - pred_in_case = predicate_sql + pred_in_case = predicate # Auto-partition by query dimensions (ColumnKey row-phase slots # only — NOT TimeTruncKey, matching legacy). @@ -8264,63 +8326,76 @@ def _emit_consecutive_periods_ctes_for_planned( # NOSONAR(S3776) — one cohesi cp_reset_alias = f"_cp_reset_{full_slot_alias}" # Build the reset CTE. - prev_cte = ctes[-1][0] + prev_cte = ctes[-1].name carry_aliases = self._carry_aliases_in_plan_order( aliases_by_slot_id, ) - carry_select = ",\n ".join(self._quote_ident(a) for a in carry_aliases) - partition_clause = ( - _SQL_PARTITION_BY + ", ".join(self._quote_ident(a) for a in partition_aliases) - if partition_aliases - else "" - ) - over_reset = " ".join(p for p in ( - partition_clause, - f'ORDER BY {self._quote_ident(time_alias)}', - "ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW", - ) if p) - reset_window_sql = ( - f'SUM(CASE WHEN {pred_in_case} THEN 0 ELSE 1 END) ' - f'OVER ({over_reset}) AS {self._quote_ident(cp_reset_alias)}' - ) + carry_cols = [exp.column(a, quoted=True) for a in carry_aliases] + running_frame = exp.WindowSpec( + kind="ROWS", + start="UNBOUNDED", start_side="PRECEDING", end="CURRENT ROW", + ) + + def _running_sum( + *, then: int, other: int, partitions: List[str], + ) -> exp.Window: + """``SUM(CASE WHEN THEN … ELSE … END) OVER (… ROWS BETWEEN + UNBOUNDED PRECEDING AND CURRENT ROW)`` — the shape both layers use, + differing only in the CASE arms and the partition set.""" + args: Dict[str, Any] = { + "this": exp.Sum(this=exp.Case( + ifs=[exp.If( + this=pred_in_case.copy(), + true=exp.Literal.number(then), + )], + default=exp.Literal.number(other), + )), + "order": exp.Order(expressions=[ + self._window_ordered(exp.column(time_alias, quoted=True)), + ]), + "spec": running_frame.copy(), + } + if partitions: + args["partition_by"] = [ + exp.column(a, quoted=True) for a in partitions + ] + return exp.Window(**args) + cp_reset_cte_name = cte_allocator.allocate_cte(f"cp_reset_{slot_alias}") - cp_reset_sql = ( - _SQL_SELECT_HEAD + carry_select - + ",\n " + reset_window_sql - + f"\nFROM {prev_cte}" - ) - ctes.append((cp_reset_cte_name, cp_reset_sql)) + ctes.append(CteEntry( + name=cp_reset_cte_name, + query=exp.Select().select( + *(c.copy() for c in carry_cols), + _running_sum( + then=0, other=1, partitions=partition_aliases, + ).as_(cp_reset_alias, quoted=True), + ).from_(prev_cte), + depends_on=[prev_cte], + )) # Build the value CTE — references the cp_reset CTE's added # column in PARTITION BY so each run of true predicate is - # counted within its own reset group. - value_partition_aliases = partition_aliases + [cp_reset_alias] - value_partition_clause = _SQL_PARTITION_BY + ", ".join( - self._quote_ident(a) for a in value_partition_aliases - ) - over_value = " ".join(( - value_partition_clause, - f'ORDER BY {self._quote_ident(time_alias)}', - "ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW", - )) - # Outer CASE WHEN guarantees rows where the predicate is false - # surface as 0 (legacy parity). - value_inner_window_sql = ( - f'SUM(CASE WHEN {pred_in_case} THEN 1 ELSE 0 END) ' - f'OVER ({over_value})' - ) - value_outer_case = ( - f'CASE WHEN {pred_in_case} ' - f'THEN {value_inner_window_sql} ELSE 0 END ' - f'AS {self._quote_ident(full_slot_alias)}' + # counted within its own reset group. The outer CASE WHEN + # guarantees rows where the predicate is false surface as 0. + value_outer_case = exp.Case( + ifs=[exp.If( + this=pred_in_case.copy(), + true=_running_sum( + then=1, other=0, + partitions=partition_aliases + [cp_reset_alias], + ), + )], + default=exp.Literal.number(0), ) cp_value_cte_name = cte_allocator.allocate_cte(f"cp_value_{slot_alias}") - cp_value_sql = ( - _SQL_SELECT_HEAD + carry_select - + ",\n " + value_outer_case - + f"\nFROM {cp_reset_cte_name}" - ) - ctes.append((cp_value_cte_name, cp_value_sql)) + ctes.append(CteEntry( + name=cp_value_cte_name, + query=exp.Select().select( + *(c.copy() for c in carry_cols), + value_outer_case.as_(full_slot_alias, quoted=True), + ).from_(cp_reset_cte_name), + depends_on=[cp_reset_cte_name], + )) # Record the slot's alias for downstream lookups. aliases_by_slot_id.setdefault(slot.id, []).append(full_slot_alias) diff --git a/tests/test_sql_generator.py b/tests/test_sql_generator.py index dc012a90..b2f650f0 100644 --- a/tests/test_sql_generator.py +++ b/tests/test_sql_generator.py @@ -2154,10 +2154,22 @@ async def test_rank_with_partition_by_list(self, generator: SQLGenerator, orders sql = await _generate(generator, query, orders_model) # PARTITION BY column order is semantically irrelevant; the typed # planner emits the keys in sorted order (customer_id before status). - assert ( + # + # Compared against the re-emitted Window node rather than the raw text: + # the transform chain is assembled as AST (DEV-1747 D8), so an OVER + # clause this long is line-broken by sqlglot's pretty printer, and + # collapsing whitespace still leaves the spaces it puts inside the + # parens. The claim here is the window's SHAPE, not its line breaks. + window = next( + w + for w in sqlglot.parse_one(sql, read="postgres").find_all( + sqlglot.exp.Window, + ) + if isinstance(w.this, sqlglot.exp.Rank) + ) + assert window.sql(dialect="postgres") == ( 'RANK() OVER (PARTITION BY "orders.customer_id", "orders.status" ' 'ORDER BY "orders.revenue_sum" DESC)' - in _norm(sql) ) async def test_percent_rank_default(self, generator: SQLGenerator, orders_model: SlayerModel) -> None: From 05aa0bd6f2241c882fc7a2f95b75b223a6ab8139 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Thu, 6 Aug 2026 18:40:59 +0200 Subject: [PATCH 60/98] DEV-1747 B6: the reroot keeps the filter AUDIT, not the filter ROUTING MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Caught by the integration suite, which the unit tests could not see: a re-rooted cross-model measure with a host filter returned an extra row. POL-003 (party_role_code = 'AG') -> measure NULL, row present The B6 commit stopped blanking the four routing lists after a reroot, on the grounds that reporting `where=[] having=[] applied=[] dropped=[]` for a reachable filter, a host-local one, and a genuinely unreachable one alike is indistinguishable. That is right about the AUDIT and wrong about the ROUTING, because the two lists mean different things: * `applied_filter_ids` records that SOME scope evaluates the filter. That is what makes an unreachable filter distinguishable, and it survives. * `where_filter_ids` / `having_filter_ids` say the filter MOVED to the forward CTE, so the host base must not apply it. A re-rooted plan has no forward CTE — the sub-plan replaces it and carries its own re-anchored filters — and the predicate is host-evaluable by construction, since it was bound against the host. Leaving the ids there told the host base to skip a filter nothing else applied there, so excluded rows came back with a NULL measure attached. So the reroot now clears where/having and keeps applied + the dropped warnings. The `CrossModelAggregatePlan` docstring said `applied` was "the audit union of where + having"; on the re-rooted path the two diverge on purpose, and it now says why. The DEV-1747 test that pinned `where_filter_ids or having_filter_ids` was pinning the fused meaning. It now asserts the audit AND that the sub-plan really carries the filter it claims to apply — a label on nothing would otherwise satisfy it. Co-Authored-By: Claude Opus 5 (1M context) --- slayer/engine/cross_model_planner.py | 23 +++++++++++++++------ slayer/engine/planned.py | 11 ++++++++-- tests/test_dev1747_reroot_filter_routing.py | 18 +++++++++++++++- 3 files changed, 43 insertions(+), 9 deletions(-) diff --git a/slayer/engine/cross_model_planner.py b/slayer/engine/cross_model_planner.py index cf65b816..f7435089 100644 --- a/slayer/engine/cross_model_planner.py +++ b/slayer/engine/cross_model_planner.py @@ -1605,14 +1605,25 @@ def _is_forward(path: Tuple[str, ...]) -> bool: if sub_agg_sid is None: return plan - # DEV-1747 B6/D6 — the routing is NOT cleared. It was decided once, in the - # coordinate system of the CTE that now exists, and the sub-plan applies - # exactly the filters it records as applied. Blanking it here is what made - # a reachable filter, a host-local one, and a genuinely unreachable one all - # report ``where=[] having=[] applied=[] dropped=[]`` — indistinguishable, - # and in the unreachable case a silent narrowing of the user's result. + # DEV-1747 B6/D6 — the AUDIT survives the reroot; the ROUTING does not, + # because they are different statements and the reroot changes only one. + # + # ``applied_filter_ids`` records what SOME scope evaluates, which is what + # makes a genuinely unreachable filter distinguishable from a reachable one + # instead of every case reporting ``applied=[] dropped=[]``. Those ids ride + # into ``sub_prebound`` above, so the sub-plan really does apply them. + # + # ``where_filter_ids`` / ``having_filter_ids`` say something narrower: this + # filter MOVED to the forward CTE, so the host base must not apply it. A + # re-rooted plan has no forward CTE — the sub-plan replaces it and carries + # its own filters — and the predicate is host-evaluable by construction + # (it was bound against the host). Leaving the ids there tells the host base + # to skip a filter nothing else applies at the host, so rows the user + # excluded come back with a NULL measure attached. return plan.model_copy(update={ "rerooted_plan": sub_plan, "rerooted_grain_pairs": grain_pairs, "rerooted_agg_slot_id": sub_agg_sid, + "where_filter_ids": [], + "having_filter_ids": [], }) diff --git a/slayer/engine/planned.py b/slayer/engine/planned.py index 96158be2..89f2ae27 100644 --- a/slayer/engine/planned.py +++ b/slayer/engine/planned.py @@ -185,8 +185,15 @@ class CrossModelAggregatePlan(BaseModel): table row: cross-model agg-ref on the same target). - ``target_model_filters`` — the target model's own ``SlayerModel.filters`` (always-applied WHERE). - ``applied_filter_ids`` is the audit union of where + having for - backward compatibility with the spec's external surface. + + ``applied_filter_ids`` is the AUDIT: which host filters some scope + evaluates. On the forward path that is exactly ``where ∪ having``. On a + RE-ROOTED plan the two diverge on purpose — ``rerooted_plan`` carries the + filters itself, and there is no forward CTE to route to, so where/having + are empty while the audit still records them. The distinction matters + because where/having are also an instruction to the host base to SKIP the + filter; a re-rooted CTE duplicates a host-evaluable predicate rather than + relocating it, so the host must keep applying it (DEV-1747 B6). ``hidden=True`` is used for order-only / filter-only refs whose aggregate value is materialised but not surfaced in the public diff --git a/tests/test_dev1747_reroot_filter_routing.py b/tests/test_dev1747_reroot_filter_routing.py index 25192092..b0d70870 100644 --- a/tests/test_dev1747_reroot_filter_routing.py +++ b/tests/test_dev1747_reroot_filter_routing.py @@ -105,7 +105,23 @@ def test_reachable_filter_is_routed_not_blanked(self) -> None: "the re-rooted CTE applies this filter, so the plan must SAY so — " "today the routing lists are cleared wholesale" ) - assert plan.where_filter_ids or plan.having_filter_ids + # ...and the SUB-PLAN is where it is applied. ``where_filter_ids`` is + # not an audit, it is an instruction to the host base to SKIP the + # filter because the FORWARD CTE took it over. A re-rooted plan has no + # forward CTE, and the predicate is host-evaluable by construction, so + # the host must keep applying it — otherwise rows the user excluded + # come back carrying a NULL measure. + assert not plan.where_filter_ids and not plan.having_filter_ids, ( + "a re-rooted plan told the host base to skip a filter that only " + "the CTE applies" + ) + # The audit has to be backed by something: the sub-plan must actually + # carry the filter it claims is applied, or "applied" is a label on + # nothing. + assert plan.rerooted_plan.filters_by_phase, ( + f"the audit claims {sorted(plan.applied_filter_ids)} applied, but " + f"the re-rooted sub-plan carries no filters at all" + ) def test_host_local_filter_is_neither_propagated_nor_warned(self) -> None: """``DROP_HOST_LOCAL``: the host base applies it and the join-back From 162e1761807f555c54767e60fba02b4659d323ed Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Thu, 6 Aug 2026 21:11:43 +0200 Subject: [PATCH 61/98] DEV-1747: docs, the design log, and a golden baseline for PRs 5-6 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Docs. Three user-visible behaviours changed and the table in `docs/concepts/queries.md` still described the old ones as rejections: * ordering by an undeclared row column in a GROUPED query used to be an HTTP 400 ("add it to dimensions"). It now sorts per group, and by the extreme the DIRECTION puts first — asc by each group's min, desc by its max. That distinction is worth stating outright, because the implicit wrap is not something a caller can see in their own query text. * the same for a JOINED row column, and for a derived column whose `sql` reaches through a join — previously "project it or order by something else". * NULLs sort last in both directions on every database, so the same query returns the same row order regardless of backend. SQL Server is the one exception and the docs say why rather than leaving it as a surprise. Also: an order target SLayer cannot resolve is now an error, never a silently unsorted result. Mirrored in `.claude/skills/slayer-query.md`. `DECISIONS.md` records the seven decisions worth consulting before changing this behaviour again — in particular why D2 was IMPOSSIBLE before the pre-bound seam rather than merely tidier (formula text cannot express a path-bearing source or a grain marker), and why the audit and the routing had to be separated after they were briefly fused. Golden baseline. `tests/test_dev1747_golden_sql.py` — same harness and same four-step blessing protocol as the DEV-1745 one, 90 entries over 5 dialects, matrix chosen to cover exactly what this PR rewires and what PR 5 builds on: all five ORDER BY render paths, both D10 directions, the D2 host-grain wrap, the three B6 reachability shapes, and both assembled WITH chains. Dialects include DuckDB for a third null-ordering regime and BigQuery + T-SQL because both mangle dotted aliases at emission. Its vacuity guard earned its place immediately: it caught an ordering case recording an exception instead of SQL, and a re-rooting case that never re-rooted, both of which would have been pinned as "correct" forever. Co-Authored-By: Claude Opus 5 (1M context) --- .claude/skills/slayer-query.md | 2 +- DECISIONS.md | 2 + docs/concepts/queries.md | 20 +- tests/golden/dev1747_sql_baseline.json | 92 +++++++ tests/test_dev1747_golden_sql.py | 327 +++++++++++++++++++++++++ 5 files changed, 439 insertions(+), 4 deletions(-) create mode 100644 tests/golden/dev1747_sql_baseline.json create mode 100644 tests/test_dev1747_golden_sql.py diff --git a/.claude/skills/slayer-query.md b/.claude/skills/slayer-query.md index d66a3ec0..70021da6 100644 --- a/.claude/skills/slayer-query.md +++ b/.claude/skills/slayer-query.md @@ -22,7 +22,7 @@ A `SlayerQuery` is a JSON/dict object. The same shape works across the REST API, `order[].column` is the short alias (`count`, `revenue_sum`) — not the colon form. -**Ordering by something you don't project.** `order` may name an undeclared column/aggregate/expression ("top-N by X, show only Y, Z"). Computed hidden, sorted on, and stripped from the result: an **aggregate** (`amount:sum`, `customers.revenue:sum`), an inline **transform** (`rank(amount:sum)`, `change(...)`, `cumsum`/`lag`/`lead`/`ntile`), an inline **composite** (`revenue:sum / cnt:sum`, `abs(amount:sum)`), and a **windowed** aggregate (`amount:sum(window='90d')`, alone or inside a composite). A **raw row column** is orderable only in a raw-rows query (`distinct_dimension_values: false`); in a grouped/dedup query it's rejected (HTTP 400 — add it to `dimensions` or order by an aggregate of it). A **joined** row column is rejected — project it. Order expressions must use formula syntax for their operands, not the `name`s of measures declared in the same query: `{"column": "revenue:sum / cnt:sum"}` works, `{"column": "rev / cnt"}` is rejected. +**Ordering by something you don't project.** `order` may name an undeclared column/aggregate/expression ("top-N by X, show only Y, Z"). Computed hidden, sorted on, and stripped from the result: an **aggregate** (`amount:sum`, `customers.revenue:sum`), an inline **transform** (`rank(amount:sum)`, `change(...)`, `cumsum`/`lag`/`lead`/`ntile`), an inline **composite** (`revenue:sum / cnt:sum`, `abs(amount:sum)`), and a **windowed** aggregate (`amount:sum(window='90d')`, alone or inside a composite). A **raw row column** sorts directly in a raw-rows query (`distinct_dimension_values: false`); in a grouped/dedup query there is no single value per group, so it sorts **per group** by the extreme the direction puts first — `asc` by each group's `min`, `desc` by each group's `max`. Write `{"column": "created_at:max", "direction": "asc"}` explicitly for the other one. A **joined** row column (`customers.regions.name`), and a derived column whose `sql` reaches through a join, behave the same way — the join is pulled in for the sort, and in a grouped query the wrap is computed per host row-group rather than globally. NULLs sort **last** in both directions on every database (SQL Server excepted: its native ordering is used, because the portable emulation makes the statement fail there). An order target SLayer cannot resolve is an error, never a silently unsorted result. Order expressions must use formula syntax for their operands, not the `name`s of measures declared in the same query: `{"column": "revenue:sum / cnt:sum"}` works, `{"column": "rev / cnt"}` is rejected. **Dim-only queries deduplicate.** A query with no measures and at least one dimension or time-dimension auto-emits `GROUP BY ` and returns the distinct combinations. The `GROUP BY` is applied before `LIMIT`, so a row cap can't silently drop unique tuples. To opt out, set `"distinct_dimension_values": false` on the query — emits raw rows (no top-level `GROUP BY`), with WHERE / ORDER BY / LIMIT applied as usual. Any measure reference in `measures` / `filters` / `order` raises `DistinctDimensionValuesError` in this mode. diff --git a/DECISIONS.md b/DECISIONS.md index 7bc3e2d2..28f6e801 100644 --- a/DECISIONS.md +++ b/DECISIONS.md @@ -103,3 +103,5 @@ implementation detail. Include issue refs when known. - 2026-08-05 — One Mode-A door, plan-time filter routing, structural reachability (DEV-1745, PR 2 of the DEV-1742 series). **P-A:** every fragment of free (Mode-A) SQL now enters a SELECT scope through ONE door — `ScopeFrame.enter_predicate` / `enter_expression`, two surfaces over one implementation differing only in the parse helper (the `SELECT 1 WHERE …` statement-keyword guard for predicates). The door prequotes, parses, scans, expands, re-parses, re-scans, unions both scans into `join_paths`, then applies Law 2. Join discovery is a side effect of entering, so a caller cannot forget it; the DEV-1494 dual-scan contract is preserved because a dotted ref whose derived column inlines to a constant is only visible to the PRE-expansion scan. The surface's grammar is a static property of the field being read (`Column.filter` and model `filters` are predicates, `Column.sql` is an expression) — deliberately no content sniffing and no "try one grammar then the other" retry, either of which would put classification back into render time. **The door adds NO qualification pass:** `expand_derived_refs_sync` already qualifies, against the OWNING model's canonical alias, and deliberately leaves a node alone when its alias path does not resolve because that is an opaque CTE / subquery reference — a blanket pass against `root_relation` would corrupt exactly those. `SQLGenerator._parse` and `_parse_predicate` were byte-identical apart from one line and now delegate to a shared `slayer/sql/render/parse.py`, which is what lets the door take over a call site without changing the SQL it emits. **Failure is loud:** an unparseable Mode-A fragment raises `ModeASqlParseError` carrying the fragment and its location. Three swallow-all `except Exception` lanes leave the production path, including `_filter_join_paths._scan`, which converted a parse failure into ZERO join paths — emitting a query missing its joins rather than reporting the problem. Two consequences on emitted SQL were accepted explicitly: undeclared bare identifiers in Mode-A text are now qualified against the scope root (the old fallbacks qualified only DECLARED columns, leaving the rest to bind to whatever was in scope after re-rendering inside a rerooted CTE), and the shifted CTE renders its model filter from the door's AST rather than passing regex-substituted text through, so sqlglot's canonical form appears. **P-D:** the outer combined-SELECT WHERE routing is decided by the planner (`PlannedQuery.outer_where_filter_ids`) and consumed verbatim; the generator's render-time re-walk of `filters_by_phase` is deleted. **Reachability is structural (5.3):** `classify_host_filter` routed derived-column references by asking whether the declaring model's NAME appeared anywhere in `target_path`. That flat membership test called a SIBLING branch reachable whenever it shared a name with the target path, and called a host-model derived column whose `Column.sql` crossed INTO the target host-local. It is replaced by one rule for every key kind — a dependency is reachable iff its anchored join path is a PREFIX of `target_path` — computed per filter at plan time by `slayer/engine/filter_reachability.py`, recursively over the whole key tree, failing CLOSED on an unhandled kind. The summary lives on `PlannedQuery`, NOT on `ColumnSqlKey` (interned, and `_reroot_path_ref` copies unknown fields through rerooting stale) and NOT on `ValueSlot` (`filter_referenced_slot_ids` silently skips keys with no interned slot — a filter-only derived column is exactly such a key — and slots are copied wholesale into nested plans); it is recomputed per plan so the paths always mean what they say relative to the root they were anchored at. **Warnings (5.5):** dropped-filter emission moves from mid-render (once per cross-model plan, so nested subplans double-fired, and silent on any path that never reached that step) to the engine boundary — collected across every plan, deduped per user filter on `(location, original text)`, with drop reasons required to AGREE (disagreement raises rather than keeping the first, which is why the reason is now target-independent). `SlayerResponse.warnings` widens to a `kind`-discriminated union and is surfaced by REST, MCP and the CLI (stderr, so stdout stays pipeable). **Two live bugs fixed in passing:** a `Column.sql` that is exactly one bare reference to another derived column was silently not expanded, because `col.replace()` is a no-op when that column IS the parsed root; and a `_cm_` CTE never registered the joins its aggregation template fragments crossed, emitting `SUM(customers.spend * regions.weight) FROM customers` — SQL no database accepts. **P-J:** the superseded helpers stay callable and test-pinned rather than deleted (state 1); newly unreferenced by production are `_render_model_filter_sql`, `_filter_join_paths`, `_render_mode_a_predicate`, `_expand_degenerate_derived_root`, `_column_ref_is_derived`, `_predicate_references_derived` and `_qualify_mode_a_sql_filter`. `_qualify_column_filter_sql` remains live via `_expand_column_filter_sql`'s bundle-less branch, and `_BARE_IDENT_RE` remains in use outside the Mode-A surfaces. - 2026-08-06 — Scope assembly: null-safe grain doctrine, pagination through the dialect, and the combined layer as one AST (DEV-1746, PR 3 of the DEV-1742 consolidation). Six consequences worth recording. (1) **One grain join-back builder** (`slayer/sql/render/joins.py`), taking EXPRESSION operand pairs and returning `None` for an empty grain so the caller emits `CROSS JOIN` — a scalar aggregate has no grain to join on, and a truthy `TRUE` would erase that distinction. It replaced a string round-trip that re-parsed pre-quoted operands: SLayer's public aliases are dotted, and on BigQuery the round-trip re-read `_base."orders.customers.status"` as a multi-part *reference*, emitting a qualifier for a table that does not exist (``_base___orders___customers`.`status``). Three golden entries recorded `ScopeLeakError` for exactly that and now record real SQL. The `_wm_` INNER grain went null-safe in the same move: it compared with a plain `=`, so a NULL-dimension group matched nothing and silently received NULL — while the outer join-back and the `_cm_` join-back were already null-safe, so the answer depended on which isolation shape a measure landed in. Executed on DuckDB and SQLite; the integration test that asserted the old behaviour said so in its NAME (`test_null_dimension_group_gets_null_windowed_value`). (2) **Pagination is a dialect-strategy method.** `apply_pagination` is the one place LIMIT/OFFSET is expressed; the T-SQL override owns its rule explicitly (TOP for a bare limit; a deterministic `ORDER BY (SELECT NULL)` before OFFSET/FETCH when the query is unordered, because SQL Server rejects OFFSET without ordering) rather than relying on sqlglot happening to apply it — so the rule survives a sqlglot upgrade and lands in the AST where callers can see it. The cross-model combined path used to append raw text and emitted a literal `LIMIT` on SQL Server, while the SAME query carrying a transform layer went through the outer wrap and came out correct. (3) **The combined statement is one `exp.Select`.** Projection, FROM/JOIN chain, WHERE, ORDER BY and pagination were all string-assembled; the WHERE in particular was glued on with a hand-rolled `"\nAND "` / `"\nWHERE "` connector that chose between them by asking whether an earlier filter existed. `Select.where` conjoins, so that choice disappears. The WITH chain is assembled by `slayer/sql/render/cte_assembly.py` from **declared** `(name, query, depends_on)` entries, never dependencies discovered by scanning the rendered statement: a scan cannot tell a CTE reference from a same-named real table, is defeated by quoting and case folding, and would silently mis-order rather than fail. **CTE bodies stay AST end to end** — the first attempt kept the renderers returning text and parsed at the seam, which re-introduced exactly the dotted-alias corruption the same PR had just removed. One documented parse seam remains: the re-rooted cross-model CTE arrives as a complete nested `WITH … SELECT` from `generate_from_planned`. The combined ORDER BY had to become AST for the same reason — rendering its terms and re-parsing corrupted them into `` `orders___customers`.`spend_sum` ``. (4) **`PlannedQuery.projection` is consumed verbatim** (B7). The combined projection was assembled in four grouped passes, so a cross-model measure declared first was emitted last; the generator admitted it in a comment. Hidden-slot trimming stops being a mechanism at all — a hidden slot is absent from the list. Two subtleties: a slot may legitimately appear MORE THAN ONCE (C13 lets one key be selected under several names, and the plan lists it once per name), so each occurrence consumes the NEXT of that slot's rendered columns — emitting the whole list per occurrence projected every alias once per name; and because pydantic's `model_copy(update=…)` skips validators and rerooting uses exactly that, one renderer-side assertion is kept at the single entry point, raising rather than skipping, since silently dropping a column the plan asked for is how a wrong answer reaches a user. (5) **One materialiser on the cross-model path.** `ScopeFrame` gained `materialize_for(expr, consumer=…)` — `resolve` anchors a ref and closes it, this is the second half alone, for a producer that anchored its own template (a date-truncated grain, a value carrying its column's declared CAST; re-deriving those from a ref would project a different expression than the aggregate consumes). Unifying the dedup tables fixed a defect: grouping a first/last cross-model aggregate by the same crossing expression it aggregates projected that expression TWICE (`regions.weight AS _val_0` and `AS _val_1`), because the grain site and the value site kept separate maps. `_build_first_last_base_select` keeps its own flow until PR 5's `RankedAggregatePlan`; the boundary is pinned by a test that fails when it moves, rather than left as an assumption. (6) **The empty-base grain is a typed plan node** (`EmptyBaseGrainPlan`, presence as the discriminator, carrying the host-local filter ids). No `grain_slot_ids` field — in that shape the grain is empty by definition — and the invariant that field would have encoded had to be restated during implementation, because `CrossModelAggregatePlan.shared_grain_slots` can name a HIDDEN row slot a filter created, which never becomes a projected grain column. Emitted SQL byte-identical. Reformatting churn was accepted and reviewed: emitting the statement in one pass means sqlglot's printer indents CTE bodies, which moved 25 golden entries and broke assertions that pinned layout rather than structure — one test helper (`_extract_src_body`, anchored on the exact text `"\n) AS _src"`) accounted for 19 failures by itself and is now whitespace-tolerant. Also fixed in passing, because it was two lines and not a refactor: a filter on a JOINED model's crossing derived column emitted invalid SQL — both the join scanner and the filter renderer expanded the column without `is_root=False`, so a further-joined ref came out bare, the scanner could not match it to a join path, the hop was never joined, and the filter referenced a table absent from the FROM. The two must stay in lockstep, since discovery scans the same expansion the renderer emits. + +- 2026-08-06 — Typed re-rooting, host-grain ordering, and one ORDER BY resolver (DEV-1747, PR 4 of the DEV-1742 consolidation). (1) **A pre-bound seam** (`slayer/engine/prebound.py`). `plan_query` split into `bind_query_inputs` → `PreboundQuery` and the planning that consumes it, so a nested sub-plan is handed TYPED, already-bound inputs instead of a regenerated formula string. `StrictQueryCarrier` forbids extra attributes and raises on any field the seam does not approve, which is what makes "the sub-plan reads only what was handed to it" checkable rather than aspirational; the negative proof is that `cross_model_planner` no longer imports `bind_expr` / `bind_filter` / `parse_expr` at all. (2) **Host-grain aggregates** (`AggregateKey.grain="host"`). A path-bearing source always routed to a TARGET-rooted CTE, which for a sort key degenerates to a scalar CROSS JOIN — every group gets the same global value and the sort silently does nothing, so the case was rejected outright instead. The marker separates where a value is READ (through the join) from where it is GROUPED (per host row-group), and the decision lives in `classify_isolation` with the other two. This was impossible before the seam, not merely tidier: formula text cannot express a path-bearing source or a grain marker, and routing one through the old text path bound `name:min` against the wrong model. (3) **Direction-aware wraps.** An undeclared row column in a grouped query wraps as `min` for ASC and `max` for DESC — the extreme the direction actually puts first. The unconditional `MAX` sorted an ascending query by each group's LARGEST member, which differs whenever groups overlap in range. (4) **One order-term resolver** (`slayer/sql/render/order_terms.py`), dispatching on the plan-assigned `OrderScope` with no precedence and no fallback. Four render sites had each re-derived the answer and disagreed on three things, each a bug: an unresolvable slot returned unsorted rows with no error anywhere on four of the five paths; null ordering skipped the dialect strategy on two, so the same query sorted NULLs differently depending on whether it carried a transform; and a PROJECTED cross-model aggregate was named by its CTE column while the SELECT projected it under the user's alias, so the term resolved only by falling through to an input column of the FROM. Null ordering is now nulls-last everywhere by policy (T-SQL excepted, where the portable emulation puts a bracketed alias inside a CASE that re-resolves against the FROM and fails); a WINDOW's internal `ORDER BY` deliberately takes the emitter's NATIVE ordering instead, since an emulation term inside a frame changes which rows the frame covers. (5) **A crossing sort key is a crossing ref.** An ORDER-BY-only target is not in `base_render_order` — materialising it there would project it and change the grain — but Law 1 does not care that ORDER BY is the only thing referencing it, so the derived-dimension scope pass walks order targets and the render-time throwaway-`ScopeFrame` probe that existed solely to DETECT the crossing is gone. (6) **The transform chains hand the assembler AST.** Both were still splicing `WITH` out of f-strings and reading each predecessor positionally (`ctes[-1][0]`); they now declare dependencies, and the bodies stay `exp.Select` end to end because this path carries dotted `.` names that a text round trip re-reads as multi-part references. (7) **Audit and routing are different statements.** The re-rooting fix that stopped blanking the routing lists was right about `applied_filter_ids` (an audit: some scope evaluates this) and wrong about `where_filter_ids` / `having_filter_ids` (an instruction: the forward CTE took it over, so the host base must skip it). A re-rooted plan has no forward CTE and its predicate is host-evaluable by construction, so keeping the routing told the host to skip a filter nothing else applied there, and rows the user excluded came back carrying a NULL measure. Only the integration suite could see it — the unit tests assert on plan fields, and the plan was self-consistent. diff --git a/docs/concepts/queries.md b/docs/concepts/queries.md index 0d2e2873..90e6d229 100644 --- a/docs/concepts/queries.md +++ b/docs/concepts/queries.md @@ -128,9 +128,23 @@ What each shape of an *undeclared* order target does: | An inline **transform** (`rank(amount:sum)`, `cumsum(...)`, `change(...)`, `lag`/`lead`/`ntile`) | Computed hidden, sorted on, stripped. | | An inline **composite** (`revenue:sum / cnt:sum`, `abs(amount:sum)`, `change(amount:sum) / 2`) | Computed hidden, sorted on, stripped. | | A **windowed** aggregate (`amount:sum(window='90d')`), alone or inside a composite | Computed hidden in its own rolling-window CTE, sorted on, stripped. | -| A raw row column, in a **raw-rows** query (`distinct_dimension_values: false`, no measures) | Sorted on directly (`ORDER BY orders.created_at`). | -| A raw row column, in an **aggregated / dedup** query | Rejected (HTTP 400): it isn't in the `GROUP BY`. Add it to `dimensions`, or order by an aggregate of it (`created_at:max`). | -| A **joined** row column (`customers.regions.name`) not projected | Rejected (HTTP 400): project it (add to `dimensions`) or order by a projected field. | +| A raw row column, in a **raw-rows** query (`distinct_dimension_values: false`, no measures) | Sorted on directly (`ORDER BY orders.created_at`). Applies to a **joined** column (`customers.regions.name`) and to a derived column whose `sql` reaches through a join — the join is pulled in for the sort. | +| A raw row column, in an **aggregated / dedup** query | Sorted on **per group**: ASC by each group's minimum, DESC by each group's maximum. The wrap is implicit — you do not write `created_at:min`. | +| A **joined** row column (`customers.regions.name`) in an aggregated query | Same per-group wrap, computed in a CTE rooted at the source model with the join pulled inside, so each group gets its own extreme rather than one global value. | + +Ordering by an undeclared row column in a grouped query is **not** the same as +ordering by the column itself — there is no single value per group to sort by. +SLayer picks the extreme the direction puts first: `asc` sorts each group by its +`min`, `desc` by its `max`. Write `{"column": "created_at:max", "direction": "asc"}` +explicitly if you want the other one. + +NULLs sort **last** in both directions, on every database, so the same query +returns the same row order regardless of backend. (SQL Server is the one +exception: its native ordering is used because the portable emulation makes the +statement fail there.) + +An order target that names nothing SLayer can resolve is an error, never a +silently unsorted result. Transform and composite order targets accept the full formula syntax, so `{"column": "revenue:sum / cnt:sum"}` and `{"column": "change(revenue:sum)"}` both diff --git a/tests/golden/dev1747_sql_baseline.json b/tests/golden/dev1747_sql_baseline.json new file mode 100644 index 00000000..2f1b8d97 --- /dev/null +++ b/tests/golden/dev1747_sql_baseline.json @@ -0,0 +1,92 @@ +{ + "chain/cross_model_window::bigquery": "SELECT\n `orders___created_at`,\n `orders___cs`,\n `orders___run`\nFROM (\nWITH _base AS (\n SELECT\n DATE_TRUNC(orders.created_at, MONTH) AS `orders___created_at`\n FROM orders AS orders\n GROUP BY\n DATE_TRUNC(orders.created_at, MONTH)\n), _cm_orders__customers__spend_sum AS (\n SELECT\n SUM(customers.spend) AS `orders___customers___spend_sum`\n FROM customers AS customers\n), base AS (\n SELECT\n _base.`orders___created_at`,\n _cm_orders__customers__spend_sum.`orders___customers___spend_sum` AS `orders___cs`\n FROM _base\n CROSS JOIN _cm_orders__customers__spend_sum\n), step1 AS (\n SELECT\n `orders___created_at`,\n `orders___cs`,\n SUM(`orders___cs`) OVER (ORDER BY `orders___created_at`) AS `orders___run`\n FROM base\n)\nSELECT\n `orders___created_at`,\n `orders___cs`,\n `orders___run`\nFROM step1\n) AS _outer", + "chain/cross_model_window::duckdb": "SELECT\n \"orders.created_at\",\n \"orders.cs\",\n \"orders.run\"\nFROM (\nWITH _base AS (\n SELECT\n DATE_TRUNC('MONTH', orders.created_at) AS \"orders.created_at\"\n FROM orders AS orders\n GROUP BY\n DATE_TRUNC('MONTH', orders.created_at)\n), _cm_orders__customers__spend_sum AS (\n SELECT\n SUM(customers.spend) AS \"orders.customers.spend_sum\"\n FROM customers AS customers\n), base AS (\n SELECT\n _base.\"orders.created_at\",\n _cm_orders__customers__spend_sum.\"orders.customers.spend_sum\" AS \"orders.cs\"\n FROM _base\n CROSS JOIN _cm_orders__customers__spend_sum\n), step1 AS (\n SELECT\n \"orders.created_at\",\n \"orders.cs\",\n SUM(\"orders.cs\") OVER (ORDER BY \"orders.created_at\") AS \"orders.run\"\n FROM base\n)\nSELECT\n \"orders.created_at\",\n \"orders.cs\",\n \"orders.run\"\nFROM step1\n) AS _outer", + "chain/cross_model_window::postgres": "SELECT\n \"orders.created_at\",\n \"orders.cs\",\n \"orders.run\"\nFROM (\nWITH _base AS (\n SELECT\n DATE_TRUNC('MONTH', orders.created_at) AS \"orders.created_at\"\n FROM orders AS orders\n GROUP BY\n DATE_TRUNC('MONTH', orders.created_at)\n), _cm_orders__customers__spend_sum AS (\n SELECT\n SUM(customers.spend) AS \"orders.customers.spend_sum\"\n FROM customers AS customers\n), base AS (\n SELECT\n _base.\"orders.created_at\",\n _cm_orders__customers__spend_sum.\"orders.customers.spend_sum\" AS \"orders.cs\"\n FROM _base\n CROSS JOIN _cm_orders__customers__spend_sum\n), step1 AS (\n SELECT\n \"orders.created_at\",\n \"orders.cs\",\n SUM(\"orders.cs\") OVER (ORDER BY \"orders.created_at\") AS \"orders.run\"\n FROM base\n)\nSELECT\n \"orders.created_at\",\n \"orders.cs\",\n \"orders.run\"\nFROM step1\n) AS _outer", + "chain/cross_model_window::sqlite": "SELECT\n \"orders.created_at\",\n \"orders.cs\",\n \"orders.run\"\nFROM (\nWITH _base AS (\n SELECT\n STRFTIME('%Y-%m-01', orders.created_at) AS \"orders.created_at\"\n FROM orders AS orders\n GROUP BY\n STRFTIME('%Y-%m-01', orders.created_at)\n), _cm_orders__customers__spend_sum AS (\n SELECT\n SUM(customers.spend) AS \"orders.customers.spend_sum\"\n FROM customers AS customers\n), base AS (\n SELECT\n _base.\"orders.created_at\",\n _cm_orders__customers__spend_sum.\"orders.customers.spend_sum\" AS \"orders.cs\"\n FROM _base\n CROSS JOIN _cm_orders__customers__spend_sum\n), step1 AS (\n SELECT\n \"orders.created_at\",\n \"orders.cs\",\n SUM(\"orders.cs\") OVER (ORDER BY \"orders.created_at\") AS \"orders.run\"\n FROM base\n)\nSELECT\n \"orders.created_at\",\n \"orders.cs\",\n \"orders.run\"\nFROM step1\n) AS _outer", + "chain/cross_model_window::tsql": "WITH _base AS (\n SELECT\n DATETRUNC(MONTH, orders.created_at) AS [orders___created_at]\n FROM orders AS orders\n GROUP BY\n DATETRUNC(MONTH, orders.created_at)\n), _cm_orders__customers__spend_sum AS (\n SELECT\n SUM(customers.spend) AS [orders___customers___spend_sum]\n FROM customers AS customers\n), base AS (\n SELECT\n _base.[orders___created_at] AS [orders___created_at],\n _cm_orders__customers__spend_sum.[orders___customers___spend_sum] AS [orders___cs]\n FROM _base\n CROSS JOIN _cm_orders__customers__spend_sum\n), step1 AS (\n SELECT\n [orders___created_at] AS [orders___created_at],\n [orders___cs] AS [orders___cs],\n SUM([orders___cs]) OVER (ORDER BY [orders___created_at]) AS [orders___run]\n FROM base\n)\nSELECT\n [orders___created_at],\n [orders___cs],\n [orders___run]\nFROM (\n SELECT\n [orders___created_at] AS [orders___created_at],\n [orders___cs] AS [orders___cs],\n [orders___run] AS [orders___run]\n FROM step1\n) AS _outer", + "chain/local_consecutive_periods::bigquery": "SELECT\n `orders___created_at`,\n `orders___rev`,\n `orders___streak`\nFROM (\nWITH base AS (\n SELECT\n DATE_TRUNC(orders.created_at, MONTH) AS `orders___created_at`,\n CAST(SUM(orders.amount) AS FLOAT64) AS `orders___rev`\n FROM orders AS orders\n GROUP BY\n DATE_TRUNC(orders.created_at, MONTH)\n), cp_reset_streak AS (\n SELECT\n `orders___created_at`,\n `orders___rev`,\n SUM(CASE WHEN NOT `orders___rev` IS NULL AND `orders___rev` <> 0 THEN 0 ELSE 1 END) OVER (ORDER BY `orders___created_at` ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS `_cp_reset_orders___streak`\n FROM base\n), cp_value_streak AS (\n SELECT\n `orders___created_at`,\n `orders___rev`,\n CASE\n WHEN NOT `orders___rev` IS NULL AND `orders___rev` <> 0\n THEN SUM(CASE WHEN NOT `orders___rev` IS NULL AND `orders___rev` <> 0 THEN 1 ELSE 0 END) OVER (\n PARTITION BY `_cp_reset_orders___streak`\n ORDER BY `orders___created_at`\n ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW\n )\n ELSE 0\n END AS `orders___streak`\n FROM cp_reset_streak\n)\nSELECT\n `orders___created_at`,\n `orders___rev`,\n `orders___streak`\nFROM cp_value_streak\n) AS _outer", + "chain/local_consecutive_periods::duckdb": "SELECT\n \"orders.created_at\",\n \"orders.rev\",\n \"orders.streak\"\nFROM (\nWITH base AS (\n SELECT\n DATE_TRUNC('MONTH', orders.created_at) AS \"orders.created_at\",\n CAST(SUM(orders.amount) AS DOUBLE) AS \"orders.rev\"\n FROM orders AS orders\n GROUP BY\n DATE_TRUNC('MONTH', orders.created_at)\n), cp_reset_streak AS (\n SELECT\n \"orders.created_at\",\n \"orders.rev\",\n SUM(CASE WHEN NOT \"orders.rev\" IS NULL AND \"orders.rev\" <> 0 THEN 0 ELSE 1 END) OVER (ORDER BY \"orders.created_at\" ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS \"_cp_reset_orders.streak\"\n FROM base\n), cp_value_streak AS (\n SELECT\n \"orders.created_at\",\n \"orders.rev\",\n CASE\n WHEN NOT \"orders.rev\" IS NULL AND \"orders.rev\" <> 0\n THEN SUM(CASE WHEN NOT \"orders.rev\" IS NULL AND \"orders.rev\" <> 0 THEN 1 ELSE 0 END) OVER (\n PARTITION BY \"_cp_reset_orders.streak\"\n ORDER BY \"orders.created_at\"\n ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW\n )\n ELSE 0\n END AS \"orders.streak\"\n FROM cp_reset_streak\n)\nSELECT\n \"orders.created_at\",\n \"orders.rev\",\n \"orders.streak\"\nFROM cp_value_streak\n) AS _outer", + "chain/local_consecutive_periods::postgres": "SELECT\n \"orders.created_at\",\n \"orders.rev\",\n \"orders.streak\"\nFROM (\nWITH base AS (\n SELECT\n DATE_TRUNC('MONTH', orders.created_at) AS \"orders.created_at\",\n CAST(SUM(orders.amount) AS DOUBLE PRECISION) AS \"orders.rev\"\n FROM orders AS orders\n GROUP BY\n DATE_TRUNC('MONTH', orders.created_at)\n), cp_reset_streak AS (\n SELECT\n \"orders.created_at\",\n \"orders.rev\",\n SUM(CASE WHEN NOT \"orders.rev\" IS NULL AND \"orders.rev\" <> 0 THEN 0 ELSE 1 END) OVER (ORDER BY \"orders.created_at\" ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS \"_cp_reset_orders.streak\"\n FROM base\n), cp_value_streak AS (\n SELECT\n \"orders.created_at\",\n \"orders.rev\",\n CASE\n WHEN NOT \"orders.rev\" IS NULL AND \"orders.rev\" <> 0\n THEN SUM(CASE WHEN NOT \"orders.rev\" IS NULL AND \"orders.rev\" <> 0 THEN 1 ELSE 0 END) OVER (\n PARTITION BY \"_cp_reset_orders.streak\"\n ORDER BY \"orders.created_at\"\n ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW\n )\n ELSE 0\n END AS \"orders.streak\"\n FROM cp_reset_streak\n)\nSELECT\n \"orders.created_at\",\n \"orders.rev\",\n \"orders.streak\"\nFROM cp_value_streak\n) AS _outer", + "chain/local_consecutive_periods::sqlite": "SELECT\n \"orders.created_at\",\n \"orders.rev\",\n \"orders.streak\"\nFROM (\nWITH base AS (\n SELECT\n STRFTIME('%Y-%m-01', orders.created_at) AS \"orders.created_at\",\n CAST(SUM(orders.amount) AS REAL) AS \"orders.rev\"\n FROM orders AS orders\n GROUP BY\n STRFTIME('%Y-%m-01', orders.created_at)\n), cp_reset_streak AS (\n SELECT\n \"orders.created_at\",\n \"orders.rev\",\n SUM(CASE WHEN NOT \"orders.rev\" IS NULL AND \"orders.rev\" <> 0 THEN 0 ELSE 1 END) OVER (ORDER BY \"orders.created_at\" ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS \"_cp_reset_orders.streak\"\n FROM base\n), cp_value_streak AS (\n SELECT\n \"orders.created_at\",\n \"orders.rev\",\n CASE\n WHEN NOT \"orders.rev\" IS NULL AND \"orders.rev\" <> 0\n THEN SUM(CASE WHEN NOT \"orders.rev\" IS NULL AND \"orders.rev\" <> 0 THEN 1 ELSE 0 END) OVER (\n PARTITION BY \"_cp_reset_orders.streak\"\n ORDER BY \"orders.created_at\"\n ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW\n )\n ELSE 0\n END AS \"orders.streak\"\n FROM cp_reset_streak\n)\nSELECT\n \"orders.created_at\",\n \"orders.rev\",\n \"orders.streak\"\nFROM cp_value_streak\n) AS _outer", + "chain/local_consecutive_periods::tsql": "WITH base AS (\n SELECT\n DATETRUNC(MONTH, orders.created_at) AS [orders___created_at],\n CAST(SUM(orders.amount) AS FLOAT) AS [orders___rev]\n FROM orders AS orders\n GROUP BY\n DATETRUNC(MONTH, orders.created_at)\n), cp_reset_streak AS (\n SELECT\n [orders___created_at] AS [orders___created_at],\n [orders___rev] AS [orders___rev],\n SUM(CASE WHEN NOT [orders___rev] IS NULL AND [orders___rev] <> 0 THEN 0 ELSE 1 END) OVER (ORDER BY [orders___created_at] ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS [_cp_reset_orders___streak]\n FROM base\n), cp_value_streak AS (\n SELECT\n [orders___created_at] AS [orders___created_at],\n [orders___rev] AS [orders___rev],\n CASE\n WHEN NOT [orders___rev] IS NULL AND [orders___rev] <> 0\n THEN SUM(CASE WHEN NOT [orders___rev] IS NULL AND [orders___rev] <> 0 THEN 1 ELSE 0 END) OVER (\n PARTITION BY [_cp_reset_orders___streak]\n ORDER BY [orders___created_at]\n ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW\n )\n ELSE 0\n END AS [orders___streak]\n FROM cp_reset_streak\n)\nSELECT\n [orders___created_at],\n [orders___rev],\n [orders___streak]\nFROM (\n SELECT\n [orders___created_at] AS [orders___created_at],\n [orders___rev] AS [orders___rev],\n [orders___streak] AS [orders___streak]\n FROM cp_value_streak\n) AS _outer", + "chain/local_multi_step::bigquery": "SELECT\n `orders___created_at`,\n `orders___rev`,\n `orders___cs`,\n `orders___ch`\nFROM (\nWITH base AS (\n SELECT\n DATE_TRUNC(orders.created_at, MONTH) AS `orders___created_at`,\n CAST(SUM(orders.amount) AS FLOAT64) AS `orders___rev`\n FROM orders AS orders\n GROUP BY\n DATE_TRUNC(orders.created_at, MONTH)\n), step1 AS (\n SELECT\n `orders___created_at`,\n `orders___rev`,\n SUM(`orders___rev`) OVER (ORDER BY `orders___created_at`) AS `orders___cs`\n FROM base\n), shifted__time_shift_inner AS (\n SELECT\n DATE_TRUNC(CAST(orders.created_at + INTERVAL 1 MONTH AS DATETIME), MONTH) AS `orders___created_at`,\n CAST(SUM(orders.amount) AS FLOAT64) AS `orders___rev`\n FROM orders AS orders\n GROUP BY\n DATE_TRUNC(CAST(orders.created_at + INTERVAL 1 MONTH AS DATETIME), MONTH)\n), sjoin__time_shift_inner AS (\n SELECT\n step1.`orders___created_at`,\n step1.`orders___rev`,\n step1.`orders___cs`,\n shifted__time_shift_inner.`orders___rev` AS `orders____time_shift_inner`\n FROM step1\n LEFT JOIN shifted__time_shift_inner\n ON step1.`orders___created_at` IS NOT DISTINCT FROM shifted__time_shift_inner.`orders___created_at`\n), step2 AS (\n SELECT\n `orders___created_at`,\n `orders___rev`,\n `orders___cs`,\n `orders____time_shift_inner`,\n `orders___rev` - `orders____time_shift_inner` AS `orders___ch`\n FROM sjoin__time_shift_inner\n)\nSELECT\n `orders___created_at`,\n `orders___rev`,\n `orders___cs`,\n `orders____time_shift_inner`,\n `orders___ch`\nFROM step2\n) AS _outer", + "chain/local_multi_step::duckdb": "SELECT\n \"orders.created_at\",\n \"orders.rev\",\n \"orders.cs\",\n \"orders.ch\"\nFROM (\nWITH base AS (\n SELECT\n DATE_TRUNC('MONTH', orders.created_at) AS \"orders.created_at\",\n CAST(SUM(orders.amount) AS DOUBLE) AS \"orders.rev\"\n FROM orders AS orders\n GROUP BY\n DATE_TRUNC('MONTH', orders.created_at)\n), step1 AS (\n SELECT\n \"orders.created_at\",\n \"orders.rev\",\n SUM(\"orders.rev\") OVER (ORDER BY \"orders.created_at\") AS \"orders.cs\"\n FROM base\n), shifted__time_shift_inner AS (\n SELECT\n DATE_TRUNC('MONTH', CAST(orders.created_at + INTERVAL 1 MONTH AS TIMESTAMP)) AS \"orders.created_at\",\n CAST(SUM(orders.amount) AS DOUBLE) AS \"orders.rev\"\n FROM orders AS orders\n GROUP BY\n DATE_TRUNC('MONTH', CAST(orders.created_at + INTERVAL 1 MONTH AS TIMESTAMP))\n), sjoin__time_shift_inner AS (\n SELECT\n step1.\"orders.created_at\",\n step1.\"orders.rev\",\n step1.\"orders.cs\",\n shifted__time_shift_inner.\"orders.rev\" AS \"orders._time_shift_inner\"\n FROM step1\n LEFT JOIN shifted__time_shift_inner\n ON step1.\"orders.created_at\" IS NOT DISTINCT FROM shifted__time_shift_inner.\"orders.created_at\"\n), step2 AS (\n SELECT\n \"orders.created_at\",\n \"orders.rev\",\n \"orders.cs\",\n \"orders._time_shift_inner\",\n \"orders.rev\" - \"orders._time_shift_inner\" AS \"orders.ch\"\n FROM sjoin__time_shift_inner\n)\nSELECT\n \"orders.created_at\",\n \"orders.rev\",\n \"orders.cs\",\n \"orders._time_shift_inner\",\n \"orders.ch\"\nFROM step2\n) AS _outer", + "chain/local_multi_step::postgres": "SELECT\n \"orders.created_at\",\n \"orders.rev\",\n \"orders.cs\",\n \"orders.ch\"\nFROM (\nWITH base AS (\n SELECT\n DATE_TRUNC('MONTH', orders.created_at) AS \"orders.created_at\",\n CAST(SUM(orders.amount) AS DOUBLE PRECISION) AS \"orders.rev\"\n FROM orders AS orders\n GROUP BY\n DATE_TRUNC('MONTH', orders.created_at)\n), step1 AS (\n SELECT\n \"orders.created_at\",\n \"orders.rev\",\n SUM(\"orders.rev\") OVER (ORDER BY \"orders.created_at\") AS \"orders.cs\"\n FROM base\n), shifted__time_shift_inner AS (\n SELECT\n DATE_TRUNC('MONTH', CAST(orders.created_at + INTERVAL '1 MONTH' AS TIMESTAMP)) AS \"orders.created_at\",\n CAST(SUM(orders.amount) AS DOUBLE PRECISION) AS \"orders.rev\"\n FROM orders AS orders\n GROUP BY\n DATE_TRUNC('MONTH', CAST(orders.created_at + INTERVAL '1 MONTH' AS TIMESTAMP))\n), sjoin__time_shift_inner AS (\n SELECT\n step1.\"orders.created_at\",\n step1.\"orders.rev\",\n step1.\"orders.cs\",\n shifted__time_shift_inner.\"orders.rev\" AS \"orders._time_shift_inner\"\n FROM step1\n LEFT JOIN shifted__time_shift_inner\n ON step1.\"orders.created_at\" IS NOT DISTINCT FROM shifted__time_shift_inner.\"orders.created_at\"\n), step2 AS (\n SELECT\n \"orders.created_at\",\n \"orders.rev\",\n \"orders.cs\",\n \"orders._time_shift_inner\",\n \"orders.rev\" - \"orders._time_shift_inner\" AS \"orders.ch\"\n FROM sjoin__time_shift_inner\n)\nSELECT\n \"orders.created_at\",\n \"orders.rev\",\n \"orders.cs\",\n \"orders._time_shift_inner\",\n \"orders.ch\"\nFROM step2\n) AS _outer", + "chain/local_multi_step::sqlite": "SELECT\n \"orders.created_at\",\n \"orders.rev\",\n \"orders.cs\",\n \"orders.ch\"\nFROM (\nWITH base AS (\n SELECT\n STRFTIME('%Y-%m-01', orders.created_at) AS \"orders.created_at\",\n CAST(SUM(orders.amount) AS REAL) AS \"orders.rev\"\n FROM orders AS orders\n GROUP BY\n STRFTIME('%Y-%m-01', orders.created_at)\n), step1 AS (\n SELECT\n \"orders.created_at\",\n \"orders.rev\",\n SUM(\"orders.rev\") OVER (ORDER BY \"orders.created_at\") AS \"orders.cs\"\n FROM base\n), shifted__time_shift_inner AS (\n SELECT\n STRFTIME('%Y-%m-01', DATE(orders.created_at, '1 months')) AS \"orders.created_at\",\n CAST(SUM(orders.amount) AS REAL) AS \"orders.rev\"\n FROM orders AS orders\n GROUP BY\n STRFTIME('%Y-%m-01', DATE(orders.created_at, '1 months'))\n), sjoin__time_shift_inner AS (\n SELECT\n step1.\"orders.created_at\",\n step1.\"orders.rev\",\n step1.\"orders.cs\",\n shifted__time_shift_inner.\"orders.rev\" AS \"orders._time_shift_inner\"\n FROM step1\n LEFT JOIN shifted__time_shift_inner\n ON step1.\"orders.created_at\" IS shifted__time_shift_inner.\"orders.created_at\"\n), step2 AS (\n SELECT\n \"orders.created_at\",\n \"orders.rev\",\n \"orders.cs\",\n \"orders._time_shift_inner\",\n \"orders.rev\" - \"orders._time_shift_inner\" AS \"orders.ch\"\n FROM sjoin__time_shift_inner\n)\nSELECT\n \"orders.created_at\",\n \"orders.rev\",\n \"orders.cs\",\n \"orders._time_shift_inner\",\n \"orders.ch\"\nFROM step2\n) AS _outer", + "chain/local_multi_step::tsql": "WITH base AS (\n SELECT\n DATETRUNC(MONTH, orders.created_at) AS [orders___created_at],\n CAST(SUM(orders.amount) AS FLOAT) AS [orders___rev]\n FROM orders AS orders\n GROUP BY\n DATETRUNC(MONTH, orders.created_at)\n), step1 AS (\n SELECT\n [orders___created_at] AS [orders___created_at],\n [orders___rev] AS [orders___rev],\n SUM([orders___rev]) OVER (ORDER BY [orders___created_at]) AS [orders___cs]\n FROM base\n), shifted__time_shift_inner AS (\n SELECT\n DATETRUNC(MONTH, CAST(DATEADD(MONTH, 1, orders.created_at) AS DATETIME2)) AS [orders___created_at],\n CAST(SUM(orders.amount) AS FLOAT) AS [orders___rev]\n FROM orders AS orders\n GROUP BY\n DATETRUNC(MONTH, CAST(DATEADD(MONTH, 1, orders.created_at) AS DATETIME2))\n), sjoin__time_shift_inner AS (\n SELECT\n step1.[orders___created_at] AS [orders___created_at],\n step1.[orders___rev] AS [orders___rev],\n step1.[orders___cs] AS [orders___cs],\n shifted__time_shift_inner.[orders___rev] AS [orders____time_shift_inner]\n FROM step1\n LEFT JOIN shifted__time_shift_inner\n ON (\n step1.[orders___created_at] = shifted__time_shift_inner.[orders___created_at]\n OR (\n step1.[orders___created_at] IS NULL\n AND shifted__time_shift_inner.[orders___created_at] IS NULL\n )\n )\n), step2 AS (\n SELECT\n [orders___created_at] AS [orders___created_at],\n [orders___rev] AS [orders___rev],\n [orders___cs] AS [orders___cs],\n [orders____time_shift_inner] AS [orders____time_shift_inner],\n [orders___rev] - [orders____time_shift_inner] AS [orders___ch]\n FROM sjoin__time_shift_inner\n)\nSELECT\n [orders___created_at],\n [orders___rev],\n [orders___cs],\n [orders___ch]\nFROM (\n SELECT\n [orders___created_at] AS [orders___created_at],\n [orders___rev] AS [orders___rev],\n [orders___cs] AS [orders___cs],\n [orders____time_shift_inner] AS [orders____time_shift_inner],\n [orders___ch] AS [orders___ch]\n FROM step2\n) AS _outer", + "order/combined_cross_model::bigquery": "WITH _base AS (\n SELECT\n orders.status AS `orders___status`,\n CAST(SUM(orders.amount) AS FLOAT64) AS `orders___rev`\n FROM orders AS orders\n GROUP BY\n orders.status\n), _cm_orders__customers__spend_sum AS (\n SELECT\n SUM(customers.spend) AS `orders___customers___spend_sum`\n FROM customers AS customers\n)\nSELECT\n _base.`orders___status`,\n _base.`orders___rev`,\n _cm_orders__customers__spend_sum.`orders___customers___spend_sum` AS `orders___cs`\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_sum\nORDER BY\n _cm_orders__customers__spend_sum.`orders___customers___spend_sum` DESC", + "order/combined_cross_model::duckdb": "WITH _base AS (\n SELECT\n orders.status AS \"orders.status\",\n CAST(SUM(orders.amount) AS DOUBLE) AS \"orders.rev\"\n FROM orders AS orders\n GROUP BY\n orders.status\n), _cm_orders__customers__spend_sum AS (\n SELECT\n SUM(customers.spend) AS \"orders.customers.spend_sum\"\n FROM customers AS customers\n)\nSELECT\n _base.\"orders.status\",\n _base.\"orders.rev\",\n _cm_orders__customers__spend_sum.\"orders.customers.spend_sum\" AS \"orders.cs\"\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_sum\nORDER BY\n _cm_orders__customers__spend_sum.\"orders.customers.spend_sum\" DESC", + "order/combined_cross_model::postgres": "WITH _base AS (\n SELECT\n orders.status AS \"orders.status\",\n CAST(SUM(orders.amount) AS DOUBLE PRECISION) AS \"orders.rev\"\n FROM orders AS orders\n GROUP BY\n orders.status\n), _cm_orders__customers__spend_sum AS (\n SELECT\n SUM(customers.spend) AS \"orders.customers.spend_sum\"\n FROM customers AS customers\n)\nSELECT\n _base.\"orders.status\",\n _base.\"orders.rev\",\n _cm_orders__customers__spend_sum.\"orders.customers.spend_sum\" AS \"orders.cs\"\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_sum\nORDER BY\n _cm_orders__customers__spend_sum.\"orders.customers.spend_sum\" DESC NULLS LAST", + "order/combined_cross_model::sqlite": "WITH _base AS (\n SELECT\n orders.status AS \"orders.status\",\n CAST(SUM(orders.amount) AS REAL) AS \"orders.rev\"\n FROM orders AS orders\n GROUP BY\n orders.status\n), _cm_orders__customers__spend_sum AS (\n SELECT\n SUM(customers.spend) AS \"orders.customers.spend_sum\"\n FROM customers AS customers\n)\nSELECT\n _base.\"orders.status\",\n _base.\"orders.rev\",\n _cm_orders__customers__spend_sum.\"orders.customers.spend_sum\" AS \"orders.cs\"\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_sum\nORDER BY\n _cm_orders__customers__spend_sum.\"orders.customers.spend_sum\" DESC", + "order/combined_cross_model::tsql": "WITH _base AS (\n SELECT\n orders.status AS [orders___status],\n CAST(SUM(orders.amount) AS FLOAT) AS [orders___rev]\n FROM orders AS orders\n GROUP BY\n orders.status\n), _cm_orders__customers__spend_sum AS (\n SELECT\n SUM(customers.spend) AS [orders___customers___spend_sum]\n FROM customers AS customers\n)\nSELECT\n _base.[orders___status],\n _base.[orders___rev],\n _cm_orders__customers__spend_sum.[orders___customers___spend_sum] AS [orders___cs]\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_sum\nORDER BY\n _cm_orders__customers__spend_sum.[orders___customers___spend_sum] DESC", + "order/grouped_derived_crossing::bigquery": "WITH _base AS (\n SELECT\n orders.status AS `orders___status`,\n CAST(SUM(orders.amount) AS FLOAT64) AS `orders___rev`\n FROM orders AS orders\n GROUP BY\n orders.status\n), _cm_orders__cust_region_min AS (\n SELECT\n orders.status AS `orders___status`,\n MIN(customers__regions.name) AS `orders___cust_region_min`\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n LEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\n GROUP BY\n orders.status\n)\nSELECT\n _base.`orders___status`,\n _base.`orders___rev`\nFROM _base\nLEFT JOIN _cm_orders__cust_region_min\n ON _base.`orders___status` IS NOT DISTINCT FROM _cm_orders__cust_region_min.`orders___status`\nORDER BY\n _cm_orders__cust_region_min.`orders___cust_region_min` ASC NULLS LAST", + "order/grouped_derived_crossing::duckdb": "WITH _base AS (\n SELECT\n orders.status AS \"orders.status\",\n CAST(SUM(orders.amount) AS DOUBLE) AS \"orders.rev\"\n FROM orders AS orders\n GROUP BY\n orders.status\n), _cm_orders__cust_region_min AS (\n SELECT\n orders.status AS \"orders.status\",\n MIN(customers__regions.name) AS \"orders.cust_region_min\"\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n LEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\n GROUP BY\n orders.status\n)\nSELECT\n _base.\"orders.status\",\n _base.\"orders.rev\"\nFROM _base\nLEFT JOIN _cm_orders__cust_region_min\n ON _base.\"orders.status\" IS NOT DISTINCT FROM _cm_orders__cust_region_min.\"orders.status\"\nORDER BY\n _cm_orders__cust_region_min.\"orders.cust_region_min\" ASC", + "order/grouped_derived_crossing::postgres": "WITH _base AS (\n SELECT\n orders.status AS \"orders.status\",\n CAST(SUM(orders.amount) AS DOUBLE PRECISION) AS \"orders.rev\"\n FROM orders AS orders\n GROUP BY\n orders.status\n), _cm_orders__cust_region_min AS (\n SELECT\n orders.status AS \"orders.status\",\n MIN(customers__regions.name) AS \"orders.cust_region_min\"\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n LEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\n GROUP BY\n orders.status\n)\nSELECT\n _base.\"orders.status\",\n _base.\"orders.rev\"\nFROM _base\nLEFT JOIN _cm_orders__cust_region_min\n ON _base.\"orders.status\" IS NOT DISTINCT FROM _cm_orders__cust_region_min.\"orders.status\"\nORDER BY\n _cm_orders__cust_region_min.\"orders.cust_region_min\" ASC", + "order/grouped_derived_crossing::sqlite": "WITH _base AS (\n SELECT\n orders.status AS \"orders.status\",\n CAST(SUM(orders.amount) AS REAL) AS \"orders.rev\"\n FROM orders AS orders\n GROUP BY\n orders.status\n), _cm_orders__cust_region_min AS (\n SELECT\n orders.status AS \"orders.status\",\n MIN(customers__regions.name) AS \"orders.cust_region_min\"\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n LEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\n GROUP BY\n orders.status\n)\nSELECT\n _base.\"orders.status\",\n _base.\"orders.rev\"\nFROM _base\nLEFT JOIN _cm_orders__cust_region_min\n ON _base.\"orders.status\" IS _cm_orders__cust_region_min.\"orders.status\"\nORDER BY\n _cm_orders__cust_region_min.\"orders.cust_region_min\" ASC NULLS LAST", + "order/grouped_derived_crossing::tsql": "WITH _base AS (\n SELECT\n orders.status AS [orders___status],\n CAST(SUM(orders.amount) AS FLOAT) AS [orders___rev]\n FROM orders AS orders\n GROUP BY\n orders.status\n), _cm_orders__cust_region_min AS (\n SELECT\n orders.status AS [orders___status],\n MIN(customers__regions.name) AS [orders___cust_region_min]\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n LEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\n GROUP BY\n orders.status\n)\nSELECT\n _base.[orders___status],\n _base.[orders___rev]\nFROM _base\nLEFT JOIN _cm_orders__cust_region_min\n ON (\n _base.[orders___status] = _cm_orders__cust_region_min.[orders___status]\n OR (\n _base.[orders___status] IS NULL\n AND _cm_orders__cust_region_min.[orders___status] IS NULL\n )\n )\nORDER BY\n _cm_orders__cust_region_min.[orders___cust_region_min] ASC", + "order/grouped_joined_row_asc::bigquery": "WITH _base AS (\n SELECT\n orders.status AS `orders___status`,\n CAST(SUM(orders.amount) AS FLOAT64) AS `orders___rev`\n FROM orders AS orders\n GROUP BY\n orders.status\n), _cm_orders__customers__regions__name_min_host AS (\n SELECT\n orders.status AS `orders___status`,\n MIN(customers__regions.name) AS `orders___customers___regions___name_min_host`\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n LEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\n GROUP BY\n orders.status\n)\nSELECT\n _base.`orders___status`,\n _base.`orders___rev`\nFROM _base\nLEFT JOIN _cm_orders__customers__regions__name_min_host\n ON _base.`orders___status` IS NOT DISTINCT FROM _cm_orders__customers__regions__name_min_host.`orders___status`\nORDER BY\n _cm_orders__customers__regions__name_min_host.`orders___customers___regions___name_min_host` ASC NULLS LAST", + "order/grouped_joined_row_asc::duckdb": "WITH _base AS (\n SELECT\n orders.status AS \"orders.status\",\n CAST(SUM(orders.amount) AS DOUBLE) AS \"orders.rev\"\n FROM orders AS orders\n GROUP BY\n orders.status\n), _cm_orders__customers__regions__name_min_host AS (\n SELECT\n orders.status AS \"orders.status\",\n MIN(customers__regions.name) AS \"orders.customers.regions.name_min_host\"\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n LEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\n GROUP BY\n orders.status\n)\nSELECT\n _base.\"orders.status\",\n _base.\"orders.rev\"\nFROM _base\nLEFT JOIN _cm_orders__customers__regions__name_min_host\n ON _base.\"orders.status\" IS NOT DISTINCT FROM _cm_orders__customers__regions__name_min_host.\"orders.status\"\nORDER BY\n _cm_orders__customers__regions__name_min_host.\"orders.customers.regions.name_min_host\" ASC", + "order/grouped_joined_row_asc::postgres": "WITH _base AS (\n SELECT\n orders.status AS \"orders.status\",\n CAST(SUM(orders.amount) AS DOUBLE PRECISION) AS \"orders.rev\"\n FROM orders AS orders\n GROUP BY\n orders.status\n), _cm_orders__customers__regions__name_min_host AS (\n SELECT\n orders.status AS \"orders.status\",\n MIN(customers__regions.name) AS \"orders.customers.regions.name_min_host\"\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n LEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\n GROUP BY\n orders.status\n)\nSELECT\n _base.\"orders.status\",\n _base.\"orders.rev\"\nFROM _base\nLEFT JOIN _cm_orders__customers__regions__name_min_host\n ON _base.\"orders.status\" IS NOT DISTINCT FROM _cm_orders__customers__regions__name_min_host.\"orders.status\"\nORDER BY\n _cm_orders__customers__regions__name_min_host.\"orders.customers.regions.name_min_host\" ASC", + "order/grouped_joined_row_asc::sqlite": "WITH _base AS (\n SELECT\n orders.status AS \"orders.status\",\n CAST(SUM(orders.amount) AS REAL) AS \"orders.rev\"\n FROM orders AS orders\n GROUP BY\n orders.status\n), _cm_orders__customers__regions__name_min_host AS (\n SELECT\n orders.status AS \"orders.status\",\n MIN(customers__regions.name) AS \"orders.customers.regions.name_min_host\"\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n LEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\n GROUP BY\n orders.status\n)\nSELECT\n _base.\"orders.status\",\n _base.\"orders.rev\"\nFROM _base\nLEFT JOIN _cm_orders__customers__regions__name_min_host\n ON _base.\"orders.status\" IS _cm_orders__customers__regions__name_min_host.\"orders.status\"\nORDER BY\n _cm_orders__customers__regions__name_min_host.\"orders.customers.regions.name_min_host\" ASC NULLS LAST", + "order/grouped_joined_row_asc::tsql": "WITH _base AS (\n SELECT\n orders.status AS [orders___status],\n CAST(SUM(orders.amount) AS FLOAT) AS [orders___rev]\n FROM orders AS orders\n GROUP BY\n orders.status\n), _cm_orders__customers__regions__name_min_host AS (\n SELECT\n orders.status AS [orders___status],\n MIN(customers__regions.name) AS [orders___customers___regions___name_min_host]\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n LEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\n GROUP BY\n orders.status\n)\nSELECT\n _base.[orders___status],\n _base.[orders___rev]\nFROM _base\nLEFT JOIN _cm_orders__customers__regions__name_min_host\n ON (\n _base.[orders___status] = _cm_orders__customers__regions__name_min_host.[orders___status]\n OR (\n _base.[orders___status] IS NULL\n AND _cm_orders__customers__regions__name_min_host.[orders___status] IS NULL\n )\n )\nORDER BY\n _cm_orders__customers__regions__name_min_host.[orders___customers___regions___name_min_host] ASC", + "order/grouped_joined_row_desc::bigquery": "WITH _base AS (\n SELECT\n orders.status AS `orders___status`,\n CAST(SUM(orders.amount) AS FLOAT64) AS `orders___rev`\n FROM orders AS orders\n GROUP BY\n orders.status\n), _cm_orders__customers__regions__name_max_host AS (\n SELECT\n orders.status AS `orders___status`,\n MAX(customers__regions.name) AS `orders___customers___regions___name_max_host`\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n LEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\n GROUP BY\n orders.status\n)\nSELECT\n _base.`orders___status`,\n _base.`orders___rev`\nFROM _base\nLEFT JOIN _cm_orders__customers__regions__name_max_host\n ON _base.`orders___status` IS NOT DISTINCT FROM _cm_orders__customers__regions__name_max_host.`orders___status`\nORDER BY\n _cm_orders__customers__regions__name_max_host.`orders___customers___regions___name_max_host` DESC", + "order/grouped_joined_row_desc::duckdb": "WITH _base AS (\n SELECT\n orders.status AS \"orders.status\",\n CAST(SUM(orders.amount) AS DOUBLE) AS \"orders.rev\"\n FROM orders AS orders\n GROUP BY\n orders.status\n), _cm_orders__customers__regions__name_max_host AS (\n SELECT\n orders.status AS \"orders.status\",\n MAX(customers__regions.name) AS \"orders.customers.regions.name_max_host\"\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n LEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\n GROUP BY\n orders.status\n)\nSELECT\n _base.\"orders.status\",\n _base.\"orders.rev\"\nFROM _base\nLEFT JOIN _cm_orders__customers__regions__name_max_host\n ON _base.\"orders.status\" IS NOT DISTINCT FROM _cm_orders__customers__regions__name_max_host.\"orders.status\"\nORDER BY\n _cm_orders__customers__regions__name_max_host.\"orders.customers.regions.name_max_host\" DESC", + "order/grouped_joined_row_desc::postgres": "WITH _base AS (\n SELECT\n orders.status AS \"orders.status\",\n CAST(SUM(orders.amount) AS DOUBLE PRECISION) AS \"orders.rev\"\n FROM orders AS orders\n GROUP BY\n orders.status\n), _cm_orders__customers__regions__name_max_host AS (\n SELECT\n orders.status AS \"orders.status\",\n MAX(customers__regions.name) AS \"orders.customers.regions.name_max_host\"\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n LEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\n GROUP BY\n orders.status\n)\nSELECT\n _base.\"orders.status\",\n _base.\"orders.rev\"\nFROM _base\nLEFT JOIN _cm_orders__customers__regions__name_max_host\n ON _base.\"orders.status\" IS NOT DISTINCT FROM _cm_orders__customers__regions__name_max_host.\"orders.status\"\nORDER BY\n _cm_orders__customers__regions__name_max_host.\"orders.customers.regions.name_max_host\" DESC NULLS LAST", + "order/grouped_joined_row_desc::sqlite": "WITH _base AS (\n SELECT\n orders.status AS \"orders.status\",\n CAST(SUM(orders.amount) AS REAL) AS \"orders.rev\"\n FROM orders AS orders\n GROUP BY\n orders.status\n), _cm_orders__customers__regions__name_max_host AS (\n SELECT\n orders.status AS \"orders.status\",\n MAX(customers__regions.name) AS \"orders.customers.regions.name_max_host\"\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n LEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\n GROUP BY\n orders.status\n)\nSELECT\n _base.\"orders.status\",\n _base.\"orders.rev\"\nFROM _base\nLEFT JOIN _cm_orders__customers__regions__name_max_host\n ON _base.\"orders.status\" IS _cm_orders__customers__regions__name_max_host.\"orders.status\"\nORDER BY\n _cm_orders__customers__regions__name_max_host.\"orders.customers.regions.name_max_host\" DESC", + "order/grouped_joined_row_desc::tsql": "WITH _base AS (\n SELECT\n orders.status AS [orders___status],\n CAST(SUM(orders.amount) AS FLOAT) AS [orders___rev]\n FROM orders AS orders\n GROUP BY\n orders.status\n), _cm_orders__customers__regions__name_max_host AS (\n SELECT\n orders.status AS [orders___status],\n MAX(customers__regions.name) AS [orders___customers___regions___name_max_host]\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n LEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\n GROUP BY\n orders.status\n)\nSELECT\n _base.[orders___status],\n _base.[orders___rev]\nFROM _base\nLEFT JOIN _cm_orders__customers__regions__name_max_host\n ON (\n _base.[orders___status] = _cm_orders__customers__regions__name_max_host.[orders___status]\n OR (\n _base.[orders___status] IS NULL\n AND _cm_orders__customers__regions__name_max_host.[orders___status] IS NULL\n )\n )\nORDER BY\n _cm_orders__customers__regions__name_max_host.[orders___customers___regions___name_max_host] DESC", + "order/grouped_local_row_asc::bigquery": "SELECT\n `orders___status`,\n `orders___rev`\nFROM (\n SELECT\n orders.status AS `orders___status`,\n CAST(SUM(orders.amount) AS FLOAT64) AS `orders___rev`,\n MIN(orders.created_at) AS `orders___created_at_min`\n FROM orders AS orders\n GROUP BY\n orders.status\n) AS _outer\nORDER BY\n `orders___created_at_min` ASC NULLS LAST", + "order/grouped_local_row_asc::duckdb": "SELECT\n \"orders.status\",\n \"orders.rev\"\nFROM (\n SELECT\n orders.status AS \"orders.status\",\n CAST(SUM(orders.amount) AS DOUBLE) AS \"orders.rev\",\n MIN(orders.created_at) AS \"orders.created_at_min\"\n FROM orders AS orders\n GROUP BY\n orders.status\n) AS _outer\nORDER BY\n \"orders.created_at_min\" ASC", + "order/grouped_local_row_asc::postgres": "SELECT\n \"orders.status\",\n \"orders.rev\"\nFROM (\n SELECT\n orders.status AS \"orders.status\",\n CAST(SUM(orders.amount) AS DOUBLE PRECISION) AS \"orders.rev\",\n MIN(orders.created_at) AS \"orders.created_at_min\"\n FROM orders AS orders\n GROUP BY\n orders.status\n) AS _outer\nORDER BY\n \"orders.created_at_min\" ASC", + "order/grouped_local_row_asc::sqlite": "SELECT\n \"orders.status\",\n \"orders.rev\"\nFROM (\n SELECT\n orders.status AS \"orders.status\",\n CAST(SUM(orders.amount) AS REAL) AS \"orders.rev\",\n MIN(orders.created_at) AS \"orders.created_at_min\"\n FROM orders AS orders\n GROUP BY\n orders.status\n) AS _outer\nORDER BY\n \"orders.created_at_min\" ASC NULLS LAST", + "order/grouped_local_row_asc::tsql": "SELECT\n [orders___status],\n [orders___rev]\nFROM (\n SELECT\n orders.status AS [orders___status],\n CAST(SUM(orders.amount) AS FLOAT) AS [orders___rev],\n MIN(orders.created_at) AS [orders___created_at_min]\n FROM orders AS orders\n GROUP BY\n orders.status\n) AS _outer\nORDER BY\n [orders___created_at_min] ASC", + "order/grouped_local_row_desc::bigquery": "SELECT\n `orders___status`,\n `orders___rev`\nFROM (\n SELECT\n orders.status AS `orders___status`,\n CAST(SUM(orders.amount) AS FLOAT64) AS `orders___rev`,\n MAX(orders.created_at) AS `orders___created_at_max`\n FROM orders AS orders\n GROUP BY\n orders.status\n) AS _outer\nORDER BY\n `orders___created_at_max` DESC", + "order/grouped_local_row_desc::duckdb": "SELECT\n \"orders.status\",\n \"orders.rev\"\nFROM (\n SELECT\n orders.status AS \"orders.status\",\n CAST(SUM(orders.amount) AS DOUBLE) AS \"orders.rev\",\n MAX(orders.created_at) AS \"orders.created_at_max\"\n FROM orders AS orders\n GROUP BY\n orders.status\n) AS _outer\nORDER BY\n \"orders.created_at_max\" DESC", + "order/grouped_local_row_desc::postgres": "SELECT\n \"orders.status\",\n \"orders.rev\"\nFROM (\n SELECT\n orders.status AS \"orders.status\",\n CAST(SUM(orders.amount) AS DOUBLE PRECISION) AS \"orders.rev\",\n MAX(orders.created_at) AS \"orders.created_at_max\"\n FROM orders AS orders\n GROUP BY\n orders.status\n) AS _outer\nORDER BY\n \"orders.created_at_max\" DESC NULLS LAST", + "order/grouped_local_row_desc::sqlite": "SELECT\n \"orders.status\",\n \"orders.rev\"\nFROM (\n SELECT\n orders.status AS \"orders.status\",\n CAST(SUM(orders.amount) AS REAL) AS \"orders.rev\",\n MAX(orders.created_at) AS \"orders.created_at_max\"\n FROM orders AS orders\n GROUP BY\n orders.status\n) AS _outer\nORDER BY\n \"orders.created_at_max\" DESC", + "order/grouped_local_row_desc::tsql": "SELECT\n [orders___status],\n [orders___rev]\nFROM (\n SELECT\n orders.status AS [orders___status],\n CAST(SUM(orders.amount) AS FLOAT) AS [orders___rev],\n MAX(orders.created_at) AS [orders___created_at_max]\n FROM orders AS orders\n GROUP BY\n orders.status\n) AS _outer\nORDER BY\n [orders___created_at_max] DESC", + "order/hidden_slot_outer_trim::bigquery": "SELECT\n `orders___status`,\n `orders___rev`\nFROM (\n SELECT\n orders.status AS `orders___status`,\n CAST(SUM(orders.amount) AS FLOAT64) AS `orders___rev`,\n MAX(orders.amount) AS `orders___amount_max`\n FROM orders AS orders\n GROUP BY\n orders.status\n) AS _outer\nORDER BY\n `orders___amount_max` DESC", + "order/hidden_slot_outer_trim::duckdb": "SELECT\n \"orders.status\",\n \"orders.rev\"\nFROM (\n SELECT\n orders.status AS \"orders.status\",\n CAST(SUM(orders.amount) AS DOUBLE) AS \"orders.rev\",\n MAX(orders.amount) AS \"orders.amount_max\"\n FROM orders AS orders\n GROUP BY\n orders.status\n) AS _outer\nORDER BY\n \"orders.amount_max\" DESC", + "order/hidden_slot_outer_trim::postgres": "SELECT\n \"orders.status\",\n \"orders.rev\"\nFROM (\n SELECT\n orders.status AS \"orders.status\",\n CAST(SUM(orders.amount) AS DOUBLE PRECISION) AS \"orders.rev\",\n MAX(orders.amount) AS \"orders.amount_max\"\n FROM orders AS orders\n GROUP BY\n orders.status\n) AS _outer\nORDER BY\n \"orders.amount_max\" DESC NULLS LAST", + "order/hidden_slot_outer_trim::sqlite": "SELECT\n \"orders.status\",\n \"orders.rev\"\nFROM (\n SELECT\n orders.status AS \"orders.status\",\n CAST(SUM(orders.amount) AS REAL) AS \"orders.rev\",\n MAX(orders.amount) AS \"orders.amount_max\"\n FROM orders AS orders\n GROUP BY\n orders.status\n) AS _outer\nORDER BY\n \"orders.amount_max\" DESC", + "order/hidden_slot_outer_trim::tsql": "SELECT\n [orders___status],\n [orders___rev]\nFROM (\n SELECT\n orders.status AS [orders___status],\n CAST(SUM(orders.amount) AS FLOAT) AS [orders___rev],\n MAX(orders.amount) AS [orders___amount_max]\n FROM orders AS orders\n GROUP BY\n orders.status\n) AS _outer\nORDER BY\n [orders___amount_max] DESC", + "order/host_base_alias::bigquery": "SELECT\n orders.status AS `orders___status`,\n CAST(SUM(orders.amount) AS FLOAT64) AS `orders___rev`\nFROM orders AS orders\nGROUP BY\n `orders___status`\nORDER BY\n `orders___rev` DESC", + "order/host_base_alias::duckdb": "SELECT\n orders.status AS \"orders.status\",\n CAST(SUM(orders.amount) AS DOUBLE) AS \"orders.rev\"\nFROM orders AS orders\nGROUP BY\n orders.status\nORDER BY\n \"orders.rev\" DESC", + "order/host_base_alias::postgres": "SELECT\n orders.status AS \"orders.status\",\n CAST(SUM(orders.amount) AS DOUBLE PRECISION) AS \"orders.rev\"\nFROM orders AS orders\nGROUP BY\n orders.status\nORDER BY\n \"orders.rev\" DESC NULLS LAST", + "order/host_base_alias::sqlite": "SELECT\n orders.status AS \"orders.status\",\n CAST(SUM(orders.amount) AS REAL) AS \"orders.rev\"\nFROM orders AS orders\nGROUP BY\n orders.status\nORDER BY\n \"orders.rev\" DESC", + "order/host_base_alias::tsql": "SELECT\n orders.status AS [orders___status],\n CAST(SUM(orders.amount) AS FLOAT) AS [orders___rev]\nFROM orders AS orders\nGROUP BY\n orders.status\nORDER BY\n [orders___rev] DESC", + "order/outer_composite::bigquery": "WITH _base AS (\n SELECT\n orders.status AS `orders___status`,\n SUM(orders.amount) AS `orders___amount_sum`\n FROM orders AS orders\n GROUP BY\n orders.status\n), _cm_orders__customers__spend_sum AS (\n SELECT\n SUM(customers.spend) AS `orders___customers___spend_sum`\n FROM customers AS customers\n)\nSELECT\n _base.`orders___status`,\n _cm_orders__customers__spend_sum.`orders___customers___spend_sum` + _base.`orders___amount_sum` AS `orders___mix`\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_sum\nORDER BY\n `orders___mix` DESC", + "order/outer_composite::duckdb": "WITH _base AS (\n SELECT\n orders.status AS \"orders.status\",\n SUM(orders.amount) AS \"orders.amount_sum\"\n FROM orders AS orders\n GROUP BY\n orders.status\n), _cm_orders__customers__spend_sum AS (\n SELECT\n SUM(customers.spend) AS \"orders.customers.spend_sum\"\n FROM customers AS customers\n)\nSELECT\n _base.\"orders.status\",\n _cm_orders__customers__spend_sum.\"orders.customers.spend_sum\" + _base.\"orders.amount_sum\" AS \"orders.mix\"\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_sum\nORDER BY\n \"orders.mix\" DESC", + "order/outer_composite::postgres": "WITH _base AS (\n SELECT\n orders.status AS \"orders.status\",\n SUM(orders.amount) AS \"orders.amount_sum\"\n FROM orders AS orders\n GROUP BY\n orders.status\n), _cm_orders__customers__spend_sum AS (\n SELECT\n SUM(customers.spend) AS \"orders.customers.spend_sum\"\n FROM customers AS customers\n)\nSELECT\n _base.\"orders.status\",\n _cm_orders__customers__spend_sum.\"orders.customers.spend_sum\" + _base.\"orders.amount_sum\" AS \"orders.mix\"\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_sum\nORDER BY\n \"orders.mix\" DESC NULLS LAST", + "order/outer_composite::sqlite": "WITH _base AS (\n SELECT\n orders.status AS \"orders.status\",\n SUM(orders.amount) AS \"orders.amount_sum\"\n FROM orders AS orders\n GROUP BY\n orders.status\n), _cm_orders__customers__spend_sum AS (\n SELECT\n SUM(customers.spend) AS \"orders.customers.spend_sum\"\n FROM customers AS customers\n)\nSELECT\n _base.\"orders.status\",\n _cm_orders__customers__spend_sum.\"orders.customers.spend_sum\" + _base.\"orders.amount_sum\" AS \"orders.mix\"\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_sum\nORDER BY\n \"orders.mix\" DESC", + "order/outer_composite::tsql": "WITH _base AS (\n SELECT\n orders.status AS [orders___status],\n SUM(orders.amount) AS [orders___amount_sum]\n FROM orders AS orders\n GROUP BY\n orders.status\n), _cm_orders__customers__spend_sum AS (\n SELECT\n SUM(customers.spend) AS [orders___customers___spend_sum]\n FROM customers AS customers\n)\nSELECT\n _base.[orders___status],\n _cm_orders__customers__spend_sum.[orders___customers___spend_sum] + _base.[orders___amount_sum] AS [orders___mix]\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_sum\nORDER BY\n [orders___mix] DESC", + "order/transform_chain_wrap::bigquery": "SELECT\n `orders___created_at`,\n `orders___cs`\nFROM (\nWITH base AS (\n SELECT\n DATE_TRUNC(orders.created_at, MONTH) AS `orders___created_at`,\n SUM(orders.amount) AS `orders___amount_sum`\n FROM orders AS orders\n GROUP BY\n DATE_TRUNC(orders.created_at, MONTH)\n), step1 AS (\n SELECT\n `orders___created_at`,\n `orders___amount_sum`,\n SUM(`orders___amount_sum`) OVER (ORDER BY `orders___created_at`) AS `orders___cs`\n FROM base\n)\nSELECT\n `orders___created_at`,\n `orders___amount_sum`,\n `orders___cs`\nFROM step1\n) AS _outer\nORDER BY\n `orders___cs` ASC NULLS LAST", + "order/transform_chain_wrap::duckdb": "SELECT\n \"orders.created_at\",\n \"orders.cs\"\nFROM (\nWITH base AS (\n SELECT\n DATE_TRUNC('MONTH', orders.created_at) AS \"orders.created_at\",\n SUM(orders.amount) AS \"orders.amount_sum\"\n FROM orders AS orders\n GROUP BY\n DATE_TRUNC('MONTH', orders.created_at)\n), step1 AS (\n SELECT\n \"orders.created_at\",\n \"orders.amount_sum\",\n SUM(\"orders.amount_sum\") OVER (ORDER BY \"orders.created_at\") AS \"orders.cs\"\n FROM base\n)\nSELECT\n \"orders.created_at\",\n \"orders.amount_sum\",\n \"orders.cs\"\nFROM step1\n) AS _outer\nORDER BY\n \"orders.cs\" ASC", + "order/transform_chain_wrap::postgres": "SELECT\n \"orders.created_at\",\n \"orders.cs\"\nFROM (\nWITH base AS (\n SELECT\n DATE_TRUNC('MONTH', orders.created_at) AS \"orders.created_at\",\n SUM(orders.amount) AS \"orders.amount_sum\"\n FROM orders AS orders\n GROUP BY\n DATE_TRUNC('MONTH', orders.created_at)\n), step1 AS (\n SELECT\n \"orders.created_at\",\n \"orders.amount_sum\",\n SUM(\"orders.amount_sum\") OVER (ORDER BY \"orders.created_at\") AS \"orders.cs\"\n FROM base\n)\nSELECT\n \"orders.created_at\",\n \"orders.amount_sum\",\n \"orders.cs\"\nFROM step1\n) AS _outer\nORDER BY\n \"orders.cs\" ASC", + "order/transform_chain_wrap::sqlite": "SELECT\n \"orders.created_at\",\n \"orders.cs\"\nFROM (\nWITH base AS (\n SELECT\n STRFTIME('%Y-%m-01', orders.created_at) AS \"orders.created_at\",\n SUM(orders.amount) AS \"orders.amount_sum\"\n FROM orders AS orders\n GROUP BY\n STRFTIME('%Y-%m-01', orders.created_at)\n), step1 AS (\n SELECT\n \"orders.created_at\",\n \"orders.amount_sum\",\n SUM(\"orders.amount_sum\") OVER (ORDER BY \"orders.created_at\") AS \"orders.cs\"\n FROM base\n)\nSELECT\n \"orders.created_at\",\n \"orders.amount_sum\",\n \"orders.cs\"\nFROM step1\n) AS _outer\nORDER BY\n \"orders.cs\" ASC NULLS LAST", + "order/transform_chain_wrap::tsql": "WITH base AS (\n SELECT\n DATETRUNC(MONTH, orders.created_at) AS [orders___created_at],\n SUM(orders.amount) AS [orders___amount_sum]\n FROM orders AS orders\n GROUP BY\n DATETRUNC(MONTH, orders.created_at)\n), step1 AS (\n SELECT\n [orders___created_at] AS [orders___created_at],\n [orders___amount_sum] AS [orders___amount_sum],\n SUM([orders___amount_sum]) OVER (ORDER BY [orders___created_at]) AS [orders___cs]\n FROM base\n)\nSELECT\n [orders___created_at],\n [orders___cs]\nFROM (\n SELECT\n [orders___created_at] AS [orders___created_at],\n [orders___amount_sum] AS [orders___amount_sum],\n [orders___cs] AS [orders___cs]\n FROM step1\n) AS _outer\nORDER BY\n [orders___cs] ASC", + "order/ungrouped_derived_crossing::bigquery": "SELECT\n orders.status AS `orders___status`\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nLEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\nORDER BY\n customers__regions.name ASC NULLS LAST", + "order/ungrouped_derived_crossing::duckdb": "SELECT\n orders.status AS \"orders.status\"\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nLEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\nORDER BY\n customers__regions.name ASC", + "order/ungrouped_derived_crossing::postgres": "SELECT\n orders.status AS \"orders.status\"\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nLEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\nORDER BY\n customers__regions.name ASC", + "order/ungrouped_derived_crossing::sqlite": "SELECT\n orders.status AS \"orders.status\"\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nLEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\nORDER BY\n customers__regions.name ASC NULLS LAST", + "order/ungrouped_derived_crossing::tsql": "SELECT\n orders.status AS [orders___status]\nFROM orders AS orders\nLEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\nLEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\nORDER BY\n customers__regions.name ASC", + "order/windowed_cte::bigquery": "WITH _base AS (\n SELECT\n DATE_TRUNC(orders.created_at, MONTH) AS `orders___created_at`\n FROM orders AS orders\n GROUP BY\n DATE_TRUNC(orders.created_at, MONTH)\n), _wm_orders__w AS (\n SELECT\n _base.`orders___created_at`,\n CAST(SUM(_src._w_value) AS FLOAT64) AS `orders___w`\n FROM _base\n LEFT JOIN (\n SELECT\n orders.created_at AS _w_time,\n orders.amount AS _w_value\n FROM orders AS orders\n ) AS _src\n ON _src._w_time >= _base.`orders___created_at` + INTERVAL 1 MONTH - INTERVAL 90 DAY\n AND _src._w_time < _base.`orders___created_at` + INTERVAL 1 MONTH\n GROUP BY\n _base.`orders___created_at`\n)\nSELECT\n _base.`orders___created_at`,\n _wm_orders__w.`orders___w`\nFROM _base\nLEFT JOIN _wm_orders__w\n ON _base.`orders___created_at` IS NOT DISTINCT FROM _wm_orders__w.`orders___created_at`\nORDER BY\n _wm_orders__w.`orders___w` DESC", + "order/windowed_cte::duckdb": "WITH _base AS (\n SELECT\n DATE_TRUNC('MONTH', orders.created_at) AS \"orders.created_at\"\n FROM orders AS orders\n GROUP BY\n DATE_TRUNC('MONTH', orders.created_at)\n), _wm_orders__w AS (\n SELECT\n _base.\"orders.created_at\",\n CAST(SUM(_src._w_value) AS DOUBLE) AS \"orders.w\"\n FROM _base\n LEFT JOIN (\n SELECT\n orders.created_at AS _w_time,\n orders.amount AS _w_value\n FROM orders AS orders\n ) AS _src\n ON _src._w_time >= _base.\"orders.created_at\" + INTERVAL 1 MONTH - INTERVAL 90 DAY\n AND _src._w_time < _base.\"orders.created_at\" + INTERVAL 1 MONTH\n GROUP BY\n _base.\"orders.created_at\"\n)\nSELECT\n _base.\"orders.created_at\",\n _wm_orders__w.\"orders.w\"\nFROM _base\nLEFT JOIN _wm_orders__w\n ON _base.\"orders.created_at\" IS NOT DISTINCT FROM _wm_orders__w.\"orders.created_at\"\nORDER BY\n _wm_orders__w.\"orders.w\" DESC", + "order/windowed_cte::postgres": "WITH _base AS (\n SELECT\n DATE_TRUNC('MONTH', orders.created_at) AS \"orders.created_at\"\n FROM orders AS orders\n GROUP BY\n DATE_TRUNC('MONTH', orders.created_at)\n), _wm_orders__w AS (\n SELECT\n _base.\"orders.created_at\",\n CAST(SUM(_src._w_value) AS DOUBLE PRECISION) AS \"orders.w\"\n FROM _base\n LEFT JOIN (\n SELECT\n orders.created_at AS _w_time,\n orders.amount AS _w_value\n FROM orders AS orders\n ) AS _src\n ON _src._w_time >= _base.\"orders.created_at\" + INTERVAL '1 MONTH' - INTERVAL '90 DAY'\n AND _src._w_time < _base.\"orders.created_at\" + INTERVAL '1 MONTH'\n GROUP BY\n _base.\"orders.created_at\"\n)\nSELECT\n _base.\"orders.created_at\",\n _wm_orders__w.\"orders.w\"\nFROM _base\nLEFT JOIN _wm_orders__w\n ON _base.\"orders.created_at\" IS NOT DISTINCT FROM _wm_orders__w.\"orders.created_at\"\nORDER BY\n _wm_orders__w.\"orders.w\" DESC NULLS LAST", + "order/windowed_cte::sqlite": "WITH _base AS (\n SELECT\n STRFTIME('%Y-%m-01', orders.created_at) AS \"orders.created_at\"\n FROM orders AS orders\n GROUP BY\n STRFTIME('%Y-%m-01', orders.created_at)\n), _wm_orders__w AS (\n SELECT\n _base.\"orders.created_at\",\n CAST(SUM(_src._w_value) AS REAL) AS \"orders.w\"\n FROM _base\n LEFT JOIN (\n SELECT\n orders.created_at AS _w_time,\n orders.amount AS _w_value\n FROM orders AS orders\n ) AS _src\n ON _src._w_time >= DATETIME(DATETIME(_base.\"orders.created_at\", '+1 months'), '-90 days')\n AND _src._w_time < DATETIME(_base.\"orders.created_at\", '+1 months')\n GROUP BY\n _base.\"orders.created_at\"\n)\nSELECT\n _base.\"orders.created_at\",\n _wm_orders__w.\"orders.w\"\nFROM _base\nLEFT JOIN _wm_orders__w\n ON _base.\"orders.created_at\" IS _wm_orders__w.\"orders.created_at\"\nORDER BY\n _wm_orders__w.\"orders.w\" DESC", + "order/windowed_cte::tsql": "WITH _base AS (\n SELECT\n DATETRUNC(month, orders.created_at) AS [orders___created_at]\n FROM orders AS orders\n GROUP BY\n DATETRUNC(month, orders.created_at)\n), _wm_orders__w AS (\n SELECT\n _base.[orders___created_at] AS [orders___created_at],\n CAST(SUM(_src._w_value) AS FLOAT) AS [orders___w]\n FROM _base\n LEFT JOIN (\n SELECT\n orders.created_at AS _w_time,\n orders.amount AS _w_value\n FROM orders AS orders\n ) AS _src\n ON _src._w_time >= DATEADD(DAY, -90, DATEADD(MONTH, 1, _base.[orders___created_at]))\n AND _src._w_time < DATEADD(MONTH, 1, _base.[orders___created_at])\n GROUP BY\n _base.[orders___created_at]\n)\nSELECT\n _base.[orders___created_at],\n _wm_orders__w.[orders___w]\nFROM _base\nLEFT JOIN _wm_orders__w\n ON (\n _base.[orders___created_at] = _wm_orders__w.[orders___created_at]\n OR (\n _base.[orders___created_at] IS NULL AND _wm_orders__w.[orders___created_at] IS NULL\n )\n )\nORDER BY\n _wm_orders__w.[orders___w] DESC", + "reroot/host_local_filter::bigquery": "WITH _base AS (\n SELECT\n customers__regions.name AS `orders___customers___regions___name`,\n CAST(SUM(orders.amount) AS FLOAT64) AS `orders___rev`\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n LEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\n WHERE\n orders.status = 'A'\n GROUP BY\n customers__regions.name\n), _cm_orders__customers__spend_sum AS (\n SELECT\n regions.name AS `customers___regions___name`,\n CAST(SUM(customers.spend) AS FLOAT64) AS `customers___spend_sum`\n FROM customers AS customers\n LEFT JOIN regions AS regions\n ON customers.region_id = regions.id\n GROUP BY\n regions.name\n)\nSELECT\n _base.`orders___customers___regions___name`,\n _base.`orders___rev`,\n _cm_orders__customers__spend_sum.`customers___spend_sum` AS `orders___cs`\nFROM _base\nLEFT JOIN _cm_orders__customers__spend_sum\n ON _base.`orders___customers___regions___name` IS NOT DISTINCT FROM _cm_orders__customers__spend_sum.`customers___regions___name`", + "reroot/host_local_filter::duckdb": "WITH _base AS (\n SELECT\n customers__regions.name AS \"orders.customers.regions.name\",\n CAST(SUM(orders.amount) AS DOUBLE) AS \"orders.rev\"\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n LEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\n WHERE\n orders.status = 'A'\n GROUP BY\n customers__regions.name\n), _cm_orders__customers__spend_sum AS (\n SELECT\n regions.name AS \"customers.regions.name\",\n CAST(SUM(customers.spend) AS DOUBLE) AS \"customers.spend_sum\"\n FROM customers AS customers\n LEFT JOIN regions AS regions\n ON customers.region_id = regions.id\n GROUP BY\n regions.name\n)\nSELECT\n _base.\"orders.customers.regions.name\",\n _base.\"orders.rev\",\n _cm_orders__customers__spend_sum.\"customers.spend_sum\" AS \"orders.cs\"\nFROM _base\nLEFT JOIN _cm_orders__customers__spend_sum\n ON _base.\"orders.customers.regions.name\" IS NOT DISTINCT FROM _cm_orders__customers__spend_sum.\"customers.regions.name\"", + "reroot/host_local_filter::postgres": "WITH _base AS (\n SELECT\n customers__regions.name AS \"orders.customers.regions.name\",\n CAST(SUM(orders.amount) AS DOUBLE PRECISION) AS \"orders.rev\"\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n LEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\n WHERE\n orders.status = 'A'\n GROUP BY\n customers__regions.name\n), _cm_orders__customers__spend_sum AS (\n SELECT\n regions.name AS \"customers.regions.name\",\n CAST(SUM(customers.spend) AS DOUBLE PRECISION) AS \"customers.spend_sum\"\n FROM customers AS customers\n LEFT JOIN regions AS regions\n ON customers.region_id = regions.id\n GROUP BY\n regions.name\n)\nSELECT\n _base.\"orders.customers.regions.name\",\n _base.\"orders.rev\",\n _cm_orders__customers__spend_sum.\"customers.spend_sum\" AS \"orders.cs\"\nFROM _base\nLEFT JOIN _cm_orders__customers__spend_sum\n ON _base.\"orders.customers.regions.name\" IS NOT DISTINCT FROM _cm_orders__customers__spend_sum.\"customers.regions.name\"", + "reroot/host_local_filter::sqlite": "WITH _base AS (\n SELECT\n customers__regions.name AS \"orders.customers.regions.name\",\n CAST(SUM(orders.amount) AS REAL) AS \"orders.rev\"\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n LEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\n WHERE\n orders.status = 'A'\n GROUP BY\n customers__regions.name\n), _cm_orders__customers__spend_sum AS (\n SELECT\n regions.name AS \"customers.regions.name\",\n CAST(SUM(customers.spend) AS REAL) AS \"customers.spend_sum\"\n FROM customers AS customers\n LEFT JOIN regions AS regions\n ON customers.region_id = regions.id\n GROUP BY\n regions.name\n)\nSELECT\n _base.\"orders.customers.regions.name\",\n _base.\"orders.rev\",\n _cm_orders__customers__spend_sum.\"customers.spend_sum\" AS \"orders.cs\"\nFROM _base\nLEFT JOIN _cm_orders__customers__spend_sum\n ON _base.\"orders.customers.regions.name\" IS _cm_orders__customers__spend_sum.\"customers.regions.name\"", + "reroot/host_local_filter::tsql": "WITH _base AS (\n SELECT\n customers__regions.name AS [orders___customers___regions___name],\n CAST(SUM(orders.amount) AS FLOAT) AS [orders___rev]\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n LEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\n WHERE\n orders.status = 'A'\n GROUP BY\n customers__regions.name\n), _cm_orders__customers__spend_sum AS (\n SELECT\n regions.name AS [customers___regions___name],\n CAST(SUM(customers.spend) AS FLOAT) AS [customers___spend_sum]\n FROM customers AS customers\n LEFT JOIN regions AS regions\n ON customers.region_id = regions.id\n GROUP BY\n regions.name\n)\nSELECT\n _base.[orders___customers___regions___name],\n _base.[orders___rev],\n _cm_orders__customers__spend_sum.[customers___spend_sum] AS [orders___cs]\nFROM _base\nLEFT JOIN _cm_orders__customers__spend_sum\n ON (\n _base.[orders___customers___regions___name] = _cm_orders__customers__spend_sum.[customers___regions___name]\n OR (\n _base.[orders___customers___regions___name] IS NULL\n AND _cm_orders__customers__spend_sum.[customers___regions___name] IS NULL\n )\n )", + "reroot/reachable_filter::bigquery": "WITH _base AS (\n SELECT\n customers__regions.name AS `orders___customers___regions___name`,\n CAST(SUM(orders.amount) AS FLOAT64) AS `orders___rev`\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n LEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\n WHERE\n customers__regions.name = 'Alpha'\n GROUP BY\n customers__regions.name\n), _cm_orders__customers__spend_sum AS (\n SELECT\n regions.name AS `customers___regions___name`,\n CAST(SUM(customers.spend) AS FLOAT64) AS `customers___spend_sum`\n FROM customers AS customers\n LEFT JOIN regions AS regions\n ON customers.region_id = regions.id\n WHERE\n regions.name = 'Alpha'\n GROUP BY\n regions.name\n)\nSELECT\n _base.`orders___customers___regions___name`,\n _base.`orders___rev`,\n _cm_orders__customers__spend_sum.`customers___spend_sum` AS `orders___cs`\nFROM _base\nLEFT JOIN _cm_orders__customers__spend_sum\n ON _base.`orders___customers___regions___name` IS NOT DISTINCT FROM _cm_orders__customers__spend_sum.`customers___regions___name`", + "reroot/reachable_filter::duckdb": "WITH _base AS (\n SELECT\n customers__regions.name AS \"orders.customers.regions.name\",\n CAST(SUM(orders.amount) AS DOUBLE) AS \"orders.rev\"\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n LEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\n WHERE\n customers__regions.name = 'Alpha'\n GROUP BY\n customers__regions.name\n), _cm_orders__customers__spend_sum AS (\n SELECT\n regions.name AS \"customers.regions.name\",\n CAST(SUM(customers.spend) AS DOUBLE) AS \"customers.spend_sum\"\n FROM customers AS customers\n LEFT JOIN regions AS regions\n ON customers.region_id = regions.id\n WHERE\n regions.name = 'Alpha'\n GROUP BY\n regions.name\n)\nSELECT\n _base.\"orders.customers.regions.name\",\n _base.\"orders.rev\",\n _cm_orders__customers__spend_sum.\"customers.spend_sum\" AS \"orders.cs\"\nFROM _base\nLEFT JOIN _cm_orders__customers__spend_sum\n ON _base.\"orders.customers.regions.name\" IS NOT DISTINCT FROM _cm_orders__customers__spend_sum.\"customers.regions.name\"", + "reroot/reachable_filter::postgres": "WITH _base AS (\n SELECT\n customers__regions.name AS \"orders.customers.regions.name\",\n CAST(SUM(orders.amount) AS DOUBLE PRECISION) AS \"orders.rev\"\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n LEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\n WHERE\n customers__regions.name = 'Alpha'\n GROUP BY\n customers__regions.name\n), _cm_orders__customers__spend_sum AS (\n SELECT\n regions.name AS \"customers.regions.name\",\n CAST(SUM(customers.spend) AS DOUBLE PRECISION) AS \"customers.spend_sum\"\n FROM customers AS customers\n LEFT JOIN regions AS regions\n ON customers.region_id = regions.id\n WHERE\n regions.name = 'Alpha'\n GROUP BY\n regions.name\n)\nSELECT\n _base.\"orders.customers.regions.name\",\n _base.\"orders.rev\",\n _cm_orders__customers__spend_sum.\"customers.spend_sum\" AS \"orders.cs\"\nFROM _base\nLEFT JOIN _cm_orders__customers__spend_sum\n ON _base.\"orders.customers.regions.name\" IS NOT DISTINCT FROM _cm_orders__customers__spend_sum.\"customers.regions.name\"", + "reroot/reachable_filter::sqlite": "WITH _base AS (\n SELECT\n customers__regions.name AS \"orders.customers.regions.name\",\n CAST(SUM(orders.amount) AS REAL) AS \"orders.rev\"\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n LEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\n WHERE\n customers__regions.name = 'Alpha'\n GROUP BY\n customers__regions.name\n), _cm_orders__customers__spend_sum AS (\n SELECT\n regions.name AS \"customers.regions.name\",\n CAST(SUM(customers.spend) AS REAL) AS \"customers.spend_sum\"\n FROM customers AS customers\n LEFT JOIN regions AS regions\n ON customers.region_id = regions.id\n WHERE\n regions.name = 'Alpha'\n GROUP BY\n regions.name\n)\nSELECT\n _base.\"orders.customers.regions.name\",\n _base.\"orders.rev\",\n _cm_orders__customers__spend_sum.\"customers.spend_sum\" AS \"orders.cs\"\nFROM _base\nLEFT JOIN _cm_orders__customers__spend_sum\n ON _base.\"orders.customers.regions.name\" IS _cm_orders__customers__spend_sum.\"customers.regions.name\"", + "reroot/reachable_filter::tsql": "WITH _base AS (\n SELECT\n customers__regions.name AS [orders___customers___regions___name],\n CAST(SUM(orders.amount) AS FLOAT) AS [orders___rev]\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n LEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\n WHERE\n customers__regions.name = 'Alpha'\n GROUP BY\n customers__regions.name\n), _cm_orders__customers__spend_sum AS (\n SELECT\n regions.name AS [customers___regions___name],\n CAST(SUM(customers.spend) AS FLOAT) AS [customers___spend_sum]\n FROM customers AS customers\n LEFT JOIN regions AS regions\n ON customers.region_id = regions.id\n WHERE\n regions.name = 'Alpha'\n GROUP BY\n regions.name\n)\nSELECT\n _base.[orders___customers___regions___name],\n _base.[orders___rev],\n _cm_orders__customers__spend_sum.[customers___spend_sum] AS [orders___cs]\nFROM _base\nLEFT JOIN _cm_orders__customers__spend_sum\n ON (\n _base.[orders___customers___regions___name] = _cm_orders__customers__spend_sum.[customers___regions___name]\n OR (\n _base.[orders___customers___regions___name] IS NULL\n AND _cm_orders__customers__spend_sum.[customers___regions___name] IS NULL\n )\n )", + "reroot/unreachable_filter::bigquery": "WITH _base AS (\n SELECT\n customers__regions.name AS `orders___customers___regions___name`,\n CAST(SUM(orders.amount) AS FLOAT64) AS `orders___rev`\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n LEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\n LEFT JOIN order_tags AS order_tags\n ON orders.id = order_tags.order_id\n WHERE\n order_tags.name = 'rush'\n GROUP BY\n customers__regions.name\n), _cm_orders__customers__spend_sum AS (\n SELECT\n regions.name AS `customers___regions___name`,\n CAST(SUM(customers.spend) AS FLOAT64) AS `customers___spend_sum`\n FROM customers AS customers\n LEFT JOIN regions AS regions\n ON customers.region_id = regions.id\n GROUP BY\n regions.name\n)\nSELECT\n _base.`orders___customers___regions___name`,\n _base.`orders___rev`,\n _cm_orders__customers__spend_sum.`customers___spend_sum` AS `orders___cs`\nFROM _base\nLEFT JOIN _cm_orders__customers__spend_sum\n ON _base.`orders___customers___regions___name` IS NOT DISTINCT FROM _cm_orders__customers__spend_sum.`customers___regions___name`", + "reroot/unreachable_filter::duckdb": "WITH _base AS (\n SELECT\n customers__regions.name AS \"orders.customers.regions.name\",\n CAST(SUM(orders.amount) AS DOUBLE) AS \"orders.rev\"\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n LEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\n LEFT JOIN order_tags AS order_tags\n ON orders.id = order_tags.order_id\n WHERE\n order_tags.name = 'rush'\n GROUP BY\n customers__regions.name\n), _cm_orders__customers__spend_sum AS (\n SELECT\n regions.name AS \"customers.regions.name\",\n CAST(SUM(customers.spend) AS DOUBLE) AS \"customers.spend_sum\"\n FROM customers AS customers\n LEFT JOIN regions AS regions\n ON customers.region_id = regions.id\n GROUP BY\n regions.name\n)\nSELECT\n _base.\"orders.customers.regions.name\",\n _base.\"orders.rev\",\n _cm_orders__customers__spend_sum.\"customers.spend_sum\" AS \"orders.cs\"\nFROM _base\nLEFT JOIN _cm_orders__customers__spend_sum\n ON _base.\"orders.customers.regions.name\" IS NOT DISTINCT FROM _cm_orders__customers__spend_sum.\"customers.regions.name\"", + "reroot/unreachable_filter::postgres": "WITH _base AS (\n SELECT\n customers__regions.name AS \"orders.customers.regions.name\",\n CAST(SUM(orders.amount) AS DOUBLE PRECISION) AS \"orders.rev\"\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n LEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\n LEFT JOIN order_tags AS order_tags\n ON orders.id = order_tags.order_id\n WHERE\n order_tags.name = 'rush'\n GROUP BY\n customers__regions.name\n), _cm_orders__customers__spend_sum AS (\n SELECT\n regions.name AS \"customers.regions.name\",\n CAST(SUM(customers.spend) AS DOUBLE PRECISION) AS \"customers.spend_sum\"\n FROM customers AS customers\n LEFT JOIN regions AS regions\n ON customers.region_id = regions.id\n GROUP BY\n regions.name\n)\nSELECT\n _base.\"orders.customers.regions.name\",\n _base.\"orders.rev\",\n _cm_orders__customers__spend_sum.\"customers.spend_sum\" AS \"orders.cs\"\nFROM _base\nLEFT JOIN _cm_orders__customers__spend_sum\n ON _base.\"orders.customers.regions.name\" IS NOT DISTINCT FROM _cm_orders__customers__spend_sum.\"customers.regions.name\"", + "reroot/unreachable_filter::sqlite": "WITH _base AS (\n SELECT\n customers__regions.name AS \"orders.customers.regions.name\",\n CAST(SUM(orders.amount) AS REAL) AS \"orders.rev\"\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n LEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\n LEFT JOIN order_tags AS order_tags\n ON orders.id = order_tags.order_id\n WHERE\n order_tags.name = 'rush'\n GROUP BY\n customers__regions.name\n), _cm_orders__customers__spend_sum AS (\n SELECT\n regions.name AS \"customers.regions.name\",\n CAST(SUM(customers.spend) AS REAL) AS \"customers.spend_sum\"\n FROM customers AS customers\n LEFT JOIN regions AS regions\n ON customers.region_id = regions.id\n GROUP BY\n regions.name\n)\nSELECT\n _base.\"orders.customers.regions.name\",\n _base.\"orders.rev\",\n _cm_orders__customers__spend_sum.\"customers.spend_sum\" AS \"orders.cs\"\nFROM _base\nLEFT JOIN _cm_orders__customers__spend_sum\n ON _base.\"orders.customers.regions.name\" IS _cm_orders__customers__spend_sum.\"customers.regions.name\"", + "reroot/unreachable_filter::tsql": "WITH _base AS (\n SELECT\n customers__regions.name AS [orders___customers___regions___name],\n CAST(SUM(orders.amount) AS FLOAT) AS [orders___rev]\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n LEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\n LEFT JOIN order_tags AS order_tags\n ON orders.id = order_tags.order_id\n WHERE\n order_tags.name = 'rush'\n GROUP BY\n customers__regions.name\n), _cm_orders__customers__spend_sum AS (\n SELECT\n regions.name AS [customers___regions___name],\n CAST(SUM(customers.spend) AS FLOAT) AS [customers___spend_sum]\n FROM customers AS customers\n LEFT JOIN regions AS regions\n ON customers.region_id = regions.id\n GROUP BY\n regions.name\n)\nSELECT\n _base.[orders___customers___regions___name],\n _base.[orders___rev],\n _cm_orders__customers__spend_sum.[customers___spend_sum] AS [orders___cs]\nFROM _base\nLEFT JOIN _cm_orders__customers__spend_sum\n ON (\n _base.[orders___customers___regions___name] = _cm_orders__customers__spend_sum.[customers___regions___name]\n OR (\n _base.[orders___customers___regions___name] IS NULL\n AND _cm_orders__customers__spend_sum.[customers___regions___name] IS NULL\n )\n )" +} diff --git a/tests/test_dev1747_golden_sql.py b/tests/test_dev1747_golden_sql.py new file mode 100644 index 00000000..b6763eb0 --- /dev/null +++ b/tests/test_dev1747_golden_sql.py @@ -0,0 +1,327 @@ +"""DEV-1747 — golden SQL baseline for the ordering / re-rooting / chain surfaces. + +Same harness and same protocol as ``tests/test_dev1745_golden_sql.py`` (read its +module docstring for the four-step blessing loop); only the matrix differs. This +one targets exactly what PR 4 rewires and what PRs 5-6 will move next: + +* every ORDER BY render path the single resolver replaced — host base, the + hidden-slot outer trim wrap, the combined cross-model SELECT, the windowed + CTE, and the transform chain's outer wrap; +* the host-grain (``grain="host"``) wrap in both directions, which is where a + regression would silently sort every group by one global value; +* the re-rooted cross-model CTE with reachable / host-local / unreachable + filters, since re-rooting is what PR 5 builds on; +* both transform chains, whose WITH clause is now assembled rather than + spliced, across the dialects that mangle dotted aliases at emission. + +The baseline is the state as of PR 4 — its purpose is to make any unintended +change in PRs 5 and 6 fail with a diff rather than pass silently. +""" + +from __future__ import annotations + +import asyncio +import json +import os +from pathlib import Path + +import pytest + +from slayer.core.query import SlayerQuery + +from tests._dev1747_fixtures import dev1747_models +from tests._engine_helpers import _engine_generate + + +GOLDEN_PATH = Path(__file__).parent / "golden" / "dev1747_sql_baseline.json" + +#: Postgres and SQLite for the common shapes; DuckDB for a third null-ordering +#: regime (``nulls_are_last``); BigQuery and T-SQL because both mangle dotted +#: aliases at emission, which is the corruption the AST-only chain prevents. +DIALECTS = ["postgres", "sqlite", "duckdb", "tsql", "bigquery"] + +# ``::`` -> why this entry is allowed to change right now. +# A PENDING list, not a log: a committed state always has this empty. +ALLOWED_DELTAS: dict[str, str] = {} + +_MONTH = [{"dimension": "created_at", "granularity": "month"}] + + +def _q(**kw) -> SlayerQuery: + kw.setdefault("source_model", "orders") + return SlayerQuery(**kw) + + +def _cases() -> dict: + """The matrix. Keys are stable ids — renaming one is a golden change.""" + rev = [{"formula": "amount:sum", "name": "rev"}] + return { + # --- the five ORDER BY render paths --- + "order/host_base_alias": _q( + dimensions=["status"], measures=rev, + order=[{"column": "rev", "direction": "desc"}], + ), + "order/hidden_slot_outer_trim": _q( + dimensions=["status"], measures=rev, + order=[{"column": "amount:max", "direction": "desc"}], + ), + "order/combined_cross_model": _q( + dimensions=["status"], + measures=[ + {"formula": "amount:sum", "name": "rev"}, + {"formula": "customers.spend:sum", "name": "cs"}, + ], + order=[{"column": "cs", "direction": "desc"}], + ), + "order/outer_composite": _q( + dimensions=["status"], + measures=[{"formula": "customers.spend:sum + amount:sum", "name": "mix"}], + order=[{"column": "mix", "direction": "desc"}], + ), + "order/windowed_cte": _q( + time_dimensions=_MONTH, + measures=[{"formula": "amount:sum(window='90d')", "name": "w"}], + order=[{"column": "w", "direction": "desc"}], + ), + "order/transform_chain_wrap": _q( + time_dimensions=_MONTH, + measures=[{"formula": "cumsum(amount:sum)", "name": "cs"}], + order=[{"column": "cs", "direction": "asc"}], + ), + # --- D10: the wrap is direction-aware, so BOTH are pinned --- + "order/grouped_local_row_asc": _q( + dimensions=["status"], measures=rev, + order=[{"column": "created_at", "direction": "asc"}], + ), + "order/grouped_local_row_desc": _q( + dimensions=["status"], measures=rev, + order=[{"column": "created_at", "direction": "desc"}], + ), + # --- D2: host-grain, where a target-rooted route would go scalar --- + "order/grouped_joined_row_asc": _q( + dimensions=["status"], measures=rev, + order=[{"column": "customers.regions.name", "direction": "asc"}], + ), + "order/grouped_joined_row_desc": _q( + dimensions=["status"], measures=rev, + order=[{"column": "customers.regions.name", "direction": "desc"}], + ), + # --- D9: a derived column whose sql crosses, both groupings --- + "order/grouped_derived_crossing": _q( + dimensions=["status"], measures=rev, + order=[{"column": "cust_region", "direction": "asc"}], + ), + "order/ungrouped_derived_crossing": _q( + dimensions=["status"], distinct_dimension_values=False, + order=[{"column": "cust_region", "direction": "asc"}], + ), + # --- B6: re-rooting, per filter reachability --- + "reroot/reachable_filter": _q( + dimensions=["customers.regions.name"], + measures=[ + {"formula": "amount:sum", "name": "rev"}, + {"formula": "customers.spend:sum", "name": "cs"}, + ], + filters=["customers.regions.name == 'Alpha'"], + ), + "reroot/host_local_filter": _q( + dimensions=["customers.regions.name"], + measures=[ + {"formula": "amount:sum", "name": "rev"}, + {"formula": "customers.spend:sum", "name": "cs"}, + ], + filters=["status == 'A'"], + ), + "reroot/unreachable_filter": _q( + dimensions=["customers.regions.name"], + measures=[ + {"formula": "amount:sum", "name": "rev"}, + {"formula": "customers.spend:sum", "name": "cs"}, + ], + filters=["order_tags.name == 'rush'"], + ), + # --- D8: both assembled WITH chains --- + "chain/local_multi_step": _q( + time_dimensions=_MONTH, + measures=[ + {"formula": "amount:sum", "name": "rev"}, + {"formula": "cumsum(amount:sum)", "name": "cs"}, + {"formula": "change(amount:sum)", "name": "ch"}, + ], + ), + "chain/local_consecutive_periods": _q( + time_dimensions=_MONTH, + measures=[ + {"formula": "amount:sum", "name": "rev"}, + {"formula": "consecutive_periods(amount:sum)", "name": "streak"}, + ], + ), + "chain/cross_model_window": _q( + time_dimensions=_MONTH, + measures=[ + {"formula": "customers.spend:sum", "name": "cs"}, + {"formula": "cumsum(customers.spend:sum)", "name": "run"}, + ], + ), + } + + +async def _generate_one(query: SlayerQuery, dialect: str): + """Emitted SQL, or a structured record of the raised error. + + The record keeps the COMPLETE message rather than the type alone: a type + name lets any NEW failure in the same case pass unnoticed, which is the + blind spot this harness exists to close. + """ + models = dev1747_models() + try: + return await _engine_generate( + query=query, model=models[0], extra_models=models[1:], + dialect=dialect, validate=False, + ) + except Exception as exc: # noqa: BLE001 — the exception itself is contract + return {"error": type(exc).__name__, "message": str(exc)} + + +def _render(value) -> str: + if isinstance(value, dict): + return f"RAISED {value.get('error')}: {value.get('message')}" + return str(value) + + +def _build_baseline() -> dict: + # conftest's autouse ``_enable_scope_validation`` is FUNCTION-scoped and so + # is not in effect while a module-scoped fixture runs. Set it explicitly, or + # a shape that trips ScopeLeakError during a test would have been recorded + # as valid SQL and every run would "fail" with a spurious diff. + previous = os.environ.get("SLAYER_VALIDATE_SCOPES") + os.environ["SLAYER_VALIDATE_SCOPES"] = "1" + + async def _run() -> dict: + out: dict = {} + for case_id, query in _cases().items(): + for dialect in DIALECTS: + out[f"{case_id}::{dialect}"] = await _generate_one(query, dialect) + return out + + try: + return asyncio.run(_run()) + finally: + if previous is None: + os.environ.pop("SLAYER_VALIDATE_SCOPES", None) + else: + os.environ["SLAYER_VALIDATE_SCOPES"] = previous + + +def _expected_keys() -> set: + return {f"{c}::{d}" for c in _cases() for d in DIALECTS} + + +def _merge_regenerated( + *, existing: dict | None, fresh: dict, allowed: dict, expected: set, +) -> dict: + """Fold ``fresh`` into ``existing``, honouring the allowed-delta manifest. + + Only keys named in ``allowed`` may overwrite a value already in the golden + file — that restriction IS the mechanism. Keys for newly added cases fold in + unconditionally (no prior approval to protect); keys for removed cases are + pruned. + """ + if existing is None: + return dict(fresh) + + unknown = sorted(set(allowed) - expected) + if unknown: + raise AssertionError( + f"ALLOWED_DELTAS names keys that are not in the matrix: {unknown}" + ) + + merged = {k: v for k, v in existing.items() if k in expected} + for key, value in fresh.items(): + if key not in merged or key in allowed: + merged[key] = value + return merged + + +@pytest.fixture(scope="module") +def baseline() -> dict: + if os.environ.get("SLAYER_UPDATE_GOLDEN"): + existing = ( + json.loads(GOLDEN_PATH.read_text()) if GOLDEN_PATH.exists() else None + ) + GOLDEN_PATH.parent.mkdir(parents=True, exist_ok=True) + GOLDEN_PATH.write_text( + json.dumps( + _merge_regenerated( + existing=existing, fresh=_build_baseline(), + allowed=ALLOWED_DELTAS, expected=_expected_keys(), + ), + indent=2, sort_keys=True, + ) + "\n" + ) + if not GOLDEN_PATH.exists(): + pytest.fail( + f"golden baseline missing at {GOLDEN_PATH}; generate it with " + f"SLAYER_UPDATE_GOLDEN=1" + ) + return json.loads(GOLDEN_PATH.read_text()) + + +@pytest.mark.parametrize("case_id", sorted(_cases())) +@pytest.mark.parametrize("dialect", DIALECTS) +def test_emitted_sql_matches_golden(case_id: str, dialect: str, baseline) -> None: + key = f"{case_id}::{dialect}" + assert key in baseline, ( + f"{key} is not in the golden baseline — a new case must be added " + f"deliberately (SLAYER_UPDATE_GOLDEN=1) and reviewed" + ) + actual = asyncio.run(_generate_one(_cases()[case_id], dialect)) + assert actual == baseline[key], ( + f"emitted SQL changed for {key}.\n" + f"--- golden ---\n{_render(baseline[key])}\n" + f"--- actual ---\n{_render(actual)}\n" + f"If this change is intended, get it approved per the DEV-1742 " + f"per-test protocol, add {key!r} to ALLOWED_DELTAS with the reason, " + f"regenerate with SLAYER_UPDATE_GOLDEN=1, then delete the entry." + ) + + +def test_baseline_covers_every_case_and_dialect(baseline) -> None: + missing = _expected_keys() - set(baseline) + assert not missing, f"golden baseline is missing entries: {sorted(missing)}" + + +def test_baseline_has_no_orphan_entries(baseline) -> None: + orphans = set(baseline) - _expected_keys() + assert not orphans, ( + f"golden baseline has entries for cases that no longer exist: " + f"{sorted(orphans)}; regenerate to prune them" + ) + + +def test_allowed_deltas_name_real_keys() -> None: + unknown = sorted(set(ALLOWED_DELTAS) - _expected_keys()) + assert not unknown, ( + f"ALLOWED_DELTAS names keys that are not in the matrix: {unknown}" + ) + + +def test_allowed_deltas_carry_a_reason() -> None: + blank = sorted(k for k, v in ALLOWED_DELTAS.items() if not str(v).strip()) + assert not blank, ( + f"every allowed delta must say WHY the SQL is permitted to change: " + f"{blank}" + ) + + +def test_ordering_cases_actually_emit_an_order_by(baseline) -> None: + """Vacuity guard. Half this matrix exists to pin ORDER BY shapes; an entry + that silently stopped emitting one would still "match golden" forever once + the empty form was blessed.""" + for key, value in baseline.items(): + if not key.startswith("order/"): + continue + assert isinstance(value, str), f"{key} records an error, not SQL: {value}" + assert "ORDER BY" in value.upper(), ( + f"{key} is an ordering case that emits no ORDER BY:\n{value}" + ) From 741dbb74ed0372ab234b2f68fc8bead6b519f443 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Thu, 6 Aug 2026 21:13:58 +0200 Subject: [PATCH 62/98] DEV-1747: mark the four superseded ORDER BY resolvers (P-J state 1) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `_build_combined_order_by_sql` + `_resolve_combined_order_term`, `_apply_order_limit_from_planned`, and `_planned_order_by_sql` now have no production caller — the first pair is a closed island whose only caller is its own partner. Banner says so, and says where the absence is pinned (raising sentinels in the test suite, not a grep of this file), so a reader who finds them does not have to work out whether they are live. Deletion is PR 6, in one sweep, per P-J. Co-Authored-By: Claude Opus 5 (1M context) --- slayer/sql/generator.py | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/slayer/sql/generator.py b/slayer/sql/generator.py index 323f389d..b986e9c7 100644 --- a/slayer/sql/generator.py +++ b/slayer/sql/generator.py @@ -6702,6 +6702,23 @@ def _build_arith_or_cmp_ast( """ return render_arithmetic(op, list(operands)) + # ----------------------------------------------------------------- + # Superseded by ``resolve_order_term`` (DEV-1742 §5.10 / P-J state 1) + # ----------------------------------------------------------------- + # + # ``_build_combined_order_by_sql`` + ``_resolve_combined_order_term`` (the + # combined SELECT), ``_apply_order_limit_from_planned`` (the host base and + # its outer trim wrap), and ``_planned_order_by_sql`` (the chain outer + # wrap) are the four per-site ORDER BY resolutions the one resolver + # replaced. All four are PRODUCTION-UNREFERENCED as of this change — the + # first pair is a closed island, since its only caller is its own partner. + # + # Their tests stay green so the two mechanisms can be compared, and + # ``tests/test_dev1747_order_resolver.py`` pins the absence with raising + # sentinels over every render shape rather than by grepping this file. + # Deletion happens in one sweep (PR 6) rather than smeared across the + # series. + def _build_combined_order_by_sql( self, *, From 373ead9654a47d0ab32e74bf0341bd7ea9c1db52 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Thu, 6 Aug 2026 21:35:18 +0200 Subject: [PATCH 63/98] =?UTF-8?q?DEV-1747=20B6:=20cover=20the=20bug=20at?= =?UTF-8?q?=20the=20layer=20it=20actually=20broke=20=E2=80=94=20the=20rows?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The B6 routing regression was fixed with a plan-level assertion. That is not the bar: the plan was self-consistent throughout, and the wrongness existed only in the rows a user gets back. Codex reviewed the coverage and found two holes, both real. **The HAVING half was uncovered.** Deleting only `"having_filter_ids": []` left the entire suite green. An AGGREGATE-phase predicate over the isolated aggregate is NOT host-evaluable — the aggregate does not live in `_base` at all — so unlike a ROW-phase filter it must stay routed. Worse, the earlier un-blanking had turned the documented `NotImplementedError` for that shape (7b.12: a cross-model aggregate ref routes via the per-plan CTE, not inline HAVING) into a silent answer: three regions returned, two carrying a NULL measure, for a query whose entire point was to keep only groups above a threshold. A wrong answer is strictly worse than an unsupported one, and the merge base agrees — it raises too. Now pinned in both directions. **"The CTE applied its own copy" was unfalsifiable.** Every fixture filtered on the very column it grouped by, which hides the question: the host keeps Alpha, the join-back picks Alpha's row, and Alpha's aggregate is right whether or not the CTE filtered anything. So the corpus gains a second customer in region Alpha, with no orders and the other tier, and a filter on `customers.tier` now changes the aggregate INSIDE a group that survives it (1040 -> 1000). Verified against a simulated wrong fix that filters at the host and not in the CTE — the new test is the only thing that catches it. Each half of the fix is now independently falsifiable: deleting the `where` clearing fails 5 tests, deleting the `having` clearing fails 1, and simulating the CTE-skips-its-copy variant fails 2. Also from the review: a vacuity guard asserting every fixture in the class still re-roots (the row assertions would otherwise pass while testing nothing), and warnings are now CAPTURED rather than suppressed — a reachable predicate misclassified as unreachable still produces right-looking rows because the host applies it anyway, so the warning is the only place that mistake surfaces. Co-Authored-By: Claude Opus 5 (1M context) --- tests/_dev1747_fixtures.py | 17 ++ tests/test_dev1747_reroot_filter_routing.py | 236 ++++++++++++++++++++ 2 files changed, 253 insertions(+) diff --git a/tests/_dev1747_fixtures.py b/tests/_dev1747_fixtures.py index 82797c78..1c68ae61 100644 --- a/tests/_dev1747_fixtures.py +++ b/tests/_dev1747_fixtures.py @@ -106,6 +106,15 @@ def seed_dev1747_sqlite(db_path: str) -> None: (101, 2, "gold", 250.0), (102, 3, "silver", 75.0), (103, 4, "silver", 50.0), + # Region Alpha's SECOND customer, with NO orders and the other + # tier. It exists so a target-side filter can change a cross-model + # aggregate WITHIN a group that survives the filter, rather than + # only removing whole groups: ``customers.tier == 'gold'`` takes + # Alpha's spend from 1040 to 1000 while Alpha stays in the result. + # Without it, a re-rooted CTE that failed to apply its copy of the + # filter would still produce the right number for every surviving + # group, because the join-back picks the group the host kept. + (104, 1, "silver", 40.0), ], ) con.executemany( @@ -502,3 +511,11 @@ def response_column_values(rows: List[dict], key: str) -> List[Optional[object]] assert key in row, f"row {i} has no key {key!r}; keys are {sorted(row)}" out.append(row[key]) return out + + +#: Region Alpha's cross-model spend. The UNFILTERED total spans both of its +#: customers; the gold-only total is customer 100 alone. The gap is what makes +#: "did the re-rooted CTE apply its copy of the filter?" observable inside a +#: group the filter does not remove. +ALPHA_SPEND_ALL = 1000.0 + 40.0 +ALPHA_SPEND_GOLD = 1000.0 diff --git a/tests/test_dev1747_reroot_filter_routing.py b/tests/test_dev1747_reroot_filter_routing.py index b0d70870..7468cfac 100644 --- a/tests/test_dev1747_reroot_filter_routing.py +++ b/tests/test_dev1747_reroot_filter_routing.py @@ -39,6 +39,12 @@ from slayer.core.query import ColumnRef, OrderItem, SlayerQuery from slayer.engine.stage_planner import plan_query from tests._dev1747_fixtures import ( + ALPHA_SPEND_ALL, + ALPHA_SPEND_GOLD, + GROUP_A_AMOUNT, + REGION_A_HIGH, + REGION_A_LOW, + REGION_B_ONLY, dev1747_bundle, make_sqlite_engine, seed_dev1747_sqlite, @@ -56,6 +62,16 @@ #: Off the target's graph entirely (``orders -> order_tags``) — unreachable #: from a CTE rooted at ``customers``. FILTER_UNREACHABLE = "order_tags.name == 'rush'" +#: Reachable from BOTH scopes and, unlike the others, it changes the aggregate +#: INSIDE a group that survives it: region Alpha keeps its gold customer and +#: loses its silver one, so ``cs`` drops from 1040 to 1000 while the Alpha row +#: stays. That is what distinguishes "the re-rooted CTE applied its own copy" +#: from "the host filtered and the join-back happened to pick the right group". +FILTER_TARGET_ATTRIBUTE = "customers.tier == 'gold'" +#: An AGGREGATE-phase predicate over the isolated aggregate itself. The host +#: base cannot evaluate it — the aggregate does not live in ``_base`` — so +#: unlike a ROW-phase filter it must STAY routed to the CTE. +FILTER_AGGREGATE_REF = "customers.spend:sum > 500" #: A HOST-ROOTED shape (``cte_root_model == "orders"``): ordering by a derived #: column whose ``Column.sql`` crosses. The DEV-1503/DEV-1709 helpers live only @@ -524,3 +540,223 @@ def test_the_path_counts_as_the_crossing_input(self) -> None: assert cma.cte_root_model == "orders", ( f"the wrap's CTE is rooted at {cma.cte_root_model!r}, not the host" ) + + +# --------------------------------------------------------------------------- +# Group 5 — the ROWS, not the plan (the regression the plan fields hid) +# --------------------------------------------------------------------------- +class TestRerootedFilterStillNarrowsTheHost: + """The first attempt at B6 stopped blanking the routing lists wholesale. + That was right about ``applied_filter_ids`` — an AUDIT, "some scope + evaluates this" — and wrong about ``where_filter_ids`` / + ``having_filter_ids``, which are an INSTRUCTION: the forward CTE took this + filter over, so the host base must not apply it. + + A re-rooted plan has no forward CTE (the sub-plan replaces it and carries + its own re-anchored filters), and the predicate is host-evaluable by + construction, since it was bound against the host. So the instruction told + the host base to skip a filter nothing else applied THERE, and rows the + user excluded came back with a NULL measure attached. + + Every plan-level assertion in this module passed throughout: the plan was + self-consistent, and the wrongness existed only in the rows. These tests + execute. + """ + + @staticmethod + async def _rows_and_warnings(*filters: str): + """Rows plus every warning the execute emitted. + + Warnings are CAPTURED rather than suppressed: a planner that + misclassified a reachable predicate as unreachable would drop it from + the CTE, warn about it, and — because the host still applies it — often + return correct-looking rows anyway. The warning is the only signal. + """ + with tempfile.TemporaryDirectory() as d: + db = os.path.join(d, "dev1747.db") + seed_dev1747_sqlite(db) + engine = await make_sqlite_engine(d, db) + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + response = await engine.execute(_query(*filters)) + dropped = [ + w for w in caught + if isinstance(w.message, UnreachableFilterDroppedWarning) + ] + return response.data, dropped + + @classmethod + async def _rows(cls, *filters: str): + rows, _ = await cls._rows_and_warnings(*filters) + return rows + + def test_every_fixture_here_actually_reroots(self) -> None: + """Vacuity guard for the whole class. Every assertion below is about + what re-rooting does; if the planner quietly stopped re-rooting and + compiled an equivalent forward plan, the row assertions would still + pass and would be testing nothing.""" + for label, flt in ( + ("reachable", FILTER_REACHABLE), + ("host-local", FILTER_HOST_LOCAL), + ("unreachable", FILTER_UNREACHABLE), + ("target-attribute", FILTER_TARGET_ATTRIBUTE), + ("aggregate-ref", FILTER_AGGREGATE_REF), + ): + assert _sole_plan(flt).rerooted_plan is not None, ( + f"the {label} fixture no longer re-roots, so its row " + f"assertions no longer test re-rooting" + ) + + async def test_a_reachable_filter_narrows_the_host_rows(self) -> None: + """``customers.regions.name == 'Alpha'`` keeps ONE region group. + + Without the fix the host base is unfiltered, so all four regions + survive and three of them carry a NULL measure — the user's filter + silently became "annotate, don't exclude". + """ + rows = await self._rows(FILTER_REACHABLE) + regions = [r["orders.customers.regions.name"] for r in rows] + assert regions == [REGION_A_LOW], ( + f"the re-rooted CTE applied the filter but the host base did not, " + f"so excluded regions came back: {regions}" + ) + + async def test_the_excluded_regions_do_not_come_back_as_nulls(self) -> None: + """States the failure mode directly rather than by row count: the + symptom is specifically a row PRESENT with a NULL measure, which a + count assertion alone would not distinguish from a genuinely empty + group the user asked to see.""" + rows = await self._rows(FILTER_REACHABLE) + leaked = [ + r for r in rows + if r["orders.customers.regions.name"] in ( + REGION_A_HIGH, REGION_B_ONLY, None, + ) + ] + assert not leaked, ( + f"regions the filter excludes are present with a NULL measure: " + f"{leaked}" + ) + + async def test_the_surviving_group_keeps_its_own_values(self) -> None: + """The other half: narrowing the host must not disturb the row that + SHOULD be there. A fix that over-filtered — applying the re-rooted + predicate at the host in the TARGET's coordinate system — would empty + the result instead, and the two assertions above would both pass.""" + rows = await self._rows(FILTER_REACHABLE) + assert len(rows) == 1, rows + assert rows[0]["orders.rev"] == 11.0, ( + f"the surviving group's own measure changed: {rows[0]}" + ) + assert rows[0]["orders.cs"] == ALPHA_SPEND_ALL, ( + f"the cross-model measure changed: {rows[0]}" + ) + + async def test_a_host_local_filter_still_narrows_the_host(self) -> None: + """The control for the branch that was ALREADY right: a host-local + filter never had routing ids, so it must be unaffected. ``status == + 'A'`` keeps the two orders of group A, which span two regions.""" + rows = await self._rows(FILTER_HOST_LOCAL) + regions = sorted( + str(r["orders.customers.regions.name"]) for r in rows + ) + assert regions == sorted([REGION_A_LOW, REGION_A_HIGH]), regions + assert sum(r["orders.rev"] for r in rows) == GROUP_A_AMOUNT, rows + + async def test_an_unreachable_filter_still_narrows_the_host(self) -> None: + """The filter the re-rooted CTE CANNOT evaluate must still apply at the + host — that is what the dropped-filter warning promises ("it still + applies at the host"). If the host skipped it too, the warning would be + describing a filter that ran nowhere at all.""" + rows = await self._rows(FILTER_UNREACHABLE) + regions = sorted( + str(r["orders.customers.regions.name"]) for r in rows + ) + # order_tags 'rush' tags orders 1 (region Alpha) and 2 (region Zulu). + assert regions == sorted([REGION_A_LOW, REGION_A_HIGH]), regions + + async def test_the_rerooted_cte_applies_its_own_copy_of_the_filter( + self, + ) -> None: + """The half a filtered-DIMENSION test cannot reach (Codex). + + Grouping by the very column the filter names hides whether the CTE + applied anything: the host keeps only Alpha, the join-back picks + Alpha's row out of the CTE, and Alpha's aggregate is right either way. + A filter on a DIFFERENT target attribute separates them — region Alpha + survives, but with only its gold customer counted. + + So this fails BOTH ways: if the host stops applying the filter, extra + regions come back; if the CTE stops applying it, Alpha's ``cs`` reads + the unfiltered 1040 instead of 1000. + """ + rows = await self._rows(FILTER_TARGET_ATTRIBUTE) + by_region = {r["orders.customers.regions.name"]: r for r in rows} + assert sorted(by_region) == sorted([REGION_A_LOW, REGION_A_HIGH]), ( + f"the host base did not narrow to the gold-tier rows: " + f"{sorted(by_region)}" + ) + assert by_region[REGION_A_LOW]["orders.cs"] == ALPHA_SPEND_GOLD, ( + f"Alpha's cross-model spend is {by_region[REGION_A_LOW]['orders.cs']}, " + f"not {ALPHA_SPEND_GOLD} — the re-rooted CTE aggregated an " + f"UNFILTERED target population (unfiltered total is " + f"{ALPHA_SPEND_ALL})" + ) + + async def test_no_warning_when_every_filter_is_reachable(self) -> None: + """A reachable predicate misclassified as unreachable still produces + right-looking rows, because the host applies it either way. The warning + is the only place that mistake surfaces.""" + for flt in (FILTER_REACHABLE, FILTER_TARGET_ATTRIBUTE, FILTER_HOST_LOCAL): + _, dropped = await self._rows_and_warnings(flt) + assert not dropped, ( + f"{flt!r} is evaluable by the re-rooted CTE but was reported " + f"dropped: {[str(w.message) for w in dropped]}" + ) + + async def test_the_unreachable_filter_warns_exactly_once(self) -> None: + rows, dropped = await self._rows_and_warnings(FILTER_UNREACHABLE) + assert len(dropped) == 1, [str(w.message) for w in dropped] + assert "order_tags" in str(dropped[0].message) + # And the promise the warning makes — "it still applies at the host" — + # holds: only the two 'rush'-tagged orders' regions survive. + regions = sorted(str(r["orders.customers.regions.name"]) for r in rows) + assert regions == sorted([REGION_A_LOW, REGION_A_HIGH]), regions + + +class TestRerootedAggregateRefFilter: + """The HAVING half, which the WHERE-phase tests above cannot reach (Codex). + + An AGGREGATE-phase predicate over the isolated aggregate + (``customers.spend:sum > 500``) is NOT host-evaluable: the aggregate does + not live in ``_base`` at all. So unlike a ROW-phase filter it must STAY + routed to the CTE, and clearing ``having_filter_ids`` alongside + ``where_filter_ids`` would be over-clearing. + + Deleting only the ``having_filter_ids`` half of the fix leaves every other + test in this module green, which is why this class exists. + """ + + def test_the_aggregate_ref_filter_stays_routed(self) -> None: + plan = _sole_plan(FILTER_AGGREGATE_REF) + assert plan.rerooted_plan is not None, "the fixture stopped re-rooting" + assert plan.applied_filter_ids, "the predicate is applied nowhere" + + async def test_it_raises_rather_than_returning_leaked_rows(self) -> None: + """This shape is NOT yet supported end to end (stage 7b.12: a + cross-model aggregate ref in a filter routes via the per-plan CTE, not + inline HAVING), and it must keep saying so. + + The un-blanking that this fix corrects turned that loud + ``NotImplementedError`` into a silent answer — three regions returned, + two of them carrying a NULL measure, for a query whose whole point was + to keep only the groups above a threshold. A wrong answer is strictly + worse than an unsupported one. + """ + with tempfile.TemporaryDirectory() as d: + db = os.path.join(d, "dev1747.db") + seed_dev1747_sqlite(db) + engine = await make_sqlite_engine(d, db) + with pytest.raises(NotImplementedError) as exc: + await engine.execute(_query(FILTER_AGGREGATE_REF)) + assert "not inline HAVING" in str(exc.value), exc.value From 09378a6649d10b938b4ee29403eeaa1a71fa07c6 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Thu, 6 Aug 2026 21:43:11 +0200 Subject: [PATCH 64/98] DEV-1747 B6 (2nd instance): a ROW-phase filter applies at the host too MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Found by following a Codex review's prediction that fixing the re-rooted route per-plan leaves a CROSS-plan hole. It does, and the query it described returns wrong rows: measures: amount:sum, customers.spend:sum, customers.regions.population:sum filter: customers.regions.name == 'Alpha' -> 4 rows (Alpha, Zulu, Bravo, NULL), three carrying a NULL measure `regions` is a FORWARD plan and routes the filter to its CTE's WHERE; `customers` RE-ROOTS and clears. The generator unions `where_filter_ids` over EVERY plan into one `routed_ids` set, so the forward plan's routing decided what the host did about a filter the re-rooted one needed. The underlying reading was wrong for both routes. A filter does not MOVE to a `_cm_` CTE: the CTE is joined back with a LEFT JOIN on the query grain, which propagates a value but never an exclusion. A host row whose group the CTE filtered away does not disappear — it arrives with a NULL measure. So the predicate silently became "blank out their measure" instead of "exclude these rows". ROW-phase predicates are now applied in BOTH places. They are host-evaluable by construction (bound against the host), and double-applying is free: the CTE's copy narrows the aggregate, the host's copy narrows the rows. Only AGGREGATE-phase (`having_filter_ids`) predicates are genuinely un-evaluable at the host — they reference an aggregate that does not live in `_base`, and trying raises the stage-7b.12 NotImplementedError. Present identically at the merge base: not a regression from the re-rooting work, the same bug class one route over. Four tests, each falsifiable — reverting the one-line change fails three: * the cross-plan case, with a vacuity guard asserting the fixture really does mix one re-rooted plan with one forward plan that routes a filter (if both re-rooted the union would be empty and the assertion would pass for the wrong reason); * P-G stated over ROWS: adding a cross-model measure must not change which rows a filter keeps. That is the invariant the skip broke, and it is what makes "apply in both places" a rule rather than a patch; * the hazard this could have introduced, ruled out — the host's copy of a filter on a 1:N path (`order_tags`, where order 1 carries three tags) pulling that join into the base FROM and multiplying a sibling `amount:sum`. Group A reads 24.0, not the 46.0 a three-way fan-out would give, because the host already applies host filters on joined paths this way when no cross-model measure is present. The two cases now agree. 11,337 unit + 508 integration tests pass. Co-Authored-By: Claude Opus 5 (1M context) --- DECISIONS.md | 2 + slayer/sql/generator.py | 40 +++++-- tests/test_dev1747_reroot_filter_routing.py | 125 ++++++++++++++++++++ 3 files changed, 157 insertions(+), 10 deletions(-) diff --git a/DECISIONS.md b/DECISIONS.md index 28f6e801..81f79a9c 100644 --- a/DECISIONS.md +++ b/DECISIONS.md @@ -105,3 +105,5 @@ implementation detail. Include issue refs when known. - 2026-08-06 — Scope assembly: null-safe grain doctrine, pagination through the dialect, and the combined layer as one AST (DEV-1746, PR 3 of the DEV-1742 consolidation). Six consequences worth recording. (1) **One grain join-back builder** (`slayer/sql/render/joins.py`), taking EXPRESSION operand pairs and returning `None` for an empty grain so the caller emits `CROSS JOIN` — a scalar aggregate has no grain to join on, and a truthy `TRUE` would erase that distinction. It replaced a string round-trip that re-parsed pre-quoted operands: SLayer's public aliases are dotted, and on BigQuery the round-trip re-read `_base."orders.customers.status"` as a multi-part *reference*, emitting a qualifier for a table that does not exist (``_base___orders___customers`.`status``). Three golden entries recorded `ScopeLeakError` for exactly that and now record real SQL. The `_wm_` INNER grain went null-safe in the same move: it compared with a plain `=`, so a NULL-dimension group matched nothing and silently received NULL — while the outer join-back and the `_cm_` join-back were already null-safe, so the answer depended on which isolation shape a measure landed in. Executed on DuckDB and SQLite; the integration test that asserted the old behaviour said so in its NAME (`test_null_dimension_group_gets_null_windowed_value`). (2) **Pagination is a dialect-strategy method.** `apply_pagination` is the one place LIMIT/OFFSET is expressed; the T-SQL override owns its rule explicitly (TOP for a bare limit; a deterministic `ORDER BY (SELECT NULL)` before OFFSET/FETCH when the query is unordered, because SQL Server rejects OFFSET without ordering) rather than relying on sqlglot happening to apply it — so the rule survives a sqlglot upgrade and lands in the AST where callers can see it. The cross-model combined path used to append raw text and emitted a literal `LIMIT` on SQL Server, while the SAME query carrying a transform layer went through the outer wrap and came out correct. (3) **The combined statement is one `exp.Select`.** Projection, FROM/JOIN chain, WHERE, ORDER BY and pagination were all string-assembled; the WHERE in particular was glued on with a hand-rolled `"\nAND "` / `"\nWHERE "` connector that chose between them by asking whether an earlier filter existed. `Select.where` conjoins, so that choice disappears. The WITH chain is assembled by `slayer/sql/render/cte_assembly.py` from **declared** `(name, query, depends_on)` entries, never dependencies discovered by scanning the rendered statement: a scan cannot tell a CTE reference from a same-named real table, is defeated by quoting and case folding, and would silently mis-order rather than fail. **CTE bodies stay AST end to end** — the first attempt kept the renderers returning text and parsed at the seam, which re-introduced exactly the dotted-alias corruption the same PR had just removed. One documented parse seam remains: the re-rooted cross-model CTE arrives as a complete nested `WITH … SELECT` from `generate_from_planned`. The combined ORDER BY had to become AST for the same reason — rendering its terms and re-parsing corrupted them into `` `orders___customers`.`spend_sum` ``. (4) **`PlannedQuery.projection` is consumed verbatim** (B7). The combined projection was assembled in four grouped passes, so a cross-model measure declared first was emitted last; the generator admitted it in a comment. Hidden-slot trimming stops being a mechanism at all — a hidden slot is absent from the list. Two subtleties: a slot may legitimately appear MORE THAN ONCE (C13 lets one key be selected under several names, and the plan lists it once per name), so each occurrence consumes the NEXT of that slot's rendered columns — emitting the whole list per occurrence projected every alias once per name; and because pydantic's `model_copy(update=…)` skips validators and rerooting uses exactly that, one renderer-side assertion is kept at the single entry point, raising rather than skipping, since silently dropping a column the plan asked for is how a wrong answer reaches a user. (5) **One materialiser on the cross-model path.** `ScopeFrame` gained `materialize_for(expr, consumer=…)` — `resolve` anchors a ref and closes it, this is the second half alone, for a producer that anchored its own template (a date-truncated grain, a value carrying its column's declared CAST; re-deriving those from a ref would project a different expression than the aggregate consumes). Unifying the dedup tables fixed a defect: grouping a first/last cross-model aggregate by the same crossing expression it aggregates projected that expression TWICE (`regions.weight AS _val_0` and `AS _val_1`), because the grain site and the value site kept separate maps. `_build_first_last_base_select` keeps its own flow until PR 5's `RankedAggregatePlan`; the boundary is pinned by a test that fails when it moves, rather than left as an assumption. (6) **The empty-base grain is a typed plan node** (`EmptyBaseGrainPlan`, presence as the discriminator, carrying the host-local filter ids). No `grain_slot_ids` field — in that shape the grain is empty by definition — and the invariant that field would have encoded had to be restated during implementation, because `CrossModelAggregatePlan.shared_grain_slots` can name a HIDDEN row slot a filter created, which never becomes a projected grain column. Emitted SQL byte-identical. Reformatting churn was accepted and reviewed: emitting the statement in one pass means sqlglot's printer indents CTE bodies, which moved 25 golden entries and broke assertions that pinned layout rather than structure — one test helper (`_extract_src_body`, anchored on the exact text `"\n) AS _src"`) accounted for 19 failures by itself and is now whitespace-tolerant. Also fixed in passing, because it was two lines and not a refactor: a filter on a JOINED model's crossing derived column emitted invalid SQL — both the join scanner and the filter renderer expanded the column without `is_root=False`, so a further-joined ref came out bare, the scanner could not match it to a join path, the hop was never joined, and the filter referenced a table absent from the FROM. The two must stay in lockstep, since discovery scans the same expansion the renderer emits. - 2026-08-06 — Typed re-rooting, host-grain ordering, and one ORDER BY resolver (DEV-1747, PR 4 of the DEV-1742 consolidation). (1) **A pre-bound seam** (`slayer/engine/prebound.py`). `plan_query` split into `bind_query_inputs` → `PreboundQuery` and the planning that consumes it, so a nested sub-plan is handed TYPED, already-bound inputs instead of a regenerated formula string. `StrictQueryCarrier` forbids extra attributes and raises on any field the seam does not approve, which is what makes "the sub-plan reads only what was handed to it" checkable rather than aspirational; the negative proof is that `cross_model_planner` no longer imports `bind_expr` / `bind_filter` / `parse_expr` at all. (2) **Host-grain aggregates** (`AggregateKey.grain="host"`). A path-bearing source always routed to a TARGET-rooted CTE, which for a sort key degenerates to a scalar CROSS JOIN — every group gets the same global value and the sort silently does nothing, so the case was rejected outright instead. The marker separates where a value is READ (through the join) from where it is GROUPED (per host row-group), and the decision lives in `classify_isolation` with the other two. This was impossible before the seam, not merely tidier: formula text cannot express a path-bearing source or a grain marker, and routing one through the old text path bound `name:min` against the wrong model. (3) **Direction-aware wraps.** An undeclared row column in a grouped query wraps as `min` for ASC and `max` for DESC — the extreme the direction actually puts first. The unconditional `MAX` sorted an ascending query by each group's LARGEST member, which differs whenever groups overlap in range. (4) **One order-term resolver** (`slayer/sql/render/order_terms.py`), dispatching on the plan-assigned `OrderScope` with no precedence and no fallback. Four render sites had each re-derived the answer and disagreed on three things, each a bug: an unresolvable slot returned unsorted rows with no error anywhere on four of the five paths; null ordering skipped the dialect strategy on two, so the same query sorted NULLs differently depending on whether it carried a transform; and a PROJECTED cross-model aggregate was named by its CTE column while the SELECT projected it under the user's alias, so the term resolved only by falling through to an input column of the FROM. Null ordering is now nulls-last everywhere by policy (T-SQL excepted, where the portable emulation puts a bracketed alias inside a CASE that re-resolves against the FROM and fails); a WINDOW's internal `ORDER BY` deliberately takes the emitter's NATIVE ordering instead, since an emulation term inside a frame changes which rows the frame covers. (5) **A crossing sort key is a crossing ref.** An ORDER-BY-only target is not in `base_render_order` — materialising it there would project it and change the grain — but Law 1 does not care that ORDER BY is the only thing referencing it, so the derived-dimension scope pass walks order targets and the render-time throwaway-`ScopeFrame` probe that existed solely to DETECT the crossing is gone. (6) **The transform chains hand the assembler AST.** Both were still splicing `WITH` out of f-strings and reading each predecessor positionally (`ctes[-1][0]`); they now declare dependencies, and the bodies stay `exp.Select` end to end because this path carries dotted `.` names that a text round trip re-reads as multi-part references. (7) **Audit and routing are different statements.** The re-rooting fix that stopped blanking the routing lists was right about `applied_filter_ids` (an audit: some scope evaluates this) and wrong about `where_filter_ids` / `having_filter_ids` (an instruction: the forward CTE took it over, so the host base must skip it). A re-rooted plan has no forward CTE and its predicate is host-evaluable by construction, so keeping the routing told the host to skip a filter nothing else applied there, and rows the user excluded came back carrying a NULL measure. Only the integration suite could see it — the unit tests assert on plan fields, and the plan was self-consistent. + +- 2026-08-06 — A ROW-phase host filter applies at the host base AND in the cross-model CTE (DEV-1747 B6, second instance). The generator unioned every plan's `where_filter_ids` into one `routed_ids` set and skipped those at `_base`, on the reading that a filter "moved" to a `_cm_` CTE. It does not move: the CTE is joined back with a LEFT JOIN on the query grain, which propagates a VALUE but never an EXCLUSION — a host row whose group the CTE filtered away does not disappear, it arrives with a NULL measure. So the predicate silently became "blank out their measure" instead of "exclude these rows". Because the set was a union across ALL plans, it was also a cross-plan defect: a forward plan routing its filter made the host skip it for a re-rooted sibling that needed it, so `filters=["customers.regions.name == 'Alpha'"]` returned four regions, three of them NULL. Only AGGREGATE-phase (`having_filter_ids`) predicates are genuinely un-evaluable at the host — they reference an aggregate that does not live in `_base`, and trying raises the stage-7b.12 `NotImplementedError`. ROW-phase ones are host-evaluable by construction (they were bound against the host), double-applying is free (the CTE's copy narrows the aggregate, the host's copy narrows the rows), and it restores P-G stated over ROWS: adding a cross-model measure no longer changes which rows a filter keeps. The hazard this could have introduced — the host's copy pulling a 1:N join into the base FROM and multiplying a sibling measure — does not occur, and is pinned by a test, because the host already applies host filters on joined paths that way when no cross-model measure is present; the two cases now simply agree. Present identically at the merge base, so not a regression from the re-rooting work — the same bug class one route over, found by taking a Codex review's prediction that a per-plan fix leaves a cross-plan hole and constructing the query it described. diff --git a/slayer/sql/generator.py b/slayer/sql/generator.py index b986e9c7..dc4444b3 100644 --- a/slayer/sql/generator.py +++ b/slayer/sql/generator.py @@ -4645,20 +4645,40 @@ def _add_local_aux_slots( base_has_agg = False base_group_by: Dict[str, exp.Expression] = {} else: - # Filters routed to any CTE (WHERE or HAVING) must NOT - # double-apply at the host base — nor pull their joins into - # ``_base`` (the predicate runs in the ``_cm_*`` CTE). - # ``applied_filter_ids`` is the audit union of where + having - # on each plan. + # Which host filters ``_base`` must NOT apply. + # + # Only the ones it CANNOT apply. A ``_cm_`` CTE is joined back with + # a LEFT JOIN on the query grain, which propagates a value but not + # an EXCLUSION: a host row whose group the CTE filtered away does + # not disappear, it arrives with a NULL measure. So a predicate + # applied only in the CTE silently turns "exclude these rows" into + # "blank out their measure", and the user gets rows they asked not + # to see (DEV-1747 B6, second instance). + # + # ROW-phase (``where_filter_ids``) predicates are therefore applied + # in BOTH places. They are host-evaluable by construction — they + # were bound against the host — and applying one at the host is + # exactly what the same query does when it carries no cross-model + # measure at all, so this is also what makes those two agree. + # Double-applying is free: the CTE's copy narrows the aggregate, + # the host's copy narrows the rows. + # + # AGGREGATE-phase (``having_filter_ids``) predicates are the real + # exclusion. They reference the isolated aggregate, which does not + # live in ``_base`` at all, so the host cannot evaluate them — + # trying raises ``NotImplementedError`` (stage 7b.12). # # DEV-1503: outer-WHERE filters (AGGREGATE-phase host filters - # referencing a filtered-local isolated aggregate) also go in - # here so ``_base`` does not double-apply them as HAVING on - # the bare local aggregate expression (which would reference - # an aggregate that no longer lives in ``_base``). + # referencing a filtered-local isolated aggregate) join them, so + # ``_base`` does not double-apply them as HAVING on a bare local + # aggregate expression that no longer lives there. + # + # The union runs across EVERY plan, which is what made this a + # cross-plan defect rather than a per-plan one: a forward plan + # routing its filter used to make the host skip it for a re-rooted + # sibling that needed it. routed_ids: Set[str] = set(outer_where_filter_ids) for plan in planned_query.cross_model_aggregate_plans: - routed_ids.update(plan.where_filter_ids) routed_ids.update(plan.having_filter_ids) ( base_select, diff --git a/tests/test_dev1747_reroot_filter_routing.py b/tests/test_dev1747_reroot_filter_routing.py index 7468cfac..60f54ad1 100644 --- a/tests/test_dev1747_reroot_filter_routing.py +++ b/tests/test_dev1747_reroot_filter_routing.py @@ -760,3 +760,128 @@ async def test_it_raises_rather_than_returning_leaked_rows(self) -> None: with pytest.raises(NotImplementedError) as exc: await engine.execute(_query(FILTER_AGGREGATE_REF)) assert "not inline HAVING" in str(exc.value), exc.value + + +class TestRowPhaseFiltersAlwaysApplyAtTheHost: + """B6, second instance — the same defect on the FORWARD route, found by + following Codex's prediction that a per-plan fix leaves a cross-plan hole. + + The generator unions ``where_filter_ids`` over EVERY cross-model plan into + one ``routed_ids`` set and skips those at the host base. So one plan's + routing decided what the host did about a filter another plan needed — + and the ``_cm_`` join-back is a LEFT JOIN, which propagates a value but + never an exclusion. A host row whose group the CTE filtered away does not + disappear; it arrives with a NULL measure. + + Present identically at the merge base, so this is not a regression from the + re-rooting work — it is the same bug class one route over, and the fix is + that a ROW-phase predicate is applied in BOTH places. It is host-evaluable + by construction, and double-applying costs nothing: the CTE's copy narrows + the aggregate, the host's copy narrows the rows. + """ + + @staticmethod + async def _rows(query: SlayerQuery): + with tempfile.TemporaryDirectory() as d: + db = os.path.join(d, "dev1747.db") + seed_dev1747_sqlite(db) + engine = await make_sqlite_engine(d, db) + with warnings.catch_warnings(): + warnings.simplefilter("ignore", UnreachableFilterDroppedWarning) + return (await engine.execute(query)).data + + #: One RE-ROOTED plan (``customers``, because the region dimension sits a + #: hop past it) and one FORWARD plan (``regions``, for which that same + #: dimension IS the forward path). The filter is reachable from both, so + #: the forward plan routes it to WHERE while the re-rooted plan clears — + #: and the union used to make the host skip it for both. + _MIXED_ROUTES = SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="name", model="customers.regions")], + measures=[ + {"formula": "amount:sum", "name": "rev"}, + {"formula": "customers.spend:sum", "name": "cs"}, + {"formula": "customers.regions.population:sum", "name": "pop"}, + ], + filters=[FILTER_REACHABLE], + ) + + def test_the_fixture_really_mixes_the_two_routes(self) -> None: + """Vacuity guard: the whole point is one re-rooted plan beside one + forward plan that routes a filter. If both re-rooted, the union would + be empty and the row assertion below would pass for the wrong reason.""" + plans = plan_query( + query=self._MIXED_ROUTES, bundle=dev1747_bundle(), + ).cross_model_aggregate_plans + rerooted = [p for p in plans if p.rerooted_plan is not None] + forward_routing = [ + p for p in plans if p.rerooted_plan is None and p.where_filter_ids + ] + assert rerooted, f"no plan re-rooted: {[p.target_model for p in plans]}" + assert forward_routing, ( + f"no forward plan routes a filter to its CTE, so there is no " + f"cross-plan union to test: " + f"{[(p.target_model, p.where_filter_ids) for p in plans]}" + ) + + async def test_one_plans_routing_does_not_unfilter_the_host(self) -> None: + rows = await self._rows(self._MIXED_ROUTES) + regions = [r["orders.customers.regions.name"] for r in rows] + assert regions == [REGION_A_LOW], ( + f"the forward plan's routing made the host skip the filter, so " + f"regions it excludes came back: {regions}" + ) + + async def test_a_row_phase_filter_matches_the_no_cross_model_answer( + self, + ) -> None: + """P-G, stated over rows rather than SQL: adding a cross-model measure + must not change which rows a filter keeps. That is the invariant the + skip broke, and it is what makes "apply it in both places" the right + rule rather than a patch — the host behaves as it always would. + """ + plain = SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="name", model="customers.regions")], + measures=[{"formula": "amount:sum", "name": "rev"}], + filters=[FILTER_REACHABLE], + ) + plain_rows = await self._rows(plain) + mixed_rows = await self._rows(self._MIXED_ROUTES) + assert ( + [r["orders.customers.regions.name"] for r in plain_rows] + == [r["orders.customers.regions.name"] for r in mixed_rows] + ), f"plain={plain_rows} mixed={mixed_rows}" + assert ( + [r["orders.rev"] for r in plain_rows] + == [r["orders.rev"] for r in mixed_rows] + ), "the sibling local measure changed when a cross-model one was added" + + async def test_applying_at_the_host_does_not_fan_out_a_sibling(self) -> None: + """The hazard this fix could plausibly introduce, ruled out: the host's + copy of a filter on a 1:N path (``order_tags``, where order 1 carries + THREE tags) pulls that join into the base FROM, which is exactly how a + sibling ``amount:sum`` gets multiplied. + + It does not, and the reason is structural rather than lucky — the host + applies host filters on joined paths this way already; the cross-model + case now simply agrees with it. Group A must read 24.0, not 46.0 + (11*3 + 13, the value a three-way fan-out on order 1 would produce). + """ + rows = await self._rows(SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status")], + measures=[ + {"formula": "amount:sum", "name": "rev"}, + {"formula": "order_tags.id:count", "name": "tags"}, + ], + filters=["order_tags.name == 'rush'"], + )) + by_status = {r["orders.status"]: r for r in rows} + assert sorted(by_status) == ["A"], ( + f"'rush' tags only orders 1 and 2, both status A: {sorted(by_status)}" + ) + assert by_status["A"]["orders.rev"] == GROUP_A_AMOUNT, ( + f"the sibling local measure fanned out: " + f"{by_status['A']['orders.rev']} != {GROUP_A_AMOUNT}" + ) From 9db990f8680586f3539ed907707b240fa7c467af Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Fri, 7 Aug 2026 11:38:15 +0200 Subject: [PATCH 65/98] =?UTF-8?q?DEV-1747:=20review=20round=201=20?= =?UTF-8?q?=E2=80=94=20two=20wrong-SQL=20bugs,=20one=20identity=20bug?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CodeRabbit and Codex on PR #290. Three of the findings were real defects in code this PR added. **A derived column over another derived column emitted a nonexistent column.** `amount_x4 = amount_x2 * 2` where `amount_x2` is itself derived: only `_expand_derived_column_sql` inlines the sibling. The projection path always used it; the hidden sort-key path resolved the raw `Column.sql` and emitted `ORDER BY CAST(amount_x2 * 2 ...)` — and `amount_x2` is not a column in the database, so the statement failed there rather than here. Both paths now share one `_derived_column_expr`, which is the only thing that stops them drifting again. Three tests, all failing without the fix, including one that executes against SQLite (it raised `no such column: amount_x2`). **An abandoned re-root shipped routing from the wrong coordinate system.** `routes` is classified with `reachable_paths` once `needs_reroot` is set, but two branches then return the FORWARD plan without a sub-plan. A path that is a strict prefix of `target_path` is reachable under the forward prefix rule yet absent from `reachable_paths`, so the filter the forward CTE can evaluate was dropped from it AND warned about. That is precisely the "classified against a coordinate system that no longer applies" failure D6 exists to remove — one branch below it, doing it again. Both abandon paths now re-classify forward. **Re-rooting a `SqlExprKey` could break interning.** `model_copy` skips validators in Pydantic v2, and the `before` validator is what sorts and de-duplicates `referenced_join_paths` — while `__hash__`/`__eq__` read the tuple directly. Stripping can both duplicate residuals and disturb the order, so two semantically equal keys would fail to intern. Constructed instead of copied. An exact-match path also stripped to `()`, which is not a join-path prefix but the field's "same-model filter" marker; it is dropped now. My own test had pinned that empty tuple — corrected, with a new test that two orderings of the same paths reroot to keys that are equal AND hash equal. Also: the membership test in the re-root vote now goes through `path_is_reachable` rather than re-implementing half of it (the rule lives in one place so the two cannot drift); `PreboundQuery` enforces that `bound_filter_texts` is parallel to `bound_filters`, since the routing pass reads them with `zip`, which would silently truncate rather than raise; and the two identical branches of `_resolve_source` are merged (Sonar S1871). Sonar test-quality issues cleared: broad `pytest.raises(Exception)` narrowed to the actual type, composite assertions split, and second throwing calls hoisted out of `pytest.raises` blocks. Invalid, replied on the thread: "verify every OrderEntry site supplies scope/phase" — the suite covers them all and `PlannedQuery` is never deserialized, so there are no dumps that could fail the new required fields. Invalid, no action (Codex): joined-dimension metadata returning type-only is pre-existing and deliberate (`response_meta` resolves format/description against the owning model); `walk_key_path`'s cycle guard matches the binder, which raises "Circular join detected" for the same shape, so a path that revisits a model can never bind in the first place. 11,341 unit + 508 integration tests pass. Co-Authored-By: Claude Opus 5 (1M context) --- slayer/core/keys.py | 26 ++++-- slayer/engine/cross_model_planner.py | 32 +++++-- slayer/engine/prebound.py | 25 +++++- slayer/sql/generator.py | 93 +++++++++++++------- tests/_dev1747_fixtures.py | 5 ++ tests/test_dev1747_derived_crossing_order.py | 85 +++++++++++++++++- tests/test_dev1747_order_entry.py | 10 ++- tests/test_dev1747_order_resolver.py | 28 ++++-- tests/test_dev1747_reroot_filter_routing.py | 9 +- tests/test_dev1747_reroot_visitor.py | 59 +++++++++++-- 10 files changed, 305 insertions(+), 67 deletions(-) diff --git a/slayer/core/keys.py b/slayer/core/keys.py index f611154b..325259a8 100644 --- a/slayer/core/keys.py +++ b/slayer/core/keys.py @@ -812,13 +812,25 @@ def _reroot_sql_expr_key( model instead and must NOT come through here — see the note in :func:`reroot_value_key`. """ - return key.model_copy(update={ - "referenced_join_paths": tuple( - path[len(target_path):] - if tuple(path[: len(target_path)]) == target_path else path - for path in key.referenced_join_paths - ), - }) + stripped = [ + path[len(target_path):] + if tuple(path[: len(target_path)]) == target_path else path + for path in key.referenced_join_paths + ] + # Constructed, NOT ``model_copy``: the ``before`` validator is what sorts + # and de-duplicates ``referenced_join_paths``, and ``model_copy`` skips + # validators in Pydantic v2. Stripping can produce both — two distinct + # paths can share a residual, and the residuals need not stay in sorted + # order — and ``__hash__`` / ``__eq__`` read the tuple directly, so two + # semantically equal keys would fail to intern (CodeRabbit). + # + # An EXACT match strips to ``()``, which is not a join-path prefix at all + # but the documented "same-model filter" marker, so it is dropped rather + # than carried as an empty tuple. + return SqlExprKey( + canonical_sql=key.canonical_sql, + referenced_join_paths=[p for p in stripped if p], + ) def reroot_value_key( diff --git a/slayer/engine/cross_model_planner.py b/slayer/engine/cross_model_planner.py index f7435089..5635558e 100644 --- a/slayer/engine/cross_model_planner.py +++ b/slayer/engine/cross_model_planner.py @@ -1506,23 +1506,43 @@ def _is_forward(path: Tuple[str, ...]) -> bool: crossed = [p for p in hf.crossed_join_paths if p] if not crossed: continue # host-local -> applied at the host base only - if not all(p in reachable_paths for p in crossed): + # Through the shared predicate, not a second copy of the membership + # half: the rule lives in ONE place precisely so the two cannot drift. + if not all( + path_is_reachable( + path=p, target_path=target_path, + reachable_paths=reachable_paths, + ) + for p in crossed + ): continue if any(not _is_forward(p) for p in crossed): needs_reroot = True break + if not needs_reroot: + return _forward_only() + routes = _route_host_filters( host_filters=host_filters, host_slots=host_slots, target_path=target_path, host_model=host_model, terminal_model=target_model, - reachable_paths=reachable_paths if needs_reroot else None, + reachable_paths=reachable_paths, ) + if not (grain_declared or routes.applied): + # The re-rooted shape lost, so the plan that ships is the FORWARD one + # and its routing must be classified under the forward PREFIX rule. + # Returning ``make_plan(routes)`` here shipped a forward plan carrying + # re-rooted-coordinate routing: a path that is a strict prefix of + # ``target_path`` is reachable under the prefix rule but absent from + # ``reachable_paths``, so the filter the forward CTE can evaluate was + # dropped from it AND warned about. That is the same "classified + # against a coordinate system that no longer applies" failure D6 + # exists to remove — one branch below it, doing it again. + return _forward_only() plan = make_plan(routes) - if not needs_reroot or not (grain_declared or routes.applied): - return plan # The CTE applies exactly what the routing says it applies — one decision, # consumed, never re-derived. ``routing_by_id`` maps those ids back to the @@ -1603,7 +1623,9 @@ def _is_forward(path: Tuple[str, ...]) -> bool: sub_agg_sid = s.id break if sub_agg_sid is None: - return plan + # Same abandon, same reason: no sub-plan means the FORWARD plan ships, + # so it must carry forward-coordinate routing. + return _forward_only() # DEV-1747 B6/D6 — the AUDIT survives the reroot; the ROUTING does not, # because they are different statements and the reroot changes only one. diff --git a/slayer/engine/prebound.py b/slayer/engine/prebound.py index 5a51c5d3..c26f2ceb 100644 --- a/slayer/engine/prebound.py +++ b/slayer/engine/prebound.py @@ -30,7 +30,7 @@ from typing import FrozenSet, List, Optional, Tuple -from pydantic import BaseModel, ConfigDict, Field +from pydantic import BaseModel, ConfigDict, Field, model_validator from slayer.core.enums import DataType from slayer.core.format import NumberFormat @@ -90,6 +90,29 @@ class PreboundQuery(BaseModel): offset: Optional[int] = None distinct_dimension_values: bool = True + @model_validator(mode="after") + def _filter_texts_are_parallel(self) -> "PreboundQuery": + """``bound_filter_texts`` is positionally parallel to + ``bound_filters``, and nothing downstream would notice if it were not: + the routing pass reads them with ``zip``, which silently TRUNCATES to + the shorter list. A short texts list would therefore drop host-filter + routings entirely rather than raise — the exact silent-narrowing class + this seam exists to make impossible, so the invariant is enforced here + rather than trusted at each construction site. + """ + if len(self.bound_filter_texts) != len(self.bound_filters): + raise ValueError( + f"PreboundQuery.bound_filter_texts must be parallel to " + f"bound_filters: got {len(self.bound_filter_texts)} texts for " + f"{len(self.bound_filters)} filters.", + ) + if self.n_date_range > len(self.bound_filters): + raise ValueError( + f"PreboundQuery.n_date_range={self.n_date_range} exceeds the " + f"{len(self.bound_filters)} bound filters it slices.", + ) + return self + class StrictQueryCarrier(BaseModel): """The post-bind ``query.*`` surface the §5.4 seam approves. diff --git a/slayer/sql/generator.py b/slayer/sql/generator.py index dc4444b3..0c73364d 100644 --- a/slayer/sql/generator.py +++ b/slayer/sql/generator.py @@ -2649,11 +2649,15 @@ def _resolve_column_filter(key) -> None: ) def _resolve_source(key) -> None: - if isinstance(key.source, ColumnSqlKey): - scope.resolve(key.source) # register-only; render re-expands - elif getattr(key.source, "path", ()): - # DEV-1747 D2 — the host-grain source IS the crossing input. - scope.resolve(key.source) # register-only + # Two shapes, one action. A DERIVED source (``ColumnSqlKey``) may + # cross inside its ``Column.sql``; a PATH-BEARING one crosses by + # the path itself, which is what a host-grain aggregate's source + # does (DEV-1747 D2). Either way the scope only needs the + # register-only resolve — the render re-expands. + if isinstance(key.source, ColumnSqlKey) or getattr( + key.source, "path", (), + ): + scope.resolve(key.source) def _resolve_kwargs(key) -> None: kw: Dict[str, ResolvedAggKwarg] = {} @@ -8873,24 +8877,12 @@ def _add(path: Tuple[str, ...]) -> None: # rooted at the ``__``-path alias of the owning joined model, with # ``is_root=False`` so a further-joined ref carries the full prefix # (``B`` reaching ``C`` → ``B__C``). - if key.path: - owner_model = bundle.get_referenced_model(key.path[-1]) - if owner_model is None: - continue - owner_relation = "__".join(key.path) - else: - owner_model = source_model - owner_relation = source_relation - expanded_sql = self._expand_derived_column_sql( - source_model=owner_model, source_relation=owner_relation, - column_name=key.column_name, bundle=bundle, is_root=not key.path, - ) - col = next( - (c for c in owner_model.columns if c.name == key.column_name), None, - ) - expr = _wrap_cast_for_type( - self._parse(expanded_sql), col.type if col is not None else None, + expr = self._derived_column_expr( + key=key, source_model=source_model, + source_relation=source_relation, bundle=bundle, ) + if expr is None: + continue derived_expr_by_sid[sid] = expr _add(key.path) # the join to the owning model itself (cross-model) for p in self._joined_paths_in_sql( @@ -8900,6 +8892,42 @@ def _add(path: Tuple[str, ...]) -> None: _add(p) return derived_expr_by_sid + def _derived_column_expr( + self, *, key, source_model, source_relation: str, bundle, + ) -> "Optional[exp.Expression]": + """The rendered expression for a derived (``ColumnSqlKey``) column. + + The ONE expansion, so a derived column renders identically wherever it + appears (P-G). A derived column's ``Column.sql`` may reference ANOTHER + derived column on the same model (``amount_x4 = amount_x2 * 2``), which + only :meth:`_expand_derived_column_sql` inlines — resolving the raw + ``Column.sql`` instead emits the sibling's NAME, and no such database + column exists. The projection path always expanded; the ORDER BY path + resolved raw, so an unprojected derived sort key emitted SQL that + failed at the database (CodeRabbit, DEV-1747). + + ``None`` when the owning model is not in the bundle — the caller + decides whether that is a skip or an error. + """ + if key.path: + owner_model = bundle.get_referenced_model(key.path[-1]) + if owner_model is None: + return None + owner_relation = "__".join(key.path) + else: + owner_model = source_model + owner_relation = source_relation + expanded_sql = self._expand_derived_column_sql( + source_model=owner_model, source_relation=owner_relation, + column_name=key.column_name, bundle=bundle, is_root=not key.path, + ) + col = next( + (c for c in owner_model.columns if c.name == key.column_name), None, + ) + return _wrap_cast_for_type( + self._parse(expanded_sql), col.type if col is not None else None, + ) + def _expand_column_filter_sql( self, *, @@ -10693,13 +10721,12 @@ def _host_base_order_ref( # NOSONAR(S3776) — per-key-kind resolution of ONE h source_relation=source_relation, bundle=bundle, ) - # A LOCAL DERIVED column (``ColumnSqlKey``, path empty). Emitted - # through the planned-dim helper, so its expansion is quoted - # identically to a projected dimension (DEV-1645 mixed-case-safe) — - # and, when the ``Column.sql`` reaches through a join, comes out as the - # same ``customers__regions.name`` reference the bare joined sort key - # emits. That equality is the point of D9: the two spellings name one - # column and must sort the same way. + # A LOCAL DERIVED column (``ColumnSqlKey``, path empty). Rendered + # through the SAME expansion a projected derived dimension gets, which + # is what makes the two spellings of one column sort identically (D9) + # — and what stops a derived column defined over ANOTHER derived column + # from emitting the sibling's name, which is not a database column at + # all (CodeRabbit). # # This used to build a throwaway ``ScopeFrame`` here purely to DETECT # the crossing at render time, and raised when it found one, because @@ -10708,14 +10735,16 @@ def _host_base_order_ref( # NOSONAR(S3776) — per-key-kind resolution of ONE h # to a sort key exactly as it does to a projected dimension. if ( source_model is not None + and bundle is not None and isinstance(row_key, ColumnSqlKey) and not row_key.path ): - return self._joined_or_local_dim_expr( - path=(), leaf=row_key.column_name, - source_model=source_model, + expr = self._derived_column_expr( + key=row_key, source_model=source_model, source_relation=source_relation, bundle=bundle, ) + if expr is not None: + return expr # Defensive: any other hidden shape should have been rejected at plan # time (transform / composite / joined / grouped-row). diff --git a/tests/_dev1747_fixtures.py b/tests/_dev1747_fixtures.py index 1c68ae61..3d89f211 100644 --- a/tests/_dev1747_fixtures.py +++ b/tests/_dev1747_fixtures.py @@ -203,6 +203,11 @@ def _orders_model(*, data_source: str = "test") -> SlayerModel: # A NON-crossing derived column — the control. Ordering by it must # keep working exactly as it does today. Column(name="amount_x2", type=DataType.DOUBLE, sql="amount * 2"), + # A derived column over ANOTHER derived column. Only the expanding + # resolver inlines the sibling; resolving the raw ``Column.sql`` + # emits the bare name ``amount_x2``, which is not a database column + # — so this is what tells the two render paths apart. + Column(name="amount_x4", type=DataType.DOUBLE, sql="amount_x2 * 2"), ], joins=[ ModelJoin(target_model="customers", join_pairs=[["customer_id", "id"]]), diff --git a/tests/test_dev1747_derived_crossing_order.py b/tests/test_dev1747_derived_crossing_order.py index a282ad03..0d139265 100644 --- a/tests/test_dev1747_derived_crossing_order.py +++ b/tests/test_dev1747_derived_crossing_order.py @@ -40,11 +40,18 @@ dev1747_models, make_sqlite_engine, order_by_text, + outermost_select, response_column_values, seed_dev1747_sqlite, ) from tests._engine_helpers import _engine_generate + +def _squash(sql: str) -> str: + """Collapse whitespace — sqlglot pretty-prints a long expression across + lines in one context and inline in another; that is not the subject.""" + return " ".join(sql.split()) + _MEASURE = [{"formula": "amount:sum", "name": "rev"}] @@ -149,8 +156,11 @@ async def test_ungrouped_emits_no_aggregate_wrap(self) -> None: and force grouping the query never asked for.""" sql = await _sql(_ungrouped("asc")) upper = sql.upper() - assert "MIN(" not in upper and "MAX(" not in upper, ( - f"ungrouped sort key must not be aggregate-wrapped:\n{sql}" + assert "MIN(" not in upper, ( + f"ungrouped sort key was wrapped in MIN:\n{sql}" + ) + assert "MAX(" not in upper, ( + f"ungrouped sort key was wrapped in MAX:\n{sql}" ) async def test_ungrouped_derived_matches_the_bare_joined_shape(self) -> None: @@ -231,3 +241,74 @@ async def test_plan_carries_the_crossing_decision(self) -> None: assert plan.order[0].scope in ( OrderScope.HOST_BASE, OrderScope.HOST_BASE_HIDDEN, ) + + +# --------------------------------------------------------------------------- +# Group 4 — a derived column defined over ANOTHER derived column +# --------------------------------------------------------------------------- +class TestDerivedOfDerivedSortKey: + """``amount_x4`` is ``amount_x2 * 2``, and ``amount_x2`` is ``amount * 2``. + + Only the EXPANDING resolver inlines the sibling. The projection path always + used it; the hidden sort-key path resolved the raw ``Column.sql`` instead + and emitted the sibling's NAME — and ``amount_x2`` is not a column in the + database, so the statement failed there rather than here (CodeRabbit). + + The two paths now share one expansion, which is the only thing that keeps + them from drifting again. + """ + + @staticmethod + def _sort_term(sql: str) -> str: + terms = order_by_text(sql) + assert terms, f"no ORDER BY emitted:\n{sql}" + return terms + + async def test_hidden_derived_of_derived_expands_its_sibling(self) -> None: + sql = await _sql(_ungrouped("asc", column="amount_x4")) + term = self._sort_term(sql) + assert "amount_x2" not in term, ( + f"the sort term names the DERIVED sibling, which is not a database " + f"column — the statement would fail at the DB:\n{sql}" + ) + assert "orders.amount" in term, ( + f"the sort term does not reach the real underlying column:\n{sql}" + ) + + async def test_it_matches_what_the_projection_would_emit(self) -> None: + """P-G over the two paths: the same derived column must render the same + expression whether it is projected or only sorted on. Compared to the + PROJECTED expansion rather than to a literal, so the two cannot drift + apart again without this failing.""" + hidden_sql = await _sql(_ungrouped("asc", column="amount_x4")) + projected_sql = await _sql(SlayerQuery( + source_model="orders", + dimensions=[ColumnRef(name="status"), ColumnRef(name="amount_x4")], + distinct_dimension_values=False, + order=[OrderItem(column=ColumnRef(name="amount_x4"), direction="asc")], + )) + # The projected query sorts by the ALIAS, so compare the sort term + # against the projected SELECT expression for the same column. + projected_expr = next( + ( + s.this.sql(dialect="postgres") + for s in outermost_select(projected_sql).expressions + if s.alias_or_name == "orders.amount_x4" + ), + None, + ) + assert projected_expr, f"amount_x4 was not projected:\n{projected_sql}" + hidden_term = self._sort_term(hidden_sql).replace(" ASC", "").strip() + assert _squash(hidden_term) == _squash(projected_expr), ( + f"hidden sort key renders {hidden_term!r}, projection renders " + f"{projected_expr!r}" + ) + + async def test_it_executes(self) -> None: + """The end of the argument: the statement the DB actually runs. Under + the old form SQLite raises ``no such column: amount_x2``.""" + response = await _execute(_ungrouped("asc", column="amount_x4")) + # 4 raw rows, ordered by amount * 4 ascending: 11, 13, 17, 19. + assert response_column_values(response.data, "orders.status") == [ + "A", "A", "B", "N", + ] diff --git a/tests/test_dev1747_order_entry.py b/tests/test_dev1747_order_entry.py index 10557df9..53ba8000 100644 --- a/tests/test_dev1747_order_entry.py +++ b/tests/test_dev1747_order_entry.py @@ -24,6 +24,7 @@ from __future__ import annotations import pytest +from pydantic import ValidationError from slayer.core.query import ColumnRef, OrderItem, SlayerQuery, TimeDimension from slayer.core.enums import TimeGranularity @@ -53,7 +54,7 @@ class TestOrderEntryShape: def test_scope_is_required(self) -> None: """No default. A planner path that forgets to classify must fail at construction, not silently order against ``_base``.""" - with pytest.raises(Exception): + with pytest.raises(ValidationError): OrderEntry(slot_id="s1", direction="asc") # type: ignore[call-arg] def test_nulls_defaults_to_dialect_default(self) -> None: @@ -68,7 +69,7 @@ def test_nulls_defaults_to_dialect_default(self) -> None: def test_nulls_rejects_an_unknown_policy(self) -> None: from slayer.core.keys import Phase - with pytest.raises(Exception): + with pytest.raises(ValidationError): OrderEntry( slot_id="s1", direction="asc", scope=OrderScope.HOST_BASE, phase=Phase.ROW, nulls="sometimes", # type: ignore[arg-type] @@ -78,7 +79,7 @@ def test_direction_validation_still_applies(self) -> None: """The existing contract must survive the enrichment.""" from slayer.core.keys import Phase - with pytest.raises(Exception): + with pytest.raises(ValidationError): OrderEntry( slot_id="s1", direction="ASC", # type: ignore[arg-type] scope=OrderScope.HOST_BASE, phase=Phase.ROW, @@ -328,7 +329,8 @@ def test_local_wrap_keeps_the_default_grain(self) -> None: s.key for s in plan.aggregate_slots if isinstance(s.key, AggregateKey) and s.hidden ] - assert wraps and wraps[0].grain == "target" + assert wraps, "no hidden aggregate wrap was interned" + assert wraps[0].grain == "target" def test_host_grain_and_target_grain_are_distinct_identities(self) -> None: """A user-declared ``customers.regions.name:max`` measure and the diff --git a/tests/test_dev1747_order_resolver.py b/tests/test_dev1747_order_resolver.py index a8f54d46..b0f96738 100644 --- a/tests/test_dev1747_order_resolver.py +++ b/tests/test_dev1747_order_resolver.py @@ -296,14 +296,19 @@ class TestUnresolvableRaises: def test_resolver_raises_when_the_scope_lookup_misses(self) -> None: from slayer.core.keys import Phase from slayer.engine.planned import OrderEntry, OrderScope - from slayer.sql.render.order_terms import OrderEnv, resolve_order_term + from slayer.sql.render.order_terms import ( + OrderEnv, + OrderSlotNotMaterialisedError, + resolve_order_term, + ) entry = OrderEntry( slot_id="missing", direction="asc", scope=OrderScope.CROSS_MODEL_CTE, phase=Phase.AGGREGATE, ) - with pytest.raises(Exception) as exc: - resolve_order_term(entry=entry, env=OrderEnv()) + env = OrderEnv() + with pytest.raises(OrderSlotNotMaterialisedError) as exc: + resolve_order_term(entry=entry, env=env) assert "missing" in str(exc.value), ( "the error must name the unresolvable slot so the wiring bug is " "findable; a bare exception is barely better than the silent drop" @@ -316,15 +321,20 @@ def test_every_scope_raises_on_a_missing_slot(self) -> None: arm added later without one.""" from slayer.core.keys import Phase from slayer.engine.planned import OrderEntry, OrderScope - from slayer.sql.render.order_terms import OrderEnv, resolve_order_term + from slayer.sql.render.order_terms import ( + OrderEnv, + OrderSlotNotMaterialisedError, + resolve_order_term, + ) for scope in OrderScope: entry = OrderEntry( slot_id="missing", direction="asc", scope=scope, phase=Phase.AGGREGATE, ) - with pytest.raises(Exception): - resolve_order_term(entry=entry, env=OrderEnv()) + env = OrderEnv() + with pytest.raises(OrderSlotNotMaterialisedError): + resolve_order_term(entry=entry, env=env) @pytest.mark.parametrize("shape", sorted(_D4_SHAPES)) def test_no_render_path_silently_drops_an_unresolvable_term( @@ -340,6 +350,7 @@ def test_no_render_path_silently_drops_an_unresolvable_term( """ from slayer.engine.stage_planner import plan_query from slayer.sql.generator import generate_from_planned + from slayer.sql.render.order_terms import OrderSlotNotMaterialisedError plan = plan_query(query=_D4_SHAPES[shape], bundle=dev1747_bundle()) assert plan.order, f"{shape} planned no order entry — test is vacuous" @@ -349,8 +360,9 @@ def test_no_render_path_silently_drops_an_unresolvable_term( for entry in plan.order ], }) - with pytest.raises(Exception) as exc: - generate_from_planned(broken, bundle=dev1747_bundle()) + bundle = dev1747_bundle() + with pytest.raises(OrderSlotNotMaterialisedError) as exc: + generate_from_planned(broken, bundle=bundle) assert "no_such_slot" in str(exc.value), ( f"{shape} raised without naming the slot: {exc.value}" ) diff --git a/tests/test_dev1747_reroot_filter_routing.py b/tests/test_dev1747_reroot_filter_routing.py index 60f54ad1..6a1131a2 100644 --- a/tests/test_dev1747_reroot_filter_routing.py +++ b/tests/test_dev1747_reroot_filter_routing.py @@ -262,7 +262,8 @@ def test_warning_carries_the_original_filter_text(self) -> None: def test_warning_carries_a_reason(self) -> None: warning = _sole_plan(FILTER_UNREACHABLE).dropped_filter_warnings[0] - assert warning.reason and "reach" in warning.reason.lower() + assert warning.reason, "the warning carries no reason at all" + assert "reach" in warning.reason.lower(), warning.reason async def test_exactly_one_warning_per_filter_per_execute(self) -> None: """The boundary dedups per filter identity. Two cross-model measures @@ -300,10 +301,11 @@ async def test_warnings_as_errors_mode_surfaces_the_drop(self) -> None: db = os.path.join(d, "dev1747.db") seed_dev1747_sqlite(db) engine = await make_sqlite_engine(d, db) + query = _query(FILTER_UNREACHABLE) with warnings.catch_warnings(): warnings.simplefilter("error", UnreachableFilterDroppedWarning) with pytest.raises(UnreachableFilterDroppedWarning): - await engine.execute(_query(FILTER_UNREACHABLE)) + await engine.execute(query) async def test_two_textually_distinct_filters_warn_separately(self) -> None: """Identity is per FILTER, not per text-dedup bucket — two different @@ -757,8 +759,9 @@ async def test_it_raises_rather_than_returning_leaked_rows(self) -> None: db = os.path.join(d, "dev1747.db") seed_dev1747_sqlite(db) engine = await make_sqlite_engine(d, db) + query = _query(FILTER_AGGREGATE_REF) with pytest.raises(NotImplementedError) as exc: - await engine.execute(_query(FILTER_AGGREGATE_REF)) + await engine.execute(query) assert "not inline HAVING" in str(exc.value), exc.value diff --git a/tests/test_dev1747_reroot_visitor.py b/tests/test_dev1747_reroot_visitor.py index b227bb0d..3016f867 100644 --- a/tests/test_dev1747_reroot_visitor.py +++ b/tests/test_dev1747_reroot_visitor.py @@ -126,7 +126,13 @@ def test_time_trunc_key_over_derived_column(self) -> None: def test_sql_expr_key_strips_referenced_join_paths(self) -> None: """§5.4 lists ``SqlExprKey`` paths explicitly. A standalone fragment - anchored at the query root must be re-anchored at the target.""" + anchored at the query root must be re-anchored at the target. + + The EXACT-match path (``("customers",)`` under target ``("customers",)``) + does not survive as ``()``: the field is documented as "non-anchor + join-path prefixes", and ``()`` is its "same-model filter, no crossing" + marker. Carrying an empty tuple in the list said the opposite of what + the strip means (CodeRabbit).""" out = reroot_value_key( SqlExprKey( canonical_sql="customers__regions.name = 'US'", @@ -135,7 +141,43 @@ def test_sql_expr_key_strips_referenced_join_paths(self) -> None: target_path=TARGET, ) assert out.canonical_sql == "customers__regions.name = 'US'" - assert out.referenced_join_paths == ((), ("regions",)) + assert out.referenced_join_paths == (("regions",),) + + def test_stripping_re_canonicalises_for_identity(self) -> None: + """``model_copy`` skips validators in Pydantic v2, and the ``before`` + validator is what sorts and de-duplicates the paths — while + ``__hash__`` / ``__eq__`` read the tuple directly. + + So a strip that leaves two paths sharing a residual, or leaves them out + of sorted order, produces a key that will not intern against its own + equal. Both inputs below reroot to the same residual set; the two + results must be equal AND hash equal, or the registry mints two slots + for one value.""" + a = reroot_value_key( + SqlExprKey( + canonical_sql="x", + # ``("customers", "regions")`` and a bare ``("regions",)`` + # collapse onto the SAME residual after the strip. + referenced_join_paths=(("customers", "regions"), ("regions",)), + ), + target_path=TARGET, + ) + assert a.referenced_join_paths == (("regions",),), ( + f"duplicate residuals survived the strip: {a.referenced_join_paths}" + ) + + b = reroot_value_key( + SqlExprKey( + canonical_sql="x", + referenced_join_paths=(("regions",), ("customers", "regions")), + ), + target_path=TARGET, + ) + assert a == b and hash(a) == hash(b), ( + "two orderings of the same paths rerooted to keys that do not " + "intern — identity depends on the canonical form the validator " + "produces, which model_copy would have skipped" + ) # --------------------------------------------------------------------------- @@ -374,9 +416,16 @@ def test_empty_target_path_is_identity(self) -> None: ) assert reroot_value_key(key, target_path=()) == key - def test_reroot_is_idempotent_at_the_fixed_point(self) -> None: - """Rerooting an ALREADY-local key again must not strip a second time — - otherwise a double-dispatch anywhere in the planner corrupts the ref.""" + def test_reroot_strips_one_prefix_per_application(self) -> None: + """Each call strips exactly ONE matching prefix. + + Rerooting is therefore NOT idempotent for a path that repeats the + target hop, so the planner must dispatch it exactly once. Pinned so a + double-dispatch shows up as a corrupted ref rather than a silent no-op. + + (The name and docstring used to claim the opposite — "must not strip a + second time" — while the assertions required the strip. A reader would + have taken the stated invariant as a real contract; CodeRabbit.)""" once = reroot_value_key( ColumnKey(path=("customers", "customers"), leaf="x"), target_path=TARGET, ) From 256ea9f1165bd9b85c2647149518a6513c579ba7 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Fri, 7 Aug 2026 11:53:19 +0200 Subject: [PATCH 66/98] DEV-1747: constrain the pre-bound seam's slice bounds (Codex round 2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every count on `PreboundQuery` is a LIST-SLICE bound, and the validator only checked the upper one. A negative `n_date_range` is not a smaller slice — it is a slice from the other end, so `bound_filters[:-1]` silently drops the LAST filter and keeps the rest. Wrong answer, no error. Constrained with `ge=0` at the field, so no construction site can pass one, and the same for `n_dims` / `n_time_dimensions`. Five tests, including the control that the parallel-list guard does not reject the valid shape. Codex's second finding — "using `assert` for model invariants disables validation under `python -O`" — does not apply: the validator raises `ValueError`, which Pydantic surfaces as a `ValidationError`. No `assert` is involved. Co-Authored-By: Claude Opus 5 (1M context) --- slayer/engine/prebound.py | 11 +++-- tests/test_dev1747_prebound_planner.py | 65 ++++++++++++++++++++++++++ 2 files changed, 73 insertions(+), 3 deletions(-) diff --git a/slayer/engine/prebound.py b/slayer/engine/prebound.py index c26f2ceb..c168fa07 100644 --- a/slayer/engine/prebound.py +++ b/slayer/engine/prebound.py @@ -81,11 +81,16 @@ class PreboundQuery(BaseModel): # ``None`` for a synthesized date-range bound. Carried so the cross-model # routing can report a filter the way the caller wrote it. bound_filter_texts: List[Optional[str]] = Field(default_factory=list) - n_date_range: int = 0 + # Every count here is a LIST SLICE bound. A negative one is not a smaller + # slice, it is a slice from the other end — ``bound_filters[:-1]`` silently + # drops the LAST filter and keeps the rest, which is a wrong answer rather + # than an error. Constrained at the field so no construction site can pass + # one (Codex). + n_date_range: int = Field(default=0, ge=0) order_specs: List[OrderSpec] = Field(default_factory=list) main_time_key: Optional[TimeTruncKey] = None - n_dims: int = 0 - n_time_dimensions: int = 0 + n_dims: int = Field(default=0, ge=0) + n_time_dimensions: int = Field(default=0, ge=0) limit: Optional[int] = None offset: Optional[int] = None distinct_dimension_values: bool = True diff --git a/tests/test_dev1747_prebound_planner.py b/tests/test_dev1747_prebound_planner.py index a959c303..4c77d9c6 100644 --- a/tests/test_dev1747_prebound_planner.py +++ b/tests/test_dev1747_prebound_planner.py @@ -33,6 +33,9 @@ from __future__ import annotations import pytest +from pydantic import ValidationError + +from slayer.core.keys import Phase from slayer.core.enums import TimeGranularity from slayer.core.query import ColumnRef, OrderItem, SlayerQuery, TimeDimension @@ -343,3 +346,65 @@ def _wrapped(query, bundle, _builder=builder): f"the nested planner was handed a SlayerQuery ({seen!r}); §5.4 " f"requires the typed pre-bound carrier" ) + + +class TestPreboundQueryInvariants: + """The seam's job is to make a malformed hand-off impossible, so its own + shape has to be checked rather than trusted (Codex). + + Every count on ``PreboundQuery`` is a LIST-SLICE bound and the filter texts + are read positionally with ``zip``. Both failure modes are silent: a + negative count slices from the other end, and a short text list truncates + the routing loop. Neither raises on its own. + """ + + @staticmethod + def _filter(text: str = "status == 'A'"): + from slayer.engine.binding import BoundFilter + from slayer.core.keys import ColumnKey + return BoundFilter( + value_key=ColumnKey(path=(), leaf="status"), + phase=Phase.ROW, + referenced_keys=(ColumnKey(path=(), leaf="status"),), + ) + + def test_filter_texts_must_be_parallel(self) -> None: + from slayer.engine.prebound import PreboundQuery + + with pytest.raises(ValidationError) as exc: + PreboundQuery( + bound_filters=[self._filter(), self._filter()], + bound_filter_texts=["only one"], + ) + assert "parallel" in str(exc.value) + + def test_parallel_lists_are_accepted(self) -> None: + """The control — the guard must not reject the valid shape.""" + from slayer.engine.prebound import PreboundQuery + + pq = PreboundQuery( + bound_filters=[self._filter()], bound_filter_texts=["status == 'A'"], + ) + assert len(pq.bound_filter_texts) == len(pq.bound_filters) + + def test_n_date_range_cannot_exceed_the_filters_it_slices(self) -> None: + from slayer.engine.prebound import PreboundQuery + + with pytest.raises(ValidationError): + PreboundQuery( + bound_filters=[self._filter()], + bound_filter_texts=[None], + n_date_range=2, + ) + + @pytest.mark.parametrize( + "field", ["n_date_range", "n_dims", "n_time_dimensions"], + ) + def test_slice_bounds_cannot_be_negative(self, field: str) -> None: + """``bound_filters[:-1]`` is not an empty slice — it keeps everything + but the last element. A negative count would therefore drop a filter + or a dimension and report nothing.""" + from slayer.engine.prebound import PreboundQuery + + with pytest.raises(ValidationError): + PreboundQuery(**{field: -1}) From 9d52792fcbf1802488aac69fec195207d6aadec5 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Fri, 7 Aug 2026 11:56:53 +0200 Subject: [PATCH 67/98] =?UTF-8?q?DEV-1747:=20review=20round=202=20?= =?UTF-8?q?=E2=80=94=20the=20grain=20prefix,=20and=20imports=20at=20the=20?= =?UTF-8?q?top?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CodeRabbit on the round-1 commit. `n_dims` + `n_time_dimensions` is a PREFIX length into `declared_measures`, and Python slicing past the end returns a SHORTER list rather than raising. An over-count therefore plans fewer dimensions than the caller declared, and misclassifies the measures it does reach as dimensions on the way. Rejected now, with a test. (The negative-count half of the same finding landed in the previous commit from Codex.) Imports hoisted to the top of `test_dev1747_order_resolver.py` per the repo guideline — nothing there monkeypatches those modules, so the local form bought nothing. CodeRabbit also re-raised the broad `pytest.raises(Exception)` nitpick on `test_dev1747_order_entry.py`; that was already fixed in the round-1 commit (narrowed to `ValidationError`), so the nitpick is stale rather than open. Co-Authored-By: Claude Opus 5 (1M context) --- slayer/engine/prebound.py | 14 ++++++++++++ tests/test_dev1747_order_resolver.py | 30 +++++++++----------------- tests/test_dev1747_prebound_planner.py | 12 +++++++++++ 3 files changed, 36 insertions(+), 20 deletions(-) diff --git a/slayer/engine/prebound.py b/slayer/engine/prebound.py index c168fa07..65973a34 100644 --- a/slayer/engine/prebound.py +++ b/slayer/engine/prebound.py @@ -116,6 +116,20 @@ def _filter_texts_are_parallel(self) -> "PreboundQuery": f"PreboundQuery.n_date_range={self.n_date_range} exceeds the " f"{len(self.bound_filters)} bound filters it slices.", ) + # ``n_dims`` and ``n_time_dimensions`` are the DIMENSION prefix lengths + # of ``declared_measures``; the rest of the list is measures. Python + # slicing past the end returns a SHORTER list rather than raising, so + # an over-count silently plans fewer dimensions than the caller + # declared — and the measures it does reach are misclassified as + # dimensions on the way (CodeRabbit). + grain = self.n_dims + self.n_time_dimensions + if grain > len(self.declared_measures): + raise ValueError( + f"PreboundQuery declares {self.n_dims} dimensions + " + f"{self.n_time_dimensions} time dimensions = {grain} grain " + f"members, but carries only {len(self.declared_measures)} " + f"declared measures for them to be a prefix of.", + ) return self diff --git a/tests/test_dev1747_order_resolver.py b/tests/test_dev1747_order_resolver.py index b0f96738..f090fa34 100644 --- a/tests/test_dev1747_order_resolver.py +++ b/tests/test_dev1747_order_resolver.py @@ -39,6 +39,16 @@ ) from tests._engine_helpers import _engine_generate +from slayer.core.keys import Phase +from slayer.engine.planned import OrderEntry, OrderScope +from slayer.engine.stage_planner import plan_query +from slayer.sql.generator import generate_from_planned +from slayer.sql.render.order_terms import ( + OrderEnv, + OrderSlotNotMaterialisedError, + resolve_order_term, +) + _MEASURE = [{"formula": "amount:sum", "name": "rev"}] _MONTH = TimeDimension( @@ -294,14 +304,6 @@ async def test_windowed_measure_orders_on_its_cte_column(self) -> None: # --------------------------------------------------------------------------- class TestUnresolvableRaises: def test_resolver_raises_when_the_scope_lookup_misses(self) -> None: - from slayer.core.keys import Phase - from slayer.engine.planned import OrderEntry, OrderScope - from slayer.sql.render.order_terms import ( - OrderEnv, - OrderSlotNotMaterialisedError, - resolve_order_term, - ) - entry = OrderEntry( slot_id="missing", direction="asc", scope=OrderScope.CROSS_MODEL_CTE, phase=Phase.AGGREGATE, @@ -319,14 +321,6 @@ def test_every_scope_raises_on_a_missing_slot(self) -> None: over its source text. A ``return None`` in ONE arm is enough to reintroduce the silent drop, and a per-scope loop is what catches an arm added later without one.""" - from slayer.core.keys import Phase - from slayer.engine.planned import OrderEntry, OrderScope - from slayer.sql.render.order_terms import ( - OrderEnv, - OrderSlotNotMaterialisedError, - resolve_order_term, - ) - for scope in OrderScope: entry = OrderEntry( slot_id="missing", direction="asc", @@ -348,10 +342,6 @@ def test_no_render_path_silently_drops_an_unresolvable_term( and rendering. Injecting at the PLAN is what makes the injection path-independent; every renderer reads the same field. """ - from slayer.engine.stage_planner import plan_query - from slayer.sql.generator import generate_from_planned - from slayer.sql.render.order_terms import OrderSlotNotMaterialisedError - plan = plan_query(query=_D4_SHAPES[shape], bundle=dev1747_bundle()) assert plan.order, f"{shape} planned no order entry — test is vacuous" broken = plan.model_copy(update={ diff --git a/tests/test_dev1747_prebound_planner.py b/tests/test_dev1747_prebound_planner.py index 4c77d9c6..2f4836fa 100644 --- a/tests/test_dev1747_prebound_planner.py +++ b/tests/test_dev1747_prebound_planner.py @@ -397,6 +397,18 @@ def test_n_date_range_cannot_exceed_the_filters_it_slices(self) -> None: n_date_range=2, ) + def test_the_grain_prefix_cannot_exceed_the_measures(self) -> None: + """``n_dims`` + ``n_time_dimensions`` is a PREFIX length into + ``declared_measures``. Slicing past the end returns a shorter list + rather than raising, so an over-count silently plans fewer dimensions + than were declared — and misclassifies the measures it does reach on + the way (CodeRabbit).""" + from slayer.engine.prebound import PreboundQuery + + with pytest.raises(ValidationError) as exc: + PreboundQuery(declared_measures=[], n_dims=1, n_time_dimensions=1) + assert "grain members" in str(exc.value) + @pytest.mark.parametrize( "field", ["n_date_range", "n_dims", "n_time_dimensions"], ) From 327b584b8650315a04ab9757a6bf8d81464b78c9 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Fri, 7 Aug 2026 12:13:08 +0200 Subject: [PATCH 68/98] =?UTF-8?q?DEV-1747:=20review=20round=203=20?= =?UTF-8?q?=E2=80=94=20the=20last=20six=20Sonar=20test-quality=20issues?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Gate was already OK; these are the remaining open issues, all in tests, and three of them were introduced by the round-1 and round-2 fixes. * **S5778** (two throwing calls inside one `pytest.raises`): the `_filter()` constructions are hoisted out, so the only thing that can raise inside the block is the call under test. Same for the `NotAKey()` instantiation. * **S9073** (composite assertion): `a == b and hash(a) == hash(b)` and `not where_filter_ids and not having_filter_ids` are split, so a failure names which half broke rather than just "the conjunction is false" — the equal-but-different-hash case in particular is a distinct bug with a distinct message. * **S8714** (try/except in a test): pytest already fails on an unexpected exception, and re-wording it through `pytest.fail` only replaced the real traceback with a summary. The test now asserts what it actually wanted — that a sort term is emitted — instead of merely that nothing raised. Co-Authored-By: Claude Opus 5 (1M context) --- tests/test_dev1747_grouped_joined_order.py | 20 ++++++++++---------- tests/test_dev1747_prebound_planner.py | 9 ++++----- tests/test_dev1747_reroot_filter_routing.py | 10 +++++++--- tests/test_dev1747_reroot_visitor.py | 13 +++++++++---- 4 files changed, 30 insertions(+), 22 deletions(-) diff --git a/tests/test_dev1747_grouped_joined_order.py b/tests/test_dev1747_grouped_joined_order.py index c73b0e94..8e08350c 100644 --- a/tests/test_dev1747_grouped_joined_order.py +++ b/tests/test_dev1747_grouped_joined_order.py @@ -408,13 +408,13 @@ class TestRejectRemoved: async def test_no_unresolvable_order_column_error( self, model: str, name: str, ) -> None: - from slayer.core.errors import UnresolvableOrderColumnError - - try: - await _sql(_grouped_order_query( - model=model, name=name, direction="asc", - )) - except UnresolvableOrderColumnError as exc: # pragma: no cover - failure path - pytest.fail( - f"grouped joined ORDER BY on {model}.{name} still rejects: {exc}" - ) + # No try/except: an ``UnresolvableOrderColumnError`` here IS the + # failure this test exists to catch, and letting it propagate gives the + # real traceback instead of a re-worded ``pytest.fail`` (Sonar S8714). + sql = await _sql(_grouped_order_query( + model=model, name=name, direction="asc", + )) + assert order_by_text(sql), ( + f"grouped joined ORDER BY on {model}.{name} emitted no sort term:" + f"\n{sql}" + ) diff --git a/tests/test_dev1747_prebound_planner.py b/tests/test_dev1747_prebound_planner.py index 2f4836fa..684126f1 100644 --- a/tests/test_dev1747_prebound_planner.py +++ b/tests/test_dev1747_prebound_planner.py @@ -371,10 +371,10 @@ def _filter(text: str = "status == 'A'"): def test_filter_texts_must_be_parallel(self) -> None: from slayer.engine.prebound import PreboundQuery + filters = [self._filter(), self._filter()] with pytest.raises(ValidationError) as exc: PreboundQuery( - bound_filters=[self._filter(), self._filter()], - bound_filter_texts=["only one"], + bound_filters=filters, bound_filter_texts=["only one"], ) assert "parallel" in str(exc.value) @@ -390,11 +390,10 @@ def test_parallel_lists_are_accepted(self) -> None: def test_n_date_range_cannot_exceed_the_filters_it_slices(self) -> None: from slayer.engine.prebound import PreboundQuery + filters = [self._filter()] with pytest.raises(ValidationError): PreboundQuery( - bound_filters=[self._filter()], - bound_filter_texts=[None], - n_date_range=2, + bound_filters=filters, bound_filter_texts=[None], n_date_range=2, ) def test_the_grain_prefix_cannot_exceed_the_measures(self) -> None: diff --git a/tests/test_dev1747_reroot_filter_routing.py b/tests/test_dev1747_reroot_filter_routing.py index 6a1131a2..da71ac3b 100644 --- a/tests/test_dev1747_reroot_filter_routing.py +++ b/tests/test_dev1747_reroot_filter_routing.py @@ -127,9 +127,13 @@ def test_reachable_filter_is_routed_not_blanked(self) -> None: # forward CTE, and the predicate is host-evaluable by construction, so # the host must keep applying it — otherwise rows the user excluded # come back carrying a NULL measure. - assert not plan.where_filter_ids and not plan.having_filter_ids, ( - "a re-rooted plan told the host base to skip a filter that only " - "the CTE applies" + assert not plan.where_filter_ids, ( + f"a re-rooted plan routed {plan.where_filter_ids} to a forward " + f"CTE's WHERE that does not exist, so the host base skips it" + ) + assert not plan.having_filter_ids, ( + f"a re-rooted plan routed {plan.having_filter_ids} to a forward " + f"CTE's HAVING that does not exist" ) # The audit has to be backed by something: the sub-plan must actually # carry the filter it claims is applied, or "applied" is a label on diff --git a/tests/test_dev1747_reroot_visitor.py b/tests/test_dev1747_reroot_visitor.py index 3016f867..396963df 100644 --- a/tests/test_dev1747_reroot_visitor.py +++ b/tests/test_dev1747_reroot_visitor.py @@ -173,11 +173,15 @@ def test_stripping_re_canonicalises_for_identity(self) -> None: ), target_path=TARGET, ) - assert a == b and hash(a) == hash(b), ( - "two orderings of the same paths rerooted to keys that do not " - "intern — identity depends on the canonical form the validator " + assert a == b, ( + "two orderings of the same paths rerooted to keys that compare " + "unequal — identity depends on the canonical form the validator " "produces, which model_copy would have skipped" ) + assert hash(a) == hash(b), ( + "the keys compare equal but hash differently, so they still land " + "in different registry buckets and mint two slots for one value" + ) # --------------------------------------------------------------------------- @@ -405,8 +409,9 @@ def test_unknown_kind_raises(self) -> None: class NotAKey: path = ("customers",) + not_a_key = NotAKey() with pytest.raises(TypeError): - reroot_value_key(NotAKey(), target_path=TARGET) + reroot_value_key(not_a_key, target_path=TARGET) def test_empty_target_path_is_identity(self) -> None: """``target_path == ()`` is the filtered-local case — the empty prefix From 6b53d6ab3aee100429697fb11ef6b9d8d5ed0c3e Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Fri, 7 Aug 2026 12:17:11 +0200 Subject: [PATCH 69/98] DEV-1747: imports at the top, across the whole DEV-1747 test pack MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CodeRabbit flagged three in-body import blocks in `test_dev1747_order_resolver.py` against the repo guideline. Fixing only those would have left the same violation in six sibling files I wrote in the same pass — 51 sites in total, 28 of them distinct — so this sweeps all seven. Nothing here needed the local form: none of these tests monkeypatch a module they import, and there is no cycle (the collection-order comment in `test_dev1747_reroot_visitor.py` explains why `reroot_value_key` is imported at module scope, and the same reasoning covers the rest). Two duplicate imports the hoist surfaced were removed. 11,348 tests still pass, ruff clean. Co-Authored-By: Claude Opus 5 (1M context) --- tests/test_dev1747_derived_crossing_order.py | 6 ++--- tests/test_dev1747_local_with_chain.py | 9 +++---- tests/test_dev1747_order_entry.py | 13 +++------- tests/test_dev1747_order_resolver.py | 8 +++--- tests/test_dev1747_prebound_planner.py | 26 ++++++-------------- tests/test_dev1747_reroot_filter_routing.py | 10 +++----- tests/test_dev1747_reroot_visitor.py | 6 ++--- 7 files changed, 27 insertions(+), 51 deletions(-) diff --git a/tests/test_dev1747_derived_crossing_order.py b/tests/test_dev1747_derived_crossing_order.py index 0d139265..eb7a0903 100644 --- a/tests/test_dev1747_derived_crossing_order.py +++ b/tests/test_dev1747_derived_crossing_order.py @@ -45,6 +45,9 @@ seed_dev1747_sqlite, ) from tests._engine_helpers import _engine_generate +from slayer.engine.planned import OrderScope +from slayer.engine.stage_planner import plan_query +from slayer.sql.generator import SQLGenerator def _squash(sql: str) -> str: @@ -211,7 +214,6 @@ async def test_the_probe_host_method_is_never_called( isolate the probe, whereas "this method is not called" is exactly the claim. """ - from slayer.sql.generator import SQLGenerator assert hasattr(SQLGenerator, "_apply_order_limit_from_planned"), ( "the method was deleted; P-J defers deletion to PR 6" @@ -233,8 +235,6 @@ async def test_plan_carries_the_crossing_decision(self) -> None: """Plan-level: the order entry's scope is decided before rendering (P-D). A HOST_BASE_HIDDEN scope on the ungrouped derived entry is what tells the renderer to split-emit rather than probe.""" - from slayer.engine.planned import OrderScope - from slayer.engine.stage_planner import plan_query plan = plan_query(query=_ungrouped("asc"), bundle=dev1747_bundle()) assert plan.order, "plan carries no order entries" diff --git a/tests/test_dev1747_local_with_chain.py b/tests/test_dev1747_local_with_chain.py index 1fb3286c..5101f333 100644 --- a/tests/test_dev1747_local_with_chain.py +++ b/tests/test_dev1747_local_with_chain.py @@ -37,6 +37,10 @@ with_node_of, ) from tests._engine_helpers import _engine_generate +from slayer.sql import generator +from slayer.sql.generator import SQLGenerator +from slayer.sql.render import cte_assembly +import inspect #: A LOCAL (single-model) transform chain — no cross-model measure, so it takes #: the f-string splice path rather than the cross-model one PR 3 already fixed. @@ -83,8 +87,6 @@ def _assembler_spy(monkeypatch) -> list: nothing and every assertion downstream would pass vacuously. The vacuity guard is the ``assert entries`` in each caller. """ - from slayer.sql import generator - from slayer.sql.render import cte_assembly seen: list = [] original = cte_assembly.assemble_with_chain @@ -152,9 +154,7 @@ class TestNoTextRoundTrip: async def test_window_transform_renderer_returns_ast(self) -> None: """``_render_window_transform_sql`` returns a string today, which is what forces a parse at the assembler seam.""" - import inspect - from slayer.sql.generator import SQLGenerator signature = inspect.signature(SQLGenerator._render_window_transform_sql) assert signature.return_annotation is not str, ( @@ -173,7 +173,6 @@ async def test_local_chain_does_not_call_the_parse_seam( round-trips — the very thing D8 removes. A raising sentinel scoped to this render is the exact claim: zero. """ - from slayer.sql.generator import SQLGenerator def _boom(self, sql): # noqa: ANN001 - signature mirrors the seam raise AssertionError( diff --git a/tests/test_dev1747_order_entry.py b/tests/test_dev1747_order_entry.py index 53ba8000..990efcb0 100644 --- a/tests/test_dev1747_order_entry.py +++ b/tests/test_dev1747_order_entry.py @@ -31,6 +31,9 @@ from slayer.engine.planned import OrderEntry, OrderScope from slayer.engine.stage_planner import plan_query from tests._dev1747_fixtures import dev1747_bundle +from slayer.core.keys import AggregateKey +from slayer.core.keys import ColumnKey +from slayer.core.keys import Phase _MEASURE = [{"formula": "amount:sum", "name": "rev"}] @@ -58,7 +61,6 @@ def test_scope_is_required(self) -> None: OrderEntry(slot_id="s1", direction="asc") # type: ignore[call-arg] def test_nulls_defaults_to_dialect_default(self) -> None: - from slayer.core.keys import Phase entry = OrderEntry( slot_id="s1", direction="asc", @@ -67,7 +69,6 @@ def test_nulls_defaults_to_dialect_default(self) -> None: assert entry.nulls == "default" def test_nulls_rejects_an_unknown_policy(self) -> None: - from slayer.core.keys import Phase with pytest.raises(ValidationError): OrderEntry( @@ -77,7 +78,6 @@ def test_nulls_rejects_an_unknown_policy(self) -> None: def test_direction_validation_still_applies(self) -> None: """The existing contract must survive the enrichment.""" - from slayer.core.keys import Phase with pytest.raises(ValidationError): OrderEntry( @@ -206,7 +206,6 @@ def test_grouped_joined_wrap_is_cross_model_cte(self) -> None: # --------------------------------------------------------------------------- class TestPhaseAndDirection: def test_row_target_carries_row_phase(self) -> None: - from slayer.core.keys import Phase entry = _sole_entry(SlayerQuery( source_model="orders", @@ -217,7 +216,6 @@ def test_row_target_carries_row_phase(self) -> None: assert entry.phase is Phase.ROW def test_aggregate_target_carries_aggregate_phase(self) -> None: - from slayer.core.keys import Phase entry = _sole_entry(SlayerQuery( source_model="orders", @@ -249,7 +247,6 @@ def test_multiple_entries_keep_their_own_scope_and_direction(self) -> None: # --------------------------------------------------------------------------- class TestDirectionAwareWrapIsPlanned: def _wrap_aggs(self, direction: str) -> list[str]: - from slayer.core.keys import AggregateKey plan = _plan(SlayerQuery( source_model="orders", @@ -273,7 +270,6 @@ def test_descending_plans_a_max_wrap(self) -> None: def test_same_column_both_directions_plans_two_slots(self) -> None: """MIN(a) and MAX(a) are different values, so they must be different slots. Keying the order remap by key alone collapses them.""" - from slayer.core.keys import AggregateKey plan = _plan(SlayerQuery( source_model="orders", @@ -297,7 +293,6 @@ def test_same_column_both_directions_plans_two_slots(self) -> None: # --------------------------------------------------------------------------- class TestHostGrainMarker: def test_joined_order_wrap_is_marked_host_grain(self) -> None: - from slayer.core.keys import AggregateKey plan = _plan(SlayerQuery( source_model="orders", @@ -317,7 +312,6 @@ def test_joined_order_wrap_is_marked_host_grain(self) -> None: assert wraps[0].source.path == ("customers", "regions") def test_local_wrap_keeps_the_default_grain(self) -> None: - from slayer.core.keys import AggregateKey plan = _plan(SlayerQuery( source_model="orders", @@ -336,7 +330,6 @@ def test_host_grain_and_target_grain_are_distinct_identities(self) -> None: """A user-declared ``customers.regions.name:max`` measure and the synthetic host-grain wrap mean different things (global vs per-group), so they must not intern onto one slot.""" - from slayer.core.keys import AggregateKey, ColumnKey source = ColumnKey(path=("customers", "regions"), leaf="name") target_rooted = AggregateKey(source=source, agg="max") diff --git a/tests/test_dev1747_order_resolver.py b/tests/test_dev1747_order_resolver.py index f090fa34..30028f1e 100644 --- a/tests/test_dev1747_order_resolver.py +++ b/tests/test_dev1747_order_resolver.py @@ -48,6 +48,10 @@ OrderSlotNotMaterialisedError, resolve_order_term, ) +from slayer.sql.dialects.base import SqlDialect +from slayer.sql.dialects.tsql import TsqlDialect +from slayer.sql.generator import SQLGenerator +import pydantic _MEASURE = [{"formula": "amount:sum", "name": "rev"}] @@ -230,7 +234,6 @@ async def test_ordinal_looking_alias_is_not_read_as_a_position(self) -> None: case is unreachable — pinned here so a future relaxation of that validator cannot open the hole silently. The nearest REACHABLE shape (``_1``) must still emit a quoted identifier, not a bare token.""" - import pydantic with pytest.raises(pydantic.ValidationError): ColumnRef(name="1") @@ -367,7 +370,6 @@ class TestNullOrdering: def test_dialect_hook_covers_every_direction_and_policy( self, direction: str, policy: str, ) -> None: - from slayer.sql.dialects.base import SqlDialect ordered = SqlDialect().build_ordered( exp.column("a", quoted=True), @@ -383,7 +385,6 @@ def test_tsql_pins_nulls_first_to_its_native_default( ) -> None: """T-SQL's ORDER BY resolver mis-resolves the bracketed alias INSIDE sqlglot's CASE-WHEN nulls emulation, so the pin suppresses it.""" - from slayer.sql.dialects.tsql import TsqlDialect ordered = TsqlDialect().build_ordered( exp.column("a", quoted=True), @@ -450,7 +451,6 @@ class TestSingleResolver: async def test_superseded_resolver_is_never_called( self, method: str, shape: str, monkeypatch, ) -> None: - from slayer.sql.generator import SQLGenerator assert hasattr(SQLGenerator, method), ( f"{method} has been deleted; P-J defers deletion to PR 6, so " diff --git a/tests/test_dev1747_prebound_planner.py b/tests/test_dev1747_prebound_planner.py index 684126f1..bc2c7675 100644 --- a/tests/test_dev1747_prebound_planner.py +++ b/tests/test_dev1747_prebound_planner.py @@ -41,6 +41,13 @@ from slayer.core.query import ColumnRef, OrderItem, SlayerQuery, TimeDimension from slayer.engine.stage_planner import plan_query from tests._dev1747_fixtures import dev1747_bundle +from slayer.core.keys import AggregateKey, ColumnKey, reroot_value_key +from slayer.engine import cross_model_planner +from slayer.engine import stage_planner +from slayer.engine.binding import BoundFilter +from slayer.engine.prebound import PreboundQuery +from slayer.engine.stage_planner import StrictQueryCarrier +from slayer.engine.stage_planner import bind_query_inputs _SHAPES = { "plain": SlayerQuery( @@ -119,7 +126,6 @@ def test_prebound_plan_matches_the_text_plan(self, shape: str) -> None: """Extract the bind product from a normal plan, feed it back through ``prebound=``, and require an identical plan. Any divergence means the seam is a second planner rather than the same one.""" - from slayer.engine.stage_planner import bind_query_inputs query = _SHAPES[shape] expected = _plan(query) @@ -137,7 +143,6 @@ def test_bind_query_inputs_carries_the_post_bind_scalars(self) -> None: """The fields ``plan_query`` reads off ``query`` AFTER binding. A missing one silently inherits a default — which is exactly the class of bug the strict carrier below is meant to make impossible.""" - from slayer.engine.stage_planner import bind_query_inputs prebound = bind_query_inputs( query=_SHAPES["ordered_paginated"], bundle=dev1747_bundle(), @@ -151,7 +156,6 @@ def test_prebound_carries_the_resolved_main_time_key(self) -> None: """``_resolve_main_time_dimension`` takes the whole ``query``; the prebound path must supply the resolved key instead of re-deriving it from a text carrier.""" - from slayer.engine.stage_planner import bind_query_inputs prebound = bind_query_inputs( query=_SHAPES["time_dimension"], bundle=dev1747_bundle(), @@ -172,14 +176,12 @@ def test_reroot_does_not_parse_inside_the_subplan_boundary( """Scoped spy: the HOST query parses legitimately, so a global counter would be meaningless. The sentinel raises only once the reroot has begun building its nested plan.""" - from slayer.engine import cross_model_planner real_builder_calls: list[int] = [] original = cross_model_planner._maybe_reroot_cross_model_plan def _wrapped(**kwargs): real_builder_calls.append(1) - from slayer.engine import stage_planner def _boom(*_a, **_kw): raise AssertionError( @@ -222,7 +224,6 @@ def test_the_text_serializers_are_never_called(self, monkeypatch) -> None: ``_local_agg_formula`` on the HOST-rooted one. Exercising a single shape would leave the other's serializer free to stay live. """ - from slayer.engine import cross_model_planner patched = [] for symbol in ("_local_agg_formula", "_reroot_ref", "_render_ref_formula"): @@ -251,7 +252,6 @@ def test_nested_plan_measure_is_a_typed_key_not_a_formula(self) -> None: """The observable end state: the nested plan's aggregate slot carries the RE-ROOTED typed key, byte-identical to what the visitor produces — not something re-derived from a string.""" - from slayer.core.keys import AggregateKey, ColumnKey, reroot_value_key plan = _plan(self._reroot_query()) cma = plan.cross_model_aggregate_plans[0] @@ -277,14 +277,12 @@ def test_unapproved_attribute_access_raises(self) -> None: """The guard Codex asked for: if ``plan_query`` grows a new post-bind ``query.*`` read and the seam does not carry it, the reroot must FAIL rather than silently plan against a default.""" - from slayer.engine.stage_planner import StrictQueryCarrier carrier = StrictQueryCarrier(source_model="orders") with pytest.raises(AttributeError): _ = carrier.some_field_the_seam_never_approved def test_approved_attributes_pass_through(self) -> None: - from slayer.engine.stage_planner import StrictQueryCarrier carrier = StrictQueryCarrier(source_model="orders") assert carrier.source_model == "orders" @@ -298,8 +296,6 @@ def test_reroot_actually_constructs_the_strict_carrier( nothing about the live path (an import, a comment, or a branch that is never taken all satisfy a grep). """ - from slayer.engine import cross_model_planner - from slayer.engine.stage_planner import StrictQueryCarrier built: list = [] @@ -322,7 +318,6 @@ def test_the_nested_planner_receives_the_carrier_not_a_text_query( """The seam's boundary condition. A nested ``plan_query`` given a real ``SlayerQuery`` would re-bind formula text no matter how the keys were built upstream.""" - from slayer.engine import cross_model_planner seen: list = [] original = cross_model_planner.IsolatedCteCrossModelPlanner.plan @@ -360,8 +355,6 @@ class TestPreboundQueryInvariants: @staticmethod def _filter(text: str = "status == 'A'"): - from slayer.engine.binding import BoundFilter - from slayer.core.keys import ColumnKey return BoundFilter( value_key=ColumnKey(path=(), leaf="status"), phase=Phase.ROW, @@ -369,7 +362,6 @@ def _filter(text: str = "status == 'A'"): ) def test_filter_texts_must_be_parallel(self) -> None: - from slayer.engine.prebound import PreboundQuery filters = [self._filter(), self._filter()] with pytest.raises(ValidationError) as exc: @@ -380,7 +372,6 @@ def test_filter_texts_must_be_parallel(self) -> None: def test_parallel_lists_are_accepted(self) -> None: """The control — the guard must not reject the valid shape.""" - from slayer.engine.prebound import PreboundQuery pq = PreboundQuery( bound_filters=[self._filter()], bound_filter_texts=["status == 'A'"], @@ -388,7 +379,6 @@ def test_parallel_lists_are_accepted(self) -> None: assert len(pq.bound_filter_texts) == len(pq.bound_filters) def test_n_date_range_cannot_exceed_the_filters_it_slices(self) -> None: - from slayer.engine.prebound import PreboundQuery filters = [self._filter()] with pytest.raises(ValidationError): @@ -402,7 +392,6 @@ def test_the_grain_prefix_cannot_exceed_the_measures(self) -> None: rather than raising, so an over-count silently plans fewer dimensions than were declared — and misclassifies the measures it does reach on the way (CodeRabbit).""" - from slayer.engine.prebound import PreboundQuery with pytest.raises(ValidationError) as exc: PreboundQuery(declared_measures=[], n_dims=1, n_time_dimensions=1) @@ -415,7 +404,6 @@ def test_slice_bounds_cannot_be_negative(self, field: str) -> None: """``bound_filters[:-1]`` is not an empty slice — it keeps everything but the last element. A negative count would therefore drop a filter or a dimension and report nothing.""" - from slayer.engine.prebound import PreboundQuery with pytest.raises(ValidationError): PreboundQuery(**{field: -1}) diff --git a/tests/test_dev1747_reroot_filter_routing.py b/tests/test_dev1747_reroot_filter_routing.py index da71ac3b..b2d5c305 100644 --- a/tests/test_dev1747_reroot_filter_routing.py +++ b/tests/test_dev1747_reroot_filter_routing.py @@ -49,6 +49,9 @@ make_sqlite_engine, seed_dev1747_sqlite, ) +from slayer.engine import cross_model_planner +from slayer.engine.cross_model_planner import IsolatedCteCrossModelPlanner +import inspect #: A cross-model aggregate PLUS a dimension one hop PAST the target, which is #: what makes the planner re-root the CTE at ``customers`` instead of using the @@ -176,7 +179,6 @@ def _classifier_spy(monkeypatch) -> list: calls it, and a future move of the call site into another module would make this spy silently record nothing, which the vacuity assertions below catch. """ - from slayer.engine import cross_model_planner calls: list = [] original = cross_model_planner.classify_host_filter @@ -347,7 +349,6 @@ def test_the_swallow_and_drop_path_is_never_taken(self, monkeypatch) -> None: still swallows there, the swallowed exception escapes here. A grep for the symbol would instead pass the moment someone renamed it. """ - from slayer.engine import cross_model_planner monkeypatch.setattr(cross_model_planner, "_REROOT_BIND_ERRORS", ()) plan = _sole_plan(FILTER_REACHABLE, FILTER_HOST_LOCAL, FILTER_UNREACHABLE) @@ -364,7 +365,6 @@ def test_the_text_filter_classifier_is_never_called(self, monkeypatch) -> None: ``_plan_filtered_local``, so a target-rooted query would leave this sentinel untripped and the test would assert nothing. """ - from slayer.engine import cross_model_planner assert hasattr(cross_model_planner, "_classify_subplan_filters"), ( "the helper was deleted; P-J defers deletion to PR 6" @@ -392,9 +392,7 @@ def test_no_bare_except_in_the_reroot_path(self) -> None: """Scoped to the reroot functions rather than the whole module, so an unrelated ``except Exception`` elsewhere in the file cannot fail this (or, worse, be deleted to make it pass).""" - import inspect - from slayer.engine import cross_model_planner for name in ( "_maybe_reroot_cross_model_plan", @@ -413,7 +411,6 @@ def test_no_bare_except_in_the_reroot_path(self) -> None: def test_planner_failure_propagates_rather_than_warning(self, monkeypatch) -> None: """A genuine internal error must not be reported as an expected drop.""" - from slayer.engine import cross_model_planner boom = RuntimeError("planner exploded") @@ -483,7 +480,6 @@ def _grouped_joined_order(self) -> SlayerQuery: ) def _dispatch_spy(self, monkeypatch) -> list: - from slayer.engine.cross_model_planner import IsolatedCteCrossModelPlanner calls: list = [] original = IsolatedCteCrossModelPlanner._dispatch_filtered_local diff --git a/tests/test_dev1747_reroot_visitor.py b/tests/test_dev1747_reroot_visitor.py index 396963df..0617863d 100644 --- a/tests/test_dev1747_reroot_visitor.py +++ b/tests/test_dev1747_reroot_visitor.py @@ -47,6 +47,9 @@ # test) so the whole module reports ONE clear collection error while it does # not exist yet, rather than N identical failures. from slayer.core.keys import reroot_value_key # noqa: E402 +from slayer.core.keys import ValueKey +from slayer.core.keys import reroot_aggregate_key +from typing import get_args TARGET = ("customers",) DEEP_TARGET = ("customers", "regions") @@ -349,9 +352,7 @@ def test_every_union_member_is_handled(self) -> None: fails this test the day it is added, which is the whole point of a total visitor. """ - from typing import get_args - from slayer.core.keys import ValueKey samples = { ColumnKey: ColumnKey(path=("customers",), leaf="tier"), @@ -521,7 +522,6 @@ def test_reroot_aggregate_key_delegates_to_the_visitor(self) -> None: """``reroot_aggregate_key`` stays (P-J state 1) but must become a thin wrapper, so the two cannot drift into two reroot semantics — which is precisely the drift §5.4 exists to end.""" - from slayer.core.keys import reroot_aggregate_key key = AggregateKey( source=ColumnKey(path=("customers",), leaf="spend"), From 640dc8d75b7cab15dde7e4eda71cf7781464d426 Mon Sep 17 00:00:00 2001 From: Egor Kraev Date: Fri, 7 Aug 2026 15:46:24 +0200 Subject: [PATCH 70/98] DEV-1748: first/last as a plan-shaped isolated CTE (B9) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit One first/last anywhere in a query wrapped the ENTIRE host base in a ROW_NUMBER subquery, so every sibling aggregate was computed over the ranked row set and every ranking in the query shared one scope. The rn-suffix scheme (_last_rn_2) and the filtered sentinel columns (_last_rn_f0 plus a _match_f0 flag consulted by alias) existed only to keep those rankings apart inside that one scope. Needing its own ROW ORDERING is one of the three things P-C says an aggregate can need its own rows for. So it isolates for that reason now: RankedAggregatePlan, a _rk_ CTE rooted where its rows live, joined back on the query grain null-safely, beside _cm_ and _wm_. One aggregate per scope leaves the suffix bookkeeping nothing to disambiguate, and makes the filtered form a plain WHERE applied BEFORE the ranking — the same answer by construction rather than by careful alias lookup. The CTE aggregates rather than filters: MAX(CASE WHEN rn = 1 THEN v END) ... GROUP BY grain, deliberately not SELECT v ... WHERE rn = 1. The two agree on every non-empty grain and disagree on the empty one, where the aggregate returns ONE row holding NULL and the filter returns none — and an empty grain is joined back with a CROSS JOIN, so a zero-row CTE would erase the whole result. A re-rooted cross-model first/last keeps its CrossModelAggregatePlan; its ranked plan belongs to the nested sub-plan. That sub-plan is rendered as a complete statement and spliced into a CTE body, and SQL Server rejects a WITH inside a CTE definition — no sub-plan had ever carried an isolated aggregate before. When a sub-plan's only isolated aggregate IS its answer at its own grain, it is emitted directly as the ranked SELECT (_collapses_to_ranked_cte). Closes the DEV-1476/DEV-1526 remnant: a time arg that is a derived column on a JOINED model used to raise NotImplementedError because the ranking ran in the host base and could not pull the residual join. The ranked CTE resolves its ranking key through its OWN scope. Two pre-existing bugs the rewrite surfaced, both fixed here: * A C13 multi-alias measure alongside any isolated aggregate raised outright. The combined-SELECT host loop iterated base_projection — which lists a slot once per declared name — and emitted that slot's whole alias list each time, rendering N^2 columns. Reachable today with two names on one key plus any cross-model measure. * A HOST column named as the ranking key of a TARGET-rooted first/last emitted ORDER BY ., a column that does not exist on the relation it names, and failed at the database with nothing pointing at the measure. It is a plan-time ValueError now. Behaviour deliberately NOT claimed: the "sibling aggregate stops being multiplied" benefit is vacuous. A first/last whose inputs cross a join already isolated under DEV-1709's widened trigger, and a purely-local one ranks over the same rows the host base holds. Pinned as strict parity instead. Cost, measured on 200k rows / 50 groups on SQLite: one source scan per ranked measure instead of one shared scan. amount:last 176 -> 194 ms (+10%); first+last 287 -> 342 ms (+19%). Non-ranked queries unchanged (sum-only control 41 -> 40 ms). Tests: 88 new (an execution matrix that passes before AND after, a plan/render suite, and a 34-case x 5-dialect golden), ~85 rewritten across nine files where they pinned the retired rn-suffix and sentinel machinery — each keeps its original question and asks it of the new shape. The golden moved on all 170 entries; the five recorded raises became working SQL, and the grain-equals-value case stopped projecting its expression twice. The superseded machinery is production-UNREFERENCED as of this commit and deleted in PR 6 (P-J). A runtime probe over the whole suite confirms no render path enters it; the deletion inventory is recorded above _build_first_last_base_select. Co-Authored-By: Claude Opus 5 (1M context) --- DECISIONS.md | 2 + docs/architecture/cross-model-aggregates.md | 45 +- docs/architecture/ranked-aggregates.md | 128 ++ docs/architecture/sql-generation.md | 6 + slayer/engine/isolation.py | 30 + slayer/engine/planned.py | 92 ++ slayer/engine/ranked_planner.py | 398 ++++++ slayer/engine/stage_planner.py | 76 +- slayer/sql/generator.py | 628 +++++++++- slayer/sql/render/order_terms.py | 1 + slayer/sql/render/ranked.py | 156 +++ tests/_dev1748_fixtures.py | 355 ++++++ tests/golden/dev1748_first_last_baseline.json | 172 +++ tests/test_carrier_scope_matrix.py | 2 +- .../test_dev1476_first_last_explicit_time.py | 33 +- tests/test_dev1645_invalid_postgres_sql.py | 8 +- .../test_dev1686_reserved_word_identifiers.py | 11 +- tests/test_dev1708_stage4_cte_scope.py | 79 +- tests/test_dev1728_derived_shared_grain.py | 44 +- .../test_dev1746_consumer_materialization.py | 114 +- tests/test_dev1748_first_last_matrix.py | 847 +++++++++++++ tests/test_dev1748_golden_sql.py | 381 ++++++ tests/test_dev1748_ranked_plan.py | 1104 +++++++++++++++++ tests/test_filtered_local_isolation.py | 73 +- tests/test_reroot_aggregate_key.py | 8 + tests/test_sql_generator.py | 1094 ++++++++-------- zensical.toml | 1 + 27 files changed, 5139 insertions(+), 749 deletions(-) create mode 100644 docs/architecture/ranked-aggregates.md create mode 100644 slayer/engine/ranked_planner.py create mode 100644 slayer/sql/render/ranked.py create mode 100644 tests/_dev1748_fixtures.py create mode 100644 tests/golden/dev1748_first_last_baseline.json create mode 100644 tests/test_dev1748_first_last_matrix.py create mode 100644 tests/test_dev1748_golden_sql.py create mode 100644 tests/test_dev1748_ranked_plan.py diff --git a/DECISIONS.md b/DECISIONS.md index 81f79a9c..97c9d16a 100644 --- a/DECISIONS.md +++ b/DECISIONS.md @@ -107,3 +107,5 @@ implementation detail. Include issue refs when known. - 2026-08-06 — Typed re-rooting, host-grain ordering, and one ORDER BY resolver (DEV-1747, PR 4 of the DEV-1742 consolidation). (1) **A pre-bound seam** (`slayer/engine/prebound.py`). `plan_query` split into `bind_query_inputs` → `PreboundQuery` and the planning that consumes it, so a nested sub-plan is handed TYPED, already-bound inputs instead of a regenerated formula string. `StrictQueryCarrier` forbids extra attributes and raises on any field the seam does not approve, which is what makes "the sub-plan reads only what was handed to it" checkable rather than aspirational; the negative proof is that `cross_model_planner` no longer imports `bind_expr` / `bind_filter` / `parse_expr` at all. (2) **Host-grain aggregates** (`AggregateKey.grain="host"`). A path-bearing source always routed to a TARGET-rooted CTE, which for a sort key degenerates to a scalar CROSS JOIN — every group gets the same global value and the sort silently does nothing, so the case was rejected outright instead. The marker separates where a value is READ (through the join) from where it is GROUPED (per host row-group), and the decision lives in `classify_isolation` with the other two. This was impossible before the seam, not merely tidier: formula text cannot express a path-bearing source or a grain marker, and routing one through the old text path bound `name:min` against the wrong model. (3) **Direction-aware wraps.** An undeclared row column in a grouped query wraps as `min` for ASC and `max` for DESC — the extreme the direction actually puts first. The unconditional `MAX` sorted an ascending query by each group's LARGEST member, which differs whenever groups overlap in range. (4) **One order-term resolver** (`slayer/sql/render/order_terms.py`), dispatching on the plan-assigned `OrderScope` with no precedence and no fallback. Four render sites had each re-derived the answer and disagreed on three things, each a bug: an unresolvable slot returned unsorted rows with no error anywhere on four of the five paths; null ordering skipped the dialect strategy on two, so the same query sorted NULLs differently depending on whether it carried a transform; and a PROJECTED cross-model aggregate was named by its CTE column while the SELECT projected it under the user's alias, so the term resolved only by falling through to an input column of the FROM. Null ordering is now nulls-last everywhere by policy (T-SQL excepted, where the portable emulation puts a bracketed alias inside a CASE that re-resolves against the FROM and fails); a WINDOW's internal `ORDER BY` deliberately takes the emitter's NATIVE ordering instead, since an emulation term inside a frame changes which rows the frame covers. (5) **A crossing sort key is a crossing ref.** An ORDER-BY-only target is not in `base_render_order` — materialising it there would project it and change the grain — but Law 1 does not care that ORDER BY is the only thing referencing it, so the derived-dimension scope pass walks order targets and the render-time throwaway-`ScopeFrame` probe that existed solely to DETECT the crossing is gone. (6) **The transform chains hand the assembler AST.** Both were still splicing `WITH` out of f-strings and reading each predecessor positionally (`ctes[-1][0]`); they now declare dependencies, and the bodies stay `exp.Select` end to end because this path carries dotted `.` names that a text round trip re-reads as multi-part references. (7) **Audit and routing are different statements.** The re-rooting fix that stopped blanking the routing lists was right about `applied_filter_ids` (an audit: some scope evaluates this) and wrong about `where_filter_ids` / `having_filter_ids` (an instruction: the forward CTE took it over, so the host base must skip it). A re-rooted plan has no forward CTE and its predicate is host-evaluable by construction, so keeping the routing told the host to skip a filter nothing else applied there, and rows the user excluded came back carrying a NULL measure. Only the integration suite could see it — the unit tests assert on plan fields, and the plan was self-consistent. - 2026-08-06 — A ROW-phase host filter applies at the host base AND in the cross-model CTE (DEV-1747 B6, second instance). The generator unioned every plan's `where_filter_ids` into one `routed_ids` set and skipped those at `_base`, on the reading that a filter "moved" to a `_cm_` CTE. It does not move: the CTE is joined back with a LEFT JOIN on the query grain, which propagates a VALUE but never an EXCLUSION — a host row whose group the CTE filtered away does not disappear, it arrives with a NULL measure. So the predicate silently became "blank out their measure" instead of "exclude these rows". Because the set was a union across ALL plans, it was also a cross-plan defect: a forward plan routing its filter made the host skip it for a re-rooted sibling that needed it, so `filters=["customers.regions.name == 'Alpha'"]` returned four regions, three of them NULL. Only AGGREGATE-phase (`having_filter_ids`) predicates are genuinely un-evaluable at the host — they reference an aggregate that does not live in `_base`, and trying raises the stage-7b.12 `NotImplementedError`. ROW-phase ones are host-evaluable by construction (they were bound against the host), double-applying is free (the CTE's copy narrows the aggregate, the host's copy narrows the rows), and it restores P-G stated over ROWS: adding a cross-model measure no longer changes which rows a filter keeps. The hazard this could have introduced — the host's copy pulling a 1:N join into the base FROM and multiplying a sibling measure — does not occur, and is pinned by a test, because the host already applies host filters on joined paths that way when no cross-model measure is present; the two cases now simply agree. Present identically at the merge base, so not a regression from the re-rooting work — the same bug class one route over, found by taking a Codex review's prediction that a per-plan fix leaves a cross-plan hole and constructing the query it described. + +- 2026-08-07 — `first`/`last` compiles to a plan-shaped isolated CTE (DEV-1748 B9). One first/last anywhere in a query used to wrap the ENTIRE host base in a `ROW_NUMBER` subquery, so every sibling aggregate was computed over the ranked row set and every ranking in the query shared one scope. The whole rn-suffix scheme (`_last_rn_2`) and the filtered sentinel columns (`_last_rn_f0` plus a `_match_f0` flag the outer aggregate consulted by ALIAS) existed only to keep those rankings apart inside that one scope. Needing its own ROW ORDERING is one of the three things P-C says an aggregate can need its own rows for, so it now isolates for that reason — `RankedAggregatePlan`, a `_rk_` CTE rooted where its rows live, joined back on the query grain null-safely, beside `_cm_` and `_wm_`. One aggregate per scope leaves the suffix bookkeeping nothing to disambiguate and makes the filtered form a plain `WHERE` applied BEFORE the ranking, which is the same answer by construction rather than by careful alias lookup. Six consequences worth naming. (1) **The CTE aggregates, it does not filter.** `MAX(CASE WHEN rn = 1 THEN v END) … GROUP BY grain`, deliberately not `SELECT v … WHERE rn = 1`: the two agree on every non-empty grain and disagree on the empty one, where the aggregate returns ONE row holding NULL and the filter returns none — and an empty grain is joined back with a CROSS JOIN, so a zero-row CTE erases the whole result instead of yielding a NULL measure. (2) **A re-rooted cross-model first/last keeps its `CrossModelAggregatePlan`**; its ranked plan belongs to the nested sub-plan. That sub-plan is rendered as a complete statement and spliced into a CTE body, and SQL Server rejects a `WITH` inside a CTE definition — no sub-plan had ever contained an isolated aggregate before, so a re-rooted first/last would have been the first to emit one. When a sub-plan's only isolated aggregate IS its answer at its own grain, it is emitted directly as the ranked SELECT. (3) **The ranking column is plan data** (`RankedAggregatePlan.ranking_time_key`), resolved once with an explicit per-scope precedence; it used to be re-derived at render time by two different precedences that agreed only on the cases anyone had tried. It carries no declared-type CAST — an ordering key is compared only to itself, and on SQLite `TIMESTAMP` has numeric affinity, so `CAST(DATE(created_at) AS TIMESTAMP)` truncates every date to its year and ties the whole partition. (4) **The DEV-1476/DEV-1526 remnant closes.** A time arg that is a derived column on a JOINED model raised `NotImplementedError`, because the ranking ran in the host base and could not pull the residual join. A ranked CTE resolves its ranking key through its OWN scope, so the join registers where it is needed. (5) **A HOST column named as the ranking key of a TARGET-rooted first/last is now an error** at plan time, naming the measure. It used to emit `ORDER BY .` — a reference to a column that does not exist on the relation it names — and fail at the database with nothing pointing at the cause. (6) **The sibling benefit is vacuous and is not claimed.** A first/last whose inputs cross a join already isolated under DEV-1709's widened trigger, and a purely-local one ranks over the same rows the host base holds, so no sibling's value moves. It is pinned as strict parity. The superseded machinery is production-UNREFERENCED as of this PR and deleted in PR 6 (P-J); a runtime probe over the whole suite confirms no render path enters it except the direct unit calls that pin it. diff --git a/docs/architecture/cross-model-aggregates.md b/docs/architecture/cross-model-aggregates.md index 34b07932..d91dd905 100644 --- a/docs/architecture/cross-model-aggregates.md +++ b/docs/architecture/cross-model-aggregates.md @@ -210,18 +210,14 @@ the leaves' projected aliases. This holds for projected composites, filter-only composites (routed to the outer WHERE), and order-only aggregate refs. -### Law 2 inside the ranked scope - -The first/last ranked subquery re-exports only `source_relation.*` plus -rank/`_td`/`_dim` columns, so any crossing expression the outer SELECT -consumes — an aggregate SOURCE or a column-ref KWARG value — is -materialised as a `_val_` projection inside the subquery -(`_build_first_last_base_select`, mirroring the Stage-4 CTE path). The -projection is the **resolved** value (qualified, with the `Column.type` -inner CAST for non-bare expressions), so `SUM(CAST(x AS t))` semantics are -preserved and same-sql-different-type aggregates keep distinct -materialisations; HAVING and composite consumers bind to the same alias -via `FirstLastRenderState.value_alias_by_sql` (keyed by resolved text). +### first/last is a route of its own + +A `first`/`last` aggregate does NOT reach the cross-model route. It needs its +own ROW ORDERING, so it isolates for that reason — which subsumes the +crossing-input trigger — and compiles to a `_rk_` CTE. See +[Ranked aggregates](ranked-aggregates.md). The one exception is a **re-rooted** +cross-model `first`/`last`, which keeps its `CrossModelAggregatePlan`: the +ranked plan belongs to its nested sub-plan. ### Filter routing for filtered-local @@ -280,19 +276,18 @@ register their joins in a **pre-pass** that walks the full `ValueKey` tree `column_filter`) before the `FROM` is built, so a HAVING — rendered later, once the ranked-subquery rank columns exist — still contributes its joins. -**First/last value materialization (Law 2).** When the CTE wraps its rows in a -`ROW_NUMBER`-ranked subquery and the first/last **source value** crosses a join, -the crossing value is materialized as a `_val_` projection *inside* the -subquery and the outer `MAX(CASE WHEN _last_rn = 1 THEN _val_ END)` references -the alias — a raw crossing ref there is bound only inside the subquery. A HAVING -on the same aggregate binds the same alias. The **shared grain** obeys the same -law (DEV-1728): a crossing derived grain is materialized as a `_val_` -projection inside the subquery, the outer `SELECT`/`GROUP BY` reference the -alias, and `PARTITION BY` keeps the raw expression (evaluated where the join is -bound). `generate_from_planned` installs one generation-wide `AliasAllocator` -(save/restore) so inline forward CTEs, grain projections, and the host base -never collide on `_val_`; the CTE reserves the target's physical column names -so a minted alias never shadows a `target.*` column. +**Projection boundaries (Law 2).** A `_cm_` CTE consumes a value from an inner +scope by ALIAS, never by re-reaching for the expression: a ref that crosses a +join is bound only where that join is, so the inner scope projects it as +`_val_` and the outer names the alias. The projection is the **resolved** +value (qualified, with the `Column.type` inner CAST for non-bare expressions), +so `SUM(CAST(x AS t))` semantics are preserved and same-sql-different-type +aggregates keep distinct materialisations. `generate_from_planned` installs one +generation-wide `AliasAllocator` (save/restore) so inline forward CTEs, grain +projections, and the host base never collide on `_val_`; each CTE reserves +its root model's physical column names so a minted alias never shadows a real +column. The ranked (`_rk_`) route obeys the same law through the same +`ScopeFrame` table — see [Ranked aggregates](ranked-aggregates.md). ### Derived shared-grain rendering (DEV-1728) diff --git a/docs/architecture/ranked-aggregates.md b/docs/architecture/ranked-aggregates.md new file mode 100644 index 00000000..6511c645 --- /dev/null +++ b/docs/architecture/ranked-aggregates.md @@ -0,0 +1,128 @@ +# Ranked aggregates (`first` / `last`) + +`first` and `last` answer "the value from the row that sorts first (or last) +within each group". That needs a row ORDERING, which is one of the three things +an aggregate can need its own rows for — alongside crossing a join and carrying +its own frame. Under the isolation doctrine (P-C) all three compile the same +way: **a plan-shaped CTE, rooted where its rows live, joined back on the query +grain**. For `first`/`last` that is `RankedAggregatePlan` and a `_rk_` CTE, +beside `_cm_` (cross-model) and `_wm_` (windowed). + +## What it looks like + +```sql +WITH _base AS ( + SELECT orders.status AS "orders.status", + CAST(SUM(orders.amount) AS DOUBLE PRECISION) AS "orders.s" + FROM orders AS orders + GROUP BY orders.status +), _rk_orders__l AS ( + SELECT _val_0 AS "orders.status", + CAST(MAX(CASE WHEN _rk_rn = 1 THEN _val_1 END) AS DOUBLE PRECISION) + AS "orders.l" + FROM ( + SELECT orders.status AS _val_0, + orders.amount AS _val_1, + ROW_NUMBER() OVER ( + PARTITION BY orders.status ORDER BY orders.created_at DESC + ) AS _rk_rn + FROM orders AS orders + ) AS _rk_src + GROUP BY _val_0 +) +SELECT _base."orders.status", _rk_orders__l."orders.l", _base."orders.s" +FROM _base +LEFT JOIN _rk_orders__l + ON _base."orders.status" IS NOT DISTINCT FROM _rk_orders__l."orders.status" +``` + +Two SELECTs. The inner one ranks the rows this aggregate is allowed to see; the +outer picks rank 1 per grain. The host base holds only purely-local aggregates, +so adding a `first`/`last` cannot change host cardinality or any sibling's +value. + +## Where each decision is made + +| Decision | Owner | +| --- | --- | +| Does this aggregate isolate, and where is it rooted | `classify_isolation` → `IsolationKind.RANKED_HOST` / `RANKED_TARGET` | +| Which column the ranking orders by | `ranked_planner.resolve_ranking_time_key` | +| What the grain is, in both coordinate systems | `RankedAggregatePlan.grain` | +| Which filters the CTE evaluates | the plan's `where_filter_ids` / `target_model_filters` | +| What the SQL looks like | `slayer/sql/render/ranked.py` | + +The renderer emits a plan; it re-derives none of the above (P-D). + +## The ranking column + +Resolved at plan time, per scope, and it raises at the end rather than falling +through: + +**Host-rooted** — an explicit positional time arg (`amount:last(shipped_at)`), +else the first `DATE`/`TIMESTAMP` row dimension, else the first time dimension's +**raw** column (never the truncated bucket: ranking within a month by the month +ties every row in it), else the model's `default_time_dimension`. + +**Target-rooted** — the same explicit arg, re-anchored in the target's +coordinates, else the TARGET model's `default_time_dimension`. Host dimensions +are deliberately not candidates: the CTE ranks target rows and a host column is +not one of their attributes. Naming one is an error, reported at plan time. + +The ranking key carries no declared-type CAST. It is compared only to itself, +and on SQLite `TIMESTAMP` has numeric affinity — `CAST(DATE(created_at) AS +TIMESTAMP)` truncates every date to its year and ties the whole partition. + +## Why the CTE aggregates instead of filtering + +`MAX(CASE WHEN _rk_rn = 1 THEN v END) … GROUP BY grain`, not `SELECT v … WHERE +_rk_rn = 1`. The two agree on every non-empty grain and disagree on the empty +one: over a source with no rows the aggregate form returns ONE row holding +NULL and the filter form returns none. An empty grain is joined back with a +`CROSS JOIN`, so a zero-row CTE erases the entire result rather than yielding a +NULL measure — and one NULL row is what `amount:sum` returns over the same +empty source. + +## Filtered variants are plan data + +A measure's `Column.filter` is a predicate on the rows this aggregate ranks, so +in its own scope it is simply a `WHERE`, applied **before** the ranking. Two +filtered measures in one query are two scopes with one predicate each. + +## Filter routing + +| Filter phase | Where it is applied | +| --- | --- | +| ROW | the host base **and** the ranked CTE | +| AGGREGATE (references the ranked value) | the outer combined SELECT's `WHERE` | +| POST | the existing post-transform wrapper | + +A ROW-phase filter is duplicated rather than relocated: the CTE is LEFT JOINed +back, which propagates a VALUE but never an EXCLUSION, so a predicate applied +only in the CTE would silently become "blank out their measure" instead of +"exclude these rows". An AGGREGATE-phase one cannot be a `HAVING` inside the +CTE for the mirror-image reason — dropping the CTE row resurrects the host row +carrying NULL. + +## Cross-model and re-rooting + +A `first`/`last` whose source names another model roots its CTE at that target +(`RANKED_TARGET`). The forward-path filter routing is the cross-model planner's +decision table, taken verbatim; only the ranking column and the grain are +computed in the target's coordinates. + +A **re-rooted** cross-model `first`/`last` keeps its `CrossModelAggregatePlan`: +its ranked plan belongs to the nested sub-plan, in the sub-plan's own +coordinates. That sub-plan is rendered as a complete statement and spliced into +a CTE body, and SQL Server rejects a `WITH` nested inside a CTE definition — so +when a sub-plan's only isolated aggregate IS its answer, at its own grain, it is +emitted directly as the ranked SELECT rather than as `_base` plus a combined +SELECT around it (`_collapses_to_ranked_cte`). + +## Internal names + +`_rk_rn` (the rank column) and `_rk_src` (the inner subquery) are private to one +CTE scope and never reach a result key. They are safe precisely because the +inner SELECT projects a **named** list rather than `.*`: a physical +column called `_rk_rn` is never re-exported, so it cannot capture the rank +column's reference. CTE names are minted by the collision-aware allocator, which +case-folds on dialects whose unquoted identifiers do. diff --git a/docs/architecture/sql-generation.md b/docs/architecture/sql-generation.md index f978c2a3..c27ae942 100644 --- a/docs/architecture/sql-generation.md +++ b/docs/architecture/sql-generation.md @@ -117,6 +117,12 @@ filters route to the combined-SELECT outer `WHERE` (`Phase.POST`). `sum`/`avg` local measures only; other shapes raise at plan time (`_guard_windowed_measures` in `stage_planner.py`). +And one `_rk_*` CTE per `RankedAggregatePlan` — a `first`/`last` measure. Rooted +at the host or at the join target, it ranks its own rows and picks rank 1 per +grain, then joins back on that grain like the other two. See +[Ranked aggregates](ranked-aggregates.md). Any of the three plan kinds routes +the whole query through this renderer. + ### Frame bounds vs population filters (DEV-1732) `_src` inherits the host's ROW-phase filters **minus their frame bounds** — a diff --git a/slayer/engine/isolation.py b/slayer/engine/isolation.py index 96164fef..ffc033d5 100644 --- a/slayer/engine/isolation.py +++ b/slayer/engine/isolation.py @@ -56,11 +56,21 @@ class IsolationKind(str, Enum): #: inputs (a ``Column.filter``, the source's ``Column.sql``, an arg or a #: kwarg) cross a join and so need their own rows. HOST_ROOTED = "host_rooted" + #: Its own ``_rk_`` CTE rooted at the HOST: a ``first`` / ``last`` whose + #: rows live on the host but which needs its OWN ROW ORDERING. + RANKED_HOST = "ranked_host" + #: Its own ``_rk_`` CTE rooted at the TARGET the aggregate's source names. + RANKED_TARGET = "ranked_target" @property def needs_own_cte(self) -> bool: return self is not IsolationKind.NONE + @property + def is_ranked(self) -> bool: + """Whether this kind compiles to a ranked (``first``/``last``) CTE.""" + return self in (IsolationKind.RANKED_HOST, IsolationKind.RANKED_TARGET) + def may_inline_crossing_inputs(crossed_paths: Sequence[Tuple[str, ...]]) -> bool: # NOSONAR(S1172) — crossed_paths is the documented DEV-1688 seam; the cardinality-aware decision reads it, hardcoded False until then. """Whether an aggregate whose inputs cross ``crossed_paths`` may stay in the @@ -99,6 +109,26 @@ def classify_isolation( if not isinstance(key, AggregateKey): return IsolationKind.NONE + if key.agg in ("first", "last"): + # Its own ROW ORDERING is one of the three things P-C says an aggregate + # can need its own rows for, so a first/last isolates whatever its + # inputs do — the trigger the pre-B9 classifier had no case for, which + # is why one first/last used to wrap the whole host base in a ranking + # instead. + # + # Deliberately ahead of the crossing-input branch: a crossing first/last + # already isolated, and routing it to the ranked CTE rather than the + # generic one is what lets the ranked scope apply its own predicate and + # rank in one place. Deliberately ahead of ``disable_host_rooted_isolation`` + # too: that guard exists because a CROSSING aggregate re-appears in its + # own nested sub-plan and would isolate forever. A ranked plan is + # RENDERED, never re-planned, and inside a sub-plan it still needs its + # own row ordering — suppressing it there would leave the sub-plan's + # first/last with no ranking at all. + if getattr(key.source, "path", ()): + return IsolationKind.RANKED_TARGET + return IsolationKind.RANKED_HOST + if getattr(key.source, "path", ()): # The source names another model: the aggregate's rows live there — # UNLESS it is marked host-grain (DEV-1747 D2), which separates where a diff --git a/slayer/engine/planned.py b/slayer/engine/planned.py index 89f2ae27..98184c3a 100644 --- a/slayer/engine/planned.py +++ b/slayer/engine/planned.py @@ -68,6 +68,8 @@ "OrderEntry", "OrderScope", "PlannedQuery", + "RankedAggregatePlan", + "RankedGrainMember", "SlotId", "TransformLayer", "ValueSlot", @@ -328,6 +330,87 @@ class SrcFilterRewrite(BaseModel): expression: BoundExpr +# --------------------------------------------------------------------------- +# RankedAggregatePlan — DEV-1748 +# --------------------------------------------------------------------------- + + +class RankedGrainMember(BaseModel): + """One member of a ranked aggregate's grain, in BOTH coordinate systems. + + ``host_slot_id`` names the host row slot the CTE joins back to; + ``ranked_key`` is the same value anchored in the RANKED scope, which for a + target-rooted plan is a different expression against a different root. + + Carried as one list because the renderer derives two things from it that + must agree: the ``PARTITION BY`` the ranking runs over, and the join-back + predicate. Deriving them separately is how a partition and a grain drift + apart — and a partition COARSER than the grain silently returns more than + one row per group, which the LEFT JOIN then multiplies into the host. + """ + + model_config = ConfigDict(arbitrary_types_allowed=True) + + host_slot_id: SlotId + ranked_key: ValueKey + + +class RankedAggregatePlan(BaseModel): + """Plan for one ``first`` / ``last`` aggregate slot (DEV-1748, B9). + + A ranked aggregate needs its OWN ROW ORDERING, which is one of the three + things P-C says an aggregate can need its own rows for — so like a crossing + aggregate (``_cm_``) and a windowed one (``_wm_``) it compiles to a + plan-shaped CTE rooted where its rows live and joined back on the query + grain null-safely. The host base keeps only purely-local aggregates, so + adding a first/last cannot change host cardinality. + + It replaces a shape that did the opposite: ONE first/last anywhere wrapped + the entire host base in a ``ROW_NUMBER`` subquery, so every sibling + aggregate in the query was computed over the ranked row set, and several + rankings shared one scope — which is what the retired rn-suffix scheme + (``_last_rn_2``) and the filtered sentinel columns (``_last_rn_f0`` plus a + ``_match_f0`` flag consulted by alias) existed to disambiguate. One + aggregate per CTE removes the need for all of it. + + ``ranking_time_key`` is resolved at PLAN time (P-D) and anchored in the + ranked scope's coordinates; the renderer emits it and never re-derives the + precedence. A measure's ``Column.filter`` is not carried here — it lives on + the aggregate's own key, and the renderer applies it as a predicate inside + this CTE, which is what "filtered variants are plan data" means. + + Filter routing uses the same vocabulary, and the same audit-versus- + instruction distinction, as ``CrossModelAggregatePlan``: ``where`` and + ``having`` are instructions about where a filter is EVALUATED, while + ``applied_filter_ids`` only records that some scope evaluates it. + """ + + model_config = ConfigDict(arbitrary_types_allowed=True) + + aggregate_slot_id: SlotId + #: ``first`` ranks ascending, ``last`` descending. The direction is not a + #: separate field because it is not a separate decision. + agg: Literal["first", "last"] + #: The model the ranked rows come FROM — the host for a local aggregate, + #: the join target for a cross-model one. + root_model: str + datasource: str + #: The host-relative join path to ``root_model``; empty when host-rooted. + target_path: Tuple[str, ...] = () + join_chain: List[JoinRequirement] = Field(default_factory=list) + ranking_time_key: ValueKey + grain: List[RankedGrainMember] = Field(default_factory=list) + where_filter_ids: List[BoundFilterId] = Field(default_factory=list) + having_filter_ids: List[BoundFilterId] = Field(default_factory=list) + applied_filter_ids: List[BoundFilterId] = Field(default_factory=list) + target_model_filters: List[str] = Field(default_factory=list) + dropped_filter_warnings: List[UnreachableFilterDroppedWarning] = Field( + default_factory=list, + ) + public_alias: Optional[str] = None + hidden: bool = False + + # --------------------------------------------------------------------------- # TransformLayer # --------------------------------------------------------------------------- @@ -404,6 +487,8 @@ class OrderScope(str, Enum): CROSS_MODEL_CTE = "cross_model_cte" #: Lives in a windowed (``_wm_``) CTE. WINDOWED_CTE = "windowed_cte" + #: Lives in a ranked (``_rk_``) first/last CTE. + RANKED_CTE = "ranked_cte" #: Produced by a step of the transform chain. TRANSFORM_STEP = "transform_step" #: A composite whose operands span scopes, so it can only be evaluated in @@ -514,6 +599,13 @@ class PlannedQuery(BaseModel): aggregate_slots: List[ValueSlot] = Field(default_factory=list) cross_model_aggregate_plans: List[CrossModelAggregatePlan] = Field(default_factory=list) windowed_aggregate_plans: List["WindowedAggregatePlan"] = Field(default_factory=list) + # DEV-1748 (B9) — one entry per ``first`` / ``last`` aggregate slot. A + # sibling of ``windowed_aggregate_plans``: both name aggregates that need + # their own rows and therefore their own CTE, joined back on the query + # grain (P-C). A RE-ROOTED cross-model first/last is NOT here — it stays on + # ``CrossModelAggregatePlan``, whose nested sub-plan carries the ranked plan + # instead, in the sub-plan's own coordinate system. + ranked_aggregate_plans: List["RankedAggregatePlan"] = Field(default_factory=list) combined_expression_slots: List[ValueSlot] = Field(default_factory=list) transform_layers: List[TransformLayer] = Field(default_factory=list) filters_by_phase: List[FilterPhase] = Field(default_factory=list) diff --git a/slayer/engine/ranked_planner.py b/slayer/engine/ranked_planner.py new file mode 100644 index 00000000..55b0ba41 --- /dev/null +++ b/slayer/engine/ranked_planner.py @@ -0,0 +1,398 @@ +"""Plan-time construction of ``RankedAggregatePlan`` — the ``first`` / ``last`` +route (P-C / P-D). + +A ranked aggregate needs its own ROW ORDERING, so under P-C it compiles to a +plan-shaped CTE rooted where its rows live and joined back on the query grain. +Everything that decision needs is settled HERE — which column the ranking runs +over, what the grain is in the ranked scope's own coordinates, which filters the +CTE evaluates — so the renderer emits a plan rather than re-deriving one. + +The ranking column in particular used to be resolved at render time, twice: once +for the host base's ranked wrap and once, by a different precedence, inside the +cross-model CTE. They agreed by accident on the cases anyone had tried. One +resolver with an explicit per-scope precedence replaces both: + +* **host-rooted** — an explicit positional time arg, else the first + ``DATE``/``TIMESTAMP`` row dimension, else the first time dimension's RAW + column (never the truncated bucket: ranking within a month by the month ties + every row in it), else the model's ``default_time_dimension``. +* **target-rooted** — the same explicit arg, re-anchored in the target's + coordinates, else the TARGET model's ``default_time_dimension``. Host + dimensions are deliberately not candidates: the CTE ranks target rows, and a + host column is not in scope there. + +Both ends of the precedence raise rather than fall through, with the message the +scope's users already see. +""" + +from __future__ import annotations + +from typing import Any, List, Optional, Sequence, Tuple + +from slayer.core.enums import DataType +from slayer.core.keys import ( + AggregateKey, + ColumnKey, + ColumnSqlKey, + TimeTruncKey, + ValueKey, + column_path, + reroot_value_key, +) +from slayer.core.models import SlayerModel +from slayer.engine.planned import ( + RankedAggregatePlan, + RankedGrainMember, + SlotId, + ValueSlot, +) +from slayer.engine.source_bundle import ResolvedSourceBundle + +__all__ = [ + "RANKED_AGGREGATIONS", + "build_host_ranked_plan", + "build_target_ranked_plan", + "explicit_ranking_time_arg", + "resolve_ranking_time_key", +] + +#: The aggregations that rank. Named once so the classifier, the planner and +#: the renderer cannot drift on what "a ranked aggregate" is. +RANKED_AGGREGATIONS = ("first", "last") + +_TEMPORAL_TYPES = (DataType.DATE, DataType.TIMESTAMP) + + +def explicit_ranking_time_arg(key: AggregateKey) -> Optional[ValueKey]: + """The explicit positional ranking-time arg of a ``first`` / ``last``, or + ``None``. + + The FIRST positional arg iff it is a column ref; ``None`` for anything else + — first/last never takes a leading non-column positional, so a different + shape means the caller passed something this aggregation does not read. + """ + if key.agg not in RANKED_AGGREGATIONS: + return None + for arg in key.args: + return arg if isinstance(arg, (ColumnKey, ColumnSqlKey)) else None + return None + + +def _ranking_key_name(key: ValueKey) -> str: + """The user-facing name of a ranking-time key, for an error message.""" + if isinstance(key, ColumnKey): + return ".".join((*key.path, key.leaf)) + if isinstance(key, ColumnSqlKey): + return ".".join((*key.path, key.column_name)) + return type(key).__name__ + + +def _resolves_on( + *, key: ValueKey, model: SlayerModel, bundle: ResolvedSourceBundle, +) -> bool: + """Whether ``key`` names something reachable FROM ``model``. + + A shallow check on purpose: a local leaf must be a column of the model, and + a path-bearing one must start at a model this one joins to. Walking the + whole chain is the renderer's job and it raises its own errors; this exists + so the common mistake — naming a HOST column as a TARGET-rooted ranking key + — is caught where the plan is made rather than at the database. + """ + if isinstance(key, ColumnKey): + leaf, path = key.leaf, key.path + elif isinstance(key, ColumnSqlKey): + leaf, path = key.column_name, key.path + else: + return True + if path: + return any(j.target_model == path[0] for j in (model.joins or [])) + return any(c.name == leaf for c in model.columns) + + +def _temporal_row_dimension_key( + *, + row_keys: Sequence[ValueKey], + source_model: SlayerModel, + bundle: ResolvedSourceBundle, +) -> Optional[ValueKey]: + """The first row dimension whose declared type is ``DATE``/``TIMESTAMP``.""" + for key in row_keys: + if not isinstance(key, ColumnKey): + continue + model: Optional[SlayerModel] = source_model + for hop in key.path: + model = bundle.get_referenced_model(hop) + if model is None: + break + if model is None: + continue + col = next((c for c in model.columns if c.name == key.leaf), None) + if col is not None and col.type in _TEMPORAL_TYPES: + return key + return None + + +def _time_dimension_raw_column( + *, row_keys: Sequence[ValueKey], +) -> Optional[ValueKey]: + """The first time dimension's RAW column, un-truncated.""" + for key in row_keys: + if isinstance(key, TimeTruncKey): + return key.column + return None + + +def resolve_ranking_time_key( + *, + key: AggregateKey, + root_model: SlayerModel, + bundle: ResolvedSourceBundle, + row_keys: Sequence[ValueKey] = (), + target_path: Tuple[str, ...] = (), +) -> ValueKey: + """The column a ranked aggregate's ``ROW_NUMBER`` orders by, in the RANKED + scope's coordinates. + + ``row_keys`` are the host's row-dimension keys in render order; they are + candidates only for a HOST-rooted plan (``target_path`` empty). A + target-rooted CTE ranks the target's rows, so a host dimension is not in + scope there and the precedence goes straight from the explicit arg to the + target's own default. + """ + arg = explicit_ranking_time_arg(key) + if arg is not None: + # The arg arrives in the HOST's coordinates; a target-rooted plan needs + # it in the target's, which is the same re-anchoring the aggregate + # source itself undergoes — done in lockstep so the ORDER BY and the + # value cannot end up rooted at different relations. + rerooted = reroot_value_key(arg, target_path=target_path) + if target_path and not _resolves_on( + key=rerooted, model=root_model, bundle=bundle, + ): + # A HOST column as the ranking key of a TARGET-rooted CTE. The rows + # being ranked are the target's, and a host column is not one of + # their attributes — the relationship runs the other way, usually + # one-to-many, so there is no single host value per target row to + # rank by. This used to emit ``ORDER BY .``, + # a reference to a column that does not exist, and fail at the + # database with no indication of which measure caused it. + raise ValueError( + f"first/last ranking column " + f"{_ranking_key_name(rerooted)!r} is not resolvable on model " + f"{root_model.name!r}, where a cross-model first/last ranks " + f"its rows. Name a column of {root_model.name!r} (or one it " + f"joins to), or drop the argument to rank by its " + f"default_time_dimension." + ) + return rerooted + + if not target_path: + temporal = _temporal_row_dimension_key( + row_keys=row_keys, source_model=root_model, bundle=bundle, + ) + if temporal is not None: + return temporal + raw = _time_dimension_raw_column(row_keys=row_keys) + if raw is not None: + return raw + if root_model.default_time_dimension: + return ColumnKey(path=(), leaf=root_model.default_time_dimension) + raise ValueError( + "first/last aggregation requires a ranking time column " + "(a time_dimension, a DATE/TIMESTAMP dimension, or the " + "model's default_time_dimension); none is resolvable for " + f"model {root_model.name!r}." + ) + + if root_model.default_time_dimension: + return ColumnKey(path=(), leaf=root_model.default_time_dimension) + raise ValueError( + f"first/last aggregation requires a ranking time column " + f"(an explicit positional time arg, or the target " + f"model's default_time_dimension); none is resolvable " + f"for cross-model aggregate on target " + f"{root_model.name!r}." + ) + + +def ordered_row_keys( + *, row_slots: Sequence[ValueSlot], public_projection: Sequence[SlotId], +) -> List[ValueKey]: + """Row-dimension keys in the order the base SELECT renders them. + + Publicly projected slots first, in projection order, then the rest. That is + the order the superseded render-time resolver walked, and the ranking-column + precedence is order-sensitive — two temporal dimensions rank by whichever + comes first. + """ + by_id = {s.id: s for s in row_slots} + seen: set = set() + ordered: List[ValueKey] = [] + for sid in public_projection: + slot = by_id.get(sid) + if slot is not None and sid not in seen: + seen.add(sid) + ordered.append(slot.key) + for slot in row_slots: + if slot.id not in seen: + seen.add(slot.id) + ordered.append(slot.key) + return ordered + + +def _host_grain(*, row_slots: Sequence[ValueSlot]) -> List[RankedGrainMember]: + """The host grain: every VISIBLE row slot, in one coordinate system. + + Hidden row slots are excluded for the same reason the windowed plan excludes + them — they are filter/order scaffolding the host base does not group by, so + partitioning the ranking over one would split groups the result never has. + """ + return [ + RankedGrainMember(host_slot_id=slot.id, ranked_key=slot.key) + for slot in row_slots + if not slot.hidden + ] + + +def _target_grain( + *, + row_slots: Sequence[ValueSlot], + shared_grain_slots: Sequence[SlotId], + public_projection: Sequence[SlotId], + target_path: Tuple[str, ...], +) -> List[RankedGrainMember]: + """The grain a TARGET-rooted ranked CTE shares with the host. + + Only host dimensions that lie ON the target's join path can be expressed in + the target's coordinates at all; the rest broadcast (the CTE is scalar and + CROSS JOINed), which is the forward cross-model path's existing semantics. + Re-rooting is what makes the rest reachable, and that route keeps its + ``CrossModelAggregatePlan``. + """ + by_id = {s.id: s for s in row_slots} + projected = set(public_projection) + members: List[RankedGrainMember] = [] + for sid in shared_grain_slots: + slot = by_id.get(sid) + if slot is None or slot.hidden or sid not in projected: + continue + path = _row_key_path(slot.key) + if not path or path != target_path: + # Empty: host-local, broadcast. Non-terminal / off-branch: not this + # target's grain. Both are the forward path's existing behaviour. + continue + members.append(RankedGrainMember( + host_slot_id=sid, + ranked_key=reroot_value_key(slot.key, target_path=target_path), + )) + return members + + +def _row_key_path(key: ValueKey) -> Tuple[str, ...]: + if isinstance(key, (ColumnKey, ColumnSqlKey)): + return key.path + if isinstance(key, TimeTruncKey): + return column_path(key.column) + return () + + +def build_host_ranked_plan( + *, + slot: ValueSlot, + row_slots: Sequence[ValueSlot], + public_projection: Sequence[SlotId], + source_model: SlayerModel, + bundle: ResolvedSourceBundle, + where_filter_ids: Sequence[str] = (), +) -> RankedAggregatePlan: + """One host-rooted ranked plan. + + ``where_filter_ids`` are the ROW-phase host filters (user filters AND the + model's own ``filters``) the CTE ALSO evaluates. They are duplicated rather + than relocated: a LEFT JOIN back propagates a value but never an exclusion, + so a row filter that only reached the CTE would silently become "blank out + their measure" instead of "exclude these rows" (the PR-4 B6 ruling, one + route over). + """ + key = slot.key + assert isinstance(key, AggregateKey) + return RankedAggregatePlan( + aggregate_slot_id=slot.id, + agg=key.agg, + root_model=source_model.name, + datasource=source_model.data_source, + target_path=(), + join_chain=[], + ranking_time_key=resolve_ranking_time_key( + key=key, + root_model=source_model, + bundle=bundle, + row_keys=ordered_row_keys( + row_slots=row_slots, public_projection=public_projection, + ), + ), + grain=_host_grain(row_slots=row_slots), + # The model's own ``filters`` are ROW-phase entries of + # ``filters_by_phase`` like any other, so they arrive here by id rather + # than as text — unlike the target-rooted case, where the TARGET's + # filters are not in the host's filter list at all. + where_filter_ids=list(where_filter_ids), + applied_filter_ids=list(where_filter_ids), + hidden=slot.hidden, + public_alias=None if slot.hidden else slot.public_name, + ) + + +def build_target_ranked_plan( + *, + slot: ValueSlot, + cross_model_plan: Any, + row_slots: Sequence[ValueSlot], + public_projection: Sequence[SlotId], + bundle: ResolvedSourceBundle, +) -> RankedAggregatePlan: + """One target-rooted ranked plan, re-shaped from the forward cross-model + plan the same strategy produced. + + The routing decisions — which host filters this CTE evaluates as WHERE, as + HAVING, which are unreachable — are the cross-model planner's decision table + and are taken verbatim. Only the two things that are genuinely about RANKING + are computed here: the ranking column in the target's coordinates, and the + grain in the same. + """ + key = slot.key + assert isinstance(key, AggregateKey) + target_path = tuple(getattr(key.source, "path", ())) + target_model = bundle.get_referenced_model(cross_model_plan.target_model) + if target_model is None: + raise ValueError( + f"Ranked cross-model target {cross_model_plan.target_model!r} is " + f"not in the resolved source bundle.", + ) + return RankedAggregatePlan( + aggregate_slot_id=slot.id, + agg=key.agg, + root_model=target_model.name, + datasource=cross_model_plan.datasource, + target_path=target_path, + join_chain=list(cross_model_plan.join_chain), + ranking_time_key=resolve_ranking_time_key( + key=key, + root_model=target_model, + bundle=bundle, + target_path=target_path, + ), + grain=_target_grain( + row_slots=row_slots, + shared_grain_slots=cross_model_plan.shared_grain_slots, + public_projection=public_projection, + target_path=target_path, + ), + where_filter_ids=list(cross_model_plan.where_filter_ids), + having_filter_ids=list(cross_model_plan.having_filter_ids), + applied_filter_ids=list(cross_model_plan.applied_filter_ids), + target_model_filters=list(cross_model_plan.target_model_filters), + dropped_filter_warnings=list(cross_model_plan.dropped_filter_warnings), + hidden=cross_model_plan.hidden, + public_alias=cross_model_plan.public_alias, + ) diff --git a/slayer/engine/stage_planner.py b/slayer/engine/stage_planner.py index 16e9a201..b2d6b5f5 100644 --- a/slayer/engine/stage_planner.py +++ b/slayer/engine/stage_planner.py @@ -25,7 +25,7 @@ from __future__ import annotations -from typing import Dict, FrozenSet, List, Optional, Set, Tuple, Union +from typing import AbstractSet, Dict, FrozenSet, List, Optional, Set, Tuple, Union from slayer.core.enums import DataType from slayer.core.format import NumberFormat @@ -90,12 +90,17 @@ OrderEntry, OrderScope, PlannedQuery, + RankedAggregatePlan, SlotId, SrcFilterRewrite, TransformLayer, ValueSlot, WindowedAggregatePlan, ) +from slayer.engine.ranked_planner import ( + build_host_ranked_plan, + build_target_ranked_plan, +) from slayer.engine.planning import ( DeclaredMeasure, OrderSpec, @@ -1501,6 +1506,14 @@ def _windowed_phase(bf: BoundFilter) -> Phase: )) cross_model_plans = [] + ranked_plans: List[RankedAggregatePlan] = [] + # ROW-phase filters (user filters AND the model's own ``filters``) are what + # a ranked CTE re-evaluates over its own rows. Computed once: every ranked + # plan inherits the same list, and the host base keeps applying them too + # (B6 — a LEFT JOIN back propagates a value, never an exclusion). + row_phase_filter_ids: List[BoundFilterId] = [ + fp.id for fp in filters_by_phase if fp.phase == Phase.ROW + ] host_slots_for_classifier = projection.registry.slots for slot in agg_slots: # ONE trigger decision (P-C / DEV-1688 seam). The windowed skip, the @@ -1516,6 +1529,18 @@ def _windowed_phase(bf: BoundFilter) -> Phase: ) if kind in (IsolationKind.NONE, IsolationKind.WINDOWED): continue + if kind is IsolationKind.RANKED_HOST: + # Its rows are the host's; nothing about the cross-model decision + # table applies, so it never reaches the strategy. + ranked_plans.append(build_host_ranked_plan( + slot=slot, + row_slots=row_slots, + public_projection=projection.public_projection, + source_model=reachability_anchor_model, + bundle=bundle, + where_filter_ids=row_phase_filter_ids, + )) + continue key = slot.key # DEV-1450 #2: re-rooting (C1) is owned by the strategy. We hand it # the host query, the public projection, and a sub-plan builder so it @@ -1559,6 +1584,25 @@ def _windowed_phase(bf: BoundFilter) -> Phase: if reroot_enabled else None ), ) + if kind is IsolationKind.RANKED_TARGET and plan.rerooted_plan is None: + # D1: the FORWARD cross-model first/last becomes a ranked plan — + # same CTE, rooted at the target, but ranking rather than a plain + # aggregation. A RE-ROOTED one keeps its ``CrossModelAggregatePlan``: + # its ranked plan belongs to the nested sub-plan, in the sub-plan's + # own coordinate system, and is built by that recursion. + # + # The strategy still runs either way. Re-rooting is its decision to + # make, and the routing it produces on the forward path — which host + # filter this CTE evaluates as WHERE, as HAVING, which is + # unreachable — is the same decision table a ranked CTE needs. + ranked_plans.append(build_target_ranked_plan( + slot=slot, + cross_model_plan=plan, + row_slots=row_slots, + public_projection=projection.public_projection, + bundle=bundle, + )) + continue cross_model_plans.append(plan) order_entries = [] @@ -1594,6 +1638,7 @@ def _windowed_phase(bf: BoundFilter) -> Phase: cross_model_slot_ids={ p.aggregate_slot_id for p in cross_model_plans }, + ranked_slot_ids={p.aggregate_slot_id for p in ranked_plans}, windowed_slot_ids=set(windowed_slot_ids), public_projection=projection.public_projection, slot_by_key={ @@ -1639,6 +1684,7 @@ def _windowed_phase(bf: BoundFilter) -> Phase: outer_where_filter_ids = _plan_outer_where_filters( filters_by_phase=filters_by_phase, cross_model_plans=cross_model_plans, + ranked_plans=ranked_plans, slots=[*row_slots, *agg_slots, *combined_slots], ) @@ -1647,6 +1693,7 @@ def _windowed_phase(bf: BoundFilter) -> Phase: agg_slots=agg_slots, cross_model_plans=cross_model_plans, windowed_plans=windowed_plans, + ranked_plans=ranked_plans, order_entries=order_entries, filters_by_phase=filters_by_phase, outer_where_filter_ids=outer_where_filter_ids, @@ -1658,6 +1705,7 @@ def _windowed_phase(bf: BoundFilter) -> Phase: aggregate_slots=agg_slots, cross_model_aggregate_plans=cross_model_plans, windowed_aggregate_plans=windowed_plans, + ranked_aggregate_plans=ranked_plans, combined_expression_slots=combined_slots, transform_layers=transform_layers, filters_by_phase=filters_by_phase, @@ -1685,6 +1733,7 @@ def _plan_empty_base_grain( order_entries: list, filters_by_phase: list, outer_where_filter_ids: List[BoundFilterId], + ranked_plans: Optional[list] = None, ) -> "EmptyBaseGrainPlan | None": """Decide the DEV-1503 empty-base spine at plan time (§5.12). @@ -1700,6 +1749,7 @@ def _plan_empty_base_grain( """ isolated = {p.aggregate_slot_id for p in cross_model_plans} isolated |= {p.aggregate_slot_id for p in windowed_plans} + isolated |= {p.aggregate_slot_id for p in (ranked_plans or [])} if not projection or any(sid not in isolated for sid in projection): return None # A host-LOCAL aggregate would have to be computed in ``_base``, which then @@ -1729,15 +1779,17 @@ def _plan_outer_where_filters( filters_by_phase: List[FilterPhase], cross_model_plans: List[CrossModelAggregatePlan], slots: List[ValueSlot], + ranked_plans: Optional[List["RankedAggregatePlan"]] = None, ) -> List[BoundFilterId]: """AGGREGATE-phase filters that must be applied on the OUTER combined - SELECT instead of as HAVING inside a ``_cm_*`` CTE (DEV-1503). + SELECT instead of as HAVING inside a ``_cm_*`` / ``_rk_*`` CTE (DEV-1503). A filter qualifies when its value-key tree references an aggregate that was - isolated into a CTE with its own root (``cte_root_model is not None``). - That CTE LEFT JOINs back to ``_base``, so a HAVING inside it drops CTE rows - and the join then resurfaces the host row carrying a NULL aggregate. The - same predicate on the outer, non-aggregating SELECT drops the row. + isolated into a CTE with its own root (``cte_root_model is not None``), or + into a ranked one — every ranked CTE has its own root by construction. That + CTE LEFT JOINs back to ``_base``, so a HAVING inside it drops CTE rows and + the join then resurfaces the host row carrying a NULL aggregate. The same + predicate on the outer, non-aggregating SELECT drops the row. Returned in ``filters_by_phase`` order so the emitted WHERE conjunct order is stable. @@ -1747,6 +1799,9 @@ def _plan_outer_where_filters( for p in cross_model_plans if p.cte_root_model is not None } + isolated_agg_slot_ids |= { + p.aggregate_slot_id for p in (ranked_plans or []) + } if not isolated_agg_slot_ids: return [] slot_by_key = {s.key: s for s in slots} @@ -2427,6 +2482,7 @@ def _classify_order_scope( windowed_slot_ids: Set[SlotId], public_projection: List[SlotId], slot_by_key: Dict[ValueKey, SlotId], + ranked_slot_ids: AbstractSet[SlotId] = frozenset(), ) -> OrderScope: """Name the scope that PRODUCES ``slot``'s value (DEV-1747 §5.10). @@ -2441,6 +2497,8 @@ def _classify_order_scope( """ if slot.id in cross_model_slot_ids: return OrderScope.CROSS_MODEL_CTE + if slot.id in ranked_slot_ids: + return OrderScope.RANKED_CTE if slot.id in windowed_slot_ids: return OrderScope.WINDOWED_CTE if isinstance(slot.key, TransformKey): @@ -2450,7 +2508,11 @@ def _classify_order_scope( if not isinstance(dep, AggregateKey): continue dep_sid = slot_by_key.get(dep) - if dep_sid in cross_model_slot_ids or dep_sid in windowed_slot_ids: + if ( + dep_sid in cross_model_slot_ids + or dep_sid in windowed_slot_ids + or dep_sid in ranked_slot_ids + ): return OrderScope.OUTER_COMPOSITE if slot.hidden or slot.id not in public_projection: return OrderScope.HOST_BASE_HIDDEN diff --git a/slayer/sql/generator.py b/slayer/sql/generator.py index 0c73364d..3ebe657b 100644 --- a/slayer/sql/generator.py +++ b/slayer/sql/generator.py @@ -26,7 +26,12 @@ from pydantic import BaseModel, ConfigDict, field_validator from slayer.core.errors import AggregationNotAllowedError, UnresolvableOrderColumnError -from slayer.core.keys import _FrozenKey, _reroot_path_ref, reroot_aggregate_key +from slayer.core.keys import ( + _FrozenKey, + _reroot_path_ref, + column_path, + reroot_aggregate_key, +) from slayer.core.models import Aggregation from slayer.core.refs import agg_kwarg_canonical_str from slayer.core.time_bounds import strip_frame_bounds @@ -66,6 +71,14 @@ OrderEnv, resolve_order_term, ) +from slayer.sql.render.ranked import ( + RANKED_CTE_PREFIX, + RankedGrainProjection, + build_rank_column, + build_ranked_cte_select, + build_ranked_pick, + ranked_ordered, +) from slayer.sql.render.value_expr import ( render_arithmetic, render_scalar_call, @@ -311,6 +324,64 @@ def _render_scalar_literal(v: Any) -> exp.Expression: return exp.Literal.string(str(v)) +def _strip_declared_cast(expr: exp.Expression) -> exp.Expression: + """Unwrap one declared-type ``CAST`` a derived-column expansion added. + + Used for a ranked aggregate's ORDER BY column. The CAST exists to make a + PROJECTED value match its declared type; an ordering key is compared only + to itself, and on SQLite the cast is not merely redundant — ``TIMESTAMP`` + carries numeric affinity, so it truncates every date to its year and ties + the partition. + """ + return expr.this if isinstance(expr, exp.Cast) else expr + + +def _collapses_to_ranked_cte(planned_query) -> bool: + """Whether this whole plan IS one ranked CTE (DEV-1748 D9). + + True when the ONLY thing the plan computes is one ranked aggregate at + exactly the grain the plan groups by, with nothing layered on top: no other + isolated aggregate, no host-local aggregate that would need a ``_base`` of + its own, no combined expression, no transform, no outer WHERE, no + pagination, and a projection that is precisely the grain plus the aggregate. + + Under those conditions the ranked CTE's body already produces the plan's + rows under the plan's names, so emitting ``_base`` + a combined SELECT + around it adds a ``WITH`` and nothing else. Every clause here is a case + where it would add something more, and the collapse would silently drop it. + """ + plans = planned_query.ranked_aggregate_plans + if len(plans) != 1: + return False + if ( + planned_query.cross_model_aggregate_plans + or planned_query.windowed_aggregate_plans + or planned_query.combined_expression_slots + or planned_query.transform_layers + or planned_query.outer_where_filter_ids + or planned_query.order + or planned_query.limit is not None + or planned_query.offset is not None + ): + return False + plan = plans[0] + if plan.hidden or plan.having_filter_ids: + return False + if len(planned_query.aggregate_slots) != 1: + return False + grain_ids = [m.host_slot_id for m in plan.grain] + visible_row_ids = [s.id for s in planned_query.row_slots if not s.hidden] + if grain_ids != visible_row_ids: + return False + if any(s.hidden for s in planned_query.row_slots): + # A hidden row slot is filter/order scaffolding ``_base`` would still + # have to materialise; the ranked CTE does not project it. + return False + return list(planned_query.projection) == [ + *grain_ids, plan.aggregate_slot_id, + ] + + def _wrap_cast_for_type(expr: exp.Expression, dt: Optional[DataType]) -> exp.Expression: """DEV-1361: wrap ``expr`` in ``CAST(expr AS )`` so the declared SLayer ``DataType`` is enforced in emitted SQL. @@ -1633,7 +1704,9 @@ def _build_stat_agg(self, spec: AggRenderSpec) -> exp.Expression: # so silent parity drift is impossible. # ====================================================================== - def generate_from_planned(self, planned_query, *, bundle) -> str: + def generate_from_planned( + self, planned_query, *, bundle, as_cte_body: bool = False, + ) -> str: """Render a typed ``PlannedQuery`` to SQL (public entry). DEV-1708 (D-E): installs a fresh generation-wide ``AliasAllocator`` for @@ -1643,13 +1716,19 @@ def generate_from_planned(self, planned_query, *, bundle) -> str: rerooted sub-generation (``_render_rerooted_cross_model_cte`` → ``generate_from_planned``) is a self-contained statement and gets its own allocator, with the parent's restored afterwards. + + ``as_cte_body`` says the result is about to become a CTE DEFINITION + rather than a statement, which forbids a ``WITH`` of its own (SQL Server + rejects a nested one outright). Only the caller knows that, so only the + caller can say it — see :func:`_collapses_to_ranked_cte` for the one + shape that currently needs it. """ self._assert_projection_is_public(planned_query) prev_allocator = getattr(self, "_gen_allocator", None) self._gen_allocator = self._new_allocator() try: return self._generate_from_planned_impl( - planned_query, bundle=bundle, + planned_query, bundle=bundle, as_cte_body=as_cte_body, ) finally: self._gen_allocator = prev_allocator @@ -1691,6 +1770,7 @@ def _generate_from_planned_impl( # NOSONAR(S3776) — top-level dispatch over c planned_query, *, bundle, + as_cte_body: bool = False, ) -> str: """Render a typed ``PlannedQuery`` to SQL. @@ -1727,9 +1807,26 @@ def _generate_from_planned_impl( # NOSONAR(S3776) — top-level dispatch over c ) source_relation = planned_query.source_relation + if as_cte_body and _collapses_to_ranked_cte(planned_query): + # D9. A re-rooted cross-model CTE renders its sub-plan as a COMPLETE + # statement and splices it into a CTE body, so a sub-plan that + # emitted ``_base`` plus a combined SELECT would put a ``WITH`` + # inside a CTE — which SQL Server rejects outright. It never + # happened before because no sub-plan ever contained an isolated + # aggregate; a re-rooted first/last is the first one that does. + # + # It also never needs to: when the sub-plan's only isolated + # aggregate IS its answer, at its own grain, the ranked CTE's body + # and the statement the long way round would produce are the same + # rows under the same names. So emit it directly. + return self._render_collapsed_ranked_plan( + planned_query=planned_query, bundle=bundle, + ) + if ( planned_query.cross_model_aggregate_plans or planned_query.windowed_aggregate_plans + or planned_query.ranked_aggregate_plans ): return self._render_with_cross_model_plans( planned_query=planned_query, bundle=bundle, @@ -3433,6 +3530,32 @@ def _build_filtered_rn_columns( filtered_match_map[m.alias] = match_alias return rn_exprs, filtered_rn_map, filtered_match_map + # ===================================================================== + # SUPERSEDED by ``RankedAggregatePlan`` (DEV-1748 B9) — production- + # UNREFERENCED, deleted in PR 6 (P-J: superseded code stops being reachable + # in the PR that replaces it, and is removed in the one that closes the + # series, so neither change has to be reviewed alongside the other). + # + # Every first/last aggregate is now excluded from ``base_render_order`` by + # ``isolated_slot_ids``, so ``_has_first_last_aggregate`` cannot return True + # and nothing below is entered. A runtime probe over the whole unit suite + # confirms it: the only callers left are the unit tests that pin these + # helpers directly. + # + # The inventory PR 6 deletes: + # _build_first_last_base_select, _has_first_last_aggregate, + # _build_ranked_subquery_from_planned, _build_unfiltered_rn_columns, + # _build_filtered_rn_columns, _iter_first_last_leaves, + # _resolve_ranking_time_column_from_planned, _resolve_explicit_time_col, + # _explicit_time_arg_of, FirstLastRenderState (+ its rn_suffix_map / + # filtered_rn_map / filtered_match_map / agg_synth_alias / + # value_alias_by_sql fields), ``_build_agg``'s first/last branch and its + # rn_suffix_map / filtered_* parameters, the rn threading through + # ``_render_aggregate_composite_expr``, and + # ``_resolve_agg_inputs_via_scope``'s ``_resolve_first_last_time_arg`` + # sub-pass. + # ===================================================================== + def _build_first_last_base_select( # NOSONAR(S3776) — single conceptual unit: dimension/td/derived-dim classification pass + agg-spec synth + ranked-subquery wrap + outer SELECT/GROUP BY assembly. Splitting forces shared mutable state (partition_exprs / extra_projections / outer_ref_by_sid / synth_by_sid) across helpers without simplifying anything. self, *, @@ -4305,6 +4428,366 @@ def _alias_of(sid: str) -> str: # text as a multi-part reference on BigQuery. return outer, grain_aliases + def _ranked_scope_expr( + self, + *, + key, + root_model, + root_relation: str, + bundle, + scope: ScopeFrame, + cast_derived: bool = True, + ) -> exp.Expression: + """One value expression anchored in a ranked CTE's own scope. + + Built through the SAME helpers the host ``_base`` uses, which is what + makes a grain member compare equal to the ``_base`` column it joins back + to: two spellings of "the same dimension" that differ only in a CAST + stop being the same value the moment a dialect rounds them differently. + + ``cast_derived=False`` for the RANKING column, which is compared only to + itself and so needs no agreement with anything. It also must not carry + the declared-type CAST: SQLite's ``TIMESTAMP`` has numeric affinity, so + ``CAST(DATE(created_at) AS TIMESTAMP)`` truncates every date to its year + and ties the whole partition. + + Registering the joins the expression crosses into ``scope`` is a side + effect here rather than a separate pass (Law 1), so the CTE's FROM can + never be missing one. + """ + from slayer.core.enums import TimeGranularity + from slayer.core.keys import ColumnKey, ColumnSqlKey, TimeTruncKey + + def _register(expr: exp.Expression, path: Tuple[str, ...]) -> None: + if path: + scope.join_paths.add(path) + for p in self._joined_paths_in_sql( + sql_expr=expr, source_relation=root_relation, + source_model=root_model, bundle=bundle, + ): + scope.join_paths.add(p) + + if isinstance(key, TimeTruncKey): + raw = self._raw_time_col_expr_for_planned( + time_column=key.column, source_model=root_model, + source_relation=root_relation, bundle=bundle, + ) + _register(raw, column_path(key.column)) + return self._build_date_trunc( + col_expr=raw, granularity=TimeGranularity(key.granularity), + ) + if isinstance(key, ColumnKey): + expr = self._joined_or_local_dim_expr( + path=key.path, leaf=key.leaf, source_model=root_model, + source_relation=root_relation, bundle=bundle, + ) + _register(expr, key.path) + return expr + if isinstance(key, ColumnSqlKey): + expr = self._derived_column_expr( + key=key, source_model=root_model, + source_relation=root_relation, bundle=bundle, + ) + if expr is None: + raise ValueError( + f"Derived column {key.column_name!r} on model " + f"{key.path[-1] if key.path else root_model.name!r} is not " + f"in the resolved source bundle.", + ) + _register(expr, key.path) + return expr if cast_derived else _strip_declared_cast(expr) + raise NotImplementedError( + f"Ranked CTE cannot anchor a {type(key).__name__} — the grain and " + f"the ranking column are columns, truncated time columns, or " + f"derived columns.", + ) + + def _ranked_value_expr( + self, *, key, root_model, root_relation: str, bundle, scope: ScopeFrame, + ) -> exp.Expression: + """The value a ranked aggregate picks, anchored in its own scope. + + Deliberately NOT ``_build_agg_render_spec_from_planned``: that builder + resolves an explicit time arg on the way past, and the ranking column is + plan data now (``RankedAggregatePlan.ranking_time_key``). Going through + it would re-derive at render time the one thing the plan exists to + decide — and would keep the residual-path raise alive on a path that no + longer has the limitation it describes. + """ + from slayer.core.keys import ColumnKey, ColumnSqlKey, StarKey + + source = key.source + if isinstance(source, StarKey): + raise ValueError( + f"Aggregation {key.agg!r} not allowed with measure " + f"'*' — use '*:count' for COUNT(*)." + ) + if not isinstance(source, (ColumnKey, ColumnSqlKey)): + raise NotImplementedError( + f"AggregateKey source {type(source).__name__} not supported.", + ) + leaf = ( + source.leaf if isinstance(source, ColumnKey) else source.column_name + ) + col = next((c for c in root_model.columns if c.name == leaf), None) + if col is None: + raise ValueError( + f"Aggregate source column {leaf!r} not found on model " + f"{root_model.name!r}", + ) + if isinstance(source, ColumnSqlKey) and col.sql is not None: + sql_text = self._expand_derived_column_sql( + source_model=root_model, source_relation=root_relation, + column_name=col.name, bundle=bundle, + ) + else: + sql_text = col.sql if col.sql else col.name + expr = self._resolve_sql( + sql=sql_text, name=col.name, model_name=root_relation, + type=col.type, + ) + for p in self._joined_paths_in_sql( + sql_expr=expr, source_relation=root_relation, + source_model=root_model, bundle=bundle, + ): + scope.join_paths.add(p) + return expr + + def _render_ranked_cte_from_planned( + self, + *, + plan, + agg_slot, + bundle, + planned_query, + slots_by_id: Dict[str, Any], + host_source_model, + host_source_relation: str, + full_agg_alias: str, + ) -> Tuple[exp.Select, List[str]]: + """Render one ``_rk_`` ranked (``first`` / ``last``) CTE (DEV-1748, B9). + + Two SELECTs: an inner one that projects the grain, the value and one + ``ROW_NUMBER`` over the rows this aggregate is allowed to see, and an + outer one that picks rank 1 per grain. Returns ``(cte_query, + grain_aliases)`` — the aliases the caller joins back on. + + The whole aggregate lives here, so the host base is untouched: adding a + ``first`` to a query cannot change what its siblings compute, and the + rn-suffix scheme that used to disambiguate several rankings sharing one + scope has nothing left to disambiguate. + """ + from slayer.core.keys import AggregateKey, reroot_aggregate_key + + key = agg_slot.key + if not isinstance(key, AggregateKey): + raise RuntimeError( + f"RankedAggregatePlan {plan.aggregate_slot_id!r} references a " + f"non-aggregate slot.", + ) + + if plan.target_path: + root_model = bundle.get_referenced_model(plan.root_model) + if root_model is None: + raise ValueError( + f"Ranked CTE root {plan.root_model!r} is not in the " + f"resolved source bundle.", + ) + root_relation = plan.root_model + else: + root_model = host_source_model + root_relation = host_source_relation + + allocator = self._gen_allocator or self._new_allocator() + self._reserve_model_column_names(allocator, root_model) + + def _frame() -> ScopeFrame: + return ScopeFrame( + scope_id=allocator.next_scope_id(root_relation), + root_model=root_model, + root_relation=root_relation, + bundle=bundle, + dialect=self._dialect, + allocator=allocator, + ) + + # Law 2's producer/consumer pair. The inner scope PRODUCES every value + # the outer one reads, so a value that is both the grain and the ranked + # expression is materialised once — the two used to keep separate alias + # maps and project it twice. + ranked_scope = _frame() + cte_scope = _frame() + + # The aggregate in the ranked scope's coordinates. For a target-rooted + # plan that means stripping the host prefix from the source (and from + # every embedded ref) in one pass, exactly as the cross-model CTE does. + local_key = reroot_aggregate_key(key, target_path=plan.target_path) + + grain: List[RankedGrainProjection] = [] + partition_by: List[exp.Expression] = [] + for member in plan.grain: + host_slot = slots_by_id.get(member.host_slot_id) + if host_slot is None: + raise RuntimeError( + f"RankedAggregatePlan grain references host slot " + f"{member.host_slot_id!r}, which this plan does not carry.", + ) + expr = self._ranked_scope_expr( + key=member.ranked_key, root_model=root_model, + root_relation=root_relation, bundle=bundle, scope=ranked_scope, + ) + # PARTITION BY takes the RAW expression: it is evaluated inside the + # ranked scope, where the joins it crosses are bound. The outer + # SELECT takes the materialised alias, because out there they are + # not. + partition_by.append(expr.copy()) + grain.append(RankedGrainProjection( + output_alias=self._full_alias_for_slot( + slot=host_slot, + source_relation=host_source_relation, + alias_index={}, + ), + inner_ref=ranked_scope.materialize_for( + expr, consumer=cte_scope, + ), + )) + + value_ref = ranked_scope.materialize_for( + self._ranked_value_expr( + key=local_key, root_model=root_model, + root_relation=root_relation, bundle=bundle, scope=ranked_scope, + ), + consumer=cte_scope, + ) + ranking_time = self._ranked_scope_expr( + key=plan.ranking_time_key, root_model=root_model, + root_relation=root_relation, bundle=bundle, scope=ranked_scope, + cast_derived=False, + ) + + where_parts: List[exp.Expression] = [] + # A measure's ``Column.filter`` is a predicate on the rows this + # aggregate ranks, so in its own scope it simply removes them — BEFORE + # the ranking, which is what the retired sentinel rank column plus its + # match flag were emulating from outside. + if local_key.column_filter_key is not None: + cfk_sql = local_key.column_filter_key.canonical_sql + if cfk_sql: + where_parts.append(self._enter_mode_a_predicate( + sql=cfk_sql, scope=ranked_scope, + location=f"Column.filter on model {root_model.name!r}", + )) + for filter_text in plan.target_model_filters: + if not filter_text: + continue + where_parts.append(self._enter_mode_a_predicate( + sql=filter_text, scope=ranked_scope, + location=f"SlayerModel.filters on model {root_model.name!r}", + )) + + # Host filters this CTE also evaluates. The two roots need different + # renderers for the same reason ``_wm_`` and ``_cm_`` do: a host-rooted + # CTE binds them in the host's own scope (so they render byte-identically + # to the copy ``_base`` keeps), while a target-rooted one re-anchors each + # leaf against the target. + if plan.target_path: + self._register_routed_filter_joins( + planned_query=planned_query, + filter_ids=list(plan.where_filter_ids), + scope=ranked_scope, + target_path=plan.target_path, + ) + routed_where = self._collect_routed_filters( + planned_query=planned_query, + filter_ids=plan.where_filter_ids, + target_relation=root_relation, + target_model=root_model, + bundle=bundle, + ) + else: + skip_ids = { + fp.id for fp in planned_query.filters_by_phase + } - set(plan.where_filter_ids) + self._resolve_where_filter_joins_via_scope( + planned_query=planned_query, scope=ranked_scope, + skip_filter_ids=skip_ids, + ) + routed_where, _routed_having = self._build_where_having_from_planned( + planned_query=planned_query, + source_relation=root_relation, + source_model=root_model, + bundle=bundle, + skip_filter_ids=skip_ids, + ) + if routed_where is not None: + where_parts.append(routed_where) + + from_expr, joins = self._build_from_and_joins( + source_model=root_model, source_relation=root_relation, + joined_paths=ranked_scope.join_paths.as_list(), bundle=bundle, + ) + inner = exp.Select() + # A NAMED projection list, never ``.*`` — the projection + # boundary (P-B) is what keeps the rank column's name private and what + # removes the need for a materialiser bolted on outside the scope. + ranked_scope.apply_materializations(inner) + inner = inner.select(build_rank_column( + partition_by=partition_by, + ranking_time=ranked_ordered( + ranking_time=ranking_time, + agg=plan.agg, + native_nulls_first=self._dialect.native_nulls_first( + descending=plan.agg == "last", + ), + ), + )) + inner = inner.from_(from_expr) + for join_expr, on_expr, join_type in joins: + inner = inner.join(join_expr, on=on_expr, join_type=join_type) + if where_parts: + inner = inner.where( + exp.and_(*where_parts) if len(where_parts) > 1 else where_parts[0], + ) + + pick = _wrap_cast_for_type( + build_ranked_pick(value_ref=value_ref), agg_slot.type, + ) + return build_ranked_cte_select( + inner=inner, grain=grain, pick=pick, agg_alias=full_agg_alias, + ) + + def _render_collapsed_ranked_plan(self, *, planned_query, bundle) -> str: + """Emit a whole plan AS its single ranked CTE body (D9). + + The collapse is not an optimisation. It is what keeps a re-rooted + cross-model first/last emitting valid SQL Server, where a ``WITH`` + nested inside a CTE definition is rejected outright — see the caller. + :func:`_collapses_to_ranked_cte` owns the precondition. + """ + source_model = bundle.source_model + source_relation = planned_query.source_relation + plan = planned_query.ranked_aggregate_plans[0] + slots_by_id = { + s.id: s + for s in ( + list(planned_query.row_slots) + list(planned_query.aggregate_slots) + ) + } + agg_slot = slots_by_id[plan.aggregate_slot_id] + cte_query, _grain_aliases = self._render_ranked_cte_from_planned( + plan=plan, + agg_slot=agg_slot, + bundle=bundle, + planned_query=planned_query, + slots_by_id=slots_by_id, + host_source_model=source_model, + host_source_relation=source_relation, + full_agg_alias=self._full_alias_for_slot( + slot=agg_slot, source_relation=source_relation, alias_index={}, + ), + ) + return cte_query.sql(dialect=self.dialect, pretty=True) + def _render_with_cross_model_plans( # NOSONAR(S3776) — orchestration of host ``_base`` CTE + per-plan ``_cm_*`` CTEs + combined SELECT + transform-chain step CTEs + outer ORDER BY/LIMIT wrap. Each block is a coherent compilation stage sharing planned_query / slots_by_id / cma_slot_ids / seen_base_ids state; extracting per-stage helpers would scatter the cross-cutting state. self, *, @@ -4366,6 +4849,15 @@ def _render_with_cross_model_plans( # NOSONAR(S3776) — orchestration of host windowed_slot_ids = { p.aggregate_slot_id for p in planned_query.windowed_aggregate_plans } + # DEV-1748 (B9) — ranked (``first`` / ``last``) aggregate slots render + # via their own ``_rk_`` CTEs. Same treatment as the two above: out of + # ``_base``, joined back on the grain. One first/last used to wrap the + # ENTIRE base in a ranking, which is what made a sibling aggregate's + # value depend on whether a first/last was in the query at all. + ranked_slot_ids = { + p.aggregate_slot_id for p in planned_query.ranked_aggregate_plans + } + isolated_slot_ids = cma_slot_ids | windowed_slot_ids | ranked_slot_ids # DEV-1503 / DEV-1745 (P-D) — the outer combined-SELECT WHERE wrapper is # routed by the PLANNER (``_plan_outer_where_filters``), which knows @@ -4428,16 +4920,13 @@ def _render_with_cross_model_plans( # NOSONAR(S3776) — orchestration of host # lives in a ``_wm_`` CTE joined back to ``_base``, so # rendering the composite inside ``_base`` would silently # substitute a PLAIN aggregate for the rolling one. - if s is not None and ( - s.id in cma_slot_ids or s.id in windowed_slot_ids - ): + if s is not None and s.id in isolated_slot_ids: outer_composite_slot_ids.add(slot.id) break base_projection = [ sid for sid in planned_query.projection - if sid not in cma_slot_ids + if sid not in isolated_slot_ids and sid not in outer_composite_slot_ids - and sid not in windowed_slot_ids ] # Hidden ORDER-BY-only LOCAL slots (``ORDER BY revenue:sum`` with @@ -4451,9 +4940,8 @@ def _render_with_cross_model_plans( # NOSONAR(S3776) — orchestration of host for order_entry in planned_query.order: sid = order_entry.slot_id if ( - sid in cma_slot_ids + sid in isolated_slot_ids or sid in outer_composite_slot_ids - or sid in windowed_slot_ids or sid in seen_base_ids ): # DEV-1714: a windowed slot lives in its ``_wm_`` CTE, never @@ -4499,7 +4987,7 @@ def _add_local_aux_slots( include_order=include_order, aggregates_only=aggregates_only, ): - if sid in cma_slot_ids or sid in seen_base_ids: + if sid in isolated_slot_ids or sid in seen_base_ids: continue slot = slots_by_id.get(sid) if slot is None: @@ -4532,11 +5020,7 @@ def _add_local_aux_slots( # Promoting it into ``_base`` would emit a dead PLAIN # aggregate under the windowed slot's alias, which the outer # composite would then read instead of the rolling value. - if ( - dep.id in cma_slot_ids - or dep.id in windowed_slot_ids - or dep.id in seen_base_ids - ): + if dep.id in isolated_slot_ids or dep.id in seen_base_ids: continue base_render_order.append(dep.id) seen_base_ids.add(dep.id) @@ -4875,6 +5359,43 @@ def _add_local_aux_slots( (a, a) for a in grain_aliases ] + # DEV-1748 (B9) — per-plan ``_rk_`` ranked first/last CTEs. Rooted where + # the ranked rows live (the host, or the join target), grouped at the + # query grain, joined back on it. Names are minted through the same + # collision-aware allocator the other two prefixes use, so two measures + # whose aliases lossy-sanitise alike get distinct CTEs (P-F). + rk_ctes: List[Tuple[str, exp.Expression]] = [] + rk_cte_name_for_plan: Dict[str, str] = {} + rk_agg_col_for_plan: Dict[str, str] = {} + rk_joinback_pairs_for_plan: Dict[str, List[Tuple[str, str]]] = {} + rk_allocator = self._gen_allocator or self._new_allocator() + for plan in planned_query.ranked_aggregate_plans: + agg_slot = slots_by_id.get(plan.aggregate_slot_id) + if agg_slot is None or not isinstance(agg_slot.key, AggregateKey): + raise RuntimeError( + f"RankedAggregatePlan {plan.aggregate_slot_id!r} references " + f"a missing or non-aggregate slot.", + ) + full_agg_alias = self._full_alias_for_slot( + slot=agg_slot, source_relation=source_relation, alias_index={}, + ) + cte_name = cte_name_from_alias( + RANKED_CTE_PREFIX, full_agg_alias, allocator=rk_allocator, + ) + cte_query, grain_aliases = self._render_ranked_cte_from_planned( + plan=plan, agg_slot=agg_slot, bundle=bundle, + planned_query=planned_query, slots_by_id=slots_by_id, + host_source_model=source_model, + host_source_relation=source_relation, + full_agg_alias=full_agg_alias, + ) + rk_ctes.append((cte_name, cte_query)) + rk_cte_name_for_plan[plan.aggregate_slot_id] = cte_name + rk_agg_col_for_plan[plan.aggregate_slot_id] = full_agg_alias + rk_joinback_pairs_for_plan[plan.aggregate_slot_id] = [ + (a, a) for a in grain_aliases + ] + # DEV-1745 (W5): dropped-filter warnings are NOT emitted here. This # emission fired once per cross-model plan — so nested subplans # double-fired for one user filter — and never fired at all on a path @@ -4909,7 +5430,17 @@ def _emit(sid: str, expr: exp.Expression) -> None: if planned_query.transform_layers else base_projection ) + # Deduped, because a C13 slot appears once per DECLARED NAME in the + # projection and its alias list already carries one entry per name. + # Visiting it twice and emitting the whole list each time renders N² + # columns, which the projection-consumption check below then rejects — + # a query mixing a two-name measure with any isolated aggregate used to + # fail outright. + _seen_host_ids: Set[str] = set() for sid in host_combined_ids: + if sid in _seen_host_ids: + continue + _seen_host_ids.add(sid) aliases = aliases_by_slot_id.get(sid, []) for full_alias in aliases: _emit(sid, grain_alias_column(alias=full_alias, table="_base")) @@ -4942,6 +5473,14 @@ def _emit(sid: str, expr: exp.Expression) -> None: wm_cte_name_for_plan[plan.aggregate_slot_id], wm_agg_col_for_plan[plan.aggregate_slot_id], ) + # A ranked operand resolves the same way (DEV-1748): its value is a + # column of a joined-in CTE, so a composite over it evaluates in the + # combined SELECT, never in ``_base``. + for plan in planned_query.ranked_aggregate_plans: + outer_composite_cm_map[plan.aggregate_slot_id] = ( + rk_cte_name_for_plan[plan.aggregate_slot_id], + rk_agg_col_for_plan[plan.aggregate_slot_id], + ) def _render_outer_composite(cslot) -> exp.Expression: rendered = self._render_filter_for_outer_wrapper( @@ -5090,6 +5629,28 @@ def _render_outer_composite(cslot) -> exp.Expression: ) combined_aliases_by_slot_id[plan.aggregate_slot_id] = list(full_aliases) + # DEV-1748 (B9) — ranked side. Identical to the windowed one above: one + # occurrence per declared user alias (C13), the hidden order-only case + # trimmed from the projection while its CTE stays joined. + for plan in planned_query.ranked_aggregate_plans: + agg_slot = slots_by_id[plan.aggregate_slot_id] + cte_name = rk_cte_name_for_plan[plan.aggregate_slot_id] + agg_col = rk_agg_col_for_plan[plan.aggregate_slot_id] + if plan.hidden and not planned_query.transform_layers: + combined_aliases_by_slot_id[plan.aggregate_slot_id] = [] + continue + public_names = list(agg_slot.public_aliases) or ( + [agg_slot.public_name] if agg_slot.public_name else [] + ) + full_aliases = [f"{source_relation}.{p}" for p in public_names] or [agg_col] + for full in full_aliases: + col = grain_alias_column(alias=agg_col, table=cte_name) + _emit( + plan.aggregate_slot_id, + col if full == agg_col else col.as_(full, quoted=True), + ) + combined_aliases_by_slot_id[plan.aggregate_slot_id] = list(full_aliases) + # Grain join-backs (P-I). Both plan kinds join back identically — on the # shared grain, null-safely, so a NULL dimension value or a nullable # truncated time bucket keeps its aggregate instead of dropping it. An @@ -5164,6 +5725,12 @@ def _render_outer_composite(cslot) -> exp.Expression: wm_joinback_pairs_for_plan.get(plan.aggregate_slot_id, []), ) for plan in planned_query.windowed_aggregate_plans + ] + [ + ( + rk_cte_name_for_plan[plan.aggregate_slot_id], + rk_joinback_pairs_for_plan.get(plan.aggregate_slot_id, []), + ) + for plan in planned_query.ranked_aggregate_plans ] for cte_name, joinback_pairs in joinback_specs: if cte_name in joined_cte_names: @@ -5211,6 +5778,15 @@ def _render_outer_composite(cslot) -> exp.Expression: cross_model_agg_slot_to_cm[plan.aggregate_slot_id] = ( cte_name, agg_col_alias, ) + # DEV-1748: a ranked aggregate is isolated for the same reason and + # resolves the same way. Its filter is HERE rather than a HAVING + # inside the CTE precisely because the join back is a LEFT JOIN — + # dropping the CTE row would resurrect the host row with a NULL. + for plan in planned_query.ranked_aggregate_plans: + cross_model_agg_slot_to_cm[plan.aggregate_slot_id] = ( + rk_cte_name_for_plan[plan.aggregate_slot_id], + rk_agg_col_for_plan[plan.aggregate_slot_id], + ) for fp in outer_where_filters: rendered = self._render_filter_for_outer_wrapper( key=fp.expression.value_key, @@ -5270,7 +5846,11 @@ def _render_outer_composite(cslot) -> exp.Expression: "CTEs.", ) return self._render_cross_model_transform_chain( - prelude_ctes=[("_base", base_select), *cm_ctes], + # ``_rk_`` CTEs join into the combined SELECT, which becomes the + # chain's base, so they belong in the prelude alongside the + # ``_cm_`` ones. Like those they are rooted at a real relation + # and depend on nothing. + prelude_ctes=[("_base", base_select), *cm_ctes, *rk_ctes], combined_select=combined_select, planned_query=planned_query, slots_by_id=slots_by_id, @@ -5291,6 +5871,11 @@ def _render_outer_composite(cslot) -> exp.Expression: CteEntry(name=name, query=query, depends_on=["_base"]) for name, query in wm_ctes ] + # A ``_rk_`` CTE is rooted at a real relation, never at ``_base``, so it + # declares no dependency — the same as a ``_cm_`` one. + cte_entries += [ + CteEntry(name=name, query=query) for name, query in rk_ctes + ] combined_statement = assemble_with_chain( entries=cte_entries, final=combined_select, ) @@ -5317,6 +5902,11 @@ def _render_outer_composite(cslot) -> exp.Expression: alias=wm_agg_col_for_plan[plan.aggregate_slot_id], table=wm_cte_name_for_plan[plan.aggregate_slot_id], ) + for plan in planned_query.ranked_aggregate_plans: + order_env.ranked_cte[plan.aggregate_slot_id] = grain_alias_column( + alias=rk_agg_col_for_plan[plan.aggregate_slot_id], + table=rk_cte_name_for_plan[plan.aggregate_slot_id], + ) # A PROJECTED outer composite orders on its combined-SELECT alias; an # order-only one has no alias and renders INLINE, so no synthetic # column leaks into the public projection. @@ -5684,7 +6274,9 @@ def _render_rerooted_cross_model_cte( rerooted_bundle = bundle.model_copy( update={"source_model": target_model}, ) - cte_sql = self.generate_from_planned(sub_plan, bundle=rerooted_bundle) + cte_sql = self.generate_from_planned( + sub_plan, bundle=rerooted_bundle, as_cte_body=True, + ) sub_slots_by_id = { s.id: s diff --git a/slayer/sql/render/order_terms.py b/slayer/sql/render/order_terms.py index 1a9579b2..fe30014d 100644 --- a/slayer/sql/render/order_terms.py +++ b/slayer/sql/render/order_terms.py @@ -77,6 +77,7 @@ class OrderEnv(BaseModel): host_base_hidden: Dict[str, exp.Expression] = Field(default_factory=dict) cross_model_cte: Dict[str, exp.Expression] = Field(default_factory=dict) windowed_cte: Dict[str, exp.Expression] = Field(default_factory=dict) + ranked_cte: Dict[str, exp.Expression] = Field(default_factory=dict) transform_step: Dict[str, exp.Expression] = Field(default_factory=dict) outer_composite: Dict[str, exp.Expression] = Field(default_factory=dict) #: Owns the null-ordering spelling (P-H). Defaults to the portable base. diff --git a/slayer/sql/render/ranked.py b/slayer/sql/render/ranked.py new file mode 100644 index 00000000..1c792b40 --- /dev/null +++ b/slayer/sql/render/ranked.py @@ -0,0 +1,156 @@ +"""The one ranked (``first`` / ``last``) CTE shape (P-C / P-G). + +A ranked aggregate is "the value from the row that sorts first (or last) within +each grain". That is two SELECTs: an inner one that ranks the rows it is allowed +to see, and an outer one that picks rank 1 per group. This module owns that +shape, so every route into it — host-rooted, target-rooted, and the collapsed +sub-plan a re-rooted cross-model CTE emits — produces the same SQL. + +Two choices are load-bearing and neither is arbitrary: + +**The outer SELECT aggregates; it does not filter.** ``MAX(CASE WHEN rn = 1 THEN +v END) … GROUP BY grain`` rather than ``SELECT v … WHERE rn = 1``. The two agree +on every non-empty grain and disagree on the empty one: over a source with no +rows the aggregate form returns ONE row holding NULL and the filter form returns +none. An empty grain is joined back with a CROSS JOIN, so a zero-row CTE erases +the entire result rather than yielding a NULL measure — and returning one NULL +row is what ``amount:sum`` does over the same empty source. + +**The inner SELECT projects a NAMED list, never ``source.*``.** Re-exporting the +source's columns is what made the superseded ranked wrap need a bolted-on +materialiser for crossing values, and it let a physical column named like an +internal rank column capture its reference. A projection boundary (P-B) makes +both unrepresentable: nothing crosses it that the scope did not choose to +publish. + +The internal names below are private to one CTE scope. They never appear in a +result key and never collide across CTEs, because each ranked CTE is its own +SELECT. +""" + +from __future__ import annotations + +from typing import List, Sequence, Tuple + +from pydantic import BaseModel, ConfigDict +from sqlglot import exp + +__all__ = [ + "RANKED_CTE_PREFIX", + "RANKED_SOURCE_ALIAS", + "RANK_COLUMN", + "RankedGrainProjection", + "build_rank_column", + "build_ranked_cte_select", + "build_ranked_pick", + "ranked_ordered", +] + +#: CTE-name prefix, alongside ``_cm_`` (cross-model) and ``_wm_`` (windowed). +RANKED_CTE_PREFIX = "_rk_" +#: Alias of the inner ranked subquery. +RANKED_SOURCE_ALIAS = "_rk_src" +#: The ``ROW_NUMBER`` column the outer SELECT picks rank 1 from. +RANK_COLUMN = "_rk_rn" + + +class RankedGrainProjection(BaseModel): + """One grain member as the ranked CTE sees it. + + ``output_alias`` is the column the CTE PUBLISHES — the same alias the host + ``_base`` projects that member under, which is what lets the join-back + compare the two by name. ``inner_ref`` is how the same value is named inside + the ranked scope, which for a materialised crossing expression is an alias + rather than the expression itself. + """ + + model_config = ConfigDict(arbitrary_types_allowed=True) + + output_alias: str + inner_ref: exp.Expression + + +def build_rank_column( + *, + partition_by: Sequence[exp.Expression], + ranking_time: exp.Ordered, +) -> exp.Expression: + """``ROW_NUMBER() OVER (PARTITION BY ORDER BY