diff --git a/.claude/skills/slayer-query.md b/.claude/skills/slayer-query.md index 4bab5cd1..b5a411c6 100644 --- a/.claude/skills/slayer-query.md +++ b/.claude/skills/slayer-query.md @@ -20,9 +20,9 @@ A `SlayerQuery` is a JSON/dict object. The same shape works across the REST API, } ``` -`order[].column` is the short alias (`count`, `revenue_sum`) — not the colon form. +`order[].column` uses the short alias (`count`, `revenue_sum`) to order by a measure declared in the same query; undeclared order targets use formula (colon) syntax — see below. -**Ordering by something you don't project.** `order` may name an undeclared column/aggregate/expression ("top-N by X, show only Y, Z"). Computed hidden, sorted on, and stripped from the result: an **aggregate** (`amount:sum`, `customers.revenue:sum`), an inline **transform** (`rank(amount:sum)`, `change(...)`, `cumsum`/`lag`/`lead`/`ntile`), an inline **composite** (`revenue:sum / cnt:sum`, `abs(amount:sum)`), and a **windowed** aggregate (`amount:sum(window='90d')`, alone or inside a composite). A **raw row column** is orderable only in a raw-rows query (`distinct_dimension_values: false`); in a grouped/dedup query it's rejected (HTTP 400 — add it to `dimensions` or order by an aggregate of it). A **joined** row column is rejected — project it. Order expressions must use formula syntax for their operands, not the `name`s of measures declared in the same query: `{"column": "revenue:sum / cnt:sum"}` works, `{"column": "rev / cnt"}` is rejected. +**Ordering by something you don't project.** `order` may name an undeclared column/aggregate/expression ("top-N by X, show only Y, Z"). Computed hidden, sorted on, and stripped from the result: an **aggregate** (`amount:sum`, `customers.revenue:sum`), an inline **transform** (`rank(amount:sum)`, `change(...)`, `cumsum`/`lag`/`lead`/`ntile`), an inline **composite** (`revenue:sum / cnt:sum`, `abs(amount:sum)`), and a **windowed** aggregate (`amount:sum(window='90d')`, alone or inside a composite). A **raw row column** sorts directly in a raw-rows query (`distinct_dimension_values: false`); in a grouped/dedup query there is no single value per group, so it sorts **per group** by the extreme the direction puts first — `asc` by each group's `min`, `desc` by each group's `max`. Write `{"column": "created_at:max", "direction": "asc"}` explicitly for the other one. A **joined** row column (`customers.regions.name`), and a derived column whose `sql` reaches through a join, behave the same way — the join is pulled in for the sort, and in a grouped query the wrap is computed per host row-group rather than globally. NULLs sort **last** in both directions on every database (SQL Server excepted: its native ordering is used, because the portable emulation makes the statement fail there). An order target SLayer cannot resolve is an error, never a silently unsorted result. Order expressions must use formula syntax for their operands, not the `name`s of measures declared in the same query: `{"column": "revenue:sum / cnt:sum"}` works, `{"column": "rev / cnt"}` is rejected. **Dim-only queries deduplicate.** A query with no measures and at least one dimension or time-dimension auto-emits `GROUP BY ` and returns the distinct combinations. The `GROUP BY` is applied before `LIMIT`, so a row cap can't silently drop unique tuples. To opt out, set `"distinct_dimension_values": false` on the query — emits raw rows (no top-level `GROUP BY`), with WHERE / ORDER BY / LIMIT applied as usual. Any measure reference in `measures` / `filters` / `order` raises `DistinctDimensionValuesError` in this mode. @@ -72,7 +72,7 @@ Result column naming: `revenue:sum` → `orders.revenue_sum` (colon becomes unde **Boolean logic**: `AND`, `OR`, `NOT` -**String-hygiene scalars** (DEV-1378, lowercase only): `lower`, `upper`, `trim`, `replace`, `substr`, `instr`, `length`, `concat`. Plus the SQL `||` operator (folded into `concat(...)`). Examples: `"lower(status) = 'active'"`, `"length(replace(x, ',', '')) > 0"`, `"substr(s, 1, instr(s, ',') - 1) = 'first'"`, `"first || ' ' || last = 'jane doe'"`. Calls outside this allowlist (`json_extract`, `coalesce`, …) belong in `Column.sql` / `Column.filter` / `SlayerModel.filters` (Mode A SQL), not query filters. +**Mode-B scalars** (matched case-insensitively): string hygiene (`lower`, `upper`, `trim`, `ltrim`, `rtrim`, `replace`, `substr`, `substring`, `instr`, `length`, `concat`), null handling (`coalesce`, `nullif`, `ifnull`), and math (`round`, `abs`, `ceil`, `floor`, `sign`, `log10`, …). Plus the SQL `||` operator (folded into `concat(...)`). Examples: `"lower(status) = 'active'"`, `"coalesce(nickname, name) = 'Ada'"`, `"length(replace(x, ',', '')) > 0"`, `"first || ' ' || last = 'jane doe'"`. Raw SQL functions outside the allowlist (`json_extract`, `date_trunc`, …) belong in `Column.sql` / `Column.filter` / `SlayerModel.filters` (Mode A SQL), not query filters. **Filtering on computed measures**: `"change(revenue:sum) > 0"`, `"last(change(revenue:sum)) < 0"`. Applied as post-filters on the outer query. @@ -88,7 +88,10 @@ Result column naming: `revenue:sum` → `orders.revenue_sum` (colon becomes unde engine = SlayerQueryEngine(storage=storage) # Async (most callers — REST/MCP): -result = await engine.execute(query=query) # SlayerResponse with .data, .columns, .row_count, .sql, .attributes +result = await engine.execute(query=query) # SlayerResponse with .data, .columns, .row_count, .sql, .attributes, .warnings +# .warnings holds advisories, each tagged with .kind — "normalization" for an input +# rewrite, "unreachable_filter_dropped" for a filter dropped from a cross-model CTE +# (it still applies at the host). Empty for a clean query. # With runtime variables (highest precedence — wins over query.variables / model defaults): result = await engine.execute(query=query, variables={"region": "US"}) @@ -180,7 +183,7 @@ Surfaces: Python SDK `engine.execute(query=[...])`; CLI `slayer query @file.json ## Result format -Column keys use `model_name.column_name` format: `"orders._count"`, `"orders.revenue_sum"`. For multi-hop joined dimensions, the full path is included: `"orders.customers.regions.name"`. An explicit `name` on a measure spec swaps the canonical leaf — local (`{"formula": "amount:sum", "name": "rev"}` → `"orders.rev"`) or cross-model (`{"formula": "customers.revenue:sum", "name": "cust_rev"}` → `"orders.customers.cust_rev"`, hop path preserved). In any downstream stage of a `query_nested` DAG the column is exposed under the bare `name` (e.g. `cust_rev`) — that's what you type in stage 2's `formula` to reference the value. The response also includes `attributes` — a `ResponseAttributes` object with `.dimensions` and `.measures` dicts, each mapping column alias → `FieldMetadata` (label, format). +Column keys use `model_name.column_name` format: `"orders._count"`, `"orders.revenue_sum"`. For multi-hop joined dimensions, the full path is included: `"orders.customers.regions.name"`. Columns come back in the order you declare them in the query — dimensions, then time dimensions, then measures — regardless of measure kind (local, cross-model, or windowed); hidden order-only / filter-only targets never appear. An explicit `name` on a measure spec swaps the canonical leaf — local (`{"formula": "amount:sum", "name": "rev"}` → `"orders.rev"`) or cross-model (`{"formula": "customers.revenue:sum", "name": "cust_rev"}` → `"orders.customers.cust_rev"`, hop path preserved). In any downstream stage of a `query_nested` DAG the column is exposed under the bare `name` (e.g. `cust_rev`) — that's what you type in stage 2's `formula` to reference the value. The response also includes `attributes` — a `ResponseAttributes` object with `.dimensions` and `.measures` dicts, each mapping column alias → `FieldMetadata` (label, format). ## Strict validation (v3) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index d1ba76f5..4bff8256 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -78,8 +78,8 @@ slayer/ core/ # Domain models, enums, query/formula parsers engine/ # Query orchestration query_engine.py # Central orchestrator (execute, model resolution) - enrichment.py # SlayerQuery → EnrichedQuery transformation - enriched.py # EnrichedQuery dataclasses + binding.py # SlayerQuery → typed bound keys + stage_planner.py # Bound query → typed PlannedQuery (rendered by sql/generator.py) ingestion.py # Auto-ingestion from database schemas sql/ # SQL generation (sqlglot) and execution (SQLAlchemy) storage/ # Storage backends (YAML, SQLite, pluggable registry) @@ -116,7 +116,7 @@ docs/ 1. Add the function name to `ALL_TRANSFORMS` and/or `TIME_TRANSFORMS` in `core/formula.py` 2. Handle it in the formula parser (`parse_formula`) 3. Add the SQL generation in `generator.py` -4. Add enrichment support in `enrichment.py` if it needs special handling +4. Add binder/planner support in `binding.py` / `stage_planner.py` if it needs special handling 5. Add unit tests in `test_sql_generator.py` and integration tests 6. Document in `docs/concepts/formulas.md` diff --git a/DECISIONS.md b/DECISIONS.md index fca4666b..e5c1c75b 100644 --- a/DECISIONS.md +++ b/DECISIONS.md @@ -95,3 +95,25 @@ implementation detail. Include issue refs when known. - 2026-08-03 — Optional blocks + Cube JS/FILTER_PARAMS import (DEV-1730 / #270): a Mode-A-only `{? ... ?}` block renders its content parenthesised when every inner `{var}` is supplied, else collapses to the neutral `(1=1)` — the SLayer form of a Cube `FILTER_PARAMS` optional pushdown. Blocks live in the same `substitute_variables` (escape="sql") scanner as `{var}`/`{{`/`}}`, must contain ≥1 var, do not nest, and are rejected in Mode-B. A block-bearing model runs substitution even on a zero-variable call so its blocks collapse (the `_substitute_model_sql_surfaces` fast-path now checks for `{?` too); a block-free, required-only model with zero variables is still left untouched (the documented DEV-1625 raw-brace-literal boundary). `extract_model_variables(model)` derives required (bare, no default) vs optional (in-block or defaulted) from the four Mode-A surfaces — structural, nothing persisted, surfaced additively in the inspect skeleton `Variables:` line. The Cube importer gains a **JavaScript front-end** (esprima ESTree parser, a new core dep) that parses `cube()`/`view()` into the same `CubeCube`/`CubeView` shapes as YAML (dynamic values → report + skip member). FILTER_PARAMS refs are carried JS→converter as structured `CubeFilterParamRef` on the transient `CubeCube` (sentinels in the surface text; no arrow-body re-parse, sidestepping the `{var}`-vs-`{FILTER_PARAMS…}` brace clash); the converter resolves sentinels AFTER `translate_cube_refs` so the introduced `{var}` are never eaten. Requiredness (bare vs block) is decided in the converter alone via `honor_required_meta` (default on; CLI `--ignore-required-meta`) AND the member's `meta.required`; with the flag off a scalar-position arrow collapses to Cube's own `(1=1)::TIMESTAMP` booby-trap, faithfully. Cross-cube refs, unknown members, and generated-name collisions (`d`→`d_from` clashing member `d_from`) drop the cube (`filter_params_unsupported`); each logical variable is reported once (`filter_params_variable`) and stashed in `meta.cube_variables`. `render_probe_text` (blocks→`(1=1)`, bare vars→`0`) is the single import-time validation renderer, matching runtime collapse. - 2026-08-04 — Dialect-aware / complete escaping for Mode-A `{variable}` substitution (DEV-1727), hardening DEV-1625. `substitute_variables(..., escape="sql")` is now **dialect-aware** and **fail-closed**: it gained a required keyword-only `backslash_escapes` signal (`bool | None`, raises if `None` in sql mode) so a caller rendering raw SQL can never silently under-escape. On backslash-escaping dialects (MySQL/ClickHouse/Snowflake/Redshift/BigQuery/Databricks/Spark) it doubles the backslash before escaping the single quote; on standard dialects it keeps the `''` quote-doubling. The double quote is deliberately left untouched — inside a single-quoted literal `\"` is NOT a recognised escape on 6 of the 7 backslash dialects (only MySQL), so escaping it would corrupt the value. The regime is DERIVED from sqlglot's own tokenizer via `SqlDialect.backslash_escapes_strings` (= `"\\" in tokenizer.STRING_ESCAPES`, guarded + 14-dialect pinned) so our escaping can never drift from the parser that reads the substituted SQL. `escape="python"` (Mode-B) additionally encodes the full C0 control range (`\t`/`\n`/`\r` named, rest `\xNN`) so raw newlines/NUL no longer break `ast.parse`. Engine fail-closed: `_substitute_model_sql_surfaces` / `_render_probe_model` require a `dialect`, threaded from the resolved datasource — no bare bool to forget. Assumes MySQL's default `sql_mode` (backslash escapes on); `NO_BACKSLASH_ESCAPES` servers are a sqlglot-layer-wide limitation, documented not fixed. The SQLite backslash end-to-end gap stays a pinned strict-xfail (pre-existing, out of scope). Bound parameters rejected (don't fit substitute-into-raw-SQL). Nested/join/cross-model lineages remain DEV-1678. - 2026-08-04 — Declared list-valued `{variable}` coercion (DEV-1730 follow-up): a scalar supplied for a variable the model declares `list_valued` is wrapped into a one-element list before Mode-A substitution, so an importer-generated `col IN ({var})` renders `IN ('US')` rather than the unquoted `IN (US)`. The generic scalar rule (author writes the quotes, so `{var}` also works in numeric/fragment positions like `amount >= {floor}` and `{d}::TIMESTAMP`) is CORRECT and unchanged — it just presumes an author who can see the SQL position, which a machine-generated fixed template does not have; the caller cannot supply per-element quotes through parentheses the importer wrote. Silent-wrong-answer risk drove the fix over a raise: `region IN (US)` parses as a column reference, so it fails at the database with a confusing message, or resolves against a real column and returns wrong rows. Opt-in is a front-end-NEUTRAL flag: the Cube converter writes `list_valued: ref.kind == "string"` into each `meta.cube_variables` entry (arrow forms splice pre-quoted scalars and stay `False`), and the engine reads only that flag — never Cube's `kind` taxonomy — so a future list-shaped front-end opts in the same way. Coercion lives at the single Mode-A choke point `_substitute_model_sql_surfaces` (execution and the `_render_probe_model` type-probe both route through it, so it cannot be bypassed) via `coerce_declared_list_variables` / `list_valued_variable_names` in `slayer/core/query.py`. Scope is deliberately narrow: only `str`/`int`/`float`/`bool` are wrapped; `list`/`tuple` pass through (the **empty list still raises** — "no filter" belongs to an optional block or a sentinel default); `None`/`dict` are left for `_render_variable_value` to reject with its own naming error; hand-written models declare nothing and are untouched. Follow-on from the same review: `declares_variables(model)` (any non-empty `meta.cube_variables`) now also defeats the DEV-1625 zero-variable fast path, via the shared `_model_needs_substitution_pass` predicate used by both `_substitute_model_sql_surfaces` and `_render_probe_model`. This closes the fast-path hole for a GENERATED model whose pushdowns are all required (no `{? ?}` block to force the pass): such a model used to emit a bare `{var}` into the SQL on a zero-variable call instead of raising the documented missing-variable error. The hole stays open — deliberately — for hand-written models, which declare nothing and keep the raw-brace-literal protection (`'{1,2,3}'`). The `list_valued` flag is matched with `is True`, not truthiness, since `meta` is user-extensible and a stray `1` or the string `"false"` must not switch substitution semantics. The bag is also SELF-IDENTIFYING — an entry counts only with a string `member` (the shape every importer writes) — so a hand-written `meta` that reuses the `cube_variables` key is not mistaken for generated SQL and silently stripped of its brace-literal protection. +- 2026-08-05 — One naming authority + one ValueKey renderer (DEV-1744, PR 1 of the DEV-1742 consolidation). Four consequences worth recording. (1) **Cross-model CTE dedup is now structural.** `_cm_` CTE names were minted by a doubly-lossy helper (`flat_name`, itself documented non-injective, then a non-identifier `re.sub`) and the resulting string doubled as the plan's identity key in `seen_cm`. Two reachable failures followed: two measures whose canonical aliases differ only in case emitted two `_cm_` names that fold together on every case-folding dialect (the collision belt raised — `_wm_` had been retrofitted onto the allocator years earlier, `_cm_` never was), and two genuinely distinct aggregates that sanitised alike made the second skip the loop body, leaving its join-back and column-alias maps unwritten and raising `KeyError` downstream. The dedup key is now the typed `AggregateKey` plus the source relation; the name is allocator-minted and stored once, so the five sites that used to re-derive it now read it. Deliberately NOT keyed on the canonical alias: `canonical_agg_name` omits `column_filter_key`, so a filtered and an unfiltered aggregate over one column share an alias while needing two CTEs — deduping on the string would silently merge them, a wrong answer rather than a crash. (2) **One ScalarCall policy.** Two of the five renderers returned `exp.Anonymous` passthrough, so `ifnull` reached Postgres — which has no `IFNULL` — while the same key emitted `COALESCE` from a projection. All six render paths now call one `render_scalar_call`. Transpiling alone was NOT sufficient: `exp.func("LOG10", x)` normalises to a generic `Log(10, x)` that re-emits as `LOG(10, x)`, wrong on dialects with a native single-arg `LOG10`, so the policy is transpile-then-log-alias-rewrite. Consequence surfaced and approved: `concat` now emits the `||` operator on Postgres/DuckDB where it previously emitted `CONCAT(...)`. That is a semantic change as well as a spelling one — Postgres `CONCAT()` ignores NULL operands, `||` propagates them. Ratified as the kept behavior: the projection path has always emitted `||`, so before this change SLayer disagreed with itself between filters and projections on the same input, and consistency was chosen over preserving the filter-side NULL tolerance. (3) **`ScopeFrame._model_for` raises** instead of falling back to the root model; the fallback turned a wiring bug into a wrong answer by expanding a different model's derived SQL. (4) **Named P-F carve-outs**, recorded rather than omitted: the T-SQL ORDER-BY-detach rewrite and the stage-schema wrapper take `_outer` / `_stage_inner` as shared CONSTANTS rather than allocator-minted names (the former is a post-generation AST pass with no allocator in reach, and a later PR rebuilds that machinery); `base` / `_base` / `_combined` keep their literal names but are reserved into the allocator. Superseded code is retained and callable per the chain's operating rule — deletion happens in the final PR. +- 2026-08-05 — Arithmetic composition joins ScalarCall as a construct that renders ONCE, ahead of the call-site migration (DEV-1744). A carve-out from the deferral below, taken because the generator's three composers (`_build_arith_or_cmp_ast`, `_compose_arithmetic_op`, `_build_arithmetic_for_filter`) were provably emitting wrong SQL, not merely duplicated SQL. sqlglot does not parenthesise by node nesting, and each composer knew a different subset of the precedence table — `_build_arith_or_cmp_ast` applied no precedence pass at all — so eight shapes emitted expressions that parse cleanly and mean something else: `not (a AND b)` → `NOT a AND b` (De Morgan); `-(a + b)` → `-a + b`; `(a = 5) IS NULL` → `a = 5 IS NULL`, read as `a = (5 IS NULL)` because `IS` binds tighter than `=`; `a AND (b OR c)` → `a AND b OR c`; `(a + b) * c` → `a + b * c`; `a - (b + c)` → `a - b + c`; `a + (b - c)` → `a + b - c` (`+` was treated as associative, which it is not over floats or fixed-precision decimals); and `(a > b) + 1` → `a > b + 1`. All three now delegate to one `render_arithmetic`. Sole exception: comparison operands in `_build_arithmetic_for_filter` keep `_paren_if_binary`, which parenthesises EVERY multi-term operand — strictly more grouping than the shared policy derives, never less, so it is a readability choice rather than a second correctness policy. The full suite passed without a single expectation edit, which is also why the bugs survived this long. Related: the renderer's precedence table now treats the comparison level as NON-associative, parenthesising an equal-precedence LEFT child there (arithmetic keeps left-associative flattening); and a bare `*` is refused as the source of any aggregation but `count`, which previously built `SUM(*)` / `COUNT(DISTINCT *)`. +- 2026-08-05 — `%` is parenthesised unconditionally when nested, and grouping is pinned by a round-trip property test (DEV-1744). SQL puts `%` on the `*` / `/` tier, and so does SLayer's own Mode-B parser (`formula.py`, `ast.Mod: 2`) — but SQLGLOT's parser puts it on the `+` / `-` tier, so it reads `a + b % c` back as `(a + b) % c`. That matters because generated SQL is re-parsed by sqlglot inside our own pipeline (reserved-word pre-quoting, the log-alias transform), so relying on precedence for `%` means the expression is silently regrouped in flight, before any database sees it. The renderer therefore emits the parens rather than deriving them. Found by the meta-test rather than by review: `TestEveryOperatorPairSurvivesTheRoundTrip` renders every ordered operator pair in both operand positions across postgres/sqlite/mysql, re-parses the emitted SQL, and compares OPERAND STRUCTURE (parens and dialect-injected casts normalised away, since those are grouping-preservation and typing respectively). Re-parse *stability* is deliberately not the assertion — `a + b * c` re-parses to a stable string while meaning something other than the `(a + b) * c` that was built. The matrix is grouped by result type; feeding a boolean to an arithmetic operator is not a shape the binder builds, and dialects wrap such an operand in their own numeric cast. +- 2026-08-05 — Renderer call-site migration deferred to the scope-assembly PR (DEV-1744 → DEV-1746). PR 1 lands the complete, tested `render_value_key` API but does NOT reroute the generator's own render paths through it; only the ScalarCall policy is genuinely shared (all six paths call `render_scalar_call`). Deferred because the filter and composite paths carry state the context does not yet consume — the local-aggregate HAVING branch with its slot lookup and inline-aggregate emission, the first/last ranked state, the filter-side CAST policy, and the rn-suffix / filtered-rank / composite-alias / resolved-kwarg maps — and because the cross-scope paths are the ones that should supply `resolve(consumer=...)` its first production caller, which is scope-assembly work by nature. Splitting the reroute across two PRs would move that state twice. Full detail, per call site, in the `slayer/sql/render/value_expr.py` module docstring. + +- 2026-08-05 — One Mode-A door, plan-time filter routing, structural reachability (DEV-1745, PR 2 of the DEV-1742 series). **P-A:** every fragment of free (Mode-A) SQL now enters a SELECT scope through ONE door — `ScopeFrame.enter_predicate` / `enter_expression`, two surfaces over one implementation differing only in the parse helper (the `SELECT 1 WHERE …` statement-keyword guard for predicates). The door prequotes, parses, scans, expands, re-parses, re-scans, unions both scans into `join_paths`, then applies Law 2. Join discovery is a side effect of entering, so a caller cannot forget it; the DEV-1494 dual-scan contract is preserved because a dotted ref whose derived column inlines to a constant is only visible to the PRE-expansion scan. The surface's grammar is a static property of the field being read (`Column.filter` and model `filters` are predicates, `Column.sql` is an expression) — deliberately no content sniffing and no "try one grammar then the other" retry, either of which would put classification back into render time. **The door adds NO qualification pass:** `expand_derived_refs_sync` already qualifies, against the OWNING model's canonical alias, and deliberately leaves a node alone when its alias path does not resolve because that is an opaque CTE / subquery reference — a blanket pass against `root_relation` would corrupt exactly those. `SQLGenerator._parse` and `_parse_predicate` were byte-identical apart from one line and now delegate to a shared `slayer/sql/render/parse.py`, which is what lets the door take over a call site without changing the SQL it emits. **Failure is loud:** an unparseable Mode-A fragment raises `ModeASqlParseError` carrying the fragment and its location. Three swallow-all `except Exception` lanes leave the production path, including `_filter_join_paths._scan`, which converted a parse failure into ZERO join paths — emitting a query missing its joins rather than reporting the problem. Two consequences on emitted SQL were accepted explicitly: undeclared bare identifiers in Mode-A text are now qualified against the scope root (the old fallbacks qualified only DECLARED columns, leaving the rest to bind to whatever was in scope after re-rendering inside a rerooted CTE), and the shifted CTE renders its model filter from the door's AST rather than passing regex-substituted text through, so sqlglot's canonical form appears. **P-D:** the outer combined-SELECT WHERE routing is decided by the planner (`PlannedQuery.outer_where_filter_ids`) and consumed verbatim; the generator's render-time re-walk of `filters_by_phase` is deleted. **Reachability is structural (5.3):** `classify_host_filter` routed derived-column references by asking whether the declaring model's NAME appeared anywhere in `target_path`. That flat membership test called a SIBLING branch reachable whenever it shared a name with the target path, and called a host-model derived column whose `Column.sql` crossed INTO the target host-local. It is replaced by one rule for every key kind — a dependency is reachable iff its anchored join path is a PREFIX of `target_path` — computed per filter at plan time by `slayer/engine/filter_reachability.py`, recursively over the whole key tree, failing CLOSED on an unhandled kind. The summary lives on `PlannedQuery`, NOT on `ColumnSqlKey` (interned, and `_reroot_path_ref` copies unknown fields through rerooting stale) and NOT on `ValueSlot` (`filter_referenced_slot_ids` silently skips keys with no interned slot — a filter-only derived column is exactly such a key — and slots are copied wholesale into nested plans); it is recomputed per plan so the paths always mean what they say relative to the root they were anchored at. **Warnings (5.5):** dropped-filter emission moves from mid-render (once per cross-model plan, so nested subplans double-fired, and silent on any path that never reached that step) to the engine boundary — collected across every plan, deduped per user filter on `(location, original text)`, with drop reasons required to AGREE (disagreement raises rather than keeping the first, which is why the reason is now target-independent). `SlayerResponse.warnings` widens to a `kind`-discriminated union and is surfaced by REST, MCP and the CLI (stderr, so stdout stays pipeable). **Two live bugs fixed in passing:** a `Column.sql` that is exactly one bare reference to another derived column was silently not expanded, because `col.replace()` is a no-op when that column IS the parsed root; and a `_cm_` CTE never registered the joins its aggregation template fragments crossed, emitting `SUM(customers.spend * regions.weight) FROM customers` — SQL no database accepts. **P-J:** the superseded helpers stay callable and test-pinned rather than deleted (state 1); newly unreferenced by production are `_render_model_filter_sql`, `_filter_join_paths`, `_render_mode_a_predicate`, `_expand_degenerate_derived_root`, `_column_ref_is_derived`, `_predicate_references_derived` and `_qualify_mode_a_sql_filter`. `_qualify_column_filter_sql` remains live via `_expand_column_filter_sql`'s bundle-less branch, and `_BARE_IDENT_RE` remains in use outside the Mode-A surfaces. +- 2026-08-06 — Five parser-only scalars admitted to the binder; four deferred (DEV-1744). `SCALAR_PASSTHROUGH` (parser, `formula.py`) and `SCALAR_FUNCTIONS` (binder, `keys.py`) had drifted: the parser admitted nine names the binder then rejected with `UnknownFunctionError`, and the parser's own "Supported scalar functions" error text advertised them. `ceiling`, `sign`, `ltrim`, `rtrim`, `substring` are now admitted — every Tier-1 dialect emits one correct form for them, so there is no semantic ruling to make; `ceiling`/`substring` are aliases rendering to the same node as `ceil`/`substr`. The other four are deferred to DEV-1753 because each needs a decision this consolidation should not make: `greatest`/`least` fall back to SQLite's `MAX(a, b)`/`MIN(a, b)`, which return NULL when an argument is NULL where Postgres ignores NULLs and Snowflake gets `GREATEST_IGNORE_NULLS` — the same class of divergence as the `concat` → `||` ruling above; `trunc` has four target forms including T-SQL `ROUND(x, 0, 1)` and a lowercase ClickHouse spelling on a case-sensitive backend; and `mod` is operator-shaped, so `exp.func("MOD", a, b)` builds a binary node the dialect rewrite rejects — it needs the special-casing `like` already has, in `render_scalar_call`, which the scope-assembly PR rewrites. Arity bounds are pinned tight for concrete reasons rather than caution: `ceiling(x, y)` silently emits `CEIL(x, y)` and `ceiling(x, y, z)` DuckDB's unrelated `CEIL(x TO z)`; a 4-arg `substring` drops one; and the 2-arg strip-these-characters trim form is excluded because sqlglot emits a literal `LTRIM(str, chars)` for some targets while MySQL's `LTRIM` takes one argument. The remaining divergence is pinned as an exact set by `TestParserAndBinderScalarSetsAgree`, so neither side can drift again unnoticed. +- 2026-08-06 — Scope assembly: null-safe grain doctrine, pagination through the dialect, and the combined layer as one AST (DEV-1746, PR 3 of the DEV-1742 consolidation). Six consequences worth recording. (1) **One grain join-back builder** (`slayer/sql/render/joins.py`), taking EXPRESSION operand pairs and returning `None` for an empty grain so the caller emits `CROSS JOIN` — a scalar aggregate has no grain to join on, and a truthy `TRUE` would erase that distinction. It replaced a string round-trip that re-parsed pre-quoted operands: SLayer's public aliases are dotted, and on BigQuery the round-trip re-read `_base."orders.customers.status"` as a multi-part *reference*, emitting a qualifier for a table that does not exist (``_base___orders___customers`.`status``). Three golden entries recorded `ScopeLeakError` for exactly that and now record real SQL. The `_wm_` INNER grain went null-safe in the same move: it compared with a plain `=`, so a NULL-dimension group matched nothing and silently received NULL — while the outer join-back and the `_cm_` join-back were already null-safe, so the answer depended on which isolation shape a measure landed in. Executed on DuckDB and SQLite; the integration test that asserted the old behaviour said so in its NAME (`test_null_dimension_group_gets_null_windowed_value`). (2) **Pagination is a dialect-strategy method.** `apply_pagination` is the one place LIMIT/OFFSET is expressed; the T-SQL override owns its rule explicitly (TOP for a bare limit; a deterministic `ORDER BY (SELECT NULL)` before OFFSET/FETCH when the query is unordered, because SQL Server rejects OFFSET without ordering) rather than relying on sqlglot happening to apply it — so the rule survives a sqlglot upgrade and lands in the AST where callers can see it. The cross-model combined path used to append raw text and emitted a literal `LIMIT` on SQL Server, while the SAME query carrying a transform layer went through the outer wrap and came out correct. (3) **The combined statement is one `exp.Select`.** Projection, FROM/JOIN chain, WHERE, ORDER BY and pagination were all string-assembled; the WHERE in particular was glued on with a hand-rolled `"\nAND "` / `"\nWHERE "` connector that chose between them by asking whether an earlier filter existed. `Select.where` conjoins, so that choice disappears. The WITH chain is assembled by `slayer/sql/render/cte_assembly.py` from **declared** `(name, query, depends_on)` entries, never dependencies discovered by scanning the rendered statement: a scan cannot tell a CTE reference from a same-named real table, is defeated by quoting and case folding, and would silently mis-order rather than fail. **CTE bodies stay AST end to end** — the first attempt kept the renderers returning text and parsed at the seam, which re-introduced exactly the dotted-alias corruption the same PR had just removed. One documented parse seam remains: the re-rooted cross-model CTE arrives as a complete nested `WITH … SELECT` from `generate_from_planned`. The combined ORDER BY had to become AST for the same reason — rendering its terms and re-parsing corrupted them into `` `orders___customers`.`spend_sum` ``. (4) **`PlannedQuery.projection` is consumed verbatim** (B7). The combined projection was assembled in four grouped passes, so a cross-model measure declared first was emitted last; the generator admitted it in a comment. Hidden-slot trimming stops being a mechanism at all — a hidden slot is absent from the list. Two subtleties: a slot may legitimately appear MORE THAN ONCE (C13 lets one key be selected under several names, and the plan lists it once per name), so each occurrence consumes the NEXT of that slot's rendered columns — emitting the whole list per occurrence projected every alias once per name; and because pydantic's `model_copy(update=…)` skips validators and rerooting uses exactly that, one renderer-side assertion is kept at the single entry point, raising rather than skipping, since silently dropping a column the plan asked for is how a wrong answer reaches a user. (5) **One materialiser on the cross-model path.** `ScopeFrame` gained `materialize_for(expr, consumer=…)` — `resolve` anchors a ref and closes it, this is the second half alone, for a producer that anchored its own template (a date-truncated grain, a value carrying its column's declared CAST; re-deriving those from a ref would project a different expression than the aggregate consumes). Unifying the dedup tables fixed a defect: grouping a first/last cross-model aggregate by the same crossing expression it aggregates projected that expression TWICE (`regions.weight AS _val_0` and `AS _val_1`), because the grain site and the value site kept separate maps. `_build_first_last_base_select` keeps its own flow until PR 5's `RankedAggregatePlan`; the boundary is pinned by a test that fails when it moves, rather than left as an assumption. (6) **The empty-base grain is a typed plan node** (`EmptyBaseGrainPlan`, presence as the discriminator, carrying the host-local filter ids). No `grain_slot_ids` field — in that shape the grain is empty by definition — and the invariant that field would have encoded had to be restated during implementation, because `CrossModelAggregatePlan.shared_grain_slots` can name a HIDDEN row slot a filter created, which never becomes a projected grain column. Emitted SQL byte-identical. Reformatting churn was accepted and reviewed: emitting the statement in one pass means sqlglot's printer indents CTE bodies, which moved 25 golden entries and broke assertions that pinned layout rather than structure — one test helper (`_extract_src_body`, anchored on the exact text `"\n) AS _src"`) accounted for 19 failures by itself and is now whitespace-tolerant. Also fixed in passing, because it was two lines and not a refactor: a filter on a JOINED model's crossing derived column emitted invalid SQL — both the join scanner and the filter renderer expanded the column without `is_root=False`, so a further-joined ref came out bare, the scanner could not match it to a join path, the hop was never joined, and the filter referenced a table absent from the FROM. The two must stay in lockstep, since discovery scans the same expansion the renderer emits. + +- 2026-08-06 — Typed re-rooting, host-grain ordering, and one ORDER BY resolver (DEV-1747, PR 4 of the DEV-1742 consolidation). (1) **A pre-bound seam** (`slayer/engine/prebound.py`). `plan_query` split into `bind_query_inputs` → `PreboundQuery` and the planning that consumes it, so a nested sub-plan is handed TYPED, already-bound inputs instead of a regenerated formula string. `StrictQueryCarrier` forbids extra attributes and raises on any field the seam does not approve, which is what makes "the sub-plan reads only what was handed to it" checkable rather than aspirational; the negative proof is that `cross_model_planner` no longer imports `bind_expr` / `bind_filter` / `parse_expr` at all. (2) **Host-grain aggregates** (`AggregateKey.grain="host"`). A path-bearing source always routed to a TARGET-rooted CTE, which for a sort key degenerates to a scalar CROSS JOIN — every group gets the same global value and the sort silently does nothing, so the case was rejected outright instead. The marker separates where a value is READ (through the join) from where it is GROUPED (per host row-group), and the decision lives in `classify_isolation` with the other two. This was impossible before the seam, not merely tidier: formula text cannot express a path-bearing source or a grain marker, and routing one through the old text path bound `name:min` against the wrong model. (3) **Direction-aware wraps.** An undeclared row column in a grouped query wraps as `min` for ASC and `max` for DESC — the extreme the direction actually puts first. The unconditional `MAX` sorted an ascending query by each group's LARGEST member, which differs whenever groups overlap in range. (4) **One order-term resolver** (`slayer/sql/render/order_terms.py`), dispatching on the plan-assigned `OrderScope` with no precedence and no fallback. Four render sites had each re-derived the answer and disagreed on three things, each a bug: an unresolvable slot returned unsorted rows with no error anywhere on four of the five paths; null ordering skipped the dialect strategy on two, so the same query sorted NULLs differently depending on whether it carried a transform; and a PROJECTED cross-model aggregate was named by its CTE column while the SELECT projected it under the user's alias, so the term resolved only by falling through to an input column of the FROM. Null ordering is now nulls-last everywhere by policy (T-SQL excepted, where the portable emulation puts a bracketed alias inside a CASE that re-resolves against the FROM and fails); a WINDOW's internal `ORDER BY` deliberately takes the emitter's NATIVE ordering instead, since an emulation term inside a frame changes which rows the frame covers. (5) **A crossing sort key is a crossing ref.** An ORDER-BY-only target is not in `base_render_order` — materialising it there would project it and change the grain — but Law 1 does not care that ORDER BY is the only thing referencing it, so the derived-dimension scope pass walks order targets and the render-time throwaway-`ScopeFrame` probe that existed solely to DETECT the crossing is gone. (6) **The transform chains hand the assembler AST.** Both were still splicing `WITH` out of f-strings and reading each predecessor positionally (`ctes[-1][0]`); they now declare dependencies, and the bodies stay `exp.Select` end to end because this path carries dotted `.` names that a text round trip re-reads as multi-part references. (7) **Audit and routing are different statements.** The re-rooting fix that stopped blanking the routing lists was right about `applied_filter_ids` (an audit: some scope evaluates this) and wrong about `where_filter_ids` / `having_filter_ids` (an instruction: the forward CTE took it over, so the host base must skip it). A re-rooted plan has no forward CTE and its predicate is host-evaluable by construction, so keeping the routing told the host to skip a filter nothing else applied there, and rows the user excluded came back carrying a NULL measure. Only the integration suite could see it — the unit tests assert on plan fields, and the plan was self-consistent. + +- 2026-08-06 — A ROW-phase host filter applies at the host base AND in the cross-model CTE (DEV-1747 B6, second instance). The generator unioned every plan's `where_filter_ids` into one `routed_ids` set and skipped those at `_base`, on the reading that a filter "moved" to a `_cm_` CTE. It does not move: the CTE is joined back with a LEFT JOIN on the query grain, which propagates a VALUE but never an EXCLUSION — a host row whose group the CTE filtered away does not disappear, it arrives with a NULL measure. So the predicate silently became "blank out their measure" instead of "exclude these rows". Because the set was a union across ALL plans, it was also a cross-plan defect: a forward plan routing its filter made the host skip it for a re-rooted sibling that needed it, so `filters=["customers.regions.name == 'Alpha'"]` returned four regions, three of them NULL. Only AGGREGATE-phase (`having_filter_ids`) predicates are genuinely un-evaluable at the host — they reference an aggregate that does not live in `_base`, and trying raises the stage-7b.12 `NotImplementedError`. ROW-phase ones are host-evaluable by construction (they were bound against the host), double-applying is free (the CTE's copy narrows the aggregate, the host's copy narrows the rows), and it restores P-G stated over ROWS: adding a cross-model measure no longer changes which rows a filter keeps. The hazard this could have introduced — the host's copy pulling a 1:N join into the base FROM and multiplying a sibling measure — does not occur, and is pinned by a test, because the host already applies host filters on joined paths that way when no cross-model measure is present; the two cases now simply agree. Present identically at the merge base, so not a regression from the re-rooting work — the same bug class one route over, found by taking a Codex review's prediction that a per-plan fix leaves a cross-plan hole and constructing the query it described. + +- 2026-08-07 — `first`/`last` compiles to a plan-shaped isolated CTE (DEV-1748 B9). One first/last anywhere in a query used to wrap the ENTIRE host base in a `ROW_NUMBER` subquery, so every sibling aggregate was computed over the ranked row set and every ranking in the query shared one scope. The whole rn-suffix scheme (`_last_rn_2`) and the filtered sentinel columns (`_last_rn_f0` plus a `_match_f0` flag the outer aggregate consulted by ALIAS) existed only to keep those rankings apart inside that one scope. Needing its own ROW ORDERING is one of the three things P-C says an aggregate can need its own rows for, so it now isolates for that reason — `RankedAggregatePlan`, a `_rk_` CTE rooted where its rows live, joined back on the query grain null-safely, beside `_cm_` and `_wm_`. One aggregate per scope leaves the suffix bookkeeping nothing to disambiguate and makes the filtered form a plain `WHERE` applied BEFORE the ranking, which is the same answer by construction rather than by careful alias lookup. Six consequences worth naming. (1) **The CTE aggregates, it does not filter.** `MAX(CASE WHEN rn = 1 THEN v END) … GROUP BY grain`, deliberately not `SELECT v … WHERE rn = 1`: the two agree on every non-empty grain and disagree on the empty one, where the aggregate returns ONE row holding NULL and the filter returns none — and an empty grain is joined back with a CROSS JOIN, so a zero-row CTE erases the whole result instead of yielding a NULL measure. (2) **A re-rooted cross-model first/last keeps its `CrossModelAggregatePlan`**; its ranked plan belongs to the nested sub-plan. That sub-plan is rendered as a complete statement and spliced into a CTE body, and SQL Server rejects a `WITH` inside a CTE definition — no sub-plan had ever contained an isolated aggregate before, so a re-rooted first/last would have been the first to emit one. When a sub-plan's only isolated aggregate IS its answer at its own grain, it is emitted directly as the ranked SELECT. (3) **The ranking column is plan data** (`RankedAggregatePlan.ranking_time_key`), resolved once with an explicit per-scope precedence; it used to be re-derived at render time by two different precedences that agreed only on the cases anyone had tried. It carries no declared-type CAST — an ordering key is compared only to itself, and on SQLite `TIMESTAMP` has numeric affinity, so `CAST(DATE(created_at) AS TIMESTAMP)` truncates every date to its year and ties the whole partition. (4) **The DEV-1476/DEV-1526 remnant closes.** A time arg that is a derived column on a JOINED model raised `NotImplementedError`, because the ranking ran in the host base and could not pull the residual join. A ranked CTE resolves its ranking key through its OWN scope, so the join registers where it is needed. (5) **A HOST column named as the ranking key of a TARGET-rooted first/last is now an error** at plan time, naming the measure. It used to emit `ORDER BY .` — a reference to a column that does not exist on the relation it names — and fail at the database with nothing pointing at the cause. (6) **The sibling benefit is vacuous and is not claimed.** A first/last whose inputs cross a join already isolated under DEV-1709's widened trigger, and a purely-local one ranks over the same rows the host base holds, so no sibling's value moves. It is pinned as strict parity. The superseded machinery is UNREACHABLE as of this PR and deleted in PR 6 (P-J); a runtime probe over the whole suite confirms no render path enters it except the direct unit calls that pin it. Two helpers are still CALLED but only on their no-op path — `_resolve_explicit_time_col` / `_explicit_time_arg_of` run for every aggregate through the render-spec builder and return `None` on every production call, because no first/last reaches that builder any more; PR 6 removes the branch rather than the function. + +- 2026-08-10 — P-G call-site migration: the five live render families route through `render_value_key` (DEV-1763, PR 5.5 of 6). All five per-path `ValueKey` renderers in `generator.py` (filter WHERE/HAVING, alias-environment, outer-wrapper, AGGREGATE composite, cross-model target-scope) now emit through the single `slayer.sql.render.value_expr.render_value_key(key, ctx)`, with byte-identical SQL (the DEV-1745/1747/1748 golden baselines are untouched; SQLite+DuckDB integration re-run). The legacy renderers survive PRODUCTION-UNREFERENCED (P-J state 1) with their direct pinning tests green — PR 6 (DEV-1749) deletes them. Design points. (1) **The context is per-concern, not per-call-site.** `RenderContext.scope` becomes Optional (the alias-environment and outer-wrapper families carry no scope and resolve every leaf from the alias maps; a scope-needing key fails closed with `RenderContextMissingFacilityError` rather than dereferencing `None`). `FilterFacilities` drops the dead `first_last_state` field and gains `agg_builder` (the HAVING seam: the renderer does the `slot_by_key` lookup + `having_full_alias` recovery — placeholder `__having_ref__` when the slot was not materialised in the base SELECT — then hands `(key, slot, having_full_alias)` to `_build_agg`, which renders the aggregate as its EXPRESSION so HAVING works where SELECT aliases are rejected), `cast_column_sql` (the filter-side `_filter_cast_type` policy — temporal suppressed — applied to derived `ColumnSqlKey` leaves; the filter family casts, target-scope does not), and `paren_comparison_operands` (the DEV-1539 `_paren_if_binary` extra grouping, moved into the render layer as a named policy). `AliasFacilities` gains `table_by_slot_id` and, when present, switches the five slotted kinds to ALIAS-EXCLUSIVE resolution (a miss RAISES — rebuilding from source in a CTE that only projects aliases is wrong SQL). (2) **The aggregate seam is byte-identical by construction.** `_build_agg` (the `AggRenderSpec` dialect emitter) is NOT superseded; the composite structure renders through `render_value_key` while the aggregate leaf delegates to it via a generator closure. (3) **`_filter_cast_type` / `_wrap_cast_for_type` move into the render package** so the CAST policy is renderer-visible; `generator.py` re-exports both, so its 26 internal call sites and their pinning tests are unchanged. (4) **Two escape hatches** stay for production-dead first/last code (probe: `FirstLastRenderState` at 0 production calls): the composite renderer keeps its `_build_first_last_base_select` caller, and `_collect_routed_filters` keeps a `first_last_state is not None` branch on the legacy target-scope renderer. Both are inventoried and deleted in PR 6. (5) **Aligned on fail-closed over quirk-replication:** a bare row-column operand inside an AGGREGATE composite now raises the fail-closed `RenderContextMissingFacilityError` (a `ValueError`) instead of the legacy terminal `NotImplementedError` — both REJECT (the security-relevant contract is "raises, never stringified"); the three DEV-1733 production-path guards widened to accept either type. (6) **State-1 is machine-checked** by `tests/test_dev1763_call_site_migration.py`: per-family runtime raising-sentinels plus a static `ast` walk asserting each legacy renderer's exact allowed external-reference set (none for filter/aliases/outer-wrapper; the one dead-code site each for composite/target-scope). + +- 2026-08-11 — Routed `ColumnSqlKey` paths validate symmetrically with `ColumnKey` in the cross-model target-scope renderer (DEV-1769, a DEV-1763 follow-up). When a filter is routed into a cross-model CTE its column leaves re-root to the CTE-local scope; the two column-leaf kinds validated their host-rooted path asymmetrically. `ColumnKey` rejected an intermediate-hop path (`path[-1] != target_relation`) with a stage-7b.12 `NotImplementedError`, but `ColumnSqlKey` checked ONLY `model == target_model.name` and never validated the path — any path was silently stripped to `()` and the derived column expanded rooted at the target. **Verdict: the silently-accepted shape (`model == target` yet `path[-1] != target`) is unreachable for keys the binder produces, so it is asserted rather than newly supported (option (b) of the ticket).** The binder builds every non-empty-path `ColumnSqlKey` with `model == path[-1]` (the terminal hop of the resolved join walk, `binding.py:_resolve_dotted`), and every routed-filter call site passes `target_relation == target_model.name` (`plan.target_model` / `plan.root_model`, both bare model names — never a `__`-mangled alias); `reroot_value_key` only strips path prefixes, preserving the terminal hop. So `model == target ⟹ path == () or path[-1] == target`, and the hole admits only hand-built / deserialized / rewritten / future-inconsistent keys. The fix adds the mirrored guard AFTER the existing model-ownership check (ordering preserves the reachable model-mismatch shape's exact message, so byte-identical SQL for every shape the planner routes today is unchanged — verified: the reachable derived-owned-by-another-model and plain-intermediate-hop cases keep their DEV-1450 messages) in BOTH `_reroot_routed_leaf` (the live P-G path) and `_render_filter_value_key_in_target_scope` (the legacy first/last escape hatch PR 6 deletes), raising a distinct `DEV-1769` `NotImplementedError`. Coverage in `tests/test_dev1769_routed_filter_path_validation.py`: end-to-end for the reachable shapes (multi-hop filter ending at the target renders; intermediate-hop and other-model filters raise the pre-existing messages — the two-hop aggregate `customers_v2.regions.population:sum` makes the CTE target `regions` so a path can end at or short of it) and direct-call for the binder-unreachable inconsistent key against both renderers (the only tests red before the guard). No construction-time `keys.py` validator was added — the invariant is load-bearing only at this seam, and a repo-wide constructor guard is a larger change deferred out of this low-priority ticket. + +- 2026-08-11 — Deletion, sweep, docs: P-J states 2+3 executed (DEV-1749, PR 6 of 6, closing the DEV-1742 consolidation). Every mechanism PRs 1–5 + DEV-1763 left production-unreferenced (P-J state 1) is now **deleted** together with its pinning tests (state 3), after confirming the desired behaviour is pinned by tests on the new code. Removed from `generator.py`: the five legacy per-path `ValueKey` renderers and the three arithmetic composer shims (`render_value_key` / `render_arithmetic` are the sole paths, P-G); the first/last host-base ranked machinery — `_build_first_last_base_select`, `_build_ranked_subquery_from_planned`, `_has_first_last_aggregate`, the rn-suffix + filtered-rn/match maps and raw-filter-leak fallback in `_build_agg`, `FirstLastRenderState` and its `first_last_state` threading, and the production-dead `is_first_or_last` arm of `_render_cross_model_cte` (first/last is a `RankedAggregatePlan` CTE since DEV-1748, P-C); the Mode-A model-filter qualify chain (`_render_model_filter_sql`, `_qualify_mode_a_sql_filter`, `_render_mode_a_predicate`, `_filter_join_paths`, `_expand_degenerate_derived_root`, `_column_ref_is_derived`, `_predicate_references_derived`) plus the dead `FilterPhase.text_columns` field — the Mode-A door (`ScopeFrame.enter_predicate`, DEV-1745) is the one door, P-A; the four legacy ORDER BY resolvers (`_build_combined_order_by_sql`, `_resolve_combined_order_term`, `_planned_order_by_sql`, `_apply_order_limit_from_planned`) superseded by `resolve_order_term`; `_null_safe_join_pair_sql` (string round-trip superseded by `render/joins.py`, P-I); `_build_transform_sql`+`_SELF_JOIN_TRANSFORMS`, `_build_outer_wrap`+`_strip_trailing_pagination` (planned outer-wrap delegates to `SqlDialect.emit_outer_wrap`, P-H), and `_cte_name_from_alias` (superseded by `naming.cte_name_from_alias`, P-F). Removed from `cross_model_planner.py`: the formula-text re-rooting island (`_local_agg_formula`, `_render_ref_formula`, `_scalar_formula_literal`, `_reroot_ref`, `_host_ref_path`, `_REROOT_BIND_ERRORS`) and two further dead helpers (`_classify_subplan_filters`, `_filter_ref_paths`) — cross-model re-rooting is typed keys end to end (`reroot_aggregate_key` / `reroot_value_key`), no text round-trip, P-E. **B12 (ratified):** `_build_agg`'s dispatch now reads the single `AGG_REGISTRY` classification table (DEV-1744) — `_AGG_FUNCTION_MAP` (with its dead `COUNT_DISTINCT`/`MEDIAN` string values), the second inline class map, and the generator-local stat-name frozenset are gone; the two-phase resolution order (own-inner builders → shared inner + filter wrap → distinct/median/simple) is preserved byte-for-byte, SQL-identical. **Consolidations landed:** transform-op registries single-sourced (`RANK_FAMILY_TRANSFORMS`, `TIME_TRANSFORMS` from `core/formula.py`), the cube identifier regex re-pointed at `core/refs.IDENTIFIER_RE`; the `_bare_column_refs` regex kept for its remaining validation role only. **Deferred to DEV-1777** (pure refactors, no SQL change, carry regression risk needing their own byte-identity checks): step-CTE emission extraction, the throwaway-`ScopeFrame` consolidation + `_resolve_explicit_time_col` dead-branch removal, and the positional-index couplings. The remaining bare-identifier divergences (`schema_drift`, `.isidentifier()` sites) went to DEV-1771. `docs/architecture/sql-generation.md` was rewritten as the P-A – P-J principles document. Every deleted symbol's docstring/comment references were swept to present-tense truth across 16 files. No emitted SQL changed; the full non-integration suite is green (3 xfails, all ticketed — DEV-1752 / DEV-1729 / DEV-1445 — retained per the F18 inventory). +- 2026-08-12 — Single-source bare-identifier detection narrowed to `schema_drift` only (DEV-1771). Of the three ad-hoc detectors the issue named, only `schema_drift._is_bare_identifier` was re-pointed at the canonical `core/refs.IDENTIFIER_RE` (its char-loop `all(c.isalnum() or c == "_")` accepted non-ASCII in every position; the regex rejects only a non-ASCII *leading* char, since `\w*` still matches Unicode after the ASCII-only lead class). The two `stage_planner` `.isidentifier()` sites (`_saved_model_measure_type`, `_bare_saved_measure_name`) were left as-is: they gate `get_measure(name)` and `ModelMeasure.name` is already ASCII-constrained by `_NAME_PATTERN`, so any string that could match a real measure passes both predicates identically — the flip is observationally a no-op, so no test could fail without it. The two `generator` `.isidentifier()` sites (`_resolve_sql`; the no-bundle branch reached via the `col.name` fallback) were also left as-is: `Column.name`/`Column.sql` are NOT ASCII-constrained, and routing a non-ASCII-leading physical column (e.g. Cyrillic `год`, common in RU/UA schemas) through `_parse()` drops the model-relation qualifier (`m."год"` → bare `год`) — a correctness regression on legitimate input, exactly the issue's "surface and stop" guard. The one shipped behavior change is a drift false-negative: a base column aliasing a bare non-ASCII-leading physical name (`Column(name="year", sql="год")`) reclassifies base→derived, so a dropped physical `год` is no longer flagged by `_diff_sql_table_columns` and is instead scanned as a ref by `_first_dropped_sql_column_ref` (author-accepted; advisory-only, never query correctness). `.match()` (not `fullmatch`) and the retained `.strip()` match the existing `cube`/`dbt`/`osi` reuse sites. +- 2026-08-16 — Aggregated slot-type and display-format inference share one classifier (DEV-1788, follow-up to DEV-1784's Option A). `aggregated_type` (slot `DataType`) and `_infer_aggregated_format` (response `NumberFormat`) had disagreed on the stat/parametric family: type said `DOUBLE` while format fell through to inherit the source column's format, so `revenue:stddev_samp` was typed `DOUBLE` yet displayed as currency. Both now read a single `classify_aggregation` (`core/enums.py`) returning one of four `AggregationValueClass` buckets, and each function maps the bucket to its own output — no per-name branching survives, so the two axes cannot drift. The four builtin frozensets (`INTEGER_AGGREGATIONS`, `PRESERVING_AGGREGATIONS`, `FLOAT_SOURCE_UNIT_AGGREGATIONS`, `FLOAT_PLAIN_AGGREGATIONS`) partition `BUILTIN_AGGREGATIONS`, pinned by a completeness test; custom/model-defined aggregations hit the `PRESERVING` fallback (inherit type & format), unchanged. **Semantics chosen (Option B, unit-correct):** `avg`/`median`/`weighted_avg`/`percentile`/`stddev*` are `DOUBLE` but keep the source's UNITS, so display format inherits the source (falling back to `FLOAT` when the source has none — keeping type `DOUBLE` and format `FLOAT` coherent for unformatted measures, and confining the change to formatted ones); `corr`/`var*`/`covar*` are dimensionless/squared/product units, so they display as plain `FLOAT` regardless of source. `aggregated_type` is behaviourally unchanged (only restructured). **Net user-visible change, all in `_infer_aggregated_format`:** avg-family of a FORMATTED measure now inherits that format (was `FLOAT`); `corr`/`var*`/`covar*` now `FLOAT` (was inherit); `stddev*`/`percentile` unchanged (already inherited). Drift guard extended to the full four-bucket table and routed through the public callers (`measure_key_type` / `measure_key_format_description`), plus response-metadata assertions for the stat/parametric family. diff --git a/docs/architecture/cross-model-aggregates.md b/docs/architecture/cross-model-aggregates.md index 34b07932..2535dda6 100644 --- a/docs/architecture/cross-model-aggregates.md +++ b/docs/architecture/cross-model-aggregates.md @@ -66,7 +66,7 @@ CTE back without changing cardinality. ### Rerooting the aggregate's embedded references (`reroot_aggregate_key`) When the forward `_cm_*` CTE (`_render_cross_model_cte`), its HAVING route, and -the re-rooted-plan formula (`_local_agg_formula`) render a cross-model aggregate +the re-rooted plan render a cross-model aggregate in its target scope, every reference embedded in the `AggregateKey` — the `source`, positional `args` (e.g. the `first`/`last` explicit time arg), keyword `kwargs` values (e.g. `weighted_avg(weight=…)`), and `column_filter_key` — must @@ -110,10 +110,10 @@ flowchart TB ``` It re-roots each host dimension/time-dimension/filter from the host's perspective -to the target's (`_reroot_ref`: host-local → `.`; on-target → bare; -through-target → strip the prefix), drops anything unreachable from the target -(matching legacy), reconstructs the local aggregate formula -(`_local_agg_formula`), builds a fresh `SlayerQuery` rooted at the target, and +to the target's (`reroot_value_key`: host-local → `.`; on-target → +bare; through-target → strip the prefix), drops anything unreachable from the +target, re-anchors the aggregate as a typed key (`reroot_aggregate_key`), builds +a fresh `SlayerQuery` rooted at the target, and compiles it via the injected `subplan_builder` callback (which `plan_query` supplies as a `plan_query` recursion — keeping `cross_model_planner.py` free of a `stage_planner` import). The sub-plan is rendered by @@ -171,8 +171,9 @@ The trigger predicate is structural: `agg_path` non-empty (forward cross-model, target-rooted) **OR** any crossing input (host-rooted). Both route through `IsolatedCteCrossModelPlanner.plan`; the host-rooted branch -calls `_plan_filtered_local`, which rebuilds the measure's formula text -via `_local_agg_formula` (round-trip-tested for every input shape) into a +calls `_plan_filtered_local`, which carries the EXISTING typed `aggregate_key` +unchanged (the sub-plan is rooted at the SAME host model, so there is nothing to +re-root) into a **host-rooted** nested `PlannedQuery` (same `source_model`, same dims/TDs, only the crossing measure as the single aggregate) and attaches it via the same `rerooted_plan` / `rerooted_grain_pairs` / `rerooted_agg_slot_id` @@ -210,18 +211,14 @@ the leaves' projected aliases. This holds for projected composites, filter-only composites (routed to the outer WHERE), and order-only aggregate refs. -### Law 2 inside the ranked scope +### first/last is a route of its own -The first/last ranked subquery re-exports only `source_relation.*` plus -rank/`_td`/`_dim` columns, so any crossing expression the outer SELECT -consumes — an aggregate SOURCE or a column-ref KWARG value — is -materialised as a `_val_` projection inside the subquery -(`_build_first_last_base_select`, mirroring the Stage-4 CTE path). The -projection is the **resolved** value (qualified, with the `Column.type` -inner CAST for non-bare expressions), so `SUM(CAST(x AS t))` semantics are -preserved and same-sql-different-type aggregates keep distinct -materialisations; HAVING and composite consumers bind to the same alias -via `FirstLastRenderState.value_alias_by_sql` (keyed by resolved text). +A `first`/`last` aggregate does NOT reach the cross-model route. It needs its +own ROW ORDERING, so it isolates for that reason — which subsumes the +crossing-input trigger — and compiles to a `_rk_` CTE. See +[Ranked aggregates](ranked-aggregates.md). The one exception is a **re-rooted** +cross-model `first`/`last`, which keeps its `CrossModelAggregatePlan`: the +ranked plan belongs to its nested sub-plan. ### Filter routing for filtered-local @@ -280,19 +277,18 @@ register their joins in a **pre-pass** that walks the full `ValueKey` tree `column_filter`) before the `FROM` is built, so a HAVING — rendered later, once the ranked-subquery rank columns exist — still contributes its joins. -**First/last value materialization (Law 2).** When the CTE wraps its rows in a -`ROW_NUMBER`-ranked subquery and the first/last **source value** crosses a join, -the crossing value is materialized as a `_val_` projection *inside* the -subquery and the outer `MAX(CASE WHEN _last_rn = 1 THEN _val_ END)` references -the alias — a raw crossing ref there is bound only inside the subquery. A HAVING -on the same aggregate binds the same alias. The **shared grain** obeys the same -law (DEV-1728): a crossing derived grain is materialized as a `_val_` -projection inside the subquery, the outer `SELECT`/`GROUP BY` reference the -alias, and `PARTITION BY` keeps the raw expression (evaluated where the join is -bound). `generate_from_planned` installs one generation-wide `AliasAllocator` -(save/restore) so inline forward CTEs, grain projections, and the host base -never collide on `_val_`; the CTE reserves the target's physical column names -so a minted alias never shadows a `target.*` column. +**Projection boundaries (Law 2).** A `_cm_` CTE consumes a value from an inner +scope by ALIAS, never by re-reaching for the expression: a ref that crosses a +join is bound only where that join is, so the inner scope projects it as +`_val_` and the outer names the alias. The projection is the **resolved** +value (qualified, with the `Column.type` inner CAST for non-bare expressions), +so `SUM(CAST(x AS t))` semantics are preserved and same-sql-different-type +aggregates keep distinct materialisations. `generate_from_planned` installs one +generation-wide `AliasAllocator` (save/restore) so inline forward CTEs, grain +projections, and the host base never collide on `_val_`; each CTE reserves +its root model's physical column names so a minted alias never shadows a real +column. The ranked (`_rk_`) route obeys the same law through the same +`ScopeFrame` table — see [Ranked aggregates](ranked-aggregates.md). ### Derived shared-grain rendering (DEV-1728) diff --git a/docs/architecture/index.md b/docs/architecture/index.md index a052ba03..135d4921 100644 --- a/docs/architecture/index.md +++ b/docs/architecture/index.md @@ -211,18 +211,19 @@ the kind of multi-path coupling the redesign set out to remove. - A `SlayerModel.filters` entry, OR a column-level `Column.filter` on an aggregated measure, referencing a non-trivial derived column. `_validate_model_filter` no longer rejects the model-filter form; the - generator inline-expands the predicate (the shared - `_render_mode_a_predicate`, used by both `_render_model_filter_sql` and the - `Column.filter` CASE-WHEN path) and pulls any join the expansion crosses - into the FROM. Inlining covers a bare derived ref (`is_eu` → + generator enters the predicate through the scope's Mode-A door + (`ScopeFrame.enter_predicate`), shared by both the `SlayerModel.filters` + WHERE term and the `Column.filter` CASE-WHEN path, which inline-expands + derived references and pulls any join the expansion crosses into the FROM. + Inlining covers a bare derived ref (`is_eu` → `customers.region`) **and** a dotted ref to a derived column on a joined model (`loss_payment.has_flag` → its `sql`), matching the query-level filter path so no dangling `.` (a non-physical column) - is emitted (DEV-1494). Join discovery for these Mode-A text filters - (`_filter_join_paths`) unions the paths of the **un-inlined** predicate - (so the dbt placeholder-join idiom — a constant `has_flag sql="1"` whose - only purpose is to force the join — keeps its alias) with the paths the - **inline-expanded** predicate crosses. The cross-model `_cm_*` CTE discovers + is emitted (DEV-1494). Join discovery for these Mode-A text filters is a + side effect of entering the door, and unions the paths of the **un-inlined** + predicate (so the dbt placeholder-join idiom — a constant `has_flag + sql="1"` whose only purpose is to force the join — keeps its alias) with the + paths the **inline-expanded** predicate crosses. The cross-model `_cm_*` CTE discovers its OWN filter joins too — the target measure's `Column.filter` and the target-model filters — and adds them to the CTE's FROM (each `_cm_*` CTE is an isolated per-(target, grain) computation, so the join resolves the diff --git a/docs/architecture/ranked-aggregates.md b/docs/architecture/ranked-aggregates.md new file mode 100644 index 00000000..6511c645 --- /dev/null +++ b/docs/architecture/ranked-aggregates.md @@ -0,0 +1,128 @@ +# Ranked aggregates (`first` / `last`) + +`first` and `last` answer "the value from the row that sorts first (or last) +within each group". That needs a row ORDERING, which is one of the three things +an aggregate can need its own rows for — alongside crossing a join and carrying +its own frame. Under the isolation doctrine (P-C) all three compile the same +way: **a plan-shaped CTE, rooted where its rows live, joined back on the query +grain**. For `first`/`last` that is `RankedAggregatePlan` and a `_rk_` CTE, +beside `_cm_` (cross-model) and `_wm_` (windowed). + +## What it looks like + +```sql +WITH _base AS ( + SELECT orders.status AS "orders.status", + CAST(SUM(orders.amount) AS DOUBLE PRECISION) AS "orders.s" + FROM orders AS orders + GROUP BY orders.status +), _rk_orders__l AS ( + SELECT _val_0 AS "orders.status", + CAST(MAX(CASE WHEN _rk_rn = 1 THEN _val_1 END) AS DOUBLE PRECISION) + AS "orders.l" + FROM ( + SELECT orders.status AS _val_0, + orders.amount AS _val_1, + ROW_NUMBER() OVER ( + PARTITION BY orders.status ORDER BY orders.created_at DESC + ) AS _rk_rn + FROM orders AS orders + ) AS _rk_src + GROUP BY _val_0 +) +SELECT _base."orders.status", _rk_orders__l."orders.l", _base."orders.s" +FROM _base +LEFT JOIN _rk_orders__l + ON _base."orders.status" IS NOT DISTINCT FROM _rk_orders__l."orders.status" +``` + +Two SELECTs. The inner one ranks the rows this aggregate is allowed to see; the +outer picks rank 1 per grain. The host base holds only purely-local aggregates, +so adding a `first`/`last` cannot change host cardinality or any sibling's +value. + +## Where each decision is made + +| Decision | Owner | +| --- | --- | +| Does this aggregate isolate, and where is it rooted | `classify_isolation` → `IsolationKind.RANKED_HOST` / `RANKED_TARGET` | +| Which column the ranking orders by | `ranked_planner.resolve_ranking_time_key` | +| What the grain is, in both coordinate systems | `RankedAggregatePlan.grain` | +| Which filters the CTE evaluates | the plan's `where_filter_ids` / `target_model_filters` | +| What the SQL looks like | `slayer/sql/render/ranked.py` | + +The renderer emits a plan; it re-derives none of the above (P-D). + +## The ranking column + +Resolved at plan time, per scope, and it raises at the end rather than falling +through: + +**Host-rooted** — an explicit positional time arg (`amount:last(shipped_at)`), +else the first `DATE`/`TIMESTAMP` row dimension, else the first time dimension's +**raw** column (never the truncated bucket: ranking within a month by the month +ties every row in it), else the model's `default_time_dimension`. + +**Target-rooted** — the same explicit arg, re-anchored in the target's +coordinates, else the TARGET model's `default_time_dimension`. Host dimensions +are deliberately not candidates: the CTE ranks target rows and a host column is +not one of their attributes. Naming one is an error, reported at plan time. + +The ranking key carries no declared-type CAST. It is compared only to itself, +and on SQLite `TIMESTAMP` has numeric affinity — `CAST(DATE(created_at) AS +TIMESTAMP)` truncates every date to its year and ties the whole partition. + +## Why the CTE aggregates instead of filtering + +`MAX(CASE WHEN _rk_rn = 1 THEN v END) … GROUP BY grain`, not `SELECT v … WHERE +_rk_rn = 1`. The two agree on every non-empty grain and disagree on the empty +one: over a source with no rows the aggregate form returns ONE row holding +NULL and the filter form returns none. An empty grain is joined back with a +`CROSS JOIN`, so a zero-row CTE erases the entire result rather than yielding a +NULL measure — and one NULL row is what `amount:sum` returns over the same +empty source. + +## Filtered variants are plan data + +A measure's `Column.filter` is a predicate on the rows this aggregate ranks, so +in its own scope it is simply a `WHERE`, applied **before** the ranking. Two +filtered measures in one query are two scopes with one predicate each. + +## Filter routing + +| Filter phase | Where it is applied | +| --- | --- | +| ROW | the host base **and** the ranked CTE | +| AGGREGATE (references the ranked value) | the outer combined SELECT's `WHERE` | +| POST | the existing post-transform wrapper | + +A ROW-phase filter is duplicated rather than relocated: the CTE is LEFT JOINed +back, which propagates a VALUE but never an EXCLUSION, so a predicate applied +only in the CTE would silently become "blank out their measure" instead of +"exclude these rows". An AGGREGATE-phase one cannot be a `HAVING` inside the +CTE for the mirror-image reason — dropping the CTE row resurrects the host row +carrying NULL. + +## Cross-model and re-rooting + +A `first`/`last` whose source names another model roots its CTE at that target +(`RANKED_TARGET`). The forward-path filter routing is the cross-model planner's +decision table, taken verbatim; only the ranking column and the grain are +computed in the target's coordinates. + +A **re-rooted** cross-model `first`/`last` keeps its `CrossModelAggregatePlan`: +its ranked plan belongs to the nested sub-plan, in the sub-plan's own +coordinates. That sub-plan is rendered as a complete statement and spliced into +a CTE body, and SQL Server rejects a `WITH` nested inside a CTE definition — so +when a sub-plan's only isolated aggregate IS its answer, at its own grain, it is +emitted directly as the ranked SELECT rather than as `_base` plus a combined +SELECT around it (`_collapses_to_ranked_cte`). + +## Internal names + +`_rk_rn` (the rank column) and `_rk_src` (the inner subquery) are private to one +CTE scope and never reach a result key. They are safe precisely because the +inner SELECT projects a **named** list rather than `.*`: a physical +column called `_rk_rn` is never re-exported, so it cannot capture the rank +column's reference. CTE names are minted by the collision-aware allocator, which +case-folds on dialects whose unquoted identifiers do. diff --git a/docs/architecture/sql-generation.md b/docs/architecture/sql-generation.md index f978c2a3..9cc93521 100644 --- a/docs/architecture/sql-generation.md +++ b/docs/architecture/sql-generation.md @@ -1,11 +1,15 @@ # SQL generation -**Modules:** `slayer/sql/generator.py` (the planned-consuming path), -`slayer/engine/response_meta.py` (response metadata) +**Modules:** `slayer/sql/generator.py` (renders a `PlannedQuery` to SQL), +`slayer/sql/render/` (the shared renderers — value keys, aggregates, order +terms, joins), `slayer/sql/scope.py` (`ScopeFrame`), `slayer/sql/naming.py` +(the allocator), `slayer/engine/response_meta.py` (response metadata). -The generator renders a `PlannedQuery` (or a list of them) to a SQL string. It -preserves the result-key contract exactly (**P10**) and emits SQL via sqlglot -AST building, not string concatenation. +The generator renders a `PlannedQuery` (or a list of them) to a SQL string via +sqlglot AST building, never string concatenation. It is organised around ten +principles (P-A – P-J); the invariant, the code that enforces it, and the key +mechanism are described together under each. The consolidation that arrived at +these is logged in `DECISIONS.md` (DEV-1742). ## Entry points @@ -16,7 +20,7 @@ flowchart TB gps -->|multi-stage| loop["render each stage → CTE; root = outer SELECT"] loop --> gfp gfp --> inst["SQLGenerator(dialect).generate_from_planned"] - inst -->|cross-model| cm["_render_with_cross_model_plans"] + inst -->|cross-model / windowed / ranked| cm["_render_with_cross_model_plans"] inst -->|transforms| tl["WITH base, step CTEs, outer wrap"] inst -->|plain| base["single SELECT"] ``` @@ -28,245 +32,283 @@ flowchart TB multi-stage DAG to one SQL string. Each non-root stage becomes a CTE; the root is the outer SELECT. -## `generate_from_planned` (instance method) - -Reads from typed `PlannedQuery` fields (`row_slots` / `aggregate_slots` / -`filters_by_phase` / `order` / `transform_layers`) and dispatches: - -- `cross_model_aggregate_plans` non-empty → `_render_with_cross_model_plans`; -- `transform_layers` present → `WITH base AS (...)`, Kahn-batched step CTEs - carrying the window functions, an outer wrap projecting in user-spec order; - POST-phase filters that reference transform slots wrap as `SELECT * FROM (...) - AS _filtered WHERE …`; `time_shift` / `consecutive_periods` emit dedicated - self-join CTE pairs; -- otherwise → a single base SELECT with WHERE/HAVING, GROUP BY, ORDER BY, LIMIT. - When the base CTE materialises any hidden aggregate (an aggregate referenced - ONLY by ORDER BY or a filter, never declared as a measure), a conditional - outer-trim wrapper projects exactly the public projection — same shape as the - transform path's outer wrap, minus the step CTEs — so the hidden alias does - not leak into the result columns (DEV-1501). - -It builds its own `slot_id_by_key` map (the `PlannedQuery` doesn't carry the -registry), materializes hidden aux slots referenced as transform inputs / -partition keys / time keys / POST-filter operands, and renders. - -### `AggRenderSpec` — the dialect-helper interface - -To render aggregations identically across all dialects, the shared dialect -helpers (`_build_agg`, `_build_percentile`, `_build_stat_agg`, -`_wrap_cast_for_type`, `_resolve_sql`, `_build_date_trunc`) consume a single -typed input: `AggRenderSpec`, built directly from planned slots by -`_build_agg_render_spec_from_planned`. - -Dialect-specific behavior (SQLite UDFs, ClickHouse `quantile`, the MySQL -`median` `NotImplementedError`, and so on) is therefore emitted by exactly one -code path. - -!!! note "Historical: the synthetic-`EnrichedMeasure` adapter" - - `generate_from_planned` originally consumed `PlannedQuery` at the top but - adapted *back* to `EnrichedMeasure` (`_synthesize_enriched_measure_from_planned`) - to reach the dialect helpers — a hybrid that coupled the new path to a - legacy type, kept deliberately so the two coexisting pipelines could not - drift on dialect SQL. DEV-1452 Stage A retyped the helpers onto - `AggRenderSpec`; DEV-1485 deleted the last adapter - (`_agg_render_spec_from_enriched`) and `_build_agg`'s `measure=` compat - surface with the rest of the legacy stack. - -## Multi-stage chaining (`generate_planned_stages`) - -Each non-root stage renders independently (against a per-stage bundle from -`_bundle_for_stage`) and is wrapped by `_stage_rename_wrapper` so its output -columns become the flat names downstream stages bound against -(`orders.customers.region` → `customers__region`). The wrapper derives those from -the *actual* rendered `named_selects` (robust to the cross-model renderer -emitting columns out of `public_projection` order) and asserts they match the -stage's `StageSchema` — a planner/generator divergence fails here rather than as -a confusing downstream bind miss. Stage CTEs are prepended before any CTEs the -root already emits (the root reads `FROM `). - -`_bundle_for_stage` picks the host model the stage renders against from the -planner's `render_source_model` (the stage's own source / overlay / -synthetic-over-sibling), falling back to a synthetic model over the upstream CTE -for a `StageSchema` chain stage — so the generator's FROM/joins bind against -exactly what the binder used. - -## Cross-model rendering - -`_render_with_cross_model_plans` emits one `_cm_*` CTE per -`CrossModelAggregatePlan` joined back to the host base. When `plan.rerooted_plan` -is set, `_render_rerooted_cross_model_cte` renders the nested re-rooted plan -(FROM target + the target's joins) preserving host grain; otherwise the -forward-path CTE renders (FROM bare target, grouped at the forward dims). -`Column.filter` on the aggregated column renders as -`SUM(CASE WHEN THEN END)`. See -[Cross-model aggregates](cross-model-aggregates.md). - -The same renderer also emits one `_wm_*` CTE per `WindowedAggregatePlan` — a -duration-windowed measure (`revenue:sum(window='90d')`). Unlike `_cm_*` (rooted -at the join target), a `_wm_*` CTE is **host-rooted**: an inner `_src` subquery -self-selects the host rows (dimensions → `_w_dim_`, other time dims → -`_w_td_`, the raw window time column → `_w_time`, the value → `_w_value`) -with its joins discovered through a host `ScopeFrame`, and -`FROM _base LEFT JOIN _src` -pairs the grain equalities with a trailing `INTERVAL` range predicate -(`_src._w_time >= bucket_end − window` / `< bucket_end`). The result groups at -the query grain and joins back to `_base` null-safe, exactly like a `_cm_*` CTE, -so windowed and cross-model measures coexist in one query. Windowed-measure -filters route to the combined-SELECT outer `WHERE` (`Phase.POST`). `sum`/`avg` -local measures only; other shapes raise at plan time (`_guard_windowed_measures` -in `stage_planner.py`). +`generate_from_planned` reads typed `PlannedQuery` fields (`row_slots` / +`aggregate_slots` / `filters_by_phase` / `order` / `transform_layers`) and +dispatches: any `cross_model_aggregate_plans` / `WindowedAggregatePlan` / +`RankedAggregatePlan` present → `_render_with_cross_model_plans`; +`transform_layers` present → `WITH base AS (...)`, Kahn-batched step CTEs, an +outer wrap projecting in user-spec order; otherwise → a single base SELECT with +WHERE/HAVING, GROUP BY, ORDER BY, LIMIT (plus a conditional outer-trim wrap when +the base materialises a hidden aggregate, so the hidden alias never leaks into +the result columns). + +## P-A — One door into a scope + +Every SQL fragment enters a SELECT scope through that scope's resolver, and +join discovery is a **side effect of rendering**, never a separate pass. There +is no string concatenation between scopes — inter-scope assembly is sqlglot AST. + +A `ScopeFrame` (`slayer/sql/scope.py`) is the door. Rendering a `ValueKey` or a +Mode-A text predicate through the frame both resolves the reference (qualifying +bare identifiers against the scope root, joined refs to their `__`-path alias, +reserved-word relations quoted) **and** registers the join paths the reference +crosses onto the frame's `join_paths`, which the FROM is then built from +(`_build_from_and_joins`). Discovery is root-scope-only, so a correlated ref +inside an `EXISTS (...)` subquery does not pull an outer join. + +Mode-A surfaces — a column-level `Column.filter` (`SUM(CASE WHEN THEN + END)`) and a `SlayerModel.filters` WHERE term — enter through the scope's +Mode-A door (`_mode_a_scope` / `ScopeFrame.enter_predicate`). The door +inline-expands references to derived columns (bare `is_eu` → its `CASE WHEN +customers.region …`; dotted `loss_payment.has_flag` → its `sql`) so the emitted +predicate is runnable and never names a non-physical `.`, +and it discovers the crossed joins as it renders. The dbt placeholder-join idiom +— a constant derived column such as `has_flag sql="1"` whose only purpose is to +force the inner join — keeps its join because discovery unions the paths of the +un-inlined and the inline-expanded predicate. + +## P-B — Scopes exchange data only through projected columns + +Cross-scope data flow uses exactly one materialisation mechanism — +`ScopeFrame.resolve(consumer=...)` + `apply_materializations` — with one dedup +key: producing-scope identity + anchored AST + dialect. A value a producing +scope computes once (a crossing grain expression, a first/last value that +crosses a join) is projected under a minted `_val_` alias and the consuming +scope references the alias; the dedup key means the same value is never +projected twice. + +At the stage level, each non-root stage renders independently (against a +per-stage bundle from `_bundle_for_stage`) and is wrapped by +`_stage_rename_wrapper`, which renames its output columns to the flat names +downstream stages bound against (`orders.customers.region` → +`customers__region`). The wrapper derives those from the *actual* rendered +`named_selects` and asserts they match the stage's `StageSchema` — a +planner/generator divergence fails at the boundary rather than as a confusing +downstream bind miss. + +## P-C — One isolation doctrine + +Any aggregate that needs its own rows — crossing inputs, its own row ordering +(first/last), or its own frame (windowed) — is a **plan-shaped CTE** rooted +where its rows live, joined back on the query grain. The host base SELECT +contains only purely-local aggregates, and host cardinality never changes. + +`_render_with_cross_model_plans` emits three CTE kinds, all joined back to the +host base on the query grain: + +- **`_cm_*`** per `CrossModelAggregatePlan` — a measure over a joined model. A + re-rooted plan (`plan.rerooted_plan`) renders FROM the target + the target's + joins preserving host grain; a forward plan renders FROM the bare target + grouped at the forward dims. See [Cross-model aggregates](cross-model-aggregates.md). +- **`_wm_*`** per `WindowedAggregatePlan` — a duration-windowed measure + (`revenue:sum(window='90d')`). Host-rooted: an inner `_src` subquery + self-selects the host rows and `FROM _base LEFT JOIN _src` pairs the grain + equalities with a trailing `INTERVAL` range predicate. `sum`/`avg` local + measures only; other shapes raise at plan time (`_guard_windowed_measures`). +- **`_rk_*`** per `RankedAggregatePlan` — a `first`/`last` measure. Rooted at the + host or the join target, it ranks its own rows (ROW_NUMBER), picks rank 1 per + grain, and joins back on that grain. See [Ranked aggregates](ranked-aggregates.md). + +Since the Law-3 trigger widened (DEV-1709), a LOCAL aggregate with **any** +crossing input — source `Column.sql`, `Column.filter`, positional args, kwargs — +never renders in the top-level host base: it isolates into a host-rooted `_cm_*` +CTE, and its join discovery runs inside that CTE's sub-render. The host base +FROM therefore pulls `LEFT JOIN`s only for purely-local sources — derived +dimension / time-dimension `Column.sql`, local aggregated-measure `Column.filter`, +and local aggregate-source `Column.sql` — each discovered through the scope door +(P-A) and fed to the shared `needed_join_paths` list, deduped by +`_build_from_and_joins`'s `emitted_aliases` guard. ### Frame bounds vs population filters (DEV-1732) -`_src` inherits the host's ROW-phase filters **minus their frame bounds** — a -relational comparison (`<`, `<=`, `>`, `>=`) between a non-hidden time -dimension's raw column and a temporal literal. Without that, the trailing window -cannot reach rows before the earliest visible bucket and that bucket -under-counts; with it, `date_range` and the explicit spelling of the same intent -produce identical numbers. - -`slayer/core/time_bounds.py` owns the analysis (dependency-free, so planner and -generator share it). `stage_planner.plan_query` computes the strippable column -set once into `PlannedQuery.frame_bound_columns` and partitions the filters into -`WindowedAggregatePlan.where_filter_ids` (applied) plus `src_filter_rewrites` -(applied as a residual — a top-level `and` is split so only its frame-bound -conjuncts drop; `or`/`not` are never descended into). The generator's -`_effective_src_filters` materialises that view **once** and feeds the same list -to both join discovery and rendering, so the two cannot disagree about what the -CTE contains. - -Hidden `TimeTruncKey` slots are excluded from `frame_bound_columns` on purpose: -`_build_windowed_plans` skips hidden row slots, so a hidden time axis is never -equality-joined into `_src` and stripping its bound would leave it -unconstrained. Mode-A `SlayerModel.filters` are exempt entirely — they define -which rows exist, not which frame the query looks at. - -The `time_shift` shifted CTE (`_shifted_where_part`) applies the same rule, -reading the same `frame_bound_columns`; its former -`isinstance(..., BetweenKey)` special case is subsumed, since a `date_range`'s -`BetweenKey` column is always a query time dimension's raw column. - -## Mode-A filter inlining and join discovery (DEV-1494) - -A column-level `Column.filter` on an aggregated measure becomes a CASE-WHEN -wrapper (`SUM(CASE WHEN THEN END)`), and a `SlayerModel.filters` -entry becomes a WHERE term. Both are Mode-A SQL and share one renderer, -`_render_mode_a_predicate`, which inline-expands references to derived columns — -bare (`is_eu` → its `CASE WHEN customers.region …`) or dotted to a derived -column on a joined model (`loss_payment.has_flag` → its `sql`) — so the emitted -predicate is runnable and never references a non-physical `.`. -A predicate with only base refs takes the cheap qualify path -(`_qualify_mode_a_sql_filter` regex for model filters, `_qualify_column_filter_sql` -AST for column filters), byte-identical to before. On sqlglot parse failure the -predicate falls through to the qualify path unchanged. - -Join discovery for these text filters (`_filter_join_paths`) is the **union** of -the join paths in the **un-inlined** predicate and those in the **inline-expanded** -predicate. Both are needed: the dbt placeholder-join idiom — a constant derived -column such as `has_flag sql="1"` whose only purpose is to force the (inner) -join — keeps its alias only in the un-inlined form (it inlines to the constant -`(1)`), while a derived ref's *crossed* joins (`is_eu` → `customers`; -`loss_payment.deep_flag` → `loss_payment__claim`) appear only after expansion. -Discovery for column filters in the base SELECT is restricted to **local** -aggregate sources (empty `AggregateKey.path`); a cross-model aggregate's filter -joins are discovered inside its `_cm_*` CTE instead — `_render_cross_model_cte` -collects the join paths of the target measure's `Column.filter` and the -target-model filters and adds them to the CTE's own FROM. Because each `_cm_*` -CTE is an isolated per-(target, grain) computation, adding the join resolves the -filter's refs without affecting sibling measures. Discovery is root-scope-only, -so a correlated ref inside an `EXISTS (...)` subquery does not pull an outer join. - -## Host-base join discovery (the three symmetric sources) - -The host base FROM at `_build_base_select_for_planned` pulls in `LEFT JOIN`s from -three symmetric sources, each handled by a dedicated collector wired in the same -call chain just before `_build_from_and_joins`: - -1. **Dimension / time-dimension `Column.sql`** (DEV-1484): `_expand_derived_row_dims` - pre-expands derived ROW slots (`ColumnSqlKey` dims and `TimeTruncKey` columns - that are themselves derived) and scans the expansion through - `_joined_paths_in_sql`, appending crossed paths to `needed_join_paths`. -2. **Aggregated-measure `Column.filter`** (DEV-1494): `_collect_column_filter_join_paths` - recurses through AGGREGATE-phase composite keys (`ArithmeticKey` / - `ScalarCallKey`) and, for each `AggregateKey` with a `column_filter_key`, - collects the paths the predicate touches via `_filter_join_paths` (the union - of un-inlined and inline-expanded predicate paths, per the section above). -3. **Aggregate-source `Column.sql`** (DEV-1502): `_collect_aggregate_source_join_paths` - mirrors the filter helper — recurses through the same composite keys, and for - each `AggregateKey` whose `source` is a `ColumnSqlKey` with `path == ()`, - expands the column via `_expand_derived_column_sql` and scans the result - through `_joined_paths_in_sql`. The render-time expansion in - `_build_agg_render_spec_from_planned` already produces `SUM()` SQL; - this collector closes the join-discovery loop so a measure source like - `customers__regions.population` emits both `LEFT JOIN`s. - -All three collectors restrict to **local** aggregate sources (empty -`AggregateKey.source.path`); cross-model aggregates own their own join -discovery inside the per-plan `_cm_*` CTE — for the `Column.filter` side -(DEV-1494 / DEV-1503) and, since Stage 4 (DEV-1708 closed DEV-1526), for a -target column's `Column.sql` that crosses a further join. All three -host-side collectors feed the shared `needed_join_paths` list, so repeated -paths surfaced by different sources dedupe naturally via -`_build_from_and_joins`'s `emitted_aliases` guard. - -Since Stage 5 (DEV-1709, widened Law-3 trigger), a LOCAL aggregate with -any crossing input — source `Column.sql`, `Column.filter`, positional -args, kwargs — never renders in the top-level host base at all: it -isolates into a host-rooted `_cm_*` CTE, and the discovery above runs -inside that CTE's sub-render (see -[cross-model-aggregates.md](cross-model-aggregates.md#strategy-3-host-rooted-isolation--any-crossing-input-dev-1503-widened-by-dev-1709)). -The host base only ever contains purely-local aggregates. - -## Result-key contract (P10) - -The generator preserves the result keys byte-for-byte: `orders.revenue_sum`, -`orders._count` (the `*` dropped, the leading `_` kept), joined dimensions as the -full dotted path `orders.customers.regions.name`, and renamed measures as -`orders.`. `_full_alias_for_slot` derives these from the slot's key / -public aliases. Two documented exceptions, both routed through the same -`canonical_agg_name` helper: cross-model parametric aggregates carry the kwarg -suffix legacy dropped, and hidden parametric `first`/`last` (DEV-1501) carry the -explicit time-arg suffix so distinct time-column specs get distinct -materialised aliases (`orders.revenue_last_created_at`, -`orders.revenue_last_updated_at`). +A `_src` (windowed) or shifted (`time_shift`) CTE inherits the host's ROW-phase +filters **minus their frame bounds** — a relational comparison between a +non-hidden time dimension's raw column and a temporal literal. Without that, the +trailing window cannot reach rows before the earliest visible bucket and that +bucket under-counts. `slayer/core/time_bounds.py` owns the analysis +(dependency-free, so planner and generator share it); `plan_query` partitions +the filters into `WindowedAggregatePlan.where_filter_ids` plus +`src_filter_rewrites` (a top-level `and` is split so only its frame-bound +conjuncts drop; `or`/`not` are never descended into), and the generator's +`_effective_src_filters` materialises that view once for both join discovery and +rendering. Hidden `TimeTruncKey` slots are excluded on purpose; Mode-A +`SlayerModel.filters` are exempt entirely — they define which rows exist, not +which frame the query looks at. + +## P-D — Plan decides, render emits + +All classification happens at plan time; the generator consumes the plan +verbatim. It never re-classifies a slot, re-parses user text, or re-walks +filters to decide policy. The isolation decision (does this aggregate cross a +join?), the ranking-time column, the frame-bound partition, and the order-term +scope all arrive as typed fields on `PlannedQuery` / its plans. The generator +builds only the environment those decisions read — for example, the render-time +crossing probe that used to build a throwaway `ScopeFrame` purely to *detect* a +crossing is gone; the crossing is decided at plan time and carried on the order +entry's `OrderScope`. + +When a filter is routed into a cross-model CTE, its column leaves re-root to the +CTE-local scope through `_reroot_routed_leaf`, which validates the host-rooted +path **symmetrically** for both `ColumnKey` and `ColumnSqlKey` (DEV-1769): an +intermediate-hop path — one not ending at the target relation — is rejected for +either kind rather than silently stripped. The `ColumnSqlKey` guard is +unreachable for binder-produced keys (the binder builds `model == path[-1]`, and +every call site passes `target_relation == target_model.name`), so it fails +closed on inconsistent hand-built / deserialized keys. + +## P-E — Identity is structural end-to-end + +Rerooting and isolation operate on typed keys, never by round-tripping through +formula text. A cross-model aggregate re-anchored from the host's coordinate +system into the target's is `reroot_aggregate_key` / `reroot_value_key` +(`slayer/core/keys.py`) producing typed keys directly — there is no +serialize-to-`revenue:sum`-and-re-bind step. Sub-plan filters are classified +structurally against the CTE's own root rather than re-derived from +`routing.text`. + +## P-F — One naming authority + +Every alias and CTE name is minted by the allocator (`slayer/sql/naming.py`); +result keys come from `result_key` / `flat_name`. The allocator reserves the +deterministic CTE families (`_cm_` / `_wm_` / `_rk_` / user CTEs) up front, then +allocates the transform families (`shifted_` / `sjoin_`) around them, so a +transform CTE whose preferred name collides with a reserved one renames rather +than shadows. + +Two ratified carve-outs, recorded rather than silently omitted. The structural +alias constants `_outer` (outer-wrapper subquery — the base `emit_outer_wrap` +and the T-SQL ORDER-BY-detach rewrite), `_stage_inner` (stage-schema wrapper) and +`_filtered` (filtered transform-chain wrapper) name their derived tables directly +rather than being allocator-minted — the T-SQL rewrite is a post-generation AST +pass with no allocator in reach, and each alias scopes a derived table its own +pass creates, so a collision could only arise inside that one subquery. The structural names `base` / `_base` / `_combined` keep their +literal spellings but are reserved into the allocator up front, so a user CTE +that folds onto one of them renames instead. + +### Result-key contract (P10) + +Result keys are preserved byte-for-byte: `orders.revenue_sum`, `orders._count` +(the `*` dropped, the leading `_` kept), joined dimensions as the full dotted +path `orders.customers.regions.name`, renamed measures as `orders.`. +`_full_alias_for_slot` derives these from the slot's key / public aliases. Two +documented exceptions, both through `canonical_agg_name`: cross-model parametric +aggregates carry the kwarg suffix, and hidden parametric `first`/`last` carry the +explicit time-arg suffix so distinct time-column specs get distinct materialised +aliases. + +## P-G — Same construct, same SQL + +A given `ValueKey` renders identically wherever it appears — one +`ValueKey`→AST renderer parameterised by scope context, not four copies. + +Every `ValueKey` tree — WHERE/HAVING predicates, AGGREGATE-phase composites, +POST-phase filters, the DEV-1503 outer combined WHERE, and cross-model CTE +routed filters — renders through `slayer.sql.render.value_expr.render_value_key(key, +ctx)`, parameterised by a `RenderContext`. This is what keeps the same +`ScalarCallKey` from emitting `IFNULL(...)` on one path (invalid on Postgres) +and `COALESCE(...)` on another. The context carries per-concern facilities: + +- **`FilterFacilities`** — the WHERE/HAVING `agg_builder` seam (renders a local + aggregate as its *expression*, not its SELECT alias, so HAVING works on + backends that reject aliases there), the filter-side CAST policy + (`cast_column_sql`), and the comparison grouping (`paren_comparison_operands`). +- **`CompositeFacilities`** — the AGGREGATE-composite `agg_builder`; the + composite *structure* renders through `render_value_key` while the aggregate + *leaf* delegates to `_build_agg`. +- **`AliasFacilities`** — POST-phase / outer-wrapper *alias-exclusive* + resolution: the slotted kinds come back as their materialised alias + (table-qualified via `table_by_slot_id`), never rebuilt from source. + +A missing facility or an unmaterialised slot raises +`RenderContextMissingFacilityError` — the renderer fails closed rather than +degrading quietly, which is how the predecessor copies drifted apart. Arithmetic +and scalar calls likewise route through single functions (`render_arithmetic`, +`render_scalar_call`); the generator no longer carries per-path composer shims. + +### The aggregation registry + +Aggregation classification is a single table, `AGG_REGISTRY` +(`slayer/sql/render/aggregates.py`): each built-in is one `AggEntry` naming the +`dispatch` mechanism (`simple` / `ranked` / `stat` / `dialect_hook` / `distinct` +/ `formula`) and, for the simple path, its sqlglot `node_class`. `_build_agg` +reads the table — `is_builtin_agg` / `resolve_agg_entry` — rather than the four +former mechanisms (a name→function-string map, a second inline class map, a +stat-name frozenset, and per-name equality intercepts). A name absent from the +table is a model-level custom aggregation and takes the formula-template path. +`window_agg_class` reads the same table's `window_class`, replacing a silent +`else AVG` catch-all. + +## P-H — Dialect differences live only in the dialect strategy + +To render identically across dialects, the shared helpers (`_build_agg`, +`_build_percentile`, `_build_stat_agg`, `_wrap_cast_for_type`, `_resolve_sql`, +`_build_date_trunc`) consume one typed input, `AggRenderSpec`, built from planned +slots by `_build_agg_render_spec_from_planned`. Dialect-specific behaviour +(SQLite UDFs, ClickHouse parametric quantiles, MySQL's unsupported-function +`NotImplementedError`, `log10`/`log2` literal preservation, JSON-extract +rewriting) is emitted by exactly one code path. The outer wrap is likewise a +dialect hook: `_emit_planned_outer_wrap` delegates to `SqlDialect.emit_outer_wrap` +(pagination arrives as detached AST from the plan), and order terms resolve +through the single `resolve_order_term`, which reads null-ordering and direction +from the dialect strategy rather than per-render-path. + +## P-I — Grain join-backs are null-safe everywhere + +Every plan-shaped CTE joins back to the host base on the query grain through the +one null-safe join builder (`slayer/sql/render/joins.py`), which pairs each grain +column with a null-safe equality so a NULL grain value on either side still +matches. A scalar CMA has an empty grain — no join predicate exists, so the shape +stays a CROSS JOIN. + +## P-J — No parity ballast + +Every superseded mechanism passes through three states: (1) production- +unreferenced, (2) deleted, (3) test-unpinned. States 2 and 3 happen only after +the desired behaviour is completely pinned by tests on the new code; byte-parity +with already-deleted legacy code is not a requirement. This PR (DEV-1749) +executed states 2 and 3 for the legacy value-key renderers and arithmetic +composers, the first/last host-base ranked machinery (`FirstLastRenderState` and +its helpers), the Mode-A model-filter qualify chain, the four legacy ORDER BY +resolvers, the `_null_safe_join_pair_sql` string round-trip, the formula-text +cross-model re-rooting island, and the double-indirection aggregation dispatch — +each with its pinning tests removed only after the live path's behaviour was +confirmed pinned. ## Response metadata (`response_meta.py`) `build_response_metadata` builds `SlayerResponse.attributes` and -`expected_columns` from the root `PlannedQuery` plus the rendered SQL (the -retired legacy engine derived both from an `EnrichedQuery`): +`expected_columns` from the root `PlannedQuery` plus the rendered SQL: - **`expected_columns`** comes from the final SQL's `named_selects` — the literal result-key columns rows come back under. Reading them from the SQL (rather than - re-deriving from slots) is bulletproof: it is exactly the outer SELECT the - generator emitted. + re-deriving from slots) is exactly the outer SELECT the generator emitted. - **`attributes`** (`ResponseAttributes.dimensions` / `.measures`) come from the - root plan's public `ValueSlot`s, classified dimension (ROW phase) vs measure - (everything else), with each public result key mapped to its - `FieldMetadata(label, format)`. `_slot_result_keys` mirrors - `_full_alias_for_slot` so the keys line up with the rendered projection; only - keys actually present in the rendered SQL are surfaced (a guard against - divergence). Aggregate formats come from `_infer_aggregated_format` (INTEGER - for count/star, FLOAT for avg-family, source-column format for sum/min/max). + root plan's public `ValueSlot`s, classified dimension (ROW phase) vs measure, + each public result key mapped to its `FieldMetadata(label, format)`. + `_slot_result_keys` mirrors `_full_alias_for_slot`; only keys actually present + in the rendered SQL are surfaced. Aggregate formats come from + `_infer_aggregated_format`, which shares one classifier (`classify_aggregation`, + DEV-1788) with `aggregated_type` (slot `DataType`) so the type and format axes + cannot drift: INTEGER for count/star; plain FLOAT for `corr`/`var`/`covar`; + the source column's format (else FLOAT) for the avg-family/`percentile`/`stddev`; + the source column's format (else None) for `sum`/`min`/`max`/`first`/`last`. `FieldMetadata` / `ResponseAttributes` / `_infer_aggregated_format` live here (not -in `query_engine`) so the module imports nothing from the engine; -`query_engine` re-exports them, keeping the public import path unchanged. +in `query_engine`) so the module imports nothing from the engine; `query_engine` +re-exports them, keeping the public import path unchanged. ## Design rationale -- **Why one shared dialect emitter?** Dialect coverage (SQLite UDFs, ClickHouse - parametric quantiles, MySQL's unsupported-function `NotImplementedError`, the - `log10`/`log2` literal preservation, JSON-extract rewriting) is large and - well-tested. Routing every caller through one emitter keeps that behaviour in - a single place. This originally also kept the two coexisting pipelines from - drifting on dialect SQL, at the cost of an `EnrichedMeasure` coupling that - DEV-1452 / DEV-1485 removed. -- **Why derive `expected_columns` from the SQL?** Because the SQL is the ground - truth for what rows come back keyed by. Re-deriving from slots risks a subtle - mismatch; reading `named_selects` cannot. -- **Why assert in `_stage_rename_wrapper`?** A leaked hidden column or a C13 - over-projection would otherwise surface as a downstream "column not found" - deep in the next stage's binding. Asserting at the boundary turns a confusing - failure into a precise one. +- **Why one shared dialect emitter (P-H)?** Dialect coverage is large and + well-tested; routing every caller through one emitter keeps that behaviour in a + single place and makes "same construct, same SQL" (P-G) hold across backends. +- **Why derive `expected_columns` from the SQL?** The SQL is the ground truth for + what rows come back keyed by. Re-deriving from slots risks a subtle mismatch; + reading `named_selects` cannot. +- **Why assert in `_stage_rename_wrapper` (P-B)?** A leaked hidden column or a C13 + over-projection would otherwise surface as a downstream "column not found" deep + in the next stage's binding. Asserting at the boundary turns a confusing failure + into a precise one. +- **Why typed keys end-to-end (P-E)?** A formula-text round-trip could silently + drift on quoting, path residuals, or operand order; typed re-rooting cannot + produce a key the binder would read differently. diff --git a/docs/architecture/stage-planning.md b/docs/architecture/stage-planning.md index 0bb6081f..2058596b 100644 --- a/docs/architecture/stage-planning.md +++ b/docs/architecture/stage-planning.md @@ -56,8 +56,9 @@ them, so SQL stays parity-stable: 2. `SlayerModel.filters` — Mode-A SQL, validated by `_validate_model_filter` (rejects DSL constructs, raw windows, measure refs, and windowed columns). A reference to a non-trivial derived column is accepted (DEV-1450 #4b): the - generator's `_render_model_filter_sql` inline-expands the predicate at render - time. These are text-only `FilterPhase` entries with no typed value-key. + generator enters the predicate through the scope's Mode-A door + (`ScopeFrame.enter_predicate`), which inline-expands it at render time. These + are text-only `FilterPhase` entries with no typed value-key. 3. user query filters — Mode-B DSL, bound with the `filter_alias_map` so renamed measures resolve by alias (DEV-1445). Two filter strings that bind to the same structural key are deduped (P2) so a HAVING isn't duplicated. diff --git a/docs/architecture/typed-keys.md b/docs/architecture/typed-keys.md index c73c9bc9..2b98e1ee 100644 --- a/docs/architecture/typed-keys.md +++ b/docs/architecture/typed-keys.md @@ -147,9 +147,12 @@ SCALAR_FUNCTIONS = frozenset({ "nullif", "coalesce", "ifnull", # math "ln", "log10", "log2", "log", "exp", "sqrt", "pow", "power", - "abs", "floor", "ceil", "round", + "abs", "floor", "ceil", "ceiling", "round", "sign", # string hygiene (was DEV-1378's STRING_HYGIENE_OPS) - "lower", "upper", "trim", "replace", "substr", "instr", "length", "concat", + "lower", "upper", "trim", "ltrim", "rtrim", + "replace", "substr", "substring", "instr", "length", "concat", + # pattern match — emits the SQL LIKE operator + "like", }) ``` diff --git a/docs/concepts/formulas.md b/docs/concepts/formulas.md index d59c3708..796b22d2 100644 --- a/docs/concepts/formulas.md +++ b/docs/concepts/formulas.md @@ -67,6 +67,12 @@ Windowed measures need exactly one resolvable time dimension (a single a windowed measure (`{"formula": "revenue:sum(window='90d') > 100"}`) applies after aggregation, and the windowed measure must also be selected. +A group whose dimension value is NULL gets its real windowed value, like any +other group. (Earlier versions returned NULL for such groups: the rolling +aggregate was matched to its group with a plain equality, and `NULL = NULL` is +not true. Grouping keys now compare null-safely everywhere, so a NULL region or +an unmatched outer join no longer silently blanks the measure.) + A windowed measure may also be used purely as an **order** target without being selected — `{"order": [{"column": "revenue:sum(window='90d')", "direction": "desc"}]}` ranks by the rolling value and keeps it out of the result. That works both for a diff --git a/docs/concepts/queries.md b/docs/concepts/queries.md index d0b095d2..d45864ea 100644 --- a/docs/concepts/queries.md +++ b/docs/concepts/queries.md @@ -101,7 +101,7 @@ A time dimension with a required granularity and an optional date range. Support ## OrderItem -A sort specification: `column` is the short alias (`status`, `revenue_sum`, `*:count`), `direction` is `asc` or `desc`. +A sort specification: `column` names a dimension (`status`), a declared measure's short alias (`revenue_sum`), or a formula (`*:count`); `direction` is `asc` or `desc`. ```json {"column": "*:count", "direction": "desc"} @@ -128,9 +128,23 @@ What each shape of an *undeclared* order target does: | An inline **transform** (`rank(amount:sum)`, `cumsum(...)`, `change(...)`, `lag`/`lead`/`ntile`) | Computed hidden, sorted on, stripped. | | An inline **composite** (`revenue:sum / cnt:sum`, `abs(amount:sum)`, `change(amount:sum) / 2`) | Computed hidden, sorted on, stripped. | | A **windowed** aggregate (`amount:sum(window='90d')`), alone or inside a composite | Computed hidden in its own rolling-window CTE, sorted on, stripped. | -| A raw row column, in a **raw-rows** query (`distinct_dimension_values: false`, no measures) | Sorted on directly (`ORDER BY orders.created_at`). | -| A raw row column, in an **aggregated / dedup** query | Rejected (HTTP 400): it isn't in the `GROUP BY`. Add it to `dimensions`, or order by an aggregate of it (`created_at:max`). | -| A **joined** row column (`customers.regions.name`) not projected | Rejected (HTTP 400): project it (add to `dimensions`) or order by a projected field. | +| A raw row column, in a **raw-rows** query (`distinct_dimension_values: false`, no measures) | Sorted on directly (`ORDER BY orders.created_at`). Applies to a **joined** column (`customers.regions.name`) and to a derived column whose `sql` reaches through a join — the join is pulled in for the sort. | +| A raw row column, in an **aggregated / dedup** query | Sorted on **per group**: ASC by each group's minimum, DESC by each group's maximum. The wrap is implicit — you do not write `created_at:min`. | +| A **joined** row column (`customers.regions.name`) in an aggregated query | Same per-group wrap, computed in a CTE rooted at the source model with the join pulled inside, so each group gets its own extreme rather than one global value. | + +Ordering by an undeclared row column in a grouped query is **not** the same as +ordering by the column itself — there is no single value per group to sort by. +SLayer picks the extreme the direction puts first: `asc` sorts each group by its +`min`, `desc` by its `max`. Write `{"column": "created_at:max", "direction": "asc"}` +explicitly if you want the other one. + +NULLs sort **last** in both directions, on every database, so the same query +returns the same row order regardless of backend. (SQL Server is the one +exception: its native ordering is used because the portable emulation makes the +statement fail there.) + +An order target that names nothing SLayer can resolve is an error, never a +silently unsorted result. Transform and composite order targets accept the full formula syntax, so `{"column": "revenue:sum / cnt:sum"}` and `{"column": "change(revenue:sum)"}` both @@ -152,11 +166,17 @@ Query results are returned as a `SlayerResponse`: | Field | Type | Description | |-------|------|-------------| | `data` | list[dict] | Rows as dictionaries | -| `columns` | list[str] | Column names in `model_name.column_name` format (e.g., `"orders._count"`, `"orders.customers.regions.name"` for multi-hop) | +| `columns` | list[str] | Column names in `model_name.column_name` format (e.g., `"orders._count"`, `"orders.customers.regions.name"` for multi-hop), **in the order you declared them** | | `row_count` | int | Number of rows | | `sql` | string | The generated SQL (useful for debugging) | | `attributes` | ResponseAttributes | Field metadata split by type: `attributes.dimensions` and `attributes.measures`, each a dict of column alias → FieldMetadata (label, format) | +`columns` — and the key order of each row in `data` — follows the order you +declared fields in the query: dimensions, then time dimensions, then measures, +each in the order given. This holds regardless of how a measure is computed, so +a measure on a joined model appears where you declared it rather than after the +local ones. Fields used only for ordering are computed but not returned. + ```json { "data": [ @@ -213,13 +233,16 @@ Use `and`, `or`, `not` within a single filter string: Multiple entries in the `filters` list are combined with AND. -### String-Hygiene Operators +### Scalar Functions in Filters -Filters in `SlayerQuery.filters` accept a small allowlist of lowercase -SQL scalar functions for case-folding, trimming, substring extraction, -and string concatenation: `lower`, `upper`, `trim`, `replace`, `substr`, -`instr`, `length`, `concat`. The SQL `||` concat operator is rewritten -to `concat(...)` automatically. +Filters in `SlayerQuery.filters` accept the closed Mode-B scalar +allowlist: string hygiene (`lower`, `upper`, `trim`, `ltrim`, +`rtrim`, `replace`, `substr`, `substring`, `instr`, `length`, `concat`), +null handling (`coalesce`, `nullif`, `ifnull`), and math (`round`, `abs`, +`ceil`, `floor`, `sign`, `log10`, …). The SQL `||` concat operator is +rewritten to `concat(...)` automatically. See +[references](references.md#scalar-functions-and-dialect-semantics) for the +full list and per-dialect semantics. ```json "filters": [ @@ -227,17 +250,17 @@ to `concat(...)` automatically. "trim(name) = 'Smith'", "replace(category, ',', '') = 'books'", "substr(s, 1, instr(s, ',') - 1) = 'first_token'", - "length(replace(x, ',', '')) > 0", + "coalesce(nickname, name) = 'Ada'", "first || ' ' || last = 'jane doe'" ] ``` -Names are lowercase only — `LOWER(...)` is rejected. sqlglot translates -each call to the target dialect's preferred spelling at SQL-generation +Names are matched case-insensitively — `LOWER(...)` and `lower(...)` both bind. +sqlglot translates each call to the target dialect's preferred spelling at SQL-generation time (`instr` → `POSITION` / `LOCATE` / `STRPOS`, `substr` → -`SUBSTRING`, `concat` → `||` on SQLite). Calls outside the allowlist -(`json_extract`, `coalesce`, …) belong in `Column.sql` / -`Column.filter` / `SlayerModel.filters` (Mode A SQL). +`SUBSTRING`, `concat` → `||` on SQLite). Raw SQL functions outside the +allowlist (`json_extract`, `date_trunc`, `CASE WHEN`, …) belong in +`Column.sql` / `Column.filter` / `SlayerModel.filters` (Mode A SQL). ### Filtering on Computed Columns diff --git a/docs/concepts/references.md b/docs/concepts/references.md index d13ff05e..2de6d99b 100644 --- a/docs/concepts/references.md +++ b/docs/concepts/references.md @@ -7,7 +7,7 @@ SLayer has two distinct expression layers and the rules for what each one accept | Mode | Fields | Parser | Accepts | Rejects | |---|---|---|---|---| | **A — SQL** | `Column.sql`, `Column.filter`, each entry of `SlayerModel.filters` | sqlglot | Any valid SQL expression for the underlying dialect — function calls (`json_extract`, `coalesce`, `nullif`, `lower`, `length`, …), arithmetic, `CASE WHEN`, string literals, comparison and boolean operators in SQL spelling (`=`, `<>`, `IS NULL`, `AND`, `OR`, `NOT`, `IN`, `LIKE`). Bare names and `__`-delimited join paths. | Aggregation colon syntax (`revenue:sum`); SLayer transform calls (`cumsum`, `change`, `rank`, …); references to `ModelMeasure` formulas; raw `OVER (...)` window functions inside `Column.filter` / `SlayerModel.filters` (allowed only in `Column.sql`). | -| **B — DSL** | `ModelMeasure.formula`, `SlayerQuery.measures`, `SlayerQuery.filters`, `SlayerQuery.dimensions`, `SlayerQuery.time_dimensions`, `SlayerQuery.order`, `SlayerQuery.main_time_dimension` | Python AST formula parser | Bare names that resolve to a `Column` or `ModelMeasure` on the model; single-dot dotted paths through joins (`customers.regions.name`, `customers.revenue:sum`); aggregation colon syntax (`:`, `*:count`, parametric forms); transform calls (`cumsum(revenue:sum)`, `rank(revenue:sum, partition_by=region)`); arithmetic / boolean / comparison operators; the SQL `\|\|` concat operator (folded into `concat(...)`); pattern matching via the `like(value, pattern)` scalar (emits the SQL `LIKE` operator — wrap in `not (...)` for `NOT LIKE`); a small allowlist of lowercase string-hygiene scalars in `SlayerQuery.filters` only — `lower`, `upper`, `trim`, `replace`, `substr`, `instr`, `length`, `concat`, `like`; `{variable}` placeholders (filters only). | `__`-delimited tokens in user input; raw SQL function calls outside the string-hygiene allowlist (`json_extract`, `coalesce`, …); raw `OVER (...)`; bare names that don't resolve to a Column / ModelMeasure / custom aggregation / query alias; **uppercase** spellings of the string-hygiene functions (`LOWER`, `TRIM`, …) — DSL is case-sensitive. | +| **B — DSL** | `ModelMeasure.formula`, `SlayerQuery.measures`, `SlayerQuery.filters`, `SlayerQuery.dimensions`, `SlayerQuery.time_dimensions`, `SlayerQuery.order`, `SlayerQuery.main_time_dimension` | Python AST formula parser | Bare names that resolve to a `Column` or `ModelMeasure` on the model; single-dot dotted paths through joins (`customers.regions.name`, `customers.revenue:sum`); aggregation colon syntax (`:`, `*:count`, parametric forms); transform calls (`cumsum(revenue:sum)`, `rank(revenue:sum, partition_by=region)`); arithmetic / boolean / comparison operators; the SQL `\|\|` concat operator (folded into `concat(...)`); pattern matching via the `like(value, pattern)` scalar (emits the SQL `LIKE` operator — wrap in `not (...)` for `NOT LIKE`); a closed allowlist of scalar functions (matched case-insensitively) — null handling (`nullif`, `coalesce`, `ifnull`), math (`ln`, `log10`, `log2`, `log`, `exp`, `sqrt`, `pow`, `power`, `abs`, `floor`, `ceil`, `ceiling`, `round`, `sign`), string hygiene (`lower`, `upper`, `trim`, `ltrim`, `rtrim`, `replace`, `substr`, `substring`, `instr`, `length`, `concat`) and `like`, each with a declared argument count that is validated (`coalesce` and `concat` are variadic); `{variable}` placeholders (filters only). | `__`-delimited tokens in user input; raw SQL function calls outside that allowlist (`json_extract`, `date_trunc`, …), and any allowlisted call with the wrong number of arguments; raw `OVER (...)`; bare names that don't resolve to a Column / ModelMeasure / custom aggregation / query alias; `NULL` inside an `in` / `not in` list (use `is null` / `is not null` instead — see below). | ## Identifier resolution @@ -114,6 +114,62 @@ Rejected at enrichment (when the formula is evaluated against a model): {"name": "bad", "formula": "json_extract(data, '$.x')"} // raw SQL fn ``` +## Scalar functions and dialect semantics + +The allowlisted scalars are rendered as typed SQL and then translated to each +backend's own spelling, so one formula stays correct across dialects rather +than being passed through verbatim. `length(x)` emits `LEN(x)` on SQL Server, +`substr(x, 1, 5)` emits `SUBSTRING(x FROM 1 FOR 5)` on Postgres, and +`ifnull(x, 0)` emits `COALESCE(x, 0)` on backends without `IFNULL`. + +Four consequences worth knowing: + +* **`concat` follows SQL string-concatenation semantics.** On dialects whose + natural spelling is the `||` operator (Postgres, DuckDB, SQLite), `concat(a, b)` + emits `a || b`, which yields `NULL` if either operand is `NULL`. That differs + from those backends' own `CONCAT()` function, which treats `NULL` as an empty + string. Wrap operands in `ifnull(...)` when you want the NULL-tolerant + behaviour: + + ```json + {"filters": ["concat(ifnull(first_name, ''), ifnull(last_name, '')) = 'AdaLovelace'"]} + ``` + +* **`log10` / `log2` keep their single-argument form** on the backends that + provide one, rather than becoming the generic two-argument `LOG(base, x)`. + +* **Argument counts are validated.** Each allowlisted scalar has a fixed arity + (`round` takes 1 or 2, `substr` / `substring` 2 or 3, `replace` exactly 3; + `coalesce` and `concat` are variadic). A call with the wrong number is + rejected with a message naming the function, rather than being silently + truncated or passed through to fail at the database. + + The bounds are deliberately tight where a wider call would translate into + something else entirely: `ceiling(x, y)` would emit `CEIL(x, y)` and + `ceiling(x, y, z)` DuckDB's unrelated `CEIL(x TO z)` rounding form, so + `ceil` / `ceiling` take exactly one argument. `ltrim` / `rtrim` likewise take + the string only — the "strip these characters" second argument is not + accepted, because MySQL's `LTRIM` takes a single argument and the call would + reach the server as SQL it rejects. + +* **Aliases render identically to their canonical spelling.** `ceiling` is + `ceil` and `substring` is `substr`; both spellings are accepted so a formula + written against either SQL convention binds, and both emit the target's own + form. + +## `NULL` inside an `in` list + +`NULL` is rejected inside an `in` / `not in` list. SQL compares it by +three-valued logic, so `status in ('a', None)` never matches on the null, and +`status not in ('a', None)` matches **no rows at all** — the filter silently +returns an empty result instead of "everything except 'a'". + +Test for null separately: + +```json +{"filters": ["status not in ('new', 'old')", "status is not null"]} +``` + ## See also * [Models](models.md) — `Column.sql`, `Column.filter`, model-level filters diff --git a/docs/interfaces/rest-api.md b/docs/interfaces/rest-api.md index 5d189c09..da2386fa 100644 --- a/docs/interfaces/rest-api.md +++ b/docs/interfaces/rest-api.md @@ -68,10 +68,26 @@ Response: {"orders.status": "pending", "orders._count": 15} ], "row_count": 2, - "columns": ["orders.status", "orders._count"] + "columns": ["orders.status", "orders._count"], + "warnings": [] } ``` +`warnings` carries advisories about the query itself; it is `[]` for a clean +query. Every entry has a `kind` discriminator — switch on it rather than on the +presence of a field: + +| `kind` | Meaning | Extra fields | +| -- | -- | -- | +| `normalization` | The input was rewritten to canonical form | `rule_id`, `original`, `normalized`, `location`, `rule_doc_url` (nullable) | +| `unreachable_filter_dropped` | A filter could not be applied inside a cross-model CTE and was dropped from it (it still applies at the host) | `filter_text`, `location`, `reason` | + +The filter still applies to the host's own rows; it is dropped only from the +cross-model CTE, whose aggregate is therefore computed without it. That measure +can come back wider than the filter implies — and where it feeds a `HAVING`, +`ORDER BY`, or pagination, that can change which rows surface — so it is worth +surfacing to whoever wrote the query. + ### Models ``` diff --git a/docs/reference/rest-api.md b/docs/reference/rest-api.md index 1ee62bb4..c163d221 100644 --- a/docs/reference/rest-api.md +++ b/docs/reference/rest-api.md @@ -49,10 +49,26 @@ Response: {"orders.status": "pending", "orders._count": 15} ], "row_count": 2, - "columns": ["orders.status", "orders._count"] + "columns": ["orders.status", "orders._count"], + "warnings": [] } ``` +`warnings` carries advisories about the query itself; it is `[]` for a clean +query. Every entry has a `kind` discriminator — switch on it rather than on the +presence of a field: + +| `kind` | Meaning | Extra fields | +| -- | -- | -- | +| `normalization` | The input was rewritten to canonical form | `rule_id`, `original`, `normalized`, `location`, `rule_doc_url` (nullable) | +| `unreachable_filter_dropped` | A filter could not be applied inside a cross-model CTE and was dropped from it (it still applies at the host) | `filter_text`, `location`, `reason` | + +The filter still applies to the host's own rows; it is dropped only from the +cross-model CTE, whose aggregate is therefore computed without it. That measure +can come back wider than the filter implies — and where it feeds a `HAVING`, +`ORDER BY`, or pagination, that can change which rows surface — so it is worth +surfacing to whoever wrote the query. + The body accepts the same fields as a `SlayerQuery`, plus `dry_run`, `explain`, and `variables`. Notable optional fields: - `whole_periods_only` (bool) — snap date filters to bucket boundaries. diff --git a/slayer/api/server.py b/slayer/api/server.py index 57769b8e..21a2d4c8 100644 --- a/slayer/api/server.py +++ b/slayer/api/server.py @@ -97,6 +97,10 @@ class QueryResponse(BaseModel): columns: list[str] sql: str | None = None attributes: AttributesResponse | None = None + # DEV-1745 (W5/D2): advisories about the query itself — slack-normalization + # rewrites and filters that were dropped as unreachable. One list, each + # entry tagged with a ``kind`` discriminator the consumer switches on. + warnings: list[dict[str, Any]] = [] class IngestRequest(BaseModel): @@ -346,6 +350,9 @@ def _convert_meta(d: dict) -> dict[str, FieldMetadataResponse]: row_count=result.row_count, columns=result.columns, attributes=attributes, + warnings=[ + w.model_dump(mode="json") for w in (result.warnings or []) + ], ) if dry_run or explain: response.sql = result.sql diff --git a/slayer/cli.py b/slayer/cli.py index c752675d..87931051 100644 --- a/slayer/cli.py +++ b/slayer/cli.py @@ -1225,6 +1225,17 @@ def _parse_cli_variables(args) -> dict: return out +def _print_query_warnings(result) -> None: + """Print query advisories to STDERR. + + Stderr specifically, so a piped ``slayer query`` keeps emitting clean data + on stdout while the operator still sees that a filter was dropped from a + cross-model CTE (it still applies at the host). + """ + for w in (getattr(result, "warnings", None) or []): + print(f"warning: {w.human_message()}", file=sys.stderr) + + def _run_query(args): # NOSONAR S3776 — argparse-driven dispatch; one straight-line function reads better than threaded helpers from slayer.engine.query_engine import SlayerQueryEngine @@ -1269,6 +1280,8 @@ def _run_query(args): # NOSONAR S3776 — argparse-driven dispatch; one straigh explain=bool(args.explain), ) + _print_query_warnings(result) + if args.dry_run: print(result.sql) return diff --git a/slayer/core/enums.py b/slayer/core/enums.py index a1f9bcac..3a760ab5 100644 --- a/slayer/core/enums.py +++ b/slayer/core/enums.py @@ -3,7 +3,7 @@ import datetime # noqa: F401 (kept for downstream imports of TimeGranularity) import difflib from enum import Enum -from typing import Any +from typing import Any, Optional class StrEnum(str, Enum): @@ -174,6 +174,62 @@ class JoinType(StrEnum): "corr", "covar_samp", "covar_pop", }) +# Aggregation value classification (DEV-1788). One classifier, +# ``classify_aggregation``, buckets every aggregation by how its result relates +# to the source column. Both ``aggregated_type`` (slot DataType) and +# ``_infer_aggregated_format`` (display NumberFormat) read the bucket and map it +# to their own output, so the type and format axes cannot drift apart. The four +# builtin sets partition ``BUILTIN_AGGREGATIONS`` (pinned by a drift-guard test); +# custom/model-defined aggregations hit the PRESERVING fallback. + +# Result is always an integer count, independent of the source column. +INTEGER_AGGREGATIONS: frozenset[str] = frozenset({ + "count", "count_distinct", "count_distinct_approx", +}) +# Result is a float in the SAME units as the source (display format inherited). +FLOAT_SOURCE_UNIT_AGGREGATIONS: frozenset[str] = frozenset({ + "avg", "weighted_avg", "median", "percentile", + "stddev_samp", "stddev_pop", +}) +# Result is a float in different units (dimensionless / squared / product), so it +# carries a plain FLOAT format, not the source's units. +FLOAT_PLAIN_AGGREGATIONS: frozenset[str] = frozenset({ + "corr", "var_samp", "var_pop", "covar_samp", "covar_pop", +}) +# Result preserves the source column's type AND format. +PRESERVING_AGGREGATIONS: frozenset[str] = frozenset({ + "sum", "min", "max", "first", "last", +}) + + +class AggregationValueClass(StrEnum): + """How an aggregation's result relates to its source column, for slot-type + and display-format inference (DEV-1788).""" + + COUNT = "count" # INT type, INTEGER format + PRESERVING = "preserving" # source type & format + FLOAT_SOURCE_UNITS = "float_source_units" # DOUBLE type, source format (else FLOAT) + FLOAT_PLAIN = "float_plain" # DOUBLE type, plain FLOAT format + + +def classify_aggregation( + *, measure_name: Optional[str], aggregation: str +) -> AggregationValueClass: + """Bucket an aggregation for slot-type / display-format inference. + + ``measure_name == "*"`` (``*:count``) is COUNT; custom/unknown aggregations + fall through to PRESERVING (inherit source type & format). + """ + if measure_name == "*": + return AggregationValueClass.COUNT + if aggregation in INTEGER_AGGREGATIONS: + return AggregationValueClass.COUNT + if aggregation in FLOAT_SOURCE_UNIT_AGGREGATIONS: + return AggregationValueClass.FLOAT_SOURCE_UNITS + if aggregation in FLOAT_PLAIN_AGGREGATIONS: + return AggregationValueClass.FLOAT_PLAIN + return AggregationValueClass.PRESERVING + # DEV-1576: unambiguous aggregation-name aliases that LLM agents routinely # emit. ``normalize_aggregation_name`` lowercases the incoming token and maps # it through this table; the result is only adopted when it lands in diff --git a/slayer/core/errors.py b/slayer/core/errors.py index 9882bd24..4928a91b 100644 --- a/slayer/core/errors.py +++ b/slayer/core/errors.py @@ -186,6 +186,42 @@ def __init__( )) +class ModeASqlParseError(SlayerError, ValueError): + """A free-SQL (Mode-A) fragment could not be parsed. + + Mode-A text used to fail soft in three places: the fragment was handed + through unparsed, or — worse, in the join-path scanner — swallowed into + ZERO join paths, so an unparseable predicate emitted a query missing its + joins instead of reporting the problem. This is now the single loud + failure, carrying the fragment verbatim and the surface it came from + (``location``) so the author can find it. + + Multi-inherits ``ValueError`` for back-compat with call sites that already + catch ``ValueError`` around SQL text handling (see + :class:`UnknownReferenceError`). + """ + + def __init__( + self, + fragment: str, + location: str, + reason: str | None = None, + ) -> None: + self.fragment = fragment + self.location = location + self.reason = reason + super().__init__(_format_error_message( + cls_name=type(self).__name__, + summary=f"Cannot parse SQL fragment {fragment!r}.", + scope=location if reason is None else f"{location}: {reason}", + suggestion=( + "Mode-A surfaces take raw SQL for the target dialect. Check " + "the fragment for balanced parentheses and quotes, and that " + "any '{variable}' placeholders were supplied." + ), + )) + + class AmbiguousReferenceError(SlayerError, ValueError): """A reference matches multiple candidates in scope and can't pick one. @@ -420,6 +456,32 @@ def __init__(self, filter_text: str, reason: str) -> None: ) +class RenderContextMissingFacilityError(SlayerError, ValueError): + """A ValueKey render needed a facility its render context did not carry. + + The single ValueKey renderer fails closed rather than degrading quietly: + silent fallbacks are how the generator's five renderer copies drifted apart. + """ + + def __init__( + self, + key_kind: str, + facility: str, + detail: str | None = None, + ) -> None: + self.key_kind = key_kind + self.facility = facility + self.detail = detail + suffix = f" ({detail})" if detail else "" + super().__init__(_format_error_message( + cls_name=type(self).__name__, + summary=( + f"Rendering a {key_kind} requires the {facility!r} " + f"render-context facility, which was not supplied{suffix}." + ), + )) + + class IdCollisionError(SlayerError, ValueError): """Raised by filename-backed (YAML) storage when saving an entity whose id differs from an existing id only by letter case — such ids diff --git a/slayer/core/keys.py b/slayer/core/keys.py index b07296cf..325259a8 100644 --- a/slayer/core/keys.py +++ b/slayer/core/keys.py @@ -24,7 +24,7 @@ from decimal import Decimal from enum import IntEnum -from typing import Optional, Tuple, Union +from typing import Literal, Optional, Tuple, TypeVar, Union, cast from pydantic import BaseModel, ConfigDict, field_validator @@ -42,15 +42,82 @@ "nullif", "coalesce", "ifnull", # Math "ln", "log10", "log2", "log", "exp", "sqrt", "pow", "power", - "abs", "floor", "ceil", "round", + "abs", "floor", "ceil", "ceiling", "round", "sign", # String hygiene (was DEV-1378's STRING_HYGIENE_OPS) - "lower", "upper", "trim", "replace", "substr", "instr", "length", "concat", + "lower", "upper", "trim", "ltrim", "rtrim", + "replace", "substr", "substring", "instr", "length", "concat", # Pattern match — ``like(value, pattern)`` emits the SQL ``LIKE`` operator # (sqlglot ``exp.Like``); see SQLGenerator scalar-call rendering. "like", }) +# Accepted argument counts per allowlisted scalar, as ``(min, max)``; ``max=None`` +# means variadic. Validated at bind time so a malformed call is a clear SLayer +# error, and again at render time as the fail-closed backstop. +# +# Needed because sqlglot is inconsistent about arity: ``exp.func("ROUND", a, b, c)`` +# SILENTLY DROPS the third argument, ``exp.func("LENGTH", a, b)`` emits invalid +# ``LENGTH(a, b)`` for the database to reject, and ``exp.func("LOWER", a, b)`` +# raises a raw sqlglot ValueError. None of those is a good answer for a user +# who mistyped a filter. +SCALAR_FUNCTION_ARITY: dict[str, tuple[int, Optional[int]]] = { + "nullif": (2, 2), + "coalesce": (1, None), + "ifnull": (2, 2), + "ln": (1, 1), "log10": (1, 1), "log2": (1, 1), "log": (1, 2), + "exp": (1, 1), "sqrt": (1, 1), + "pow": (2, 2), "power": (2, 2), + "abs": (1, 1), "floor": (1, 1), "ceil": (1, 1), "round": (1, 2), + # ``ceiling`` is the T-SQL spelling of ``ceil`` and renders to the same + # node. Pinned at 1: a 2-arg call silently emits ``CEIL(x, y)``, and a + # 3-arg one becomes DuckDB's unrelated ``CEIL(x TO z)`` rounding form. + "ceiling": (1, 1), "sign": (1, 1), + "lower": (1, 1), "upper": (1, 1), "trim": (1, 1), "length": (1, 1), + # The trims take the string only, matching ``trim``. The 2-arg + # strip-these-characters form is deliberately NOT admitted: sqlglot emits + # a literal ``LTRIM(str, chars)`` for some targets, and MySQL's ``LTRIM`` + # accepts one argument — so it would be SQL the server rejects. + "ltrim": (1, 1), "rtrim": (1, 1), + "replace": (3, 3), "substr": (2, 3), "substring": (2, 3), "instr": (2, 2), + "concat": (1, None), + "like": (2, 2), +} + +# Not a second allowlist: the table above must cover ``SCALAR_FUNCTIONS`` +# exactly. Checked BOTH ways at import — a missing entry would let a wrong-arity +# call through to sqlglot's inconsistent handling, and an entry for a name that +# is not allowlisted would be dead weight that reads as though it were. +_arity_missing = SCALAR_FUNCTIONS - set(SCALAR_FUNCTION_ARITY) +_arity_unknown = set(SCALAR_FUNCTION_ARITY) - SCALAR_FUNCTIONS +if _arity_missing or _arity_unknown: # pragma: no cover — import-time invariant + raise RuntimeError( + f"SCALAR_FUNCTION_ARITY disagrees with SCALAR_FUNCTIONS: " + f"missing={sorted(_arity_missing)}, unknown={sorted(_arity_unknown)}", + ) + + +def check_scalar_arity(*, name: str, argc: int) -> Optional[str]: + """Return an error message when ``name`` cannot take ``argc`` arguments.""" + bounds = SCALAR_FUNCTION_ARITY.get(name) + if bounds is None: + return None + low, high = bounds + if low <= argc and (high is None or argc <= high): + return None + if low == high: + expected = f"{low}" + elif high is None: + expected = f"{low} or more" + else: + expected = f"{low} to {high}" + plural = "" if low == high == 1 else "s" + return ( + f"Scalar function {name!r} takes {expected} argument{plural}; " + f"got {argc}." + ) + + # --------------------------------------------------------------------------- # Phase # --------------------------------------------------------------------------- @@ -403,6 +470,20 @@ class AggregateKey(_FrozenKey): ``column_filter_key`` is the ``Column.filter`` attached to the aggregated column, if any — pulled into the structural key so two aggregates with different attached filters do not collide. + + ``grain`` names WHERE the aggregate is evaluated when ``source.path`` is + non-empty (DEV-1747 D2). ``"target"`` — the default and the meaning of + every user-declared cross-model aggregate — evaluates it in a CTE rooted at + the target, one value per target row-group. ``"host"`` evaluates it in a + CTE rooted at the HOST with the path's joins pulled in, grouped on the + query grain: one value per HOST group. The DEV-1735 order wrap needs the + latter, because a target-rooted CTE for a host-grain sort key degenerates + to a scalar CROSS JOIN and sorts every group by one constant. + + It participates in identity deliberately: a declared + ``customers.regions.name:max`` and the synthetic host-grain wrap over the + same column are different values (global vs per-group), so interning them + onto one slot would silently give the user the wrong one. """ source: _AggregateSource @@ -410,6 +491,7 @@ class AggregateKey(_FrozenKey): args: Tuple[_AggregateArgValue, ...] = () kwargs: Tuple[Tuple[str, _AggregateKwargValue], ...] = () column_filter_key: Optional[SqlExprKey] = None + grain: Literal["target", "host"] = "target" @field_validator("kwargs", mode="before") @classmethod @@ -428,6 +510,7 @@ def __hash__(self) -> int: _typed_args(self.args), _typed_kwargs(self.kwargs), self.column_filter_key, + self.grain, )) def __eq__(self, other: object) -> bool: @@ -439,9 +522,16 @@ def __eq__(self, other: object) -> bool: and _typed_args(self.args) == _typed_args(other.args) and _typed_kwargs(self.kwargs) == _typed_kwargs(other.kwargs) and self.column_filter_key == other.column_filter_key + and self.grain == other.grain ) +#: Rerooting is type-preserving — a ``ColumnKey`` in, a ``ColumnKey`` out. +#: Expressing that keeps call sites precisely typed rather than collapsing +#: every rerooted key to the union. +_RerootableT = TypeVar("_RerootableT") + + def _reroot_path_ref(ref, *, target_path: Tuple[str, ...]): """Re-anchor a single embedded reference from the host coordinate system into the target's local scope (DEV-1707). @@ -471,68 +561,20 @@ def _reroot_path_ref(ref, *, target_path: Tuple[str, ...]): def reroot_aggregate_key( key: "AggregateKey", *, target_path: Tuple[str, ...], ) -> "AggregateKey": - """Re-anchor EVERY embedded reference of a cross-model ``AggregateKey`` - from the host's coordinate system into its target's local scope + """Re-anchor a cross-model ``AggregateKey`` into its target's local scope (DEV-1707 / DEV-1703 Stage 3). - When a cross-model aggregate (``customers.revenue:sum``, - ``customers.amount:last(customers.signup_at)``) is rendered inside its - target-rooted CTE, its ``source``, positional ``args``, keyword ``kwargs`` - values, and — invariantly — its ``column_filter_key`` must all be - expressed relative to the target rather than the query root. This is the - single, symmetric replacement for the per-field strip logic that used to - live in ``slayer/engine/cross_model_planner.py`` (``_local_agg_formula`` / - ``_reroot_col_kwarg``) and ``slayer/sql/generator.py`` (the inline - ``_reroot_kwarg`` / ``local_args`` / ``_reroot_having`` blocks) with two - divergent semantics. - - ``target_path`` is the join path of the aggregate's source (the hops from - the query root to the target model). Semantics — prefix-strip with - residual, applied uniformly (see ``_reroot_path_ref``): - - * ``source``, each positional arg, and each kwarg VALUE whose ``path`` - starts with ``target_path`` drops that prefix (exact match → local); - * a ref whose ``path`` does not start with ``target_path`` is left - unchanged — the function is TOTAL and never raises; a genuinely - mis-pathed ref surfaces at the downstream binder / kwarg-path validator - exactly as before; - * scalar args / kwargs pass through untouched; kwarg NAMES are preserved - (and the ``AggregateKey`` validator keeps them canonically sorted); - * ``target_path == ()`` is the identity (the filtered-local case, where - the source is already host-local — the empty prefix strips zero hops). - - ``column_filter_key`` is copied UNCHANGED. Its ``canonical_sql`` and - ``referenced_join_paths`` are anchored at the OWNING MODEL of the source - column (stamped by ``slayer.engine.binding._resolve_column_filter_key`` - via ``compute_column_filter_join_paths`` with ``anchor_model`` = that - owning model). Rerooting only changes how that owner is REACHED from the - query root; it never moves the owner, so the filter's owner-relative SQL - and paths are invariant under reroot. After rerooting a filtered - cross-model aggregate the source reads local (``path == ()``) while - ``referenced_join_paths`` stays non-empty — precisely the DEV-1503 - filtered-local isolation trigger shape. + A thin alias for :func:`reroot_value_key`, which applies the same + prefix-strip rule over the whole ``ValueKey`` union. Two implementations + would be free to drift into two reroot semantics — the drift §5.4 removes. + + ``column_filter_key`` rides through unchanged: its paths are anchored at + the OWNING model of the source column, and rerooting changes only how that + owner is reached. After rerooting a filtered cross-model aggregate the + source reads local while ``referenced_join_paths`` stays non-empty — + exactly the filtered-local isolation trigger shape. """ - target_path = tuple(target_path) - if not target_path: - return key - # Rebuild from the existing key so fields reroot does NOT own (``agg``, - # ``column_filter_key``, and any field added to ``AggregateKey`` later) - # ride through automatically rather than being silently dropped. Only - # ``source`` / ``args`` / ``kwargs`` carry rerootable paths. ``model_copy`` - # skips the ``_canonicalize_kwargs`` validator, which is a no-op here: - # reroot preserves kwarg names and order, so the input's already-canonical - # sort is unchanged (pinned by - # ``test_kwargs_canonical_sort_preserved_after_reroot``). - return key.model_copy(update={ - "source": _reroot_path_ref(key.source, target_path=target_path), - "args": tuple( - _reroot_path_ref(a, target_path=target_path) for a in key.args - ), - "kwargs": tuple( - (k, _reroot_path_ref(v, target_path=target_path)) - for k, v in key.kwargs - ), - }) + return reroot_value_key(key, target_path=target_path) class TransformKey(_FrozenKey): @@ -753,3 +795,144 @@ def phase(self) -> Phase: InKey.model_rebuild() # TimeTruncKey.column is a Union[ColumnKey, ColumnSqlKey] (DEV-1450 #4a). TimeTruncKey.model_rebuild() + + +# --------------------------------------------------------------------------- +# The total reroot visitor +# --------------------------------------------------------------------------- + + +def _reroot_sql_expr_key( + key: SqlExprKey, *, target_path: Tuple[str, ...], +) -> SqlExprKey: + """Re-anchor a STANDALONE Mode-A fragment's referenced join paths. + + Only correct when the fragment is anchored at the QUERY ROOT. A fragment + reached as ``AggregateKey.column_filter_key`` is anchored at the owning + model instead and must NOT come through here — see the note in + :func:`reroot_value_key`. + """ + stripped = [ + path[len(target_path):] + if tuple(path[: len(target_path)]) == target_path else path + for path in key.referenced_join_paths + ] + # Constructed, NOT ``model_copy``: the ``before`` validator is what sorts + # and de-duplicates ``referenced_join_paths``, and ``model_copy`` skips + # validators in Pydantic v2. Stripping can produce both — two distinct + # paths can share a residual, and the residuals need not stay in sorted + # order — and ``__hash__`` / ``__eq__`` read the tuple directly, so two + # semantically equal keys would fail to intern (CodeRabbit). + # + # An EXACT match strips to ``()``, which is not a join-path prefix at all + # but the documented "same-model filter" marker, so it is dropped rather + # than carried as an empty tuple. + return SqlExprKey( + canonical_sql=key.canonical_sql, + referenced_join_paths=[p for p in stripped if p], + ) + + +def reroot_value_key( + key: _RerootableT, *, target_path: Tuple[str, ...], +) -> _RerootableT: + """Re-anchor every embedded reference in ``key`` from the query root into + ``target_path``'s local scope. + + The generalisation of :func:`reroot_aggregate_key` over the whole + ``ValueKey`` union (plus the standalone ``SqlExprKey``). Two properties + make it safe to reroot a plan structurally instead of via formula text: + + **Total.** Every union member has an explicit case. A kind added to + ``ValueKey`` later has none, so it lands in the fail-closed arm rather than + riding through unrerooted. + + **Fail-closed.** An unhandled kind raises ``TypeError``. Returning it + unchanged would be indistinguishable from "correctly identity", which is + how a mis-anchored ref reaches the SQL generator looking well-formed. + + The rule is prefix-strip-with-residual, applied per position: a ``path`` + starting with ``target_path`` drops that prefix and keeps the residual + hops; any other ``path``, and any scalar, is returned unchanged. + ``target_path == ()`` is the identity — the empty prefix strips zero hops. + + ``AggregateKey.column_filter_key`` is deliberately copied UNCHANGED. + ``binding._resolve_column_filter_key`` walks ``source.path`` first and only + then stamps the anchor, so the fragment's paths are expressed relative to + the model that OWNS the filtered column. Rerooting changes how that owner + is reached from the query root; it never moves the owner, so those paths + are invariant. A standalone ``SqlExprKey`` is anchored at the query root + and therefore does strip — the asymmetry is per position, not per type. + """ + target_path = tuple(target_path) + if not target_path: + return key + + def _recurse(value): + return reroot_value_key(value, target_path=target_path) + + # Scalars ride through untouched — they appear as ScalarCallKey args and + # as AggregateKey kwarg values. + if key is None or isinstance(key, (Decimal, str, bool, int, float)): + return key + + # --- leaves --------------------------------------------------------- + if isinstance(key, (ColumnKey, ColumnSqlKey, StarKey)): + # ``_reroot_path_ref`` also accepts bare scalars, so it cannot carry the + # type-preserving annotation; the isinstance guard above establishes it. + return cast(_RerootableT, _reroot_path_ref(key, target_path=target_path)) + if isinstance(key, LiteralKey): + return key + if isinstance(key, TimeTruncKey): + # The path lives on the WRAPPED column, which is why walk_value_keys + # needs a special case here; the visitor must not inherit that blind + # spot. + return key.model_copy(update={"column": _recurse(key.column)}) + if isinstance(key, SqlExprKey): + return _reroot_sql_expr_key(key, target_path=target_path) + + # --- composites ----------------------------------------------------- + if isinstance(key, AggregateKey): + return key.model_copy(update={ + "source": _recurse(key.source), + "args": tuple(_recurse(a) for a in key.args), + "kwargs": tuple((n, _recurse(v)) for n, v in key.kwargs), + }) + if isinstance(key, TransformKey): + # ``args`` / ``kwargs`` are Tuple[Scalar, ...] — type-prohibited from + # holding a ValueKey, so there is nothing to traverse there. + return key.model_copy(update={ + "input": _recurse(key.input), + "partition_keys": frozenset( + _recurse(p) for p in key.partition_keys + ), + "time_key": ( + None if key.time_key is None else _recurse(key.time_key) + ), + }) + if isinstance(key, ArithmeticKey): + return key.model_copy(update={ + "operands": tuple(_recurse(o) for o in key.operands), + }) + if isinstance(key, ScalarCallKey): + return key.model_copy(update={ + "args": tuple(_recurse(a) for a in key.args), + }) + if isinstance(key, BetweenKey): + return key.model_copy(update={ + "column": _recurse(key.column), + "low": _recurse(key.low), + "high": _recurse(key.high), + }) + if isinstance(key, InKey): + return key.model_copy(update={ + "column": _recurse(key.column), + "values": tuple(_recurse(v) for v in key.values), + }) + + raise TypeError( + f"reroot_value_key has no case for {type(key).__name__!r}. The visitor " + f"is total over ValueKey by design: add an explicit case rather than " + f"letting an unrerooted key through, which the SQL generator cannot " + f"distinguish from a correctly-local one." + ) diff --git a/slayer/core/refs.py b/slayer/core/refs.py index 1ac659a0..68cdd344 100644 --- a/slayer/core/refs.py +++ b/slayer/core/refs.py @@ -116,16 +116,14 @@ def _decimal_to_plain_str(value: Decimal) -> str: def agg_kwarg_canonical_str(value: Any) -> str: """Canonicalize an AggregateKey kwarg / arg value to SQL-string form. - DEV-1450 stage 7b.13: ``EnrichedMeasure.agg_kwargs`` is typed as - ``Dict[str, str]`` and every value flows through - ``_validate_agg_param_value`` (``slayer/sql/generator.py:172``) which - accepts only identifiers, qualified names, or numeric literals. - Sites that build the synth ``EnrichedMeasure`` from a typed - ``AggregateKey`` -- AND the two canonical-alias renderers that - previously called ``str(v)`` directly (``slayer/sql/generator.py:3753`` - and ``slayer/engine/cross_model_planner.py:286``) -- route every - kwarg value through this helper instead, so a ``ColumnKey`` never - surfaces as Pydantic-repr noise. + Agg kwarg / arg values must reach ``_build_agg`` as SQL strings that + ``_validate_agg_param_value`` (``slayer/sql/generator.py``) accepts — + identifiers, qualified names, or numeric literals. Sites that build the + synth ``AggRenderSpec`` from a typed ``AggregateKey`` -- AND the two + canonical-alias renderers that previously called ``str(v)`` directly + (``slayer/sql/generator.py`` and ``slayer/engine/cross_model_planner.py``) + -- route every kwarg value through this helper instead, so a ``ColumnKey`` + never surfaces as Pydantic-repr noise. Conversion rules: diff --git a/slayer/core/warnings.py b/slayer/core/warnings.py index 0f5a1659..21909887 100644 --- a/slayer/core/warnings.py +++ b/slayer/core/warnings.py @@ -17,13 +17,38 @@ from __future__ import annotations -from typing import Optional +from typing import Annotated, Literal, Optional, Union -from pydantic import BaseModel +from pydantic import BaseModel, Field -class NormalizationWarning(BaseModel): - """Structured payload describing one slack-normalization rewrite. +class SlayerWarning(BaseModel): + """Base of the warning family carried on ``SlayerResponse.warnings``. + + ``SlayerResponse.warnings`` used to be normalization-only, so consumers + could assume every element had a ``rule_id``. It now carries more than one + kind of advisory, so every payload declares a ``kind`` discriminator and a + consumer switches on it rather than on the presence of a field. + """ + + kind: str + + def human_message(self) -> str: + """One operator-readable line describing this advisory. + + Lives on the payload so MCP, the CLI, and any future surface render a + given kind identically, and so a NEW kind cannot silently fall back to + a Pydantic repr on one surface and a hand-written string on another. + Subclasses override; the base is the honest last resort. + """ + return f"{self.kind}: {self.model_dump(exclude={'kind'})}" + + +class NormalizationWarning(SlayerWarning): + """Structured payload describing one slack-normalization event — a REWRITE + (``rewritten=True``, the default) or a report-only advisory + (``rewritten=False``, e.g. ``MALFORMED_DATE_RANGE``, which the planner + silently no-ops rather than rewriting). ``rule_id`` identifies the rule that fired (``FUNC_STYLE_AGG``, ``DOT_PATH_IN_SQL``, ``MISPLACED_MEASURE``). ``location`` is a @@ -32,11 +57,56 @@ class NormalizationWarning(BaseModel): into ``docs/agent_input_slack.md``. """ + kind: Literal["normalization"] = "normalization" rule_id: str original: str normalized: str location: str rule_doc_url: Optional[str] = None + # Some rules (MALFORMED_DATE_RANGE) REPORT without rewriting; the message + # must not claim a transform that never happened (DEV-1783). + rewritten: bool = True + + def human_message(self) -> str: + if not self.rewritten: + return ( + f"[{self.rule_id}] flagged {self.original}: {self.normalized} " + f"(at {self.location})" + ) + return ( + f"[{self.rule_id}] rewrote {self.original} → {self.normalized} " + f"(at {self.location})" + ) + + +class DroppedFilterWarning(SlayerWarning): + """A user filter that could not be applied where it was routed. + + Carries the filter's ORIGINAL author text (not the normalized, prequoted + or re-rendered form — the author has to recognise it), the surface it came + from, and why it was dropped. + """ + + kind: Literal["unreachable_filter_dropped"] = "unreachable_filter_dropped" + filter_text: str + location: str + reason: str + + def human_message(self) -> str: + return ( + f"dropped filter {self.filter_text!r} (at {self.location}): " + f"{self.reason}" + ) + + +# The response carries a DISCRIMINATED union, not the bare base class: Pydantic +# validates a ``List[SlayerWarning]`` down to the base type and would silently +# drop every subclass field on the way through. Keyed on ``kind``, each payload +# round-trips as itself. +AnySlayerWarning = Annotated[ + Union[NormalizationWarning, DroppedFilterWarning], + Field(discriminator="kind"), +] class SlayerNormalizationWarning(UserWarning): @@ -49,7 +119,7 @@ class SlayerNormalizationWarning(UserWarning): def __init__(self, payload: NormalizationWarning) -> None: self.payload = payload - super().__init__( - f"[{payload.rule_id}] {payload.original!s} → {payload.normalized!s} " - f"(at {payload.location})" - ) + # One source of truth for the wording, so a non-rewrite rule reads the + # same on the Python-warnings channel as in the structured payload + # (DEV-1783). + super().__init__(payload.human_message()) diff --git a/slayer/cube/refs.py b/slayer/cube/refs.py index a149380b..4ace8383 100644 --- a/slayer/cube/refs.py +++ b/slayer/cube/refs.py @@ -7,6 +7,8 @@ import re +from slayer.core.refs import IDENTIFIER_RE + _JINJA_RE = re.compile(r"\{\{|\{%|%\}|\}\}") _LITERAL_RE = re.compile(r"'(?:''|[^'])*'") # SQL string literal, doubled-quote aware _REF_RE = re.compile(r"\{([^{}]+)\}") @@ -17,7 +19,6 @@ # `\bAND\b` (no surrounding `\s+` quantifiers) avoids the polynomial-backtracking # shape Sonar S5852 flags; operands are whitespace-stripped after the split. _AND_SPLIT = re.compile(r"\bAND\b", re.IGNORECASE) -_IDENTIFIER_RE = re.compile(r"^[A-Za-z_]\w*$") def contains_jinja(text: str) -> bool: @@ -126,4 +127,4 @@ def _equality_pair(part: str, *, source_cube: str, target_cube: str) -> list[str def is_bare_identifier(sql: str) -> bool: """True if ``sql`` is a single bare column identifier (usable in join_pairs).""" - return bool(_IDENTIFIER_RE.match(sql.strip())) + return bool(IDENTIFIER_RE.match(sql.strip())) diff --git a/slayer/engine/agg_registry.py b/slayer/engine/agg_registry.py index 4b7c224a..c80aa883 100644 --- a/slayer/engine/agg_registry.py +++ b/slayer/engine/agg_registry.py @@ -1,8 +1,8 @@ """Stage 4 (DEV-1450) — aggregation registry helpers. -Lifts the agg-name collection BFS from ``enrichment.py`` and the -parameter-resolution helpers from ``sql/generator.py`` so the new binder -modules don't have to reach into those tangles. The helpers are pure: +Collects the agg-name BFS and the parameter-resolution helpers so the new +binder modules don't have to reach into ``sql/generator.py``'s tangles. +The helpers are pure: given a model + a resolve_join_target callback, they produce structured results without touching storage or spawning side maps. diff --git a/slayer/engine/binding.py b/slayer/engine/binding.py index 0a8eb811..23e7f53d 100644 --- a/slayer/engine/binding.py +++ b/slayer/engine/binding.py @@ -54,6 +54,7 @@ ) from slayer.core.keys import ( SCALAR_FUNCTIONS, + check_scalar_arity, AggregateKey, ArithmeticKey, BetweenKey, @@ -505,6 +506,19 @@ def _bind_in( LiteralKey(value=normalize_scalar(elt.value)) for elt in parsed.right.elements ) + # SQL's three-valued logic makes a NULL in the list a trap rather than a + # member test. ``col IN (a, NULL)`` never matches on the NULL, and + # ``col NOT IN (a, NULL)`` evaluates to NULL for EVERY row — so the filter + # silently returns zero rows instead of "everything except a". Neither is + # what the author meant, and neither announces itself. + if any(v.value is None for v in values): + raise ValueError( + f"NULL is not allowed inside an {parsed.op!r} list: SQL compares " + f"it by three-valued logic, so 'not in' with a NULL matches NO " + f"rows at all. Test for null separately — e.g. " + f"`col is null` / `col is not null` — combined with the " + f"{parsed.op!r} over the non-null values." + ) return InKey( column=column, values=values, @@ -828,8 +842,7 @@ def _bind_agg( column_filter_key = _resolve_column_filter_key( source=source, bundle=bundle, ) - # Codex review: enforce per-column aggregation eligibility gates - # the legacy enrichment site at ``enrichment.py:401-417`` enforced. + # Codex review: enforce the per-column aggregation eligibility gates. # Without this, ``id:sum`` (a PK) or ``status:avg`` (text) compile # silently in the typed pipeline. The check is best-effort against # the bundle — sources whose target model can't be resolved (e.g. @@ -939,7 +952,7 @@ def _validate_agg_eligibility( token exactly matches a custom aggregation registered on the owning model, so a custom ``countd`` wins over the ``countd -> count_distinct`` alias. - Gate order (mirrors the legacy ``enrichment.py`` v2 contract): + Gate order (the binding contract): 0. Unknown-name-first: a name that is neither a built-in nor a model custom aggregation raises ``"Unknown aggregation ..."`` **before** the @@ -1285,11 +1298,20 @@ def _bind_scalar( f"{sorted(SCALAR_FUNCTIONS)}." ), ) - if parsed.name == "like" and len(parsed.args) != 2: - raise ValueError( - f"Scalar function 'like' takes exactly 2 arguments " - f"(value, pattern); got {len(parsed.args)}." - ) + # Arity for EVERY allowlisted scalar, not just like. sqlglot's own + # handling is inconsistent — a wrong-arity round silently drops the + # extra argument, length emits SQL the database rejects — so a + # mistyped filter deserves a clear error here instead. + arity_error = check_scalar_arity( + name=parsed.name, argc=len(parsed.args), + ) + if arity_error is not None: + if parsed.name == "like": + raise ValueError( + f"Scalar function 'like' takes exactly 2 arguments " + f"(value, pattern); got {len(parsed.args)}." + ) + raise ValueError(arity_error) args = tuple( _bind(a, scope=scope, bundle=bundle, in_filter=in_filter, alias_map=alias_map) for a in parsed.args diff --git a/slayer/engine/column_expansion.py b/slayer/engine/column_expansion.py index 27b890e3..6168b15e 100644 --- a/slayer/engine/column_expansion.py +++ b/slayer/engine/column_expansion.py @@ -196,10 +196,7 @@ def collect_root_scope_joined_paths( # already loaded every referenced model (P11: storage consulted once, up # front), so it expands derived refs through a *sync* model resolver. # -# DEV-1485: the async twins (``_walk_path_to_target`` / ``_process_column_node`` -# / ``expand_derived_refs``) resolved join targets through ``storage.get_model`` -# for the legacy enrichment path and are deleted with it, as this comment always -# said they would be. Nothing awaits model resolution here any more. +# Nothing awaits model resolution here — the expansion is fully synchronous. SyncResolveModel = Callable[[str], Optional[SlayerModel]] @@ -245,14 +242,20 @@ def _process_column_node_sync( visited: Tuple[Tuple[str, str], ...], is_root: bool, root_scope_ids: Set[int], -) -> None: +) -> Optional[exp.Expression]: """Expand one ``exp.Column`` node in place if it names a derived column. - Returns ``True`` when the node was rewritten, ``False`` when it was left - alone (a physical column, or an unresolvable/opaque alias path). + Returns the replacement expression when the node was rewritten, ``None`` + when it was left alone (a physical column — which is qualified in place — + or an unresolvable/opaque alias path). + + The return value matters only when ``col`` is the ROOT of the tree being + walked: ``Expression.replace`` swaps a node inside its parent, so on a + parentless root it is a no-op and the expansion would be computed and then + silently discarded. The caller rebinds the root from this return value. """ if col.args.get("db") or col.args.get("catalog"): - return + return None table_id = col.args.get("table") col_name = col.name table_alias = table_id.name if table_id is not None else alias_path @@ -264,13 +267,13 @@ def _process_column_node_sync( is_root=is_root, ) if target_model is None or canonical_alias is None: - return + return None target_col = target_model.get_column(col_name) if target_col is None or _is_trivial_base(column=target_col): col.set("table", exp.to_identifier(canonical_alias)) - return + return None if id(col) not in root_scope_ids: - return + return None next_is_root = is_root and (target_model is model) key = (target_model.name, col_name) if key in visited: @@ -287,9 +290,11 @@ def _process_column_node_sync( is_root=next_is_root, ) if expanded_sql is None: - return + return None expanded_ast = sqlglot.parse_one(expanded_sql, dialect=dialect) - col.replace(exp.Paren(this=expanded_ast)) + replacement = exp.Paren(this=expanded_ast) + col.replace(replacement) + return replacement def expand_derived_refs_sync( @@ -320,7 +325,7 @@ def expand_derived_refs_sync( column_nodes = list(parsed.find_all(exp.Column)) root_scope_ids = _root_scope_column_ids(parsed=parsed) for col in column_nodes: - _process_column_node_sync( + replacement = _process_column_node_sync( col=col, model=model, alias_path=alias_path, @@ -330,4 +335,12 @@ def expand_derived_refs_sync( is_root=is_root, root_scope_ids=root_scope_ids, ) + # ``col.replace`` mutates the node's PARENT. When the whole fragment is + # a single column reference — ``Column.sql = "other_derived_col"``, an + # alias of another derived column — that column IS ``parsed`` and has + # no parent, so the replace is a silent no-op. Rebind the root here, or + # the correctly-expanded SQL is computed and thrown away and the + # emitted query references a derived column as though it were physical. + if replacement is not None and col is parsed: + parsed = replacement return parsed.sql(dialect=dialect) diff --git a/slayer/engine/cross_model_planner.py b/slayer/engine/cross_model_planner.py index 15ff9d8e..2438f1bf 100644 --- a/slayer/engine/cross_model_planner.py +++ b/slayer/engine/cross_model_planner.py @@ -49,9 +49,6 @@ from slayer.core.enums import DataType from slayer.core.errors import ( - AmbiguousReferenceError, - IllegalScopeReferenceError, - UnknownReferenceError, UnreachableFilterDroppedWarning, ) from slayer.core.keys import ( @@ -63,21 +60,20 @@ TimeTruncKey, ValueKey, column_path, - reroot_aggregate_key, + reroot_value_key, ) -from slayer.core.models import ModelMeasure, SlayerModel -from slayer.core.query import ColumnRef, SlayerQuery, TimeDimension -from slayer.core.refs import agg_kwarg_canonical_str, canonical_agg_name -from slayer.core.scope import ModelScope, StageColumn, StageSchema +from slayer.core.models import SlayerModel +from slayer.sql.naming import canonical_aggregate_alias +from slayer.core.scope import StageColumn, StageSchema from slayer.engine.aggregate_input_paths import ( compute_aggregate_input_join_paths, ) from slayer.engine.binding import ( - bind_expr, - bind_filter, - bind_time_dimension, + BoundExpr, + BoundFilter, walk_value_keys, ) +from slayer.engine.filter_reachability import path_is_reachable from slayer.engine.planned import ( BoundFilterId, CrossModelAggregatePlan, @@ -86,8 +82,16 @@ SlotId, ValueSlot, ) +from slayer.engine.planning import DeclaredMeasure, _canonical_name +from slayer.engine.prebound import ( + PreboundQuery, + StrictQueryCarrier, + dimension_key_metadata, + measure_key_format_description, + measure_key_type, + walk_key_path, +) from slayer.engine.source_bundle import ResolvedSourceBundle -from slayer.engine.syntax import parse_expr, parse_filter_expr # --------------------------------------------------------------------------- @@ -119,6 +123,18 @@ class HostFilterRouting(BaseModel): phase: Phase referenced_slot_ids: List[SlotId] = Field(default_factory=list) text: Optional[str] = None + # §5.4 — the typed predicate behind ``text``. A sub-plan that inherits a + # host filter re-roots THIS rather than re-parsing the string, so the + # inherited predicate keeps its structural identity. Optional because + # direct callers and test doubles build routings without one. + bound: Optional[BoundFilter] = None + # DEV-1745 (W4 / D9) — the filter's structural reachability summary, in the + # host plan's coordinate system. ``crossed_join_paths`` is every join path + # its dependency tree is anchored at; ``has_host_local_ref`` marks a + # dependency anchored at the host root, which no CTE rooted elsewhere can + # evaluate. Computed at plan time by ``filter_reachability``. + crossed_join_paths: Tuple[Tuple[str, ...], ...] = () + has_host_local_ref: bool = False # --------------------------------------------------------------------------- @@ -126,94 +142,98 @@ class HostFilterRouting(BaseModel): # --------------------------------------------------------------------------- +def _classify_referenced_slots( + *, + referenced_slot_ids: List[SlotId], + host_slots: List[ValueSlot], + target_path: Tuple[str, ...], +) -> Tuple[List[SlotId], List[SlotId], List[SlotId]]: + """Split a filter's referenced slots into + ``(unknown, aggregate_on_target, aggregate_other)``. + + Only AGGREGATE slots are sorted here — row-level reachability comes from + the structural summary, not from slot keys. An aggregate is routed by WHERE + it is computed: one whose source path IS the target can be propagated as a + HAVING inside that CTE, one computed anywhere else cannot be evaluated + there at all. + """ + by_id = {s.id: s for s in host_slots} + unknown: List[SlotId] = [] + on_target: List[SlotId] = [] + other: List[SlotId] = [] + for sid in referenced_slot_ids: + slot = by_id.get(sid) + if slot is None: + # Unknown slot id — be conservative, treat as unreachable. + unknown.append(sid) + elif isinstance(slot.key, AggregateKey): + agg_path = getattr(slot.key.source, "path", ()) + bucket = on_target if agg_path == target_path else other + bucket.append(sid) + return unknown, on_target, other + + def classify_host_filter( *, host_filter: HostFilterRouting, host_slots: List[ValueSlot], target_path: Tuple[str, ...], host_model_name: Optional[str] = None, + reachable_paths: "Optional[frozenset]" = None, ) -> FilterRoute: """Classify one host filter for cross-model CTE propagation. See the module docstring for the decision table. The classifier is pure: same inputs → same output, no side effects. - ``host_model_name`` is used to route ``ColumnSqlKey`` refs (derived - columns): the key carries its host model name but not a path, so we - compare it against the host model name and the path to decide - reachable / local / unreachable. When ``host_model_name`` is None, - ColumnSqlKey refs default to local — conservative for callers that - don't have the host model in scope. + Row-level reachability is decided EXCLUSIVELY from the filter's structural + summary (``crossed_join_paths`` / ``has_host_local_ref``, computed at plan + time by :mod:`slayer.engine.filter_reachability`) under one rule for every + key kind: a dependency is reachable iff its anchored path is a PREFIX of + ``target_path``. It is an ALL-DEPENDENCIES predicate — one unreachable + dependency drops the filter, however many others are reachable. + + This replaces a flat model-NAME membership test for derived columns, which + counted a SIBLING branch as reachable whenever it happened to share a model + name with the target path, and counted a host-model derived column whose + ``Column.sql`` crossed INTO the target as host-local. + + Aggregates keep their own arm: an aggregate is routed by WHERE it is + computed (on the target vs elsewhere), not by the reachability of its + inputs. ``host_model_name`` is accepted for signature compatibility and is + no longer consulted — the structural summary carries what it approximated. """ if host_filter.phase == Phase.POST: return FilterRoute.STAY_AT_HOST_POST - if not host_filter.referenced_slot_ids: - # No referenced slots — nothing to route into the CTE. - return FilterRoute.STAY_AT_HOST_POST - by_id = {s.id: s for s in host_slots} + unknown, aggregate_on_target, aggregate_other = _classify_referenced_slots( + referenced_slot_ids=host_filter.referenced_slot_ids, + host_slots=host_slots, + target_path=target_path, + ) - local_row: List[SlotId] = [] - reachable_path: List[SlotId] = [] - unreachable: List[SlotId] = [] - aggregate_on_target: List[SlotId] = [] - aggregate_other: List[SlotId] = [] + crossed = tuple(host_filter.crossed_join_paths) + # THE rule lives in one place. Repeating the prefix comparison here would + # be a second copy free to drift from it — the exact failure this PR removes. + unreachable_paths = [ + p for p in crossed + if not path_is_reachable( + path=p, target_path=target_path, reachable_paths=reachable_paths, + ) + ] - for sid in host_filter.referenced_slot_ids: - s = by_id.get(sid) - if s is None: - # Unknown slot id — be conservative, treat as unreachable. - unreachable.append(sid) - continue - if isinstance(s.key, AggregateKey): - agg_source = s.key.source - agg_path = getattr(agg_source, "path", ()) - if agg_path == target_path: - aggregate_on_target.append(sid) - else: - aggregate_other.append(sid) - elif isinstance(s.key, ColumnKey): - if not s.key.path: - local_row.append(sid) - elif s.key.path == target_path[: len(s.key.path)]: - reachable_path.append(sid) - else: - unreachable.append(sid) - elif isinstance(s.key, ColumnSqlKey): - # Derived column. Route by its host model: on host → local; - # on any model in target_path → reachable; otherwise → - # unreachable. - cm = s.key.model - if host_model_name is not None and cm == host_model_name: - local_row.append(sid) - elif cm in target_path: - reachable_path.append(sid) - elif host_model_name is None: - # No host name to compare against — conservative default. - local_row.append(sid) - else: - unreachable.append(sid) - else: - # Transform / Arithmetic / ScalarCall: phase already decided. - # POST was checked above; ROW/AGGREGATE land here from - # arithmetic / scalar calls. Treat as local for routing. - local_row.append(sid) - - if unreachable or aggregate_other: - # Any unreachable ref → drop + warn (covers pure-unreachable AND - # mixed-with-reachable cases per decision table rows 6/7). + if unknown or aggregate_other or unreachable_paths: return FilterRoute.DROP_UNREACHABLE - if local_row and not (aggregate_on_target or reachable_path): + if host_filter.has_host_local_ref: + # Mixed host-local + reachable. The local refs cannot be evaluated in + # a CTE rooted elsewhere, so the whole filter stays at the host. return FilterRoute.DROP_HOST_LOCAL - if local_row: - # Mixed local + (target-path / target-agg). The local refs can't - # be evaluated in the CTE, so the filter stays at host. + if not crossed and not aggregate_on_target: + # Nothing crosses and no aggregate to propagate — purely host-local. return FilterRoute.DROP_HOST_LOCAL if aggregate_on_target: return FilterRoute.PROPAGATE_HAVING - if reachable_path: - return FilterRoute.PROPAGATE_WHERE - return FilterRoute.STAY_AT_HOST_POST + return FilterRoute.PROPAGATE_WHERE # --------------------------------------------------------------------------- @@ -244,10 +264,10 @@ def plan( host_filters: List[HostFilterRouting], public_alias: Optional[str] = None, hidden: bool = False, - host_query: Optional[SlayerQuery] = None, + host_query: Optional[StrictQueryCarrier] = None, public_projection: Optional[List[SlotId]] = None, subplan_builder: Optional[ - Callable[[SlayerQuery, ResolvedSourceBundle], PlannedQuery] + Callable[[StrictQueryCarrier, ResolvedSourceBundle], PlannedQuery] ] = None, ) -> CrossModelAggregatePlan: ... @@ -312,36 +332,18 @@ def _aggregate_alias(*, key: AggregateKey) -> str: suffix matches the rest of the engine (legacy enrichment, search, DBT converter). """ - # ColumnKey -> leaf, ColumnSqlKey (derived agg source) -> column_name, - # StarKey -> "*" (CR / Codex: a derived source must alias as - # ``net_sum``, not ``_sum``). - measure_name = ( - getattr(key.source, "leaf", None) - or getattr(key.source, "column_name", None) - or "*" - ) - # AggregateKey.args / kwargs are normalised tuples of scalars / - # ColumnKey-shaped values; convert to the (List[str], - # Dict[str, Any]) shape ``canonical_agg_name`` expects. DEV-1450 - # stage 7b.13: route through ``agg_kwarg_canonical_str`` so a - # ColumnKey kwarg renders as ``leaf`` (or ``path.leaf`` for joined - # paths) instead of Pydantic-repr noise from naive ``str(v)``. - # The kwarg suffix is preserved here -- the legacy enrichment at - # ``query_engine.py:2160`` drops it, causing two parametric aggs - # with different kwargs to collide on CTE alias (legacy bug). - # The 7b.5 fix added kwarg-aware aliases here as a correctness - # improvement -- ``test_cross_model_planner_wiring.py:: - # test_parameterized_aggregates_get_distinct_cte_aliases`` pins - # this. Parity with legacy for cross-model parametric aggs is - # not achievable on this axis. - args_list = [agg_kwarg_canonical_str(a) for a in key.args] - kwargs_dict = {k: agg_kwarg_canonical_str(v) for k, v in key.kwargs} - return canonical_agg_name( - measure_name=measure_name, - aggregation_name=key.agg, - agg_args=args_list or None, - agg_kwargs=kwargs_dict or None, - ) + # The derivation itself lives in ``slayer.sql.naming`` (P-F, one naming + # authority). This is the ``cte_schema`` profile — the bare canonical + # name with no relation or path prefix, since the alias names a column + # INSIDE the CTE. + # + # The kwarg suffix is preserved -- the deleted legacy enrichment dropped it, + # causing two parametric aggs with different kwargs to collide on CTE alias. + # ``test_cross_model_planner_wiring.py:: + # test_parameterized_aggregates_get_distinct_cte_aliases`` pins this; + # parity with legacy for cross-model parametric aggs is not achievable on + # this axis. + return canonical_aggregate_alias(key, profile="cte_schema") def _make_cte_schema( @@ -439,15 +441,20 @@ def _match_filtered_local_grain_pairs( def _find_filtered_local_sub_agg_slot( *, sub_plan: PlannedQuery, - formula: str, + aggregate_key: AggregateKey, host_model: SlayerModel, ) -> SlotId: - """Locate the sub-plan's single local aggregate slot. + """Locate the sub-plan's slot for the isolated aggregate. - Recursion suppression guarantees no nested cross-model plans so the - sub-plan has exactly one local aggregate — the filtered measure being - isolated. + The exact key match comes first (§5.4: the sub-plan is planned FROM this + key, so it interns under the same identity). The path-less fallback covers + a sub-plan whose own pass rewrote the key — recursion suppression + guarantees no nested cross-model plans, so at most one local aggregate is + present to match. """ + for s in sub_plan.aggregate_slots: + if s.key == aggregate_key: + return s.id for s in sub_plan.aggregate_slots: if isinstance(s.key, AggregateKey) and not getattr( s.key.source, "path", (), @@ -455,7 +462,7 @@ def _find_filtered_local_sub_agg_slot( return s.id raise ValueError( "DEV-1503 sub-plan produced no local aggregate slot for " - f"{formula!r} on {host_model.name!r} — planner bug." + f"{aggregate_key!r} on {host_model.name!r} — planner bug." ) @@ -493,43 +500,60 @@ def _build_filtered_local_cte_schema( ) -def _classify_subplan_filters( - *, - host_filters: List[HostFilterRouting], -) -> Optional[List[str]]: - """Decide which host-query filters propagate into the DEV-1503 sub-plan. - - ROW: pass through — the sub-plan applies them to the aggregate's rowset - (otherwise a non-dim filter like ``status = 'active'`` has no effect on - the join-back aggregate value). - AGGREGATE (any slot ref): skip. Pushing such a filter into the sub-plan - as HAVING would drop CTE rows where the aggregate fails the test; the - outer LEFT JOIN then surfaces the host row with a NULL aggregate - instead of dropping it — wrong semantics. The generator's outer-WHERE - wrapper applies the filter on the joined-back column so the row is - actually dropped (DEV-1503 spec). - POST: skip — stays at the existing host post-transform wrapper. - - Consume ``routing.text`` directly — it carries the original user-filter - string for user-filter routings (None for date_range bounds) and is - populated by ``stage_planner`` from the deduped ``bound_filters`` list, - so it stays in lock-step with ``host_filters`` even after Mode-B - dedup-by-bound-key collapses two textually-different filter spellings - onto one routing (CR PR #153 thread r3350000254). Slicing - ``host_query.filters`` here would mis-pair phases when a ``date_range``- - bearing time_dimension is present (Codex review) OR when dedup drops - user-filter entries. +def _route_host_rooted_filters( + *, host_filters: List[HostFilterRouting], +) -> _FilterRoutes: + """Route host filters for a HOST-ROOTED CTE (DEV-1503 / DEV-1747 D6). + + The re-rooted table's question is reachability — can a CTE rooted at the + TARGET evaluate this predicate? A host-rooted CTE is rooted at the host, so + reachability is never in doubt and the question is PHASE instead: + + * ROW — propagate. The sub-plan applies it to the aggregate's rowset; + without it a predicate like ``status = 'active'`` would not affect the + value that joins back. + * AGGREGATE — do NOT propagate. As a HAVING inside the CTE it would drop + CTE rows where the aggregate fails, and the outer LEFT JOIN would then + surface the host row with a NULL aggregate instead of dropping it. The + generator's outer-WHERE wrapper applies it on the joined-back column, so + the row is actually dropped. + * POST — do NOT propagate; it stays at the host's post-transform wrapper. + + Nothing is ever DROPPED here, so no warning can arise: every filter is + applied somewhere, either in the CTE or at the host. + + ``where_ids`` stays EMPTY. It is the forward-CTE delegation instruction — + "the CTE took this over, so the host base must SKIP it" — but a host-rooted + CTE has no forward CTE, and the sub-plan already carries these predicates as + its own filters. Only ``applied`` (the audit) is populated (DEV-1783). """ - sub_filter_texts: List[str] = [] + applied_ids: List[BoundFilterId] = [] for routing in host_filters: if routing.text is None: - # date_range bound — not a user filter, do not propagate. - continue + continue # date_range bound — re-attached by the caller, in order if routing.phase in (Phase.POST, Phase.AGGREGATE): continue - # ROW phase — propagate. - sub_filter_texts.append(routing.text) - return sub_filter_texts or None + if routing.bound is None: + continue + applied_ids.append(routing.filter_id) + return _FilterRoutes(applied=applied_ids, where_ids=[]) + + + + +class _FilterRoutes(BaseModel): + """The routing decision for one CTE's whole host-filter set. + + Grouped into a record so it can be produced once and threaded to the plan + constructor without four positional lists (DEV-1747 D6). + """ + + model_config = ConfigDict(arbitrary_types_allowed=True) + + applied: List[BoundFilterId] = Field(default_factory=list) + where_ids: List[BoundFilterId] = Field(default_factory=list) + having_ids: List[BoundFilterId] = Field(default_factory=list) + dropped: List[UnreachableFilterDroppedWarning] = Field(default_factory=list) def _route_host_filters( @@ -539,15 +563,18 @@ def _route_host_filters( target_path: Tuple[str, ...], host_model: SlayerModel, terminal_model: SlayerModel, -) -> Tuple[ - List[BoundFilterId], List[BoundFilterId], List[BoundFilterId], - List[UnreachableFilterDroppedWarning], -]: + reachable_paths: "Optional[frozenset]" = None, +) -> _FilterRoutes: """Classify each host filter via the ``inherited_filter_policy`` decision - table (``classify_host_filter``) into ``(applied, where_ids, having_ids, - dropped)`` — extracted from ``IsolatedCteCrossModelPlanner.plan`` (DEV-1708) - to keep that method focused. ``DROP_HOST_LOCAL`` / ``STAY_AT_HOST_POST`` are - neither propagated nor warned.""" + table (``classify_host_filter``) — extracted from + ``IsolatedCteCrossModelPlanner.plan`` (DEV-1708) to keep that method + focused. ``DROP_HOST_LOCAL`` / ``STAY_AT_HOST_POST`` are neither propagated + nor warned. + + ``reachable_paths``, when supplied, is the set of host-coordinate join + paths a RE-ROOTED CTE can actually evaluate — walked from the target's own + join graph by the caller. It replaces the forward-path prefix test, so a + filter the re-rooted CTE will genuinely apply is not reported as dropped.""" applied: List[BoundFilterId] = [] where_ids: List[BoundFilterId] = [] having_ids: List[BoundFilterId] = [] @@ -558,6 +585,7 @@ def _route_host_filters( host_slots=host_slots, target_path=target_path, host_model_name=host_model.name, + reachable_paths=reachable_paths, ) if route is FilterRoute.PROPAGATE_WHERE: where_ids.append(hf.filter_id) @@ -568,13 +596,21 @@ def _route_host_filters( elif route is FilterRoute.DROP_UNREACHABLE: dropped.append(UnreachableFilterDroppedWarning( filter_text=hf.text or hf.filter_id, + # Deliberately target-INDEPENDENT (D8). The same user filter is + # classified once per cross-model plan, and the boundary dedups + # those to one warning while asserting the reasons AGREE. A + # reason naming this plan's target would make two plans + # disagree about one filter and trip that assertion. reason=( - f"filter {hf.filter_id!r} references slot(s) outside " - f"the join path to {terminal_model.name!r}; " - f"unreachable filters are dropped." + f"filter {hf.filter_id!r} depends on join path(s) that are " + f"not reachable from the cross-model aggregate's CTE root; " + f"it still applies at the host, and is dropped from the CTE." ), )) - return applied, where_ids, having_ids, dropped + return _FilterRoutes( + applied=applied, where_ids=where_ids, + having_ids=having_ids, dropped=dropped, + ) def _compute_shared_grain_slots( @@ -628,10 +664,10 @@ def plan( host_filters: List[HostFilterRouting], public_alias: Optional[str] = None, hidden: bool = False, - host_query: Optional[SlayerQuery] = None, + host_query: Optional[StrictQueryCarrier] = None, public_projection: Optional[List[SlotId]] = None, subplan_builder: Optional[ - Callable[[SlayerQuery, ResolvedSourceBundle], PlannedQuery] + Callable[[StrictQueryCarrier, ResolvedSourceBundle], PlannedQuery] ] = None, ) -> CrossModelAggregatePlan: host_model = bundle.source_model @@ -644,7 +680,12 @@ def plan( agg_source = aggregate_key.source path = getattr(agg_source, "path", ()) - if not path: + # DEV-1747 D2 — ``grain="host"`` routes to the HOST-rooted CTE even + # though the source carries a path. The path says WHERE the value is + # read from; the grain says WHERE it is grouped. A joined ORDER BY wrap + # reads through the join but must be grouped per HOST row-group, so it + # belongs on the same route as a crossing-input local aggregate. + if not path or getattr(aggregate_key, "grain", "target") == "host": return self._dispatch_filtered_local( aggregate_slot_id=aggregate_slot_id, aggregate_key=aggregate_key, @@ -677,14 +718,6 @@ def plan( )) target_path = path - applied, where_ids, having_ids, dropped = _route_host_filters( - host_filters=host_filters, - host_slots=host_slots, - target_path=target_path, - host_model=host_model, - terminal_model=terminal_model, - ) - target_model_filters = list(terminal_model.filters or []) # Shared grain: host ROW dimensions / time-dimensions on the target's @@ -706,38 +739,55 @@ def plan( join_back_pairs=join_back_pairs, ) - forward_plan = CrossModelAggregatePlan( - aggregate_slot_id=aggregate_slot_id, - target_model=terminal_model.name, - datasource=host_model.data_source, - join_chain=join_chain, - join_back_pairs=join_back_pairs, - cte_stage_schema=cte_schema, - shared_grain_slots=shared_grain, - applied_filter_ids=applied, - where_filter_ids=where_ids, - having_filter_ids=having_ids, - target_model_filters=target_model_filters, - dropped_filter_warnings=dropped, - hidden=hidden, - public_alias=public_alias, - ) + def _make_plan(routes: "_FilterRoutes") -> CrossModelAggregatePlan: + return CrossModelAggregatePlan( + aggregate_slot_id=aggregate_slot_id, + target_model=terminal_model.name, + datasource=host_model.data_source, + join_chain=join_chain, + join_back_pairs=join_back_pairs, + cte_stage_schema=cte_schema, + shared_grain_slots=shared_grain, + applied_filter_ids=routes.applied, + where_filter_ids=routes.where_ids, + having_filter_ids=routes.having_ids, + target_model_filters=target_model_filters, + dropped_filter_warnings=routes.dropped, + hidden=hidden, + public_alias=public_alias, + ) # DEV-1450 #2: re-rooting is the strategy's call. When the caller # supplies the host query + a sub-plan builder, decide forward-plan # vs re-rooted-plan here; without them (direct ``plan(...)`` callers / # test doubles) return the forward plan unchanged. + # + # DEV-1747 D6 — the reroot decision is made BEFORE the filters are + # classified, so the one classification runs in the coordinate system + # of the CTE that will actually exist. Classifying against the forward + # path and then re-rooting is how a reachable filter ended up judged + # unreachable, and then had that judgement blanked. if subplan_builder is not None and host_query is not None: return _maybe_reroot_cross_model_plan( - plan=forward_plan, + make_plan=_make_plan, query=host_query, agg_key=aggregate_key, bundle=bundle, host_model=host_model, + host_slots=host_slots, + host_filters=host_filters, public_projection=public_projection or [], subplan_builder=subplan_builder, + target_model_name=terminal_model.name, + target_path=target_path, ) - return forward_plan + return _make_plan(_route_host_filters( + host_filters=host_filters, + host_slots=host_slots, + target_path=target_path, + host_model=host_model, + terminal_model=terminal_model, + )) # ---------------------------------------------------------------------- # DEV-1503 — filtered-local isolation @@ -754,10 +804,10 @@ def _dispatch_filtered_local( host_filters: List[HostFilterRouting], public_alias: Optional[str], hidden: bool, - host_query: Optional[SlayerQuery], + host_query: Optional[StrictQueryCarrier], public_projection: Optional[List[SlotId]], subplan_builder: Optional[ - Callable[[SlayerQuery, ResolvedSourceBundle], PlannedQuery] + Callable[[StrictQueryCarrier, ResolvedSourceBundle], PlannedQuery] ], ) -> CrossModelAggregatePlan: """Validate the host-rooted trigger preconditions and dispatch @@ -772,7 +822,16 @@ def _dispatch_filtered_local( has_crossing_filter = cfk is not None and bool( cfk.referenced_join_paths, ) - has_crossing_input = has_crossing_filter or bool( + # DEV-1747 D2 — for a ``grain="host"`` aggregate the SOURCE PATH is + # itself the crossing input: the value is read through a join that the + # CTE has to pull in. Omitting it here would make the check below + # reject the wrap as "a plain local aggregate", since a path-bearing + # source carries no ``column_filter_key`` and no crossing arg. + has_crossing_source = ( + getattr(aggregate_key, "grain", "target") == "host" + and bool(getattr(agg_source, "path", ())) + ) + has_crossing_input = has_crossing_filter or has_crossing_source or bool( compute_aggregate_input_join_paths( key=aggregate_key, anchor_model=host_model, @@ -821,49 +880,66 @@ def _plan_filtered_local( host_model: SlayerModel, host_slots: List[ValueSlot], host_filters: List[HostFilterRouting], - host_query: SlayerQuery, + host_query: StrictQueryCarrier, public_alias: Optional[str], public_projection: List[SlotId], hidden: bool, subplan_builder: Callable[ - [SlayerQuery, ResolvedSourceBundle], PlannedQuery, + [StrictQueryCarrier, ResolvedSourceBundle], PlannedQuery, ], ) -> CrossModelAggregatePlan: """Build a host-rooted nested sub-plan for a cross-model-FILTERED local measure (DEV-1503). - The sub-plan is a ``SlayerQuery`` rooted at the SAME host model with - ``measures=[]`` and the host's dimensions / - time_dimensions. The sub-plan's ``plan_query`` recursion handles - the filter-target join (its ``Column.filter`` will pull in the - joined table at the generator's inline path), the host model's own + The sub-plan is a ``PreboundQuery`` rooted at the SAME host model, + carrying the filtered measure and the host's bound dimensions / + time dimensions. The sub-plan's ``plan_query`` recursion handles the + filter-target join (its ``Column.filter`` pulls in the joined table at + the generator's inline path), the host model's own ``SlayerModel.filters``, and the per-dimension GROUP BY — producing a per-grain aggregate that the host base LEFT JOINs back. - Host query filters are NOT propagated into the sub-plan here — the - host base CTE applies them. The generator's outer-WHERE wrapper - handles aggregate-referencing filters separately (DEV-1503 spec). + Only ROW-phase host filters propagate; the rest stay at the host base + or at the generator's outer-WHERE wrapper (DEV-1503 spec). """ - # Reconstruct the local measure formula from the AggregateKey. The - # source.path is empty so ``_local_agg_formula`` emits a bare - # ``leaf:agg`` shape (plus any args / kwargs). Carry the user- - # supplied alias through so a host filter referencing the rename - # (``latest_pmt > 500`` for a measure named ``latest_pmt``) binds - # against the same alias in the sub-plan rather than the canonical - # ``latest_payment_last_updated_at`` form. - formula = _local_agg_formula(aggregate_key) - measure_name_for_subplan = public_alias - sub_filters = _classify_subplan_filters(host_filters=host_filters) - rerooted_query = SlayerQuery( - source_model=host_model.name, - measures=[ModelMeasure( - formula=formula, name=measure_name_for_subplan, - )], - dimensions=list(host_query.dimensions or []) or None, - time_dimensions=list(host_query.time_dimensions or []) or None, - filters=sub_filters, + # §5.4 — the sub-plan is rooted at the SAME host model, so the + # aggregate key and the host's bound dimensions carry over VERBATIM; + # there is nothing to re-root and so nothing to serialize. The + # user-supplied alias rides along so a host filter referencing the + # rename (``latest_pmt > 500`` for a measure named ``latest_pmt``) + # resolves against the same alias in the sub-plan rather than the + # canonical ``latest_payment_last_updated_at`` form. + host_prebound = host_query.prebound + if host_prebound is None: + raise ValueError( + "DEV-1503 filtered-local isolation needs the host's typed " + "bind product; the carrier arrived without one. The " + "stage_planner must pass the PreboundQuery it planned from." + ) + host_rooted_routes = _route_host_rooted_filters( + host_filters=host_filters, + ) + routing_by_id = {hf.filter_id: hf for hf in host_filters} + sub_prebound = _nested_prebound( + host_prebound=host_prebound, + aggregate_measure=_aggregate_declared_measure( + key=aggregate_key, + model=host_model, + public_alias=public_alias, + ), + grain_measures=list(_grain_declared_measures(host_prebound)), + inherited_filters=[ + routing_by_id[fid].bound + for fid in host_rooted_routes.applied + if routing_by_id[fid].bound is not None + ], + ) + sub_plan = subplan_builder( + StrictQueryCarrier( + source_model=host_model.name, prebound=sub_prebound, + ), + bundle, ) - sub_plan = subplan_builder(rerooted_query, bundle) grain_pairs = _match_filtered_local_grain_pairs( host_slots=host_slots, @@ -871,7 +947,8 @@ def _plan_filtered_local( sub_plan=sub_plan, ) sub_agg_sid = _find_filtered_local_sub_agg_slot( - sub_plan=sub_plan, formula=formula, host_model=host_model, + sub_plan=sub_plan, aggregate_key=aggregate_key, + host_model=host_model, ) cte_schema = _build_filtered_local_cte_schema( aggregate_key=aggregate_key, host_model=host_model, @@ -889,7 +966,13 @@ def _plan_filtered_local( join_back_pairs=[], cte_stage_schema=cte_schema, shared_grain_slots=[host_sid for host_sid, _ in grain_pairs], - applied_filter_ids=[], + # DEV-1747 D6 — the plan states which host filters the CTE applies + # rather than reporting an empty routing while the sub-plan quietly + # carries them. ``where_filter_ids`` stays empty: the sub-plan + # already holds these predicates as its OWN filters (they were + # handed to it typed), so listing them again would render them + # twice. Nothing is dropped on this route, so no warning can arise. + applied_filter_ids=host_rooted_routes.applied, where_filter_ids=[], having_filter_ids=[], target_model_filters=[], @@ -913,13 +996,18 @@ def _plan_filtered_local( # collapses the host dimension to a scalar CROSS JOIN -- every host row gets # the global aggregate. # -# The fix mirrors legacy ``_build_rerooted_enriched``: build a full nested -# ``SlayerQuery`` rooted at the target (so all of the target's joins are in -# scope for dimensions AND filters), compile it via ``subplan_builder``, and -# attach the sub-plan to the ``CrossModelAggregatePlan``. The generator -# renders the sub-plan as the ``_cm_*`` CTE and joins it back to the host base -# on the (re-rooted) dimension. Dimensions / filters that don't resolve from -# the target are dropped -- matching legacy's drop-unreachable behaviour. +# The fix: re-anchor the host's bound keys into the target's coordinate +# system (so all of the target's joins are in scope for dimensions AND +# filters), plan them via ``subplan_builder``, and attach the sub-plan to the +# ``CrossModelAggregatePlan``. The generator renders the sub-plan as the +# ``_cm_*`` CTE and joins it back to the host base on the re-rooted dimension. +# Dimensions / filters that don't resolve from the target are dropped. +# +# DEV-1742 §5.4: the re-anchoring is STRUCTURAL. Until then this pass +# regenerated formula text for every ref and let the planner re-bind it, so a +# key's identity survived only as far as the string could carry it — which is +# why a path-bearing source or a host-grain marker could not be expressed at +# all. ``_reroot_host_key`` transforms the key; nothing is re-parsed. # # DEV-1450 #2: this used to be a post-hoc pass in ``stage_planner.plan_query``; # it now lives behind ``IsolatedCteCrossModelPlanner.plan`` so the @@ -928,248 +1016,450 @@ def _plan_filtered_local( # module does not import ``stage_planner`` (no cycle). -def _reroot_ref( - *, model_prefix: Optional[str], name: str, host_model_name: str, - target_model_name: str, -) -> str: - """Re-root one Mode-B ref from the host's perspective to the target's. +def _grain_declared_measures(prebound: PreboundQuery) -> List[DeclaredMeasure]: + """The host's dimension + time-dimension declarations — the grain prefix of + ``declared_measures``, which the nested plan groups by unchanged.""" + n = prebound.n_dims + prebound.n_time_dimensions + return list(prebound.declared_measures[:n]) + + +def _aggregate_declared_measure( + *, + key: AggregateKey, + model: SlayerModel, + public_alias: Optional[str], +) -> DeclaredMeasure: + """The nested plan's single measure declaration, built from the key. + + Reproduces what ``_declared_measures_from_query`` derives for a measure — + canonical alias, type, format, description — without a formula string to + re-parse. An explicit ``public_alias`` surfaces as the name while the + canonical form is retained as ``canonical_alias``, so a colon-form filter + or ORDER BY still resolves onto the same slot (DEV-1443). + """ + canonical = canonical_aggregate_alias(key, profile="stage_formula") + if canonical is None: # pragma: no cover — binder restricts source shapes + canonical = _canonical_name(key) + fmt, desc = measure_key_format_description(model=model, key=key) + return DeclaredMeasure( + bound=BoundExpr(value_key=key), + declared_name=public_alias or canonical, + public_name=public_alias or canonical, + canonical_alias=canonical if public_alias else None, + type=measure_key_type(model=model, key=key), + format=fmt, + description=desc, + ) + + +def _nested_prebound( + *, + host_prebound: PreboundQuery, + aggregate_measure: DeclaredMeasure, + grain_measures: List[DeclaredMeasure], + inherited_filters: List[BoundFilter], + date_range_filters: Optional[List[BoundFilter]] = None, + n_dims: Optional[int] = None, + n_time_dimensions: Optional[int] = None, + main_time_key: Optional[TimeTruncKey] = None, +) -> PreboundQuery: + """Assemble the nested plan's bind product from typed pieces (§5.4). + + Filter order mirrors ``bind_query_inputs``: date-range bounds first (so + ``n_date_range`` still slices them off), then inherited user filters. The + nested plan is never ordered or paginated — the host owns both. + """ + bounds = list( + host_prebound.bound_filters[: host_prebound.n_date_range] + if date_range_filters is None else date_range_filters + ) + return PreboundQuery( + declared_measures=[*grain_measures, aggregate_measure], + bound_filters=[*bounds, *inherited_filters], + bound_filter_texts=( + [None] * len(bounds) + [None] * len(inherited_filters) + ), + n_date_range=len(bounds), + order_specs=[], + main_time_key=( + host_prebound.main_time_key if main_time_key is None + else main_time_key + ), + n_dims=host_prebound.n_dims if n_dims is None else n_dims, + n_time_dimensions=( + host_prebound.n_time_dimensions if n_time_dimensions is None + else n_time_dimensions + ), + distinct_dimension_values=True, + ) - Mirrors legacy ``_build_rerooted_enriched``: - * host-local (``model_prefix is None``) -> ``.`` (now a - cross-model dim from the target's view), - * on the target itself -> bare ```` (local on target), - * a path through the target -> strip the target prefix, - * any other dotted ref -> kept as-is (resolved via the target's joins). +def _reroot_host_path( + path: Tuple[str, ...], *, target_path: Tuple[str, ...], + host_model_name: str, +) -> Tuple[str, ...]: + """The path-level half of :func:`_reroot_host_key` — the same three rules, + applied to a bare join path so reachability can be decided without a key.""" + path = tuple(path) + if not path: + return (host_model_name,) + if path[: len(target_path)] == tuple(target_path): + return path[len(target_path):] + return path + + +def _rerooted_reachable_paths( + *, + host_filters: List[HostFilterRouting], + target_path: Tuple[str, ...], + target_model: SlayerModel, + host_model_name: str, + bundle: ResolvedSourceBundle, +) -> frozenset: + """Which host-coordinate join paths a RE-ROOTED CTE can evaluate. + + A prefix test cannot answer this: the CTE is rooted at the target with the + target's whole join graph in scope, so a host-side SIBLING branch is + reachable whenever the target happens to join to it too — common in star + schemas, where several fact-adjacent tables share a dimension. Walking the + graph is the only honest test, and getting it wrong in either direction is + a correctness bug: too narrow drops a filter the user wrote, too wide + emits SQL referencing an unbound table. """ - if model_prefix is None: - return f"{host_model_name}.{name}" - if model_prefix == target_model_name: - return name - if model_prefix.startswith(target_model_name + "."): - return f"{model_prefix[len(target_model_name) + 1:]}.{name}" - return f"{model_prefix}.{name}" - - -def _host_ref_path(model_prefix: Optional[str]) -> Tuple[str, ...]: - """The join path a host ColumnRef / TimeDimension prefix denotes.""" - if not model_prefix: - return () - return tuple(model_prefix.split(".")) - - -def _scalar_formula_literal(value) -> str: - """Render a normalized scalar back into formula text.""" - if isinstance(value, bool): - return "True" if value else "False" - if value is None: - return "None" - if isinstance(value, str): - return repr(value) - return str(value) - - -def _filter_ref_paths(value_key: ValueKey) -> List[Tuple[str, ...]]: - """Join paths of every column-like leaf a (bound) filter references.""" - paths: List[Tuple[str, ...]] = [] - for k in walk_value_keys(value_key): - if isinstance(k, (ColumnKey, ColumnSqlKey, StarKey)): - paths.append(tuple(k.path)) - elif isinstance(k, TimeTruncKey): - paths.append(tuple(column_path(k.column))) - return paths - - -def _render_ref_formula(ref) -> str: - """Render one already-rerooted embedded reference back into formula text. - - Column-like refs dot-join their (residual) path with the leaf; scalars - fall through to ``_scalar_formula_literal``. Contains NO path-stripping - decisions — the reroot has already happened (DEV-1707). + reachable = set() + for hf in host_filters: + for p in hf.crossed_join_paths: + if not p or p in reachable: + continue + rr = _reroot_host_path( + p, target_path=target_path, host_model_name=host_model_name, + ) + if walk_key_path( + model=target_model, path=rr, bundle=bundle, + ) is not None: + reachable.add(tuple(p)) + return frozenset(reachable) + + +def _reroot_host_key( + key: ValueKey, *, target_path: Tuple[str, ...], host_model_name: str, +) -> ValueKey: + """Re-anchor one host-coordinate key into the target's coordinate system. + + Three rules (the typed reroot; the superseded formula-text round-trip it + replaced was deleted in PR 6, DEV-1749): + + * host-local (empty path) → reached FROM the target by naming the host as + the first hop, so ``status`` becomes ``orders.status``; + * on or through the target → the target prefix is stripped, which is + exactly ``reroot_value_key``; + * anywhere else → unchanged, to be resolved through the target's own joins. + + The host-local prepend is per-key rather than per-leaf: a composite whose + leaves sit at different depths would need each leaf re-anchored + separately, and no such shape reaches re-rooting today (dimensions and + time dimensions are single references). ``_key_reaches_from`` rejects + anything that does not resolve, so a future composite is dropped rather + than mis-anchored. """ - if isinstance(ref, ColumnSqlKey): - return ".".join((*ref.path, ref.column_name)) - if isinstance(ref, ColumnKey): - return ".".join((*ref.path, ref.leaf)) - return _scalar_formula_literal(ref) - - -def _local_agg_formula(key: AggregateKey) -> str: - """Reconstruct the LOCAL colon-formula for a cross-model aggregate - (``customers.revenue:sum`` -> ``revenue:sum``) so it can be re-planned - against the target model as a plain local measure. - - Every embedded reference — source, positional args, and column-valued - kwargs — is re-anchored symmetrically via the unified - ``reroot_aggregate_key`` (DEV-1707), then rendered by the path-free - ``_render_ref_formula``. A kwarg / arg one hop past the target keeps its - residual path (``other=regions.code``); an exact match becomes local - (``other=region_id``). The public string contract is unchanged — the - strip logic simply no longer lives here. + inner = key.column if isinstance(key, TimeTruncKey) else key + path = tuple(getattr(inner, "path", ()) or ()) + if not path: + if not hasattr(inner, "path"): + return key + rerooted = inner.model_copy(update={"path": (host_model_name,)}) + if isinstance(key, TimeTruncKey): + return key.model_copy(update={"column": rerooted}) + return rerooted + return reroot_value_key(key, target_path=target_path) + + +def _key_reaches_from( + *, key: ValueKey, model: SlayerModel, bundle: ResolvedSourceBundle, +) -> bool: + """Whether every column-like leaf of ``key`` resolves from ``model``. + + The structural stand-in for "does this bind against the target scope?" — + re-rooting must not call the binder (§5.4), so reachability is decided by + walking the join graph and checking the terminal model owns the leaf. """ - local = reroot_aggregate_key( - key, target_path=tuple(getattr(key.source, "path", ())), + saw_column = False + for k in walk_value_keys(key): + if isinstance(k, TimeTruncKey): + continue # its wrapped column is walked in its own right + if not isinstance(k, (ColumnKey, ColumnSqlKey, StarKey)): + continue + saw_column = True + terminal = walk_key_path( + model=model, path=tuple(k.path), bundle=bundle, + ) + if terminal is None: + return False + if isinstance(k, StarKey): + continue + leaf = getattr(k, "leaf", None) or getattr(k, "column_name", None) + if leaf is None or terminal.get_column(leaf) is None: + return False + return saw_column + + +def _rerooted_dimension_measure( + *, + key: ValueKey, + label: Optional[str], + target_model: SlayerModel, + bundle: ResolvedSourceBundle, +) -> DeclaredMeasure: + """One re-rooted dimension / time-dimension declaration for the sub-plan. + + ``_canonical_name`` produces the same ``__``-flattened alias the text path + derived from the re-rooted dotted reference, so the CTE's column names and + the host's join-back are unchanged by the switch to typed re-rooting. + """ + if isinstance(key, TimeTruncKey): + return DeclaredMeasure( + bound=BoundExpr(value_key=key), + declared_name=_canonical_name(key), + public_name=_canonical_name(key), + label=label, + type=DataType.TIMESTAMP, + ) + dim_type, fmt, desc = dimension_key_metadata( + model=target_model, key=key, bundle=bundle, ) - src = local.source - if isinstance(src, StarKey): - base = "*" - elif isinstance(src, ColumnSqlKey): - base = ".".join((*src.path, src.column_name)) - else: # ColumnKey - base = ".".join((*src.path, src.leaf)) - - formula = f"{base}:{local.agg}" - parts: List[str] = [_render_ref_formula(a) for a in local.args] - parts += [f"{k}={_render_ref_formula(v)}" for k, v in local.kwargs] - if parts: - formula += "(" + ", ".join(parts) + ")" - return formula - - -_REROOT_BIND_ERRORS = ( - UnknownReferenceError, - AmbiguousReferenceError, - IllegalScopeReferenceError, - ValueError, - NotImplementedError, -) + return DeclaredMeasure( + bound=BoundExpr(value_key=key), + declared_name=_canonical_name(key), + public_name=_canonical_name(key), + label=label, + type=dim_type, + format=fmt, + description=desc, + ) + + + + + + + + -def _maybe_reroot_cross_model_plan( + + + + + + + + +def _maybe_reroot_cross_model_plan( # NOSONAR(S3776) — one re-rooting decision over three parallel input kinds (dimensions, time dimensions, filters). Each loop re-anchors, tests target-reachability, and votes on `needs_reroot`; the vote is the shared state that makes them one pass rather than three functions. *, - plan, - query: SlayerQuery, + make_plan: Callable[["_FilterRoutes"], CrossModelAggregatePlan], + query: StrictQueryCarrier, agg_key: AggregateKey, bundle: ResolvedSourceBundle, host_model: SlayerModel, + host_slots: List[ValueSlot], + host_filters: List[HostFilterRouting], public_projection: List[str], - subplan_builder: Callable[[SlayerQuery, ResolvedSourceBundle], PlannedQuery], + subplan_builder: Callable[ + [StrictQueryCarrier, ResolvedSourceBundle], PlannedQuery, + ], + target_model_name: str, + target_path: Tuple[str, ...], ): - """Attach a re-rooted sub-``PlannedQuery`` to ``plan`` when the host - query carries dimensions reachable from the target by re-rooting through - the target's join graph. Returns ``plan`` unchanged when re-rooting is - unnecessary (only forward-path or genuinely unreachable dims).""" - target_model_name = plan.target_model + """Decide forward-vs-re-rooted, classify the host filters ONCE for whichever + shape won, and build the plan. + + Re-rooting applies when the host carries dimensions or filters reachable + from the target only by walking the TARGET's own join graph — off the + host→target forward path, which the forward CTE cannot evaluate. + + §5.4 — every input arrives already bound, on the carrier's + ``PreboundQuery``. Re-rooting re-anchors those keys structurally and hands + them straight back to the planner, so the sub-plan's slot identities are + the host's, transformed — never re-derived from a regenerated string. + + D6 — the decision precedes the classification. The two used to run in the + other order, with the reroot then BLANKING what the classifier had decided + against a coordinate system that no longer applied. + """ target_model = bundle.get_referenced_model(target_model_name) - if target_model is None: - return plan - target_path = tuple(getattr(agg_key.source, "path", ())) + host_prebound = query.prebound + + def _forward_only(): + return make_plan(_route_host_filters( + host_filters=host_filters, host_slots=host_slots, + target_path=target_path, host_model=host_model, + terminal_model=target_model or host_model, + )) + + if target_model is None or host_prebound is None: + return _forward_only() rerooted_bundle = bundle.model_copy(update={"source_model": target_model}) - target_scope = ModelScope(source_model=target_model) - - def _resolvable_ref(ref_str: str) -> Optional[ValueKey]: - try: - return bind_expr( - parse_expr(ref_str), - scope=target_scope, - bundle=rerooted_bundle, - ).value_key - except _REROOT_BIND_ERRORS: - return None + + def _reroot(key: ValueKey) -> ValueKey: + return _reroot_host_key( + key, target_path=target_path, host_model_name=host_model.name, + ) + + def _reaches(key: ValueKey) -> bool: + return _key_reaches_from( + key=key, model=target_model, bundle=rerooted_bundle, + ) def _is_forward(path: Tuple[str, ...]) -> bool: # On the host->target path (handled by the forward-path CTE already). return bool(path) and path == target_path[: len(path)] - n_dims = len(query.dimensions or []) - rerooted_dims: List[ColumnRef] = [] - rerooted_tds: List[TimeDimension] = [] + n_dims = host_prebound.n_dims + n_tds = host_prebound.n_time_dimensions + grain_declared: List[DeclaredMeasure] = [] grain_host_sids: List[str] = [] grain_rerooted_keys: List[ValueKey] = [] needs_reroot = False - for i, dim in enumerate(query.dimensions or []): + for i, dm in enumerate(host_prebound.declared_measures[: n_dims + n_tds]): host_sid = public_projection[i] if i < len(public_projection) else None - host_path = _host_ref_path(dim.model) - rr = _reroot_ref( - model_prefix=dim.model, name=dim.name, - host_model_name=host_model.name, target_model_name=target_model_name, + host_key = dm.bound.value_key + inner = ( + host_key.column if isinstance(host_key, TimeTruncKey) else host_key ) - rr_key = _resolvable_ref(rr) - if rr_key is None: + host_path = tuple(getattr(inner, "path", ()) or ()) + rr_key = _reroot(host_key) + if not _reaches(rr_key): continue # unreachable from target -> drop if not _is_forward(host_path): needs_reroot = True if host_sid is None: continue - rerooted_dims.append(ColumnRef(name=rr, label=dim.label)) - grain_host_sids.append(host_sid) - grain_rerooted_keys.append(rr_key) - - for j, td in enumerate(query.time_dimensions or []): - idx = n_dims + j - host_sid = public_projection[idx] if idx < len(public_projection) else None - host_path = _host_ref_path(td.dimension.model) - rr = _reroot_ref( - model_prefix=td.dimension.model, name=td.dimension.name, - host_model_name=host_model.name, target_model_name=target_model_name, - ) - rr_td = TimeDimension( - dimension=ColumnRef(name=rr), - granularity=td.granularity, - date_range=td.date_range, - label=td.label, - ) - try: - rr_key = bind_time_dimension( - rr_td, scope=target_scope, bundle=rerooted_bundle, - ).value_key - except _REROOT_BIND_ERRORS: - continue - if not _is_forward(host_path): - needs_reroot = True - if host_sid is None: - continue - rerooted_tds.append(rr_td) + grain_declared.append(_rerooted_dimension_measure( + key=rr_key, label=dm.label, target_model=target_model, + bundle=rerooted_bundle, + )) grain_host_sids.append(host_sid) grain_rerooted_keys.append(rr_key) - # Filters. A purely host-local filter (every ref on the host's own - # columns) filters host rows -- it stays at the host base; the join-back - # propagates the cardinality reduction, so adding it to the CTE would risk - # binding a bare name to a same-named TARGET column. A join-traversing - # filter affects the aggregate value and rides into the re-rooted CTE; one - # that reaches OFF the host->target forward path is exactly what the - # forward-path classifier drops, so it also triggers re-rooting (covers a - # cross-model agg filtered through the target's graph with no dimensions). - host_scope = ModelScope(source_model=host_model) - rerooted_filters: List[str] = [] - for f in (query.filters or []): - try: - host_bound = bind_filter( - parse_filter_expr(f), scope=host_scope, bundle=bundle, - ) - except _REROOT_BIND_ERRORS: - continue - host_paths = _filter_ref_paths(host_bound.value_key) - if all(p == () for p in host_paths): + # Filters vote structurally, before any classification. A filter that + # reaches OFF the host→target forward path is exactly what the forward CTE + # cannot evaluate, so wanting it is a reason to re-root — the same vote a + # non-forward dimension casts. Only a filter the RE-ROOTED CTE could + # actually evaluate votes: an ``order_tags`` predicate is unreachable + # either way and must not drag the plan into a shape that does not help it. + reachable_paths = _rerooted_reachable_paths( + host_filters=host_filters, + target_path=target_path, + target_model=target_model, + host_model_name=host_model.name, + bundle=rerooted_bundle, + ) + for hf in host_filters: + crossed = [p for p in hf.crossed_join_paths if p] + if not crossed: continue # host-local -> applied at the host base only - # The binder strips a same-model self-prefix (C14), so a - # ``.col`` ref binds locally against the target scope without - # any string surgery -- pass the filter through verbatim. - try: - bind_filter( - parse_filter_expr(f), scope=target_scope, bundle=rerooted_bundle, + # Through the shared predicate, not a second copy of the membership + # half: the rule lives in ONE place precisely so the two cannot drift. + if not all( + path_is_reachable( + path=p, target_path=target_path, + reachable_paths=reachable_paths, ) - except _REROOT_BIND_ERRORS: + for p in crossed + ): continue - rerooted_filters.append(f) - if any(p != target_path[: len(p)] for p in host_paths if p): + if any(not _is_forward(p) for p in crossed): needs_reroot = True + break + + if not needs_reroot: + return _forward_only() - if not needs_reroot or not ( - rerooted_dims or rerooted_tds or rerooted_filters - ): - return plan - - rerooted_query = SlayerQuery( - source_model=target_model_name, - measures=[ModelMeasure(formula=_local_agg_formula(agg_key))], - dimensions=rerooted_dims or None, - time_dimensions=rerooted_tds or None, - filters=rerooted_filters or None, + routes = _route_host_filters( + host_filters=host_filters, + host_slots=host_slots, + target_path=target_path, + host_model=host_model, + terminal_model=target_model, + reachable_paths=reachable_paths, + ) + if not (grain_declared or routes.applied): + # The re-rooted shape lost, so the plan that ships is the FORWARD one + # and its routing must be classified under the forward PREFIX rule. + # Returning ``make_plan(routes)`` here shipped a forward plan carrying + # re-rooted-coordinate routing: a path that is a strict prefix of + # ``target_path`` is reachable under the prefix rule but absent from + # ``reachable_paths``, so the filter the forward CTE can evaluate was + # dropped from it AND warned about. That is the same "classified + # against a coordinate system that no longer applies" failure D6 + # exists to remove — one branch below it, doing it again. + return _forward_only() + plan = make_plan(routes) + + # The CTE applies exactly what the routing says it applies — one decision, + # consumed, never re-derived. ``routing_by_id`` maps those ids back to the + # typed predicates so each rides in re-anchored rather than re-parsed. + # + # Date-range bounds (the ``[:n_date_range]`` prefix of ``bound_filters``) + # are not user filters and carry no routing id; they are re-anchored below + # alongside their time dimension. + routing_by_id = {hf.filter_id: hf for hf in host_filters} + rerooted_filters: List[BoundFilter] = [] + for fid in routes.applied: + hf = routing_by_id.get(fid) + if hf is None or hf.bound is None: + continue + rr_key = reroot_value_key(hf.bound.value_key, target_path=target_path) + rerooted_filters.append(BoundFilter( + value_key=rr_key, + phase=hf.bound.phase, + referenced_keys=tuple(walk_value_keys(rr_key)), + )) + + # Date-range bounds ride into the CTE alongside their re-rooted time + # dimension, so the sub-plan applies the same window the host does. + rerooted_bounds = [ + BoundFilter( + value_key=reroot_value_key(bf.value_key, target_path=target_path), + phase=bf.phase, + referenced_keys=tuple(walk_value_keys( + reroot_value_key(bf.value_key, target_path=target_path), + )), + ) + for bf in host_prebound.bound_filters[: host_prebound.n_date_range] + if _reaches(reroot_value_key(bf.value_key, target_path=target_path)) + ] + n_rerooted_tds = sum( + 1 for dm in grain_declared + if isinstance(dm.bound.value_key, TimeTruncKey) + ) + sub_prebound = _nested_prebound( + host_prebound=host_prebound, + aggregate_measure=_aggregate_declared_measure( + key=reroot_value_key(agg_key, target_path=target_path), + model=target_model, + public_alias=None, + ), + grain_measures=grain_declared, + inherited_filters=rerooted_filters, + date_range_filters=rerooted_bounds, + n_dims=len(grain_declared) - n_rerooted_tds, + n_time_dimensions=n_rerooted_tds, + main_time_key=next( + ( + dm.bound.value_key for dm in grain_declared + if isinstance(dm.bound.value_key, TimeTruncKey) + ), + None, + ), + ) + sub_plan = subplan_builder( + StrictQueryCarrier( + source_model=target_model_name, prebound=sub_prebound, + ), + rerooted_bundle, ) - sub_plan = subplan_builder(rerooted_query, rerooted_bundle) sub_row_by_key = {s.key: s.id for s in sub_plan.row_slots} grain_pairs: List[Tuple[str, str]] = [] @@ -1186,19 +1476,29 @@ def _is_forward(path: Tuple[str, ...]) -> bool: sub_agg_sid = s.id break if sub_agg_sid is None: - return plan - + # Same abandon, same reason: no sub-plan means the FORWARD plan ships, + # so it must carry forward-coordinate routing. + return _forward_only() + + # DEV-1747 B6/D6 — the AUDIT survives the reroot; the ROUTING does not, + # because they are different statements and the reroot changes only one. + # + # ``applied_filter_ids`` records what SOME scope evaluates, which is what + # makes a genuinely unreachable filter distinguishable from a reachable one + # instead of every case reporting ``applied=[] dropped=[]``. Those ids ride + # into ``sub_prebound`` above, so the sub-plan really does apply them. + # + # ``where_filter_ids`` / ``having_filter_ids`` say something narrower: this + # filter MOVED to the forward CTE, so the host base must not apply it. A + # re-rooted plan has no forward CTE — the sub-plan replaces it and carries + # its own filters — and the predicate is host-evaluable by construction + # (it was bound against the host). Leaving the ids there tells the host base + # to skip a filter nothing else applies at the host, so rows the user + # excluded come back with a NULL measure attached. return plan.model_copy(update={ "rerooted_plan": sub_plan, "rerooted_grain_pairs": grain_pairs, "rerooted_agg_slot_id": sub_agg_sid, - # The forward-path classifier marked these host filters - # DROP_UNREACHABLE, but the re-rooted CTE re-applies every - # target-reachable filter (and the host base keeps the rest for - # cardinality), so nothing is silently dropped -- clear the now-stale - # warnings and forward-only routing ids. - "dropped_filter_warnings": [], "where_filter_ids": [], "having_filter_ids": [], - "applied_filter_ids": [], }) diff --git a/slayer/engine/filter_reachability.py b/slayer/engine/filter_reachability.py new file mode 100644 index 00000000..585930ce --- /dev/null +++ b/slayer/engine/filter_reachability.py @@ -0,0 +1,446 @@ +"""DEV-1745 (W4 / mechanism contract 5.3) — per-filter structural reachability. + +Cross-model routing asks one question of every host filter: can this predicate +be evaluated inside a CTE rooted at ``target_path``? The answer is structural — +it depends on which join paths the filter's dependencies are anchored at — so +it is computed HERE, at plan time, and ``classify_host_filter`` reads it. + +What this replaces: a flat model-NAME membership test (``cm in target_path``) +used for derived columns. It got two shapes wrong. A model reachable on a +SIBLING branch counted as reachable, because its name appeared in the target +path even though no prefix of the path led to it. And a host-model derived +column whose ``Column.sql`` crossed INTO the target counted as host-local, +because only the declaring model's name was consulted, never the SQL. + +One rule for every key kind: a dependency is reachable iff its anchored join +path is a PREFIX of ``target_path`` (``path == target_path[:len(path)]``). +Reachability is an ALL-DEPENDENCIES predicate — any unreachable dependency +drops the filter. + +Storage (D9). The summary lives per-filter on ``PlannedQuery``. NOT on +``ColumnSqlKey``: that key is interned and ``_reroot_path_ref`` re-anchors it +with ``model_copy(update={"path": ...})``, which carries any extra field +through rerooting stale. NOT on ``ValueSlot``: ``filter_referenced_slot_ids`` +silently skips keys with no interned slot, and a derived column referenced only +inside a filter is exactly such a key — plus slots are copied wholesale into +nested plans, which would import the PARENT's coordinate system. + +Invariant: every summary is expressed in the coordinate system of the +``PlannedQuery`` that owns it, and is recomputed per plan, never copied. +""" + +from __future__ import annotations + +from decimal import Decimal +from typing import List, Optional, Tuple + +from sqlglot import exp + +from slayer.core.errors import SlayerError +from slayer.core.keys import ( + AggregateKey, + ArithmeticKey, + BetweenKey, + ColumnKey, + ColumnSqlKey, + InKey, + LiteralKey, + ScalarCallKey, + SqlExprKey, + StarKey, + TimeTruncKey, + TransformKey, +) +from slayer.engine.column_expansion import collect_root_scope_joined_paths +from slayer.engine.column_filter_paths import ( + _expand_derived_refs_any_dialect, + _parse_filter_sql_any_dialect, +) +from slayer.engine.planned import FilterReachability + +Path = Tuple[str, ...] + + +class UnhandledValueKindError(SlayerError, TypeError): + """A ValueKey kind the reachability scan does not know how to walk. + + Fails CLOSED, mirroring the total-visitor discipline the ValueKey renderer + uses: a new key kind that silently contributed no paths would read as + "crosses nothing", and a filter depending on it would propagate into a CTE + that cannot evaluate it. + """ + + def __init__(self, key: object) -> None: + self.key_type = type(key).__name__ + super().__init__( + f"UnhandledValueKindError: reachability scan has no rule for key " + f"kind {self.key_type!r}. Add an explicit arm — a silent empty " + f"result would route the filter as if it crossed nothing." + ) + + +def _prefixes(path: Path) -> List[Path]: + """Every non-empty prefix of ``path``. + + The FROM builder needs each intermediate join to reach the last one, and + reachability is judged per hop, so a two-hop reference contributes both + ``("a",)`` and ``("a", "b")``. + """ + return [tuple(path[: i + 1]) for i in range(len(path))] + + +def _expanded_derived_ast( + *, key: ColumnSqlKey, anchor_model, anchor_relation: str, bundle, + cache: "Optional[dict]" = None, +): + """The parsed, expanded AST of a derived column's ``Column.sql``. + + Expanded with the same anchoring convention ``ScopeFrame`` uses — at the + ``__``-path alias with ``is_root=False`` when the column lives on a joined + model — so the refs inside come out already prefixed by the key's own path + and an anchor-rooted scan resolves them without further adjustment. + + Memoised through the caller-supplied ``cache``. Both visitors ask for the + same key's expansion — ``_derived_sql_paths`` for the crossed set and + ``_derived_sql_touches_anchor`` for host-locality — and both run for every + filter on every plan, while ``_expand_derived_refs_any_dialect`` itself + re-parses per hop of a derived-of-derived chain. + + The cache is passed IN rather than held module-level on purpose. A global + keyed by ``id(bundle)`` would be unsound: CPython reuses ids once an object + is collected, so a fresh bundle could be handed a dead one's entry. A dict + owned by one plan-level call cannot outlive the bundle it was built for. + """ + if cache is None: + return _expanded_derived_ast_uncached( + key=key, anchor_model=anchor_model, + anchor_relation=anchor_relation, bundle=bundle, + ) + cache_key = (key.model, key.column_name, key.path, anchor_relation) + if cache_key not in cache: + cache[cache_key] = _expanded_derived_ast_uncached( + key=key, anchor_model=anchor_model, + anchor_relation=anchor_relation, bundle=bundle, + ) + return cache[cache_key] + + +def _expanded_derived_ast_uncached( + *, key: ColumnSqlKey, anchor_model, anchor_relation: str, bundle, +): + model = ( + anchor_model if key.model == getattr(anchor_model, "name", None) + else bundle.get_referenced_model(key.model) + ) + if model is None: + return None + col = next((c for c in model.columns if c.name == key.column_name), None) + if col is None or not col.sql: + return None + + alias_path = "__".join(key.path) if key.path else anchor_relation + expanded = _expand_derived_refs_any_dialect( + sql=col.sql, model=model, alias_path=alias_path, bundle=bundle, + ) + return _parse_filter_sql_any_dialect(expanded or col.sql) + + +def _derived_sql_paths( + *, key: ColumnSqlKey, anchor_model, anchor_relation: str, bundle, + cache: "Optional[dict]" = None, +) -> List[Path]: + """Join paths the expansion of a derived column's ``Column.sql`` crosses.""" + parsed = _expanded_derived_ast( + key=key, anchor_model=anchor_model, + anchor_relation=anchor_relation, bundle=bundle, cache=cache, + ) + if parsed is None: + return [] + return list(collect_root_scope_joined_paths( + parsed=parsed, + source_model=anchor_model, + source_relation=anchor_relation, + bundle=bundle, + )) + + +def _derived_sql_touches_anchor( + *, key: ColumnSqlKey, anchor_model, anchor_relation: str, bundle, + cache: "Optional[dict]" = None, +) -> bool: + """Whether a derived column's expansion references the ANCHOR relation. + + Tested directly rather than inferred from "it crossed nothing". A derived + column can do BOTH: ``amount * customers.rate`` crosses into ``customers`` + AND depends on the host-local ``amount``. Treating a non-empty crossed set + as proof of non-locality would propagate that filter into a + ``customers``-rooted CTE, where ``orders.amount`` is not bound. + + Expansion qualifies host-local refs to ``anchor_relation``, so those are + exactly the columns carrying that table (or, defensively, none at all). + """ + parsed = _expanded_derived_ast( + key=key, anchor_model=anchor_model, + anchor_relation=anchor_relation, bundle=bundle, cache=cache, + ) + if parsed is None: + # Nothing resolvable to inspect — a bare column name on the anchor. + return True + for col in parsed.find_all(exp.Column): + table = col.args.get("table") + if table is None or table.name == anchor_relation: + return True + return False + + +# Values a key tree can carry INLINE — plain data, not references, so they +# cannot cross a join. ``Decimal`` is load-bearing: ``AggregateKey.args`` / +# ``kwargs`` and ``ScalarCallKey.args`` normalise numeric literals to it, so a +# parametric aggregate like ``price:percentile(p=0.9)`` puts a Decimal in the +# tree. Omitting it made the fail-closed visitor reject a legitimate key. +_INLINE_SCALARS = (str, int, float, bool, Decimal) + +# Leaf kinds: they carry references but no child keys. +_LEAF_KINDS = (LiteralKey, StarKey, SqlExprKey, ColumnKey, ColumnSqlKey) + + +def _child_keys(node, *, descend_aggregates: bool = True) -> List: + """The child keys of a composite node, in a STABLE order. + + A variable-length SEQUENCE, not a fixed record — hence a list. + + One dispatch shared by both visitors, so a new key kind is handled — or + rejected — identically by each. Fails CLOSED on an unknown kind: a silent + empty result would read as "crosses nothing" and route a filter into a + scope that cannot evaluate it. + + ``partition_keys`` is a frozenset, whose iteration order varies between + runs; sorted here because the discovered paths drive JOIN emission order, + and non-deterministic SQL is its own bug. + + ``descend_aggregates=False`` stops at an aggregate: for host-locality, an + aggregate is routed by WHERE it is computed, not by its inputs. + """ + if isinstance(node, _LEAF_KINDS) or isinstance(node, _INLINE_SCALARS): + return [] + if isinstance(node, TimeTruncKey): + return [node.column] + if isinstance(node, AggregateKey): + if not descend_aggregates: + return [] + # ``column_filter_key`` is deliberately NOT a plain child: its + # ``referenced_join_paths`` are OWNER-relative and must be re-anchored + # by prefixing ``source.path``, which ``compute_key_join_paths`` does + # explicitly (DEV-1783). Descending it here would record them + # anchor-rooted and mis-route the filter. + return [ + node.source, + *node.args, + *(v for _name, v in node.kwargs), + ] + if isinstance(node, TransformKey): + return [ + node.input, + *sorted(node.partition_keys, key=repr), + node.time_key, + ] + if isinstance(node, ArithmeticKey): + return list(node.operands) + if isinstance(node, ScalarCallKey): + return list(node.args) + if isinstance(node, InKey): + return [node.column, *node.values] + if isinstance(node, BetweenKey): + return [node.column, node.low, node.high] + raise UnhandledValueKindError(node) + + +def _leaf_paths( + node, *, anchor_model, anchor_relation: str, bundle, + cache: "Optional[dict]" = None, +) -> List[Path]: + """Join paths a key contributes ITSELF (not via its children). + + Composites contribute nothing here — their dependencies arrive through + ``_child_keys`` — EXCEPT an ``AggregateKey``'s ``column_filter_key``, which + is not a plain child (its paths are OWNER-relative and re-anchored here). + Split out of the traversal so the walk stays a two-line "collect, then + descend". + """ + if isinstance(node, AggregateKey) and node.column_filter_key is not None: + # ``column_filter_key.referenced_join_paths`` are OWNER-relative + # (anchored at the aggregated column's owner, reached via + # ``source.path``). Re-anchor to the query root by prefixing + # ``source.path`` (DEV-1783); the reroot visitor leaves the owner in + # place for the same reason (keys.py: cfk copied unchanged). + source_path = tuple(getattr(node.source, "path", ()) or ()) + return [ + pre + for p in node.column_filter_key.referenced_join_paths + for pre in _prefixes(source_path + tuple(p)) + ] + if isinstance(node, ColumnSqlKey): + # Own anchored path first, then whatever its expansion reaches — the + # order the FROM builder consumes. Built as a NEW list rather than + # appending to ``_prefixes``' return, so this cannot corrupt that + # result if it ever becomes cached. + return [ + *_prefixes(node.path), + *_derived_sql_paths( + key=node, anchor_model=anchor_model, + anchor_relation=anchor_relation, bundle=bundle, cache=cache, + ), + ] + if isinstance(node, ColumnKey): + return _prefixes(node.path) + if isinstance(node, SqlExprKey): + return [ + pre + for p in node.referenced_join_paths + for pre in _prefixes(tuple(p)) + ] + return [] + + +def compute_key_join_paths( + *, key, anchor_model, anchor_relation: str, bundle, + cache: "Optional[dict]" = None, +) -> Tuple[Path, ...]: + """Every join path ``key``'s dependency tree crosses, anchored at + ``anchor_relation``. + + Recursive over the WHOLE key tree, not just the top node: a crossing + reference buried under arithmetic or inside an aggregate's kwargs is still + a dependency the destination scope has to satisfy. Returns an + insertion-ordered, de-duplicated tuple; empty means the key is evaluable + wherever the anchor is. + """ + seen: "dict[Path, None]" = {} + + def _add(path: Path) -> None: + if path: + seen.setdefault(tuple(path), None) + + def _walk(node) -> None: + if node is None: + return + for path in _leaf_paths( + node, anchor_model=anchor_model, + anchor_relation=anchor_relation, bundle=bundle, cache=cache, + ): + _add(path) + for child in _child_keys(node): + _walk(child) + + _walk(key) + return tuple(seen) + + +def key_has_host_local_ref( + *, key, anchor_model, anchor_relation: str, bundle, + cache: "Optional[dict]" = None, +) -> bool: + """Whether ``key`` depends on anything anchored AT the host root. + + A host-local dependency cannot be evaluated inside a CTE rooted elsewhere, + so a filter carrying one stays at the host even when its other dependencies + are reachable. Distinguished from "crosses nothing" deliberately: a derived + column declared on the host whose ``Column.sql`` reaches INTO the target + has an empty anchored path but is NOT host-local — inside the target's + scope its expansion resolves. + """ + + def _is_local(node) -> bool: + if isinstance(node, ColumnKey): + return not node.path + if isinstance(node, ColumnSqlKey): + return not node.path and _derived_sql_touches_anchor( + key=node, anchor_model=anchor_model, + anchor_relation=anchor_relation, bundle=bundle, cache=cache, + ) + return False + + def _walk(node) -> bool: + if node is None: + return False + if _is_local(node): + return True + return any( + _walk(child) + for child in _child_keys(node, descend_aggregates=False) + ) + + return _walk(key) + + +def path_is_reachable( + *, + path: Path, + target_path: Path, + reachable_paths: "Optional[frozenset]" = None, +) -> bool: + """The ONE reachability rule, for every key kind. + + A FORWARD-path CTE selects from the bare target and carries only the + host→target hops, so ``path`` is reachable iff it is a PREFIX of + ``target_path``. A path deeper than the target is not available (the + target's scope stops there); a sibling branch that happens to share a model + name is not available either, which is precisely what the old flat + membership test got wrong. + + A RE-ROOTED CTE (DEV-1747 D6) is planned against the TARGET as its own + root, so the target's WHOLE join graph is in scope and the prefix test no + longer describes it: a host-side sibling branch can be reachable from the + target by a different route entirely. That question is about the model + graph, not about string prefixes, so the caller — which holds the bundle — + walks it and passes the answer in as ``reachable_paths``. Membership then + replaces the prefix test outright. + """ + if reachable_paths is not None: + return tuple(path) in reachable_paths + return tuple(path) == tuple(target_path[: len(path)]) + + +def recompute_filter_reachability(planned_query, *, bundle) -> List: + """Recompute every filter's summary from scratch, anchored at + ``planned_query``'s OWN root. + + Used to verify the coordinate-system invariant: a plan's stored summary + must equal this. If a parent had copied its summary into a nested plan, the + stored value would still be anchored at the parent root and the two would + differ. + """ + anchor_model = planned_query.render_source_model or bundle.source_model + anchor_relation = planned_query.source_relation + cache: dict = {} + out: List = [] + for fp in planned_query.filters_by_phase: + if fp.expression is None: + continue + out.append(FilterReachability( + filter_id=fp.id, + crossed_join_paths=compute_key_join_paths( + key=fp.expression.value_key, + anchor_model=anchor_model, + anchor_relation=anchor_relation, + bundle=bundle, + cache=cache, + ), + has_host_local_ref=key_has_host_local_ref( + key=fp.expression.value_key, + anchor_model=anchor_model, + anchor_relation=anchor_relation, + bundle=bundle, + cache=cache, + ), + )) + return out + + +def filter_reachability_for(planned_query) -> List: + """The summary ``planned_query`` CARRIES — read, never recomputed. + + The accessor exists so consumers cannot accidentally recompute against a + different anchor and get a summary in the wrong coordinate system. + """ + return list(planned_query.filter_reachability) diff --git a/slayer/engine/isolation.py b/slayer/engine/isolation.py new file mode 100644 index 00000000..16db696b --- /dev/null +++ b/slayer/engine/isolation.py @@ -0,0 +1,197 @@ +"""One plan-time decision about how an aggregate is isolated (P-C). + +An aggregate that needs its own rows — because it crosses inputs, orders its +own rows (first/last), or carries its own frame (windowed) — is compiled as a +plan-shaped CTE rooted where its rows live and joined back on the query grain. +The host base contains only purely-local aggregates, so host cardinality never +changes. + +*Whether* an aggregate needs that, and *where* its CTE is rooted, used to be +decided by three predicates inlined in the planner's aggregate loop. They ran in +a fixed order and each knew about the others by omission — the windowed skip +existed because a windowed measure would otherwise trip the crossing-input +trigger, and the crossing-input trigger excluded path-bearing sources because +the target-rooted branch had already claimed them. Reading any one of them meant +reading all three. + +They are one function here, returning one value. Nothing about the decision +changed: the same inputs produce the same kind, which is what +``tests/test_dev1746_isolation_classifier.py`` pins. + +This is also where cardinality-aware inlining will land (DEV-1688). +:func:`may_inline_crossing_inputs` is the seam — hardcoded ``False``, so every +crossing aggregate isolates, which is today's behaviour. When a future change +lets a provably 1:1 crossing input stay inline, it changes there and every +isolation kind sees it at once, rather than in the four places that used to +decide independently. The render-time counterpart is ``ScopeFrame.may_inline``, +which guards the projection boundary for individual values; this one guards +whole aggregates. +""" + +from __future__ import annotations + +from enum import Enum +from typing import TYPE_CHECKING, List, Sequence, Set, Tuple + +from slayer.core.keys import AggregateKey +from slayer.engine.aggregate_input_paths import compute_aggregate_input_join_paths + +if TYPE_CHECKING: # pragma: no cover — typing only, keeps the import leaf clean + from slayer.engine.planned import ValueSlot + from slayer.engine.source_bundle import ResolvedSourceBundle + +__all__ = [ + "IsolationKind", + "classify_isolation", + "may_inline_crossing_inputs", +] + + +class IsolationKind(str, Enum): + """How one aggregate slot is compiled.""" + + #: Purely local — renders inline in the host base SELECT. + NONE = "none" + #: Its own ``_wm_`` CTE: a host-rooted range join carrying its own frame. + WINDOWED = "windowed" + #: Its own ``_cm_`` CTE rooted at the TARGET the aggregate's source names. + TARGET_ROOTED = "target_rooted" + #: Its own ``_cm_`` CTE rooted at the HOST, because a LOCAL aggregate's + #: inputs (a ``Column.filter``, the source's ``Column.sql``, an arg or a + #: kwarg) cross a join and so need their own rows. + HOST_ROOTED = "host_rooted" + #: Its own ``_rk_`` CTE rooted at the HOST: a ``first`` / ``last`` whose + #: rows live on the host but which needs its OWN ROW ORDERING. + RANKED_HOST = "ranked_host" + #: Its own ``_rk_`` CTE rooted at the TARGET the aggregate's source names. + RANKED_TARGET = "ranked_target" + + @property + def needs_own_cte(self) -> bool: + return self is not IsolationKind.NONE + + @property + def is_ranked(self) -> bool: + """Whether this kind compiles to a ranked (``first``/``last``) CTE.""" + return self in (IsolationKind.RANKED_HOST, IsolationKind.RANKED_TARGET) + + +def may_inline_crossing_inputs(crossed_paths: Sequence[Tuple[str, ...]]) -> bool: # NOSONAR(S1172) — crossed_paths is the documented DEV-1688 seam; the cardinality-aware decision reads it, hardcoded False until then. + """Whether an aggregate whose inputs cross ``crossed_paths`` may stay in the + host base instead of being isolated into its own CTE. + + Hardcoded ``False``: a crossing input is isolated, always. Inlining one is + only safe when the crossed join is provably 1:N-free for this aggregate, + which needs the cardinality metadata DEV-1688 is about. Until then this is + the single place that answer is given, so the future change has one site + rather than one per isolation kind. + """ + return False + + +def classify_isolation( + *, + slot: "ValueSlot", + windowed_slot_ids: Set[str], + bundle: "ResolvedSourceBundle", + disable_host_rooted_isolation: bool = False, +) -> IsolationKind: + """How ``slot`` is compiled. The single trigger decision. + + ``disable_host_rooted_isolation`` is set when planning a nested sub-plan: + that sub-plan contains the same crossing measure and would otherwise recurse + forever, and inside a CTE the crossing input renders inline legally, because + the CTE is the aggregate's own scope. + """ + if slot.id in windowed_slot_ids: + # A windowed measure always compiles to its own ``_wm_`` CTE, even when + # its ``Column.filter`` crosses a join — that crossing is carried inside + # the windowed CTE, not by a second isolation on top of it. + return IsolationKind.WINDOWED + + key = slot.key + if not isinstance(key, AggregateKey): + return IsolationKind.NONE + + if key.agg in ("first", "last"): + # Its own ROW ORDERING is one of the three things P-C says an aggregate + # can need its own rows for, so a first/last isolates whatever its + # inputs do — the trigger the pre-B9 classifier had no case for, which + # is why one first/last used to wrap the whole host base in a ranking + # instead. + # + # Deliberately ahead of the crossing-input branch: a crossing first/last + # already isolated, and routing it to the ranked CTE rather than the + # generic one is what lets the ranked scope apply its own predicate and + # rank in one place. Deliberately ahead of ``disable_host_rooted_isolation`` + # too: that guard exists because a CROSSING aggregate re-appears in its + # own nested sub-plan and would isolate forever. A ranked plan is + # RENDERED, never re-planned, and inside a sub-plan it still needs its + # own row ordering — suppressing it there would leave the sub-plan's + # first/last with no ranking at all. + if getattr(key.source, "path", ()): + return IsolationKind.RANKED_TARGET + return IsolationKind.RANKED_HOST + + if getattr(key.source, "path", ()): + # The source names another model: the aggregate's rows live there — + # UNLESS it is marked host-grain (DEV-1747 D2), which separates where a + # value is READ from where it is GROUPED. A joined ORDER BY wrap reads + # through the join but must be computed per HOST row-group, so its + # crossing IS the path and it belongs on the host-rooted route. Sending + # it to a target-rooted CTE would collapse it to a scalar CROSS JOIN: + # every group gets the same value and the sort silently does nothing. + if getattr(key, "grain", "target") == "host": + # Same recursion guard as the crossing-input branch below — inside + # the nested sub-plan the CTE is already this aggregate's own scope, + # so it renders inline (base-pull) rather than isolating forever. + if disable_host_rooted_isolation: + return IsolationKind.NONE + return IsolationKind.HOST_ROOTED + # Target-rooted isolation is deliberately NOT suppressed by the guard: + # inlining a joined SUM into the host base would multiply it by the + # join's fan-out. + return IsolationKind.TARGET_ROOTED + + if disable_host_rooted_isolation: + return IsolationKind.NONE + + crossed = _crossing_input_paths(key=key, bundle=bundle) + if not crossed: + return IsolationKind.NONE + if may_inline_crossing_inputs(crossed): + return IsolationKind.NONE + return IsolationKind.HOST_ROOTED + + +def _crossing_input_paths( + *, key: AggregateKey, bundle: "ResolvedSourceBundle", +) -> List[Tuple[str, ...]]: + """Join paths a LOCAL aggregate's inputs cross. + + A ``Column.filter`` carries its crossings as typed + ``referenced_join_paths`` from binder time; everything else (the source's + ``Column.sql``, positional args including an explicit first/last time arg, + and kwargs — column refs, user template fragments, and non-overridden + model-default aggregation params) is computed structurally. + + Both sources are UNIONED (DEV-1783): an aggregate whose column filter AND + whose source/args/kwargs each cross a different join must report both, or + the isolation decision under-counts and a fan-out-multiplying input inlines. + Order-stable (filter paths first) and de-duplicated. + """ + out: List[Tuple[str, ...]] = [] + if key.column_filter_key is not None: + for p in key.column_filter_key.referenced_join_paths: + if p not in out: + out.append(p) + source_model = getattr(bundle, "source_model", None) + for p in compute_aggregate_input_join_paths( + key=key, + anchor_model=source_model, + anchor_relation=source_model.name if source_model is not None else "", + bundle=bundle, + ): + if p not in out: + out.append(p) + return out diff --git a/slayer/engine/normalization.py b/slayer/engine/normalization.py index f31b1ea4..cd2328a1 100644 --- a/slayer/engine/normalization.py +++ b/slayer/engine/normalization.py @@ -469,6 +469,9 @@ def _apply_dot_path_in_sql( original=original, normalized="(ambiguous: shadowed by local alias or CTE — not rewritten)", location=location, + # Shadowed ref is left untouched — reported, not rewritten + # (DEV-1783). + rewritten=False, rule_doc_url="docs/agent_input_slack.md#dot-path-in-sql", ) emitted.append(payload) @@ -565,9 +568,51 @@ def normalize_query( # the model. Wiring is preserved so future activations need no # plumbing changes. + # Rule 4: MALFORMED_DATE_RANGE. + all_warnings.extend(_apply_malformed_date_range(query)) + return NormalizationResult(query=query, warnings=all_warnings) +def _apply_malformed_date_range( + query: SlayerQuery, +) -> List[NormalizationWarning]: + """Warn when a ``time_dimensions[i].date_range`` is present but is not the + two-element ``[start, end]`` the planner requires. + + The planner's silent ``continue`` on such a range is deliberate and stays + exactly as it is — this rule changes NO behaviour, it only stops the drop + from being invisible. The trigger is the planner's own drop condition + (``date_range is not None and len(date_range) != 2``), so the warning fires + if and only if the range is actually ignored: ``[]``, one element, or three + or more. An absent ``date_range`` is legitimately optional and never warns. + + Reports, but does not rewrite: there is no unambiguous canonical form to + rewrite a malformed range TO, and inventing one would change results. + """ + emitted: List[NormalizationWarning] = [] + for i, td in enumerate(query.time_dimensions or []): + date_range = getattr(td, "date_range", None) + if date_range is None or len(date_range) == 2: + continue + payload = NormalizationWarning( + rule_id="MALFORMED_DATE_RANGE", + original=f"time_dimensions[{i}].date_range={list(date_range)!r}", + normalized="(ignored — no date filter emitted)", + location=f"time_dimensions[{i}].date_range", + # Reports but does NOT rewrite (planner silently no-ops the range); + # the message must not claim a transform (DEV-1783). + rewritten=False, + # No rule_doc_url: docs/agent_input_slack.md does not exist, and a + # link to a missing page is worse than no link. + ) + emitted.append(payload) + _warnings_module.warn( + SlayerNormalizationWarning(payload), stacklevel=2, + ) + return emitted + + def _normalize_model_measures( model: SlayerModel, *, custom_agg_names: Optional[frozenset[str]], ) -> Tuple[SlayerModel, List[NormalizationWarning]]: diff --git a/slayer/engine/planned.py b/slayer/engine/planned.py index a55b48a6..735df069 100644 --- a/slayer/engine/planned.py +++ b/slayer/engine/planned.py @@ -23,7 +23,8 @@ from __future__ import annotations -from typing import List, Optional, Tuple +from enum import Enum +from typing import Dict, List, Literal, Optional, Tuple from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator @@ -61,10 +62,15 @@ "BoundExpr", "BoundFilterId", "CrossModelAggregatePlan", + "EmptyBaseGrainPlan", "FilterPhase", + "FilterReachability", "JoinRequirement", "OrderEntry", + "OrderScope", "PlannedQuery", + "RankedAggregatePlan", + "RankedGrainMember", "SlotId", "TransformLayer", "ValueSlot", @@ -182,8 +188,15 @@ class CrossModelAggregatePlan(BaseModel): table row: cross-model agg-ref on the same target). - ``target_model_filters`` — the target model's own ``SlayerModel.filters`` (always-applied WHERE). - ``applied_filter_ids`` is the audit union of where + having for - backward compatibility with the spec's external surface. + + ``applied_filter_ids`` is the AUDIT: which host filters some scope + evaluates. On the forward path that is exactly ``where ∪ having``. On a + RE-ROOTED plan the two diverge on purpose — ``rerooted_plan`` carries the + filters itself, and there is no forward CTE to route to, so where/having + are empty while the audit still records them. The distinction matters + because where/having are also an instruction to the host base to SKIP the + filter; a re-rooted CTE duplicates a host-evaluable predicate rather than + relocating it, so the host must keep applying it (DEV-1747 B6). ``hidden=True`` is used for order-only / filter-only refs whose aggregate value is materialised but not surfaced in the public @@ -318,6 +331,87 @@ class SrcFilterRewrite(BaseModel): expression: BoundExpr +# --------------------------------------------------------------------------- +# RankedAggregatePlan — DEV-1748 +# --------------------------------------------------------------------------- + + +class RankedGrainMember(BaseModel): + """One member of a ranked aggregate's grain, in BOTH coordinate systems. + + ``host_slot_id`` names the host row slot the CTE joins back to; + ``ranked_key`` is the same value anchored in the RANKED scope, which for a + target-rooted plan is a different expression against a different root. + + Carried as one list because the renderer derives two things from it that + must agree: the ``PARTITION BY`` the ranking runs over, and the join-back + predicate. Deriving them separately is how a partition and a grain drift + apart — and a partition COARSER than the grain silently returns more than + one row per group, which the LEFT JOIN then multiplies into the host. + """ + + model_config = ConfigDict(arbitrary_types_allowed=True) + + host_slot_id: SlotId + ranked_key: ValueKey + + +class RankedAggregatePlan(BaseModel): + """Plan for one ``first`` / ``last`` aggregate slot (DEV-1748, B9). + + A ranked aggregate needs its OWN ROW ORDERING, which is one of the three + things P-C says an aggregate can need its own rows for — so like a crossing + aggregate (``_cm_``) and a windowed one (``_wm_``) it compiles to a + plan-shaped CTE rooted where its rows live and joined back on the query + grain null-safely. The host base keeps only purely-local aggregates, so + adding a first/last cannot change host cardinality. + + It replaces a shape that did the opposite: ONE first/last anywhere wrapped + the entire host base in a ``ROW_NUMBER`` subquery, so every sibling + aggregate in the query was computed over the ranked row set, and several + rankings shared one scope — which is what the retired rn-suffix scheme + (``_last_rn_2``) and the filtered sentinel columns (``_last_rn_f0`` plus a + ``_match_f0`` flag consulted by alias) existed to disambiguate. One + aggregate per CTE removes the need for all of it. + + ``ranking_time_key`` is resolved at PLAN time (P-D) and anchored in the + ranked scope's coordinates; the renderer emits it and never re-derives the + precedence. A measure's ``Column.filter`` is not carried here — it lives on + the aggregate's own key, and the renderer applies it as a predicate inside + this CTE, which is what "filtered variants are plan data" means. + + Filter routing uses the same vocabulary, and the same audit-versus- + instruction distinction, as ``CrossModelAggregatePlan``: ``where`` and + ``having`` are instructions about where a filter is EVALUATED, while + ``applied_filter_ids`` only records that some scope evaluates it. + """ + + model_config = ConfigDict(arbitrary_types_allowed=True) + + aggregate_slot_id: SlotId + #: ``first`` ranks ascending, ``last`` descending. The direction is not a + #: separate field because it is not a separate decision. + agg: Literal["first", "last"] + #: The model the ranked rows come FROM — the host for a local aggregate, + #: the join target for a cross-model one. + root_model: str + datasource: str + #: The host-relative join path to ``root_model``; empty when host-rooted. + target_path: Tuple[str, ...] = () + join_chain: List[JoinRequirement] = Field(default_factory=list) + ranking_time_key: ValueKey + grain: List[RankedGrainMember] = Field(default_factory=list) + where_filter_ids: List[BoundFilterId] = Field(default_factory=list) + having_filter_ids: List[BoundFilterId] = Field(default_factory=list) + applied_filter_ids: List[BoundFilterId] = Field(default_factory=list) + target_model_filters: List[str] = Field(default_factory=list) + dropped_filter_warnings: List[UnreachableFilterDroppedWarning] = Field( + default_factory=list, + ) + public_alias: Optional[str] = None + hidden: bool = False + + # --------------------------------------------------------------------------- # TransformLayer # --------------------------------------------------------------------------- @@ -358,15 +452,14 @@ class FilterPhase(BaseModel): walks the typed value-key tree. * ``text`` is a Mode-A SQL fragment — used for ``SlayerModel.filters`` (always-applied WHERE). The renderer - qualifies bare-identifier column refs in ``text_columns`` with - the source-relation alias and emits the result verbatim - (matching legacy ``_build_where_and_having`` qualification). + enters it through the source scope's Mode-A door, which + qualifies bare-identifier column refs and discovers crossed + joins as a side effect of rendering. """ id: BoundFilterId phase: Phase text: Optional[str] = None - text_columns: Tuple[str, ...] = () expression: Optional[BoundExpr] = None @@ -375,20 +468,54 @@ class FilterPhase(BaseModel): # --------------------------------------------------------------------------- +class OrderScope(str, Enum): + """WHERE the ordered value lives — the one thing a renderer needs to know + to build a sort term (DEV-1747 §5.10). + + Every render site used to re-derive this, and they disagreed: one + dispatched on the slot KIND, one ran a five-way precedence chain over + alias maps, and one knew about neither. Naming the producing scope in the + plan is what lets a single resolver replace all of them (P-D). + """ + + #: Materialised in ``_base`` and projected publicly. + HOST_BASE = "host_base" + #: Materialised in ``_base`` but trimmed from the public projection — + #: an order-only aggregate or an unprojected host dimension. + HOST_BASE_HIDDEN = "host_base_hidden" + #: Lives in a cross-model / host-rooted isolated (``_cm_``) CTE. + CROSS_MODEL_CTE = "cross_model_cte" + #: Lives in a windowed (``_wm_``) CTE. + WINDOWED_CTE = "windowed_cte" + #: Lives in a ranked (``_rk_``) first/last CTE. + RANKED_CTE = "ranked_cte" + #: Produced by a step of the transform chain. + TRANSFORM_STEP = "transform_step" + #: A composite whose operands span scopes, so it can only be evaluated in + #: the outer combined SELECT — never inside ``_base``. + OUTER_COMPOSITE = "outer_composite" + + class OrderEntry(BaseModel): - """One entry in the ORDER BY of a planned query.""" + """One entry in the ORDER BY of a planned query. - slot_id: SlotId - direction: str # "asc" or "desc" + ``scope`` and ``phase`` are REQUIRED and have no default: a planner path + that forgets to classify must fail at construction rather than fall through + to the ``_base.``-qualified branch, which is how an order term silently + attached to the wrong scope. + """ - @field_validator("direction") - @classmethod - def _validate_direction(cls, v: str) -> str: - if v not in ("asc", "desc"): - raise ValueError( - f"OrderEntry.direction must be 'asc' or 'desc', got {v!r}" - ) - return v + slot_id: SlotId + direction: Literal["asc", "desc"] + scope: OrderScope + phase: Phase + #: Null-ordering policy. ``"default"`` is NULLs last, which is what SLayer + #: means by unstated on every dialect — a semantic layer whose NULLs sort + #: first on SQLite and last on Postgres answers the same question two ways. + #: The dialect strategy owns the SPELLING (P-H) — an explicit clause, an + #: emulation, or T-SQL's native pin where the emulation does not run — so + #: no render site emits a NULLS clause of its own. + nulls: Literal["default", "first", "last"] = "default" # --------------------------------------------------------------------------- @@ -396,6 +523,53 @@ def _validate_direction(cls, v: str) -> str: # --------------------------------------------------------------------------- +class FilterReachability(BaseModel): + """DEV-1745 (W4 / D9) — one filter's structural reachability summary. + + ``crossed_join_paths`` is every join path the filter's dependency tree is + anchored at, in THIS plan's coordinate system. ``has_host_local_ref`` marks + a dependency anchored at the plan's own root, which cannot be evaluated + inside a CTE rooted elsewhere. + + Carried per filter on the owning ``PlannedQuery`` rather than on + ``ColumnSqlKey`` (interned, and rerooting copies unknown fields through + stale) or ``ValueSlot`` (slot-less filter-only keys are silently skipped, + and slots are copied into nested plans). + """ + + model_config = ConfigDict(frozen=True) + + filter_id: BoundFilterId + crossed_join_paths: Tuple[Tuple[str, ...], ...] = () + has_host_local_ref: bool = False + + +class EmptyBaseGrainPlan(BaseModel): + """The host base has no columns of its own (DEV-1503, §5.12). + + Set when every projected value is an isolated aggregate — no host row + slots, no host-local aggregates — so ``_base`` has nothing to project and + becomes a one-row spine for the combined ``CROSS JOIN`` to hang off. Its + PRESENCE is the discriminator; there is no ``grain_slot_ids`` field because + in this shape the grain is empty by definition, which is precisely why the + join-back degenerates to a CROSS JOIN. + + ``host_filter_ids`` are the ROW-phase filters that stay host-local (not + routed into a ``_cm_*`` CTE or the outer WHERE). When any exist the spine is + emitted as ``SELECT 1 AS _placeholder FROM WHERE ... LIMIT 1``; + otherwise as a bare ``SELECT 1 AS _placeholder`` with no FROM at all. + + The ``LIMIT 1`` is load-bearing rather than an optimisation: the filtered + form keeps the host FROM so the WHERE can gate the result, but a host FROM + yields N rows and CROSS JOINing N rows to a one-row scalar aggregate would + repeat the answer N times. ``LIMIT 1`` collapses the spine to a single row + while an empty match still yields zero rows overall. The unfiltered form + drops the FROM entirely for the same reason. + """ + + host_filter_ids: List[BoundFilterId] = Field(default_factory=list) + + class PlannedQuery(BaseModel): """The fully typed plan for one query stage (P7). @@ -416,6 +590,13 @@ class PlannedQuery(BaseModel): aggregate_slots: List[ValueSlot] = Field(default_factory=list) cross_model_aggregate_plans: List[CrossModelAggregatePlan] = Field(default_factory=list) windowed_aggregate_plans: List["WindowedAggregatePlan"] = Field(default_factory=list) + # DEV-1748 (B9) — one entry per ``first`` / ``last`` aggregate slot. A + # sibling of ``windowed_aggregate_plans``: both name aggregates that need + # their own rows and therefore their own CTE, joined back on the query + # grain (P-C). A RE-ROOTED cross-model first/last is NOT here — it stays on + # ``CrossModelAggregatePlan``, whose nested sub-plan carries the ranked plan + # instead, in the sub-plan's own coordinate system. + ranked_aggregate_plans: List["RankedAggregatePlan"] = Field(default_factory=list) combined_expression_slots: List[ValueSlot] = Field(default_factory=list) transform_layers: List[TransformLayer] = Field(default_factory=list) filters_by_phase: List[FilterPhase] = Field(default_factory=list) @@ -454,6 +635,82 @@ class PlannedQuery(BaseModel): # equality-joined into ``_src`` (``_build_windowed_plans`` skips them), so # stripping a bound on one would leave that axis unconstrained. frame_bound_columns: List[ValueKey] = Field(default_factory=list) + # DEV-1745 (W3 / P-D) — ids of the AGGREGATE-phase filters that must be + # applied as a plain WHERE on the OUTER combined SELECT rather than as + # HAVING inside a ``_cm_*`` CTE (DEV-1503). + # + # A filtered-local ISOLATED aggregate lives in a CTE that LEFT JOINs back + # to ``_base``. Applying the comparison as HAVING inside that CTE drops CTE + # rows, but the LEFT JOIN then resurfaces the host row with a NULL + # aggregate — the wrong semantic. On the outer, non-aggregating SELECT the + # same comparison drops the row. + # + # Decided HERE because it is a routing decision, not an emission detail: + # the generator used to re-walk ``filters_by_phase`` at render time to + # rediscover it, which is policy chosen during emission. The generator now + # reads this field and never re-derives it, so clearing the field removes + # the outer WHERE. + outer_where_filter_ids: List[BoundFilterId] = Field(default_factory=list) + # DEV-1745 (W4 / D9) — per-filter structural reachability, in THIS plan's + # coordinate system. Recomputed for every plan (including the nested + # rerooted plan a cross-model CTE compiles), never copied down from a + # parent: the paths only mean anything relative to the root they were + # anchored at. Read via ``filter_reachability_for``. + filter_reachability: List[FilterReachability] = Field(default_factory=list) + # DEV-1746 (§5.12) — set when the host base has no columns of its own and + # is emitted as a one-row placeholder spine. Decided at plan time; the + # generator consumes it and never re-derives the shape. + empty_base_plan: Optional[EmptyBaseGrainPlan] = None + + @model_validator(mode="after") + def _projection_is_public_and_well_formed(self) -> "PlannedQuery": + """``projection`` is the ONE authoritative public column list (§5.2). + + Every renderer consumes it verbatim, which is what makes hidden-slot + trimming the absence of a step rather than a step. Two ways that could + break, both checked here rather than discovered as wrong SQL: + + * a HIDDEN slot appearing in it — hidden slots carry no public name, so + the renderer would have nothing to alias the column as; + * a slot appearing MORE times than it has declared names. A slot may + legitimately repeat: C13 lets one key be selected under several user + names, and the plan lists it once per name, each occurrence consuming + the next alias. One occurrence too many means a column emitted twice + under the same name. + """ + by_id = { + slot.id: slot + for slot in ( + list(self.row_slots) + + list(self.aggregate_slots) + + list(self.combined_expression_slots) + ) + } + counts: Dict[SlotId, int] = {} + for sid in self.projection: + counts[sid] = counts.get(sid, 0) + 1 + for sid, count in counts.items(): + slot = by_id.get(sid) + if slot is None: + # Slot tables can legitimately be partial in nested plans; the + # renderer resolves what it needs. Only slots we can SEE are + # checked, so this validator never rejects a plan for a reason + # it cannot substantiate. + continue + if slot.hidden: + raise ValueError( + f"hidden slot {sid!r} appears in the public projection; " + f"hidden slots carry no public name and must be absent", + ) + declared = len(slot.public_aliases) or (1 if slot.public_name else 0) + if declared and count > declared: + raise ValueError( + f"slot {sid!r} appears {count} times in the public " + f"projection but declares only {declared} public name(s) " + f"{list(slot.public_aliases) or [slot.public_name]!r} — " + f"the extra occurrence would emit a duplicate column", + ) + return self # ``CrossModelAggregatePlan.rerooted_plan`` is a forward reference to diff --git a/slayer/engine/planning.py b/slayer/engine/planning.py index 604996d9..dd56cf58 100644 --- a/slayer/engine/planning.py +++ b/slayer/engine/planning.py @@ -55,7 +55,7 @@ normalize_scalar, ) from slayer.core.formula import RANK_FAMILY_TRANSFORMS -from slayer.core.refs import agg_kwarg_canonical_str, canonical_agg_name +from slayer.sql.naming import canonical_aggregate_alias from slayer.engine.binding import BoundExpr, BoundFilter from slayer.engine.planned import SlotId, ValueSlot @@ -569,10 +569,8 @@ class DeclaredMeasure(BaseModel): DEV-1452 Stage B decisions #2 + #8: ``type``, ``format``, and ``description`` carry typed display + slot metadata from the source - ``ModelMeasure`` / ``Column`` so the public slot retains the same - contract the legacy enrichment pipeline produced. ``type`` mirrors - the legacy ``EnrichedMeasure.type`` (count → INT, avg → DOUBLE, - sum/min/max → source column type). + ``ModelMeasure`` / ``Column``. ``type`` follows the aggregation + (count → INT, avg → DOUBLE, sum/min/max → source column type). """ model_config = ConfigDict(arbitrary_types_allowed=True) @@ -799,31 +797,16 @@ def _canonical_name(key: ValueKey) -> str: # NOSONAR(S3776) — sequential isin # mirrors the cross-model parametric P10 exception. Without this, # two hidden parametric aggregates collide on a single base-CTE # alias when materialised. - if isinstance(key.source, StarKey): - measure_name = "*" - else: - leaf = getattr(key.source, "leaf", None) or getattr( - key.source, "column_name", None, - ) - if leaf is None: - return f"_agg_{key.agg}" - measure_name = leaf - agg_args = ( - [agg_kwarg_canonical_str(a) for a in key.args] - if key.args - else None - ) - agg_kwargs = ( - {k: agg_kwarg_canonical_str(v) for k, v in key.kwargs} - if key.kwargs - else None - ) - return canonical_agg_name( - measure_name=measure_name, - aggregation_name=key.agg, - agg_args=agg_args, - agg_kwargs=agg_kwargs, - ) + # The derivation lives in ``slayer.sql.naming`` (P-F). The + # ``declared_name`` profile is the bare canonical name, with the + # explicit ``_agg_`` placeholder for a source exposing neither a + # leaf nor a column name (deliberately NOT the star form, so such a + # slot stays distinguishable from a real ``*:count``). + alias = canonical_aggregate_alias(key, profile="declared_name") + # Only the ``stage_formula`` profile ever declines (returns None); + # ``declared_name`` always yields a name. + assert alias is not None + return alias if isinstance(key, TransformKey): return f"_{key.op}_inner" if isinstance(key, ArithmeticKey): diff --git a/slayer/engine/prebound.py b/slayer/engine/prebound.py new file mode 100644 index 00000000..7d813c21 --- /dev/null +++ b/slayer/engine/prebound.py @@ -0,0 +1,300 @@ +"""The pre-bound planner seam (DEV-1742 §5.4, P-E). + +``plan_query`` used to be the only door into binding: hand it a +``SlayerQuery`` and it parsed every measure / filter / order string, bound +each against a scope, and planned the result in one pass. Re-rooting needed a +nested plan built from keys it already held, so it SERIALIZED them back to +formula text and let ``plan_query`` re-derive the very identities it had just +thrown away. + +``PreboundQuery`` is that bind product made explicit. ``bind_query_inputs`` +produces it; ``plan_query(prebound=…)`` consumes it and skips binding +entirely. A caller holding typed keys re-roots them structurally and hands +them straight back — no text in the loop. + +``StrictQueryCarrier`` closes the second half. ``plan_query`` reads a handful +of query-level scalars that are not bind products (``source_model``, +``name``); a pre-bound caller that forgot one would silently inherit a Pydantic +default and plan the wrong thing. The carrier approves exactly those two and +raises on everything else, so a new post-bind ``query.*`` read fails loudly +instead of quietly. + +Also home to the key → slot-metadata lifts (``type`` / ``format`` / +``description``). They live here rather than in ``stage_planner`` because both +the planner and the re-rooting strategy need them, and the strategy cannot +import the planner (the recursion is injected as ``subplan_builder`` precisely +to avoid that cycle). +""" + +from __future__ import annotations + +from typing import List, Optional, Tuple + +from pydantic import BaseModel, ConfigDict, Field, model_validator + +from slayer.core.enums import ( + AggregationValueClass, + DataType, + classify_aggregation, +) +from slayer.core.format import NumberFormat +from slayer.core.keys import ( + AggregateKey, + ColumnKey, + ColumnSqlKey, + StarKey, + TimeTruncKey, + ValueKey, +) +from slayer.core.models import SlayerModel +from slayer.engine.binding import BoundFilter +from slayer.engine.planning import DeclaredMeasure, OrderSpec +from slayer.engine.response_meta import _infer_aggregated_format + + +__all__ = [ + "PreboundQuery", + "StrictQueryCarrier", + "aggregated_type", + "dimension_key_metadata", + "measure_key_format_description", + "measure_key_type", + "walk_key_path", +] + + +# --------------------------------------------------------------------------- +# The seam types +# --------------------------------------------------------------------------- + + +class PreboundQuery(BaseModel): + """The typed product of ``plan_query``'s bind block. + + Everything downstream of binding reads from here, so a caller that already + holds bound keys can plan without a parser. The ``n_*`` counts are the + dimension / time-dimension prefix lengths of ``declared_measures``, which + the projection and partition-key passes slice by. + """ + + model_config = ConfigDict(arbitrary_types_allowed=True) + + declared_measures: List[DeclaredMeasure] = Field(default_factory=list) + bound_filters: List[BoundFilter] = Field(default_factory=list) + # Parallel to ``bound_filters``: the originating user-filter text, or + # ``None`` for a synthesized date-range bound. Carried so the cross-model + # routing can report a filter the way the caller wrote it. + bound_filter_texts: List[Optional[str]] = Field(default_factory=list) + # Every count here is a LIST SLICE bound. A negative one is not a smaller + # slice, it is a slice from the other end — ``bound_filters[:-1]`` silently + # drops the LAST filter and keeps the rest, which is a wrong answer rather + # than an error. Constrained at the field so no construction site can pass + # one (Codex). + n_date_range: int = Field(default=0, ge=0) + order_specs: List[OrderSpec] = Field(default_factory=list) + main_time_key: Optional[TimeTruncKey] = None + n_dims: int = Field(default=0, ge=0) + n_time_dimensions: int = Field(default=0, ge=0) + limit: Optional[int] = None + offset: Optional[int] = None + distinct_dimension_values: bool = True + + @model_validator(mode="after") + def _filter_texts_are_parallel(self) -> "PreboundQuery": + """``bound_filter_texts`` is positionally parallel to + ``bound_filters``, and nothing downstream would notice if it were not: + the routing pass reads them with ``zip``, which silently TRUNCATES to + the shorter list. A short texts list would therefore drop host-filter + routings entirely rather than raise — the exact silent-narrowing class + this seam exists to make impossible, so the invariant is enforced here + rather than trusted at each construction site. + """ + if len(self.bound_filter_texts) != len(self.bound_filters): + raise ValueError( + f"PreboundQuery.bound_filter_texts must be parallel to " + f"bound_filters: got {len(self.bound_filter_texts)} texts for " + f"{len(self.bound_filters)} filters.", + ) + if self.n_date_range > len(self.bound_filters): + raise ValueError( + f"PreboundQuery.n_date_range={self.n_date_range} exceeds the " + f"{len(self.bound_filters)} bound filters it slices.", + ) + # ``n_dims`` and ``n_time_dimensions`` are the DIMENSION prefix lengths + # of ``declared_measures``; the rest of the list is measures. Python + # slicing past the end returns a SHORTER list rather than raising, so + # an over-count silently plans fewer dimensions than the caller + # declared — and the measures it does reach are misclassified as + # dimensions on the way (CodeRabbit). + grain = self.n_dims + self.n_time_dimensions + if grain > len(self.declared_measures): + raise ValueError( + f"PreboundQuery declares {self.n_dims} dimensions + " + f"{self.n_time_dimensions} time dimensions = {grain} grain " + f"members, but carries only {len(self.declared_measures)} " + f"declared measures for them to be a prefix of.", + ) + return self + + +class StrictQueryCarrier(BaseModel): + """The post-bind ``query.*`` surface the §5.4 seam approves. + + Anything not declared here raises rather than returning a default, so a + new post-bind read in ``plan_query`` cannot silently plan a re-rooted + sub-query against the wrong value. + """ + + model_config = ConfigDict(extra="forbid") + + source_model: Optional[str] = None + name: Optional[str] = None + prebound: Optional[PreboundQuery] = None + + def __getattr__(self, item: str): + if item.startswith("_"): + return super().__getattr__(item) + raise AttributeError( + f"{type(self).__name__} does not carry {item!r}. The pre-bound " + f"seam approves only " + f"{sorted(type(self).model_fields)}; add the field here (and " + f"populate it at every construction site) rather than letting the " + f"planner read a default." + ) + + +# --------------------------------------------------------------------------- +# Key -> slot metadata +# --------------------------------------------------------------------------- + +def aggregated_type( + *, + model: SlayerModel, + measure_name: Optional[str], + aggregation: str, +) -> Optional[DataType]: + """Type for an aggregated measure slot, via the shared + ``classify_aggregation`` (DEV-1788), so it cannot drift from + ``_infer_aggregated_format``: + + * ``COUNT`` (``*:count`` / count-family) → ``INT`` + * ``FLOAT_SOURCE_UNITS`` / ``FLOAT_PLAIN`` (avg-family, stat, parametric) → + ``DOUBLE`` + * ``PRESERVING`` (sum / min / max / first / last, and custom aggs) → inherit + source column type (``None`` if absent). + """ + cls = classify_aggregation(measure_name=measure_name, aggregation=aggregation) + if cls is AggregationValueClass.COUNT: + return DataType.INT + if cls in ( + AggregationValueClass.FLOAT_SOURCE_UNITS, + AggregationValueClass.FLOAT_PLAIN, + ): + return DataType.DOUBLE + # PRESERVING — inherit source column type. + if measure_name is None: + return None + col = model.get_column(measure_name) + if col is not None and col.type is not None: + return col.type + return None + + +def _local_aggregate_source_name(key: ValueKey) -> Optional[str]: + """The source column name of a LOCAL aggregate, or ``None``. + + ``None`` for anything that isn't a bare local aggregate — a non-aggregate + key, an unsupported source shape, or a cross-model source (whose metadata + is lifted by ``response_meta`` against the target model instead). + """ + if not isinstance(key, AggregateKey): + return None + src = key.source + if isinstance(src, StarKey): + return "*" + if not isinstance(src, (ColumnKey, ColumnSqlKey)): + return None + if getattr(src, "path", ()): + return None + return getattr(src, "leaf", None) or getattr(src, "column_name", None) + + +def measure_key_type( + *, model: SlayerModel, key: ValueKey, +) -> Optional[DataType]: + """``type`` for a measure slot, from its bound key alone.""" + name = _local_aggregate_source_name(key) + if name is None: + return None + return aggregated_type( + model=model, measure_name=name, aggregation=key.agg, + ) + + +def measure_key_format_description( + *, model: SlayerModel, key: ValueKey, +) -> Tuple[Optional[NumberFormat], Optional[str]]: + """``format`` / ``description`` for a measure slot, from its bound key. + + ``*:count`` has an inferred INTEGER format but no description — there is + no source column to document it. + """ + name = _local_aggregate_source_name(key) + if name is None: + return None, None + fmt = _infer_aggregated_format( + model=model, measure_name=name, aggregation=key.agg, + ) + if name == "*": + return fmt, None + col = model.get_column(name) + return fmt, (col.description if col is not None else None) + + +def walk_key_path( + *, model: SlayerModel, path: Tuple[str, ...], bundle, +) -> Optional[SlayerModel]: + """Walk ``path`` as join hops from ``model``; ``None`` on any miss. + + The structural counterpart to binding a dotted reference: it answers + "is this join path traversable?" without a parser and without raising, + which is what re-rooting needs to decide reachability. + """ + current = model + visited = {current.name} + for hop in path: + if not any(j.target_model == hop for j in current.joins): + return None + nxt = bundle.get_referenced_model(hop) + if nxt is None or nxt.name in visited: + return None + visited.add(nxt.name) + current = nxt + return current + + +def dimension_key_metadata( + *, model: SlayerModel, key: ValueKey, bundle, +) -> Tuple[Optional[DataType], Optional[NumberFormat], Optional[str]]: + """``(type, format, description)`` for a dimension slot, from its key. + + A LOCAL dimension carries the source column's full display contract; a + JOINED one carries only its type. That asymmetry is deliberate and + pre-existing: joined refs surface format / description through + ``response_meta``, which resolves them against the owning model. + """ + inner = key.column if isinstance(key, TimeTruncKey) else key + path = tuple(getattr(inner, "path", ()) or ()) + leaf = getattr(inner, "leaf", None) or getattr(inner, "column_name", None) + if leaf is None: + return None, None, None + if not path: + col = model.get_column(leaf) + if col is None: + return None, None, None + return col.type, col.format, col.description + terminal = walk_key_path(model=model, path=path, bundle=bundle) + if terminal is None: + return None, None, None + col = terminal.get_column(leaf) + return (col.type if col is not None else None), None, None diff --git a/slayer/engine/query_engine.py b/slayer/engine/query_engine.py index 9ebfbbee..4d57649d 100644 --- a/slayer/engine/query_engine.py +++ b/slayer/engine/query_engine.py @@ -7,6 +7,7 @@ import decimal import logging import re +import warnings as _warnings_module from collections.abc import Callable from typing import Any, Dict, List, Optional @@ -36,7 +37,11 @@ list_valued_variable_names, substitute_variables, ) -from slayer.core.warnings import NormalizationWarning +from slayer.core.warnings import ( + AnySlayerWarning, + DroppedFilterWarning, + NormalizationWarning, +) from slayer.core.recommend import ( CandidateCoverage, ItemPath, @@ -362,6 +367,81 @@ def _build_explain_sql(dialect: str, sql: str) -> str: return get_dialect(dialect).build_explain_sql(sql) +def _walk_cross_model_plans(planned): + """Every cross-model plan on ``planned``, including nested rerooted plans. + + A nested plan carries its OWN dropped-filter warnings for the same user + filter, which is why the old per-plan emission double-fired. + """ + for plan in getattr(planned, "cross_model_aggregate_plans", ()) or (): + yield plan + nested = getattr(plan, "rerooted_plan", None) + if nested is not None: + yield from _walk_cross_model_plans(nested) + + +def _stage_location(stages, index: int) -> str: + """Human-readable pointer to the stage a filter came from. + + Part of the dedup identity (D8), so it must distinguish two stages that + carry the SAME filter text — those are two distinct user filters. + """ + name = getattr(stages[index], "name", None) if index < len(stages) else None + return f"stage {name!r}.filters" if name else f"stages[{index}].filters" + + +def _collect_dropped_filter_warnings( + *, planned_list, stages, +) -> List[DroppedFilterWarning]: + """Dropped-filter payloads for the whole pipeline, one per user filter. + + Identity is ``(location, original filter text)`` — stated in the terms the + author sees, because the contract is user-facing. The same filter dropped + by several cross-model plans is ONE warning. + + Reasons for the same filter must AGREE. A disagreement means two plans + reached different conclusions about one filter, which is a planner + inconsistency; raising beats silently keeping whichever came first. + """ + by_identity: "dict[tuple[str, str], DroppedFilterWarning]" = {} + for index, planned in enumerate(planned_list): + location = _stage_location(stages, index) + for plan in _walk_cross_model_plans(planned): + for w in plan.dropped_filter_warnings or (): + identity = (location, w.filter_text) + existing = by_identity.get(identity) + if existing is None: + by_identity[identity] = DroppedFilterWarning( + filter_text=w.filter_text, + location=location, + reason=w.reason, + ) + elif existing.reason != w.reason: + raise ValueError( + f"Planner inconsistency: filter {w.filter_text!r} at " + f"{location} was dropped for two different reasons — " + f"{existing.reason!r} vs {w.reason!r}." + ) + return list(by_identity.values()) + + +def _emit_dropped_filter_warnings(response) -> None: + """Emit one ``UnreachableFilterDroppedWarning`` per dropped user filter. + + Called once, at the outermost boundary, AFTER the response is built. + """ + from slayer.core.errors import UnreachableFilterDroppedWarning + + for w in response.warnings or (): + if isinstance(w, DroppedFilterWarning): + _warnings_module.warn( + UnreachableFilterDroppedWarning( + filter_text=w.filter_text, reason=w.reason, + ), + stacklevel=3, + ) + + class SlayerResponse(BaseModel): """Response from a SLayer query.""" @@ -369,13 +449,14 @@ class SlayerResponse(BaseModel): columns: List[str] = PydanticField(default_factory=list) sql: Optional[str] = None attributes: ResponseAttributes = PydanticField(default_factory=ResponseAttributes) - # DEV-1450 stage 6 — slack-normalization warnings. - # Structured payload for each slack rewrite the normalization layer - # performed on the input (function-style aggs, misplaced measures, - # AST-resolvable dotted refs in raw SQL). Empty for queries that - # arrived in canonical form. Surfaced alongside the result so - # REST / MCP / CLI consumers can echo the rewrites back to authors. - warnings: List[NormalizationWarning] = PydanticField(default_factory=list) + # Advisories about the query itself, discriminated on ``kind``: + # ``normalization`` for a slack rewrite the normalization layer performed + # on the input (function-style aggs, misplaced measures, AST-resolvable + # dotted refs in raw SQL), and ``unreachable_filter_dropped`` for a user + # filter dropped from a cross-model CTE. Empty for a clean query. Surfaced + # alongside the result so REST / MCP / CLI consumers can echo them back to + # authors; switch on ``kind`` rather than on the presence of a field. + warnings: List[AnySlayerWarning] = PydanticField(default_factory=list) @model_validator(mode="after") def _populate_columns(self) -> "SlayerResponse": @@ -433,7 +514,7 @@ class _Prepared(BaseModel): """DB-free product of ``_prepare_pipeline`` (DEV-1715). Everything the execute / cache-hook / evict / refresh paths need after - resolve→enrich→plan→SQL-gen→policy but BEFORE any SQL-client construction. + resolve→bind→plan→SQL-gen→policy but BEFORE any SQL-client construction. ``sql`` is the FINAL, policy-rewritten SQL that is actually executed, so a cache key computed from it (``make_key(sql, ds_fingerprint)``) always matches the executed statement. ``resolved_data_source`` is @@ -744,7 +825,7 @@ async def execute( main_query, named_queries, prefer_data_source = await self._normalize_input( query, runtime_kwarg=runtime_kwarg, prefer_data_source=data_source ) - return await self._execute_pipeline( + response = await self._execute_pipeline( query=main_query, named_queries=named_queries, runtime_kwarg=runtime_kwarg, @@ -755,6 +836,14 @@ async def execute( original_input=query, original_data_source=data_source, ) + # DEV-1745 (W5): the ONE Python-warnings emission, at the outermost + # boundary and after the structured response exists. Ordering is + # load-bearing — under ``-W error`` this raises, and the contract is + # that the payload was fully built first. Emitting here rather than + # mid-render also makes it path-independent: dry_run, explain and a + # real execute all reach it. + _emit_dropped_filter_warnings(response) + return response async def _normalize_input( # NOSONAR S3776 — public dispatch over str/dict/list/SlayerQuery; splitting hides the input-shape contract self, @@ -866,7 +955,7 @@ async def _normalize_by_name( return main_query, named_queries, model.data_source or prefer_data_source - async def _prepare_pipeline( # NOSONAR S3776 — linear pipeline (resolve→enrich→generate→policy); breaking it up obscures the order of operations + async def _prepare_pipeline( # NOSONAR S3776 — linear pipeline (resolve→bind→generate→policy); breaking it up obscures the order of operations self, query: SlayerQuery, named_queries: Dict[str, SlayerQuery], @@ -876,7 +965,7 @@ async def _prepare_pipeline( # NOSONAR S3776 — linear pipeline (resolve→enr override_datasource: Optional[DatasourceConfig] = None, ) -> _Prepared: """DB-free-ish prepare portion shared by execute / evict / refresh - (DEV-1715): resolve→enrich→normalize→plan→SQL-gen→ClickHouse-preflight→ + (DEV-1715): resolve→bind→normalize→plan→SQL-gen→ClickHouse-preflight→ policy-rewrite→response-metadata. Produces the FINAL executed SQL and the datasource fingerprint but constructs **no** SQL client on the common (no-policy) path — so ``evict()`` recomputes a cache key without @@ -1027,6 +1116,14 @@ async def _prepare_pipeline( # NOSONAR S3776 — linear pipeline (resolve→enr planned_list = plan_stages(queries=stages, bundle=bundle) root_planned = planned_list[-1] + # DEV-1745 (W5): collect dropped-filter payloads across EVERY plan in + # the pipeline (including nested rerooted subplans) and dedup them per + # user filter. Collection happens here, with the plans in hand; the + # Python-warnings emission happens later, at the outermost boundary. + slack_warnings.extend(_collect_dropped_filter_warnings( + planned_list=planned_list, stages=stages, + )) + dialect = self._dialect_for_type(datasource.type) sql = generate_planned_stages( planned_list, bundle=bundle, dialect=dialect, @@ -1051,7 +1148,7 @@ async def _prepare_pipeline( # NOSONAR S3776 — linear pipeline (resolve→enr ) # Models whose live schema a query-time DBAPI error could be attributed - # to (the typed-plan equivalent of the legacy enriched-derived set). + # to, derived from the typed plan. touched = self._touched_models_for_plan( bundle=bundle, planned_list=planned_list, @@ -1312,7 +1409,7 @@ async def evict( *, data_source: Optional[str] = None, ) -> bool: - """Remove one cached entry, recomputing its key DB-free (resolve→enrich + """Remove one cached entry, recomputing its key DB-free (resolve→bind →SQL-gen→policy). Returns ``True`` if an entry was present. Never constructs a SQL client on the no-policy path.""" runtime_kwarg = variables or {} @@ -1669,8 +1766,6 @@ async def _maybe_raise_schema_drift( ``touched_models`` is computed from the resolved bundle / plan; the join graph is widened via ``_expand_join_graph`` before attribution. - (DEV-1485 Stage D removed the alternative ``enriched=`` mode, which - derived the same set from a legacy ``EnrichedQuery``.) Any error from ``validate_models`` itself is swallowed so the original exception is never masked. @@ -1746,7 +1841,7 @@ async def get_column_types( # NOSONAR(S3776) — linear probe pipeline: query-b ) -> Dict[str, str]: """Infer column types for a model's columns via a type-probe query. - Builds a real query through the engine's enrich+generate pipeline + Builds a real query through the engine's bind+generate pipeline so cross-model measures (with JOINs) are resolved correctly. Returns {column_name: type_category} where type_category is @@ -2443,8 +2538,7 @@ async def _expand_query_backed_model( # NOSONAR S3776 — linear render pipelin named_q = {q.name: q for q in stages[:-1] if q.name} # 2. Build resolved source bundle for the final stage. Stage B - # mirrors the legacy ``_query_as_model`` data_source behaviour: the - # bundle is built WITHOUT a DS hint, so the inner resolution falls + # builds the bundle WITHOUT a DS hint, so the inner resolution falls # back to the unique-match / priority-list resolver. This lets # ``get_column_types`` recover from a stale persisted # ``model.data_source`` (the cache populator may not have refreshed @@ -2654,7 +2748,7 @@ async def _resolve_model_inner( ) raise ValueError(f"Model '{model_name}' not found") - # If model has source_queries, re-enrich from stored queries. + # If model has source_queries, re-expand from stored queries. # Model-level defaults are folded into outer_vars by the helper # (precedence: runtime > stage > outer > model_defaults). return await self._expand_query_backed_model( @@ -2808,10 +2902,10 @@ async def save_model(self, model: SlayerModel) -> SlayerModel: query-backed models, persists as-is. DEV-1450 stage 6 — runs the slack-normalization layer over the - incoming model so persisted formulas land in canonical form. The - (Before DEV-1485 the legacy in-tree rewriters also fired during - enrichment for callers that loaded and re-executed persisted models; - normalization at save time is now the only such rewrite.) + incoming model so persisted formulas land in canonical form. + (Before DEV-1485 the legacy in-tree rewriters also fired when + callers loaded and re-executed persisted models; normalization at + save time is now the only such rewrite.) DEV-1500 — the FUNC_STYLE_AGG rewrite recognises custom aggregations defined on joined models via ``_reachable_aggs_for_save`` (best-effort diff --git a/slayer/engine/ranked_planner.py b/slayer/engine/ranked_planner.py new file mode 100644 index 00000000..6b7082f7 --- /dev/null +++ b/slayer/engine/ranked_planner.py @@ -0,0 +1,404 @@ +"""Plan-time construction of ``RankedAggregatePlan`` — the ``first`` / ``last`` +route (P-C / P-D). + +A ranked aggregate needs its own ROW ORDERING, so under P-C it compiles to a +plan-shaped CTE rooted where its rows live and joined back on the query grain. +Everything that decision needs is settled HERE — which column the ranking runs +over, what the grain is in the ranked scope's own coordinates, which filters the +CTE evaluates — so the renderer emits a plan rather than re-deriving one. + +The ranking column in particular used to be resolved at render time, twice: once +for the host base's ranked wrap and once, by a different precedence, inside the +cross-model CTE. They agreed by accident on the cases anyone had tried. One +resolver with an explicit per-scope precedence replaces both: + +* **host-rooted** — an explicit positional time arg, else the first + ``DATE``/``TIMESTAMP`` row dimension, else the first time dimension's RAW + column (never the truncated bucket: ranking within a month by the month ties + every row in it), else the model's ``default_time_dimension``. +* **target-rooted** — the same explicit arg, re-anchored in the target's + coordinates, else the TARGET model's ``default_time_dimension``. Host + dimensions are deliberately not candidates: the CTE ranks target rows, and a + host column is not in scope there. + +Both ends of the precedence raise rather than fall through, with the message the +scope's users already see. +""" + +from __future__ import annotations + +from typing import Any, List, Optional, Sequence, Tuple + +from slayer.core.enums import DataType +from slayer.core.keys import ( + AggregateKey, + ColumnKey, + ColumnSqlKey, + TimeTruncKey, + ValueKey, + column_path, + reroot_value_key, +) +from slayer.core.models import SlayerModel +from slayer.engine.planned import ( + RankedAggregatePlan, + RankedGrainMember, + SlotId, + ValueSlot, +) +from slayer.engine.source_bundle import ResolvedSourceBundle + +__all__ = [ + "RANKED_AGGREGATIONS", + "build_host_ranked_plan", + "build_target_ranked_plan", + "explicit_ranking_time_arg", + "resolve_ranking_time_key", +] + +#: The aggregations that rank. Named once so the classifier, the planner and +#: the renderer cannot drift on what "a ranked aggregate" is. +RANKED_AGGREGATIONS = ("first", "last") + +_TEMPORAL_TYPES = (DataType.DATE, DataType.TIMESTAMP) + + +def explicit_ranking_time_arg(key: AggregateKey) -> Optional[ValueKey]: + """The explicit positional ranking-time arg of a ``first`` / ``last``, or + ``None``. + + The FIRST positional arg iff it is a column ref; ``None`` for anything else + — first/last never takes a leading non-column positional, so a different + shape means the caller passed something this aggregation does not read. + """ + if key.agg not in RANKED_AGGREGATIONS: + return None + for arg in key.args: + return arg if isinstance(arg, (ColumnKey, ColumnSqlKey)) else None + return None + + +def _ranking_key_name(key: ValueKey) -> str: + """The user-facing name of a ranking-time key, for an error message.""" + if isinstance(key, ColumnKey): + return ".".join((*key.path, key.leaf)) + if isinstance(key, ColumnSqlKey): + return ".".join((*key.path, key.column_name)) + return type(key).__name__ + + +def _resolves_on(*, key: ValueKey, model: SlayerModel) -> bool: + """Whether ``key`` names something reachable FROM ``model``. + + A shallow check on purpose: a local leaf must be a column of the model, and + a path-bearing one must start at a model this one joins to. Walking the + whole chain is the renderer's job and it raises its own errors; this exists + so the common mistake — naming a HOST column as a TARGET-rooted ranking key + — is caught where the plan is made rather than at the database. + """ + if isinstance(key, ColumnKey): + leaf, path = key.leaf, key.path + elif isinstance(key, ColumnSqlKey): + leaf, path = key.column_name, key.path + else: + return True + if path: + return any(j.target_model == path[0] for j in (model.joins or [])) + return any(c.name == leaf for c in model.columns) + + +def _temporal_row_dimension_key( + *, + row_keys: Sequence[ValueKey], + source_model: SlayerModel, + bundle: ResolvedSourceBundle, +) -> Optional[ValueKey]: + """The first row dimension whose declared type is ``DATE``/``TIMESTAMP``.""" + for key in row_keys: + if not isinstance(key, ColumnKey): + continue + model: Optional[SlayerModel] = source_model + for hop in key.path: + model = bundle.get_referenced_model(hop) + if model is None: + break + if model is None: + continue + col = next((c for c in model.columns if c.name == key.leaf), None) + if col is not None and col.type in _TEMPORAL_TYPES: + return key + return None + + +def _time_dimension_raw_column( + *, row_keys: Sequence[ValueKey], +) -> Optional[ValueKey]: + """The first time dimension's RAW column, un-truncated.""" + for key in row_keys: + if isinstance(key, TimeTruncKey): + return key.column + return None + + +def resolve_ranking_time_key( + *, + key: AggregateKey, + root_model: SlayerModel, + bundle: ResolvedSourceBundle, + row_keys: Sequence[ValueKey] = (), + target_path: Tuple[str, ...] = (), +) -> ValueKey: + """The column a ranked aggregate's ``ROW_NUMBER`` orders by, in the RANKED + scope's coordinates. + + ``row_keys`` are the host's row-dimension keys in render order; they are + candidates only for a HOST-rooted plan (``target_path`` empty). A + target-rooted CTE ranks the target's rows, so a host dimension is not in + scope there and the precedence goes straight from the explicit arg to the + target's own default. + """ + arg = explicit_ranking_time_arg(key) + if arg is not None: + # The arg arrives in the HOST's coordinates; a target-rooted plan needs + # it in the target's, which is the same re-anchoring the aggregate + # source itself undergoes — done in lockstep so the ORDER BY and the + # value cannot end up rooted at different relations. + rerooted = reroot_value_key(arg, target_path=target_path) + if target_path and not _resolves_on(key=rerooted, model=root_model): + # A HOST column as the ranking key of a TARGET-rooted CTE. The rows + # being ranked are the target's, and a host column is not one of + # their attributes — the relationship runs the other way, usually + # one-to-many, so there is no single host value per target row to + # rank by. This used to emit ``ORDER BY .``, + # a reference to a column that does not exist, and fail at the + # database with no indication of which measure caused it. + raise ValueError( + f"first/last ranking column " + f"{_ranking_key_name(rerooted)!r} is not resolvable on model " + f"{root_model.name!r}, where a cross-model first/last ranks " + f"its rows. Name a column of {root_model.name!r} (or one it " + f"joins to), or drop the argument to rank by its " + f"default_time_dimension." + ) + return rerooted + + if not target_path: + temporal = _temporal_row_dimension_key( + row_keys=row_keys, source_model=root_model, bundle=bundle, + ) + if temporal is not None: + return temporal + raw = _time_dimension_raw_column(row_keys=row_keys) + if raw is not None: + return raw + if root_model.default_time_dimension: + return ColumnKey(path=(), leaf=root_model.default_time_dimension) + raise ValueError( + "first/last aggregation requires a ranking time column " + "(a time_dimension, a DATE/TIMESTAMP dimension, or the " + "model's default_time_dimension); none is resolvable for " + f"model {root_model.name!r}." + ) + + if root_model.default_time_dimension: + return ColumnKey(path=(), leaf=root_model.default_time_dimension) + raise ValueError( + f"first/last aggregation requires a ranking time column " + f"(an explicit positional time arg, or the target " + f"model's default_time_dimension); none is resolvable " + f"for cross-model aggregate on target " + f"{root_model.name!r}." + ) + + +def ordered_row_keys( + *, row_slots: Sequence[ValueSlot], public_projection: Sequence[SlotId], +) -> List[ValueKey]: + """Row-dimension keys in the order the base SELECT renders them. + + Publicly projected slots first, in projection order, then the rest. That is + the order the superseded render-time resolver walked, and the ranking-column + precedence is order-sensitive — two temporal dimensions rank by whichever + comes first. + """ + by_id = {s.id: s for s in row_slots} + seen: set = set() + ordered: List[ValueKey] = [] + for sid in public_projection: + slot = by_id.get(sid) + if slot is not None and sid not in seen: + seen.add(sid) + ordered.append(slot.key) + for slot in row_slots: + if slot.id not in seen: + seen.add(slot.id) + ordered.append(slot.key) + return ordered + + +def _host_grain(*, row_slots: Sequence[ValueSlot]) -> List[RankedGrainMember]: + """The host grain: every VISIBLE row slot, in one coordinate system. + + Hidden row slots are excluded for the same reason the windowed plan excludes + them — they are filter/order scaffolding the host base does not group by, so + partitioning the ranking over one would split groups the result never has. + """ + return [ + RankedGrainMember(host_slot_id=slot.id, ranked_key=slot.key) + for slot in row_slots + if not slot.hidden + ] + + +def _target_grain( + *, + row_slots: Sequence[ValueSlot], + shared_grain_slots: Sequence[SlotId], + public_projection: Sequence[SlotId], + target_path: Tuple[str, ...], +) -> List[RankedGrainMember]: + """The grain a TARGET-rooted ranked CTE shares with the host. + + Only host dimensions that lie ON the target's join path can be expressed in + the target's coordinates at all; the rest broadcast (the CTE is scalar and + CROSS JOINed), which is the forward cross-model path's existing semantics. + Re-rooting is what makes the rest reachable, and that route keeps its + ``CrossModelAggregatePlan``. + """ + by_id = {s.id: s for s in row_slots} + projected = set(public_projection) + members: List[RankedGrainMember] = [] + for sid in shared_grain_slots: + slot = by_id.get(sid) + if slot is None or slot.hidden or sid not in projected: + continue + path = _row_key_path(slot.key) + if not path or path != target_path: + # Empty: host-local, broadcast. Non-terminal / off-branch: not this + # target's grain. Both are the forward path's existing behaviour. + continue + members.append(RankedGrainMember( + host_slot_id=sid, + ranked_key=reroot_value_key(slot.key, target_path=target_path), + )) + return members + + +def _row_key_path(key: ValueKey) -> Tuple[str, ...]: + if isinstance(key, (ColumnKey, ColumnSqlKey)): + return key.path + if isinstance(key, TimeTruncKey): + return column_path(key.column) + return () + + +def build_host_ranked_plan( + *, + slot: ValueSlot, + row_slots: Sequence[ValueSlot], + public_projection: Sequence[SlotId], + source_model: SlayerModel, + bundle: ResolvedSourceBundle, + where_filter_ids: Sequence[str] = (), +) -> RankedAggregatePlan: + """One host-rooted ranked plan. + + ``where_filter_ids`` are the ROW-phase host filters (user filters AND the + model's own ``filters``) the CTE ALSO evaluates. They are duplicated rather + than relocated: a LEFT JOIN back propagates a value but never an exclusion, + so a row filter that only reached the CTE would silently become "blank out + their measure" instead of "exclude these rows" (the PR-4 B6 ruling, one + route over). + """ + key = slot.key + assert isinstance(key, AggregateKey) + return RankedAggregatePlan( + aggregate_slot_id=slot.id, + agg=key.agg, + root_model=source_model.name, + datasource=source_model.data_source, + target_path=(), + join_chain=[], + ranking_time_key=resolve_ranking_time_key( + key=key, + root_model=source_model, + bundle=bundle, + row_keys=ordered_row_keys( + row_slots=row_slots, public_projection=public_projection, + ), + ), + grain=_host_grain(row_slots=row_slots), + # The model's own ``filters`` are ROW-phase entries of + # ``filters_by_phase`` like any other, so they arrive here by id rather + # than as text — unlike the target-rooted case, where the TARGET's + # filters are not in the host's filter list at all. + where_filter_ids=list(where_filter_ids), + applied_filter_ids=list(where_filter_ids), + hidden=slot.hidden, + public_alias=None if slot.hidden else slot.public_name, + ) + + +def build_target_ranked_plan( + *, + slot: ValueSlot, + cross_model_plan: Any, + row_slots: Sequence[ValueSlot], + public_projection: Sequence[SlotId], + bundle: ResolvedSourceBundle, +) -> RankedAggregatePlan: + """One target-rooted ranked plan, re-shaped from the forward cross-model + plan the same strategy produced. + + The routing decisions — which host filters this CTE evaluates as WHERE, as + HAVING, which are unreachable — are the cross-model planner's decision table + and are taken verbatim. Only the two things that are genuinely about RANKING + are computed here: the ranking column in the target's coordinates, and the + grain in the same. + """ + key = slot.key + assert isinstance(key, AggregateKey) + target_path = tuple(getattr(key.source, "path", ())) + target_model = bundle.get_referenced_model(cross_model_plan.target_model) + if target_model is None: + raise ValueError( + f"Ranked cross-model target {cross_model_plan.target_model!r} is " + f"not in the resolved source bundle.", + ) + return RankedAggregatePlan( + aggregate_slot_id=slot.id, + agg=key.agg, + root_model=target_model.name, + datasource=cross_model_plan.datasource, + target_path=target_path, + join_chain=list(cross_model_plan.join_chain), + ranking_time_key=resolve_ranking_time_key( + key=key, + root_model=target_model, + bundle=bundle, + target_path=target_path, + ), + grain=_target_grain( + row_slots=row_slots, + shared_grain_slots=cross_model_plan.shared_grain_slots, + public_projection=public_projection, + target_path=target_path, + ), + where_filter_ids=list(cross_model_plan.where_filter_ids), + # A ranked CTE NEVER emits a HAVING, so it must not claim one. The + # strategy routes an aggregate-phase filter there for a plain + # cross-model CTE; on a ranked one the same predicate goes to the outer + # combined SELECT instead, because this CTE is LEFT JOINed back and + # dropping its row would resurrect the host row carrying NULL + # (DEV-1503). ``where``/``having`` are INSTRUCTIONS about where a filter + # is evaluated — carrying an id the renderer ignores would be an + # instruction nothing follows. The audit (``applied_filter_ids``) keeps + # the full record, and ``_assert_ranked_having_is_covered`` proves the + # predicate really is applied somewhere. + having_filter_ids=[], + applied_filter_ids=list(cross_model_plan.applied_filter_ids), + target_model_filters=list(cross_model_plan.target_model_filters), + dropped_filter_warnings=list(cross_model_plan.dropped_filter_warnings), + hidden=cross_model_plan.hidden, + public_alias=cross_model_plan.public_alias, + ) diff --git a/slayer/engine/response_meta.py b/slayer/engine/response_meta.py index 7a5ed70a..6eee9eee 100644 --- a/slayer/engine/response_meta.py +++ b/slayer/engine/response_meta.py @@ -1,9 +1,7 @@ """DEV-1450 stage 7b.15d — response metadata from the typed plan. -The legacy engine derived ``SlayerResponse.attributes`` and -``expected_columns`` from an ``EnrichedQuery``. The typed pipeline has no -``EnrichedQuery``; this module rebuilds the same two artefacts from the root -``PlannedQuery`` plus the final rendered SQL. +This module builds ``SlayerResponse.attributes`` and ``expected_columns`` +from the root ``PlannedQuery`` plus the final rendered SQL. * ``expected_columns`` comes from the final SQL's ``named_selects`` — the literal result-key columns the rows come back keyed by. Deriving them from @@ -28,6 +26,7 @@ import sqlglot from pydantic import BaseModel, Field as PydanticField +from slayer.core.enums import AggregationValueClass, classify_aggregation from slayer.core.format import NumberFormat, NumberFormatType from slayer.core.keys import ( AggregateKey, @@ -74,28 +73,28 @@ def _infer_aggregated_format( measure_name: str, aggregation: str, ) -> Optional[NumberFormat]: - """Infer NumberFormat for an aggregated measure based on aggregation type and source measure format. - - Rules: - - count, count_distinct, count_distinct_approx: always INTEGER - - avg, weighted_avg, median: always FLOAT - - sum, min, max, first, last: inherit from source measure - - *:count (measure_name="*"): INTEGER + """Infer the display NumberFormat for an aggregated measure via the shared + ``classify_aggregation`` (DEV-1788), so it cannot drift from + ``aggregated_type``: + + - COUNT (``*:count`` / count-family): INTEGER + - FLOAT_PLAIN (corr / var / covar): plain FLOAT + - FLOAT_SOURCE_UNITS (avg-family / percentile / stddev): inherit source + format, else FLOAT (the result is fractional even absent source units) + - PRESERVING (sum / min / max / first / last, and custom aggs): inherit + source format, else None """ - if measure_name == "*": - return NumberFormat(type=NumberFormatType.INTEGER) - - if aggregation in ("count", "count_distinct", "count_distinct_approx"): + cls = classify_aggregation(measure_name=measure_name, aggregation=aggregation) + if cls is AggregationValueClass.COUNT: return NumberFormat(type=NumberFormatType.INTEGER) - - if aggregation in ("avg", "weighted_avg", "median"): + if cls is AggregationValueClass.FLOAT_PLAIN: return NumberFormat(type=NumberFormatType.FLOAT) - # sum, min, max, first, last: inherit from source column's format source_col = model.get_column(measure_name) if source_col and source_col.format: return source_col.format - + if cls is AggregationValueClass.FLOAT_SOURCE_UNITS: + return NumberFormat(type=NumberFormatType.FLOAT) return None @@ -203,9 +202,9 @@ def _measure_format( """Number format for a measure slot. Aggregate slots inherit via ``_infer_aggregated_format`` (INTEGER for - count(-distinct) / star, FLOAT for avg-family, source-column format for - sum/min/max). Transform / arithmetic / scalar-call slots default to FLOAT, - matching the legacy ``EnrichedQuery`` expression/transform handling. + count(-distinct) / star, plain FLOAT for corr / var / covar, source format + for the avg-family / percentile / stddev and for sum / min / max). + Transform / arithmetic / scalar-call slots default to FLOAT. """ key = slot.key if isinstance(key, AggregateKey): diff --git a/slayer/engine/schema_drift.py b/slayer/engine/schema_drift.py index 0dff9a04..a011b18c 100644 --- a/slayer/engine/schema_drift.py +++ b/slayer/engine/schema_drift.py @@ -37,6 +37,7 @@ SlayerModel, ) from slayer.core.query import SlayerQuery +from slayer.core.refs import IDENTIFIER_RE from slayer.sql.sql_predicate import parse_sql_predicate from slayer.engine.introspect_utils import _safe_get_columns from slayer.engine.ingestion import ( @@ -233,13 +234,10 @@ def _type_buckets_conflict(*, persisted: DataType, live: DataType) -> bool: def _is_bare_identifier(s: str | None) -> bool: - """``s`` is a bare SQL identifier (alphanumeric + underscore, no leading digit).""" + """``s`` is a bare SQL identifier per the canonical ``IDENTIFIER_RE``.""" if not s: return False - s = s.strip() - if not s or s[0].isdigit(): - return False - return all(c.isalnum() or c == "_" for c in s) + return IDENTIFIER_RE.match(s.strip()) is not None def _column_is_base(col_sql: str | None) -> bool: diff --git a/slayer/engine/stage_planner.py b/slayer/engine/stage_planner.py index 10228061..c102762b 100644 --- a/slayer/engine/stage_planner.py +++ b/slayer/engine/stage_planner.py @@ -25,15 +25,15 @@ from __future__ import annotations -from typing import Dict, FrozenSet, List, Optional, Tuple, Union +from typing import AbstractSet, Dict, FrozenSet, List, Optional, Set, Tuple, Union from slayer.core.enums import DataType +from slayer.core.formula import TIME_TRANSFORMS from slayer.core.format import NumberFormat from slayer.core.errors import ( AmbiguousReferenceError, DistinctDimensionValuesError, UnknownReferenceError, - UnresolvableOrderColumnError, ) from slayer.core.keys import ( AggregateKey, @@ -45,7 +45,6 @@ LiteralKey, Phase, ScalarCallKey, - StarKey, TimeTruncKey, TransformKey, ValueKey, @@ -58,13 +57,12 @@ SlayerQuery, TimeDimension, ) -from slayer.core.refs import agg_kwarg_canonical_str, canonical_agg_name +from slayer.core.refs import canonical_agg_name +from slayer.sql.naming import canonical_aggregate_alias from slayer.core.time_bounds import strip_frame_bounds from slayer.core.window_duration import parse_window_duration from slayer.core.scope import ModelScope, StageColumn, StageSchema -from slayer.engine.aggregate_input_paths import ( - compute_aggregate_input_join_paths, -) +from slayer.engine.isolation import IsolationKind, classify_isolation from slayer.engine.binding import ( BoundExpr as BinderBoundExpr, BoundFilter, @@ -79,17 +77,31 @@ IsolatedCteCrossModelPlanner, ) from slayer.engine.measure_expansion import expand_model_measures -from slayer.engine.response_meta import _infer_aggregated_format +from slayer.engine.filter_reachability import ( + compute_key_join_paths, + key_has_host_local_ref, +) from slayer.engine.planned import ( BoundExpr as PlannedBoundExpr, + BoundFilterId, + CrossModelAggregatePlan, + EmptyBaseGrainPlan, FilterPhase, + FilterReachability, OrderEntry, + OrderScope, PlannedQuery, + RankedAggregatePlan, + SlotId, SrcFilterRewrite, TransformLayer, ValueSlot, WindowedAggregatePlan, ) +from slayer.engine.ranked_planner import ( + build_host_ranked_plan, + build_target_ranked_plan, +) from slayer.engine.planning import ( DeclaredMeasure, OrderSpec, @@ -100,6 +112,12 @@ lower_sugar_transforms, rewrite_rank_partition_keys, ) +from slayer.engine.prebound import ( + PreboundQuery, + StrictQueryCarrier, + measure_key_format_description, + measure_key_type, +) from slayer.engine.source_bundle import ( ResolvedSourceBundle, _apply_extension_overlay, @@ -114,23 +132,19 @@ from slayer.sql.sql_predicate import parse_sql_predicate -__all__ = ["plan_query", "plan_stages"] +__all__ = [ + "PreboundQuery", + "StrictQueryCarrier", + "bind_query_inputs", + "plan_query", + "plan_stages", +] -# Stage 7b.10 — TIME_NEEDING transform ops that require a resolvable -# time dimension to render their OVER ``ORDER BY``. Mirrors the legacy -# ``TIME_TRANSFORMS`` set at ``slayer/core/formula.py:33``. -_TIME_NEEDING_TRANSFORM_OPS = frozenset({ - "cumsum", - "change", - "change_pct", - "time_shift", - "first", - "last", - "lag", - "lead", - "consecutive_periods", -}) +# Stage 7b.10 — transform ops that require a resolvable time dimension to render +# their OVER ``ORDER BY``: the canonical ``TIME_TRANSFORMS`` set (single-sourced +# from ``slayer/core/formula.py``). +_TIME_NEEDING_TRANSFORM_OPS = TIME_TRANSFORMS def _attach_time_keys( @@ -635,52 +649,48 @@ def _reject_measure_refs_in_order( ) -def plan_query( # NOSONAR(S3776) — planner entry-point dispatcher. The DEV-1503 addition is a small trigger-predicate branch + a kwarg pass-through; the function's pre-existing complexity is owned by the multi-stage scope / bundle / projection / filter-routing wiring it orchestrates and is tracked as a separate refactor. +def _resolve_scope( + *, + query: SlayerQuery, + bundle: ResolvedSourceBundle, + stage_schemas: Optional[Dict[str, StageSchema]], +) -> Union[ModelScope, StageSchema]: + """Default binding scope: an upstream ``StageSchema`` when the query's + ``source_model`` names a sibling stage, else a ``ModelScope`` over the + bundle's host model.""" + source = query.source_model + if isinstance(source, str) and source in (stage_schemas or {}): + return (stage_schemas or {})[source] + return ModelScope(source_model=bundle.source_model) + + +def bind_query_inputs( # NOSONAR(S3776) — one cohesive bind pass. The stages are strictly sequential and share the growing `declared_measures` / `bound_filters` / `order_specs` triple: parse+bind, time-key attachment, sugar lowering, rank-partition validation. Splitting them would thread the same three lists through four signatures without removing a branch. *, query: SlayerQuery, bundle: ResolvedSourceBundle, scope: Optional[Union[ModelScope, StageSchema]] = None, - cross_model_planner: Optional[CrossModelPlanner] = None, stage_schemas: Optional[Dict[str, StageSchema]] = None, - disable_host_rooted_isolation: bool = False, -) -> PlannedQuery: - """Compile one ``SlayerQuery`` into a typed ``PlannedQuery``. - - ``scope`` defaults to a ``ModelScope`` over ``bundle.source_model``; - pass an explicit ``StageSchema`` to bind against an upstream stage. - ``stage_schemas`` is a name → StageSchema map used by - ``plan_stages`` to wire multi-stage references. - - ``disable_host_rooted_isolation`` (DEV-1503; renamed and widened by - DEV-1709) suppresses the HOST-ROOTED half of the Law-3 trigger — the - isolation of a LOCAL aggregate whose ``Column.filter`` or any other - input (source ``Column.sql``, positional args, kwargs) crosses a join. - It never affects target-rooted isolation (``source.path`` non-empty). - Threaded through ``subplan_builder`` whenever a cross-model strategy - recurses for a host-rooted (or target-rooted) nested sub-plan, so the - sub-plan's same crossing measure is rendered inline (not infinitely - re-isolated) and so re-rooting of a genuine cross-model aggregate - doesn't redundantly isolate the target's own filter joins. +) -> PreboundQuery: + """Parse and bind every text surface of a ``SlayerQuery`` (DEV-1742 §5.4). + + This is the ONLY door into the parser. ``plan_query`` calls it when no + ``prebound`` is supplied; a caller that already holds typed keys builds a + ``PreboundQuery`` structurally and skips it entirely, which is what makes + re-rooting free of formula-text round-trips (P-E). + + The returned keys are fully normalized — time keys attached, ``change`` / + ``change_pct`` sugar lowered, rank ``partition_by`` columns validated and + rewritten to their time buckets — so planning sees exactly one key shape. + + Model filters are deliberately NOT included: they are a property of the + SCOPE, not the query, and ``plan_query`` lifts them from ``scope`` + directly so a pre-bound caller inherits its own model's filters rather + than the host's. """ - stage_schemas = stage_schemas or {} - cross_model_planner = ( - cross_model_planner or IsolatedCteCrossModelPlanner() - ) - if scope is None: - source = query.source_model - if isinstance(source, str) and source in stage_schemas: - scope = stage_schemas[source] - else: - scope = ModelScope(source_model=bundle.source_model) - - # The generator must render this stage's FROM / joins against the SAME - # model the binder used. For a ModelScope that's the (possibly overlaid / - # synthetic) host; for a StageSchema chain stage it's None (the generator - # builds a synthetic model from the upstream schema). - render_source_model = ( - scope.source_model if isinstance(scope, ModelScope) else None - ) + scope = _resolve_scope( + query=query, bundle=bundle, stage_schemas=stage_schemas, + ) # DEV-1543: raw-rows mode rejects measure references in filters / order. # Runs BEFORE binding so the targeted, actionable error wins over the @@ -736,10 +746,9 @@ def plan_query( # NOSONAR(S3776) — planner entry-point dispatcher. The DEV-15 # DEV-1450 stage 7b.9 — filter list construction in legacy WHERE # order: date_range filters first, then SlayerModel.filters - # (Mode-A SQL), then user query filters (Mode-B DSL). The legacy - # generator emits date_range BEFORE iterating ``enriched.filters`` - # (slayer/sql/generator.py:2527 vs :2540), and ``enriched.filters`` - # itself is model filters then query filters (enrichment.py:1192). + # (Mode-A SQL), then user query filters (Mode-B DSL). date_range is + # emitted before the model/query filters, and model filters precede + # query filters. # # ``bound_filters`` carries the typed-BoundFilter entries (date_range # + query filters) for the cross-model routing and projection @@ -756,7 +765,6 @@ def plan_query( # NOSONAR(S3776) — planner entry-point dispatcher. The DEV-15 # (which has not been deduped, unlike bound_filters) — CR PR #153 # thread r3350000254. bound_filter_texts: List[Optional[str]] = [] - text_filter_entries: List[FilterPhase] = [] # 1. date_range filters (one per TD with a 2-element date_range) for td in (query.time_dimensions or []): @@ -769,12 +777,11 @@ def plan_query( # NOSONAR(S3776) — planner entry-point dispatcher. The DEV-15 bound_filter_texts.append(None) n_date_range = len(bound_filters) - # 2. SlayerModel.filters — Mode-A SQL, always-applied WHERE. - if isinstance(scope, ModelScope) and scope.source_model is not None: - for j, mf in enumerate(scope.source_model.filters or []): - text_filter_entries.append(_validate_model_filter( - mf=mf, idx=j, model=scope.source_model, - )) + # 2. SlayerModel.filters — Mode-A SQL, always-applied WHERE. Lifted from + # the SCOPE in ``plan_query``, not here (§5.4): they belong to the model + # being planned against, so a re-rooted sub-plan must pick up the + # TARGET's model filters rather than inherit the host's through the + # carrier. # 3. user query filters (Mode-B DSL). # @@ -891,9 +898,8 @@ def plan_query( # NOSONAR(S3776) — planner entry-point dispatcher. The DEV-15 # binder-output left ``time_key`` as ``None``. Closes the 7b.4 # carry-over gap: ``_bind_transform`` does not have query / scope # context to resolve the TD, so the planner does it here after all - # binding completes. Validation mirrors legacy - # ``enrichment.py:564-569`` -- any time-needing transform with no - # resolvable TD raises with the legacy phrase. + # binding completes. Any time-needing transform with no resolvable + # time dimension raises. active_td_key: Optional[TimeTruncKey] = None if isinstance(scope, ModelScope) and scope.source_model is not None: active_td = _resolve_main_time_dimension( @@ -1093,6 +1099,101 @@ def _rw(vk: ValueKey) -> ValueKey: for spec in order_specs ] + return PreboundQuery( + declared_measures=declared_measures, + bound_filters=bound_filters, + bound_filter_texts=bound_filter_texts, + n_date_range=n_date_range, + order_specs=order_specs, + main_time_key=active_td_key, + n_dims=n_dims, + n_time_dimensions=n_tds, + limit=query.limit, + offset=query.offset, + distinct_dimension_values=query.distinct_dimension_values, + ) + + +def plan_query( # NOSONAR(S3776) — planner entry-point dispatcher. The DEV-1503 addition is a small trigger-predicate branch + a kwarg pass-through; the function's pre-existing complexity is owned by the multi-stage scope / bundle / projection / filter-routing wiring it orchestrates and is tracked as a separate refactor. + *, + query: Union[SlayerQuery, StrictQueryCarrier], + bundle: ResolvedSourceBundle, + scope: Optional[Union[ModelScope, StageSchema]] = None, + cross_model_planner: Optional[CrossModelPlanner] = None, + stage_schemas: Optional[Dict[str, StageSchema]] = None, + disable_host_rooted_isolation: bool = False, + prebound: Optional[PreboundQuery] = None, +) -> PlannedQuery: + """Compile one query into a typed ``PlannedQuery``. + + ``scope`` defaults to a ``ModelScope`` over ``bundle.source_model``; + pass an explicit ``StageSchema`` to bind against an upstream stage. + ``stage_schemas`` is a name → StageSchema map used by + ``plan_stages`` to wire multi-stage references. + + ``prebound`` (DEV-1742 §5.4) supplies the bind product directly, skipping + the parser. ``query`` is then a ``StrictQueryCarrier`` holding only the + post-bind scalars the planner still needs — anything else it is asked for + raises, so a new read cannot silently fall back to a default. + + ``disable_host_rooted_isolation`` (DEV-1503; renamed and widened by + DEV-1709) suppresses the HOST-ROOTED half of the Law-3 trigger — the + isolation of a LOCAL aggregate whose ``Column.filter`` or any other + input (source ``Column.sql``, positional args, kwargs) crosses a join. + It never affects target-rooted isolation (``source.path`` non-empty). + Threaded through ``subplan_builder`` whenever a cross-model strategy + recurses for a host-rooted (or target-rooted) nested sub-plan, so the + sub-plan's same crossing measure is rendered inline (not infinitely + re-isolated) and so re-rooting of a genuine cross-model aggregate + doesn't redundantly isolate the target's own filter joins. + """ + stage_schemas = stage_schemas or {} + cross_model_planner = ( + cross_model_planner or IsolatedCteCrossModelPlanner() + ) + + if scope is None: + scope = _resolve_scope( + query=query, bundle=bundle, stage_schemas=stage_schemas, + ) + + # The generator must render this stage's FROM / joins against the SAME + # model the binder used. For a ModelScope that's the (possibly overlaid / + # synthetic) host; for a StageSchema chain stage it's None (the generator + # builds a synthetic model from the upstream schema). + render_source_model = ( + scope.source_model if isinstance(scope, ModelScope) else None + ) + + if prebound is None: + # A raw SlayerQuery is the only bindable input; a StrictQueryCarrier + # arrives only paired with its own prebound product (§5.4), so reaching + # the parser with one is a wiring bug, not a fall-through. + assert isinstance(query, SlayerQuery) + prebound = bind_query_inputs( + query=query, bundle=bundle, scope=scope, + stage_schemas=stage_schemas, + ) + declared_measures = list(prebound.declared_measures) + bound_filters = list(prebound.bound_filters) + bound_filter_texts = list(prebound.bound_filter_texts) + n_date_range = prebound.n_date_range + order_specs = list(prebound.order_specs) + active_td_key = prebound.main_time_key + n_dims = prebound.n_dims + n_tds = prebound.n_time_dimensions + distinct_dimension_values = prebound.distinct_dimension_values + + # SlayerModel.filters — Mode-A SQL, always-applied WHERE. Scope-derived + # (see ``bind_query_inputs``), so a pre-bound sub-plan picks up its OWN + # model's filters. + text_filter_entries: List[FilterPhase] = [] + if isinstance(scope, ModelScope) and scope.source_model is not None: + for j, mf in enumerate(scope.source_model.filters or []): + text_filter_entries.append(_validate_model_filter( + mf=mf, idx=j, model=scope.source_model, + )) + source_col_names = _source_column_names(scope) host_model_name = _host_model_name(scope) @@ -1134,7 +1235,7 @@ def _rw(vk: ValueKey) -> ValueKey: # raw-rows query materialised a hidden aggregate and emitted # ``GROUP BY ... HAVING ...``, the exact auto-aggregation the flag exists # to turn off. - if query.distinct_dimension_values is False and agg_slots: + if distinct_dimension_values is False and agg_slots: offender = _canonical_name(agg_slots[0].key) raise DistinctDimensionValuesError( f"distinct_dimension_values=False rejects measure references, but " @@ -1190,21 +1291,21 @@ def _rw(vk: ValueKey) -> ValueKey: # slot, ordered by its alias (unchanged); # * row column, UNGROUPED query -> split ``orders.created_at`` / # ``customers__regions.name`` emission in the generator's - # ``_apply_order_limit_from_planned`` (the row IS the grain, so the + # ``_apply_planned_order_limit`` (the row IS the grain, so the # bare reference is legal); # * LOCAL row column, GROUPED query -> the bare reference is NOT legal # (the column is not in GROUP BY), so it materialises as a hidden - # ``:max`` aggregate slot and the order entry is repointed at it. - # MAX is order-preserving per group and portable across every Tier-1 - # dialect; - # * JOINED row column, GROUPED query -> still - # ``UnresolvableOrderColumnError``. A host-rooted MAX over a joined - # column is not expressible today: an ``AggregateKey`` with a - # non-empty ``source.path`` always routes to a TARGET-rooted CTE - # (Law 3), which for a host-grain sort key degenerates to a scalar - # CROSS JOIN — every group would get the same global value and the - # sort would silently do nothing. Failing loudly beats sorting by a - # constant. Full support (host-rooted crossing MAX) is DEV-1735; + # direction-aware aggregate slot (``:min`` for ASC, ``:max`` for DESC) + # and the order entry is repointed at it. Both are order-preserving + # per group and portable across every Tier-1 dialect; + # * JOINED row column, GROUPED query -> the same wrap, marked + # ``grain="host"`` (DEV-1747 D2). The marker is what separates WHERE + # the value is READ (through the join, per ``source.path``) from WHERE + # it is GROUPED (per host row-group). Without it a path-bearing source + # always routed to a TARGET-rooted CTE, which for a host-grain sort key + # degenerates to a scalar CROSS JOIN — every group gets the same global + # value and the sort silently does nothing. That case used to be + # rejected outright rather than sorted wrongly; # * transform / composite -> materialised as a hidden slot and ordered at # the outer wrap (DEV-1733), same Law-2 discipline as aggregates. # @@ -1214,11 +1315,14 @@ def _rw(vk: ValueKey) -> ValueKey: # it, and a sort must not fail because ``max`` is not whitelisted on the # column being sorted. _has_grouping = bool(agg_slots) or ( - bool(query.dimensions or query.time_dimensions) - and query.distinct_dimension_values + bool(n_dims or n_tds) and distinct_dimension_values ) - # ORDER BY targets rewritten to a hidden aggregate: original key -> MAX key. - order_key_remap: Dict[ValueKey, ValueKey] = {} + # ORDER BY targets rewritten to a hidden aggregate wrap, keyed by + # (original key, DIRECTION). DEV-1747 D10 makes the wrap direction-aware, + # so ``ORDER BY a ASC, a DESC`` needs MIN(a) and MAX(a) — two different + # values over one column. Keying by the value key alone would collapse them + # onto whichever slot was interned first. + order_key_remap: Dict[Tuple[ValueKey, str], ValueKey] = {} for spec in order_specs: okey = spec.bound.value_key osid = projection.registry.find_by_key(okey) @@ -1230,29 +1334,35 @@ def _rw(vk: ValueKey) -> ValueKey: if not _has_grouping: continue # raw-rows query -> split emission, no wrap needed path = _row_key_path(okey) - if path: - # ``UnresolvableOrderColumnError`` formats ``qualifier.column``; - # pass the bare leaf as ``column`` and the joined path as the - # qualifier so the message reads ``customers.regions.name`` and - # not a duplicated ``customers.customers.regions.name``. - disp = _partition_key_display(okey) - raise UnresolvableOrderColumnError( - column=disp.rsplit(".", 1)[-1], qualifier=".".join(path), - ) # A TimeTruncKey is not a legal aggregate source; wrap its # underlying column instead. DATE_TRUNC is monotonic - # non-decreasing, so MAX(col) and MAX(trunc(col)) sort a group - # identically. + # non-decreasing, so the wrap over the trunc and over the raw + # column sort a group identically. src = okey.column if isinstance(okey, TimeTruncKey) else okey - max_key = AggregateKey(source=src, agg="max") - if projection.registry.find_by_key(max_key) is None: + # DEV-1747 D10 — ASC orders each group by its MINIMUM and DESC by + # its MAXIMUM: the extreme the direction actually puts first. + # An unconditional MAX (the pre-DEV-1747 behaviour) sorts ASC by + # each group's LARGEST member, which is not what the user asked + # for whenever groups overlap in range. + wrap_key = AggregateKey( + source=src, + agg="min" if spec.direction == "asc" else "max", + # DEV-1747 D2 — a JOINED sort key is host-grain: it must be + # computed in a CTE rooted at the HOST with the crossed join + # pulled inside, grouped on the query grain. Routing it to a + # target-rooted CTE (which a bare path-bearing source does) + # degenerates to a scalar CROSS JOIN, giving every group the + # same global value. + grain="host" if path else "target", + ) + if projection.registry.find_by_key(wrap_key) is None: projection.registry.intern( - key=max_key, - declared_name=_canonical_name(max_key), + key=wrap_key, + declared_name=_canonical_name(wrap_key), hidden=True, - phase=max_key.phase, + phase=wrap_key.phase, ) - order_key_remap[okey] = max_key + order_key_remap[(okey, spec.direction)] = wrap_key # DEV-1733: TransformKey / ArithmeticKey / ScalarCallKey — an inline # transform or composite expression referenced only in ORDER BY. These # materialise as hidden slots (a step CTE on the transform path, a @@ -1319,10 +1429,57 @@ def _windowed_phase(bf: BoundFilter) -> Phase: # BoundFilter (date_range + user filters). Model.filters (text-only) # are always row-phase host-local WHERE and never need to be routed # to a cross-model CTE — they're skipped here. + # DEV-1745 (W4 / D9) — the per-filter structural reachability summary, in + # THIS plan's coordinate system. Computed once here and carried on the plan; + # ``classify_host_filter`` routes from it rather than re-deriving anything + # from model names at classification time. + reachability_anchor_model = render_source_model or bundle.source_model + source_relation = ( + query.source_model + if isinstance(query.source_model, str) + else host_model_name + ) + # The post-bind ``query.*`` surface a re-rooted sub-plan is allowed to see. + # Built here so the SAME object carries the typed bind product into the + # cross-model strategy and back out through ``subplan_builder`` (§5.4). + host_carrier = StrictQueryCarrier( + source_model=( + query.source_model if isinstance(query.source_model, str) else None + ), + name=query.name, + prebound=prebound, + ) + filter_reachability: List[FilterReachability] = [] + # One expansion cache for the whole plan — the two visitors ask for the + # same derived column's expansion, and so does every filter that mentions it. + reachability_cache: dict = {} + for fp in filters_by_phase: + if fp.expression is None: + continue + filter_reachability.append(FilterReachability( + filter_id=fp.id, + crossed_join_paths=compute_key_join_paths( + key=fp.expression.value_key, + anchor_model=reachability_anchor_model, + anchor_relation=source_relation, + bundle=bundle, + cache=reachability_cache, + ), + has_host_local_ref=key_has_host_local_ref( + key=fp.expression.value_key, + anchor_model=reachability_anchor_model, + anchor_relation=source_relation, + bundle=bundle, + cache=reachability_cache, + ), + )) + reachability_by_fid = {r.filter_id: r for r in filter_reachability} + host_filter_routings: List[HostFilterRouting] = [] for fid, bf, ftext in zip( - bound_filter_ids, bound_filters, bound_filter_texts, + bound_filter_ids, bound_filters, bound_filter_texts, strict=True, ): + summary = reachability_by_fid.get(fid) host_filter_routings.append(HostFilterRouting( filter_id=fid, phase=bf.phase, @@ -1330,53 +1487,55 @@ def _windowed_phase(bf: BoundFilter) -> Phase: bf, projection.registry, )), text=ftext, + # §5.4 — the typed predicate, so a sub-plan that inherits this + # filter re-roots the KEY rather than re-parsing ``text``. + bound=bf, + crossed_join_paths=( + summary.crossed_join_paths if summary is not None else () + ), + has_host_local_ref=( + summary.has_host_local_ref if summary is not None else False + ), )) cross_model_plans = [] + ranked_plans: List[RankedAggregatePlan] = [] + ranked_having_ids: Set[BoundFilterId] = set() + # ROW-phase filters (user filters AND the model's own ``filters``) are what + # a ranked CTE re-evaluates over its own rows. Computed once: every ranked + # plan inherits the same list, and the host base keeps applying them too + # (B6 — a LEFT JOIN back propagates a value, never an exclusion). + row_phase_filter_ids: List[BoundFilterId] = [ + fp.id for fp in filters_by_phase if fp.phase == Phase.ROW + ] host_slots_for_classifier = projection.registry.slots for slot in agg_slots: - # DEV-1714 Stage 10 — a windowed slot renders via its own ``_wm_`` CTE - # (host-rooted range join), never a cross-model ``_cm_`` CTE, even when - # its ``Column.filter`` crosses a join (which would otherwise trip the - # host-rooted isolation trigger below). - if slot.id in windowed_slot_ids: - continue - key = slot.key - if not isinstance(key, AggregateKey): - continue - agg_path = getattr(key.source, "path", ()) - # DEV-1503 / DEV-1709 — Law-3 trigger predicate. Invoke the - # cross-model planner when the aggregate's source carries a - # non-empty join path (target-rooted, existing behaviour) OR when - # ANY other input of a LOCAL aggregate crosses a join (host-rooted - # isolation): ``Column.filter`` (typed ``referenced_join_paths`` - # from binder time — DEV-1503), source ``Column.sql``, positional - # args incl. the explicit first/last time arg, kwargs (column - # refs, user template fragments, and non-overridden model-default - # ``AggregationParam`` fragments) — DEV-1709's widened trigger, - # computed plan-time by ``compute_aggregate_input_join_paths``. - has_crossing_filter = ( - key.column_filter_key is not None - and bool(key.column_filter_key.referenced_join_paths) - ) - has_crossing_input = ( - not disable_host_rooted_isolation - and not agg_path - and ( - has_crossing_filter - or bool(compute_aggregate_input_join_paths( - key=key, - anchor_model=bundle.source_model, - anchor_relation=( - bundle.source_model.name - if bundle.source_model is not None else "" - ), - bundle=bundle, - )) - ) + # ONE trigger decision (P-C / DEV-1688 seam). The windowed skip, the + # target-rooted branch and the host-rooted crossing trigger were three + # predicates here that each knew about the others by omission; they are + # one classifier now, and the cardinality-aware inlining decision has a + # single place to land. + kind = classify_isolation( + slot=slot, + windowed_slot_ids=windowed_slot_ids, + bundle=bundle, + disable_host_rooted_isolation=disable_host_rooted_isolation, ) - if not agg_path and not has_crossing_input: + if kind in (IsolationKind.NONE, IsolationKind.WINDOWED): + continue + if kind is IsolationKind.RANKED_HOST: + # Its rows are the host's; nothing about the cross-model decision + # table applies, so it never reaches the strategy. + ranked_plans.append(build_host_ranked_plan( + slot=slot, + row_slots=row_slots, + public_projection=projection.public_projection, + source_model=reachability_anchor_model, + bundle=bundle, + where_filter_ids=row_phase_filter_ids, + )) continue + key = slot.key # DEV-1450 #2: re-rooting (C1) is owned by the strategy. We hand it # the host query, the public projection, and a sub-plan builder so it # can compile a nested re-rooted PlannedQuery when the host carries @@ -1404,7 +1563,7 @@ def _windowed_phase(bf: BoundFilter) -> Phase: host_filters=host_filter_routings, public_alias=slot.public_name, hidden=slot.hidden, - host_query=query if reroot_enabled else None, + host_query=host_carrier if reroot_enabled else None, public_projection=( projection.public_projection if reroot_enabled else None ), @@ -1412,17 +1571,52 @@ def _windowed_phase(bf: BoundFilter) -> Phase: (lambda q, b: plan_query( query=q, bundle=b, cross_model_planner=cross_model_planner, disable_host_rooted_isolation=True, + # The carrier IS the hand-off: its ``prebound`` is the + # typed sub-query the strategy re-rooted structurally. + prebound=q.prebound, )) if reroot_enabled else None ), ) + if kind is IsolationKind.RANKED_TARGET and plan.rerooted_plan is None: + # What the strategy WOULD have made this CTE evaluate as HAVING. The + # ranked plan drops it (a ranked CTE never emits one); this records + # it so the coverage guard below can prove the predicate is still + # applied by some scope rather than silently dropped. + ranked_having_ids.update(plan.having_filter_ids) + # D1: the FORWARD cross-model first/last becomes a ranked plan — + # same CTE, rooted at the target, but ranking rather than a plain + # aggregation. A RE-ROOTED one keeps its ``CrossModelAggregatePlan``: + # its ranked plan belongs to the nested sub-plan, in the sub-plan's + # own coordinate system, and is built by that recursion. + # + # The strategy still runs either way. Re-rooting is its decision to + # make, and the routing it produces on the forward path — which host + # filter this CTE evaluates as WHERE, as HAVING, which is + # unreachable — is the same decision table a ranked CTE needs. + ranked_plans.append(build_target_ranked_plan( + slot=slot, + cross_model_plan=plan, + row_slots=row_slots, + public_projection=projection.public_projection, + bundle=bundle, + )) + continue cross_model_plans.append(plan) + # Loop-invariant lookups for order-scope classification, hoisted out of the + # per-spec loop below (none depend on ``spec``). + order_cross_model_slot_ids = {p.aggregate_slot_id for p in cross_model_plans} + order_ranked_slot_ids = {p.aggregate_slot_id for p in ranked_plans} + order_windowed_slot_ids = set(windowed_slot_ids) + order_slot_by_key = {s.key: s.id for s in projection.registry.slots} order_entries = [] for spec in order_specs: - # A grouped row-column sort key was rewritten to a hidden ``:max`` - # aggregate above; order on that slot, not the bare row key. - okey = order_key_remap.get(spec.bound.value_key, spec.bound.value_key) + # A grouped row-column sort key was rewritten to a hidden direction- + # aware aggregate wrap above; order on that slot, not the bare row key. + okey = order_key_remap.get( + (spec.bound.value_key, spec.direction), spec.bound.value_key, + ) sid = projection.registry.find_by_key(okey) if sid is None: # DEV-1733: an order target that reached here without a slot would @@ -1440,20 +1634,25 @@ def _windowed_phase(bf: BoundFilter) -> Phase: f"arithmetic / scalar expression, a dimension, or declare the " f"expression as a measure and order by its name." ) - order_entries.append( - OrderEntry(slot_id=sid, direction=spec.direction), - ) + order_slot = projection.registry.get(sid) + order_entries.append(OrderEntry( + slot_id=sid, + direction=spec.direction, + scope=_classify_order_scope( + slot=order_slot, + cross_model_slot_ids=order_cross_model_slot_ids, + ranked_slot_ids=order_ranked_slot_ids, + windowed_slot_ids=order_windowed_slot_ids, + public_projection=projection.public_projection, + slot_by_key=order_slot_by_key, + ), + phase=order_slot.key.phase, + )) transform_layers = _emit_transform_layers(slots=projection.registry.slots) stage_schema = _emit_stage_schema( - query=query, projection=projection, - ) - source_relation = ( - query.source_model - if isinstance(query.source_model, str) - else host_model_name + stage_name=query.name, projection=projection, ) - # Stage 7b.10 — the active TD's slot id (``active_td_slot_id``) is resolved # right after projection above so the windowed-plan builder can use it. @@ -1479,27 +1678,188 @@ def _windowed_phase(bf: BoundFilter) -> Phase: wp.where_filter_ids = src_where_ids wp.src_filter_rewrites = src_rewrites + # DEV-1745 (W3 / P-D) — decide the outer-WHERE routing HERE, where the + # cross-model plans (and so ``cte_root_model``) are already known. The + # generator used to rediscover this by re-walking the filters at render + # time; it now consumes the field. + outer_where_filter_ids = _plan_outer_where_filters( + filters_by_phase=filters_by_phase, + cross_model_plans=cross_model_plans, + ranked_plans=ranked_plans, + slots=[*row_slots, *agg_slots, *combined_slots], + ) + + _assert_ranked_having_is_covered( + ranked_having_ids=ranked_having_ids, + outer_where_filter_ids=outer_where_filter_ids, + cross_model_plans=cross_model_plans, + ) + + empty_base_plan = _plan_empty_base_grain( + projection=projection.public_projection, + agg_slots=agg_slots, + cross_model_plans=cross_model_plans, + windowed_plans=windowed_plans, + ranked_plans=ranked_plans, + order_entries=order_entries, + filters_by_phase=filters_by_phase, + outer_where_filter_ids=outer_where_filter_ids, + ) + return PlannedQuery( source_relation=source_relation, row_slots=row_slots, aggregate_slots=agg_slots, cross_model_aggregate_plans=cross_model_plans, windowed_aggregate_plans=windowed_plans, + ranked_aggregate_plans=ranked_plans, combined_expression_slots=combined_slots, transform_layers=transform_layers, filters_by_phase=filters_by_phase, projection=projection.public_projection, order=order_entries, - limit=query.limit, - offset=query.offset, + limit=prebound.limit, + offset=prebound.offset, stage_schema=stage_schema, active_time_dimension_slot_id=active_td_slot_id, render_source_model=render_source_model, - distinct_dimension_values=query.distinct_dimension_values, + distinct_dimension_values=distinct_dimension_values, frame_bound_columns=frame_bound_columns, + outer_where_filter_ids=outer_where_filter_ids, + filter_reachability=filter_reachability, + empty_base_plan=empty_base_plan, ) +def _assert_ranked_having_is_covered( + *, + ranked_having_ids: Set[BoundFilterId], + outer_where_filter_ids: List[BoundFilterId], + cross_model_plans: List[CrossModelAggregatePlan], +) -> None: + """Every filter the strategy routed to a RANKED CTE's HAVING must still be + evaluated somewhere (DEV-1748). + + A ranked CTE emits no HAVING — the predicate belongs on the outer combined + SELECT, because the CTE is LEFT JOINed back and dropping its row would + resurrect the host row carrying NULL. Two scopes pick these up in practice: + the outer WHERE (when the filter references the ranked aggregate itself) and + a sibling ``_cm_`` CTE (when it references a different aggregate on the same + target, which has a plan of its own). + + No query is known to escape both. The guard exists because the failure mode + if one did is a filter that silently stops applying — a wrong answer with no + error — and that is worth converting into a loud one. + """ + covered = set(outer_where_filter_ids) + for plan in cross_model_plans: + covered.update(plan.having_filter_ids) + covered.update(plan.where_filter_ids) + orphaned = sorted(ranked_having_ids - covered) + if orphaned: + raise RuntimeError( + f"Filter(s) {orphaned} were routed to a ranked first/last CTE's " + f"HAVING, which a ranked CTE never emits, and no other scope " + f"evaluates them. Applying nothing would silently widen the " + f"result; planner/renderer drift (DEV-1748).", + ) + + +def _plan_empty_base_grain( + *, + projection: List[SlotId], + agg_slots: list, + cross_model_plans: list, + windowed_plans: list, + order_entries: list, + filters_by_phase: list, + outer_where_filter_ids: List[BoundFilterId], + ranked_plans: Optional[list] = None, +) -> "EmptyBaseGrainPlan | None": + """Decide the DEV-1503 empty-base spine at plan time (§5.12). + + The host base has nothing of its own exactly when every value the query + asks for is an isolated aggregate: no row slots, no host-LOCAL aggregates, + no combined expressions, and nothing ordered that would have to be + materialised there. The generator used to re-derive this from its own + render order; deciding it here keeps the policy on the plan (P-D). + + ``host_filter_ids`` are the ROW-phase filters that remain host-local — not + routed into a ``_cm_*`` CTE and not lifted to the outer WHERE. Without them + the spine would aggregate across host rows the user filtered out. + """ + isolated = {p.aggregate_slot_id for p in cross_model_plans} + isolated |= {p.aggregate_slot_id for p in windowed_plans} + isolated |= {p.aggregate_slot_id for p in (ranked_plans or [])} + if not projection or any(sid not in isolated for sid in projection): + return None + # A host-LOCAL aggregate would have to be computed in ``_base``, which then + # has a column of its own and is not a placeholder spine. + if any(slot.id not in isolated for slot in agg_slots): + return None + # An order target that is not itself isolated must be materialised in + # ``_base`` too, for the same reason. + if any(entry.slot_id not in isolated for entry in order_entries): + return None + routed: set = set(outer_where_filter_ids) + for plan in cross_model_plans: + routed.update(plan.where_filter_ids) + routed.update(plan.having_filter_ids) + host_filter_ids = [ + fp.id + for fp in filters_by_phase + if fp.phase == Phase.ROW + and fp.id not in routed + and (fp.expression is not None or fp.text is not None) + ] + return EmptyBaseGrainPlan(host_filter_ids=host_filter_ids) + + +def _plan_outer_where_filters( + *, + filters_by_phase: List[FilterPhase], + cross_model_plans: List[CrossModelAggregatePlan], + slots: List[ValueSlot], + ranked_plans: Optional[List["RankedAggregatePlan"]] = None, +) -> List[BoundFilterId]: + """AGGREGATE-phase filters that must be applied on the OUTER combined + SELECT instead of as HAVING inside a ``_cm_*`` / ``_rk_*`` CTE (DEV-1503). + + A filter qualifies when its value-key tree references an aggregate that was + isolated into a CTE with its own root (``cte_root_model is not None``), or + into a ranked one — every ranked CTE has its own root by construction. That + CTE LEFT JOINs back to ``_base``, so a HAVING inside it drops CTE rows and + the join then resurfaces the host row carrying a NULL aggregate. The same + predicate on the outer, non-aggregating SELECT drops the row. + + Returned in ``filters_by_phase`` order so the emitted WHERE conjunct order + is stable. + """ + isolated_agg_slot_ids = { + p.aggregate_slot_id + for p in cross_model_plans + if p.cte_root_model is not None + } + isolated_agg_slot_ids |= { + p.aggregate_slot_id for p in (ranked_plans or []) + } + if not isolated_agg_slot_ids: + return [] + slot_by_key = {s.key: s for s in slots} + routed: List[BoundFilterId] = [] + for fp in filters_by_phase: + if fp.phase != Phase.AGGREGATE or fp.expression is None: + continue + for k in walk_value_keys(fp.expression.value_key): + if not isinstance(k, AggregateKey): + continue + slot = slot_by_key.get(k) + if slot is not None and slot.id in isolated_agg_slot_ids: + routed.append(fp.id) + break + return routed + + def _frame_bound_columns(*, row_slots: list) -> List[ValueKey]: """Raw column keys of the stage's NON-HIDDEN time dimensions (DEV-1732). @@ -1719,123 +2079,30 @@ def _format_description_for_dimension( return col.format, col.description -_COUNT_AGGREGATIONS: FrozenSet[str] = frozenset( - {"count", "count_distinct", "count_distinct_approx"} -) -_FLOAT_AGGREGATIONS: FrozenSet[str] = frozenset({ - "avg", "weighted_avg", "median", - "stddev_samp", "stddev_pop", "var_samp", "var_pop", - "corr", "covar_samp", "covar_pop", "percentile", -}) - - -def _infer_aggregated_type( - *, - model: SlayerModel, - measure_name: Optional[str], - aggregation: str, -) -> Optional[DataType]: - """Type for an aggregated measure slot. Mirrors - ``_infer_aggregated_format`` (decision #2 of the Stage B plan): - - * ``*:count`` (measure_name=``"*"``) → ``INT`` - * ``count`` / ``count_distinct`` / ``count_distinct_approx`` → ``INT`` - * ``avg`` / ``weighted_avg`` / ``median`` / parametric / stat aggs → - ``DOUBLE`` - * ``sum`` / ``min`` / ``max`` / ``first`` / ``last`` → inherit from - source column type (DOUBLE if absent). - """ - if measure_name == "*": - return DataType.INT - if aggregation in _COUNT_AGGREGATIONS: - return DataType.INT - if aggregation in _FLOAT_AGGREGATIONS: - return DataType.DOUBLE - # sum / min / max / first / last — preserve source column type. - if measure_name is None: - return None - col = model.get_column(measure_name) - if col is not None and col.type is not None: - return col.type - return None - - def _format_description_for_measure_formula( *, scope: Union[ModelScope, StageSchema], bound, ) -> Tuple[Optional[NumberFormat], Optional[str]]: - """Lift ``format`` / ``description`` for a measure formula. The - aggregation-aware format comes from ``_infer_aggregated_format`` when - the bound expression is a bare local aggregate; description follows - the source ``Column`` (sum / min / max preserve documentation). - """ + """Scope adapter over ``measure_key_format_description`` — a StageSchema + scope has no source model to lift a column's display contract from.""" if not isinstance(scope, ModelScope) or scope.source_model is None: return None, None - if not isinstance(bound.value_key, AggregateKey): - return None, None - src = bound.value_key.source - if isinstance(src, StarKey): - # ``*:count`` — INTEGER format inferred by helper; no description. - return ( - _infer_aggregated_format( - model=scope.source_model, - measure_name="*", - aggregation=bound.value_key.agg, - ), - None, - ) - if not isinstance(src, (ColumnKey, ColumnSqlKey)): - return None, None - if getattr(src, "path", ()): # cross-model — handled by response_meta - return None, None - bare = getattr(src, "leaf", None) or getattr(src, "column_name", None) - if bare is None: - return None, None - fmt = _infer_aggregated_format( - model=scope.source_model, - measure_name=bare, - aggregation=bound.value_key.agg, + return measure_key_format_description( + model=scope.source_model, key=bound.value_key, ) - col = scope.source_model.get_column(bare) - desc = col.description if col is not None else None - return fmt, desc def _type_for_measure_formula( *, scope: Union[ModelScope, StageSchema], bound, ) -> Optional[DataType]: - """Lift ``type`` for a measure-formula slot. - - Mirrors ``_format_description_for_measure_formula`` — sources the - type from ``_infer_aggregated_type`` for local aggregates so the - migrated query-backed virtual model carries ``*:count → INT``, - ``avg → DOUBLE``, ``sum → source column type``. Declared - ``ModelMeasure.type`` overrides — that flow runs through - ``expand_model_measures`` so the bound key already carries the - declared type via the source column lookup. + """Scope adapter over ``measure_key_type``. + + Declared ``ModelMeasure.type`` overrides — that flow runs through + ``expand_model_measures`` so the bound key already carries the declared + type via the source column lookup. """ if not isinstance(scope, ModelScope) or scope.source_model is None: return None - if not isinstance(bound.value_key, AggregateKey): - return None - src = bound.value_key.source - if isinstance(src, StarKey): - return _infer_aggregated_type( - model=scope.source_model, - measure_name="*", - aggregation=bound.value_key.agg, - ) - if not isinstance(src, (ColumnKey, ColumnSqlKey)): - return None - if getattr(src, "path", ()): # cross-model — handled elsewhere - return None - bare = getattr(src, "leaf", None) or getattr(src, "column_name", None) - if bare is None: - return None - return _infer_aggregated_type( - model=scope.source_model, - measure_name=bare, - aggregation=bound.value_key.agg, - ) + return measure_key_type(model=scope.source_model, key=bound.value_key) def _joined_column_type( @@ -2093,8 +2360,7 @@ def _declared_measures_from_query( # ``expand_model_measures`` rewrites the AST but drops the # source measure's type metadata; re-look-up here. # 3. ``_type_for_measure_formula`` — aggregation-aware inference. - # Mirrors how the legacy ``EnrichedMeasure.type`` honored an - # explicit type before falling back to inference. + # An explicit type wins over inference at every level of this chain. m_type = ( m.type or _saved_model_measure_type(scope=scope, formula=formula) @@ -2196,48 +2462,27 @@ def _canonical_alias_for_formula( alias (``amount_percentile_p=0_5_``), breaking parity. """ if bound is not None and isinstance(bound.value_key, AggregateKey): - key = bound.value_key - if isinstance(key.source, StarKey): - measure_name: Optional[str] = "*" - # Cross-model star (``customers.*:count``) carries its join - # path so the canonical alias keeps the ``customers.`` prefix - # (result key ``orders.customers._count``). - path: Tuple[str, ...] = key.source.path - else: - # ColumnKey exposes ``.leaf``; ColumnSqlKey exposes - # ``.column_name``. Both shapes can appear as aggregate - # sources (the synth adapter rejects ``ColumnSqlKey`` with - # a typed deferral; the planner still needs to derive an - # alias before the generator runs). Mirror ``_canonical_name`` - # at ``planning.py:540-545``. - measure_name = ( - getattr(key.source, "leaf", None) - or getattr(key.source, "column_name", None) - ) - path = getattr(key.source, "path", ()) - if measure_name is not None: - prefix = ".".join(path) + "." if path else "" - # Local aggregates retain the kwarg suffix to match legacy - # ``enrichment.py:349`` (``percentile(p=0.5)`` -> - # ``_p_0_5``). Cross-model aggregates ALSO retain it -- the - # legacy ``query_engine.py:2160`` drops it, causing CTE alias - # collision on two parametric variants, which the 7b.5 fix - # corrected at the planner layer. Result-key shape for - # cross-model parametric aggs therefore diverges from - # legacy in this slice (no parity tests for that combination; - # structural correctness over bit-identical legacy output). - return prefix + canonical_agg_name( - measure_name=measure_name, - aggregation_name=key.agg, - agg_args=[agg_kwarg_canonical_str(a) for a in key.args] or None, - agg_kwargs={ - k: agg_kwarg_canonical_str(v) for k, v in key.kwargs - } or None, - ) - # Fall through to text-based path -- AggregateKey source is - # neither StarKey, ColumnKey, nor ColumnSqlKey (shouldn't be - # reachable in practice; the binder restricts sources to - # those three shapes). + # The derivation lives in ``slayer.sql.naming`` (P-F). The + # ``stage_formula`` profile prefixes the join path RELATIVE to the stage + # (no source relation) and keeps a cross-model star's own path, so + # ``customers.*:count`` aliases as ``customers._count`` and surfaces as + # the result key ``orders.customers._count``. + # + # Both local and cross-model aggregates retain the kwarg suffix + # (``percentile(p=0.5)`` -> ``_p_0_5``). For cross-model parametric + # aggregates that DIVERGES from the deleted legacy pipeline, which + # dropped the suffix and thereby collided two parametric variants onto + # one alias — a ratified divergence, pinned by + # tests/test_dev1744_result_key_contract.py. + alias = canonical_aggregate_alias( + bound.value_key, profile="stage_formula", + ) + if alias is not None: + return alias + # ``None`` means the aggregate's source exposes neither a leaf nor a + # column name — not reachable in practice (the binder restricts sources + # to ColumnKey / ColumnSqlKey / StarKey) — so fall through to the + # text-shape path below. text = formula.strip() if ":" in text and "(" not in text: base, agg = text.rsplit(":", 1) @@ -2270,6 +2515,66 @@ def _host_model_name( return "(stage)" +def _composite_reads_an_isolated_cte( + *, + key: ValueKey, + slot_by_key: Dict[ValueKey, SlotId], + isolated_slot_ids: AbstractSet[SlotId], +) -> bool: + """Whether any aggregate leaf of a composite lives in an isolated CTE. + + One such leaf is enough: the composite then cannot be evaluated inside + ``_base`` at all, because that leaf's value is a column of a CTE joined back + to it. Falling back to a host-base scope would silently substitute a plain + aggregate for the cross-model, rolling, or ranked one. + """ + for dep in walk_value_keys(key): + if isinstance(dep, AggregateKey) and slot_by_key.get(dep) in isolated_slot_ids: + return True + return False + + +def _classify_order_scope( + *, + slot: ValueSlot, + cross_model_slot_ids: Set[SlotId], + windowed_slot_ids: Set[SlotId], + public_projection: List[SlotId], + slot_by_key: Dict[ValueKey, SlotId], + ranked_slot_ids: AbstractSet[SlotId] = frozenset(), +) -> OrderScope: + """Name the scope that PRODUCES ``slot``'s value (DEV-1747 §5.10). + + Order matters. A slot can satisfy more than one test — the DEV-1735 order + wrap is both hidden and cross-model — and the producing scope is the + narrower fact, so isolated scopes are checked before the host base. + + A composite is classified OUTER_COMPOSITE when any operand lives in an + isolated CTE: it cannot be evaluated inside ``_base`` at all, and falling + back to a host-base scope would silently substitute a plain aggregate for + the cross-model or rolling one. + """ + if slot.id in cross_model_slot_ids: + return OrderScope.CROSS_MODEL_CTE + if slot.id in ranked_slot_ids: + return OrderScope.RANKED_CTE + if slot.id in windowed_slot_ids: + return OrderScope.WINDOWED_CTE + if isinstance(slot.key, TransformKey): + return OrderScope.TRANSFORM_STEP + if isinstance(slot.key, (ArithmeticKey, ScalarCallKey)) and _composite_reads_an_isolated_cte( + key=slot.key, + slot_by_key=slot_by_key, + isolated_slot_ids=( + cross_model_slot_ids | windowed_slot_ids | set(ranked_slot_ids) + ), + ): + return OrderScope.OUTER_COMPOSITE + if slot.hidden or slot.id not in public_projection: + return OrderScope.HOST_BASE_HIDDEN + return OrderScope.HOST_BASE + + def _bucket_slots(slots: List[ValueSlot]): row: List[ValueSlot] = [] agg: List[ValueSlot] = [] @@ -2286,7 +2591,7 @@ def _bucket_slots(slots: List[ValueSlot]): def _emit_stage_schema( *, - query: SlayerQuery, + stage_name: Optional[str], projection, ) -> StageSchema: """Build the StageSchema from the projection plan. @@ -2337,8 +2642,9 @@ def _emit_stage_schema( format=slot.format, description=slot.description, )) - relation_name = query.name or "(unnamed_stage)" - return StageSchema(relation_name=relation_name, columns=columns) + return StageSchema( + relation_name=stage_name or "(unnamed_stage)", columns=columns, + ) def _emit_transform_layers(*, slots: List[ValueSlot]) -> List[TransformLayer]: @@ -2413,21 +2719,20 @@ def _validate_model_filter( """Validate a ``SlayerModel.filters`` entry and emit a text-only ``FilterPhase`` for it. - Replicates legacy validation (``slayer/engine/enrichment.py:1138-1219``): + Validation: * ``parse_sql_predicate`` rejects DSL constructs (colon aggregation, transform calls) and raw ``OVER(...)`` window functions. * Reject references to a ``ModelMeasure`` declared on the same model — model filters are WHERE-clause SQL, can't reference - aggregates (legacy ``enrichment.py:1147-1153``). + aggregates. * Reject references to a column whose ``Column.sql`` contains a - window function (legacy ``enrichment.py:1205-1219``). + window function. * DEV-1450 follow-up #4b: references to a NON-windowed derived - ``Column.sql`` column are now accepted — the generator inlines the - column's expanded SQL at render time - (``SQLGenerator._render_model_filter_sql``) and pulls any joins the - expansion crosses into the FROM, matching legacy - ``resolve_filter_columns``. + ``Column.sql`` column are accepted — the generator inlines the + column's expanded SQL at render time through the Mode-A door + (``ScopeFrame.enter_predicate``) and pulls any joins the expansion + crosses into the FROM. """ parsed = parse_sql_predicate(mf) measure_names = {m.name for m in (model.measures or [])} @@ -2453,7 +2758,6 @@ def _validate_model_filter( id=f"mf{idx}", phase=Phase.ROW, text=mf, - text_columns=tuple(parsed.columns), expression=None, ) diff --git a/slayer/engine/variables.py b/slayer/engine/variables.py index c7a38fe6..19f2fae6 100644 --- a/slayer/engine/variables.py +++ b/slayer/engine/variables.py @@ -1,8 +1,7 @@ """Stage 7b.1 (DEV-1450) — variable substitution in the new pipeline. -Moves the ``{var}`` placeholder substitution that previously lived inside -the legacy enrichment path (``slayer.engine.enrichment.py:1162``) into a -small, pipeline-friendly module. +Handles the ``{var}`` placeholder substitution in a small, +pipeline-friendly module. Public surface: diff --git a/slayer/mcp/server.py b/slayer/mcp/server.py index d7e4ab37..ccc49d4a 100644 --- a/slayer/mcp/server.py +++ b/slayer/mcp/server.py @@ -1955,11 +1955,21 @@ def _format_table(data: list[dict[str, Any]], columns: list[str], max_rows: int return result -def _format_json(data: list[dict[str, Any]], columns: list[str]) -> str: - """Format data as JSON array.""" - import json +def _format_json( + data: list[dict[str, Any]], + warnings: list[dict[str, Any]] | None = None, +) -> str: + """Format data as JSON. - return json.dumps(data, default=str) + A bare array when there is nothing to report, so the long-standing shape is + unchanged for every clean query. When warnings exist they go INSIDE the + JSON as ``{"data": [...], "warnings": [...]}`` — appending them as prose + would break ``json.loads`` on exactly the queries a caller most needs to + inspect (DEV-1745 W5). + """ + if not warnings: + return json.dumps(data, default=str) + return json.dumps({"data": data, "warnings": warnings}, default=str) def _format_csv(data: list[dict[str, Any]], columns: list[str]) -> str: @@ -1978,13 +1988,53 @@ def _format_csv(data: list[dict[str, Any]], columns: list[str]) -> str: return "\n".join(lines) +def _csv_warning_comments(result: SlayerResponse) -> str: + """Warnings as leading `#` comment lines for CSV output. + + Comments precede the header, so every DATA record keeps a uniform column + count and the advisory is still visible to whoever reads the output. + """ + lines = [f"# warning: {w.human_message()}" for w in (result.warnings or [])] + return "" if not lines else "\n".join(lines) + "\n" + + +def _format_warnings(result: SlayerResponse) -> str: + """Advisories about the query, appended to the TEXT output formats. + + A dropped filter changes which rows the answer covers, so it cannot be + left to a field the caller might not read (DEV-1745 W5 / D2). Rendering + goes through each payload's ``human_message`` so this surface and the CLI + cannot describe the same warning differently. + """ + lines = [f" - {w.human_message()}" for w in (result.warnings or [])] + return "" if not lines else "\n\nWarnings:\n" + "\n".join(lines) + + def _format_output(result: SlayerResponse, fmt: str) -> str: - """Format query output in the requested format.""" + """Format query output in the requested format. + + Warnings never corrupt a machine-readable format: for ``json`` they go + INSIDE the payload under a ``warnings`` key, and for ``csv`` they become + leading ``#`` comment lines. Only ``markdown`` gets a prose block. + + Note this covers the warnings only. The ``query`` tool still prepends + ``SQL:`` text for ``show_sql`` / ``explain`` and appends an attributes + block, which has always made those combinations non-JSON; that predates + this change and is not addressed here. + """ if fmt == "csv": - return _format_csv(data=result.data, columns=result.columns) + # Leading `#` comment lines, never trailing prose: appending the block + # turned each warning into a record with the wrong column count and + # broke every CSV reader on exactly the queries worth inspecting. + return _csv_warning_comments(result) + _format_csv( + data=result.data, columns=result.columns, + ) if fmt == "markdown": - return result.to_markdown() - return _format_json(data=result.data, columns=result.columns) + return result.to_markdown() + _format_warnings(result) + return _format_json( + data=result.data, + warnings=[w.model_dump(mode="json") for w in (result.warnings or [])], + ) def _format_field_meta(entries: dict[str, Any]) -> list[str]: diff --git a/slayer/sql/dialects/base.py b/slayer/sql/dialects/base.py index 6bb8a8a2..25925d2d 100644 --- a/slayer/sql/dialects/base.py +++ b/slayer/sql/dialects/base.py @@ -13,7 +13,7 @@ from __future__ import annotations from functools import lru_cache -from typing import TYPE_CHECKING, Any +from typing import TYPE_CHECKING, Any, Literal, Optional from collections.abc import Callable from pydantic import BaseModel, ConfigDict @@ -211,12 +211,84 @@ def build_null_safe_eq( Base (Postgres-family) uses sqlglot's ``NullSafeEQ`` → ``IS NOT DISTINCT FROM``, which sqlglot also transpiles correctly for DuckDB / Snowflake / - BigQuery / Trino / Databricks / ClickHouse. MySQL overrides to ``<=>``; - SQLite to bare ``IS``; dialects with no native form (T-SQL / Oracle / + BigQuery / Trino / Databricks / ClickHouse — and for MySQL, where it + emits ``<=>``. MySQL therefore needs no override here (an earlier + version of this docstring claimed one existed). ``SqliteDialect`` + overrides to bare ``IS``; dialects with no native form (T-SQL / Oracle / Redshift) to the expanded ``a = b OR (a IS NULL AND b IS NULL)``. """ return exp.NullSafeEQ(this=left, expression=right) + # ------------------------------------------------------------------ + # ORDER BY term construction (DEV-1747 D5 / P-H) + # ------------------------------------------------------------------ + + def build_ordered( + self, + order_col: exp.Expression, + *, + descending: bool, + nulls: Literal["default", "first", "last"] = "default", + ) -> exp.Ordered: + """Build one ``ORDER BY`` term with its null-ordering policy applied. + + The single place any render site turns a resolved column plus a + direction into an ``exp.Ordered`` (P-H). It previously lived on the + generator as ``_ordered``, which meant the combined and transform-chain + paths — which built their own ``exp.Ordered`` — silently skipped it. + + ``nulls="default"`` leaves ``nulls_first`` unset, which sqlglot renders + as **nulls last on every dialect** — an explicit ``NULLS LAST`` where + the native default differs and the syntax exists, a ``CASE WHEN + IS NULL …`` emulation where it does not (MySQL / SQLite). That + uniformity is the point: a semantic layer whose NULLs sort first on + SQLite and last on Postgres answers the same question two ways. + + T-SQL is the one exception and overrides this, because its emulation + does not merely look different — the bracketed alias inside the CASE + re-resolves against the FROM scope and the statement fails. + + ``"first"`` / ``"last"`` are an explicit intent and are honoured as + asked, emulation included — that is the only way to express them on a + dialect with no NULLS syntax. + """ + kwargs: dict = {"this": order_col, "desc": descending} + if nulls == "first": + kwargs["nulls_first"] = True + elif nulls == "last": + kwargs["nulls_first"] = False + return exp.Ordered(**kwargs) + + def native_nulls_first(self, *, descending: bool) -> bool: + """Where NULLs sort in this dialect's OWN ordering for ``descending``. + + Setting ``nulls_first`` to this value is what makes sqlglot emit a bare + ``ORDER BY``: no NULLS clause, no ``CASE WHEN … IS NULL`` emulation. + That is wanted for orderings that are internal machinery rather than a + user-visible sort — a window frame's ``OVER (ORDER BY …)``, where an + emulation term would change which rows the frame covers. + + Read from the same dialect class that GENERATES the clause, for the + same reason :func:`_sqlglot_backslash_escapes` reads the tokenizer: a + hand-kept table would silently disagree with the emitter, and the + symptom is a wrong sort rather than an error. + """ + ordering = getattr( + _SqlglotDialect.get_or_raise(self.sqlglot_name), + "NULL_ORDERING", None, + ) + if ordering == "nulls_are_last": + return False + if ordering == "nulls_are_small": + return not descending + if ordering == "nulls_are_large": + return descending + raise RuntimeError( + f"Cannot derive the native null ordering for sqlglot dialect " + f"{self.sqlglot_name!r}: NULL_ORDERING is {ordering!r}. A sqlglot " + f"upgrade may have changed this API.", + ) + @staticmethod def _expanded_null_safe_eq( left: exp.Expression, right: exp.Expression, @@ -451,6 +523,33 @@ def rewrite_target_ast(self, tree: exp.Expression) -> exp.Expression: """ return tree + def apply_pagination( + self, + select: exp.Select, + *, + limit: Optional[int], + offset: Optional[int], + ) -> exp.Select: + """Apply LIMIT/OFFSET to a completed ``SELECT`` (P-H). + + The single place pagination is expressed. Every render path routes + here, so a dialect that spells pagination differently is handled once + rather than per path — the cross-model combined statement used to append + raw ``LIMIT``/``OFFSET`` text and emitted literal ``LIMIT`` on SQL + Server, while the same query carrying a transform layer went through the + outer wrap and came out correct. + + Setting the bounds on the ``Select`` is what makes transposition work: + sqlglot rewrites them per dialect only when generating the wrapping + SELECT, never from a free-standing ``Limit`` node. + """ + out = select + if limit is not None: + out = out.limit(limit) + if offset is not None: + out = out.offset(offset) + return out + def emit_outer_wrap( self, *, @@ -464,13 +563,14 @@ def emit_outer_wrap( """Emit the DEV-1444 outer-projection wrap around ``inner_sql``. Contract: ``inner_sql`` is the inner SELECT with **trailing - pagination already detached** (``SQLGenerator._build_outer_wrap`` - owns the strip). ``order`` / ``limit`` / ``offset_arg`` are the - detached sqlglot AST nodes the caller pulled off the inner; the - hook re-emits them on the outer statement. + pagination already detached** (the planned outer-wrap path, + ``SQLGenerator._emit_planned_outer_wrap``, owns it — pagination + arrives as detached AST from the plan). ``order`` / ``limit`` / + ``offset_arg`` are the detached sqlglot AST nodes the caller pulled + off the inner; the hook re-emits them on the outer statement. ``parse`` is the generator's ``_parse`` callback when the - generator is the caller (``SQLGenerator._build_outer_wrap``). + generator is the caller (``SQLGenerator._emit_planned_outer_wrap``). T-SQL needs it to preserve SLayer-specific AST rewrites (LOG10/ LOG2 alias preservation, SQLite JSONExtract function-form) when the override re-parses ``inner_sql`` to detach the WITH clause. @@ -528,8 +628,8 @@ def rewrite_emitted_sql(self, sql: str) -> str: generator output. Symmetric companion to ``rewrite_parsed_ast`` (the input-side - hook): write-side, applied at the end of - ``SQLGenerator.generate()`` AFTER ``_apply_outer_projection_trim``. + hook): write-side, applied at the end of the generator's terminal + SQL emit (``generate_planned_stages``). Contract: preserve query semantics. Suitable for alias renames, identifier mangling/escape, dialect-quoting fixes. Do NOT change diff --git a/slayer/sql/dialects/tsql.py b/slayer/sql/dialects/tsql.py index 8cf1a854..1afbf2a3 100644 --- a/slayer/sql/dialects/tsql.py +++ b/slayer/sql/dialects/tsql.py @@ -27,14 +27,14 @@ from __future__ import annotations import re -from typing import Any +from typing import Any, Literal from collections.abc import Callable import sqlglot from sqlglot import exp from slayer.core.enums import TimeGranularity -from slayer.sql.naming import decode_alias, encode_alias +from slayer.sql.naming import OUTER_WRAP_ALIAS, decode_alias, encode_alias from slayer.sql.dialects.base import SqlDialect, _build_covar_decomposition @@ -63,6 +63,20 @@ _TSQL_DOTTED_ALIAS_RE = re.compile(r"\[(\w+(?:\.\w+)+)\]", re.ASCII) +def _offset_ordering_fallback( + order: "exp.Expression | None", offset_arg: "exp.Expression | None", +) -> "exp.Expression | None": + """The ORDER BY an OFFSET-bearing outer wrap must carry: the caller's, or a + synthesized ``ORDER BY (SELECT NULL)`` no-op when there is none (SQL Server + rejects OFFSET without ORDER BY). Returns ``order`` unchanged otherwise, so + a user's ordering is never replaced (DEV-1783).""" + if order is not None or offset_arg is None: + return order + return exp.Order(expressions=[ + exp.Ordered(this=exp.Subquery(this=exp.Select().select(exp.Null()))), + ]) + + class TsqlDialect(SqlDialect): sqlglot_name: str = "tsql" ds_type_aliases: frozenset[str] = frozenset({"mssql", "sqlserver", "tsql"}) @@ -78,6 +92,34 @@ def build_null_safe_eq( portable expanded ``a = b OR (a IS NULL AND b IS NULL)``.""" return self._expanded_null_safe_eq(left, right) + def build_ordered( + self, + order_col: exp.Expression, + *, + descending: bool, + nulls: Literal["default", "first", "last"] = "default", + ) -> exp.Ordered: + """DEV-1571 Bug 2 / DEV-1716 — pin ``nulls_first`` to T-SQL's native + default for the direction (FIRST on ASC, LAST on DESC). + + Left unset, sqlglot emits ``CASE WHEN IS NULL THEN 1 ELSE 0 + END, `` to emulate the nulls-last ordering every other dialect + gets; the bracketed alias INSIDE the CASE WHEN mis-resolves against the + FROM scope (``Invalid column name``). So T-SQL trades null-ordering + parity for a statement that runs — the one place SLayer's null ordering + is dialect-specific, and only because the portable form is unavailable. + + An EXPLICIT ``first`` / ``last`` policy is honoured as asked — the pin + exists to avoid the emulation, not to override a stated intent. + """ + if nulls == "default": + return exp.Ordered( + this=order_col, desc=descending, nulls_first=not descending, + ) + return super().build_ordered( + order_col, descending=descending, nulls=nulls, + ) + def build_approx_count_distinct( self, col_sql: str, @@ -243,6 +285,37 @@ def build_covar_2arg( # DEV-1571 Bug 1: emit_outer_wrap hoists inner top-level CTEs # ------------------------------------------------------------------ + def apply_pagination( + self, + select: exp.Select, + *, + limit: "int | None", + offset: "int | None", + ) -> exp.Select: + """T-SQL pagination, with the ``OFFSET`` ordering requirement made + explicit. + + SQL Server rejects ``OFFSET`` without an ``ORDER BY``. When the query is + genuinely unordered we supply ``ORDER BY (SELECT NULL)`` — the + conventional no-op ordering, which adds no semantics because there were + none to preserve, and only makes the statement legal. + + sqlglot happens to inject the same thing today, but that is its + behaviour and not our contract: doing it here means the rule survives a + sqlglot upgrade, and it puts the ordering in the AST where a caller (and + our tests) can see it rather than only in the generated string. A user's + own ORDER BY is never replaced. + + ``TOP`` versus ``FETCH`` needs no special handling — sqlglot picks + ``TOP`` for a bare limit and ``OFFSET … FETCH`` once an offset is + present, which is the correct T-SQL in both cases. + """ + if offset is not None and select.args.get("order") is None: + select = select.order_by( + exp.Subquery(this=exp.Select().select(exp.Null())), + ) + return super().apply_pagination(select, limit=limit, offset=offset) + def emit_outer_wrap( self, *, @@ -288,6 +361,10 @@ def emit_outer_wrap( to the base impl — T-SQL will still reject malformed SQL at the DB layer, but we don't make it worse. """ + # SQL Server rejects OFFSET without ORDER BY. Resolve the effective + # ordering BEFORE branching, so BOTH the AST path AND the base-impl + # fallback (a non-Select inner, base.py also emits a bare OFFSET) get it. + order = _offset_ordering_fallback(order, offset_arg) parse_fn = parse if parse is not None else ( lambda s: sqlglot.parse_one(s, dialect=self.sqlglot_name) ) @@ -325,7 +402,7 @@ def emit_outer_wrap( col.set("table", None) derived = exp.Subquery( this=parsed, - alias=exp.TableAlias(this=exp.to_identifier("_outer")), + alias=exp.TableAlias(this=exp.to_identifier(OUTER_WRAP_ALIAS)), ) outer = exp.Select() for a in public: diff --git a/slayer/sql/generator.py b/slayer/sql/generator.py index f00ca7f5..da944eaa 100644 --- a/slayer/sql/generator.py +++ b/slayer/sql/generator.py @@ -25,15 +25,20 @@ ) from pydantic import BaseModel, ConfigDict, field_validator -from slayer.core.errors import AggregationNotAllowedError, UnresolvableOrderColumnError -from slayer.core.keys import _FrozenKey, _reroot_path_ref, reroot_aggregate_key +from slayer.core.errors import AggregationNotAllowedError +from slayer.core.formula import RANK_FAMILY_TRANSFORMS +from slayer.core.keys import ( + _FrozenKey, + _reroot_path_ref, + column_path, + reroot_aggregate_key, +) from slayer.core.models import Aggregation from slayer.core.refs import agg_kwarg_canonical_str from slayer.core.time_bounds import strip_frame_bounds from slayer.core.window_duration import parse_window_duration as _parse_window_duration from slayer.engine.column_expansion import ( _is_trivial_base, - _walk_path_to_target_sync, collect_root_scope_joined_paths, expand_derived_refs_sync, ) @@ -43,14 +48,53 @@ ) from slayer.sql.dialects import SqlDialect, get_dialect from slayer.sql.naming import ( + FILTERED_ALIAS, + OUTER_WRAP_ALIAS, AliasAllocator, + canonical_aggregate_alias, + cte_name_from_alias, dialect_folds_case, - flat_name, maybe_quote_ident, quote_mixed_case_identifiers, result_key, result_key_from_alias, ) +from slayer.sql.render.aggregates import window_agg_class +from slayer.sql.render.cte_assembly import CteEntry, assemble_with_chain +from slayer.sql.render.joins import ( + build_grain_joinback_condition, + grain_alias_column, +) +from slayer.sql.render.order_terms import ( + HOST_BASE_SCOPES, + OrderEnv, + resolve_order_term, +) +from slayer.sql.render.ranked import ( + RANKED_CTE_PREFIX, + RankedGrainProjection, + build_rank_column, + build_ranked_cte_select, + build_ranked_pick, + ranked_ordered, +) +from slayer.sql.render.aggregates import ( + DISPATCH_DISTINCT, + DISPATCH_FORMULA, + DISPATCH_STAT, + is_builtin_agg, + resolve_agg_entry, +) +from slayer.sql.render.value_expr import ( + AliasFacilities, + CompositeFacilities, + FilterFacilities, + RenderContext, + _wrap_cast_for_type, + contains_aggregate, + render_value_key, + rewrite_log_alias, +) from slayer.sql.reserved_keywords import prequote_reserved_identifiers from slayer.sql.scope import ScopeFrame from slayer.sql.scope_check import maybe_validate_scopes @@ -80,18 +124,16 @@ class ResolvedAggKwarg(BaseModel): class AggRenderSpec(BaseModel): """DEV-1452 — typed input record for the dialect-aware aggregation helpers (``_build_agg``, ``_build_percentile``, ``_build_stat_agg``, - ``_build_formula_agg``, ``_resolve_value_sql``, ``_resolve_agg_param``, - ``_build_ranked_subquery_from_planned``). - - Decouples the helpers from ``EnrichedMeasure`` so the legacy enrichment - pipeline can be deleted without forking dialect SQL emission. Carries - exactly the 11 fields the helpers empirically read; ``EnrichedMeasure`` - fields outside this set (``agg_args``, ``source_measure_name``, - ``distinct``, ``window``, ``user_declared``, ``label``, - ``filter_columns``) are deliberately NOT carried — ``count_distinct`` - dispatches on the agg name, and the positional time arg for - ``first`` / ``last`` is pre-resolved into ``time_column`` at spec-build - time. + ``_build_formula_agg``, ``_resolve_value_sql``, ``_resolve_agg_param``). + + Gives dialect SQL emission a single typed input, decoupled from the + query representation. Carries exactly the 11 fields the helpers + empirically read; other measure attributes (``agg_args``, + ``source_measure_name``, ``distinct``, ``window``, ``user_declared``, + ``label``, ``filter_columns``) are deliberately NOT carried — + ``count_distinct`` dispatches on the agg name, and the positional time + arg for ``first`` / ``last`` is pre-resolved into ``time_column`` at + spec-build time. """ model_config = ConfigDict(frozen=True, arbitrary_types_allowed=True) @@ -172,110 +214,6 @@ def _coerce_agg_kwargs(cls, v: Any) -> Any: aggregate.""" -class FirstLastRenderState(BaseModel): - """DEV-1501 — bundle of maps produced by - ``_build_first_last_base_select`` (host base) or - ``_render_cross_model_cte`` (cross-model CTE) that the HAVING render - path needs to thread into ``_build_agg`` so a HAVING aggregate - references the same ``_first_rn`` / ``_last_rn{suffix}`` column the - SELECT projects (instead of bare ``_last_rn``, which collapses - distinct time-column specs). - """ - - model_config = ConfigDict(frozen=True, arbitrary_types_allowed=True) - - rn_suffix_map: Dict[str, str] = {} - """Effective-time-column → rn suffix (``""`` / ``"_2"`` / …). Empty - when no first/last aggregates are in scope.""" - - default_time_col_sql: Optional[str] = None - """Fallback time column when a spec has no explicit ``time_column``. - ``None`` when every first/last spec carries an explicit arg.""" - - filtered_rn_map: Dict[str, str] = {} - """Per-spec-alias → dedicated rn column for filtered first/last - aggregates (Column.filter wired in at aggregation time).""" - - filtered_match_map: Dict[str, str] = {} - """Per-spec-alias → match-flag column for filtered first/last.""" - - agg_synth_alias: Optional[str] = None - """DEV-1501 Group A.3 — only set for the cross-model CTE single-agg - case. The cross-model CTE projects exactly one aggregate; if HAVING - references the same key, ``_render_filter_value_key_in_target_scope`` - must rebuild the synth with THIS alias so the ``filtered_rn_map`` / - ``filtered_match_map`` lookups (keyed by the synth alias) hit. Host- - base callers leave this ``None`` and rely on ``aliases_by_slot_id`` - threaded through ``_build_where_having_from_planned`` instead.""" - - value_alias_by_sql: Dict[str, str] = {} - """DEV-1708 Law 2: RESOLVED value text → ``_val_`` materialisation alias - for an aggregate whose SOURCE crosses a join. Keyed by the resolved - (qualified + ``Column.type`` inner-CAST) value emission — DEV-1709 — so - same-sql-different-type aggregates map to distinct materialisations. The - projected aggregate materialises the crossing value inside the ranked - subquery; a HAVING referencing the same aggregate must bind to the SAME - alias (not re-emit the raw crossing ref, which is out of scope in the - outer SELECT). Empty when every source is local.""" - - -def _iter_first_last_leaves(key) -> "list": # NOSONAR(S3776) — sequential isinstance dispatch over the closed ValueKey union; each branch is the per-type recursion contract for surfacing first/last AggregateKey leaves. Extracting per-type helpers would scatter the contract. - """DEV-1501 (Codex round 3): walk a composite ValueKey for first / - last ``AggregateKey`` leaves. - - Composite aggregate slots (``ArithmeticKey`` / ``ScalarCallKey``) - aren't separately materialised — their operand AggregateKeys are - inlined at the composite render path. Without surfacing the leaves - here, the ranked-subquery builder wouldn't see their distinct time - columns and the composite render would resolve every operand to - bare ``_last_rn``. - - Returns local first/last leaves only (cross-model operands raise in - the composite render path; row / literal / scalar-call / - transform / between / in branches recurse into operands without - surfacing themselves). - """ - from slayer.core.keys import ( - AggregateKey, - ArithmeticKey, - BetweenKey, - InKey, - ScalarCallKey, - ) - - out: list = [] - - def _walk(k) -> None: - if isinstance(k, AggregateKey): - if k.agg in ("first", "last") and not getattr( - k.source, "path", (), - ): - out.append(k) - return - if isinstance(k, ArithmeticKey): - for o in k.operands: - _walk(o) - return - if isinstance(k, ScalarCallKey): - for a in k.args: - _walk(a) - return - if isinstance(k, BetweenKey): - _walk(k.column) - _walk(k.low) - _walk(k.high) - return - if isinstance(k, InKey): - _walk(k.column) - # LiteralKey / ColumnKey / TimeTruncKey / TransformKey / etc.: - # not a first/last operand carrier; stop recursing. - - _walk(key) - return out - - - - def _render_scalar_literal(v: Any) -> exp.Expression: """Render a Python scalar (None / bool / int / float / Decimal / str) as a bare sqlglot literal node. Used by the POST-phase filter renderer @@ -291,84 +229,82 @@ def _render_scalar_literal(v: Any) -> exp.Expression: return exp.Literal.string(str(v)) -def _wrap_cast_for_type(expr: exp.Expression, dt: Optional[DataType]) -> exp.Expression: - """DEV-1361: wrap ``expr`` in ``CAST(expr AS )`` so the - declared SLayer ``DataType`` is enforced in emitted SQL. - - Skipped when ``dt`` is ``None`` (no declared type) or ``DataType.TEXT`` - (cosmetic — SQL TEXT/VARCHAR roundtripping is already a no-op for our - purposes and ``CAST(... AS TEXT)`` does not unwrap SQLite's - JSON-quoted-string return values anyway). Also skipped when ``dt`` is - opaque (``DataType.UNKNOWN``) — there is no such SQL type, so - ``CAST(x AS UNKNOWN)`` is invalid in every dialect. Skipped when ``expr`` is a - plain ``exp.Column`` (possibly qualified ``model.col``) — those are - bare column references whose runtime type already matches the declared - type by definition; wrapping them in CAST is dead noise and on SQLite - can be lossy (e.g. ``CAST(text_timestamp AS TIMESTAMP)`` truncating - to a year). Idempotent: if ``expr`` is already a CAST to the same - target, return it unchanged. +def _strip_declared_cast(expr: exp.Expression) -> exp.Expression: + """Unwrap one declared-type ``CAST`` a derived-column expansion added. + + Used for a ranked aggregate's ORDER BY column. The CAST exists to make a + PROJECTED value match its declared type; an ordering key is compared only + to itself, and on SQLite the cast is not merely redundant — ``TIMESTAMP`` + carries numeric affinity, so it truncates every date to its year and ties + the partition. """ - if dt is None or dt == DataType.TEXT or dt.is_opaque: - return expr - if isinstance(expr, exp.Column): - return expr - target = exp.DataType.Type(dt.value) - if isinstance(expr, exp.Cast): - existing = expr.args.get("to") - if isinstance(existing, exp.DataType) and existing.this == target: - return expr - return exp.Cast(this=expr, to=exp.DataType(this=target)) + return expr.this if isinstance(expr, exp.Cast) else expr + +def _collapses_to_ranked_cte(planned_query) -> bool: + """Whether this whole plan IS one ranked CTE (DEV-1748 D9). -def _filter_cast_type(dt: Optional[DataType]) -> Optional[DataType]: - """The CAST target to use when rendering a derived column inside a - WHERE / HAVING predicate (DEV-1450 #4a). + True when the ONLY thing the plan computes is one ranked aggregate at + exactly the grain the plan groups by, with nothing layered on top: no other + isolated aggregate, no host-local aggregate that would need a ``_base`` of + its own, no combined expression, no transform, no outer WHERE, no + pagination, and a projection that is precisely the grain plus the aggregate. - Temporal types (``DATE`` / ``TIMESTAMP``) are suppressed: in a filter - the derived expression is COMPARED, not type-enforced, and - ``CAST(text AS TIMESTAMP)`` on SQLite gives the expression NUMERIC - affinity — truncating a string timestamp to its leading year and - breaking ``BETWEEN`` / comparison. A base temporal column in the same - position is never cast (it renders as a bare ``exp.Column``), so this - keeps the derived form on par. Non-temporal types pass through so a - derived numeric / boolean column still gets its enforcing CAST. + Under those conditions the ranked CTE's body already produces the plan's + rows under the plan's names, so emitting ``_base`` + a combined SELECT + around it adds a ``WITH`` and nothing else. Every clause here is a case + where it would add something more, and the collapse would silently drop it. """ - if dt in (DataType.DATE, DataType.TIMESTAMP): - return None - return dt + plans = planned_query.ranked_aggregate_plans + if len(plans) != 1: + return False + if ( + planned_query.cross_model_aggregate_plans + or planned_query.windowed_aggregate_plans + or planned_query.combined_expression_slots + or planned_query.transform_layers + or planned_query.outer_where_filter_ids + or planned_query.order + or planned_query.limit is not None + or planned_query.offset is not None + ): + return False + plan = plans[0] + if plan.hidden or plan.having_filter_ids: + return False + if len(planned_query.aggregate_slots) != 1: + return False + grain_ids = [m.host_slot_id for m in plan.grain] + visible_row_ids = [s.id for s in planned_query.row_slots if not s.hidden] + if grain_ids != visible_row_ids: + return False + # HIDDEN row slots are deliberately not a reason to refuse. They are filter + # scaffolding — a ``WHERE customers.tier = 'gold'`` interns ``tier`` as a + # hidden ROW slot — and ``_base`` does not project or group by them either: + # the no-transform aux pass is ``aggregates_only``, and a transform layer is + # already excluded above. The ranked CTE applies the very same ROW filters + # (``where_filter_ids``), so the two renderings agree. + return list(planned_query.projection) == [ + *grain_ids, plan.aggregate_slot_id, + ] -logger = logging.getLogger(__name__) -# Maps aggregation name (string) → SQL function name. -_AGG_FUNCTION_MAP: dict[str, str] = { - "count": "COUNT", - "count_distinct": "COUNT_DISTINCT", - "sum": "SUM", - "avg": "AVG", - "min": "MIN", - "max": "MAX", - "median": "MEDIAN", - # "first", "last" use special ROW_NUMBER + conditional aggregate - # "weighted_avg" and custom aggregations use formula substitution - # "percentile", "stddev_samp", "stddev_pop", "var_samp", "var_pop", - # "corr" are dialect-dependent and routed through dedicated builders - # (_build_percentile / _build_stat_agg) — they are intentionally - # absent from this map. -} +# ``_wrap_cast_for_type`` / ``_filter_cast_type`` moved to +# ``slayer.sql.render.value_expr`` (DEV-1763 P-G): the filter-CAST policy is now +# renderer-visible, and they are re-exported above so this module's call sites +# and their pinning tests are unchanged. + +logger = logging.getLogger(__name__) -# DEV-1317: statistical aggregations routed through _build_stat_agg. -# stddev_samp/_pop and var_samp/_pop are 1-arg; corr / covar_samp / -# covar_pop are 2-arg via the `other=` kwarg. SQLite gets these through -# registered Python UDFs; Postgres/DuckDB/MySQL/ClickHouse use the -# native function emitted via sqlglot transpilation. MySQL has no -# native CORR / COVAR_SAMP / COVAR_POP — _build_stat_agg raises -# NotImplementedError there, mirroring _build_median. -_STAT_AGG_NAMES: frozenset[str] = frozenset({ - "stddev_samp", "stddev_pop", "var_samp", "var_pop", - "corr", "covar_samp", "covar_pop", -}) - -# Subset of _STAT_AGG_NAMES that take two columns (LHS + `other=` kwarg). +# DEV-1317: statistical aggregations (``DISPATCH_STAT`` in ``AGG_REGISTRY``) are +# routed through _build_stat_agg. stddev_samp/_pop and var_samp/_pop are 1-arg; +# corr / covar_samp / covar_pop are 2-arg via the `other=` kwarg. SQLite gets +# these through registered Python UDFs; Postgres/DuckDB/MySQL/ClickHouse use the +# native function emitted via sqlglot transpilation. MySQL has no native CORR / +# COVAR_SAMP / COVAR_POP — _build_stat_agg raises NotImplementedError there, +# mirroring _build_median. +# +# The two-column subset (LHS + `other=` kwarg). _TWO_ARG_STAT_AGGS: frozenset[str] = frozenset({"corr", "covar_samp", "covar_pop"}) # DEV-1450 stage 7b.13: aggregations dispatched through the built-in @@ -399,10 +335,6 @@ def _filter_cast_type(dt: Optional[DataType]) -> Optional[DataType]: # to ``Anonymous(this='log10'|'log2', ...)``; the per-dialect native-alias # decision is delegated to ``SqlDialect.should_use_native_log`` (DEV-1716). -# Transforms that use self-join CTEs instead of window functions. -# This gives correct results at result-set edges (no NULLs when the DB has the data) -# and handles gaps in time series correctly. -_SELF_JOIN_TRANSFORMS = {"time_shift"} # Separator used when joining pre-rendered SQL fragments into a conjunctive # WHERE/HAVING clause; extracted as a constant so Sonar S1192 doesn't flag it @@ -444,6 +376,27 @@ def _wrap_filter(sql_str: str, filter_sql: Optional[str]) -> str: return f"(CASE WHEN {filter_sql} THEN {sql_str} END)" +def _is_host_grain(key) -> bool: + """True for an ``AggregateKey`` marked ``grain="host"`` (DEV-1747 D2). + + The marker separates WHERE a value is READ from WHERE it is GROUPED: the + source ``path`` says the value comes through a join, ``grain="host"`` says + the aggregate is nonetheless computed per HOST row-group. Such a key + renders INLINE over the joined relation inside its own scope, rather than + in a target-rooted CTE that would collapse it to one global value. + """ + return getattr(key, "grain", "target") == "host" + + +def _host_grain_join_alias(path) -> str: + """The FROM alias a join ``path`` is emitted under. + + Mirrors ``_build_from_and_joins``: the first hop uses the target's bare + name, later hops the ``__``-delimited path alias. + """ + return "__".join(path) + + def _first_bare_column_name(key) -> Optional[str]: """Return the leaf name of the first bare column reference inside a ROW-phase composite key (DEV-1576 / DEV-1717 error messages). @@ -530,21 +483,26 @@ def _validate_agg_param_value(value: str, param_name: str, agg_name: str) -> Non -def _cte_name_from_alias(prefix: str, alias: str) -> str: - """Build a unique CTE name from a measure alias. +def _cm_plan_identity(*, source_relation: str, plan, agg_slot) -> tuple: + """The dedup identity for a cross-model CTE. - Dots are replaced with ``__`` (double underscore) to avoid collision - with aliases that already contain underscores. E.g.: - - ``orders.revenue_sum`` -> ``_fm_orders__revenue_sum`` - - ``orders_v2.revenue_sum`` -> ``_fm_orders_v2__revenue_sum`` + Structural, never the sanitised name string: the canonical alias omits the + aggregate's column filter, and the name is doubly lossy, so either would + merge plans that must render separately. - DEV-1713: the ``.`` -> ``__`` flatten delegates to - :func:`slayer.sql.naming.flat_name` (single owner); this adds only the - non-identifier-character sanitisation on top. + The reroot shape is part of the identity because the two render paths + produce DIFFERENT join-back pairs and a different aggregate column alias — + forward uses the canonical alias, rerooted uses the sub-plan's. Sharing a + CTE across them would join at the wrong grain or read the wrong column. + The planner interns each key to one slot and emits one plan per slot, so + two plans cannot collide here today; keeping the shape in the identity + means a future planner change cannot make that silently wrong. """ - sanitized = flat_name(alias) - sanitized = re.sub(r"[^a-zA-Z0-9_]", "_", sanitized) - return prefix + sanitized + return ( + source_relation, + agg_slot.key, + plan.rerooted_plan is not None, + ) def _effective_src_filters(*, planned_query, plan) -> list: @@ -604,53 +562,8 @@ def _effective_src_filters(*, planned_query, plan) -> list: _BARE_IDENT_RE = re.compile(r"[A-Za-z_]\w*") -def _strip_trailing_pagination(sql: str) -> str: - """DEV-1444: remove trailing ORDER BY / LIMIT / OFFSET clauses that - SLayer's generator appends as raw string segments after the inner - SELECT body. Used by ``_apply_outer_projection_trim`` so the outer - wrapper owns pagination without it appearing twice. - - Works on the trailing tail only — preserves any ORDER BY / LIMIT / - OFFSET that appears inside nested CTEs or sub-queries (they have a - closing ``)`` after them). - """ - s = sql.rstrip() - # OFFSET / LIMIT use narrow digit-bounded regexes. LIMIT-OFFSET is - # checked before bare OFFSET / LIMIT so the combined form is peeled - # in a single pass. - for pattern in ( - _TRAILING_LIMIT_OFFSET_RE, - _TRAILING_OFFSET_RE, - _TRAILING_LIMIT_RE, - ): - m = pattern.search(s) - if not m or m.start() == 0: - continue - tail = s[m.start():] - if tail.count("(") != tail.count(")"): - continue - s = s[:m.start()].rstrip() - # ORDER BY: use rfind on the upper-cased copy (case-insensitive - # match) instead of a regex with an unbounded character class. Same - # paren-balance check confirms the clause is at the outermost - # nesting level. - upper = s.upper() - pos = upper.rfind("ORDER BY") - if pos > 0: - # Word-boundary on the left (preceding whitespace or newline) - # and after (the BY must be followed by whitespace or end). - left_ok = upper[pos - 1] in " \t\n\r" - right_idx = pos + len("ORDER BY") - right_ok = right_idx >= len(upper) or upper[right_idx] in " \t\n\r" - if left_ok and right_ok: - tail = s[pos:] - if tail.count("(") == tail.count(")"): - s = s[:pos].rstrip() - return s - - class SQLGenerator: - """Generates SQL from an EnrichedQuery.""" + """Generates SQL from a typed ``PlannedQuery`` (from ``stage_planner``).""" def __init__(self, dialect: "str | SqlDialect" = "postgres"): if isinstance(dialect, SqlDialect): @@ -835,37 +748,6 @@ def _parse_predicate(self, sql: str, *, dialect: Optional[str] = None) -> exp.Ex - def _build_outer_wrap( - self, - *, - inner_sql: str, - public: List[str], - order, - limit, - offset_arg, - ) -> str: - """Thin delegate to ``self._dialect.emit_outer_wrap`` (DEV-1716). - - Strips trailing ORDER BY / LIMIT / OFFSET from ``inner_sql`` - (text-level) before handing off to the dialect hook, then passes - the detached AST nodes for re-emission on the outer statement. The - hook owns the wrap shape (base derived-table form; ``TsqlDialect`` - hoists inner CTEs) AND the dialect-correct identifier quoting of the - public-alias list (backticks / brackets / ANSI double quotes). - """ - if order is None and limit is None and offset_arg is None: - stripped = inner_sql - else: - stripped = _strip_trailing_pagination(inner_sql) - return self._dialect.emit_outer_wrap( - inner_sql=stripped, - public=public, - order=order, - limit=limit, - offset_arg=offset_arg, - parse=self._parse, - ) - def _quote_ident(self, name: str) -> str: """Render ``name`` as ONE dialect-quoted identifier string (DEV-1716). @@ -880,32 +762,80 @@ def _quote_ident(self, name: str) -> str: """ return exp.to_identifier(name, quoted=True).sql(dialect=self.dialect) - def _null_safe_join_pair_sql(self, *, left_sql: str, right_sql: str) -> str: - """Render one dialect-aware null-safe equality (DEV-1708 / Codex F2) for - a grain join-back ``ON`` clause. ``left_sql`` / ``right_sql`` are the - already-quoted qualified column strings (``_base."x"`` / ``_cm."x"``); - they are parsed back to AST so the dialect strategy's - ``build_null_safe_eq`` can wrap them (native ``IS NOT DISTINCT FROM`` / - ``<=>`` / ``IS``, or the expanded ``= … OR (… IS NULL AND … IS NULL)``).""" - left = self._parse(left_sql) - right = self._parse(right_sql) - return self._dialect.build_null_safe_eq(left, right).sql(dialect=self.dialect) - - def _ordered(self, order_col: exp.Expression, *, ascending: bool) -> exp.Ordered: - """Build an ``exp.Ordered`` node, suppressing sqlglot's NULLS-emulation - ``CASE WHEN`` on T-SQL (DEV-1571 Bug 2 / DEV-1716). - - On T-SQL, sqlglot emits ``CASE WHEN IS NULL THEN 1 ELSE 0 END, - `` to emulate NULLS ordering whenever ``nulls_first`` is unset; - the bracketed alias INSIDE the CASE WHEN mis-resolves against the FROM - scope (``Invalid column name``). Pinning ``nulls_first`` to T-SQL's - native default for the direction (FIRST on ASC, LAST on DESC) - suppresses the wrapper. No-op on every other dialect. + def _parse_cte_body(self, sql: str) -> exp.Expression: + """Parse a rendered CTE body back into AST for the WITH assembler. + + The deliberate seam. The CTE renderers still return SQL text, and one of + them (the re-rooted cross-model CTE) returns a COMPLETE ``WITH … SELECT`` + statement produced by a nested ``generate_from_planned`` — threading AST + out through that whole pipeline is a larger change than this PR takes on. + Parsing once here keeps the assembly between scopes on AST, which is + what the doctrine is about; the alternative was splicing statement text + into an f-string, which is what it replaced. + + ``sqlglot.parse_one`` rather than :meth:`_parse`: this text is our own + freshly-emitted output, so it needs no prequoting or derived-ref + expansion — only structure. + """ + return sqlglot.parse_one(sql, dialect=self.dialect) + + @staticmethod + def _carry_aliases_in_plan_order( + aliases_by_slot_id: Dict[str, List[str]], + ) -> List[str]: + """Aliases an inner stage carries forward, in PLAN order (B8). + + These lists used to be ``sorted(...)`` — one site still carried the + comment "matches legacy ``_generate_with_computed:1607``", i.e. it was + byte-parity ballast rather than a requirement. Alphabetical order is + unrelated to anything the query means, and it made a step CTE project + its columns in a different order from the base it selects them from. + + ``aliases_by_slot_id`` is populated as slots are rendered, so its + insertion order IS the plan's render order; iterating it directly is + what "plan order" means here. + + A duplicate alias RAISES. Two slots sharing a rendered alias is an + allocator invariant violation: the old ``sorted(...)`` emitted the + column twice, which leaves the downstream ``SELECT "x" FROM step1`` + ambiguous, and silently collapsing it instead would change the stage's + arity while hiding the violation that caused it. + """ + out: List[str] = [] + owner_of: Dict[str, str] = {} + for sid, aliases in aliases_by_slot_id.items(): + for alias in aliases: + owner = owner_of.get(alias) + if owner == sid: + raise ValueError( + f"slot {sid!r} renders the alias {alias!r} more than " + f"once; an inner stage cannot carry the same output " + f"name twice", + ) + if owner is not None: + raise ValueError( + f"slots {owner!r} and {sid!r} both render the alias " + f"{alias!r}; an inner stage cannot carry the same " + f"output name twice", + ) + owner_of[alias] = sid + out.append(alias) + return out + + def _ordered( + self, order_col: exp.Expression, *, ascending: bool, + nulls: str = "default", + ) -> exp.Ordered: + """Build an ``exp.Ordered`` node via the dialect strategy. + + DEV-1747 D5 — the T-SQL ``nulls_first`` pin used to live here, which + left the combined and transform-chain paths (which build their own + ``exp.Ordered``) without it. It now lives in ``SqlDialect.build_ordered`` + so every render site gets identical null ordering (P-H). """ - kwargs: dict = {"this": order_col, "desc": not ascending} - if self.dialect == "tsql": - kwargs["nulls_first"] = ascending - return exp.Ordered(**kwargs) + return self._dialect.build_ordered( + order_col, descending=not ascending, nulls=nulls, + ) @@ -985,102 +915,15 @@ def _build_date_trunc(self, col_expr: exp.Expression, granularity: TimeGranulari col_expr=col_expr, granularity=granularity, parse=self._parse, ) - def _build_transform_sql(self, t) -> str: # NOSONAR S3776 — flat dispatch over transform names; per-transform SQL forms read better as one if/elif tree than as named helpers - """Build a window function SQL expression for a transform. - - DEV-1716: identifier refs are dialect-quoted (``_quote_ident``) so - MySQL/T-SQL/BigQuery get correct quotes; the subsequent - ``self._parse(window_sql)`` reads them back as identifiers (backticks - on MySQL, brackets on T-SQL) rather than string literals. - """ - measure = self._quote_ident(t.measure_alias) - time_col = self._quote_ident(t.time_alias) if t.time_alias else None - partition_cols = getattr(t, "partition_aliases", []) or [] - partition_clause = ( - _SQL_PARTITION_BY + ", ".join(self._quote_ident(a) for a in partition_cols) - if partition_cols - else "" - ) - order_clause = f"ORDER BY {time_col}" if time_col else "" - over_parts = " ".join(p for p in (partition_clause, order_clause) if p) - - # Rank-family OVER clauses always order by the inner measure DESC; their - # partition is empty unless the user passed partition_by= on the call. - rank_order = f"ORDER BY {measure} DESC" - rank_over = " ".join(p for p in (partition_clause, rank_order) if p) - - if t.transform == "cumsum": - return f"SUM({measure}) OVER ({over_parts})" - elif t.transform == "consecutive_periods": - raise ValueError("consecutive_periods should be materialized with staged CTEs") - elif t.transform in _SELF_JOIN_TRANSFORMS: - raise ValueError(f"{t.transform} should not reach _build_transform_sql; it uses self-join CTE") - elif t.transform == "lag": - return f"LAG({measure}, {abs(t.offset)}) OVER ({over_parts})" - elif t.transform == "lead": - return f"LEAD({measure}, {abs(t.offset)}) OVER ({over_parts})" - elif t.transform == "rank": - return f"RANK() OVER ({rank_over})" - elif t.transform == "percent_rank": - return f"PERCENT_RANK() OVER ({rank_over})" - elif t.transform == "dense_rank": - return f"DENSE_RANK() OVER ({rank_over})" - elif t.transform == "ntile": - n = getattr(t, "n", None) - if not isinstance(n, int) or n <= 0: - raise ValueError(f"ntile requires a positive integer n, got {n!r}") - return f"NTILE({n}) OVER ({rank_over})" - elif t.transform == "first": - return ( - f"FIRST_VALUE({measure}) OVER ({over_parts} " - f"ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING)" - ) - elif t.transform == "last": - return ( - f"FIRST_VALUE({measure}) OVER ({partition_clause} ORDER BY {time_col} DESC " - f"ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING)" - ) - else: - raise ValueError(f"Unsupported transform: {t.transform}") - - - - - - # ------------------------------------------------------------------ - # FROM / JOIN building - # ------------------------------------------------------------------ - - - - # ------------------------------------------------------------------ - # Column / measure resolution (from enriched SQL expressions) - # ------------------------------------------------------------------ - def _rewrite_log_aliases(self, node: exp.Expression) -> exp.Expression: - """DEV-1337: rewrite ``Log(this=Literal(10|2), expression=X)`` back to - ``Anonymous(this='log10'|'log2', expressions=[X])`` for dialects with - native single-arg aliases. Walked over every parsed AST so the - rewrite survives sqlglot's re-parse passes (which would otherwise - turn ``LOG10(x)`` back into a generic ``Log`` node and re-emit as - ``LOG(10, x)``). No-op on non-``Log`` nodes and on ``Log`` nodes - with a non-literal or non-{10,2} base. + """Thin delegator to the shared log-alias policy in + ``slayer.sql.render.value_expr``. + + Kept as a method so the existing ``tree.transform(...)`` call sites, + which walk every parsed AST so the rewrite survives sqlglot's re-parse + passes, stay unchanged. """ - if not isinstance(node, exp.Log): - return node - base = node.args.get("this") - arg = node.args.get("expression") - if arg is None or not isinstance(base, exp.Literal) or base.is_string: - return node - try: - base_val = float(base.this) - except (TypeError, ValueError): - return node - if base_val == 10 and self._dialect.should_use_native_log(10): - return exp.Anonymous(this="log10", expressions=[arg.copy()]) - if base_val == 2 and self._dialect.should_use_native_log(2): - return exp.Anonymous(this="log2", expressions=[arg.copy()]) - return node + return rewrite_log_alias(node, dialect=self._dialect) def _resolve_sql( self, @@ -1091,8 +934,8 @@ def _resolve_sql( ) -> exp.Expression: """Resolve an enriched SQL expression to a sqlglot AST node. - DEV-1361: when the caller has a typed object in scope (an - ``EnrichedDimension``, a ``Column``), it passes ``type=`` so the + DEV-1361: when the caller has a typed object in scope (a typed + slot, a ``Column``), it passes ``type=`` so the generator wraps non-trivial expressions in ``CAST(... AS )``. Bare identifiers (``sql=None`` or ``sql`` is a single identifier) trust the DB schema and sqlglot — no CAST is emitted regardless of @@ -1170,8 +1013,8 @@ def _resolve_agg_param( if name in spec.agg_kwargs: value = spec.agg_kwargs[name] # Guard the untrusted string forms: a ``kind="str"`` wrapper OR a - # bare ``str`` (the legacy ``EnrichedMeasure`` adapter and model-level - # defaults reach here unwrapped). ``kind="expr"`` is a trusted, + # bare ``str`` (model-level defaults and direct-construction call + # sites reach here unwrapped). ``kind="expr"`` is a trusted, # bind-time-resolved expression and is embedded verbatim. if isinstance(value, ResolvedAggKwarg): if value.kind == "str": @@ -1196,12 +1039,11 @@ def _resolve_agg_param( def _build_agg( self, spec: "AggRenderSpec | None" = None, - rn_suffix_map: Optional[dict[str, str]] = None, - default_time_col: Optional[str] = None, - filtered_rn_map: Optional[dict[str, str]] = None, - filtered_match_map: Optional[dict[str, str]] = None, ) -> tuple[exp.Expression, bool]: - """Build an aggregation expression from an ``AggRenderSpec``.""" + """Build an aggregation expression from an ``AggRenderSpec``. + + First/last aggregates never reach here — they render as a plan-shaped + ``RankedAggregatePlan`` CTE (DEV-1748 B9), not through this emitter.""" if spec is None: # pragma: no cover — defensive raise ValueError("_build_agg requires a 'spec'.") agg_name = spec.aggregation @@ -1219,84 +1061,40 @@ def _build_agg( table=exp.to_identifier(spec.model_name), ), False - # --- first/last: MAX(CASE WHEN _rn = 1 THEN col END) --- - if agg_name in ("first", "last"): - col_expr = self._resolve_sql( - sql=spec.sql, - name=spec.name, - model_name=spec.model_name, - type=spec.column_type, - ) - col = col_expr.sql(dialect=self.dialect) - suffix = "" - if rn_suffix_map is not None: - # DEV-1501: when no default ranking time column is in scope, - # every first/last spec is guaranteed to carry an explicit - # ``time_column`` (validated in - # ``_build_first_last_base_select``); so the suffix lookup - # must not gate on ``default_time_col`` being truthy, else - # distinct-time-column specs all collapse to ``_last_rn``. - effective_tc = spec.time_column or default_time_col - if effective_tc is not None: - suffix = rn_suffix_map.get(effective_tc, "") - rn_col = f"_first_rn{suffix}" if agg_name == "first" else f"_last_rn{suffix}" - # For filtered first/last, use the dedicated ROW_NUMBER column - # that pushes non-matching rows to the bottom of the ranking. - # Look up by alias (unique per spec) so two filtered specs - # sharing source/agg but with different filters map to their - # own respective rank columns. Use the per-spec match flag - # (also projected by the ranked subquery) instead of - # re-emitting spec.filter_sql here — the filter can reference - # joined-table columns that are not in scope outside the - # subquery. - if spec.filter_sql and filtered_rn_map: - filtered_rn = filtered_rn_map.get(spec.alias, rn_col) - match_col = ( - filtered_match_map.get(spec.alias) - if filtered_match_map - else None - ) - # Fall back to the raw filter expression only if no match flag - # was projected (legacy callers); accepts the leak risk. - filter_clause = f"{match_col} = 1" if match_col else spec.filter_sql - case_sql = ( - f"MAX(CASE WHEN {filtered_rn} = 1 AND {filter_clause} " - f"THEN {col} END)" - ) - else: - # ``col`` is already a fully-qualified SQL expression resolved - # via ``_resolve_sql`` earlier in this branch, so we don't need - # to re-prefix ``spec.model_name``. (DEV-1333.) - case_sql = f"MAX(CASE WHEN {rn_col} = 1 THEN {col} END)" - return self._parse(case_sql), True - - # --- Custom or parameterized aggregation (formula-based) --- - if agg_name not in _AGG_FUNCTION_MAP: - # percentile is dialect-dependent (no static formula works on - # SQLite/ClickHouse/MySQL) so it gets its own builder rather than - # going through the BUILTIN_AGGREGATION_FORMULAS path. - if agg_name == "percentile": - return self._build_percentile(spec), True - # Statistical aggregates also dispatch to a dedicated builder so - # the SQLite-UDF / native-function / NotImplementedError split - # mirrors _build_median. - if agg_name in _STAT_AGG_NAMES: - return self._build_stat_agg(spec), True - # count_distinct_approx (DEV-1595): dialect-aware approximate- - # distinct — native function (DuckDB/ClickHouse/BigQuery/…) or the - # exact COUNT(DISTINCT) fallback (Postgres/SQLite/MySQL). Built like - # percentile/stat-agg (via _wrap_filter + _resolve_value_sql) so a - # row-level filter wraps as COUNT(DISTINCT (CASE WHEN ... END)). - if agg_name == "count_distinct_approx": - col_expr = _wrap_filter( - self._resolve_value_sql(spec), spec.filter_sql - ) - return self._dialect.build_approx_count_distinct( - col_sql=col_expr, parse=self._parse - ), True + # Classification comes from the single ``AGG_REGISTRY`` table (DEV-1744): + # a name not registered is a model-level custom aggregation and takes the + # formula-template path. + if not is_builtin_agg(agg_name): return self._build_formula_agg(spec, agg_name), True - # --- Resolve inner expression --- + entry = resolve_agg_entry(agg_name) + dispatch = entry.dispatch + + # --- Builders that resolve (and filter-wrap) their OWN inner --- + # These are dialect-dependent or template-based, so they cannot share the + # plain inner resolution below and run BEFORE it (which also keeps them + # from triggering its join-discovery side effect). + if dispatch == DISPATCH_STAT: + # DEV-1317: SQLite-UDF / native-function / NotImplementedError split. + return self._build_stat_agg(spec), True + if dispatch == DISPATCH_FORMULA: + # ``weighted_avg`` and any other {value}/{param} template built-in. + return self._build_formula_agg(spec, agg_name), True + if agg_name == "percentile": + # Dialect-dependent (no static formula works on + # SQLite/ClickHouse/MySQL) so it gets its own builder. + return self._build_percentile(spec), True + if agg_name == "count_distinct_approx": + # DEV-1595: native approx-distinct or the exact COUNT(DISTINCT) + # fallback. A row-level filter wraps as COUNT(DISTINCT (CASE ...)). + col_expr = _wrap_filter( + self._resolve_value_sql(spec), spec.filter_sql + ) + return self._dialect.build_approx_count_distinct( + col_sql=col_expr, parse=self._parse + ), True + + # --- Resolve inner expression (SIMPLE / DISTINCT / median paths) --- if agg_name == "count" and spec.sql is None: # COUNT(*) — if filtered, use COUNT(CASE WHEN filter THEN 1 END) if spec.filter_sql: @@ -1324,7 +1122,7 @@ def _build_agg( inner = self._parse(case_sql) # --- count_distinct --- - if agg_name == "count_distinct": + if dispatch == DISPATCH_DISTINCT: return exp.Count(this=exp.Distinct(expressions=[inner])), True # --- median (dialect-dependent) --- @@ -1332,16 +1130,8 @@ def _build_agg( return self._build_median(inner), True # --- Standard aggregations (sum, avg, min, max, count) --- - agg_class_map = { - "COUNT": exp.Count, - "SUM": exp.Sum, - "AVG": exp.Avg, - "MIN": exp.Min, - "MAX": exp.Max, - } - agg_func = _AGG_FUNCTION_MAP[agg_name] - agg_class = agg_class_map[agg_func] - return agg_class(this=inner), True + # ``node_class`` is the sqlglot class the registry entry carries. + return entry.node_class(this=inner), True def _build_formula_agg(self, spec: AggRenderSpec, agg_name: str) -> exp.Expression: # NOSONAR(S3776) — sequential dispatch over formula source (aggregation_def vs built-in) and per-kind ResolvedAggKwarg substitution (DEV-1527); one cohesive template-substitution contract. """Build SQL for formula-based aggregations (weighted_avg, custom).""" @@ -1514,19 +1304,16 @@ def _build_stat_agg(self, spec: AggRenderSpec) -> exp.Expression: # ====================================================================== # DEV-1450 stage 7b.8 — PlannedQuery → SQL. # - # The legacy generator (everything above) consumes EnrichedQuery. This - # new entry point consumes the typed PlannedQuery from - # slayer/engine/stage_planner.py. The two paths coexist until the - # engine cutover (stage 7b.15) flips the default path. - # - # 7b.8 scope: local-only single-model queries — row-phase dims, local - # aggregates, Mode-B row filters, ORDER BY / LIMIT / OFFSET, dim-only - # dedup. Cross-model, time dimensions, transforms, and aggregate - # filtering raise NotImplementedError with an explicit stage marker - # so silent parity drift is impossible. + # This entry point consumes the typed PlannedQuery from + # slayer/engine/stage_planner.py and renders the full pipeline: + # row-phase dims, local aggregates, Mode-B row filters, ORDER BY / + # LIMIT / OFFSET, dim-only dedup, plus cross-model, time dimensions, + # transforms, and aggregate filtering. # ====================================================================== - def generate_from_planned(self, planned_query, *, bundle) -> str: + def generate_from_planned( + self, planned_query, *, bundle, as_cte_body: bool = False, + ) -> str: """Render a typed ``PlannedQuery`` to SQL (public entry). DEV-1708 (D-E): installs a fresh generation-wide ``AliasAllocator`` for @@ -1536,21 +1323,61 @@ def generate_from_planned(self, planned_query, *, bundle) -> str: rerooted sub-generation (``_render_rerooted_cross_model_cte`` → ``generate_from_planned``) is a self-contained statement and gets its own allocator, with the parent's restored afterwards. + + ``as_cte_body`` says the result is about to become a CTE DEFINITION + rather than a statement, which forbids a ``WITH`` of its own (SQL Server + rejects a nested one outright). Only the caller knows that, so only the + caller can say it — see :func:`_collapses_to_ranked_cte` for the one + shape that currently needs it. """ + self._assert_projection_is_public(planned_query) prev_allocator = getattr(self, "_gen_allocator", None) self._gen_allocator = self._new_allocator() try: return self._generate_from_planned_impl( - planned_query, bundle=bundle, + planned_query, bundle=bundle, as_cte_body=as_cte_body, ) finally: self._gen_allocator = prev_allocator + @staticmethod + def _assert_projection_is_public(planned_query) -> None: + """The renderer-side belt for the public-projection invariant (§5.2). + + ``PlannedQuery`` validates this at construction, but pydantic's + ``model_copy(update=...)`` skips validators — and rerooting a plan uses + exactly that. So the ONE place every render path passes through checks + it again. It RAISES rather than skipping the offending slot: silently + dropping a column the plan asked for is how a wrong answer reaches a + user, whereas a raise names the slot. + + This is the only such check left; the defensive ``if slot.hidden: + continue`` guards the renderers used to carry are redundant now that + the projection is authoritative. + """ + slots = { + slot.id: slot + for slot in ( + list(planned_query.row_slots) + + list(planned_query.aggregate_slots) + + list(planned_query.combined_expression_slots) + ) + } + for sid in planned_query.projection: + slot = slots.get(sid) + if slot is not None and slot.hidden: + raise ValueError( + f"hidden slot {sid!r} reached the public projection; the " + f"plan's projection must contain only public slots " + f"(a model_copy that skips validation is the usual cause)", + ) + def _generate_from_planned_impl( # NOSONAR(S3776) — top-level dispatch over cross-model / transform-chain / plain branches plus the conditional outer-trim wrap. Each branch is a coherent compilation strategy; extracting would scatter the shared planned_query / slots_by_id / aliases_by_slot_id state across helpers without simplifying anything. self, planned_query, *, bundle, + as_cte_body: bool = False, ) -> str: """Render a typed ``PlannedQuery`` to SQL. @@ -1560,15 +1387,12 @@ def _generate_from_planned_impl( # NOSONAR(S3776) — top-level dispatch over c the DB-bound terminal (``generate_planned_stages``), NOT here. Mangling a stage's column names would break the downstream flat-name binding. - Mirrors the local-only branch of ``_generate_base`` but reads - from typed PlannedQuery fields (``row_slots`` / ``aggregate_slots`` - / ``filters_by_phase`` / ``order`` / ``transform_layers``) - instead of ``EnrichedQuery``. Reuses legacy dialect helpers + Reads from typed PlannedQuery fields (``row_slots`` / + ``aggregate_slots`` / ``filters_by_phase`` / ``order`` / + ``transform_layers``) and renders through the dialect helpers (``_resolve_sql`` / ``_build_agg`` / ``_wrap_cast_for_type`` / ``_parse_predicate`` / ``_build_date_trunc``) so dialect-specific - behavior is rendered identically to the legacy ``generate()`` - path — the parity oracle in ``tests/parity_oracle.py`` pins - this contract. + behavior is emitted consistently across the pipeline. Stage 7b.10 adds window-transform rendering: when ``planned_query.transform_layers`` is non-empty, the base SELECT @@ -1587,9 +1411,41 @@ def _generate_from_planned_impl( # NOSONAR(S3776) — top-level dispatch over c ) source_relation = planned_query.source_relation + if as_cte_body and _collapses_to_ranked_cte(planned_query): + # D9. A re-rooted cross-model CTE renders its sub-plan as a COMPLETE + # statement and splices it into a CTE body, so a sub-plan that + # emitted ``_base`` plus a combined SELECT would put a ``WITH`` + # inside a CTE — which SQL Server rejects outright. It never + # happened before because no sub-plan ever contained an isolated + # aggregate; a re-rooted first/last is the first one that does. + # + # It also never needs to: when the sub-plan's only isolated + # aggregate IS its answer, at its own grain, the ranked CTE's body + # and the statement the long way round would produce are the same + # rows under the same names. So emit it directly. + return self._render_collapsed_ranked_plan( + planned_query=planned_query, bundle=bundle, + ) + if as_cte_body and planned_query.ranked_aggregate_plans: + # The residual, made loud. Anything a ranked sub-plan cannot collapse + # would go through ``_render_with_cross_model_plans`` and emit its own + # ``WITH`` — a nested one, which SQL Server rejects, and which sqlglot + # otherwise FLATTENS into the parent chain where the two ``_base`` + # CTEs then collide. Both outcomes are invalid SQL that no unit test + # reads, so a shape that escapes the collapse must stop here rather + # than reach a database. + raise NotImplementedError( + "A re-rooted cross-model first/last whose sub-plan needs more " + "than the ranked CTE itself is not yet supported: the sub-plan " + "renders into a CTE body, which cannot carry a WITH of its own. " + "Split the measure into an earlier stage, or drop the part of " + "the query the sub-plan cannot express in one SELECT.", + ) + if ( planned_query.cross_model_aggregate_plans or planned_query.windowed_aggregate_plans + or planned_query.ranked_aggregate_plans ): return self._render_with_cross_model_plans( planned_query=planned_query, bundle=bundle, @@ -1659,8 +1515,6 @@ def _generate_from_planned_impl( # NOSONAR(S3776) — top-level dispatch over c aliases_by_slot_id, has_aggregation, group_by_keys, - where_consumed, - first_last_state, ) = self._build_base_select_for_planned( planned_query=planned_query, bundle=bundle, @@ -1675,16 +1529,10 @@ def _generate_from_planned_impl( # NOSONAR(S3776) — top-level dispatch over c source_relation=source_relation, source_model=source_model, bundle=bundle, - first_last_state=first_last_state, aliases_by_slot_id=aliases_by_slot_id, ) - # ``where_consumed`` is True for the first/last ranked-subquery path: - # the WHERE is applied INSIDE the ranked subquery (it must filter raw - # rows before ranking), so re-applying it on the outer SELECT would be - # both redundant and — for filters that should narrow the ranked set — - # semantically wrong. - if where_clause is not None and not where_consumed: + if where_clause is not None: base_select = base_select.where(where_clause) # Match legacy _generate_base:1375 — dim-only-dedup OR @@ -1728,7 +1576,7 @@ def _generate_from_planned_impl( # NOSONAR(S3776) — top-level dispatch over c slots_by_id=slots_by_id, bundle=bundle, ) - base_select = self._apply_order_limit_from_planned( + base_select = self._apply_planned_order_limit( select=base_select, planned_query=planned_query, source_relation=source_relation, @@ -1739,9 +1587,12 @@ def _generate_from_planned_impl( # NOSONAR(S3776) — top-level dispatch over c ) return base_select.sql(dialect=self.dialect, pretty=True) - # 7b.10 — transform layers present. Build the CTE chain. - base_cte_sql = base_select.sql(dialect=self.dialect, pretty=True) - ctes: list[tuple[str, str]] = [("base", base_cte_sql)] + # 7b.10 — transform layers present. Build the CTE chain. Bodies stay + # ``exp.Select`` from renderer to assembler (D8): this chain carries + # dotted ``.`` names throughout, and a render-to-text- + # and-re-parse seam re-reads one as a multi-part reference on any + # dialect that mangles dots at emission. + ctes: List[CteEntry] = [CteEntry(name="base", query=base_select)] # DEV-1692: collision-safe CTE-name allocator for the whole transform # chain. The hoisted time_shift slot alias (``_time_shift_inner``) # repeats across arithmetic-wrapped shifts, so two ``shifted_`` / @@ -1749,7 +1600,7 @@ def _generate_from_planned_impl( # NOSONAR(S3776) — top-level dispatch over c # CTE name is reserved/allocated through this one allocator so the # ``step`` / ``shifted_`` / ``sjoin_`` / ``cp_`` families never collide. cte_allocator = self._new_allocator() - cte_allocator.reserve(*(name for name, _ in ctes)) + cte_allocator.reserve(*(entry.name for entry in ctes)) # Codex (PR #269): also reserve every already-projected column alias's # BARE form so a hidden transform alias minted below # (``_time_shift_inner`` / ``_consecutive_periods_inner``) can never @@ -1785,6 +1636,10 @@ def _generate_from_planned_impl( # NOSONAR(S3776) — top-level dispatch over c bundle=bundle, ) ) + # Explicit chain tail: the CTE the next transform step reads from, + # tracked directly rather than as ``ctes[-1]`` so an append elsewhere in + # the list can never silently retarget where the chain continues. + chain_tail = ctes[-1].name while pending_layers: ready_window: list = [] ready_time_shift: list = [] @@ -1814,11 +1669,13 @@ def _generate_from_planned_impl( # NOSONAR(S3776) — top-level dispatch over c if ready_window: step_num += 1 step_name = cte_allocator.allocate_cte(f"step{step_num}") - prev_cte = ctes[-1][0] - carry_aliases_sorted = sorted( - a for aliases in aliases_by_slot_id.values() for a in aliases + prev_cte = chain_tail + carry_aliases = self._carry_aliases_in_plan_order( + aliases_by_slot_id, ) - step_parts = [self._quote_ident(a) for a in carry_aliases_sorted] + step_parts = [ + exp.column(a, quoted=True) for a in carry_aliases + ] for layer in ready_window: for slot_id in layer.slot_ids: slot = slots_by_id[slot_id] @@ -1828,7 +1685,7 @@ def _generate_from_planned_impl( # NOSONAR(S3776) — top-level dispatch over c else slot.declared_name ) full_alias = f"{source_relation}.{alias}" - window_sql = self._render_window_transform_sql( + window_expr = self._render_window_transform_sql( slot=slot, slots_by_id=slots_by_id, slot_id_by_key=slot_id_by_key, @@ -1836,30 +1693,32 @@ def _generate_from_planned_impl( # NOSONAR(S3776) — top-level dispatch over c planned_query=planned_query, ) if slot.type is not None: - wrapped = _wrap_cast_for_type( - self._parse(window_sql), slot.type, + window_expr = _wrap_cast_for_type( + window_expr, slot.type, ) - window_sql = wrapped.sql(dialect=self.dialect) - step_parts.append(f'{window_sql} AS {self._quote_ident(full_alias)}') + step_parts.append( + window_expr.as_(full_alias, quoted=True), + ) aliases_by_slot_id.setdefault(slot_id, []).append( full_alias, ) available_alias_by_slot_id.setdefault( slot_id, full_alias, ) - step_sql = ( - "SELECT\n " - + _SQL_COL_SEP.join(step_parts) - + f"\nFROM {prev_cte}" - ) - ctes.append((step_name, step_sql)) + ctes.append(CteEntry( + name=step_name, + query=exp.Select().select(*step_parts).from_(prev_cte), + depends_on=[prev_cte], + )) + chain_tail = step_name # --- time_shift layers (each gets shifted_ + sjoin_ pair) - for layer in ready_time_shift: for slot_id in layer.slot_ids: slot = slots_by_id[slot_id] - self._emit_time_shift_ctes_for_planned( + chain_tail = self._emit_time_shift_ctes_for_planned( slot=slot, ctes=ctes, + chain_tail=chain_tail, cte_allocator=cte_allocator, slots_by_id=slots_by_id, slot_id_by_key=slot_id_by_key, @@ -1876,9 +1735,10 @@ def _generate_from_planned_impl( # NOSONAR(S3776) — top-level dispatch over c for layer in ready_cp: for slot_id in layer.slot_ids: slot = slots_by_id[slot_id] - self._emit_consecutive_periods_ctes_for_planned( + chain_tail = self._emit_consecutive_periods_ctes_for_planned( slot=slot, ctes=ctes, + chain_tail=chain_tail, cte_allocator=cte_allocator, slots_by_id=slots_by_id, slot_id_by_key=slot_id_by_key, @@ -1913,12 +1773,12 @@ def _generate_from_planned_impl( # NOSONAR(S3776) — top-level dispatch over c unmaterialised.append(cslot) if unmaterialised: step_num += 1 - step_name = f"step{step_num}" - prev_cte = ctes[-1][0] - carry_aliases_sorted = sorted( - a for aliases in aliases_by_slot_id.values() for a in aliases + step_name = cte_allocator.allocate_cte(f"step{step_num}") + prev_cte = chain_tail + carry_aliases = self._carry_aliases_in_plan_order( + aliases_by_slot_id, ) - step_parts = [self._quote_ident(a) for a in carry_aliases_sorted] + step_parts = [exp.column(a, quoted=True) for a in carry_aliases] for cslot in unmaterialised: alias = ( cslot.public_aliases[0] @@ -1926,48 +1786,44 @@ def _generate_from_planned_impl( # NOSONAR(S3776) — top-level dispatch over c else cslot.declared_name ) full_alias = f"{source_relation}.{alias}" - rendered = self._render_value_key_against_aliases( + rendered = render_value_key( key=cslot.key, - slot_id_by_key=slot_id_by_key, - available_alias_by_slot_id=available_alias_by_slot_id, + ctx=RenderContext( + dialect=self._dialect, + aliases=AliasFacilities( + slot_id_by_key=slot_id_by_key, + available_alias_by_slot_id=available_alias_by_slot_id, + ), + ), ) - expr_sql = rendered.sql(dialect=self.dialect) if cslot.type is not None: - wrapped = _wrap_cast_for_type( - self._parse(expr_sql), cslot.type, - ) - expr_sql = wrapped.sql(dialect=self.dialect) - step_parts.append(f'{expr_sql} AS {self._quote_ident(full_alias)}') + rendered = _wrap_cast_for_type(rendered, cslot.type) + step_parts.append(rendered.as_(full_alias, quoted=True)) aliases_by_slot_id.setdefault(cslot.id, []).append( full_alias, ) available_alias_by_slot_id.setdefault( cslot.id, full_alias, ) - step_sql = ( - "SELECT\n " - + _SQL_COL_SEP.join(step_parts) - + f"\nFROM {prev_cte}" - ) - ctes.append((step_name, step_sql)) + ctes.append(CteEntry( + name=step_name, + query=exp.Select().select(*step_parts).from_(prev_cte), + depends_on=[prev_cte], + )) + chain_tail = step_name # Inner SELECT inside _outer wrap: ALL carried aliases sorted - # (matches legacy _generate_with_computed:1607). - final_cte = ctes[-1][0] - inner_sorted = sorted( - a for aliases in aliases_by_slot_id.values() for a in aliases - ) - inner_sql = ( - "SELECT\n " - + _SQL_COL_SEP.join(self._quote_ident(a) for a in inner_sorted) - + f"\nFROM {final_cte}" - ) - - cte_clause = ( - _SQL_WITH - + ",\n".join(f"{name} AS (\n{sql}\n)" for name, sql in ctes) - ) - chain_sql = f"{cte_clause}\n{inner_sql}" + # in PLAN order (B8 — this list used to be sorted alphabetically to + # match the legacy renderer byte-for-byte). + final_cte = chain_tail + inner_aliases = self._carry_aliases_in_plan_order(aliases_by_slot_id) + inner_select = exp.Select().select( + *(exp.column(a, quoted=True) for a in inner_aliases), + ).from_(final_cte) + + chain_sql = assemble_with_chain( + entries=ctes, final=inner_select, + ).sql(dialect=self.dialect, pretty=True) # POST-phase filter wrap (filters referencing transform / arith # slots). Mirrors legacy _generate_with_computed:1627-1648 — @@ -1979,7 +1835,7 @@ def _generate_from_planned_impl( # NOSONAR(S3776) — top-level dispatch over c ) if post_filter_conditions: chain_sql = ( - f"SELECT *\nFROM (\n{chain_sql}\n) AS _filtered" + f"SELECT *\nFROM (\n{chain_sql}\n) AS {FILTERED_ALIAS}" f"\nWHERE {_SQL_AND_JOINER.join(post_filter_conditions)}" ) @@ -2053,7 +1909,7 @@ def _validate_window_transform_ops_for_7b10(*, planned_query) -> None: leaf_kinds = (ColumnKey, ColumnSqlKey, AggregateKey, TimeTruncKey) # Keep aligned with _emit_consecutive_periods_ctes_for_planned — - # the renderer dispatches arithmetic ops via _compose_arithmetic_op + # the renderer dispatches arithmetic ops via render_arithmetic # which supports these binary comparisons only. _COMPARISON_OPS = {"==", "!=", "<", "<=", ">", ">="} @@ -2426,6 +2282,7 @@ def _ready(key) -> bool: def _resolve_agg_inputs_via_scope( # NOSONAR(S3776) — one cohesive Law-1 discovery pass: three ordered sub-passes (Column.filter → source → kwargs) over the local aggregates via small closures sharing scope/resolved. Extracting them would scatter the ordered-registration contract that keeps the base FROM byte-identical. self, *, base_render_order, slots_by_id, scope: ScopeFrame, + skip_cross_model_aggs: bool = False, ) -> "Dict[Any, Dict[str, ResolvedAggKwarg]]": """Resolve every LOCAL aggregate's join-crossing inputs through the host ``scope`` (Law 1) — ``scope.resolve`` anchors each ref and registers the @@ -2436,10 +2293,11 @@ def _resolve_agg_inputs_via_scope( # NOSONAR(S3776) — one cohesive Law-1 disc resolver join-registration order (Column.filter → source → kwargs): 1. **``Column.filter`` predicates** (DEV-1494; replaces - ``_collect_column_filter_join_paths``). The Mode-A predicate is - dual-scanned via ``_filter_join_paths`` (raw + inline-expanded, so a - placeholder dotted ref that inlines to a constant still pulls its - join) and the paths registered into the scope. + ``_collect_column_filter_join_paths``). The Mode-A predicate enters + through the door (``ScopeFrame.enter_predicate``), whose dual-scan + (raw + inline-expanded, so a placeholder dotted ref that inlines to + a constant still pulls its join) registers the crossed paths into + the scope as a side effect. 2. **derived aggregate SOURCES** (``ColumnSqlKey`` whose ``Column.sql`` crosses a join — DEV-1502; replaces ``_collect_aggregate_source_ join_paths``). Discovery only; the render spec re-expands the source. @@ -2476,7 +2334,16 @@ def _resolve_agg_inputs_via_scope( # NOSONAR(S3776) — one cohesive Law-1 disc def _walk(key, fn) -> None: if isinstance(key, AggregateKey): - if not getattr(key.source, "path", ()): + # DEV-1747 D2 — a HOST-GRAIN aggregate reads through a join but + # is grouped at the host grain, so it renders INLINE here and + # its source join has to register like any other crossing input + # (Law 1). When the caller owns it in a ``_cm_*`` CTE + # (``skip_cross_model_aggs``) the join belongs to that CTE, not + # to this base — registering it here would add an unused, and + # for a one-to-many join cardinality-changing, LEFT JOIN. + if not getattr(key.source, "path", ()) or ( + _is_host_grain(key) and not skip_cross_model_aggs + ): fn(key) elif isinstance(key, ArithmeticKey): for o in key.operands: @@ -2495,15 +2362,24 @@ def _resolve_column_filter(key) -> None: cfk = key.column_filter_key if cfk is None or not cfk.canonical_sql: return - for p in self._filter_join_paths( - sql=cfk.canonical_sql, source_relation=scope.root_relation, - source_model=scope.root_model, bundle=scope.bundle, - ): - scope.join_paths.add(p) + # Entering registers the crossed joins on ``scope`` as a side + # effect (Law 1 / P-A); the AST itself is re-rendered later by the + # aggregate CASE-WHEN wrapper, so it is discarded here. + self._enter_mode_a_predicate( + sql=cfk.canonical_sql, scope=scope, + location=f"Column.filter on model {scope.root_model.name!r}", + ) def _resolve_source(key) -> None: - if isinstance(key.source, ColumnSqlKey): - scope.resolve(key.source) # register-only; render re-expands + # Two shapes, one action. A DERIVED source (``ColumnSqlKey``) may + # cross inside its ``Column.sql``; a PATH-BEARING one crosses by + # the path itself, which is what a host-grain aggregate's source + # does (DEV-1747 D2). Either way the scope only needs the + # register-only resolve — the render re-expands. + if isinstance(key.source, ColumnSqlKey) or getattr( + key.source, "path", (), + ): + scope.resolve(key.source) def _resolve_kwargs(key) -> None: kw: Dict[str, ResolvedAggKwarg] = {} @@ -2520,27 +2396,11 @@ def _resolve_fragment_kwargs(key) -> None: # aggregation template as qualified SQL text, so their crossed # joins must register exactly like ``Column.filter`` predicates # do (the widened Law-3 trigger isolates on them, and the CTE - # sub-render lands here). Scanned with the same - # ``_filter_join_paths`` pipeline; unparseable fragments - # contribute nothing. - fragments = [v for _, v in key.kwargs if isinstance(v, str)] - agg_def = next( - (a for a in (scope.root_model.aggregations or []) - if a.name == key.agg), - None, + # sub-render lands here). Shared with the ``_cm_*`` CTE path so the + # two cannot drift apart again (DEV-1745 W2). + self._register_fragment_kwarg_joins( + key=key, scope=scope, model=scope.root_model, ) - if agg_def is not None: - overridden = {name for name, _ in key.kwargs} - fragments.extend( - p.sql for p in (agg_def.params or []) - if p.name not in overridden and p.sql - ) - for frag in fragments: - for p in self._filter_join_paths( - sql=frag, source_relation=scope.root_relation, - source_model=scope.root_model, bundle=scope.bundle, - ): - scope.join_paths.add(p) def _resolve_first_last_time_arg(key) -> None: # DEV-1710 Stage 6 — a first/last explicit ranking-time arg @@ -2576,7 +2436,7 @@ def _resolve_agg_kwargs_for_key( The base SELECT uses the batch ``_resolve_agg_inputs_via_scope`` pass over a shared host scope (which also registers the crossed joins). The HAVING - render path (``_render_value_key_for_filter``) has no such scope, so it + render path (``render_value_key``) has no such scope, so it builds a throwaway one here purely to reproduce the SAME anchored kwarg expression the SELECT emits — the crossed join is already base-pulled (the HAVING aggregate is also a ``base_render_order`` slot), so the @@ -2683,6 +2543,7 @@ def _build_base_select_for_planned( # NOSONAR(S3776) — join-path collection a base_render_order=base_render_order, slots_by_id=slots_by_id, source_relation=source_relation, source_model=source_model, bundle=bundle, scope=host_scope, + order_slot_ids=[e.slot_id for e in planned_query.order], ) # WHERE-phase filters referencing joined columns (direct, derived, or # Mode-A ``__`` paths) register their joins into the scope too (position @@ -2706,6 +2567,7 @@ def _build_base_select_for_planned( # NOSONAR(S3776) — join-path collection a base_render_order=base_render_order, slots_by_id=slots_by_id, scope=host_scope, + skip_cross_model_aggs=skip_cross_model_aggs, ) # Merge the scope's registered paths (positions 2-7, in first-seen order) # after the dimension paths (position 1) → byte-identical FROM. @@ -2719,31 +2581,6 @@ def _build_base_select_for_planned( # NOSONAR(S3776) — join-path collection a bundle=bundle, ) - # DEV-1450: first/last AGGREGATIONS rank rows via a ROW_NUMBER - # subquery (mirrors legacy ``_generate_base`` + ``_build_last_ - # ranked_from``). - if self._has_first_last_aggregate( - base_render_order=base_render_order, slots_by_id=slots_by_id, - ): - # DEV-1503 — local first/last in ``_base`` alongside cross-model - # CTEs is supported: the ranked subquery wraps ``_base`` for the - # local measures; cross-model / filtered-local aggregates are - # deferred to their per-plan ``_cm_*`` CTEs (their slot ids are - # excluded from ``base_render_order`` by the caller, so - # ``_build_first_last_base_select`` never sees them and emits no - # dangling references). - return self._build_first_last_base_select( - planned_query=planned_query, - bundle=bundle, - source_model=source_model, - source_relation=source_relation, - base_render_order=base_render_order, - slots_by_id=slots_by_id, - from_clause=from_clause, - base_joins=base_joins, - skip_filter_ids=skip_filter_ids, - ) - select_columns: list[exp.Expression] = [] group_by_keys: Dict[str, exp.Expression] = {} has_aggregation = False @@ -2848,15 +2685,22 @@ def _record_alias(sid: str, full_alias: str) -> None: # (``amount:weighted_avg(weight=) + quantity:sum``) # embeds its expanded join-anchored expression instead of a # bare, non-existent name. - composite, any_agg = self._render_aggregate_composite_expr( + composite = render_value_key( key=key, - slot=slot, - source_model=source_model, - source_relation=source_relation, - bundle=bundle, - resolved_agg_kwargs=resolved_agg_kwargs, + ctx=RenderContext( + dialect=self._dialect, + composites=CompositeFacilities( + agg_builder=self._composite_agg_builder( + slot=slot, + source_model=source_model, + source_relation=source_relation, + bundle=bundle, + resolved_agg_kwargs=resolved_agg_kwargs, + ), + ), + ), ) - if any_agg: + if contains_aggregate(key): composite = _wrap_cast_for_type(composite, slot.type) has_aggregation = True select_columns.append(composite.copy().as_(full_alias)) @@ -2865,18 +2709,24 @@ def _record_alias(sid: str, full_alias: str) -> None: agg_path = getattr(key.source, "path", ()) if agg_path: if skip_cross_model_aggs: - # Cross-model aggregate; rendered by the per-plan - # ``_cm_*`` CTE. Skip in the host base. + # Owned by a per-plan ``_cm_*`` CTE — target-rooted or + # (DEV-1747 D2) host-rooted. Skip in the host base. continue - raise NotImplementedError( - f"DEV-1450 stage 7b.12: cross-model aggregate " - f"(source.path={agg_path!r}) reached the local " - f"base SELECT path. The cross-model orchestrator " - f"should have routed this through `_render_with_" - f"cross_model_plans`." - ) + if not _is_host_grain(key): + raise NotImplementedError( + f"DEV-1450 stage 7b.12: cross-model aggregate " + f"(source.path={agg_path!r}) reached the local " + f"base SELECT path. The cross-model orchestrator " + f"should have routed this through `_render_with_" + f"cross_model_plans`." + ) + # DEV-1747 D2 — a HOST-GRAIN aggregate inside its own CTE: + # the crossed join is already in this scope's FROM, so the + # aggregate renders inline over the joined relation and + # GROUPs at the query grain. This is the base-pull the + # recursion guard exists to reach. # DEV-1450 stage 7b.12: ``column_filter_key`` is now - # propagated into the synthetic EnrichedMeasure's + # propagated into the synthetic ``AggRenderSpec``'s # ``filter_sql`` field so ``_build_agg`` wraps the # aggregate as ``SUM(CASE WHEN THEN col END)``. synth = self._build_agg_render_spec_from_planned( @@ -2909,105 +2759,8 @@ def _record_alias(sid: str, full_alias: str) -> None: ) return ( base_select, aliases_by_slot_id, has_aggregation, group_by_keys, - False, None, ) - def _has_first_last_aggregate( - self, *, base_render_order: List[str], slots_by_id: Dict[str, Any], - ) -> bool: - """True if any LOCAL ``first`` / ``last`` AGGREGATE slot appears in - the base render order — directly as an ``AggregateKey`` slot OR - as an operand inside a composite (``ArithmeticKey`` / - ``ScalarCallKey``) aggregate slot. - - Cross-model first/last (non-empty ``source.path``) is excluded — - it is not rendered by the ranked-subquery path (each cross-model - aggregate has its own CTE). DEV-1501 (Codex round 4): composite- - only first/last (e.g. ``last(created_at) + last(updated_at)`` - with no direct sibling) must still trigger the ranked-subquery - path; without composite-aware detection the composite render - would emit ``MAX(CASE WHEN _last_rn = 1 …)`` referencing a - column the bare-FROM never projects. - """ - from slayer.core.keys import AggregateKey, Phase - - for sid in base_render_order: - slot = slots_by_id.get(sid) - if slot is None or slot.phase != Phase.AGGREGATE: - continue - key = slot.key - if ( - isinstance(key, AggregateKey) - and key.agg in ("first", "last") - and not getattr(key.source, "path", ()) - ): - return True - # Composite slot (no direct AggregateKey): walk for first/ - # last AggregateKey leaves. The composite render needs the - # ranked subquery so each operand's ``_first_rn`` / - # ``_last_rn{suffix}`` column exists. - if not isinstance(key, AggregateKey) and _iter_first_last_leaves(key): - return True - return False - - def _resolve_ranking_time_column_from_planned( - self, - *, - base_render_order: List[str], - slots_by_id: Dict[str, Any], - source_model, - source_relation: str, - bundle, - ) -> Optional[str]: - """Resolve the default ORDER-BY time column for first/last - ROW_NUMBER ranking (mirrors legacy ``_resolve_last_agg_time``). - - Precedence (matching legacy): the first ``DATE`` / ``TIMESTAMP`` - regular dimension, then the first time-dimension slot's raw column, - then the model's ``default_time_dimension``. Returns the qualified - SQL string (e.g. ``"orders.created_at"`` / ``"stores.opened_at"``), - or ``None`` when nothing temporal is in scope. - - (The legacy ``main_time_dimension`` short-circuit and the - filter-referenced-date fallback are corner cases the spec permits - diverging on; they are not reproduced here.) - """ - from slayer.core.keys import ColumnKey, Phase, TimeTruncKey - - for sid in base_render_order: - slot = slots_by_id[sid] - if slot.phase == Phase.ROW and isinstance(slot.key, ColumnKey): - model = source_model - for hop in slot.key.path: - nxt = bundle.get_referenced_model(hop) - if nxt is None: - model = None - break - model = nxt - if model is None: - continue - col_def = next( - (c for c in model.columns if c.name == slot.key.leaf), None, - ) - if col_def is not None and col_def.type in ( - DataType.DATE, DataType.TIMESTAMP, - ): - return self._joined_or_local_dim_expr( - path=slot.key.path, leaf=slot.key.leaf, - source_model=source_model, - source_relation=source_relation, bundle=bundle, - ).sql(dialect=self.dialect) - for sid in base_render_order: - slot = slots_by_id[sid] - if slot.phase == Phase.ROW and isinstance(slot.key, TimeTruncKey): - return self._raw_time_col_expr_for_planned( - time_column=slot.key.column, source_model=source_model, - source_relation=source_relation, bundle=bundle, - ).sql(dialect=self.dialect) - if source_model.default_time_dimension: - return f"{source_relation}.{source_model.default_time_dimension}" - return None - @staticmethod def _explicit_time_arg_of(key): """The explicit positional ranking-time arg of a ``first`` / ``last`` @@ -3015,9 +2768,9 @@ def _explicit_time_arg_of(key): The SINGLE arg-selection contract shared by the three sites that must never disagree on WHICH positional arg is the time column (DEV-1710 / - Codex F1): the raise-gate in ``_build_first_last_base_select``, the - join-discovery pass in ``_resolve_agg_inputs_via_scope``, and the render - seam ``_resolve_explicit_time_col``. Returns the FIRST positional arg + Codex F1): the ranked-plan builder in ``slayer/engine/ranked_planner.py``, + the join-discovery pass in ``_resolve_agg_inputs_via_scope``, and the + render seam ``_resolve_explicit_time_col``. Returns the FIRST positional arg iff it is a ``ColumnKey`` / ``ColumnSqlKey``; ``None`` for a non-first/last agg, empty args, or a first positional arg of any other type (first/last never takes a leading non-column positional). @@ -3121,848 +2874,50 @@ def _resolve_explicit_time_col( return f"{source_relation}.{col_sql}" return self._parse(col_sql).sql(dialect=self.dialect) - def _build_ranked_subquery_from_planned( # NOSONAR(S3776) — Group 2 already factored the per-spec ROW_NUMBER passes into _build_unfiltered_rn_columns / _build_filtered_rn_columns; what's left is exp.Select / from / joins / where assembly that has to live in one place. + def _composite_agg_builder( + self, *, slot, source_model, source_relation: str, bundle, + resolved_agg_kwargs, + ): + """The AGGREGATE-phase composite seam (DEV-1763 P-G): render one + aggregate LEAF of a composite inline via the same synth + ``_build_agg`` + path the single-aggregate branch uses. ``render_value_key`` owns the + composite STRUCTURE (arithmetic / scalar calls); only the aggregate leaf + needs the generator's spec builder + resolved column-ref kwargs. The + live base-SELECT site threads no rn-state (that is the dead first/last + path); the ``__op__`` placeholder alias is inert without it.""" + + def build(agg_key) -> exp.Expression: + if getattr(agg_key.source, "path", ()): + raise NotImplementedError( + "DEV-1450: cross-model aggregate operand inside an " + "AGGREGATE-phase composite is not yet supported; factor it " + "into a multi-stage source_queries model." + ) + synth = self._build_agg_render_spec_from_planned( + slot=slot, key=agg_key, source_model=source_model, + source_relation=source_relation, full_alias="__op__", + bundle=bundle, + resolved_agg_kwargs=(resolved_agg_kwargs or {}).get(agg_key), + ) + agg_expr, _is_agg = self._build_agg(synth) + return agg_expr + + return build + + def _render_window_measure_cte_from_planned( # NOSONAR(S3776) — one cohesive host-rooted range-join CTE build: ``_src`` projection (dims / other-time-dims / raw-window-time / value) with Law-1 join discovery, WHERE inheritance minus date_range, and the ``_base LEFT JOIN _src`` interval range join. Splitting scatters the shared scope / grain-alias / join-eq state. self, *, + plan, + agg_slot, + source_model, source_relation: str, - default_time_col_sql: str, - partition_exprs: List[exp.Expression], - extra_projections: List[Tuple[str, exp.Expression]], - synth_specs: List[AggRenderSpec], - from_clause: exp.Expression, - base_joins: List, - where_clause: Optional[exp.Expression], - ) -> Tuple[exp.Expression, dict, dict, dict]: - """Build the ROW_NUMBER-ranked subquery that wraps the source for - first/last aggregation (planned-native port of - ``_build_last_ranked_from``). - - Projects ``source_relation.*`` plus the supplied ``extra_projections`` - (truncated time dimensions / joined dimensions referenced by the - outer SELECT) plus one ``ROW_NUMBER`` column per distinct - (effective-time-column, agg) pair. Filtered first/last measures get - a dedicated ranking column (non-matching rows pushed to the bottom) - and a boolean match flag. WHERE is applied INSIDE so it filters raw - rows before ranking. Returns ``(subquery, rn_suffix_map, - filtered_rn_map, filtered_match_map)``. - """ - partition_clause = "" - if partition_exprs: - partition_clause = _SQL_PARTITION_BY + ", ".join( - p.sql(dialect=self.dialect) for p in partition_exprs - ) - - select_exprs: List[exp.Expression] = [ - exp.Column(this=exp.Star(), table=exp.to_identifier(source_relation)), - ] - for alias, e in extra_projections: - select_exprs.append(e.copy().as_(alias)) - - unfiltered_exprs, rn_suffix_map = self._build_unfiltered_rn_columns( - synth_specs=synth_specs, - default_time_col_sql=default_time_col_sql, - partition_clause=partition_clause, - ) - select_exprs.extend(unfiltered_exprs) - - filtered_exprs, filtered_rn_map, filtered_match_map = ( - self._build_filtered_rn_columns( - synth_specs=synth_specs, - default_time_col_sql=default_time_col_sql, - partition_clause=partition_clause, - ) - ) - select_exprs.extend(filtered_exprs) - - inner = exp.Select() - for e in select_exprs: - inner = inner.select(e) - inner = inner.from_(from_clause) - for join_expr, on_expr, join_type in base_joins: - inner = inner.join(join_expr, on=on_expr, join_type=join_type) - if where_clause is not None: - inner = inner.where(where_clause) - subquery = exp.Subquery( - this=inner, alias=exp.to_identifier(source_relation), - ) - return subquery, rn_suffix_map, filtered_rn_map, filtered_match_map - - def _build_unfiltered_rn_columns( - self, - *, - synth_specs: List[AggRenderSpec], - default_time_col_sql: str, - partition_clause: str, - ) -> Tuple[List[exp.Expression], Dict[str, str]]: - """One ``ROW_NUMBER`` projection per distinct effective time column - for the unfiltered ``first`` / ``last`` specs. - - Each unique effective time column gets a stable suffix in render - order (first sorted gets ``""``, then ``"_2"``, ...); the same - time column shared by both ``first`` and ``last`` produces two - projections (`_first_rn{suffix}` ASC, `_last_rn{suffix}` DESC). - Returns ``(rn_select_exprs, rn_suffix_map)``. - """ - time_col_agg_types: Dict[str, set] = {} - for m in synth_specs: - if m.aggregation in ("first", "last") and not m.filter_sql: - eff = m.time_column or default_time_col_sql - time_col_agg_types.setdefault(eff, set()).add(m.aggregation) - sorted_tcs = sorted(time_col_agg_types) - rn_suffix_map: Dict[str, str] = { - tc: ("" if i == 0 else f"_{i + 1}") - for i, tc in enumerate(sorted_tcs) - } - rn_exprs: List[exp.Expression] = [] - for tc in sorted_tcs: - suffix = rn_suffix_map[tc] - if "last" in time_col_agg_types[tc]: - rn_exprs.append( - self._parse( - f"ROW_NUMBER() OVER ({partition_clause} " - f"ORDER BY {tc} DESC)" - ).as_(f"_last_rn{suffix}") - ) - if "first" in time_col_agg_types[tc]: - rn_exprs.append( - self._parse( - f"ROW_NUMBER() OVER ({partition_clause} " - f"ORDER BY {tc} ASC)" - ).as_(f"_first_rn{suffix}") - ) - return rn_exprs, rn_suffix_map - - def _build_filtered_rn_columns( - self, - *, - synth_specs: List[AggRenderSpec], - default_time_col_sql: str, - partition_clause: str, - ) -> Tuple[List[exp.Expression], Dict[str, str], Dict[str, str]]: - """One dedicated ``ROW_NUMBER`` + match-flag projection per distinct - ``(filter, time, agg)`` triple for the filtered ``first`` / ``last`` - specs. - - Filtered first/last needs to push non-matching rows past the - winners; emits ``ROW_NUMBER() OVER (... ORDER BY CASE WHEN - THEN 0 ELSE 1 END,