diff --git a/.claude/skills/slayer-overview.md b/.claude/skills/slayer-overview.md index 353955ab..8ce11e97 100644 --- a/.claude/skills/slayer-overview.md +++ b/.claude/skills/slayer-overview.md @@ -8,8 +8,8 @@ SLayer is a lightweight, agent-first semantic layer. Instead of writing raw SQL, ## Architecture -- **SlayerQueryEngine** — central orchestrator. Its `_enrich()` method resolves a SlayerQuery + SlayerModel into an EnrichedQuery (fully resolved SQL expressions), then passes it to SQLGenerator for SQL generation -- **SQLGenerator** — takes an EnrichedQuery (not SlayerQuery) and converts it to SQL via sqlglot (dialect-aware: postgres, mysql, bigquery, etc.) +- **SlayerQueryEngine** — central orchestrator. It resolves a SlayerQuery + SlayerModel into a PlannedQuery (typed value keys interned into slots, each carrying its resolved expression, join path and phase) via `stage_planner.plan_query`, then passes it to SQLGenerator +- **SQLGenerator** — takes a PlannedQuery (not SlayerQuery) and converts it to SQL via sqlglot (dialect-aware: postgres, mysql, bigquery, etc.) - **SlayerSQLClient** — executes SQL via SQLAlchemy with retry logic and statement timeouts - **Storage** — YAML or SQLite backends for model and datasource configs - **Ingestion** — auto-generates models from DB schema with rollup-style FK joins (denormalized LEFT JOINs). It can be triggered manually (`slayer ingest`, `ingest_datasource_models`, `POST /ingest`) or **on every server boot** via `slayer serve --ingest-on-startup` / `slayer mcp --ingest-on-startup` (also `SLAYER_INGEST_ON_STARTUP=1`, or `create_app/create_mcp_server(ingest_on_startup=True)` programmatically). It is idempotent and continues on per-datasource failures. @@ -42,7 +42,7 @@ Search: `search` (three-channel: entity-overlap BM25 over memory tags + tantivy slayer/ core/ — DataType, SlayerModel, SlayerQuery, formula parser (formula.py), etc. sql/ — SQLGenerator, SlayerSQLClient - engine/ — SlayerQueryEngine, EnrichedQuery, auto-ingestion with rollup joins + engine/ — SlayerQueryEngine, PlannedQuery, auto-ingestion with rollup joins storage/ — YAMLStorage, SQLiteStorage, StorageBackend protocol api/ — FastAPI server mcp/ — MCP server (FastMCP) diff --git a/.claude/skills/slayer-query.md b/.claude/skills/slayer-query.md index b6950970..4bab5cd1 100644 --- a/.claude/skills/slayer-query.md +++ b/.claude/skills/slayer-query.md @@ -22,6 +22,8 @@ A `SlayerQuery` is a JSON/dict object. The same shape works across the REST API, `order[].column` is the short alias (`count`, `revenue_sum`) — not the colon form. +**Ordering by something you don't project.** `order` may name an undeclared column/aggregate/expression ("top-N by X, show only Y, Z"). Computed hidden, sorted on, and stripped from the result: an **aggregate** (`amount:sum`, `customers.revenue:sum`), an inline **transform** (`rank(amount:sum)`, `change(...)`, `cumsum`/`lag`/`lead`/`ntile`), an inline **composite** (`revenue:sum / cnt:sum`, `abs(amount:sum)`), and a **windowed** aggregate (`amount:sum(window='90d')`, alone or inside a composite). A **raw row column** is orderable only in a raw-rows query (`distinct_dimension_values: false`); in a grouped/dedup query it's rejected (HTTP 400 — add it to `dimensions` or order by an aggregate of it). A **joined** row column is rejected — project it. Order expressions must use formula syntax for their operands, not the `name`s of measures declared in the same query: `{"column": "revenue:sum / cnt:sum"}` works, `{"column": "rev / cnt"}` is rejected. + **Dim-only queries deduplicate.** A query with no measures and at least one dimension or time-dimension auto-emits `GROUP BY ` and returns the distinct combinations. The `GROUP BY` is applied before `LIMIT`, so a row cap can't silently drop unique tuples. To opt out, set `"distinct_dimension_values": false` on the query — emits raw rows (no top-level `GROUP BY`), with WHERE / ORDER BY / LIMIT applied as usual. Any measure reference in `measures` / `filters` / `order` raises `DistinctDimensionValuesError` in this mode. ## Measures — colon aggregation @@ -46,7 +48,7 @@ Each entry in `measures` is either a bare formula string or a `{"formula": ..., ] ``` -Built-in aggregations: `sum`, `avg`, `min`, `max`, `count`, `count_distinct`, `count_distinct_approx`, `first`, `last`, `weighted_avg`, `median`, `percentile`, `stddev_samp`, `stddev_pop`, `var_samp`, `var_pop`, `corr`, `covar_samp`, `covar_pop`. `count_distinct_approx` is dialect-aware (native approximate-distinct where available, exact `COUNT(DISTINCT)` fallback otherwise). Two-column `corr`/`covar_samp`/`covar_pop` take the second column as a named param: `price:corr(other=quantity)`. `sum` and `avg` accept an optional trailing-window: `revenue:sum(window='30d')`. +Built-in aggregations: `sum`, `avg`, `min`, `max`, `count`, `count_distinct`, `count_distinct_approx`, `first`, `last`, `weighted_avg`, `median`, `percentile`, `stddev_samp`, `stddev_pop`, `var_samp`, `var_pop`, `corr`, `covar_samp`, `covar_pop`. `count_distinct_approx` is dialect-aware (native approximate-distinct where available, exact `COUNT(DISTINCT)` fallback otherwise). Two-column `corr`/`covar_samp`/`covar_pop` take the second column as a named param: `price:corr(other=quantity)`. `sum` and `avg` accept an optional trailing-window: `revenue:sum(window='30d')`. A time bound narrows which buckets come back, not which rows the window may reach — so `date_range` and an equivalent explicit filter (`created_at >= '2025-01-01'`) give identical windowed numbers. Only `<`/`<=`/`>`/`>=` against a time dimension's own column and a literal counts; other operators, non-time-dimension columns, bounds under `or`/`not`, and model-level `filters` all restrict the window's input as usual. Same rule for `time_shift`. For month-over-month / period-over-period growth use `change_pct(x)` (absolute delta: `change(x)`) — both are calendar-aware and partition-safe (the underlying self-join matches on all non-time dimensions, so per-group series reset cleanly). Reach for `time_shift` only when you need the shifted value itself as a term in custom arithmetic or at a different grain (`time_shift(revenue:sum, -1, 'year')` for year-over-year). diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 26af9d5c..fe093aee 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -7,7 +7,10 @@ on: push: branches: [main] pull_request: - branches: [main] + # Run CI for every PR regardless of base branch — stacked PRs + # targeting feature branches (e.g. `egor/dev-1450-…`) need the + # same lint + test gate as PRs to main. + branches: ['**'] jobs: lint-and-test: diff --git a/DECISIONS.md b/DECISIONS.md index 496c8b6f..fca4666b 100644 --- a/DECISIONS.md +++ b/DECISIONS.md @@ -64,7 +64,33 @@ implementation detail. Include issue refs when known. - 2026-07-20 — Demo datasource ships curated semantic enrichment (labels, formats, saved measures), additive-only and idempotent so user edits survive re-runs; jafgen invoked via `sys.executable` so the demo works under pipx/uv. - 2026-07-22 — Case-colliding ids rejected in the YAML backend only (#249): ids are filenames there and case variants alias on macOS/Windows; SQLite keeps case variants as distinct rows (so such stores can't export to YAML without renames); backends opt in via the `_ids_collide_as_filenames` flag + shared collision helpers; case-variant reads/deletes are exact-directory-entry checked (not-found / no-op, never the wrong file). - 2026-07-31 — RLS policy restructured around one required `ruleset` (DEV-1718 / #260), superseding the `data_filters` rule list of DEV-1578/DEV-1627: a `SessionPolicy` carries exactly one `ColumnFilterRuleset` or `JoinFilterRuleset`, discriminated on an explicit `kind` (no inference — a kind-less dict fails to discriminate), and no-filtering is `policy=None` rather than an empty policy. The join ruleset hoists the tenant anchor (`table`/`column`/`value`) up from the rules, so nested rules carry only `target_table` + `join_path` and classification becomes fully structural and DB-free (`has_column` is never probed); an explicit `whitelist` replaces the mandatory blocking-column backstop, so a table that is neither the anchor, a join target, nor whitelisted fails closed. Join paths may be written from either endpoint and are normalized target-first, so the correlated EXISTS always lands its terminal predicate on the anchor. +- 2026-08-01 — Scope-closure validator (DEV-1705, DEV-1703 Stage 1): `slayer/sql/scope_check.py::assert_scope_closed` walks every sqlglot scope and flags a *provable* out-of-scope reference — a table qualifier not bound in that scope's FROM/JOINs (C1) or a cross-scope reference naming a column an inner scope does not project (C2, with a plain/`REPLACE` star exporting every name and `* EXCEPT (c)` dropping `c`). Deliberately sound-on-corpus: unqualified/ambiguous refs and physical-table column names are unverifiable and never flagged (zero false positives), so it can gate every currently-passing statement. Runs on **post-mangle, pre-RLS** generator output: dialect alias mangling (`.`→`___`) runs first, so BigQuery/T-SQL names carry no dotted output columns — pre-mangle those dotted refs parse as `table.column` (false unbound leaks) and trigger BigQuery's `TypeError`; mangling is identity for non-mangling dialects. RLS's correlated `_rls_src` EXISTS is applied downstream by the engine and whitelisted only via `allow_rls_correlation=True`. The generator terminals call `maybe_validate_scopes` env-gated by `SLAYER_VALIDATE_SCOPES`; the test harness sets it suite-wide (autouse) so a scope leak fails at generation time. A validator misfire is a validator bug to fix, never silenced. BigQuery `TypeError` on parse is a bounded, reported skip owned by Stage 9 (DEV-1713). +- 2026-08-02 — DEV-1703 Stage 2 (DEV-1706): `ScopeFrame` + single resolver + minimal `AliasAllocator` (`slayer/sql/scope.py`, `slayer/sql/naming.py`). `ScopeFrame.resolve(ref, consumer=None)` anchors a ref at the scope root or a `__`-path alias and REGISTERS every join it crosses into the scope's ordered `join_paths` in the same call (Law 1 — discovery is a side effect of rendering, never a separate step), with the Law-2 materialise branch built and unit-proven ahead of its first e2e consumer (Stage 4). The host base SELECT builds a host `ScopeFrame` and resolves column-ref aggregation kwargs (`weighted_avg(weight=)`, `corr(other=)`) through it: the resolve base-pulls the crossed LEFT JOIN and the resolved expression is embedded verbatim as a typed `ResolvedAggKwarg(kind="expr")`, deleting the `agg_kwarg_canonical_str` emission round-trip that collapsed a derived kwarg to a bare (non-existent) column name (DEV-1527 local half; cross-model remainder is Stage 4). Pulled forward because they are hard dependencies of the reserved-word fix landing here: DEV-1686 FROM/JOIN alias quoting (Identifier-node aliases + `prequote_reserved_identifiers` at every `_parse`/`_parse_predicate` re-parse) and DEV-1645 mixed-case *identifier* quoting (Flavor B — `_quote_mixed_case_identifiers` / `_to_ident` / `_to_table`); DEV-1645 ORDER-BY placement policies remain Stage 8. DEV-1539 predicate outer-parens land as a comparison-operand wrap in `_build_arithmetic_for_filter` (multi-term operands parenthesised) — derived-column filters were already precedence-safe via their type CAST. Of the four host-base join collectors, the two AGGREGATE-phase ones are folded into the resolver here — `_collect_aggregate_source_join_paths` (DEV-1502 derived sources) and `_collect_column_filter_join_paths` (DEV-1494 Column.filter) now register through the host scope in `_resolve_agg_inputs_via_scope`. The two ROW-phase ones (`_collect_filter_join_paths` WHERE-filter ValueKey trees + the join-collection half of `_expand_derived_row_dims`) are an order-coupled pair (derived-dims must register before WHERE-filters) and fold in together in Stage 4 (DEV-1708), where filter routing through the scope is reworked; the host-rooted placeholder's LIMIT-1 filter discovery (D-J) moves with them. +- 2026-08-02 — Unified cross-model reroot (DEV-1707, DEV-1703 Stage 3): one pure `slayer.core.keys.reroot_aggregate_key(key, *, target_path)` re-anchors ALL embedded references of an `AggregateKey` (source, positional args, kwarg values, `column_filter_key`) symmetrically when a cross-model aggregate renders in its target scope — replacing the three scattered per-field strip implementations (`_local_agg_formula`/`_reroot_col_kwarg` in `cross_model_planner.py`; the inline `_reroot_kwarg`/`local_args` and `_reroot_having` blocks in `generator.py`) that had diverged into two semantics. Unified on the planner's prefix-strip-with-residual (`('customers','regions')` under target `('customers',)` → `('regions',)`; exact match → local); the generator's old exact-match is subsumed. Non-matching paths pass through unchanged (function is total, never raises). `column_filter_key` is owner-anchored (its `canonical_sql`/`referenced_join_paths` are relative to the model owning the filtered column) so it is invariant under reroot and copied through unchanged — a rerooted filtered cross-model aggregate still reads local-source + non-empty filter paths, matching the DEV-1503 trigger. Closes DEV-1476 (c) (args now strip symmetrically with kwargs) and (d-cross) (path-bearing `ColumnSqlKey` time args become target-local before `_resolve_explicit_time_col`). A time arg left with a *residual* path (a hop past the target) stays a loud gap: the derived-column case raises `NotImplementedError` pointing at DEV-1526 (Stage 4 — the isolated CTE does not yet pull the deeper join), the bare-column case is caught by the `SLAYER_VALIDATE_SCOPES` scope-closure validator. +- 2026-08-02 — Cross-model / isolation CTE renderer on `ScopeFrame` (DEV-1708, DEV-1703 Stage 4). The forward `_cm_*` CTE renderer (`generator.py::_render_cross_model_cte`) and its routed-filter renderer now build a per-CTE `ScopeFrame` rooted at the target relation and route every expression through it (Law 1): the rerooted aggregate source, positional args, column-ref kwargs, `Column.filter`, shared-grain dimensions, target-model filters, and routed host WHERE/HAVING filters all `resolve()`/register their crossed joins into the CTE's single ordered `join_paths` set — the ad-hoc `_add_cte_join_paths` closure is deleted and the CTE FROM is built from that set. Closes **DEV-1526** (source `Column.sql` crossing a further join), **DEV-1527** cross-model remainder (a column-ref kwarg naming a derived target column now expands through the scope and is embedded as a typed `ResolvedAggKwarg(kind="expr")`, not a bare non-existent column), and the WHERE/HAVING routed-filter derived-ref gap. A routed-filter **pre-pass** (`_register_routed_filter_joins`) walks the full `ValueKey` tree (nested arithmetic/boolean/IN operands + aggregate leaves' source/args/kwargs/`column_filter`) before the FROM is built, so a HAVING (rendered later, after the ranked-subquery rn maps exist) still contributes its joins. **Law 2** (DEV-1702 B2, forward variant): when the CTE contains a first/last ranked subquery whose SOURCE value crosses a join, the crossing value is materialised as a `_val_` projection INSIDE the subquery (the outer `MAX(CASE WHEN _last_rn = 1 THEN … END)` would otherwise reference a table bound only inside the subquery); the routed-HAVING variant binds the SAME alias via `FirstLastRenderState.value_alias_by_sql`. A generation-wide `AliasAllocator` is installed by `generate_from_planned` (save/restore) so inline forward CTEs and the host base share `_val_` naming; recursive rerooted sub-generations get their own. **DEV-1701** host-side fix: a joined derived TIME dimension whose `Column.sql` crosses a further join expands with `is_root=False` (host-path alias `customers_v2__regions`, not bare `regions`) in the shared `_raw_time_col_expr_for_planned`, fixing host base and CTE from one helper — deliberately touching host-base rendering (nominally out of the issue's scope) because the e2e query is otherwise invalid and the suite-wide scope validator rejects it. **Null-safe grain join-back** (Codex F2): the combined-SELECT `LEFT JOIN _cm_* ON` uses `SqlDialect.build_null_safe_eq` — base `NullSafeEQ` → `IS NOT DISTINCT FROM` (Postgres/DuckDB/Snowflake/BigQuery/Trino/Presto/Databricks/Spark/ClickHouse), MySQL `<=>`, SQLite `IS` override (native form needs ≥3.39), and an expanded `a = b OR (a IS NULL AND b IS NULL)` for T-SQL/Oracle/Redshift — so NULL dimension values and nullable truncated time grains join back instead of dropping. **Decision (user-approved):** a PLAIN derived (non-time) dimension used as cross-model shared grain now raises `NotImplementedError` (planner, gated on `not hidden` so filter-only derived refs are unaffected) instead of silently CROSS-JOIN-broadcasting the global aggregate across groups — full support rides with DEV-1495-b1 (Stage 8/9). Out of scope, deferred to Stage 5 (DEV-1531/1709): the DEV-1702-B2 filtered-local (host-rooted) variant, whose value materialisation lives in `_build_first_last_base_select`. +- 2026-08-02 — first/last explicit-time completion (DEV-1710, DEV-1703 Stage 6): a first/last explicit ranking-time arg (`amount:last(customers.signup_at)`) now discovers its crossed join through the host `ScopeFrame` like every other input, not through a bespoke collector. Three sites are unified on one arg-selection contract, `SQLGenerator._explicit_time_arg_of(key)` (first positional arg iff it is a `ColumnKey`/`ColumnSqlKey`, else `None` — first/last never takes a leading non-column positional): the raise-gate in `_build_first_last_base_select` (was a divergent `any(isinstance(a,(ColumnKey,ColumnSqlKey)))` scan-all-args, so a scalar-first/col-later shape slipped the "requires a ranking time column" raise and then crashed building the `ROW_NUMBER` map — Codex F1), the new discovery sub-pass, and the render seam. `_resolve_agg_inputs_via_scope` gains sub-pass 4 (position 7): for each LOCAL first/last it `scope.resolve(arg)` register-only, so the crossed LEFT JOIN base-pulls as a Law-1 side effect (bare single-hop, bare multi-hop — every prefix registered — and local derived args whose `Column.sql` reaches a joined table); a path-bearing `ColumnSqlKey` (the DEV-1526 residual) is skipped here, not anchored. `_resolve_explicit_time_col` renders through a throwaway host-rooted `ScopeFrame` when a bundle is present (its `join_paths` discarded — discovery is owned by the base pass; same throwaway pattern as `_resolve_agg_kwargs_for_key`), which also gains DEV-1686 reserved-word qualifier quoting for free; the early returns/raises (None for no explicit arg, `NotImplementedError` DEV-1526 before any resolve, `ValueError` for a not-found derived column before any resolve) are preserved in order, and the pre-existing `bundle=None` fallback (bare-ident f-string / verbatim emit) stays verbatim because the `_build_agg_render_spec_from_planned` unit pins and the two direct-call guard tests invoke it bundle-less. Safe because `synth.time_column` is re-parsed downstream (interpolated into a `ROW_NUMBER() OVER (... ORDER BY {tc} ...)` string that `_parse` re-emits), so resolver normalization (`date()`→`DATE()`) washes out. The `Phase.AGGREGATE` arm of `_collect_joined_paths_for_base` is deleted and its signature narrowed to `(base_render_order, slots_by_id)` — it now collects only ROW dimension paths. Closes DEV-1476 fully (all four acceptance reprose green: local no-TD, cross-model bare, cross-model derived, plus the Stage-A local cases). Deliberately NOT done under Option A: the residual-hop `NotImplementedError` is kept (narrowed, owned by DEV-1526/Stage 4) rather than removed, and Stage 5 arg-isolation is not pulled forward. Out of scope and flagged separately: a derived time column whose `Column.sql` references ANOTHER derived column (nested inlining) — `expand_derived_refs_sync` does not recursively inline bare sibling-derived refs, and this fails identically for a plain dimension, so it is a general column-expansion limitation, not a time-arg concern. +- 2026-08-03 — Widened Law-3 isolation trigger (DEV-1709, DEV-1703 Stage 5): a LOCAL aggregate isolates into a host-rooted `_cm_*` CTE when ANY explicit input crosses a join — source `Column.sql` (dotted / `__` / sibling derived chains), `Column.filter` (the pre-existing DEV-1503 half), positional args incl. the explicit first/last time arg (D2), and kwargs (column refs, user template-fragment strings, and non-overridden model-default `AggregationParam.sql` fragments — an unparseable fragment contributes nothing, parity with the filter scan's fallback, a documented D1 carve-out). Non-filter kinds are computed plan-time by `slayer/engine/aggregate_input_paths.py::compute_aggregate_input_join_paths` (crossing info recomputed from the bundle, never cached on keys — DEV-1703 Q3); the recursion flag is renamed `disable_dev1503_isolation` → `disable_host_rooted_isolation` and gates ONLY the host-rooted half. Headline consequence: the top-level host base only ever contains purely-local aggregates, so a measure-pulled 1:N join can no longer multiply the rows sibling measures see (sibling protection — pinned by executed DuckDB values); the crossing measure itself keeps multiply-per-match semantics inside its CTE (F1). Aggregate-phase filters referencing a newly-isolated aggregate route to the combined-SELECT outer WHERE (never HAVING-into-the-CTE); host ROW filters (local and pathed) propagate into the host-rooted sub-plan (F4); composite crossing leaves isolate individually per interned `AggregateKey` slot (identical keys share one CTE, distinct keys get distinct CTEs — merging is DEV-1688/`may_inline` territory). Inside the CTE's sub-render, `_build_first_last_base_select` gains the Law-2 materialisation (closing DEV-1531 and DEV-1702-B1): every crossing input expression the ranked outer scope consumes — aggregate SOURCE and column-ref KWARG values (also closing the DEV-1527/DEV-1476 first/last kwarg deferral) — is projected inside the ranked subquery as a `_val_` whose body is the RESOLVED value (qualified + `Column.type` inner CAST for non-bare expressions, so `SUM(CAST(x AS t))` semantics survive materialisation and same-sql-different-type aggregates keep distinct `_val`s); alias maps are keyed by that resolved text end-to-end (host path, Stage-4 CTE path, HAVING/composite consumers). `_validate_aggregate_kwarg_paths` is relaxed for LOCAL sources (structurally-crossing kwargs are now supported inputs; the cross-model path-mismatch rejection survives). Deferred with a strict-xfail + follow-up ticket: an IMPLICITLY-resolved crossing time column (model `default_time_dimension` pointing at a crossing derived column) does not trigger — D2 covers explicit args only, and plan-time duplication of the render-time time-fallback resolution was judged not worth the drift risk. +- 2026-08-03 — time_shift CTEs on `ScopeFrame` (DEV-1711, DEV-1703 Stage 7): the shifted CTE resolves every partition key and the shift-axis time expression through a per-slot `ScopeFrame`, so its FROM pulls exactly the joins they cross. This closes **DEV-1474** (cross-model partitions like `change(order_total:sum)` by `stores.name`) and makes the sjoin grain uniformly *every projected dimension* — joined, derived, and secondary-time alike — joined back **null-safe** so NULL dimension / NULL time-bucket groups keep their prior-period value instead of dropping. A joined-column ROW filter now pulls its join into the shifted CTE instead of raising. +- 2026-08-03 — Full naming module (DEV-1713, DEV-1703 Stage 9): `slayer/sql/naming.py` becomes the single owner of every alias / result-key decision — `result_key()` (dotted FINAL-stage keys, hops via `path`, dot-free `leaf`), `result_key_from_alias()` (an already-canonical relative alias that may embed hop dots, e.g. a cross-model measure `customers.revenue_sum`), `flat_name()` (the `__`-joined INNER-stage downstream bind names), the relocated BigQuery/T-SQL `encode_alias`/`decode_alias` mangling bijection (was `slayer/sql/dialects/_alias_mangle.py`, now deleted), the relocated DEV-1645 mixed-case identifier-quoting policy (`quote_mixed_case_identifiers`/`maybe_quote_ident`, generator keeps thin delegators), and the DEV-1692 `assert_unique_cte_names` per-`WITH`-scope collision belt. The legacy flatteners (`_alias_to_short`, `_alias_to_short_local`, `_flatten_dotted`, `_cte_name_from_alias`, the stage-wrapper strip) all delegate to `flat_name` (byte-identical) so the two forms can't drift while the legacy stack lives (deleted in Stage 11). +- 2026-08-03 — D3 dotted joined-dimension result keys (DEV-1495 bug 1, DEV-1713): a joined DERIVED dimension (`ColumnSqlKey` with a non-empty path) now projects and returns under the DOTTED key `orders.customers.rev_x2` — matching cross-model measures and the documented result-key contract — not the flat `orders.customers__revenue`. The generator's `_full_alias_for_slot` and `response_meta._slot_result_keys` both route the three ROW key shapes (`ColumnKey`/`ColumnSqlKey`/`TimeTruncKey`) through `result_key`, so the SQL alias and the response key cannot diverge. The planner's flat `declared_name`/`StageColumn.name` (the downstream-bind contract) is untouched; only FINAL-stage public keys change. **Deliberate breaking change** for any consumer adapted to the buggy flat form. ORDER BY on a projected joined dimension follows the same dotted alias (plain-path resolver routed through `_full_alias_for_slot`) so the sort key names a real projected column. +- 2026-08-03 — Bare named-measure aliasing (DEV-1713): a query measure that is a bare identifier resolving to a saved `ModelMeasure` surfaces under the measure NAME (`orders.rev_total`), not the formula-derived canonical (`orders.revenue_sum`) that `expand_model_measures` would otherwise leave; explicit query `name` still wins, and the canonical alias is retained for DEV-1443 colon-form filter/ORDER-BY resolution. A self-qualified reference (`orders.rev_total`) normalizes to the bare form (`strip_source_model_prefix`) and behaves identically. +- 2026-08-03 — DEV-1692 time_shift de-collision (DEV-1713): the fix has two halves, both in the TYPED pipeline's `_emit_time_shift_ctes_for_planned` (the hoisted placeholder name `_time_shift_inner` repeats across arithmetic-wrapped shifts). A per-generation `AliasAllocator` (reserve every deterministic CTE name, allocate the `shifted_`/`sjoin_`/`step`/`cp_` families around them) makes CTE NAMES unique, AND the hidden slot's projected value alias is allocated uniquely (`_time_shift_inner`, `_time_shift_inner_2`, …) so downstream arithmetic (resolved by slot id) reads each shift's own value instead of collapsing both onto the first — the value corruption the duplicate CTE name had masked. User-facing (`public_aliases`) shift aliases are already unique and left untouched. +- 2026-08-03 — BigQuery scope-validator carve-out removed (DEV-1713, closing the DEV-1705 inherited item): with BigQuery naming/mangling finalized, no real BigQuery generator output makes sqlglot raise `TypeError` on parse (the dotted-alias shapes are collapsed to `___` before validation, and the stale calendar-time_shift INTERVAL round-trip bug no longer reproduces). The `_SQLGLOT_TYPEERROR_DIALECTS` skip-set is deleted from `scope_check.py` and all three test harnesses; BigQuery output is now scope-validated and CTE-collision-checked like every other dialect, and a parse `TypeError` propagates for every dialect with no exception. Verified empirically by running the full BigQuery test surface with the skip-set emptied — only the self-referential monkeypatch test (which forced a TypeError) reacted; zero real residual, so no follow-up ticket. +- 2026-08-03 — CTE-name collision detection stays case-sensitive for now (DEV-1713 Codex review, deferred to DEV-1726): `AliasAllocator` / `assert_unique_cte_names` compare CTE names by exact string, but generated CTE names are emitted unquoted and case-fold on Postgres/Snowflake/Redshift — so two user measure aliases differing only in case, both generating CTEs, can still collide there. Deferred rather than fixed in Stage 9 because it is pre-existing (pre-DEV-1713 the names weren't deduped at all), an edge case, and the correct fix is dialect-aware (fold only for case-folding dialects; a blanket case-insensitive dedup would wrongly merge genuinely-distinct names on case-sensitive BigQuery / quoted SQLite). Tracked in DEV-1726. - 2026-08-03 — Mode-A `{variable}` substitution (DEV-1625 / #270): the `{var}` mechanism now fills a query's DIRECT source model's four raw-SQL surfaces — `SlayerModel.sql`/`.filters`, `Column.sql`/`.filter` — not just query-level filters; the primitive for parameterizing hand-written model SQL (e.g. Cube `FILTER_PARAMS`). `substitute_variables(..., escape="sql"|"python")` picks the escaping regime by layer: SQL quote-doubling for sqlglot-parsed Mode-A, Python backslash-escaping for the Python-AST Mode-B filters (SQL doubling would silently corrupt a Mode-B value via adjacent-literal concatenation). Contract: raise-on-missing once any variable is in play; a fully variable-free execution leaves braces as literals so raw brace literals (`'{1,2,3}'`) survive. Values are trusted input; the escaping is not dialect-aware (MySQL backslash caveat). Nested source_queries stages / join-target / cross-model-target lineages are deferred (DEV-1678); dialect-aware + control-char escaping deferred (DEV-1727). +- 2026-08-03 — Merge resolution, Stage 4 × Stage 9 (DEV-1708 × DEV-1713): the user-approved derived-shared-grain raise (DEV-1708) wins over Stage 9's combined test vehicle — D3's dotted final-stage keys do NOT by themselves make a plain derived joined dim legal as cross-model shared grain (the CTE join-back rendering is still unbuilt; full support remains DEV-1495). The two Stage-9 Codex-F6 alias/key-agreement tests split their derived-dim and cross-model-aggregate coverage into separate queries. +- 2026-08-03 — Dialect-aware CTE-name case-folding (DEV-1726): `AliasAllocator` gains `folds_case` (resolved via `naming.dialect_folds_case`, threaded by the single `SQLGenerator._new_allocator` factory — the only construction site, test-pinned) and `assert_unique_cte_names` folds per dialect. Comparison-only folding with `str.lower()` (not `casefold`; sqlglot `normalize_identifier` parity) — allocated names keep original case, so output changes only when a genuine fold-collision forces a `_2` walk. Fold set = every registry dialect EXCEPT ClickHouse; unknown strings stay exact. Issue-text corrections (GoogleSQL docs + sqlglot + empirics): BigQuery FOLDS (CTE names are query aliases, case-insensitive — only real table names are CS) and SQLite/DuckDB fold even quoted names; MySQL/T-SQL fold deliberately despite config-dependence (folding is rename-only-safe, not folding leaves the bug live on majority configs). The belt folds regardless of quoting — over-strict by design on allocator-sanitized output, never a general SQL validator. Public result keys are reserved, never allocator-minted, so folding can never rename them. +- 2026-08-03 — Order-only hidden slots + plan-time order/partition validations (DEV-1712, DEV-1703 Stage 8). An ORDER BY ref that is not a declared dimension/measure is classified at **plan time** in `stage_planner.plan_query` (right after `_bucket_slots`): an **aggregate** (local or cross-model) always materialises hidden and orders — never rejected; a **local row column** is allowed only in a raw-rows query (`distinct_dimension_values=False`, no grouping) and emits a SPLIT `orders.` reference in the generator's `_apply_order_limit_from_planned` (the old `NotImplementedError` becomes a defensive internal assertion — the plan-time pass guarantees only that shape reaches it); a **grouped** local row column raises `ValueError` (not in GROUP BY — add it to dims or order by an aggregate of it); a **joined** row column raises `UnresolvableOrderColumnError`; an inline **transform/composite** (`change(amount:sum)`, only reachable as `raw_formula` — composite arithmetic is unexpressible via `OrderItem.column`, Pydantic rejects it) raises `ValueError` pointing at "declare it as a measure" — full support deferred to **DEV-1733** with strict-xfail future tests + a matching worktree. The grouping predicate is planner-semantic: `bool(agg_slots) or (dims/tds present and distinct_dimension_values)` — a hidden order aggregate that induces grouping counts, so a row column in that same query is correctly rejected. **Hidden cross-model aggregate trim** (DEV-1495 bug 2): an order-only CMA gets `hidden=True`/`public_alias=None` from the planner; the generator's combined-projection loop skips it (`public_aliases=[]`) **only when there is no transform chain** — a hidden CMA feeding a `cumsum(...)` step must stay projected for the step CTE to consume (the transform outer-wrap does the public-vs-hidden trim there). Its ORDER BY term is CTE-qualified (`_cm_*.""`) since the bare alias is no longer projected. The DEV-1495-bug-1 malformed-alias half (`orders.customers._sum`) was already fixed by Stage 9's naming module — only the projection leak remained. **partition_by grain guard** (DEV-1497): a pre-intern pass (`planning.rewrite_rank_partition_keys`, mirroring `lower_sugar_transforms`' identity-preserving rebuild) validates every rank-family (`rank`/`dense_rank`/`percent_rank`/`ntile`) `partition_by` key resolves to a query dimension/time-dimension by **exact ValueKey membership** — the typed binder resolves `partition_by` to a ValueKey before validation, so the legacy string-matching ambiguity can't arise. A time-dimension **source column** is rewritten to its `TimeTruncKey` so `PARTITION BY` uses the truncated bucket (not the raw timestamp, which had silently widened the GROUP BY grain and emitted a duplicate alias); a non-dimension raises `ValueError` naming the transform + column + available dims (the legacy `enrichment._resolve_rank_partition` message, restored). The 7 DEV-1645 Flavor-A ORDER-BY unit pins (split-not-composite for unprojected sort keys; `UnresolvableOrderColumnError` for joined sort keys) were made green by porting main's legacy `_OrderColRef`/`_order_split_sql`/`_resolve_order_column` fix (lost in the Stage-0 merge) into the legacy generator — throwaway parity, deleted with the legacy stack in Stage 11 — so `tests/parity_xfails.py` is now empty (the DEV-1485 end-state). Deliberate divergence from main: the typed pipeline rejects a grouped raw-row order at plan time (HTTP 400) instead of emitting SQL the database rejects at execution. A follow-up review sweep (PR #274) extended the same joined-ORDER-BY host-local guard into the legacy `_resolve_order_column` and rejected an unprojected sort key in the CTE-wrapped `_apply_pagination_to_sql`; added a partition ambiguity guard for a time column carried at two granularities. +- 2026-08-04 — Duration-windowed measures on the typed pipeline (DEV-1714, DEV-1703 Stage 10): `revenue:sum(window='90d')` is reimplemented as a plan-time `WindowedAggregatePlan` (symmetric with `CrossModelAggregatePlan`) plus a host-rooted `_wm___` range-join CTE rendered as a `ScopeFrame` client, closing DEV-1496 (all pinned strict-xfails promote). The `window` kwarg is a globally reserved aggregation-kwarg name (legacy parity — enrichment pops it unconditionally) and triggers the plan. The CTE's `_src` subquery self-selects host rows — dims → `_w_dim_`, other time dims date-trunc'd → `_w_td_` (grain-preserving), the raw window time column → `_w_time`, the value → `_w_value` (CASE-wrapped by `Column.filter`) — discovers its joins through a host `ScopeFrame` (Law 1, replacing the legacy regex `_window_referenced_aliases` scanner: isolates from unrelated query joins, keeps filter-referenced joins incl. multi-hop), and range-joins to `_base` on the grain (`_src._w_time >= bucket_end − window` / `< bucket_end`; per-unit `INTERVAL` via the DEV-1716 dialect strategy, SQLite `DATETIME` modifiers). It LEFT-JOINs back to `_base` null-safe on the grain, reusing the `_cm_*` orchestration in `_render_with_cross_model_plans` so windowed and cross-model measures coexist in one query. The compact-duration parser moved to `slayer/core/window_duration.py` (dependency-free) so the ENGINE planner validates durations at plan time without importing the SQL layer; the plan carries the parsed `(amount, unit)` parts so the renderer never re-parses. A filter referencing a windowed measure reclassifies to `Phase.POST` (outer WHERE on the joined-back column, never HAVING on the plain base aggregate). Scope is **sum/avg local measures only**: eight plan-time guards (precedence G1→G8→G3→G4→G5→G7→G6→G2, running on the ORIGINAL value-key trees before hidden-slot interning so transform/composite win over hidden) raise loudly on non-sum/avg, no-time-dim, cross-model, transform (input or sibling), arithmetic/composite, hidden filter-only, mixed windowed+plain filter, and malformed/empty/non-string duration — the DEV-1504 shapes stay guarded, never silently degraded. `_src` row-filter semantics are exact legacy parity: model + WHERE-phase filters apply inside `_src`, only the typed `date_range` is stripped (the trailing window must reach rows before the range start); an explicit raw-time-column filter still truncates the window near the boundary — a documented inconsistency tracked as DEV-1732. Windowed CAST follows the base path (casts the inferred slot type, matching plain aggregates — not legacy explicit-type-only, whose distinction the typed pipeline no longer has). NULL-dimension groups get a NULL windowed value (the plain `=` inside the CTE never matches NULL — the same F1-style documented cardinality decision as the cross-model CTEs). An ORDER-BY-only windowed reference is not a reachable hidden shape — `OrderItem` coercion drops the window kwarg from the column name before the planner sees it. Post-merge with DEV-1712 Stage 8: the plan-time order-by validation and the windowed-plan build sit side by side after `_bucket_slots`; a selected windowed measure ordered-by is an `AggregateKey` so the Stage-8 pass lets it through, and the windowed slot resolves in the combined ORDER BY via its bare projected `_wm_` alias (never a dangling `_base.` ref). +- 2026-08-04 — Render a derived cross-model shared grain (DEV-1728, closes DEV-1495-b1): the DEV-1708 `derived_shared_grain_not_implemented` gate is DELETED (both the planner raise in `_compute_shared_grain_slots` and the generator backstop). A plain derived (non-time) joined dimension used as a cross-model shared grain now renders through the same forward `_cm_*` CTE path a base column and a TimeTrunc-derived grain already used: `_compute_shared_grain_slots`' `ColumnSqlKey` branch mirrors the `ColumnKey` branch exactly (append the slot; the `not s.hidden` guard is dropped because the generator's `base_projection_ids` intersection already excludes hidden filter-only refs), and `_render_cross_model_cte` expands the derived `Column.sql` rooted at the target, groups by it under the DOTTED host alias (`orders.customers.rev_x2` — the naming half DEV-1713 fixed, which is what unblocked this), and joins back null-safe. A plain derived grain is CAST to its declared type to match the host base's `_wrap_cast_for_type` (bare-column / TEXT grains skip the cast identically on both sides), so the join-back compares identically-typed values instead of silently dropping groups on an INT-vs-float mismatch. **Law 2 for grains:** a first/last aggregate grouped by a CROSSING derived grain materialises that grain as a `_val_` projection inside the ranked subquery (outer `SELECT`/`GROUP BY` reference the alias; `PARTITION BY` keeps the raw expression where the join is bound) — the same treatment the crossing first/last SOURCE value already got, and this also fixes a confirmed live bug where a crossing derived TIME grain + first/last emitted invalid SQL (the ranked subquery re-exported only `target.*`, leaking an unbound `regions.opened_at` into the outer `DATE_TRUNC`). The CTE now reserves the target model's physical column names on the shared allocator so a minted `_val_` can never shadow a `target.*` column (Codex F6; previously a latent gap on the DEV-1709 source materialisation too). A target-LOCAL derived grain needs no materialisation (its refs are re-exported by `target.*`). **Out of scope (unchanged):** an intermediate-hop shared grain (a grain on a middle hop of a multi-hop aggregate target path) still raises the pre-existing 7b.12 `NotImplementedError` — it hits base columns identically, so lifting it is a separate feature, not part of removing the derived-grain gate. This supersedes the 2026-08-02 DEV-1708 user-approved raise and the 2026-08-03 Stage-4×Stage-9 merge-resolution note (which deferred full support to DEV-1495): the CTE join-back rendering that was "still unbuilt" there is now built. The re-rooted and filtered-local sibling paths were probed and already handle a derived grain correctly (regression-guarded). +- 2026-08-04 — Order-only refs resolve like filter refs (DEV-1703 Phase 1, narrowing DEV-1712 Stage 8): ONE rule — Law 1 pulls whatever joins an ORDER BY ref crosses into the scope that owns its rows, even when ORDER BY is the sole referencer, and the ref is emitted in whatever form that scope makes legal. Two of Stage 8's rejections become resolutions: a LOCAL row column in a GROUPED query materialises a hidden `:max` aggregate slot and orders on its alias (MAX is order-preserving per group and portable across every Tier-1 dialect; a `TimeTruncKey` wraps its UNDERLYING column since DATE_TRUNC is monotonic, and is not a legal aggregate source anyway), and a JOINED row column in a RAW-ROWS query pulls its join and split-emits `customers__regions.name` (`_collect_joined_paths_for_base` now also walks ORDER BY targets — an order-only joined column is deliberately NOT added to `base_render_order`, which would project it and widen the GROUP BY grain). The wrap is interned POST-bind, so the bind-time aggregation gate (PK columns, `allowed_aggregations`, per-type defaults) deliberately does not apply — the caller asked to SORT a column, not to aggregate it, and a sort must not fail because `max` is not whitelisted. Interning happens after `_bucket_slots`, so the buckets are recomputed when any wrap is minted. Two rejections deliberately REMAIN (DEV-1735): a GROUPED query with a JOINED sort key, because an `AggregateKey` with a non-empty `source.path` always routes to a TARGET-rooted CTE (Law 3) which for a host-grain sort key degenerates to a scalar CROSS JOIN — every group would get the same global value and the sort would silently do nothing, strictly worse than a clear error; and an order-only LOCAL DERIVED column whose `Column.sql` crosses, whose join is likewise never pulled (now inconsistent with the bare joined case — same ticket). Grain preservation is the invariant these tests have always really been about and is pinned explicitly: a sort key must never reach GROUP BY. Verified by executed values on SQLite and real Postgres, not just emitted SQL. +- 2026-08-04 — Frame bounds vs population filters (DEV-1732). A ROW-phase filter conjunct that compares a **non-hidden query time dimension's raw column** against a **temporal literal** using `<`/`<=`/`>`/`>=` (either operand order), or a `BetweenKey` over such a column, is a **FRAME bound**: it narrows which buckets are returned, not which rows a CTE may read. Frame bounds are therefore stripped from every CTE that must reach outside the visible frame, so the two spellings of one intent agree — `date_range=[A,B]` and `filters=["created_at >= A and created_at <= B"]` now produce identical windowed numbers, where before the explicit spelling silently truncated the trailing window at the boundary (Stage 10 pinned that truncation as documented legacy parity; this ticket inverts the pin). **Both** bounds are stripped, not just the lower one: `date_range` strips a single `BetweenKey` node — i.e. both — so equivalence demands the same, accepting that a mid-bucket frame end lets the last bucket read rows past the stated end exactly as `date_range` already did (pinned by VALUE on SQLite + DuckDB, not just SQL shape). The strippable set is **every** non-hidden time dimension's raw column, not just the window axis — precisely the set for which a `date_range` spelling exists; matching is by ValueKey identity, so derived (`ColumnSqlKey`) temporal columns are covered for free. Hidden `TimeTruncKey` slots are excluded and the exclusion is **load-bearing**: `_build_windowed_plans` skips hidden row slots, so a hidden time axis is never equality-joined into `_src` and stripping its bound would leave it wholly unconstrained (over-count) where keeping it merely preserves prior behaviour. A **temporal literal** is a bare `LiteralKey` holding a non-`None` `str` — a deliberate whitelist of one shape, mirroring `BetweenKey`'s endpoints, rather than "contains no column ref" (which would admit dynamic expressions, and would strip `created_at < None`, turning an empty result into the full population, and `created_at >= 5`). A top-level `and` is **split** (n-ary flat, nested `and` recursed; survivors rebuilt in order) so `"created_at >= X and status = 'paid'"` keeps constraining `_src` to paid rows while reaching back before X; `or`/`not` are never descended into — no sound split exists, and keeping the predicate whole preserves prior numbers. NOT frame bounds: `==`/`!=`/`in`/`is` (equality on a raw timestamp means "this instant", never a range), a non-literal RHS, a `ScalarCallKey` LHS, and a time column that is not a query time dimension (no `date_range` spelling exists for it, so dropping it would over-count against every other measure — the core cardinality principle). **Mode-A `SlayerModel.filters` are exempt entirely** (deliberate, not an oversight): a model filter defines which rows EXIST rather than which frame the query looks at, there is no model-level `date_range` to be inconsistent with, and analysing arbitrary dialect SQL with `__` join-path aliases would make a silent mis-strip possible — so a time-scoped model still clips the window. **No escape hatch** (no opt-out kwarg, no legacy flag): one intent, one meaning; genuine population clipping goes in an inner stage of a multi-stage query. Accepted risk: a caller who depended on the old truncation sees numbers move with no error. Implementation: `slayer/core/time_bounds.py` (dependency-free so planner and generator share it) holds the analysis; `plan_query` computes `PlannedQuery.frame_bound_columns` **once** and partitions filters into `WindowedAggregatePlan.where_filter_ids` + `src_filter_rewrites` (residuals); the generator's `_effective_src_filters` materialises that view once and feeds the SAME list to both join discovery and rendering, so the two cannot disagree about the CTE's contents. Note that stripping can never orphan a join — the strippable columns are exactly the time dimensions `_src` always projects (`_w_time` / `_w_td_`), so their joins are independently required. The `date_range` filter-id skip is kept alongside the helper as a redundant floor, making a Stage-10 regression structurally impossible. +- 2026-08-04 — Scope amendment: DEV-1732's frame-bound rule also fixes the **`time_shift` shifted CTE** (`_shifted_where_part`), which carried the identical asymmetry — `BetweenKey` omitted, every other ROW filter propagated — so the earliest visible bucket's shifted value was correct under `date_range` and NULL under the explicit spelling. The `isinstance(..., BetweenKey)` special case is subsumed by `strip_frame_bounds` (a `date_range`'s column is always a query time dimension's raw column, so the helper returns `None` for it — same behaviour, one rule), and the join-path scan now runs on the residual. Deliberately widened past the issue's title after weighing it: the rule is a semantic decision, not a `_wm_` detail, and shipping it honoured in only one of the two places it applies would leave the next reader to re-derive the analysis. Codex flagged the widening as unrequested scope during plan review; the scope was amended explicitly rather than implicitly, which was Codex's own prescribed remedy. This is a user-visible behaviour change for anyone who wrote an explicit bound and relied on the truncated shifted CTE. +- 2026-08-04 — Order-only transform / composite / windowed ORDER BY targets (DEV-1733, DEV-1703 Stage 8 follow-up). Stage 8 rejected an ORDER BY ref bound to a **transform** or **composite** slot with an actionable `ValueError`; that raise is deleted. The full contract for an undeclared order target is now: local aggregate, cross-model aggregate, **transform** (`rank`/`cumsum`/`lag`/`lead`/`ntile`/`change`/`change_pct`), **composite** (`revenue:sum / cnt:sum`, `abs(a:sum)`, `change(a:sum) / 2`) and **windowed** (`a:sum(window='90d')`, bare or inside a composite) are all supported as HIDDEN order targets, stripped from response columns + `StageSchema`. They reach that via TWO materialisation paths, not one (detail under **Materialisation** below): a standalone target materialises a hidden slot that an outer wrap then trims, whereas a composite whose operands include a `_cm_` / `_wm_` value stays **inline** in the combined ORDER BY — the combined path has no outer trim wrap, so a materialised hidden column there would leak as a public result column. Row-column and joined-column shapes keep their Stage-8 behaviour verbatim. **Entry point** (D1): `_coerce_order_column` emits an `_expr_pending` placeholder `ColumnRef` when the canonical name fails `ColumnRef` validation AND the string is a *formula candidate* (contains `:` or matches the func-style call pattern), with `raw_formula` carrying the original; `_order_formula_candidate` is the single predicate shared with `_capture_raw_formula` so the two validators cannot drift. The boundary is deliberate: `order=[{"column": "rev / cnt"}]` — a composite over declared measure **aliases** — is not a candidate and keeps its pre-existing Pydantic error, because alias references inside expressions are unsupported everywhere in SLayer (a measure `{"formula": "rev / id:count"}` fails the same way). The planner routes on the sentinel **AND** a non-empty `raw_formula`, so a model with a genuine `_expr_pending` column, or a hand-built/deserialized `OrderItem`, still resolves normally. **Silent-drop fix**: `_iter_slot_deps` yields a composite's operands but never the composite itself, so `find_by_key` returned `None` in `plan_query` and the ORDER BY entry was *silently discarded* — `order=[{"column": "change(a:sum)"}]` ran unsorted with no error. `ProjectionPlanner.plan` now interns the top-level `ArithmeticKey`/`ScalarCallKey` of an **order** spec as a hidden slot; filters keep the operands-only walk (their top-level composite is rendered inline into WHERE/HAVING). **Materialisation** (D4) is per-path and not uniform: the transform path already emits a step CTE for unmaterialised composites; the no-transform path adds the composite's own slot id to `base_render_order` so it renders in the base SELECT and `_build_outer_trim_wrap_sql` trims it; the cross-model/windowed combined path keeps DEV-1503's **inline** `outer_composite_order_expressions` term, because that path has no outer trim wrap and a materialised hidden column would leak as a public result column. A composite with a `_cm_`/`_wm_` operand is therefore excluded from base materialisation (`_composite_has_remote_operand`) and from the `_add_local_aux_slots` promotion — otherwise it renders in `_base` from a **plain** aggregate while the CTE sits joined but unused, silently substituting a non-rolling value. `_apply_order_limit_from_planned`'s hidden branch dispatches on an explicit `(AggregateKey, ArithmeticKey, ScalarCallKey, TransformKey)` tuple rather than "any hidden slot with an alias", so a hidden ROW slot still hits the split-emission / invariant branches. **Windowed** (DEV-1714 Stage 10 x DEV-1733): `_guard_windowed_measures` returns `dict[key, hidden]` and gains an order-vk pass registering order-only windowed keys as hidden plans; `WindowedAggregatePlan.hidden`/`public_alias` (present but unused since Stage 10) are now populated, the combined SELECT trims a hidden windowed column under the same `plan.hidden and not transform_layers` predicate the hidden-CMA branch uses, and `hidden_cma_order_ref` was generalised to `hidden_cte_order_refs` — checked **before** the `cma_slot_ids` gate, since a windowed slot is not a cross-model slot and would otherwise fall through to a bare alias the SELECT no longer emits. G5 is relaxed for **order** vks only (deliberate asymmetry: a windowed composite is legal in `order`, still 400 in `measures`, since projecting it surfaces the rolling value's NULLs as user-visible results — DEV-1504); G4 (windowed + any transform) and cross-model/non-sum-avg windowed guards are untouched. This fixed a live silent-wrong-answer: an order-only windowed ref rendered a **plain** `SUM` and ordered by it (the pre-existing planner comment asserting the shape was unreachable was wrong — `OrderItem.raw_formula` preserves the `window=` kwarg). **Hidden-alias uniqueness** (D5) moved from the renderer to the PLAN: `ValueRegistry` uniquifies a hidden slot's `declared_name` (`_2`/`_3`/…) against every taken name, and `ProjectionPlanner.plan` pre-reserves all declared public/canonical names before the first intern so the outcome is intern-order independent (a hidden dep of measure *i* must not claim a name measure *j>i* declares publicly — public names are never renamed). This fixed a second live silent-wrong-answer: two hidden transform slots of the same op both took `_cumsum_inner`, so `cumsum(a:sum) + cumsum(b:sum)` projected two step-CTE columns under one alias and computed `cumsum(a) + cumsum(a)` (52/79 came back as 100/150). DEV-1692 had fixed this class inside the `time_shift`/`consecutive_periods` emitters only; owning it at intern time covers every renderer at once. Two DEV-1501-era tests pinning the old contract were inverted, not deleted: `test_order_arithmetic_walks_to_aggregate` (the arithmetic root is now slotted) and `test_hidden_composite_order_rejected_at_input_validation` (the composite string is now accepted). Out of scope, still guarded: windowed composites in `measures` and windowed + transform (DEV-1504); `time_shift` combined with a cross-model aggregate (DEV-1450 stage 7b.15e), which rejects the declared-measure form too. +- 2026-08-04 — Merge resolution, Phase 1 × DEV-1733 (both amend the same plan-time ORDER BY classifier in `plan_query`). The two branches improved DIFFERENT rows of the same behaviour table and the merge takes both: Phase 1 turned a GROUPED **local row column** from `ValueError` into a hidden `:max` wrap, and DEV-1733 turned a **transform / composite** target from `ValueError` into hidden materialisation. Neither change touches the other's row, so no behaviour was traded away. One ordering difference WAS material and had to be decided rather than merged textually: DEV-1733 tests the joined-`path` check BEFORE the grouping check, so it rejects an **ungrouped** joined row column (`UnresolvableOrderColumnError`); Phase 1 tests grouping first, so an ungrouped joined column pulls its join (Law 1) and split-emits. Phase 1's order wins — the raw-rows case is legal SQL (the row IS the grain), it is pinned by executed values, and rejecting it was a Stage-8 conservatism that Phase 1 deliberately lifted. The joined + GROUPED rejection that both branches share is untouched and remains DEV-1735. In `order_entries`, DEV-1733's loud raise for an unslotted order key is kept and composed with Phase 1's `order_key_remap` lookup, so a remapped `:max` slot resolves and any genuinely unslotted shape still fails loudly instead of being silently dropped. + +- 2026-08-04 — Legacy enrichment stack deleted (DEV-1485 = DEV-1703 Stage D / Phase 3). `slayer/engine/enrichment.py` (3369 LOC) and `enriched.py` (316) are gone, along with the legacy subgraph in `query_engine.py` (3942 → 2842) and `generator.py` (11817 → 10504) — ~6800 lines total. The typed pipeline is now the ONLY rendering path: top-level planning, query-backed expansion (execute *and* save), join-target rendering, and dialect emission. **Reachability was measured, not assumed.** Both legacy entry points were instrumented with hit COUNTERS — not raises, so a swallowed exception in a broad `except` could not hide a live path — and the full suite run against the merged tree: `_resolve_model_inner`'s named-query branch got **0 hits**, `_query_as_model` got **1** (a BigQuery test calling it directly). Confirmed statically too: `_resolve_query_model`'s only callers were itself and `_query_as_model`, `_walk_join_chain`'s only callers were the legacy resolvers, and no module outside `query_engine.py` calls any engine private method — the subgraph was closed, entered solely through that one dead branch. **Two DEV-1485 premises were wrong and are corrected here.** (1) It required porting `_resolve_model_inner`'s named-query branch onto `_expand_query_backed_model` before `_query_as_model` could go; that branch has no caller (the typed pipeline resolves siblings via `_follow_sibling_chain` in `source_bundle.py`), so it is deleted rather than ported — zero adapters, not one. (2) It listed `parse_formula` + `FieldSpec`/`AggregatedMeasureRef`/`ArithmeticField`/`TransformField`/`MixedArithmeticField` in `core/formula.py` as dying symbols; `parse_formula` did NOT die with enrichment — `slayer/dbt/converter.py` and `slayer/osi/converter.py` both call it as a formula VALIDATOR (dbt categorises its failures as `dangling_reference`), and the union members are its return-type contract. A per-symbol scan of `formula.py` found exactly one genuinely dead name, `FILTER_FUNCTIONS` (an unused duplicate of `_LIKE_INTERNAL_NAMES`), which was removed; the rest of the module stays. Collateral collapses now that their legacy mode is gone: `_maybe_raise_schema_drift` loses its `enriched=` branch (both callers already passed `touched_models`) and with it the orphaned `_collect_models_touched`; `_build_agg` loses its `measure=` compat surface and `_agg_render_spec_from_enriched`; three zero-caller helpers go (`_has_cross_model_filter`, `_is_windowed_measure`, and `_window_referenced_aliases`, the last already superseded by `ScopeFrame` in DEV-1714); three dicts annotated `Dict[str, "EnrichedMeasure"]` actually held `AggRenderSpec` and were retyped rather than kept alive. `tests/parity_xfails.py` + its `pytest_collection_modifyitems` hook are deleted — per Codex F9 the hook was inventoried first and does exactly two things (apply the strict-xfail markers, self-police stale keys), both no-ops with an empty registry, which was the DEV-1485 gate. Test migrations preserved every original invariant rather than dropping coverage: the BigQuery mangled-alias wrap moved to `_expand_query_backed_model` (same backtick invariant, only the rename target differs); two `strip_source_model_prefix` tests moved from poking internal resolvers to end-to-end through the typed pipeline (strictly better — `execute` applies the strip itself); the ContextVar per-task recursion-guard test became a concurrent-expansion isolation test (the typed path threads `_resolving` as a parameter, so isolation is structural); `TestContextVarSafety` was deleted since it asserted the migrated path never touches ContextVars that no longer exist. Codex F11 audit beyond greps: all 164 `slayer` modules import cleanly in isolated subprocesses, no dynamic/string-based imports of the deleted modules, no deleted name in any `__init__` or `__all__`. +- 2026-08-04 — `path_resolution.py` deleted (DEV-1485 Stage D follow-up, user-approved). `walk_join_chain` + `NoJoinError` had ZERO production callers once the legacy resolvers went: the typed pipeline walks join hops in `binding.py` against the resolved bundle (`bundle.get_referenced_model`), never through this module. Deliberately widened past DEV-1485's file list because leaving it was an active trap, not merely dead weight — it had already absorbed one test and made it look meaningful. `test_inner_path_resolvable_by_engine_walker` exists to prove that the INNER hop `recommend_root_model` emits is walkable BY THE QUERY-TIME RESOLVER (the storage symmetry invariant); when the legacy stack was deleted it was rewired from `engine._walk_join_chain` onto `path_resolution.walk_join_chain`, which *was* the query-time resolver when the test was written but no longer is — so it pinned a function no query touches and would have kept passing while the property it names silently broke. It now feeds the recommendation back in as a query and asserts the join renders, which fails if any layer (binder, planner, generator) cannot traverse the hop. `recommend_root_model` itself was never affected — it uses `JoinGraph` + `min_hops_root`. Knock-on: `_resolve_model`'s `named_queries` parameter is removed, since its only remaining justification was the `resolve_model` callback contract that `walk_join_chain` defined. **Method note, for the next deletion of this kind:** the Stage-D pass keyed on "method takes an `enriched` parameter", which structurally misses legacy-only helpers that don't take one — eleven were left orphaned and swept up afterwards (`_OrderColRef`, `_order_split_sql`, `_alias_prefixes`, `_filter_dotted_columns`, `_filter_references_available`, `_safe_parse_outer`, `_deps_available`, `_build_consecutive_periods_ctes`, `_build_self_join_column`, `_apply_placeholder_fill`, `_apply_order_limit_to_planned_sql_string`). The reachability scan that found them counts AST `Name`/`Attribute` references, so it reports FRAMEWORK-INVOKED symbols as dead: deleting `SlayerResponse._populate_columns` (a Pydantic `@model_validator` that fills `columns` from `data[0].keys()`) broke 32 tests. Always check `decorator_list` before trusting a zero-reference verdict. - 2026-08-03 — List-valued `{variable}` substitution (DEV-1730 / #270): a `list`/`tuple` variable renders an injection-safe `IN`-list body through the same `_render_variable_value` choke point, so the one branch covers every consumer (Mode-A engine pass, Mode-B enrichment, `get_column_types` defaults probe) with no schema change. Quoting is deliberately **asymmetric** vs scalars: a scalar string's quotes are author-written (`status = '{v}'`), but list elements are **auto-quoted** at render time — a single `{var}` placeholder can't carry per-element quotes, so `col IN ({regions})` renders `IN ('US', 'CA')`. Mode-B (`escape="python"`) appends a **trailing comma** (`('US',)`) so the Python-AST parser always reads a tuple, never `str` containment for a 1-element list. An **empty list raises** rather than emitting `IN ()` (invalid SQL): "no filter" semantics belong to a sentinel default, not an empty list (DEV-1730 acceptance). Per-element escaping reuses `_escape_string_value`, so DEV-1727's dialect-aware escaping composes automatically. - 2026-08-03 — Optional blocks + Cube JS/FILTER_PARAMS import (DEV-1730 / #270): a Mode-A-only `{? ... ?}` block renders its content parenthesised when every inner `{var}` is supplied, else collapses to the neutral `(1=1)` — the SLayer form of a Cube `FILTER_PARAMS` optional pushdown. Blocks live in the same `substitute_variables` (escape="sql") scanner as `{var}`/`{{`/`}}`, must contain ≥1 var, do not nest, and are rejected in Mode-B. A block-bearing model runs substitution even on a zero-variable call so its blocks collapse (the `_substitute_model_sql_surfaces` fast-path now checks for `{?` too); a block-free, required-only model with zero variables is still left untouched (the documented DEV-1625 raw-brace-literal boundary). `extract_model_variables(model)` derives required (bare, no default) vs optional (in-block or defaulted) from the four Mode-A surfaces — structural, nothing persisted, surfaced additively in the inspect skeleton `Variables:` line. The Cube importer gains a **JavaScript front-end** (esprima ESTree parser, a new core dep) that parses `cube()`/`view()` into the same `CubeCube`/`CubeView` shapes as YAML (dynamic values → report + skip member). FILTER_PARAMS refs are carried JS→converter as structured `CubeFilterParamRef` on the transient `CubeCube` (sentinels in the surface text; no arrow-body re-parse, sidestepping the `{var}`-vs-`{FILTER_PARAMS…}` brace clash); the converter resolves sentinels AFTER `translate_cube_refs` so the introduced `{var}` are never eaten. Requiredness (bare vs block) is decided in the converter alone via `honor_required_meta` (default on; CLI `--ignore-required-meta`) AND the member's `meta.required`; with the flag off a scalar-position arrow collapses to Cube's own `(1=1)::TIMESTAMP` booby-trap, faithfully. Cross-cube refs, unknown members, and generated-name collisions (`d`→`d_from` clashing member `d_from`) drop the cube (`filter_params_unsupported`); each logical variable is reported once (`filter_params_variable`) and stashed in `meta.cube_variables`. `render_probe_text` (blocks→`(1=1)`, bare vars→`0`) is the single import-time validation renderer, matching runtime collapse. - 2026-08-04 — Dialect-aware / complete escaping for Mode-A `{variable}` substitution (DEV-1727), hardening DEV-1625. `substitute_variables(..., escape="sql")` is now **dialect-aware** and **fail-closed**: it gained a required keyword-only `backslash_escapes` signal (`bool | None`, raises if `None` in sql mode) so a caller rendering raw SQL can never silently under-escape. On backslash-escaping dialects (MySQL/ClickHouse/Snowflake/Redshift/BigQuery/Databricks/Spark) it doubles the backslash before escaping the single quote; on standard dialects it keeps the `''` quote-doubling. The double quote is deliberately left untouched — inside a single-quoted literal `\"` is NOT a recognised escape on 6 of the 7 backslash dialects (only MySQL), so escaping it would corrupt the value. The regime is DERIVED from sqlglot's own tokenizer via `SqlDialect.backslash_escapes_strings` (= `"\\" in tokenizer.STRING_ESCAPES`, guarded + 14-dialect pinned) so our escaping can never drift from the parser that reads the substituted SQL. `escape="python"` (Mode-B) additionally encodes the full C0 control range (`\t`/`\n`/`\r` named, rest `\xNN`) so raw newlines/NUL no longer break `ast.parse`. Engine fail-closed: `_substitute_model_sql_surfaces` / `_render_probe_model` require a `dialect`, threaded from the resolved datasource — no bare bool to forget. Assumes MySQL's default `sql_mode` (backslash escapes on); `NO_BACKSLASH_ESCAPES` servers are a sqlglot-layer-wide limitation, documented not fixed. The SQLite backslash end-to-end gap stays a pinned strict-xfail (pre-existing, out of scope). Bound parameters rejected (don't fit substitute-into-raw-SQL). Nested/join/cross-model lineages remain DEV-1678. diff --git a/docs/architecture/binding.md b/docs/architecture/binding.md new file mode 100644 index 00000000..1e7dd1f1 --- /dev/null +++ b/docs/architecture/binding.md @@ -0,0 +1,165 @@ +# Binding + +**Module:** `slayer/engine/binding.py` + +The binder takes a `ParsedExpr`, a scope (`ModelScope` or `StageSchema`), and a +`ResolvedSourceBundle`, and produces a `BoundExpr` whose leaves are resolved +`ValueKey`s. It is the stage that turns *names* into *structural identity* — and +it is a pure function of its inputs (P11). + +```mermaid +flowchart LR + parsed["ParsedExpr"] --> bind["_bind (recursive)"] + scope["scope: ModelScope | StageSchema"] --> bind + bundle["ResolvedSourceBundle"] --> bind + bind --> bk["BoundExpr(value_key: ValueKey)"] +``` + +## Output types + +- `BoundExpr(value_key)` — the whole expression's structural identity. `.phase` + is lifted from `value_key.phase`. +- `BoundFilter(value_key, phase, referenced_keys)` — adds the max phase any + referenced slot reaches (**P8**) and the full tuple of `ValueKey`s touched + anywhere in the tree (used by cross-model filter routing). + +Public entry points: `bind_expr`, `bind_filter`, `bind_time_dimension`, plus the +`walk_value_keys` traversal helper. + +## Resolving a reference + +`_resolve_ref` (bare) and `_resolve_dotted` (path) are where scope semantics +live (**P5**): + +**Against a `StageSchema`:** a bare name must be a column in the flat schema +(else `UnknownReferenceError`); a dotted ref raises `IllegalScopeReferenceError` +("downstream stages see a flat schema"). This is the DEV-1449 guard. + +**Against a `ModelScope`:** + +- A bare name resolves to a `ColumnKey(path=(), leaf=name)`, or to a + `ColumnSqlKey` if the column is derived (`col.sql` set and not a trivial + self-remap). If the name matches a `ModelMeasure` instead of a column, the + binder raises with a suggestion to expand it first — measure expansion is the + [parser stage](parsing.md)'s job, not the binder's. +- A `__`-bearing bare name is legal *only* if it exact-matches a literal column + name (the C11 carve-out); otherwise `IllegalScopeReferenceError`. +- A dotted ref walks the join graph: `parts[:-1]` are join targets, `parts[-1]` + is the leaf. Each hop must have a matching `ModelJoin` on the current model and + the target must be in the bundle; revisiting a model raises a legacy-compatible + `Circular join` error. The terminal column becomes a `ColumnKey` (or + `ColumnSqlKey`) carrying the hop path. + +### C14 — self-prefix stripping + +`_resolve_dotted` strips a leading segment equal to the host model's name before +walking: `orders.status` on an `orders`-rooted query → `status`; +`orders.customers.name` → `customers.name`. This is principle **C14**, +preserving the legacy convenience of qualifying with your own model name. + +## Binding aggregates + +`_bind_agg` builds an `AggregateKey`. The source is a `StarKey` for `*`, a +path-carrying `StarKey` for a cross-model star (`customers.*:count`, via +`_resolve_dotted_star`), or the bound column otherwise. Positional/kwarg +arguments bind through `_bind_agg_arg` — identifier args become `ColumnKey`, +literals normalize via `normalize_scalar`. Crucially, `_resolve_column_filter_key` +looks up the resolved source column's `Column.filter` and folds it into the key +as `column_filter_key` (a `SqlExprKey`) — so an aggregate over a filtered column +has a distinct identity (P3 / the `column_filter_key` invariant from +[Typed keys](typed-keys.md)). + +## Binding transforms (P9) + +`_bind_transform` produces a `TransformKey` whose `input` is the bound value to +transform — a `ValueSlotRef`, not a string. Two whitelists govern it: + +- `_TRANSFORM_KWARG_RULES` — per-op accepted kwargs. It is deliberately broader + than the legacy whitelist: every transform implicitly accepts `partition_by` + (so `change(measure, partition_by=…)` threads through to the desugared + `time_shift`, **C6**), and the rank family / `time_shift` / `lag` / `lead` / + `ntile` / `consecutive_periods` add their own. +- `_TRANSFORM_POSITIONAL_KWARGS` — the transforms whose documented DSL form + accepts positional params after the value (`time_shift(x, periods, + granularity)`, `lag(x, periods)`, `lead(x, periods)`). A name supplied both + positionally and as a kwarg is an error. + +`partition_by` values must bind to a `ColumnKey` / `ColumnSqlKey` (and become +`partition_keys`); other kwargs must fold to a scalar literal +(`_fold_to_scalar` also handles unary-minus over a numeric literal, the AST shape +of `periods=-1`). `_apply_transform_kwarg_defaults` validates required kwargs +(`ntile` needs a positive-integer `n`; `time_shift` needs `periods`) and applies +defaults (`lag`/`lead` default `periods=1`). Note the binder does **not** set +`time_key` — it lacks query context; the [stage planner](stage-planning.md) +attaches it after all binding completes. + +## Binding scalar calls (P1 / C12) + +`_bind_scalar` re-checks `SCALAR_FUNCTIONS` membership (defence-in-depth against +direct `ParsedExpr` construction that bypasses the parser) and builds a +`ScalarCallKey`. Arithmetic, comparison, boolean, and unary ops all become +`ArithmeticKey` with the operator string. + +## Filters and phase classification (P8) + +`bind_filter` binds the predicate, walks it via `walk_value_keys` to gather every +referenced `ValueKey`, takes `phase = max(referenced phases)`, and rejects raw +windows. The phase is computed entirely from the slots referenced — no text +analysis. `_reject_windowed_column_sql` raises `IllegalWindowInFilterError` if +any referenced `ColumnSqlKey` has a windowed `Column.sql` body — DEV-1369 +removed predicate promotion, so filtering on a window is an error, not an +auto-hoist. (Against a `StageSchema` this check is skipped — window detection +already happened when the upstream stage was bound.) + +### `alias_map` — filter/order refs by declared name (P4 / DEV-1445) + +`bind_filter` accepts an optional `alias_map: Dict[str, ValueKey]` mapping a +stage's declared-measure names (user `name`, declared name, canonical alias) to +their bound `ValueKey`. A bare ref matching an alias interns onto that exact slot +*before* any column lookup. This is what lets a filter reference a renamed measure +by alias: `filters=["rev >= 100"]` for a measure declared +`{"formula": "customers.revenue:sum", "name": "rev"}` binds `rev` onto the +cross-model aggregate slot — and because the dotted/colon form interns +structurally onto the *same* `AggregateKey`, both forms share one slot. Only +*measure* aliases enter the map (never dimension / time-dimension names), because +a time dimension's declared name is its raw column and a `created_at <= '…'` +filter must resolve to the raw column, not the truncated dimension slot. See +[Stage planning](stage-planning.md) for how the map is built. + +## `bind_time_dimension` + +Binds a `TimeDimension` into a `BoundExpr` carrying a `TimeTruncKey`. The +underlying column resolves against scope exactly like an identifier ref and must +have a temporal `Column.type` (`DATE` / `TIMESTAMP`). It may be a base +`ColumnKey` OR a **derived** `ColumnSqlKey` (DEV-1450 follow-up #4a): +`TimeTruncKey.column` is `Union[ColumnKey, ColumnSqlKey]`, and the generator +applies the `DATE_TRUNC` over the expanded `Column.sql` everywhere it would +over a bare column. + +> **Limitation.** Only `ModelScope` is accepted (a `StageSchema` raises — +> downstream stages already see the truncated column as a flat name). + +## `walk_value_keys` + +Yields every `ValueKey` reachable from a key (including the key itself), +recursing through `AggregateKey` source/args/kwargs, `TransformKey` +input/args/kwargs/partition_keys/time_key, `ArithmeticKey` operands, +`ScalarCallKey` args, and `BetweenKey` column/low/high. It is the typed +counterpart to the parser's `walk_parsed_refs`, used for phase computation and +cross-model filter routing. + +## Design rationale + +- **Why is the binder pure?** Everything it needs is in the bundle (P11). No + storage access, no `ContextVar`, no callback re-resolution — so binding a given + `(parsed, scope, bundle)` is deterministic and order-independent. This is the + single biggest simplification over the legacy enrichment closures. +- **Why fold `Column.filter` into the aggregate key rather than handle it at + render time?** Because two aggregates over the same column differ *as values* + when their filters differ; making that part of identity means the registry + interns correctly and the generator never has to reconcile two slots that are + "the same column but filtered differently". +- **Why does the binder leave `time_key` unset?** Resolving the active time + dimension needs query-level context (`main_time_dimension`, + `default_time_dimension`, the set of TDs in the query). The binder is + expression-local; the planner has the query, so it patches `time_key` there. diff --git a/docs/architecture/cross-model-aggregates.md b/docs/architecture/cross-model-aggregates.md new file mode 100644 index 00000000..34b07932 --- /dev/null +++ b/docs/architecture/cross-model-aggregates.md @@ -0,0 +1,365 @@ +# Cross-model aggregates + +**Modules:** `slayer/engine/cross_model_planner.py` (strategy + +`_maybe_reroot_cross_model_plan`), +`slayer/sql/generator.py` (`_render_with_cross_model_plans`, +`_render_rerooted_cross_model_cte`) + +A cross-model aggregate is `customers.revenue:sum` on an `orders`-rooted query — +an aggregate whose source carries a non-empty join path. Principle **P3** says it +shares the `AggregateKey` shape with a local aggregate (only `source.path` +differs), and that "base CTE vs cross-model CTE" is a *render strategy* decided +downstream, not a semantic split. The identity side of P3 holds cleanly. The +**render** side turned out to need two strategies — the most significant +deviation from the plan. + +## The identity is uniform; the rendering is not + +```mermaid +flowchart TB + agg["AggregateKey(source.path = ('customers',), agg='sum')"] + agg --> cmp["cross_model_planner.plan(...)"] + cmp --> plan["CrossModelAggregatePlan"] + plan -->|rerooted_plan is None| fwd["forward-path CTE
FROM bare target, GROUP BY forward dims"] + plan -->|rerooted_plan set| rr["re-rooted nested PlannedQuery
FROM target + target's joins"] +``` + +The planner detects the cross-model case structurally (`agg_path` non-empty in +`plan_query`) and invokes the strategy. The strategy is a substitutable +component — **I1**: `CrossModelPlanner` is a `Protocol`, +`IsolatedCteCrossModelPlanner` is the default — so the *shape* of the result +(`CrossModelAggregatePlan` in `planned.py`) is strategy-agnostic and only the +populating planner changes. + +## Strategy 1: `IsolatedCteCrossModelPlanner` (the plan's design) + +This is the planned design: one CTE per `(target_model, shared_grain)`. It walks +the join chain from host to target (`_walk_chain` → `JoinRequirement`s), groups +the aggregate at the first hop's target grain, and builds `join_back_pairs` so +the host LEFT JOINs the CTE back on the first-hop columns. `_make_cte_schema` +produces the CTE's typed projection. `_aggregate_alias` derives the output +column name via `canonical_agg_name`. + +### Host-filter routing (the `inherited_filter_policy` decision table) + +`classify_host_filter` is a pure classifier mapping each host filter to a +`FilterRoute`: + +| Filter references | Route | +| --- | --- | +| host-local row slot only | `DROP_HOST_LOCAL` (applied at host) | +| all on the joined-target path (row) | `PROPAGATE_WHERE` | +| cross-model agg-ref on the same target | `PROPAGATE_HAVING` | +| slots on a different joined branch | `DROP_UNREACHABLE` (+ warn) | +| mixed reachable + unreachable | `DROP_UNREACHABLE` (+ warn) | +| transform / POST phase | `STAY_AT_HOST_POST` | + +The planner threads each route into the explicit +`where_filter_ids` / `having_filter_ids` lists on `CrossModelAggregatePlan` so +the generator never re-classifies. The target model's own `SlayerModel.filters` +ride into `target_model_filters` (always-applied WHERE), and a `Column.filter` on +the aggregated column rides on the `AggregateKey` itself as a CASE-WHEN — neither +goes through host-filter classification. `shared_grain_slots` is the set of host +dimension/time-dimension slots reachable from the target, used to LEFT JOIN the +CTE back without changing cardinality. + +### Rerooting the aggregate's embedded references (`reroot_aggregate_key`) + +When the forward `_cm_*` CTE (`_render_cross_model_cte`), its HAVING route, and +the re-rooted-plan formula (`_local_agg_formula`) render a cross-model aggregate +in its target scope, every reference embedded in the `AggregateKey` — the +`source`, positional `args` (e.g. the `first`/`last` explicit time arg), keyword +`kwargs` values (e.g. `weighted_avg(weight=…)`), and `column_filter_key` — must +be re-anchored from the query root's coordinate system to the target's. This is +one symmetric transform, `slayer.core.keys.reroot_aggregate_key(key, *, +target_path)` (DEV-1707), which prefix-strips `target_path` off each ref's join +path and keeps the residual (`('customers','regions')` under target +`('customers',)` → `('regions',)`; an exact match → local). `column_filter_key` +is owner-anchored (stamped against the model that owns the filtered column) and +therefore invariant under reroot — it is carried through unchanged. A time arg +left with a *residual* path after reroot (a hop past the target) is a +[DEV-1526](https://linear.app/motley-ai/issue/DEV-1526) Stage-4 gap: the isolated +CTE does not yet pull that deeper join, so `_resolve_explicit_time_col` raises +for the derived-column case and the scope-closure validator catches the +bare-column case. + +## Strategy 2: re-rooting (the deviation) + +`IsolatedCteCrossModelPlanner` alone is insufficient. When the host query carries +dimensions that are reachable from the target through the **target's own** join +graph (the legacy `_build_rerooted_enriched` case — e.g. +`policy_amount → policy → policy_number`), the forward-path CTE +("FROM bare target, GROUP BY forward-path dims only") collapses the host +dimension to a scalar `CROSS JOIN`: every host row gets the global aggregate +instead of a per-dimension value. + +`_maybe_reroot_cross_model_plan` detects this and attaches a nested re-rooted +plan. As of DEV-1450 follow-up #2 it lives in `cross_model_planner.py` and runs +**inside** `IsolatedCteCrossModelPlanner.plan` — the strategy owns the +forward-vs-re-rooted choice rather than `plan_query` patching the plan after the +fact: + +```mermaid +flowchart TB + detect["host dims/filters reachable from target
via the target's join graph?"] + detect -->|no| keep["keep forward-path plan"] + detect -->|yes| build["build a full SlayerQuery rooted at the target"] + build --> replan["subplan_builder(rerooted_query, rerooted_bundle)"] + replan --> attach["attach rerooted_plan / rerooted_grain_pairs / rerooted_agg_slot_id"] + attach --> gen["generator: _render_rerooted_cross_model_cte"] +``` + +It re-roots each host dimension/time-dimension/filter from the host's perspective +to the target's (`_reroot_ref`: host-local → `.`; on-target → bare; +through-target → strip the prefix), drops anything unreachable from the target +(matching legacy), reconstructs the local aggregate formula +(`_local_agg_formula`), builds a fresh `SlayerQuery` rooted at the target, and +compiles it via the injected `subplan_builder` callback (which `plan_query` +supplies as a `plan_query` recursion — keeping `cross_model_planner.py` free of a +`stage_planner` import). The sub-plan is rendered by +`_render_rerooted_cross_model_cte` as the `_cm_*` CTE (FROM target + the target's +joins, preserving host grain) and joined back on the re-rooted dimension via +`rerooted_grain_pairs`. + +### Why this is flagged as a deviation + +The plan envisioned the `inherited_filter_policy` decision table plus +`IsolatedCteCrossModelPlanner` as **the** cross-model mechanism. In practice +there are now **two** cross-model render strategies, both owned by the strategy +(`IsolatedCteCrossModelPlanner.plan` → `_maybe_reroot_cross_model_plan`), with +the re-rooting one bolted onto `CrossModelAggregatePlan` via `rerooted_plan` / +`rerooted_grain_pairs` / `rerooted_agg_slot_id`. P3's "one shape, render strategy +chosen downstream" holds for *identity* but not for *rendering* — and the +re-rooted path is, structurally, the legacy `_build_rerooted_enriched` shape +brought across to the typed plan. +This reintroduces (in a contained, typed form) the kind of "second resolution +path for a permutation" the redesign set out to eliminate. It works and is +tested, but it is the place a future reviewer should look first when reasoning +about cross-model behavior. + +## Strategy 3: host-rooted isolation — any crossing input (DEV-1503, widened by DEV-1709) + +A LOCAL aggregate (empty `source.path`) isolates into a **host-rooted** CTE +when **any** of its inputs crosses a join (Law 3, DEV-1703 D1/D2): + +- its `Column.filter` references a joined table — the original DEV-1503 + case (`loss_payment_amt:sum` with `filter="loss_payment.has_flag = 1"`), + read from the bind-time `column_filter_key.referenced_join_paths`; +- its **source `Column.sql`** crosses a join (`region_pay` with + `sql="customers__regions.payment_amount"`, single-dot forms, and sibling + derived chains); +- a **positional arg** crosses — including the explicit first/last time arg + (`amount:last(customers.signup_at)` and derived variants); +- a **kwarg** crosses — a column ref (`weighted_avg(weight=customers.w)` or + a crossing derived column), a user-supplied template-fragment string, or + a non-overridden model-default `AggregationParam.sql` fragment. + +The non-filter kinds are computed plan-time by +`slayer/engine/aggregate_input_paths.py::compute_aggregate_input_join_paths` +(the same parse → derived-expansion → root-scope-walk pipeline the filter +scan uses; an unparseable template fragment contributes nothing — parity +with the filter scan's defensive fallback). Without isolation, a crossing +input emitted inline in the host base SELECT would pull its join into the +host's FROM: two measures whose filter targets are different INNER joins +would intersect the base to rows present in BOTH targets, and any 1:N +crossing join would **multiply the host rows seen by sibling measures** — +the sibling-protection guarantee is the point of Law 3. The crossing +measure itself keeps multiply-per-match semantics inside its CTE (F1 +decision — 1:N semantics unchanged, only the scope moved). + +The trigger predicate is structural: +`agg_path` non-empty (forward cross-model, target-rooted) **OR** any +crossing input (host-rooted). Both route through +`IsolatedCteCrossModelPlanner.plan`; the host-rooted branch +calls `_plan_filtered_local`, which rebuilds the measure's formula text +via `_local_agg_formula` (round-trip-tested for every input shape) into a +**host-rooted** nested `PlannedQuery` (same `source_model`, same dims/TDs, +only the crossing measure as the single aggregate) and attaches it via the +same `rerooted_plan` / `rerooted_grain_pairs` / `rerooted_agg_slot_id` +slots the re-rooted path uses. The plan carries +`cte_root_model = host_model.name` as the disambiguator the renderer +reads; `_render_rerooted_cross_model_cte` short-circuits the source-model +swap when `cte_root_model` is set. Isolation is strictly +per-`AggregateKey`-slot: identical keys intern to one slot and share one +CTE; distinct keys get distinct CTEs (cross-CTE merging is DEV-1688 / +`may_inline` territory). + +```mermaid +flowchart TB + detect["any crossing input?\n(filter / source sql / arg / kwarg)"] + detect -->|yes| build["_plan_filtered_local builds host-rooted SlayerQuery"] + build --> replan["subplan_builder(rerooted_query, bundle)"] + replan --> attach["attach with cte_root_model = host.name"] + attach --> gen["generator: _render_rerooted_cross_model_cte (host-rooted branch)"] +``` + +`subplan_builder` always passes `disable_host_rooted_isolation=True` +(DEV-1709 rename of `disable_dev1503_isolation`) so the recursive +`plan_query` call inside the sub-plan does NOT re-trigger isolation on the +same measure — inside the CTE the crossing inputs render inline +(base-pull), which is legal there because the CTE is the aggregate's own +scope. The flag never affects target-rooted isolation. + +### Composite lowering (F3) + +In an AGGREGATE-phase composite (`a:sum + b:sum`, `coalesce(a:sum, 0)`), +each **crossing leaf** isolates individually (the leaves are hidden +aggregate slots that traverse the same trigger loop); local leaves stay in +`_base`; the composite expression renders only in the combined SELECT via +the leaves' projected aliases. This holds for projected composites, +filter-only composites (routed to the outer WHERE), and order-only +aggregate refs. + +### Law 2 inside the ranked scope + +The first/last ranked subquery re-exports only `source_relation.*` plus +rank/`_td`/`_dim` columns, so any crossing expression the outer SELECT +consumes — an aggregate SOURCE or a column-ref KWARG value — is +materialised as a `_val_` projection inside the subquery +(`_build_first_last_base_select`, mirroring the Stage-4 CTE path). The +projection is the **resolved** value (qualified, with the `Column.type` +inner CAST for non-bare expressions), so `SUM(CAST(x AS t))` semantics are +preserved and same-sql-different-type aggregates keep distinct +materialisations; HAVING and composite consumers bind to the same alias +via `FirstLastRenderState.value_alias_by_sql` (keyed by resolved text). + +### Filter routing for filtered-local + +| Host filter phase | Route | +| --- | --- | +| ROW | propagate into the host-rooted sub-plan (so a non-dim filter like `status = 'active'` affects the isolated aggregate's rowset) | +| AGGREGATE | **outer combined-SELECT WHERE wrapper** (see below) | +| POST | stay at the existing host post-transform wrapper | + +### Outer combined-SELECT WHERE wrapper + +An AGGREGATE-phase host filter referencing an isolated aggregate +(`loss_payment_amt:sum > 1000`) cannot route as HAVING inside the `_cm_*` CTE: +the LEFT JOIN back to `_base` would surface host rows whose filtered +aggregate didn't meet the predicate with a NULL value instead of dropping +them. The renderer (`_render_with_cross_model_plans`) classifies each +AGGREGATE-phase filter; any that walks an `AggregateKey` matching an +isolated slot is routed to an outer WHERE on the **combined SELECT** (which +is non-aggregating — plain WHERE is legal). The renderer +(`_render_filter_for_outer_wrapper`) substitutes: + +- isolated `AggregateKey` → `.""` (the joined-back column), +- any other slot → `_base.""` (the host base's projection). + +Non-isolated aggregate operands of a mixed filter (`loss_payment_amt:sum > +1000 AND total_amount:sum > 10` where `total_amount:sum` isn't a public +measure) are promoted to hidden aux slots in `base_render_order` by the +existing `_add_local_aux_slots(aggregates_only=True)` pass — `_base` +materialises them so the outer WHERE can reference them, and the combined +public projection trims them out. + +## Generator side + +`generate_from_planned` delegates to `_render_with_cross_model_plans` when +`cross_model_aggregate_plans` is non-empty. Each plan renders as a `_cm_*` CTE +(forward-path or re-rooted), joined back to the host base. `Column.filter` on the +aggregated column renders as `SUM(CASE WHEN THEN END)`. See +[SQL generation](sql-generation.md). + +### The forward CTE is a `ScopeFrame` (DEV-1708, DEV-1703 Stage 4) + +`_render_cross_model_cte` builds one `ScopeFrame` rooted at the target relation +and routes **every** expression it renders — the rerooted aggregate source, +positional args, column-ref kwargs, `Column.filter`, shared-grain dimensions, +target-model filters, and routed host WHERE/HAVING filters — through +`resolve()` (Law 1). Each `resolve` anchors the ref at the target and registers +the joins it crosses into the CTE's single ordered `join_paths` set, from which +the CTE `FROM` is built. Discovery can no longer be forgotten per carrier: a +cross-model aggregate whose target column's `Column.sql` crosses a *further* +join (`customers.deep_pop:sum` where `deep_pop` is `regions.population`) now +pulls that `LEFT JOIN regions` into the `_cm_*` CTE, and a parametric-agg +column-ref kwarg naming a derived target column expands through the scope +instead of emitting a bare, non-existent column. Routed WHERE/HAVING filters +register their joins in a **pre-pass** that walks the full `ValueKey` tree +(nested arithmetic/boolean/IN operands + aggregate leaves' source/args/kwargs/ +`column_filter`) before the `FROM` is built, so a HAVING — rendered later, once +the ranked-subquery rank columns exist — still contributes its joins. + +**First/last value materialization (Law 2).** When the CTE wraps its rows in a +`ROW_NUMBER`-ranked subquery and the first/last **source value** crosses a join, +the crossing value is materialized as a `_val_` projection *inside* the +subquery and the outer `MAX(CASE WHEN _last_rn = 1 THEN _val_ END)` references +the alias — a raw crossing ref there is bound only inside the subquery. A HAVING +on the same aggregate binds the same alias. The **shared grain** obeys the same +law (DEV-1728): a crossing derived grain is materialized as a `_val_` +projection inside the subquery, the outer `SELECT`/`GROUP BY` reference the +alias, and `PARTITION BY` keeps the raw expression (evaluated where the join is +bound). `generate_from_planned` installs one generation-wide `AliasAllocator` +(save/restore) so inline forward CTEs, grain projections, and the host base +never collide on `_val_`; the CTE reserves the target's physical column names +so a minted alias never shadows a `target.*` column. + +### Derived shared-grain rendering (DEV-1728) + +A cross-model aggregate can be grouped by a joined **derived** dimension +(`{"dimensions": ["customers.rev_x2"], "measures": ["customers.revenue:sum"]}`, +where `rev_x2` is a `Column.sql` on `customers`). The grain loop expands the +derived column's `Column.sql` rooted at the target relation, adds it to the CTE +`SELECT` + `GROUP BY` under the **dotted** host alias +(`orders.customers.rev_x2` — the same alias the host base projects since +DEV-1713), and joins back null-safe. A derived expression that crosses a +*further* join (`deep_pop` = `regions.population`) pulls that join into the CTE +`FROM` via the same `ScopeFrame` machinery the derived-time-dimension grain uses; +a plain derived grain is wrapped in the same `CAST(... AS )` the host base +applies so the join-back compares identically-typed values. Only an +**intermediate-hop** grain (a grain on a middle hop of a multi-hop aggregate +target path) is still unrendered — it raises the same 7b.12 `NotImplementedError` +a base column on an intermediate hop does. + +### Null-safe grain join-back + +The combined-SELECT `LEFT JOIN _cm_* ON` grain equality uses a dialect-aware +null-safe predicate (`SqlDialect.build_null_safe_eq`): `IS NOT DISTINCT FROM` +on Postgres/DuckDB/Snowflake/BigQuery/Trino/Presto/Databricks/Spark/ClickHouse, +`<=>` on MySQL, bare `IS` on SQLite (the native form needs SQLite ≥3.39), and +the expanded `a = b OR (a IS NULL AND b IS NULL)` on T-SQL/Oracle/Redshift. A +plain `=` would yield `NULL` for `NULL = NULL`, so a NULL dimension value or a +nullable truncated time grain would drop its joined-back aggregate; the +null-safe form retains it. + +## Known limitations (documented, not blocking) + +- A host-local filter on a **no-dimension** cross-model-agg query is applied + nowhere (the empty `_base` placeholder doesn't filter; host-local filters are + excluded from the re-rooted CTE). Semantically ambiguous; rare. +- `time_shift` / `consecutive_periods` / `change` / `change_pct` over (or + alongside) a cross-model aggregate raise `NotImplementedError` — factor the + temporal transform into an earlier stage. +- Cross-model parametric-agg result keys diverge from legacy **by design**: + `customers.revenue:percentile(p=0.5)` → `…revenue_percentile_p_0_5` where + legacy dropped the kwarg suffix (`…revenue_percentile`). Legacy's drop was a + collision bug; the new path keeps the suffix. This violates **P10** for this + one combination and is tested structurally, not by parity. See + [the deviations list](index.md#deviations-from-the-plan). +- A cross-model parametric-agg kwarg naming a **target** column + (`customers.revenue:weighted_avg(weight=customers.qty)`) is supported and + expands through the CTE `ScopeFrame` (DEV-1708). The kwarg must be + **relation-qualified** — a bare `weight=qty` resolves against the host by DSL + rule and raises at bind time. A *host-local* weight column evaluated inside + the target CTE remains unsupported. +- A **shared-grain dimension on an intermediate hop** of a multi-hop aggregate + target path (base or derived) raises the 7b.12 `NotImplementedError` — use the + terminal-target path or pull the dimension to the host base. (A plain derived + grain on the *terminal* target path is fully rendered — see + [Derived shared-grain rendering](#derived-shared-grain-rendering-dev-1728).) + +## Design rationale + +- **Why a Protocol (I1)?** So the cross-model strategy is substitutable without + touching the plan shape or the generator. The re-rooting case shows the value: + it was added as a *second* population path for the same `CrossModelAggregatePlan` + struct, not as a new struct. +- **Why route filters in the planner, not the generator?** So the generator + renders each route mechanically. Classification needs the slot graph (which + slot is on which branch); putting it in the planner keeps the generator a + straight `WHERE`/`HAVING`/`CASE-WHEN` emitter. +- **Why re-root rather than emit a literal JOIN chain inside the CTE?** Parity + with legacy `_build_rerooted_enriched` for the grain-preserving case; emitting + the chain directly was the path not taken, and re-rooting reuses the whole + planner recursively, which is less code than a bespoke chain emitter — at the + cost of the second-strategy complexity above. diff --git a/docs/architecture/engine-orchestration.md b/docs/architecture/engine-orchestration.md new file mode 100644 index 00000000..106c6a42 --- /dev/null +++ b/docs/architecture/engine-orchestration.md @@ -0,0 +1,121 @@ +# Engine orchestration + +**Modules:** `slayer/engine/query_engine.py` (`_execute_pipeline`, +`save_model`), `slayer/engine/variables.py` + +`SlayerQueryEngine` is where the pipeline is wired into a runnable execution. It +also marks the boundary between the new typed pipeline and the legacy stack that +still co-exists. + +## `execute` → `_execute_pipeline` + +`execute(query, …)` dispatches over the input shape (str run-by-name, dict, list +DAG, `SlayerQuery`), then `_execute_pipeline` runs the linear pipeline: + +```mermaid +flowchart TB + pre["strip_source_model_prefix + snap_to_whole_periods"] + pre --> bundle["build_resolved_source_bundle (P11)"] + bundle --> qbexp["_expand_query_backed_model (LEGACY path)
source / referenced / stage-source models"] + qbexp --> norm["_normalize_stage → normalize_query (P0)"] + norm --> vars["apply_variables_to_query"] + vars --> plan["plan_stages (root last)"] + plan --> gen["generate_planned_stages → SQL"] + gen --> meta["build_response_metadata"] + meta --> exec["client.execute → SlayerResponse"] +``` + +The new typed pipeline is `build_resolved_source_bundle → _normalize_stage → +apply_variables_to_query → plan_stages → generate_planned_stages → +build_response_metadata`. Storage is consulted once, in +`build_resolved_source_bundle` (P11). + +`_normalize_stage` resolves each stage's source model from the bundle so +`MISPLACED_MEASURE` and custom-aggregation-aware `FUNC_STYLE_AGG` see the right +column / aggregation names; a sibling-sourced stage normalizes with `model=None`. +Slack warnings from every stage are collected and surface on +`SlayerResponse.warnings`. + +`_touched_models_for_plan` collects the model names a query-time DBAPI error +could be attributed to (bundle referenced models + cross-model targets + +query-backed base names) for schema-drift attribution. + +## Variables (`variables.py`) + +`merge_query_variables` collapses the four layers — **runtime > stage > outer > +model defaults** — into the effective dict that populates +`ResolvedSourceBundle.query_variables`. `apply_variables_to_query` returns a fresh +`SlayerQuery` with `{var}` substituted in `filters` (the only field legacy +substituted; formula text, `Column.sql`, `Column.filter`, `SlayerModel.filters` +are deliberately not substituted). `dry_run_placeholders=True` fills unresolved +valid placeholders with `"0"` (the legacy save-time dry-run behavior); invalid +names still raise. + +## `save_model` + +`save_model` runs `normalize_model` (the [slack layer](slack-normalization.md)) +so persisted formulas land canonical, then persists. For a **query-backed** model +it rejects user-supplied cache fields and calls `_validate_and_populate_cache`, +which renders the backing query and stores `columns` / `backing_query_sql` / +`data_source`. + +## Query-backed model expansion + +`_expand_query_backed_model` turns a model's `source_queries` into a virtual +`sql`-mode model whose `.sql` is the rendered backing query. It mirrors +`_execute_pipeline`'s mid-section (bundle → expand-nested → normalize → +variables → `plan_stages` → `generate_planned_stages`) and wraps the result in a +flat-rename SELECT so the virtual model exposes downstream-bindable flat +columns. The pipeline then treats that virtual model as a plain `sql`-mode model +and plans/renders the **outer** query the same way. + +`_execute_pipeline` invokes it for the source model, for query-backed referenced +(join/cross-model target) models, and for non-root stage sources; `save_model` → +`_validate_and_populate_cache` uses the same path for its save-time dry run. One +renderer, one set of semantics, in every case. + +!!! note "Historical: the two-pipeline period" + + Between the DEV-1450 cutover and DEV-1485 there were **two** rendering + stacks. The cutover routed top-level query planning through the typed + pipeline, but query-backed expansion kept running on the legacy + `_query_as_model` → `enrich_query` → `SQLGenerator.generate(enriched=…)` + path in production. DEV-1452 Stage B migrated expansion onto the typed + pipeline, and DEV-1485 (Stage D) deleted the legacy stack outright — + `enrichment.py`, `enriched.py` (`EnrichedQuery` / `EnrichedMeasure`), + `_query_as_model`, the legacy `SQLGenerator.generate`, and the + `_forbidden_sibling_refs_var` / `_join_target_resolving_var` `ContextVar`s + are all gone. Sibling stages resolve through `_follow_sibling_chain` in + `source_bundle.py`; forward / self / cycle references are caught by + `topologically_order_stages` up front. + + Documentation, commit messages, and issues written during that period may + still describe the legacy path as load-bearing. It is not — it no longer + exists. + +## Pre-processing before the "single" slack pass + +`_execute_pipeline` runs `strip_source_model_prefix()` and (when +`whole_periods_only`) `snap_to_whole_periods()` *before* `_normalize_stage`. +These are query-shape transforms rather than slack-token rewrites, but they mean +the pipeline does not literally "begin with a single slack-normalization pass" +(**P0**). A minor deviation, noted for completeness in +[the deviations list](index.md#deviations-from-the-plan). + +## Design rationale + +- **Why split the cutover this way?** Bisectability. Flipping the outer query + while leaving query-backed expansion on legacy let the cutover land with all + non-integration tests green and integration green, without a single + thousand-line "delete everything" commit. The cost is the temporary + two-pipeline coexistence above. +- **Why does query-backed expansion produce a virtual `sql`-mode model rather + than planning the inner stages directly?** Because the outer typed pipeline + already knows how to consume a `sql`-mode model. Re-expressing a query-backed + model as `{sql_table: None, sql: }` lets the outer + planner stay oblivious to query-backedness — at the cost of rendering the inner + SQL through the legacy generator for now. DEV-1452's job is to make the inner + rendering go through `plan_stages` / `generate_planned_stages` too. +- **Why consult storage only in the bundle builder (P11)?** So everything after + it is pure and order-independent. The legacy `ContextVar` re-resolution exists + *only* on the query-backed/legacy path; the new path has none. diff --git a/docs/architecture/errors-and-warnings.md b/docs/architecture/errors-and-warnings.md new file mode 100644 index 00000000..8431e482 --- /dev/null +++ b/docs/architecture/errors-and-warnings.md @@ -0,0 +1,80 @@ +# Errors and warnings + +**Modules:** `slayer/core/errors.py`, `slayer/core/warnings.py` + +The redesign replaced anonymous `ValueError`s scattered through enrichment with a +typed error vocabulary. Each error carries the offending input, a scope summary, +and (where feasible) a did-you-mean suggestion, and renders with a **stable +`str()` format** so tests can snapshot it. + +## The stable message format + +`_format_error_message` builds every stage-5 error's message in one shape: + +```text +: + at + scope: + suggestion: +``` + +The first line always begins with the class name, so log greps and snapshot +tests bind to a stable prefix; the indented lines are optional. + +## The error classes + +| Class | Raised when | +| --- | --- | +| `UnknownReferenceError` | a bare or dotted ref doesn't resolve in scope | +| `AmbiguousReferenceError` | a ref matches multiple candidates in scope | +| `IllegalScopeReferenceError` | a dotted ref against a `StageSchema`, or `__` in a `ModelScope` without an exact match | +| `IllegalWindowInFilterError` | raw `OVER(...)` in a DSL filter, or a filter referencing a windowed `Column.sql` | +| `AggregationNotAllowedError` | type-bucket / PK / `allowed_aggregations` violation | +| `UnknownFunctionError` | a Mode-B call not in `SCALAR_FUNCTIONS` / transforms / aggregations | +| `MeasureRecursionLimitError` | named-measure expansion exceeded depth (32, env-configurable) | +| `MeasureCycleError` | a cycle in named-measure expansion | +| `DuplicateMeasureNameError` | two measures declare the same `name` | +| `MeasureNameCollidesWithColumnError` | a declared `name` matches a source column | +| `CanonicalAliasShadowsColumnError` | a formula's canonical alias shadows a source column | + +### `ValueError` multi-inheritance for back-compat + +`UnknownReferenceError`, `AmbiguousReferenceError`, `IllegalScopeReferenceError`, +and `IllegalWindowInFilterError` multi-inherit `ValueError` (alongside +`SlayerError`). This is deliberate: the cutover replaced legacy `ValueError` +resolution paths with these typed errors, and many pre-existing call sites and +tests catch `ValueError` (or use `pytest.raises(ValueError)`). Multi-inheriting +keeps them working unchanged. `ColumnCycleError` (DEV-1410) does the same. + +`SlayerError` is the base for SLayer's intentional failure modes, so callers can +distinguish them from unexpected `Exception` paths (driver errors, IO errors). + +## Warnings + +Two warning types are **not** exceptions: + +- `SlayerNormalizationWarning` (`core/warnings.py`) — a `UserWarning` carrying a + `NormalizationWarning` payload, emitted by the + [slack layer](slack-normalization.md) on every rewrite. Surfaced both via + `warnings.warn(...)` and on `SlayerResponse.warnings`. +- `UnreachableFilterDroppedWarning` (`core/errors.py`) — a `UserWarning` emitted + by the [cross-model planner](cross-model-aggregates.md) when a host filter + references slots unreachable from a CTE's root, so the filter is dropped from + the CTE (the host still applies it to its own rows). A visibility/debug + warning, not an error. + +## Design rationale + +- **Why typed errors over `ValueError`?** The legacy enrichment raised bare + `ValueError`s that callers couldn't distinguish, so error handling was + string-matching on messages. Typed classes let surfaces (REST/MCP/CLI) and + tests react to the *kind* of failure, and the `.name` / `.scope_summary` / + `.suggestion` attributes make programmatic remediation possible. +- **Why a stable `str()` format?** So snapshot tests can pin the message + (including suggestion text and scope summary) without brittle substring + matches, and so agents reading the error get a consistent, parseable shape. +- **Why keep `ValueError` in the MRO?** A clean break would have churned every + `except ValueError` call site and test in the same PR as the cutover. Multi- + inheriting defers that churn without weakening the new typed surface — callers + that want the specific class can catch it; callers that catch `ValueError` + still work. diff --git a/docs/architecture/index.md b/docs/architecture/index.md new file mode 100644 index 00000000..a052ba03 --- /dev/null +++ b/docs/architecture/index.md @@ -0,0 +1,248 @@ +# Architecture: the typed resolution pipeline + +This section documents how SLayer turns a `SlayerQuery` into SQL — the typed, +composable pipeline introduced by the DEV-1450 redesign. It describes the code +**as currently implemented**, not the original plan; where the implementation +diverges from the plan, [Deviations from the plan](#deviations-from-the-plan) +calls it out. + +The audience is contributors. If you only write queries, read +[Concepts](../concepts/queries.md) instead. + +## Why the redesign + +The expressive surface syntax (dotted joins, colon aggregation, transforms, +renamed measures, cross-model aggregates) used to be resolved by a single large +enrichment pass (`slayer/engine/enrichment.py`, ~2300 lines) that interleaved +string rewriting, alias remapping, a parallel `cross_model_measures` track, +virtual-model flattening, and implicit passthrough. Every new permutation of +"custom name × join × transform" added another resolution path, and the paths +interacted in ways that produced corner-case bugs (DEV-1445/1446/1448/1449). + +The redesign replaces that with a pipeline of small stages, each taking +well-typed input and producing a well-typed intermediate object that carries +everything the next stage needs. **Identity is structural, not textual** — the +single idea that makes the four bugs structurally impossible rather than +individually patched. + +## The pipeline at a glance + +```mermaid +flowchart LR + raw["SlayerQuery
(raw, slack-tolerant)"] --> norm["slack normalize
(normalization.py)"] + norm --> parse["parse_expr
(syntax.py)"] + parse --> expand["expand measures
(measure_expansion.py)"] + expand --> bind["bind_expr / bind_filter
(binding.py)"] + bind --> plan["plan_query / plan_stages
(stage_planner.py)"] + plan --> planned["PlannedQuery
(planned.py)"] + planned --> gen["generate_planned_stages
(generator.py)"] + gen --> sql["SQL string"] +``` + +Each arrow is the typed-object boundary from principle **P7** +(`raw → NormalizedInput → ParsedExpr → BoundExpr → ValueSlot → PlannedQuery → +SQL`). No stage string-rewrites the output of a previous stage after parsing. + +A more detailed view, showing the planner's internal sub-stages and the source +bundle that feeds resolution: + +```mermaid +flowchart TB + subgraph orchestrator["engine.execute → _execute_pipeline (query_engine.py)"] + bundle["build_resolved_source_bundle
(source_bundle.py) — storage read once (P11)"] + norm["_normalize_stage → normalize_query (P0)"] + vars["apply_variables_to_query (variables.py)"] + plan["plan_stages (stage_planner.py)"] + gen["generate_planned_stages (generator.py)"] + meta["build_response_metadata (response_meta.py)"] + end + + bundle --> norm --> vars --> plan --> gen --> meta + + subgraph perstage["plan_query — per stage"] + decl["parse + expand + bind
declared measures / dims / TDs"] + filt["bind_filter
filters (phase-classified, P8)"] + td["attach time_key → transforms"] + sugar["lower_sugar_transforms (change/change_pct)"] + proj["ProjectionPlanner + ValueRegistry
intern slots (P2/P4)"] + cm["cross_model_planner.plan
per cross-model aggregate (P3/I1)"] + end + + plan --> decl --> filt --> td --> sugar --> proj --> cm +``` + +## Principles, and where they live + +The redesign was specified as 12 principles (P0–P11). Each maps to a concrete +module: + +| Principle | Statement | Where | +| --- | --- | --- | +| **P0** | Pipeline begins with a single slack-normalization pass; rewrites are returned as typed warnings | [`normalization.py`](slack-normalization.md) | +| **P1** | Two surface languages (Mode A SQL / Mode B DSL), never mixed mid-expression; closed `SCALAR_FUNCTIONS` allowlist | [`syntax.py`](parsing.md), `keys.py` | +| **P2** | Identity is structural, not textual — two equal keys intern to one slot | [`keys.py`](typed-keys.md), `planning.py` | +| **P3** | Local and cross-model aggregates share one `AggregateKey` shape (path empty vs non-empty) | [`keys.py`](typed-keys.md), [`cross_model_planner.py`](cross-model-aggregates.md) | +| **P4** | Public names are a separate namespace; a slot has one declared name + many public aliases | [`planning.py`](planning.md) | +| **P5** | Scope determines what dots mean — `ModelScope` vs `StageSchema`, never confused | [`scope.py`](scopes-and-bundle.md), [`binding.py`](binding.md) | +| **P6** | Each stage emits an explicit `StageSchema`; stages compose only through schemas | [`scope.py`](scopes-and-bundle.md), [`stage_planner.py`](stage-planning.md) | +| **P7** | Typed pipeline; no string rewriting after parse | whole pipeline | +| **P8** | Phase (WHERE/HAVING/post) is a property of the slot, not the filter text | [`keys.py`](typed-keys.md) `Phase`, [`binding.py`](binding.md) | +| **P9** | Transforms are operators over slots, not over strings | [`planning.py`](planning.md), [`binding.py`](binding.md) | +| **P10** | Result-key contract preserved exactly | [`generator.py`](sql-generation.md), `response_meta.py` | +| **P11** | Resolution is pure — storage consulted once, no `ContextVar` re-resolution | [`source_bundle.py`](scopes-and-bundle.md) | + +## Module map + +The new pipeline modules, in dependency order: + +| Module | Role | Doc | +| --- | --- | --- | +| `slayer/core/keys.py` | The `ValueKey` family — structural identity primitives | [Typed keys](typed-keys.md) | +| `slayer/core/scope.py` | `ModelScope`, `StageSchema`, `StageColumn` | [Scopes & bundle](scopes-and-bundle.md) | +| `slayer/core/errors.py`, `warnings.py` | Typed errors + slack-warning carriers | [Errors & warnings](errors-and-warnings.md) | +| `slayer/engine/source_bundle.py` | `ResolvedSourceBundle` + eager builder (P11) | [Scopes & bundle](scopes-and-bundle.md) | +| `slayer/engine/normalization.py` | Slack-normalization layer (P0) | [Slack normalization](slack-normalization.md) | +| `slayer/engine/syntax.py` | Mode-B Python-AST parser → `ParsedExpr` | [Parsing](parsing.md) | +| `slayer/sql/sql_expr.py` | Mode-A sqlglot wrapper | [Parsing](parsing.md) | +| `slayer/engine/measure_expansion.py` | Pre-bind named-`ModelMeasure` expansion | [Parsing](parsing.md) | +| `slayer/engine/binding.py` | `ExpressionBinder` / `FilterBinder` → `BoundExpr` | [Binding](binding.md) | +| `slayer/engine/planning.py` | `ValueRegistry`, `ProjectionPlanner`, transform lowering | [Planning](planning.md) | +| `slayer/engine/cross_model_planner.py` | Cross-model aggregate strategy (I1) | [Cross-model aggregates](cross-model-aggregates.md) | +| `slayer/engine/planned.py` | `PlannedQuery` and its parts | [Planning](planning.md) | +| `slayer/engine/stage_planner.py` | `plan_query` / `plan_stages` orchestrators | [Stage planning](stage-planning.md) | +| `slayer/engine/variables.py` | `{var}` substitution + 4-layer merge | [Engine orchestration](engine-orchestration.md) | +| `slayer/sql/generator.py` | `generate_from_planned` / `generate_planned_stages` | [SQL generation](sql-generation.md) | +| `slayer/engine/response_meta.py` | `attributes` / `expected_columns` from the plan | [SQL generation](sql-generation.md) | +| `slayer/engine/query_engine.py` | `_execute_pipeline` orchestration + cutover | [Engine orchestration](engine-orchestration.md) | + +## The four bugs, made structurally impossible + +The acceptance criterion was that DEV-1445/1446/1448/1449 stop being reachable, +not that each gets a patch. How structural identity achieves that: + +- **DEV-1446** (transform-wrapped agg-ref of a renamed measure deduping): + `change(amount:sum)` and `amount:sum` share the same inner `AggregateKey` + instance, so the `ValueRegistry` interns one slot — `SUM(amount)` appears once. + See [Planning](planning.md). +- **DEV-1445** (cross-model renamed-measure filter by alias *or* dotted form): + `customers.revenue:sum` and the user alias `rev` both bind to one + `AggregateKey`; the filter's `rev` ref resolves through `alias_map` onto that + same slot. See [Binding](binding.md), [Stage planning](stage-planning.md). +- **DEV-1448** (user `name` on a join-traversed measure governs the stage column): + `StageColumn.name` is the declared name, flattened — downstream stages bind + against it. See [Stage planning](stage-planning.md). +- **DEV-1449** (downstream stages see upstream stages as flat schemas): + binding against a `StageSchema` rejects dotted refs with + `IllegalScopeReferenceError`; only flat `__` names resolve. See + [Scopes & bundle](scopes-and-bundle.md), [Binding](binding.md). + +Each has an `engine.execute`-level acceptance test in +`tests/test_dev1445_*.py` / `1446` / `1448` / `1449`. + +## Current state: one pipeline + +The typed pipeline is the only rendering path. Top-level query planning, +query-backed model expansion (execute *and* save), join-target rendering, and +dialect SQL emission all run through it. + +The legacy enrichment stack is **deleted** (DEV-1485, Stage D of DEV-1703): +`enrichment.py`, `enriched.py` (`EnrichedQuery` / `EnrichedMeasure`), +`_query_as_model`, the legacy `SQLGenerator.generate(enriched=…)`, and the +`_forbidden_sibling_refs_var` / `_join_target_resolving_var` `ContextVar`s no +longer exist. Sibling stages resolve through `_follow_sibling_chain` in +`source_bundle.py`; forward / self / cycle references are caught up front by +`topologically_order_stages`. + +Older documents, commit messages, and Linear issues describe a period when two +pipelines coexisted and the legacy stack was load-bearing for query-backed +inner rendering. That is history — see the note in +[Engine orchestration](engine-orchestration.md). + +## Deviations from the plan + +These are places where the implemented code departs from the DEV-1450 plan. +They are documented here so reviewers don't mistake them for the intended end +state. All are deliberate and tracked, but several reintroduce — temporarily — +the kind of multi-path coupling the redesign set out to remove. + +1. ~~**Legacy stack still load-bearing for query-backed models.**~~ **Resolved.** + This was the largest gap between plan and reality: the cutover deferred every + deletion, and the legacy stack rendered the backing SQL of every query-backed + model. DEV-1452 Stage B migrated expansion onto the typed pipeline and + DEV-1485 (Stage D) deleted the legacy stack, so the plan's stage-7b bullet + ("delete `EnrichedQuery`, `EnrichedMeasure`, … `_query_as_model`, … legacy + `SQLGenerator.generate`") is now satisfied in full. + +2. **A second cross-model rendering path was needed** (re-rooting). The plan's + cross-model design was a single strategy: `IsolatedCteCrossModelPlanner` plus + the `inherited_filter_policy` decision table, producing one CTE per + `(target, grain)`. That proved insufficient: when host dimensions are + reachable from the target through the *target's own* join graph, the + forward-path CTE collapses the host grain to a scalar `CROSS JOIN`. The fix + (`_maybe_reroot_cross_model_plan` in `stage_planner.py`, rendered by + `_render_rerooted_cross_model_cte` in `generator.py`) builds a full nested + re-rooted `PlannedQuery` — mirroring legacy `_build_rerooted_enriched`. So + there are now **two** cross-model render strategies selected heuristically, + bolted onto `CrossModelAggregatePlan` via `rerooted_plan` / + `rerooted_grain_pairs` / `rerooted_agg_slot_id`. This is the most significant + architectural compromise — the "one shape, render strategy chosen downstream" + abstraction (P3) holds for identity but not for rendering. See + [Cross-model aggregates](cross-model-aggregates.md). + +3. ~~**The new generator adapts back to `EnrichedMeasure`.**~~ **Resolved.** + `generate_from_planned` briefly synthesized `EnrichedMeasure` objects to + reuse the dialect helpers (`_build_agg`, `_build_percentile`, + `_build_stat_agg`, …), coupling the new path to a legacy type. DEV-1452 + Stage A retyped those helpers onto `AggRenderSpec`, built directly from + planned slots by `_build_agg_render_spec_from_planned`, and DEV-1485 removed + the last adapter (`_agg_render_spec_from_enriched`) along with `_build_agg`'s + `measure=` compat surface. See [SQL generation](sql-generation.md). + +4. **Derived-column parity with legacy restored (DEV-1450 follow-ups #4a / #4b).** + Two cases that legacy handled, and the typed pipeline briefly narrowed to + `NotImplementedError`, now work again: + - A `TimeDimension` over a derived (`Column.sql`) temporal column. + `TimeTruncKey.column` is `Union[ColumnKey, ColumnSqlKey]`; the binder + (`bind_time_dimension`) and every generator render site apply the + `DATE_TRUNC` over the EXPANDED derived expression (base SELECT, ORDER BY, + window/OVER transforms, the time_shift self-join CTE, `date_range` + BETWEEN, and the cross-model shared-grain CTE). See + [Typed keys](typed-keys.md) and [SQL generation](sql-generation.md). + - A `SlayerModel.filters` entry, OR a column-level `Column.filter` on an + aggregated measure, referencing a non-trivial derived column. + `_validate_model_filter` no longer rejects the model-filter form; the + generator inline-expands the predicate (the shared + `_render_mode_a_predicate`, used by both `_render_model_filter_sql` and the + `Column.filter` CASE-WHEN path) and pulls any join the expansion crosses + into the FROM. Inlining covers a bare derived ref (`is_eu` → + `customers.region`) **and** a dotted ref to a derived column on a joined + model (`loss_payment.has_flag` → its `sql`), matching the query-level + filter path so no dangling `.` (a non-physical column) + is emitted (DEV-1494). Join discovery for these Mode-A text filters + (`_filter_join_paths`) unions the paths of the **un-inlined** predicate + (so the dbt placeholder-join idiom — a constant `has_flag sql="1"` whose + only purpose is to force the join — keeps its alias) with the paths the + **inline-expanded** predicate crosses. The cross-model `_cm_*` CTE discovers + its OWN filter joins too — the target measure's `Column.filter` and the + target-model filters — and adds them to the CTE's FROM (each `_cm_*` CTE is + an isolated per-(target, grain) computation, so the join resolves the + filter's refs without affecting sibling measures). The windowed-`Column.sql` + and same-model `ModelMeasure`-ref rejects remain. + +5. **P10 is intentionally violated for cross-model parametric aggregates.** + Result keys for `customers.revenue:percentile(p=0.5)` now carry the kwarg + signature (`…revenue_percentile_p_0_5`) where legacy dropped it + (`…revenue_percentile`). Legacy's drop was a collision bug (two parametric + variants on one column produced the same alias); the new path fixes it but + at the cost of bit-identical parity. Tested structurally, not by parity. + +6. **Pre-processing runs before the "single" slack pass (P0).** + `_execute_pipeline` runs `strip_source_model_prefix()` and + `snap_to_whole_periods()` on the query *before* `_normalize_stage`. These are + query-shape transforms rather than slack-token rewrites, but they mean the + pipeline does not literally "begin with a single slack-normalization pass". + +Test-only deviations (parity oracle replacing the planned parity adapter; the +two retained `@pytest.mark.skip`s in `tests/test_filter_renamed_measure.py`; +production extractors using a scope-free `walk_parsed_refs` instead of binding) +are noted in the relevant component docs and are not architectural concerns. diff --git a/docs/architecture/parsing.md b/docs/architecture/parsing.md new file mode 100644 index 00000000..1129ab8e --- /dev/null +++ b/docs/architecture/parsing.md @@ -0,0 +1,158 @@ +# Parsing + +**Modules:** `slayer/engine/syntax.py` (Mode B), `slayer/sql/sql_expr.py` +(Mode A), `slayer/engine/measure_expansion.py` (pre-bind expansion) + +Parsing turns expression strings into typed `ParsedExpr` trees. It is **pure +syntax** — no scope resolution, no named-measure expansion, no function-style +rewriting. Those are separate stages (the [slack layer](slack-normalization.md) +does function-style → colon; the [binder](binding.md) does scope; expansion is +its own step). This separation is what keeps each stage small. + +```mermaid +flowchart LR + text["expr string
'change(amount:sum) > 0'"] --> pp["_preprocess_colons
amount:sum → placeholder"] + pp --> ast["ast.parse(mode='eval')"] + ast --> conv["_convert
AST → ParsedExpr"] + conv --> tree["ParsedExpr tree"] +``` + +## The `ParsedExpr` family + +Eleven frozen Pydantic node types with value-based equality (so tests assert via +`==`): + +| Node | Shape | +| --- | --- | +| `Ref` | `name` — a bare identifier | +| `DottedRef` | `parts` — a dotted path | +| `StarSource` | `*` | +| `Literal` | `value` (`Decimal` / `str` / `bool` / `None`) | +| `AggCall` | `source, agg, args, kwargs` — colon aggregation | +| `TransformCall` | `op, input, args, kwargs` | +| `ScalarCall` | `name, args` | +| `Arith` | `op, left, right` | +| `UnaryOp` | `op, operand` | +| `Cmp` | `op, left, right` | +| `BoolOp` | `op, operands` | + +## How `parse_expr` works + +Mode B is a *Python-AST* DSL — the grammar is a deliberate subset of Python +expression syntax, so the parser leans on `ast.parse(..., mode="eval")` rather +than a hand-rolled grammar. Two pre/post steps make the colon and `__` rules +work: + +1. **`_preprocess_colons`** replaces `:` with a placeholder + identifier (`__slayer_agg_N__`) before handing the text to Python's parser, + capturing the source kind (`*` / `Ref` / `DottedRef`) and agg name in a side + map. Any trailing `(args)` is left in place so Python parses it as a `Call` + naturally. String-literal spans are skipped so quoted contents aren't touched. +2. **`_reject_dunder_in_ast`** walks the parsed AST and rejects any user + identifier containing `__` (on `Name`, `Attribute.attr`, and `keyword.arg`), + unless `allow_dunder=True`. `__` is reserved for internal join-path aliases on + the SQL side; users write single-dot DSL paths. + +`_convert` then maps AST nodes to `ParsedExpr` nodes. A `Call` dispatches in a +fixed order: aggregation placeholder → transform (in `ALL_TRANSFORMS`, requires +≥1 positional) → scalar (in `SCALAR_FUNCTIONS`, rejects kwargs) → otherwise +`UnknownFunctionError`. List/tuple kwarg values (e.g. `partition_by=[a, b]`) +convert to a tuple of converted elements. + +### Rejections (P1) + +The parser is where the Mode-B contract is enforced: + +- a function call not in `SCALAR_FUNCTIONS` / `ALL_TRANSFORMS` / aggregations → + `UnknownFunctionError`; +- a raw `OVER(...)` clause anywhere in the text → `IllegalWindowInFilterError` + (checked by regex before AST parsing); +- `__` in a user identifier → `ValueError` (unless `allow_dunder`); +- chained comparisons (`1 < x < 10`) → `ValueError` (split into `1 < x and x < + 10`); the binder can't give a chained comparison a single phase. + +### `allow_dunder` — the StageSchema escape hatch (DEV-1449) + +`parse_expr(text, *, allow_dunder=False)` defaults to rejecting `__`. The +[stage planner](stage-planning.md) sets `allow_dunder=True` *only* when binding a +downstream stage against a flat `StageSchema`, whose columns **are** the +`__`-flattened multi-hop aliases of the upstream stage (`customers__region`). +Legality there is the binder's concern (the column must exist in the upstream +schema). This is the one place `__` is legal in a Mode-B ref, and it is exactly +what makes a downstream stage able to name an upstream joined dimension. + +### `parse_filter_expr` — SQL-operator leniency + +Filters historically accepted SQL operator spellings (`=`, `<>`, `NULL`, keyword +`AND`/`OR`/`NOT`/`IS`/`IN`) alongside Python ones. `parse_filter_expr` normalizes +those to Python equivalents (string-literal-aware) via +`_normalize_sql_filter_operators`, then delegates to `parse_expr`. Measures and +order parse with `parse_expr` directly; only filters get the leniency — matching +the legacy `parse_filter` contract. + +### `walk_parsed_refs` — scope-free reference extraction + +`walk_parsed_refs(parsed)` yields the reference-bearing leaves (`Ref`, +`DottedRef`, `AggCall`) of a tree without binding it. It is the scope-free +counterpart to the binder's `walk_value_keys`: production extractors that only +need the *names* a formula touches — schema-drift cascade attribution and memory +entity tagging — walk the parse tree directly instead of binding against a scope. +Its descent rules match the legacy `parse_formula` walk exactly (an `AggCall` is +yielded as a unit and its args/kwargs are *not* descended, so +`weighted_avg(weight=quantity)` surfaces `price`, never `quantity`). + +> **Deviation note.** The plan specified walking the typed-key `BoundExpr` via +> `walk_value_keys` for these extractors. That is infeasible: binding raises on +> bare named-measure refs (which need planner-side expansion) and resolves the +> very refs drift detection must find *pre*-resolution. `parse_expr` + +> `walk_parsed_refs` was the user-approved alternative. + +## Mode A — `sql_expr.py` + +Mode A (`Column.sql`, `Column.filter`, `SlayerModel.filters`) is sqlglot-native. +`parse_sql_expr` wraps the fragment as `SELECT () AS _` before sqlglot +parses it — necessary because sqlglot's SQLite/MySQL parser otherwise falls back +to a `Command` node for a top-level `replace(...)`. `has_window_function` is the +predicate the binder uses to reject filters that touch a windowed `Column.sql` +(DEV-1369). Mode A keeps full SQL expressiveness; the typed pipeline only needs +to detect windows and (in the slack layer) rewrite multi-dot paths. + +## Pre-bind measure expansion — `measure_expansion.py` + +The binder raises `UnknownReferenceError` for a bare *measure* name (measures +aren't columns). `expand_model_measures` runs *before* binding: it is an +AST → AST rewrite that replaces every `Ref(name=X)` whose `X` is a saved +`ModelMeasure` with the recursively-expanded `parse_expr(measure.formula)` tree, +turning measure refs into binder-resolvable column/aggregation nodes. + +```mermaid +flowchart LR + r["Ref('aov')"] -->|aov is a ModelMeasure| sub["parse_expr(aov.formula)"] + sub --> walk["recursively expand its refs"] + walk --> out["expanded ParsedExpr"] +``` + +Eligibility is principled: expansion fires at the root and in `Arith` / `UnaryOp` +/ `Cmp` / `BoolOp` operands, `ScalarCall.args`, and `TransformCall.input` / args +/ kwarg values. It does **not** fire on `DottedRef` segments (those resolve +through joins), `AggCall` in any position (sources/args/kwargs are column-level +by contract), or function-name slots. Recursion is bounded: depth limit 32 +(configurable via `SLAYER_MEASURE_EXPANSION_DEPTH`) raising +`MeasureRecursionLimitError`, plus per-chain cycle detection raising +`MeasureCycleError` with the offending chain. A `parse_cache` memoizes each +measure's parse. The node-type tuple is derived from the `ParsedExpr` union via +`get_args`, so a new node type added to `syntax.py` is automatically walked. + +## Design rationale + +- **Why reuse Python's AST for Mode B?** The DSL was always a Python-expression + subset; `ast.parse` gives precedence, grouping, and operator handling for free, + and the conversion layer stays small. The colon preprocessor is the only piece + that bridges the one construct Python doesn't have. +- **Why is parsing pure (no scope)?** So the same parser serves the binder, the + measure expander, and the scope-free extractors. Mixing in resolution would + re-couple parsing to the model graph — the coupling the redesign removes. +- **Why a separate expansion pass instead of expanding in the binder?** Expansion + is an AST → AST rewrite with its own recursion/cycle concerns; keeping it + before binding means the binder only ever sees column/aggregation refs and can + stay a straight scope lookup. diff --git a/docs/architecture/planning.md b/docs/architecture/planning.md new file mode 100644 index 00000000..5937d598 --- /dev/null +++ b/docs/architecture/planning.md @@ -0,0 +1,170 @@ +# Planning: interning, projection, and the plan shape + +**Modules:** `slayer/engine/planning.py` (ValueRegistry, ProjectionPlanner, +transform lowering), `slayer/engine/planned.py` (the `PlannedQuery` types) + +Planning turns bound expressions into a `PlannedQuery` — the fully resolved, +render-ready target. Three composable concerns live in `planning.py`; the typed +result types live in `planned.py`. The [stage planner](stage-planning.md) +composes them. + +## `ValueRegistry` — interning by structural identity (P2 / P4) + +The registry maps `ValueKey → ValueSlot`. `intern(...)` either returns the +existing slot for a structurally-equal key or allocates a fresh one. This is the +mechanism behind **P2**: `change(amount:sum)` and a filter `amount:sum` build the +same inner `AggregateKey`, so they intern to one slot and `SUM(amount)` is +emitted once (DEV-1446). + +```mermaid +flowchart TB + k1["AggregateKey(amount, sum)
from measure"] --> reg + k2["AggregateKey(amount, sum)
inner of change()"] --> reg + k3["AggregateKey(amount, sum)
from filter"] --> reg + reg["ValueRegistry.intern"] --> slot["one ValueSlot
id=s1"] +``` + +### Public names are a separate namespace (P4 / C13) + +A slot has at most one *declared name* but can accumulate **multiple** +`public_aliases` — if the same structural key is declared with two different +user `name`s, both aliases appear in the projection pointing at one slot +(`_merge_into_existing`). A filter/order expression may reference a declared name +as an alias for the slot, but cannot *synthesize* a new slot from a +canonical-looking bare name when no corresponding measure was declared. + +### Alias-collision validations (DEV-1443) + +`intern` enforces three validations against the host model's column names: + +- a declared `public_name` colliding with a source column → + `MeasureNameCollidesWithColumnError`; +- a `canonical_alias` (e.g. `amount_sum`) shadowing a source column → + `CanonicalAliasShadowsColumnError`; +- two different keys declaring the same `public_name` → + `DuplicateMeasureNameError`. + +With carefully chosen exemptions: a *self-named dimension* (a `ColumnKey` / +`ColumnSqlKey` / `TimeTruncKey` whose public name **is** its own column name) is +the column, not a rename, so the collision check is skipped; and an unnamed +`*:` re-aggregation (whose canonical `_count` is a structural marker, not a +column ref) is exempt. + +## Transform lowering (P9 / C6) + +`change` and `change_pct` are sugar. `desugar_change` rewrites +`change(x)` → `x - time_shift(x, periods=-1)` and `desugar_change_pct` +→ `(x - time_shift(x, -1)) / time_shift(x, -1)`. The inner `x` is the **same +`ValueKey` instance** across the arithmetic and the time_shift, so a downstream +registry interns it once — this is the identity-preservation that makes DEV-1446 +hold even through desugaring. `partition_by` and `time_key` thread through to the +underlying `time_shift` (**C6**). + +`lower_sugar_transforms(key)` is the recursive walker that applies the desugar +functions anywhere in a `ValueKey` tree (`TransformKey` / `ArithmeticKey` / +`ScalarCallKey` / `BetweenKey`), rebuilding only the path that contains a +change/change_pct so identity is preserved elsewhere. The stage planner runs it +*after* `time_key` patching so the desugared `time_shift` inherits the patched +key. + +## `ProjectionPlanner` — declared + hidden slots + +`ProjectionPlanner.plan(...)` interns each declared measure (in dim → time-dim → +measure order) into the registry, builds the public projection, and then +materializes **hidden** slots for any value referenced only in filters / order +or as an auxiliary dependency of a declared measure. + +The dependency-selection rule is `_iter_slot_deps`, and it encodes which keys +need a materialised slot versus which the generator inlines: + +| Key | Slotted? | +| --- | --- | +| `ColumnKey` / `ColumnSqlKey` / `TimeTruncKey` | yes (row slot) | +| `AggregateKey` | yes (stops — its inner source materializes inside the aggregate) | +| `TransformKey` | yes, **and** recurse into `input`, `partition_keys`, `time_key` | +| `ArithmeticKey` / `ScalarCallKey` | no — recurse into operands/args; the op/call is inlined | +| `BetweenKey` | no — inlined into WHERE; recurse into column/low/high | +| `LiteralKey` / `StarKey` | never slottable alone | + +So `ORDER BY revenue:sum DESC LIMIT 10` with no declared `revenue:sum` measure +interns the aggregate as a `hidden=True` slot: the base CTE materializes it, the +outer SELECT trims it from the public projection, and `StageSchema.columns` +excludes it (downstream stages see no extra column). The same rule covers +filter-only refs. The no-transform "plain" path follows the same pattern via a +conditional outer-trim wrapper (DEV-1501): the wrap fires only when the base +materialises a hidden slot, so simple flat queries stay flat. Hidden parametric +aggregates (`revenue:last(created_at)` vs `revenue:last(updated_at)`, +`revenue:percentile(p=0.5)` vs `…(p=0.95)`) route their declared name through +`canonical_agg_name` so the args/kwargs surface in the materialised alias — +two distinct hidden parametric aggregates get distinct base-CTE aliases instead +of colliding on `revenue_last` / `revenue_percentile`. + +`filter_referenced_slot_ids(bound_filter, registry)` walks the predicate via +`_iter_slot_deps` and looks each dep up in the registry, returning `set[SlotId]` +— the input the [cross-model planner](cross-model-aggregates.md) needs for filter +routing (it gets slot ids, not pre-interning `ValueKey`s, and it sees +composite-predicate leaves rather than just the top-level key). + +## The `PlannedQuery` shape (planned.py) + +`PlannedQuery` is the typed target the [SQL generator](sql-generation.md) +consumes. It carries everything needed to emit SQL without re-walking the model +graph (**P7**): + +| Field | Role | +| --- | --- | +| `source_relation` | the FROM relation name (model name or stage CTE) | +| `join_plan` | `JoinRequirement` hops | +| `row_slots` / `aggregate_slots` / `combined_expression_slots` | slots bucketed by phase | +| `cross_model_aggregate_plans` | one `CrossModelAggregatePlan` per cross-model aggregate | +| `transform_layers` | one `TransformLayer` per transform slot, in dependency order | +| `filters_by_phase` | `FilterPhase` entries (WHERE / HAVING / post) | +| `projection` / `order` / `limit` / `offset` | output shape | +| `stage_schema` | the projection downstream stages bind against (P6) | +| `active_time_dimension_slot_id` | the TD slot used for OVER `ORDER BY` | +| `render_source_model` | the concrete `SlayerModel` this stage renders against | + +A `ValueSlot` carries `id`, `key`, `declared_name`, `public_name`, +`public_aliases`, `hidden`, `phase`, `label`, `type`, and `expression` +(a `BoundExpr`). A model-validator enforces the hidden invariant: a hidden slot +must have `public_name=None` and `public_aliases=[]`, so the generator can never +accidentally emit it in the public projection. + +`FilterPhase` has two mutually-exclusive carrier modes: a typed `expression` +(`BoundExpr`, for Mode-B DSL filters and the planner-emitted `BetweenKey` +date_range) or `text` + `text_columns` (a Mode-A SQL fragment, for +`SlayerModel.filters` — the renderer qualifies the named columns and emits the +text verbatim). + +### `BoundExpr` unification + +`planned.py` re-exports `binding.BoundExpr` as the canonical class. Earlier the +planned side had a separate `BoundExpr` with an optional `sql_text` cache; that +was folded into the binder's `BoundExpr(value_key=ValueKey)` so +`ValueSlot.expression` and `FilterPhase.expression` store binder output directly. +There is no cached SQL string — the generator renders from the typed `value_key` +against the slot registry. + +### `CrossModelAggregatePlan` — the re-rooting fields + +The struct carries the route-explicit filter ids +(`where_filter_ids` / `having_filter_ids` / `target_model_filters`) plus, for the +re-rooting case, `rerooted_plan` (a nested `PlannedQuery`), `rerooted_grain_pairs`, +and `rerooted_agg_slot_id`. See [Cross-model aggregates](cross-model-aggregates.md) +— the re-rooting fields are the largest deviation from the plan's single-strategy +design. + +## Design rationale + +- **Why intern at all?** Because the four bugs are all "the same value got two + slots" or "two values shared one". Structural interning is the single mechanism + that resolves both directions, instead of per-permutation alias bookkeeping. +- **Why hidden slots rather than special-casing order/filter refs?** A hidden + slot is materialised in the base CTE like any other, then trimmed from the + public projection — so the generator has one uniform notion of "a value to + compute" and the projection logic decides visibility. Order-only and + filter-only aggregates fall out of this for free. +- **Why does `_iter_slot_deps` inline `ArithmeticKey` / `ScalarCallKey`?** + Because they have no independent column to materialise — they're operators over + their operands. Slotting them would create spurious hidden columns; the + generator inlines the operator into the SELECT/WHERE and slots only the leaves. diff --git a/docs/architecture/scopes-and-bundle.md b/docs/architecture/scopes-and-bundle.md new file mode 100644 index 00000000..28b0e377 --- /dev/null +++ b/docs/architecture/scopes-and-bundle.md @@ -0,0 +1,138 @@ +# Scopes and the source bundle + +**Modules:** `slayer/core/scope.py`, `slayer/engine/source_bundle.py` + +Binding needs two things the keys don't carry: *what a name resolves against* +(the scope) and *the resolved models it resolves through* (the bundle). These +two modules supply them, and together they implement principles **P5**, **P6**, +and **P11**. + +## Two scope kinds (P5) + +A reference like `customers.regions.name` means different things depending on +where it is being resolved. The redesign makes that explicit with two scope +types that are never confused: + +```mermaid +flowchart TB + subgraph ModelScope + direction TB + ms["source_model: SlayerModel"] + msd["dotted refs walk the join graph
customers.regions.name → hop, hop, leaf"] + msu["__ in a ref → IllegalScopeReferenceError
(unless it exact-matches a literal column name)"] + end + subgraph StageSchema + direction TB + ss["columns: List[StageColumn] (flat)"] + ssd["dotted refs → IllegalScopeReferenceError"] + ssu["__-bearing names are legal flat names
customers__regions__name"] + end +``` + +- **`ModelScope`** — joins exist. Dotted refs walk the join graph rooted at + `source_model`. A `__` in a Mode-B ref is illegal *unless* it exact-matches a + column literally named that way (the C11 carve-out for legacy persisted + query-backed columns). +- **`StageSchema`** — a flat namespace. Dots are *not* join syntax (a dotted ref + raises); `__`-bearing identifiers are ordinary flat names. This is what a + downstream stage binds against — and exactly why DEV-1449 is impossible: a + downstream ref to an upstream multi-hop dimension must use the flat form + (`robot_details__modelseriesval`), and the dotted form raises. + +`ModelScope.source_model` is `Optional` from day one (the **I2** extension +point). The binder asserts `source_model is not None` at use sites today; a +future anchor-less mode would set it `None` and take a different binder branch. +Keeping the type optional avoids a breaking change later. + +## `StageSchema` and `StageColumn` (P6) + +`StageSchema` is the typed projection a stage emits. Downstream stages bind only +against it — they never re-walk the upstream join graph. The fields that make +DEV-1448/1449 work are on `StageColumn`: + +| Field | Meaning | +| --- | --- | +| `name` | the downstream **bind** name — flat (`customers__revenue_sum`, or a user `rev`) | +| `sql_alias` | the identifier emitted in the stage's SELECT projection | +| `public_alias` | the result-key piece returned to the user (dotted form, or the user name) | +| `type, label, format, hidden, description, meta, sampled, provenance` | per-column metadata carried downstream | + +The split between `name`, `sql_alias`, and `public_alias` is what lets the +planner reserve a hidden or alias-bearing form without coupling the downstream +bind name to the public result key. `StageSchema` supports `__getitem__`, `get`, +and `__contains__` for name lookup; `relation_name` is the CTE name / subquery +alias used when a downstream stage references it. + +## `ResolvedSourceBundle` and "storage consulted once" (P11) + +`ResolvedSourceBundle` is the eagerly-resolved set of everything the binder +might need, built **once** at the top of execution by +`build_resolved_source_bundle`. After that, the binder is provably scope-only — +no `ContextVar`, no callback that re-enters storage. This is **P11**, the +principle that most directly kills the old tangle: the legacy enrichment path +re-resolved models lazily through `ContextVar`-threaded callbacks, which is why +concurrent and nested resolution was so hard to reason about. + +The bundle carries: + +| Field | Contents | +| --- | --- | +| `source_model` | the host of the query (the real base the root chain bottoms out at) | +| `referenced_models` | transitive join-graph walk + each sibling stage's base; host first | +| `inline_extensions` | a root `ModelExtension` overlay over a non-sibling base, re-applied after query-backed expansion | +| `named_queries` | the raw sibling `SlayerQuery`s of a multi-stage DAG | +| `stage_source_models` | per-named-stage resolved source model (for heterogeneous DAGs) | +| `query_variables` | merged variables (runtime > stage > outer > model defaults) | +| `datasource_hint` | the `data_source=` kwarg that wins over the priority list | + +### How the builder resolves the source + +`build_resolved_source_bundle` (`source_bundle.py:189`) handles every input +shape the public API accepts — stored-model name, inline `SlayerModel`, +`ModelExtension` overlay, and the dict forms of both — via `_resolve_source_spec`. + +Two subtleties worth knowing: + +- **Sibling-chain following.** A root whose `source_model` points at a named + sibling is followed down (`_follow_sibling_chain`) to the real base it + ultimately reads from, so the bundle's `source_model` is always a concrete + base, not a sibling name. A cycle raises (mirrors the legacy circular-reference + guard). +- **Root overlay preservation.** When the root source is a `ModelExtension` over + a non-sibling base, the overlay is recorded in `inline_extensions` so the + engine can re-apply it *after* a query-backed base expands (expansion derives + columns from the backing query and would otherwise drop the overlay's extra + columns). + +The join-graph walk (`_collect_referenced_models`) is a best-effort BFS scoped +to the source model's own `data_source` — joins never cross datasource +boundaries. Absent join targets are skipped silently (matching the legacy +`_expand_join_graph`). The source model is returned first so +`get_referenced_model` finds the host before any same-named join target. + +### Synthetic models for sibling stages + +When a downstream stage joins to — or cross-model-references — a *sibling* stage +(materialised elsewhere as a CTE), the planner needs a `SlayerModel` to resolve +against. `synthetic_model_from_stage_schema` builds a stand-in whose +`sql_table` is the stage's CTE name and whose columns are the stage's flat +output columns. `stage_bundle_with_siblings` threads these synthetic models into +a per-stage bundle so a join/cross-model ref to a sibling resolves to its CTE +relation. These two helpers are what let the [stage planner](stage-planning.md) +treat sibling stages uniformly with stored models. + +## Design rationale + +- **Why a bundle at all, rather than passing `storage` to the binder?** Purity. + If the binder can reach storage, it can re-resolve, and re-resolution is where + the old order-dependence and `ContextVar` machinery came from. Resolving + everything up front makes the binder a pure function of `(parsed, scope, + bundle)`. +- **Why optional `source_model` everywhere (I2)?** So a future + "resolve against a whole datasource, no anchor model" mode is a type-additive + change, not a breaking one. The cost today is a handful of + `assert source_model is not None` lines. +- **Why two scope classes instead of a flag?** A flag (`is_stage: bool`) would + re-merge the two resolution rules into one function with internal branching — + precisely the shape the redesign is removing. Distinct types force the binder + to dispatch, and force callers to be explicit about which world they're in. diff --git a/docs/architecture/slack-normalization.md b/docs/architecture/slack-normalization.md new file mode 100644 index 00000000..210a50ac --- /dev/null +++ b/docs/architecture/slack-normalization.md @@ -0,0 +1,144 @@ +# Slack normalization + +**Module:** `slayer/engine/normalization.py` (warning types in +`slayer/core/warnings.py`) + +The pipeline begins (principle **P0**) with a single pass that rewrites +*slack-but-unambiguous* agent input into canonical form, so every downstream +stage sees only the canonical shape. Each rewrite is returned as a typed +`NormalizationWarning` and surfaced two ways at once. + +This is how SLayer stays tolerant of the natural things agents type +(`sum(revenue)`, a bare column listed under `measures`) without letting that +tolerance leak into the resolution logic — the parser, binder, and planner never +have to know that `sum(revenue)` is even a thing. + +## The three rules + +```mermaid +flowchart LR + subgraph "Mode B (DSL fields)" + f["FUNC_STYLE_AGG
sum(revenue) → revenue:sum
count(*) → *:count"] + end + subgraph "Query shape" + m["MISPLACED_MEASURE
bare column in query.measures
→ moved to query.dimensions"] + end + subgraph "Mode A (SQL fields)" + d["DOT_PATH_IN_SQL
customers.regions.name
→ customers__regions.name"] + end +``` + +| Rule | Mode | Detects | Rewrites to | +| --- | --- | --- | --- | +| `FUNC_STYLE_AGG` | Mode B | `sum(col)`, `count(*)`, `percentile(amount, p=0.5)` | colon form (`col:sum`, `*:count`, `amount:percentile(p=0.5)`) | +| `MISPLACED_MEASURE` | query shape | a bare (no colon, no call) entry in `query.measures` that names a column | moved to `query.dimensions` | +| `DOT_PATH_IN_SQL` | Mode A | a root-scope dotted ref whose leading segment is a known join target | the `__` alias form | + +### `FUNC_STYLE_AGG` + +Applies to Mode-B fields (`ModelMeasure.formula`, `SlayerQuery.measures[].formula`, +`SlayerQuery.filters`). It scans for `(` where `` is a builtin or +custom aggregation name (and not already preceded by `:`), finds the balanced +close paren (string-literal-aware), and rewrites the first argument into colon +form, keeping any remaining args as the parametric tail. `first` / `last` are +also transform names, so the rewrite skips them when the inner is already a +colon-form aggregate (`_AMBIGUOUS_AGG_TRANSFORMS`). Custom aggregation names are +threaded in via `custom_agg_names` so model-defined aggregations are recognized. + +`func_style_agg_to_colon` is the **quiet** variant for read-only consumers +(schema-drift attribution, memory entity tagging) that need the rewrite but must +not re-surface slack advice to the user — it suppresses the warning. + +### `MISPLACED_MEASURE` + +Mirrors the legacy `_auto_move_fields_to_dimensions` heuristic but emits a +structured warning. A bare token in `measures` that names a known `ModelMeasure` +stays a measure; one that names a column moves to `dimensions`; an unknown token +is left for the downstream resolver to error on. It is a no-op when the stage has +no resolved model (a sibling-sourced stage), because column classification needs +the model's column names. + +### `DOT_PATH_IN_SQL` + +The subtle one. It rewrites `customers.regions.name` → `customers__regions.name` +in Mode-A SQL (`Column.sql`, `Column.filter`, `SlayerModel.filters`), but only +when the leading segment is a real join target on the host model — and it is +**AST-based and scope-aware**, not a regex: + +- It parses with sqlglot and identifies the **root-scope** `Column` nodes by + walking lexical ancestors (`_dot_path_root_scope_analysis`), *not* by trusting + `Scope.columns` (which would pull in correlated subquery refs). Refs inside + subqueries, CTE bodies, and set-op branches are left alone. +- It collects shadow names — CTE definitions, explicit `AS` aliases, + Subquery/CTE sources, and schema/catalog qualifiers on FROM tables — and a ref + whose leading segment matches both a join target and a shadow is flagged + *ambiguous*: no rewrite, a warning carrying + `normalized="(ambiguous: …)"`. + +The scope-guard reuses the `column_expansion.py` precedent from DEV-1410. Why +AST and not the old construction-time regex: the rewrite needs the model's join +graph to know whether the first segment is a join target (vs. a +catalog/schema-qualified name like `mydb.customers.x`), which a `Column.sql` +field validator has no access to. So multi-dot normalization is **boundary-only, +by design** — it runs in the slack pass at `engine.execute` / `engine.save_model`, +not at Pydantic construction. A consequence to state honestly: a `SlayerModel` +built in memory and read back without crossing execute/save shows the raw +multi-dot form; `save_model` canonicalizes before persisting. + +## Warning shape and dual surfacing + +```python +class NormalizationWarning(BaseModel): # slayer/core/warnings.py + rule_id: str # "FUNC_STYLE_AGG" + original: str # "sum(revenue)" + normalized: str # "revenue:sum" + location: str # "measures[2].formula" + rule_doc_url: Optional[str] # "docs/agent_input_slack.md#func-style-agg" + +class SlayerNormalizationWarning(UserWarning): + """Carrier UserWarning around a NormalizationWarning payload.""" +``` + +Every rewrite is surfaced **both** as a Python warning +(`warnings.warn(SlayerNormalizationWarning(payload))`, so +`warnings.catch_warnings()` callers see it) **and** appended to +`SlayerResponse.warnings: List[NormalizationWarning]` (so REST/MCP/CLI consumers +get the structured payload alongside the result). One source of truth, two +surfaces. The payload Pydantic type lives in `slayer.core.warnings` rather than +in the engine module so storage/REST schemas can reference it without importing +engine code. + +## Entry points and boundaries + +- `normalize_query(query, *, model, custom_agg_names)` — runs `FUNC_STYLE_AGG` + over Mode-B fields and `MISPLACED_MEASURE` over the query shape, returning a + `NormalizationResult(query, warnings)`. (The query-side `DOT_PATH_IN_SQL` + wiring is present but a no-op — Mode-A on a query is rare; most Mode-A lives on + the model.) +- `normalize_model(model)` — runs `FUNC_STYLE_AGG` over `ModelMeasure.formula` + and `DOT_PATH_IN_SQL` over `Column.sql` / `Column.filter` / + `SlayerModel.filters`, returning `NormalizationResult(model, warnings)`. + +These are invoked at the engine boundaries: `engine.execute` (per stage, via +`_normalize_stage`) and `engine.save_model`. CLI / REST / MCP go through those +entry points automatically. See [Engine orchestration](engine-orchestration.md) +for the call sites. + +## Design rationale + +- **Why normalize before parsing rather than teaching the parser to accept slack + forms?** Keeping the slack rules in one pass means the rest of the pipeline has + exactly one shape to reason about. If `parse_expr` accepted `sum(revenue)`, the + binder and planner would each have to handle both spellings. +- **Why typed warnings rather than logging?** Agents (and the REST/MCP consumers + driving them) need to *see* that their input was rewritten, structurally, so + they can learn the canonical form. A log line is invisible to them; the + `rule_doc_url` points at the canonical-form documentation. +- **Why AST for `DOT_PATH_IN_SQL`?** The old construction-time regex blindly + rewrote any `a.b.c`, including `mydb.customers.x` — a latent bug. Being + scope-aware and join-graph-aware is only possible at a boundary that has the + model in hand. + +The reference page for the rules (with the `#func-style-agg` / +`#dot-path-in-sql` / `#misplaced-measure` anchors that `rule_doc_url` points at) +is `docs/agent_input_slack.md`, authored as part of the user-facing docs update. diff --git a/docs/architecture/sql-generation.md b/docs/architecture/sql-generation.md new file mode 100644 index 00000000..f978c2a3 --- /dev/null +++ b/docs/architecture/sql-generation.md @@ -0,0 +1,272 @@ +# SQL generation + +**Modules:** `slayer/sql/generator.py` (the planned-consuming path), +`slayer/engine/response_meta.py` (response metadata) + +The generator renders a `PlannedQuery` (or a list of them) to a SQL string. It +preserves the result-key contract exactly (**P10**) and emits SQL via sqlglot +AST building, not string concatenation. + +## Entry points + +```mermaid +flowchart TB + gps["generate_planned_stages(planned_list, bundle, dialect)"] + gps -->|single stage| gfp["generate_from_planned(planned, bundle, dialect)"] + gps -->|multi-stage| loop["render each stage → CTE; root = outer SELECT"] + loop --> gfp + gfp --> inst["SQLGenerator(dialect).generate_from_planned"] + inst -->|cross-model| cm["_render_with_cross_model_plans"] + inst -->|transforms| tl["WITH base, step CTEs, outer wrap"] + inst -->|plain| base["single SELECT"] +``` + +- `generate_from_planned(planned_query, *, bundle, dialect)` — module-level + entry that constructs an `SQLGenerator` and delegates to the instance method. + Renders **one** stage. +- `generate_planned_stages(planned_queries, *, bundle, dialect)` — renders a + multi-stage DAG to one SQL string. Each non-root stage becomes a CTE; the root + is the outer SELECT. + +## `generate_from_planned` (instance method) + +Reads from typed `PlannedQuery` fields (`row_slots` / `aggregate_slots` / +`filters_by_phase` / `order` / `transform_layers`) and dispatches: + +- `cross_model_aggregate_plans` non-empty → `_render_with_cross_model_plans`; +- `transform_layers` present → `WITH base AS (...)`, Kahn-batched step CTEs + carrying the window functions, an outer wrap projecting in user-spec order; + POST-phase filters that reference transform slots wrap as `SELECT * FROM (...) + AS _filtered WHERE …`; `time_shift` / `consecutive_periods` emit dedicated + self-join CTE pairs; +- otherwise → a single base SELECT with WHERE/HAVING, GROUP BY, ORDER BY, LIMIT. + When the base CTE materialises any hidden aggregate (an aggregate referenced + ONLY by ORDER BY or a filter, never declared as a measure), a conditional + outer-trim wrapper projects exactly the public projection — same shape as the + transform path's outer wrap, minus the step CTEs — so the hidden alias does + not leak into the result columns (DEV-1501). + +It builds its own `slot_id_by_key` map (the `PlannedQuery` doesn't carry the +registry), materializes hidden aux slots referenced as transform inputs / +partition keys / time keys / POST-filter operands, and renders. + +### `AggRenderSpec` — the dialect-helper interface + +To render aggregations identically across all dialects, the shared dialect +helpers (`_build_agg`, `_build_percentile`, `_build_stat_agg`, +`_wrap_cast_for_type`, `_resolve_sql`, `_build_date_trunc`) consume a single +typed input: `AggRenderSpec`, built directly from planned slots by +`_build_agg_render_spec_from_planned`. + +Dialect-specific behavior (SQLite UDFs, ClickHouse `quantile`, the MySQL +`median` `NotImplementedError`, and so on) is therefore emitted by exactly one +code path. + +!!! note "Historical: the synthetic-`EnrichedMeasure` adapter" + + `generate_from_planned` originally consumed `PlannedQuery` at the top but + adapted *back* to `EnrichedMeasure` (`_synthesize_enriched_measure_from_planned`) + to reach the dialect helpers — a hybrid that coupled the new path to a + legacy type, kept deliberately so the two coexisting pipelines could not + drift on dialect SQL. DEV-1452 Stage A retyped the helpers onto + `AggRenderSpec`; DEV-1485 deleted the last adapter + (`_agg_render_spec_from_enriched`) and `_build_agg`'s `measure=` compat + surface with the rest of the legacy stack. + +## Multi-stage chaining (`generate_planned_stages`) + +Each non-root stage renders independently (against a per-stage bundle from +`_bundle_for_stage`) and is wrapped by `_stage_rename_wrapper` so its output +columns become the flat names downstream stages bound against +(`orders.customers.region` → `customers__region`). The wrapper derives those from +the *actual* rendered `named_selects` (robust to the cross-model renderer +emitting columns out of `public_projection` order) and asserts they match the +stage's `StageSchema` — a planner/generator divergence fails here rather than as +a confusing downstream bind miss. Stage CTEs are prepended before any CTEs the +root already emits (the root reads `FROM `). + +`_bundle_for_stage` picks the host model the stage renders against from the +planner's `render_source_model` (the stage's own source / overlay / +synthetic-over-sibling), falling back to a synthetic model over the upstream CTE +for a `StageSchema` chain stage — so the generator's FROM/joins bind against +exactly what the binder used. + +## Cross-model rendering + +`_render_with_cross_model_plans` emits one `_cm_*` CTE per +`CrossModelAggregatePlan` joined back to the host base. When `plan.rerooted_plan` +is set, `_render_rerooted_cross_model_cte` renders the nested re-rooted plan +(FROM target + the target's joins) preserving host grain; otherwise the +forward-path CTE renders (FROM bare target, grouped at the forward dims). +`Column.filter` on the aggregated column renders as +`SUM(CASE WHEN THEN END)`. See +[Cross-model aggregates](cross-model-aggregates.md). + +The same renderer also emits one `_wm_*` CTE per `WindowedAggregatePlan` — a +duration-windowed measure (`revenue:sum(window='90d')`). Unlike `_cm_*` (rooted +at the join target), a `_wm_*` CTE is **host-rooted**: an inner `_src` subquery +self-selects the host rows (dimensions → `_w_dim_`, other time dims → +`_w_td_`, the raw window time column → `_w_time`, the value → `_w_value`) +with its joins discovered through a host `ScopeFrame`, and +`FROM _base LEFT JOIN _src` +pairs the grain equalities with a trailing `INTERVAL` range predicate +(`_src._w_time >= bucket_end − window` / `< bucket_end`). The result groups at +the query grain and joins back to `_base` null-safe, exactly like a `_cm_*` CTE, +so windowed and cross-model measures coexist in one query. Windowed-measure +filters route to the combined-SELECT outer `WHERE` (`Phase.POST`). `sum`/`avg` +local measures only; other shapes raise at plan time (`_guard_windowed_measures` +in `stage_planner.py`). + +### Frame bounds vs population filters (DEV-1732) + +`_src` inherits the host's ROW-phase filters **minus their frame bounds** — a +relational comparison (`<`, `<=`, `>`, `>=`) between a non-hidden time +dimension's raw column and a temporal literal. Without that, the trailing window +cannot reach rows before the earliest visible bucket and that bucket +under-counts; with it, `date_range` and the explicit spelling of the same intent +produce identical numbers. + +`slayer/core/time_bounds.py` owns the analysis (dependency-free, so planner and +generator share it). `stage_planner.plan_query` computes the strippable column +set once into `PlannedQuery.frame_bound_columns` and partitions the filters into +`WindowedAggregatePlan.where_filter_ids` (applied) plus `src_filter_rewrites` +(applied as a residual — a top-level `and` is split so only its frame-bound +conjuncts drop; `or`/`not` are never descended into). The generator's +`_effective_src_filters` materialises that view **once** and feeds the same list +to both join discovery and rendering, so the two cannot disagree about what the +CTE contains. + +Hidden `TimeTruncKey` slots are excluded from `frame_bound_columns` on purpose: +`_build_windowed_plans` skips hidden row slots, so a hidden time axis is never +equality-joined into `_src` and stripping its bound would leave it +unconstrained. Mode-A `SlayerModel.filters` are exempt entirely — they define +which rows exist, not which frame the query looks at. + +The `time_shift` shifted CTE (`_shifted_where_part`) applies the same rule, +reading the same `frame_bound_columns`; its former +`isinstance(..., BetweenKey)` special case is subsumed, since a `date_range`'s +`BetweenKey` column is always a query time dimension's raw column. + +## Mode-A filter inlining and join discovery (DEV-1494) + +A column-level `Column.filter` on an aggregated measure becomes a CASE-WHEN +wrapper (`SUM(CASE WHEN THEN END)`), and a `SlayerModel.filters` +entry becomes a WHERE term. Both are Mode-A SQL and share one renderer, +`_render_mode_a_predicate`, which inline-expands references to derived columns — +bare (`is_eu` → its `CASE WHEN customers.region …`) or dotted to a derived +column on a joined model (`loss_payment.has_flag` → its `sql`) — so the emitted +predicate is runnable and never references a non-physical `.`. +A predicate with only base refs takes the cheap qualify path +(`_qualify_mode_a_sql_filter` regex for model filters, `_qualify_column_filter_sql` +AST for column filters), byte-identical to before. On sqlglot parse failure the +predicate falls through to the qualify path unchanged. + +Join discovery for these text filters (`_filter_join_paths`) is the **union** of +the join paths in the **un-inlined** predicate and those in the **inline-expanded** +predicate. Both are needed: the dbt placeholder-join idiom — a constant derived +column such as `has_flag sql="1"` whose only purpose is to force the (inner) +join — keeps its alias only in the un-inlined form (it inlines to the constant +`(1)`), while a derived ref's *crossed* joins (`is_eu` → `customers`; +`loss_payment.deep_flag` → `loss_payment__claim`) appear only after expansion. +Discovery for column filters in the base SELECT is restricted to **local** +aggregate sources (empty `AggregateKey.path`); a cross-model aggregate's filter +joins are discovered inside its `_cm_*` CTE instead — `_render_cross_model_cte` +collects the join paths of the target measure's `Column.filter` and the +target-model filters and adds them to the CTE's own FROM. Because each `_cm_*` +CTE is an isolated per-(target, grain) computation, adding the join resolves the +filter's refs without affecting sibling measures. Discovery is root-scope-only, +so a correlated ref inside an `EXISTS (...)` subquery does not pull an outer join. + +## Host-base join discovery (the three symmetric sources) + +The host base FROM at `_build_base_select_for_planned` pulls in `LEFT JOIN`s from +three symmetric sources, each handled by a dedicated collector wired in the same +call chain just before `_build_from_and_joins`: + +1. **Dimension / time-dimension `Column.sql`** (DEV-1484): `_expand_derived_row_dims` + pre-expands derived ROW slots (`ColumnSqlKey` dims and `TimeTruncKey` columns + that are themselves derived) and scans the expansion through + `_joined_paths_in_sql`, appending crossed paths to `needed_join_paths`. +2. **Aggregated-measure `Column.filter`** (DEV-1494): `_collect_column_filter_join_paths` + recurses through AGGREGATE-phase composite keys (`ArithmeticKey` / + `ScalarCallKey`) and, for each `AggregateKey` with a `column_filter_key`, + collects the paths the predicate touches via `_filter_join_paths` (the union + of un-inlined and inline-expanded predicate paths, per the section above). +3. **Aggregate-source `Column.sql`** (DEV-1502): `_collect_aggregate_source_join_paths` + mirrors the filter helper — recurses through the same composite keys, and for + each `AggregateKey` whose `source` is a `ColumnSqlKey` with `path == ()`, + expands the column via `_expand_derived_column_sql` and scans the result + through `_joined_paths_in_sql`. The render-time expansion in + `_build_agg_render_spec_from_planned` already produces `SUM()` SQL; + this collector closes the join-discovery loop so a measure source like + `customers__regions.population` emits both `LEFT JOIN`s. + +All three collectors restrict to **local** aggregate sources (empty +`AggregateKey.source.path`); cross-model aggregates own their own join +discovery inside the per-plan `_cm_*` CTE — for the `Column.filter` side +(DEV-1494 / DEV-1503) and, since Stage 4 (DEV-1708 closed DEV-1526), for a +target column's `Column.sql` that crosses a further join. All three +host-side collectors feed the shared `needed_join_paths` list, so repeated +paths surfaced by different sources dedupe naturally via +`_build_from_and_joins`'s `emitted_aliases` guard. + +Since Stage 5 (DEV-1709, widened Law-3 trigger), a LOCAL aggregate with +any crossing input — source `Column.sql`, `Column.filter`, positional +args, kwargs — never renders in the top-level host base at all: it +isolates into a host-rooted `_cm_*` CTE, and the discovery above runs +inside that CTE's sub-render (see +[cross-model-aggregates.md](cross-model-aggregates.md#strategy-3-host-rooted-isolation--any-crossing-input-dev-1503-widened-by-dev-1709)). +The host base only ever contains purely-local aggregates. + +## Result-key contract (P10) + +The generator preserves the result keys byte-for-byte: `orders.revenue_sum`, +`orders._count` (the `*` dropped, the leading `_` kept), joined dimensions as the +full dotted path `orders.customers.regions.name`, and renamed measures as +`orders.`. `_full_alias_for_slot` derives these from the slot's key / +public aliases. Two documented exceptions, both routed through the same +`canonical_agg_name` helper: cross-model parametric aggregates carry the kwarg +suffix legacy dropped, and hidden parametric `first`/`last` (DEV-1501) carry the +explicit time-arg suffix so distinct time-column specs get distinct +materialised aliases (`orders.revenue_last_created_at`, +`orders.revenue_last_updated_at`). + +## Response metadata (`response_meta.py`) + +`build_response_metadata` builds `SlayerResponse.attributes` and +`expected_columns` from the root `PlannedQuery` plus the rendered SQL (the +retired legacy engine derived both from an `EnrichedQuery`): + +- **`expected_columns`** comes from the final SQL's `named_selects` — the literal + result-key columns rows come back under. Reading them from the SQL (rather than + re-deriving from slots) is bulletproof: it is exactly the outer SELECT the + generator emitted. +- **`attributes`** (`ResponseAttributes.dimensions` / `.measures`) come from the + root plan's public `ValueSlot`s, classified dimension (ROW phase) vs measure + (everything else), with each public result key mapped to its + `FieldMetadata(label, format)`. `_slot_result_keys` mirrors + `_full_alias_for_slot` so the keys line up with the rendered projection; only + keys actually present in the rendered SQL are surfaced (a guard against + divergence). Aggregate formats come from `_infer_aggregated_format` (INTEGER + for count/star, FLOAT for avg-family, source-column format for sum/min/max). + +`FieldMetadata` / `ResponseAttributes` / `_infer_aggregated_format` live here (not +in `query_engine`) so the module imports nothing from the engine; +`query_engine` re-exports them, keeping the public import path unchanged. + +## Design rationale + +- **Why one shared dialect emitter?** Dialect coverage (SQLite UDFs, ClickHouse + parametric quantiles, MySQL's unsupported-function `NotImplementedError`, the + `log10`/`log2` literal preservation, JSON-extract rewriting) is large and + well-tested. Routing every caller through one emitter keeps that behaviour in + a single place. This originally also kept the two coexisting pipelines from + drifting on dialect SQL, at the cost of an `EnrichedMeasure` coupling that + DEV-1452 / DEV-1485 removed. +- **Why derive `expected_columns` from the SQL?** Because the SQL is the ground + truth for what rows come back keyed by. Re-deriving from slots risks a subtle + mismatch; reading `named_selects` cannot. +- **Why assert in `_stage_rename_wrapper`?** A leaked hidden column or a C13 + over-projection would otherwise surface as a downstream "column not found" + deep in the next stage's binding. Asserting at the boundary turns a confusing + failure into a precise one. diff --git a/docs/architecture/stage-planning.md b/docs/architecture/stage-planning.md new file mode 100644 index 00000000..0bb6081f --- /dev/null +++ b/docs/architecture/stage-planning.md @@ -0,0 +1,164 @@ +# Stage planning + +**Module:** `slayer/engine/stage_planner.py` + +The stage planner is the orchestrator that turns `SlayerQuery` stages into +`PlannedQuery`s. `plan_query` compiles one stage; `plan_stages` compiles a +multi-stage DAG. It composes the [binder](binding.md), the +[planning](planning.md) primitives, and the +[cross-model planner](cross-model-aggregates.md), and emits each stage's +`StageSchema` (**P6**). + +## `plan_query` — one stage end to end + +```mermaid +flowchart TB + q["SlayerQuery + scope + bundle"] --> dm["_declared_measures_from_query
parse + expand + bind dims/TDs/measures"] + dm --> filt["bind filters (date_range, model.filters, user filters)"] + filt --> ord["resolve ORDER BY (alias map → bind)"] + ord --> td["resolve active TD → _attach_time_keys"] + td --> sugar["lower_sugar_transforms (change/change_pct)"] + sugar --> tval["validate: every time-needing transform has a time_key"] + tval --> proj["ProjectionPlanner.plan → registry + projection"] + proj --> cmp["per cross-model aggregate: cross_model_planner.plan + maybe re-root"] + cmp --> emit["emit transform_layers, filters_by_phase, stage_schema"] + emit --> pq["PlannedQuery"] +``` + +### Declared measures, in projection order + +`_declared_measures_from_query` builds the `DeclaredMeasure` list in **dim → +time-dim → measure** order (matching the legacy `user_projection` order). Each +dimension/time-dimension binds and is declared under its flattened `__` name +(`stores.opened_at` → `stores__opened_at`) — that flat name is the +`StageColumn.name` a downstream stage binds against (DEV-1448/1449). Measures +run through `expand_model_measures` first (against a `ModelScope` only — +downstream `StageSchema` stages don't expose saved measures), then bind, then get +a canonical alias via `_canonical_alias_for_formula`. + +`_canonical_alias_for_formula` routes any aggregate-rooted formula (including +parametric `revenue:percentile(p=0.5)`) through `canonical_agg_name` so kwargs are +sanitized consistently (`p=0.5` → `_p_0_5`); a cross-model star keeps its +`customers.` prefix. (This is also where cross-model parametric aliases keep the +kwarg suffix legacy dropped — the documented P10 divergence.) + +### Filters, in legacy WHERE order + +The planner constructs filters in the exact order the legacy generator emitted +them, so SQL stays parity-stable: + +1. `date_range` filters — one per time dimension with a 2-element range, built by + `_build_date_range_filter` as a `BetweenKey` over the **bare** underlying + column (a `ColumnKey`, or a `ColumnSqlKey` for a derived temporal column — + DEV-1450 #4a — which the generator renders as ` BETWEEN …`), + not the `TimeTruncKey`, so the self-join CTE path can read raw data while the + filter applies to the outer projection. +2. `SlayerModel.filters` — Mode-A SQL, validated by `_validate_model_filter` + (rejects DSL constructs, raw windows, measure refs, and windowed columns). + A reference to a non-trivial derived column is accepted (DEV-1450 #4b): the + generator's `_render_model_filter_sql` inline-expands the predicate at render + time. These are text-only `FilterPhase` entries with no typed value-key. +3. user query filters — Mode-B DSL, bound with the `filter_alias_map` so renamed + measures resolve by alias (DEV-1445). Two filter strings that bind to the same + structural key are deduped (P2) so a HAVING isn't duplicated. + +The `filter_alias_map` is built from **measure** aliases only (the tail of +`declared_measures` past the dim/time-dim prefix) — never dimension/time-dimension +names, because a time dimension's declared name is its raw column and a +`created_at <= '…'` filter must resolve to the raw column. + +### ORDER BY resolution + +A user order column may name a declared measure by user `name`, declared name, +canonical alias, the flattened dotted form (a joined dimension), or the `_count` +form of `*:count`. `plan_query` checks `declared_alias_to_bound` in that order +before falling back to `bind_expr` on the preserved `raw_formula` — so an +aggregate alias like `amount_sum` (not a column on the model) interns onto the +projection slot rather than raising. + +### Time-key attachment + +`_attach_time_keys` walks every measure/filter/order value-key and, for each +time-needing `TransformKey` (`cumsum` / `change` / `time_shift` / `first` / +`last` / `lag` / `lead` / `consecutive_periods`) whose `time_key` is `None`, +patches in the active TD's key. The active TD is resolved by +`_resolve_main_time_dimension` (0 TDs → none; 1 TD → that one; +2+ → `main_time_dimension` by full-name then leaf, else +`model.default_time_dimension` host-local). After patching, +`_find_unresolved_time_needing_op` validates that no time-needing transform was +left without a TD, raising the legacy error phrase. Sugar lowering runs *after* +this so the desugared `time_shift` inherits the patched `time_key`. + +### Emitting the plan + +`ProjectionPlanner.plan` builds the registry; `_bucket_slots` splits slots into +row / aggregate / combined by phase; the cross-model loop runs the strategy once +per cross-model aggregate — passing `host_query` / `public_projection` / a +`subplan_builder` callback so the strategy itself decides forward-vs-re-rooted +(DEV-1450 #2; the re-root logic moved into `cross_model_planner.py`); +`_emit_transform_layers` +emits one `TransformLayer` per transform slot in Kahn-topological dependency +order (so `cumsum(change(...))` renders inner before outer); and +`_emit_stage_schema` builds the `StageSchema` from public slots. + +## `_emit_stage_schema` — the downstream contract (P6) + +Only public (non-hidden) slots appear, one column per `public_projection` +occurrence (so a C13 multi-alias slot emits one column per alias). Each column's +downstream `name` and `sql_alias` are the `__`-flattened alias +(`customers.revenue_sum` → `customers__revenue_sum`), while `public_alias` keeps +the dotted result-key form. Two distinct public columns that flatten to the same +downstream name raise a collision error rather than silently binding the first +match. This flat-name schema is precisely what a downstream stage binds against — +the reason DEV-1449's dotted-ref-downstream raises. + +## `plan_stages` — the multi-stage DAG + +```mermaid +flowchart LR + qs["queries: List[SlayerQuery]"] --> topo["_topo_sort (Kahn, root last)"] + topo --> loop["for each stage in order"] + loop --> sb["_stage_scope_and_bundle
resolve scope + per-stage bundle"] + sb --> pq["plan_query"] + pq --> sch["record stage_schema by name"] + sch --> loop +``` + +`_topo_sort` orders stages so each appears after the siblings it references via +`source_model` (Kahn's algorithm; rejects duplicate names and cycles; unnamed +stages — typically the root — go last). For each stage, +`_stage_scope_and_bundle` resolves the right `(scope, bundle)`: + +- a `ModelExtension`/dict **over a sibling** → overlay the extra columns onto a + synthetic model of the sibling CTE and bind `ModelScope`-style; +- a **bare-string sibling** source (a chain) → bind against the upstream flat + `StageSchema` (P6 / DEV-1449), with the synthetic upstream model as the host + for cross-model/generation consistency; +- otherwise **model-scoped** → the stage's own resolved source (the root uses the + bundle's `source_model`; a named sibling uses its pre-resolved + `stage_source_models` entry). + +Each per-stage bundle threads in synthetic models for the *other* already-planned +siblings (`stage_bundle_with_siblings`), so a join/cross-model ref to a sibling +resolves to its CTE. After planning a named stage, its `StageSchema` is recorded +so later stages can bind against it. + +## Design rationale + +- **Why build declared measures in dim/time-dim/measure order?** So the public + projection order matches legacy exactly (P10), and so the `filter_alias_map` + can slice off the measure tail without tracking indices separately. +- **Why attach `time_key` in the planner rather than the binder?** Only the + planner has the query (the set of TDs, `main_time_dimension`, + `default_time_dimension`). The binder is expression-local. See + [Binding](binding.md). +- **Why emit `StageSchema` per stage rather than let downstream re-walk joins?** + Because re-walking is the legacy tangle. A stage composes with the next *only* + through its schema (P6); the downstream binder sees a flat namespace and + literally cannot reach the upstream join graph — which is what makes DEV-1449 + a structural impossibility rather than a guarded special case. +- **Why topo-sort here when the engine also sorts the runtime list?** The engine + sorts the user-submitted list (and validates root-as-sink) before planning; + `_topo_sort` re-establishes the planning order from `source_model` references so + `plan_stages` is correct regardless of how it's called (it shares the algorithm + but is the planner's own guarantee). diff --git a/docs/architecture/typed-keys.md b/docs/architecture/typed-keys.md new file mode 100644 index 00000000..c73c9bc9 --- /dev/null +++ b/docs/architecture/typed-keys.md @@ -0,0 +1,163 @@ +# Typed keys — structural identity + +**Module:** `slayer/core/keys.py` + +The `ValueKey` family is the foundation of the whole pipeline. A key answers +exactly one question — *"are these two expression occurrences the same value?"* +— and carries nothing else. Rendering state (SQL text, alias, projection +position, hidden-ness) lives on `ValueSlot` (in `planned.py`), never on the key. + +This separation is principle **P2**: identity is structural, not textual. +`revenue:sum`, the inner `revenue:sum` in `change(revenue:sum)`, and a filter +occurrence of `revenue:sum` all build the same `AggregateKey`, so the +`ValueRegistry` interns them to one slot. That is what makes the dedup bugs +(DEV-1446) structurally impossible rather than patched. + +## The family + +```mermaid +classDiagram + class ValueKey { + <> + } + ValueKey <|-- ColumnKey + ValueKey <|-- ColumnSqlKey + ValueKey <|-- TimeTruncKey + ValueKey <|-- StarKey + ValueKey <|-- LiteralKey + ValueKey <|-- SqlExprKey + ValueKey <|-- AggregateKey + ValueKey <|-- TransformKey + ValueKey <|-- ArithmeticKey + ValueKey <|-- ScalarCallKey + ValueKey <|-- BetweenKey +``` + +| Key | Phase | Identifies | +| --- | --- | --- | +| `ColumnKey(path, leaf)` | ROW | a base column; `path` empty for local, non-empty for joined | +| `ColumnSqlKey(path, model, column_name)` | ROW | a derived column (`Column.sql` set) | +| `TimeTruncKey(column, granularity)` | ROW | a time-truncated column at one grain | +| `StarKey(path)` | ROW | the `*` source for `*:count` (local or cross-model) | +| `LiteralKey(value)` | ROW | a literal operand inside an expression tree | +| `SqlExprKey(canonical_sql)` | ROW | a Mode-A SQL fragment (a `Column.filter`) | +| `AggregateKey(source, agg, args, kwargs, column_filter_key)` | AGGREGATE | one aggregation slot | +| `TransformKey(op, input, args, kwargs, partition_keys, time_key)` | POST | a window/temporal transform over a value | +| `ArithmeticKey(op, operands)` | max(operands) | arithmetic / comparison / boolean | +| `ScalarCallKey(name, args)` | max(args) | a closed-allowlist scalar function call | +| `BetweenKey(column, low, high)` | ROW | a `BETWEEN` predicate (today only `date_range`) | + +All are frozen Pydantic models (`_FrozenKey` sets `frozen=True`), so they are +hashable and immutable — usable directly as `dict` keys in the `ValueRegistry`. + +## Phase + +`Phase` is an `IntEnum` (`ROW=0 < AGGREGATE=1 < POST=2`). It is the engine of +filter routing (**P8**): a composite key's phase is the **max** of its operands' +phases, and a filter routes to `WHERE` / `HAVING` / post-filter by the highest +phase it reaches. Phase is computed, not stored — `ArithmeticKey.phase` is +`max(o.phase for o in operands)`, `ScalarCallKey.phase` is the max over args +that carry a phase, and the leaf keys hard-code their level. Keeping phase a +*property of the key* means no separate "is this a HAVING filter?" text analysis +exists anywhere. + +## Design choices + +### Local and cross-model share one shape (P3) + +`ColumnKey`, `ColumnSqlKey`, and `StarKey` all carry a `path: Tuple[str, ...]`. +Empty path = local; non-empty = a join walk from the query's source model +(`("customers",)`, `("customers", "regions")`). `AggregateKey` inherits this via +its `source`. There is **no separate `cross_model_measures` track** in the +intermediate representation — `path == ()` is the only thing distinguishing a +local aggregate from a cross-model one, and "base CTE vs cross-model CTE" is a +*render* decision made downstream by the planner, not a semantic split baked +into the key. + +### Structural identity has to survive Python's scalar coercion + +Python collapses `True == 1 == Decimal("1")` (and `False == 0`). A naive +tuple-of-bare-values hash would intern `args=(True,)` with `args=(Decimal("1"),)` +— wrong. `AggregateKey`, `TransformKey`, `ScalarCallKey`, and `LiteralKey` +therefore override `__hash__` / `__eq__` to wrap each scalar leaf in a +`(type_tag, value)` pair via `_typed_leaf` (`"__bool__"` / `"__num__"` / +`"__str__"` / `"__none__"`). This restores the type distinction at hash/eq time +without changing the stored representation users see via `key.args[0]`. This was +a review fix (the original keys interned distinct values together). + +### Kwargs are canonicalized to sorted order + +`AggregateKey.kwargs` and `TransformKey.kwargs` run through a `before`-validator +(`_sort_kwargs_tuple`) that sorts by key name, so `weighted_avg(weight=qty)` +interns regardless of input order. Numeric scalars are expected to already be +normalized to `Decimal` (via `normalize_scalar`) so `percentile(p=0.5)` and +`percentile(p=0.50)` are the same key; identifier kwargs arrive as `ColumnKey` +so `weight=quantity` and `weight=quantity_v2` differ. + +### `normalize_scalar` is the one place ints/floats become `Decimal` + +`int → Decimal(value)`; `float → Decimal(str(value))` (via `str`, so floats land +on their displayed decimal form, not their binary approximation); `bool` / `str` +/ `None` / `Decimal` pass through. Booleans are checked *before* int (because +`bool` is-a `int` in Python). Anything else raises `TypeError`. + +### `column_filter_key` folds `Column.filter` into aggregate identity + +A column's `Column.filter` (a Mode-A CASE-WHEN applied at aggregation time) +becomes part of the `AggregateKey` via `column_filter_key: Optional[SqlExprKey]`. +Two aggregates over the same column with different attached filters are therefore +different slots; same-filter ones intern. `*:count` (a `StarKey` source) has no +column, so `column_filter_key` stays `None`. + +### `TimeTruncKey` is a distinct key, not a slot flag + +A time dimension is identified by `(column, granularity)`. Month, day, and +raw uses of the same column are distinct slots automatically, with no +special-casing in the registry. The granularity is stored as a plain `str` +(the `TimeGranularity` member's value) so the key stays a pure-data frozen model +without an enum import. The underlying column is recoverable, so a +`date_range` filter can bind against the raw column independently of the +truncation. (Codex weighed three encodings — a new key, slot metadata, or +`TransformKey(op="date_trunc")` — and the new key won for keeping the registry +uniform.) `TimeTruncKey.column` is `Union[ColumnKey, ColumnSqlKey]` (DEV-1450 +follow-up #4a): a derived (`Column.sql`) temporal column is a first-class time +dimension — the generator applies the `DATE_TRUNC` over the expanded +expression. The kind-agnostic helpers `column_leaf` / `column_path` (in +`keys.py`) unwrap either form. See [Binding](binding.md). + +### `BetweenKey` exists only for legacy SQL parity + +A `col BETWEEN low AND high` and the Mode-B compound `col >= low and col <= high` +render to different SQL text. The legacy generator emits `BETWEEN` for +`date_range`, so `BetweenKey` marks exactly that spot. The Mode-B parser never +produces it — a user-written `col >= a and col <= b` stays an `ArithmeticKey`, +preserving its parity with legacy (which keeps the AND form verbatim). This is a +deliberately narrow key whose only job is "don't drift from legacy on this one +construct". + +## The closed scalar allowlist (P1 / C12) + +`SCALAR_FUNCTIONS` is a `frozenset` living here (not in `formula.py`) so the keys +module is the single source of truth for what counts as a structurally-keyed +scalar call: + +```python +SCALAR_FUNCTIONS = frozenset({ + # null handling + "nullif", "coalesce", "ifnull", + # math + "ln", "log10", "log2", "log", "exp", "sqrt", "pow", "power", + "abs", "floor", "ceil", "round", + # string hygiene (was DEV-1378's STRING_HYGIENE_OPS) + "lower", "upper", "trim", "replace", "substr", "instr", "length", "concat", +}) +``` + +Anything outside this set (plus the transform and aggregation registries) raises +`UnknownFunctionError` in Mode B. This replaces the deleted +`MixedArithmeticField` implicit passthrough: arbitrary dialect-specific +functions (`regexp_match`, `date_part`, JSON ops) belong in Mode A — the user +moves them into a derived `Column.sql`. The `keys.py` constant is imported by +both the parser (`syntax.py`) and the binder (`binding.py`); the binder +re-checks membership as defence-in-depth against direct `ParsedExpr` +construction that bypasses the parser. diff --git a/docs/concepts/formulas.md b/docs/concepts/formulas.md index af4e5113..d59c3708 100644 --- a/docs/concepts/formulas.md +++ b/docs/concepts/formulas.md @@ -62,6 +62,70 @@ Units can be combined in descending or practical order, for example `'1y2m3w5d6h7min8s'`, `'90d'`, `'6h'`, or `'15min'`. Quote the duration value inside the formula. +Windowed measures need exactly one resolvable time dimension (a single +`time_dimensions` entry, or `main_time_dimension` to disambiguate). Filtering on +a windowed measure (`{"formula": "revenue:sum(window='90d') > 100"}`) applies +after aggregation, and the windowed measure must also be selected. + +A windowed measure may also be used purely as an **order** target without being +selected — `{"order": [{"column": "revenue:sum(window='90d')", "direction": "desc"}]}` +ranks by the rolling value and keeps it out of the result. That works both for a +bare windowed measure and for one inside an order-only composite +(`{"column": "revenue:sum(window='90d') / cnt:sum"}`). + +Note the deliberate asymmetry: a windowed measure inside a composite is allowed +in `order` but not yet in `measures`. Ordering needs only a single scalar +comparison, whereas projecting the composite surfaces the rolling value's NULLs +(a grain bucket with no matching source rows yields NULL) as user-visible +result values — settling those semantics is part of the follow-up below. + +#### Time bounds do not clip the window + +A trailing window has to read rows from *before* the earliest bucket you asked +for — otherwise that bucket silently under-counts. So a **time bound narrows +which buckets come back, not which rows the window may reach**. These two +queries return identical numbers: + +```json +{"time_dimensions": [{"dimension": "created_at", "granularity": "month", + "date_range": ["2025-01-01", "2025-12-31"]}]} +``` +```json +{"time_dimensions": [{"dimension": "created_at", "granularity": "month"}], + "filters": ["created_at >= '2025-01-01' and created_at <= '2025-12-31'"]} +``` + +A bound counts as a *frame* bound when it compares a **time dimension's own +column** against a **literal** using `<`, `<=`, `>`, or `>=`. Everything else is +an ordinary row filter and does restrict the window's input, including: + +- other operators on that column — `created_at == '2025-01-01'`, `IN (…)`, + `IS NOT NULL`; +- a bound on a time column that is not one of the query's time dimensions; +- a comparison against another column rather than a literal; +- a bound wrapped in `or` or `not`, which cannot be separated out safely; +- `filters` declared on the **model** — those define which rows exist at all, so + a model scoped to `created_at >= '2024-01-01'` does clip the window there. + +Mixed filters are split, so only the time part is set aside: +`"created_at >= '2025-01-01' and status = 'paid'"` restricts the window's input +to paid rows while still reaching back before January. + +The same rule applies to [`time_shift`](#transform-functions) — the earliest +visible bucket still gets its prior-period value under either spelling. + +If you genuinely want to clip the underlying rows, apply the bound in an inner +stage of a multi-stage query so the windowed stage never sees the raw column. + +The following windowed-measure shapes raise a clear error rather than returning +wrong numbers, and are planned follow-ups: a windowed aggregation other than +`sum`/`avg`; a cross-model windowed measure (`customers.revenue:sum(window=…)`); +a windowed measure combined with a transform (`cumsum`, `time_shift`, …) in any +position; a windowed measure nested in an arithmetic/composite expression in +`measures` (`{"formula": "revenue:sum(window='90d') / 2"}`); or one compared +against a plain aggregate inside one filter +(`revenue:sum(window='90d') > 100 and revenue:sum > 50`). + --- ## Field Formulas @@ -155,7 +219,7 @@ total per status, not one running total across the whole result set. `time_shift` uses a **self-join CTE** with an INTERVAL-shifted time column. `change` and `change_pct` are desugared into a hidden `time_shift` + arithmetic expression at query enrichment time. The shifted sub-query applies the time offset everywhere (WHERE, GROUP BY, SELECT), so it can reach outside the current result set — no edge NULLs when the database has the data, and correct handling of gaps in time series. -The self-join matches on **all non-time dimensions as well as the shifted time column** (e.g. `ON base.month = shifted.month AND base.store = shifted.store`), so these transforms are partition-safe: each group's series is compared only against itself, and per-group series reset cleanly. One store's first month is never diffed against another store's last month. +The self-join matches on **every projected dimension as well as the shifted time column** — plain columns, joined columns (`stores.name`), derived columns, and any secondary time dimension all take part in the join grain (e.g. `ON base.month IS NOT DISTINCT FROM shifted.month AND base.store IS NOT DISTINCT FROM shifted.store`). So these transforms are partition-safe: each group's series is compared only against itself, and per-group series reset cleanly. One store's first month is never diffed against another store's last month. The grain match is **null-safe** (`IS NOT DISTINCT FROM`, or the dialect equivalent), so a group with a NULL dimension value — for example rows with no matching row across a LEFT join — still lines up against its own prior period instead of dropping to a NULL shifted value. **Intent recipes:** @@ -230,7 +294,7 @@ Combine with a filter to get "top N": **Ranking within a partition (`partition_by=`):** -To rank within groups instead of across the whole result set, pass `partition_by=` referencing one or more **query dimensions** (or time dimensions). The columns must already be grouped on — partitioning by a column that's not a dimension errors at enrichment time. +To rank within groups instead of across the whole result set, pass `partition_by=` referencing one or more **query dimensions** (or time dimensions). The columns must already be grouped on — partitioning by a column that's not a dimension errors at plan time (HTTP 400). Naming a query time-dimension partitions by its truncated bucket, not the raw timestamp. ```json { diff --git a/docs/concepts/queries.md b/docs/concepts/queries.md index 055ea85c..d0b095d2 100644 --- a/docs/concepts/queries.md +++ b/docs/concepts/queries.md @@ -97,6 +97,8 @@ A time dimension with a required granularity and an optional date range. Support `week` is Monday-anchored (ISO-8601); `week_sunday` is Sunday-anchored (weeks start Sunday, end Saturday) for tools that use Sunday weeks. Both are model granularities you set on a `TimeDimension` — `week_sunday` is the SLayer value, not a wire keyword sent by a BI tool. +`date_range` and an equivalent explicit filter (`"created_at >= '2024-01-01' and created_at <= '2024-12-31'"`) are interchangeable — including for trailing-window measures and `time_shift`, which still read rows from before the range so the earliest bucket isn't short-changed. See [Time bounds do not clip the window](formulas.md#time-bounds-do-not-clip-the-window) for exactly which predicates count as a time bound. + ## OrderItem A sort specification: `column` is the short alias (`status`, `revenue_sum`, `*:count`), `direction` is `asc` or `desc`. @@ -107,6 +109,42 @@ A sort specification: `column` is the short alias (`status`, `revenue_sum`, `*:c Via MCP: `{"column": "*:count", "direction": "desc"}` +### Ordering by something you don't project + +`order` may reference a column or aggregate that is **not** declared as a dimension/measure — the classic "top-N by metric X, display only Y, Z" pattern: + +```json +{"source_model": "orders", "dimensions": ["status"], "measures": [{"formula": "*:count"}], + "order": [{"column": "amount:sum", "direction": "desc"}], "limit": 10} +``` + +The `amount:sum` aggregate is computed as a hidden column, sorted on, and **stripped from the result** — the response projects only `status` and `_count`. This works for local aggregates, cross-model aggregates (`customers.revenue:sum`), and inner-stage columns re-aggregated in a later DAG stage (`customers__revenue_sum:max`). + +What each shape of an *undeclared* order target does: + +| Order target | Behavior | +| --- | --- | +| An aggregate (`amount:sum`, `customers.revenue:sum`) | Computed hidden, sorted on, stripped from the result. Always allowed. | +| An inline **transform** (`rank(amount:sum)`, `cumsum(...)`, `change(...)`, `lag`/`lead`/`ntile`) | Computed hidden, sorted on, stripped. | +| An inline **composite** (`revenue:sum / cnt:sum`, `abs(amount:sum)`, `change(amount:sum) / 2`) | Computed hidden, sorted on, stripped. | +| A **windowed** aggregate (`amount:sum(window='90d')`), alone or inside a composite | Computed hidden in its own rolling-window CTE, sorted on, stripped. | +| A raw row column, in a **raw-rows** query (`distinct_dimension_values: false`, no measures) | Sorted on directly (`ORDER BY orders.created_at`). | +| A raw row column, in an **aggregated / dedup** query | Rejected (HTTP 400): it isn't in the `GROUP BY`. Add it to `dimensions`, or order by an aggregate of it (`created_at:max`). | +| A **joined** row column (`customers.regions.name`) not projected | Rejected (HTTP 400): project it (add to `dimensions`) or order by a projected field. | + +Transform and composite order targets accept the full formula syntax, so +`{"column": "revenue:sum / cnt:sum"}` and `{"column": "change(revenue:sum)"}` both +work without declaring a measure. One limit: the operands must be written as +formulas, not as the *names* of measures you declared in the same query — +`{"column": "rev / cnt"}` is rejected at validation, because referencing a +declared measure by its alias inside an expression is not supported anywhere in +SLayer. Write `{"column": "revenue:sum / cnt:sum"}` instead. + +A windowed measure inside a **declared** composite measure +(`{"formula": "revenue:sum(window='90d') / cnt:sum"}`), and any combination of a +windowed measure with a transform, are still rejected — see +[formulas](formulas.md#windowed-sum-and-average). + ## Response Query results are returned as a `SlayerResponse`: @@ -152,11 +190,16 @@ Filter formulas define conditions for the query. They go in the `filters` parame | `<` | `"amount < 1000"` | | `<=` | `"amount <= 1000"` | | `in` | `"status in ('active', 'pending')"` | +| `not in` | `"status not in ('cancelled', 'expired')"` | | `IS NULL` | `"discount IS NULL"` | | `IS NOT NULL` | `"discount IS NOT NULL"` | | `like` | `"name like '%acme%'"` | | `not like` | `"name not like '%test%'"` | +The right-hand side of `in` / `not in` must be a non-empty tuple of literal +values (strings, numbers, or booleans) — references and expressions on the +RHS are not supported. Both `(...)` and `[...]` syntax are accepted. + ### Boolean Logic Use `and`, `or`, `not` within a single filter string: @@ -475,6 +518,21 @@ When models have [joins](models.md#joins), you can reference measures from joine This generates a sub-query for the joined measure, scoped to shared dimensions, and LEFT JOINs it to the main query — avoiding aggregation errors from row multiplication. +A cross-model **parametric** aggregate keeps its kwarg signature in the result key, so two variants on the same target column do not collide: + +```json +{ + "source_model": "orders", + "dimensions": ["customers.region"], + "measures": [ + "customers.revenue:percentile(p=0.5)", + "customers.revenue:percentile(p=0.95)" + ] +} +``` + +surfaces two distinct result keys — `orders.customers.revenue_percentile_p_0_5` and `orders.customers.revenue_percentile_p_0_95`. (A non-parametric `customers.revenue:sum` surfaces as `orders.customers.revenue_sum`.) + ### Query lists Pass a list of queries to `execute()`. Earlier queries are named sub-queries, the last is the main query. Named queries can be referenced by `source_model` name or joined via `joins`: diff --git a/docs/concepts/query-cache.md b/docs/concepts/query-cache.md index cf401b9d..df6e6c8a 100644 --- a/docs/concepts/query-cache.md +++ b/docs/concepts/query-cache.md @@ -117,6 +117,17 @@ re-snap and model/schema edits are picked up. If the freshly-prepared SQL differ `refresh()` is continue-on-failure: a failed refresh-key scan or re-execution leaves the existing entry unchanged and records a `RefreshError`. +`refresh()` scans and re-executes each entry against the **datasource identity +it was cached under** (its SQL-client fingerprint), reusing the client created +at write time — so a `datasource`-priority change or a same-name connection edit +between caching and `refresh()` can never migrate a table-backed entry or read +from a different database. One residual: a run-by-name **query-backed** model +resolves its inner `source_queries` metadata through the current datasource +priority, so if the same backing-model name lives in two datasources and the +priority changes between caching and `refresh()`, the inner SQL may be shaped +from the other datasource. After repointing or re-prioritising a datasource +while a cache is live, call `clear_cache()`. + The synchronous wrappers `execute_sync(..., cache=True, data_source=...)`, `refresh_sync()`, and `evict_sync(...)` are available for CLI / notebook / script use. diff --git a/docs/concepts/references.md b/docs/concepts/references.md index 9e3a292f..d13ff05e 100644 --- a/docs/concepts/references.md +++ b/docs/concepts/references.md @@ -7,14 +7,14 @@ SLayer has two distinct expression layers and the rules for what each one accept | Mode | Fields | Parser | Accepts | Rejects | |---|---|---|---|---| | **A — SQL** | `Column.sql`, `Column.filter`, each entry of `SlayerModel.filters` | sqlglot | Any valid SQL expression for the underlying dialect — function calls (`json_extract`, `coalesce`, `nullif`, `lower`, `length`, …), arithmetic, `CASE WHEN`, string literals, comparison and boolean operators in SQL spelling (`=`, `<>`, `IS NULL`, `AND`, `OR`, `NOT`, `IN`, `LIKE`). Bare names and `__`-delimited join paths. | Aggregation colon syntax (`revenue:sum`); SLayer transform calls (`cumsum`, `change`, `rank`, …); references to `ModelMeasure` formulas; raw `OVER (...)` window functions inside `Column.filter` / `SlayerModel.filters` (allowed only in `Column.sql`). | -| **B — DSL** | `ModelMeasure.formula`, `SlayerQuery.measures`, `SlayerQuery.filters`, `SlayerQuery.dimensions`, `SlayerQuery.time_dimensions`, `SlayerQuery.order`, `SlayerQuery.main_time_dimension` | Python AST formula parser | Bare names that resolve to a `Column` or `ModelMeasure` on the model; single-dot dotted paths through joins (`customers.regions.name`, `customers.revenue:sum`); aggregation colon syntax (`:`, `*:count`, parametric forms); transform calls (`cumsum(revenue:sum)`, `rank(revenue:sum, partition_by=region)`); arithmetic / boolean / comparison operators; `LIKE` / `NOT LIKE`; the SQL `\|\|` concat operator (folded into `concat(...)`); a small allowlist of lowercase string-hygiene scalars in `SlayerQuery.filters` only — `lower`, `upper`, `trim`, `replace`, `substr`, `instr`, `length`, `concat`; `{variable}` placeholders (filters only). | `__`-delimited tokens in user input; raw SQL function calls outside the string-hygiene allowlist (`json_extract`, `coalesce`, …); raw `OVER (...)`; bare names that don't resolve to a Column / ModelMeasure / custom aggregation / query alias; **uppercase** spellings of the string-hygiene functions (`LOWER`, `TRIM`, …) — DSL is case-sensitive. | +| **B — DSL** | `ModelMeasure.formula`, `SlayerQuery.measures`, `SlayerQuery.filters`, `SlayerQuery.dimensions`, `SlayerQuery.time_dimensions`, `SlayerQuery.order`, `SlayerQuery.main_time_dimension` | Python AST formula parser | Bare names that resolve to a `Column` or `ModelMeasure` on the model; single-dot dotted paths through joins (`customers.regions.name`, `customers.revenue:sum`); aggregation colon syntax (`:`, `*:count`, parametric forms); transform calls (`cumsum(revenue:sum)`, `rank(revenue:sum, partition_by=region)`); arithmetic / boolean / comparison operators; the SQL `\|\|` concat operator (folded into `concat(...)`); pattern matching via the `like(value, pattern)` scalar (emits the SQL `LIKE` operator — wrap in `not (...)` for `NOT LIKE`); a small allowlist of lowercase string-hygiene scalars in `SlayerQuery.filters` only — `lower`, `upper`, `trim`, `replace`, `substr`, `instr`, `length`, `concat`, `like`; `{variable}` placeholders (filters only). | `__`-delimited tokens in user input; raw SQL function calls outside the string-hygiene allowlist (`json_extract`, `coalesce`, …); raw `OVER (...)`; bare names that don't resolve to a Column / ModelMeasure / custom aggregation / query alias; **uppercase** spellings of the string-hygiene functions (`LOWER`, `TRIM`, …) — DSL is case-sensitive. | ## Identifier resolution ### SQL mode (`Column.sql`, `Column.filter`, `SlayerModel.filters`) * A bare identifier `col` resolves to the column named `col` on the underlying table or SQL of this model. -* A path `a__b__c.col` resolves through the join graph: `a__b__c` is the SQL table alias produced by walking `model → a → b → c`, and `.col` is the leaf column on the final model. **`__` separates join hops only**; the leaf column always follows a single dot. The flattened form `a__b__c__col` does **not** exist in SQL mode — it appears only inside virtual-model column names produced by `_query_as_model` (see below). +* A path `a__b__c.col` resolves through the join graph: `a__b__c` is the SQL table alias produced by walking `model → a → b → c`, and `.col` is the leaf column on the final model. **`__` separates join hops only**; the leaf column always follows a single dot. The flattened form `a__b__c__col` does **not** exist in SQL mode — it appears only inside virtual-model column names produced by the query-backed model wrap (see below). * Single-dot `t.col` is a literal `.` SQL reference (sqlglot's normal behavior). * User-supplied multi-dot input (`a.b.c`) is auto-rewritten to `a__b.c` at validation time with a warning. * Other derived columns of the same model (or of a joined model via `__`) are recursively expanded so chains like `A.ratio = "A.bar / B.foo_normalized"` (where `B.foo_normalized` is itself derived) work. @@ -31,9 +31,9 @@ SLayer has two distinct expression layers and the rules for what each one accept ## The internal `__` carve-out -The `Column._validate_name` validator allows `__` inside `Column.name`. This is required by `_query_as_model`, which flattens joined-model columns into virtual-model column names like `stores__name` or `customers__regions__name` — the entire dotted path becomes one SQL identifier on the synthetic table. +The `Column._validate_name` validator allows `__` inside `Column.name`. This is required by the query-backed model wrap (`_expand_query_backed_model`, via `flat_name` in `slayer/sql/naming.py`), which flattens joined-model columns into virtual-model column names like `stores__name` or `customers__regions__name` — the entire dotted path becomes one SQL identifier on the synthetic table. -`__` is **not** rejected at SlayerQuery / ModelMeasure construction. A user-authored DSL formula or filter that references such a virtual column by name (e.g. a downstream stage filtering on `kpis__total_amount_sum`) needs to remain constructible. Instead, **strict resolution at enrichment time** catches the cases that are actually wrong: any bare name in a query measure / filter / dimension that doesn't resolve to a `Column` / `ModelMeasure` / custom aggregation / canonical agg alias / query-level alias on the source model raises `ReferenceError`. Typos like `customers__region` (against a model that has `customers` joined to `region`, but no virtual column with that flattened name) are surfaced at execution time, not at construction. +`__` is **not** rejected at SlayerQuery / ModelMeasure construction. A user-authored DSL formula or filter that references such a virtual column by name (e.g. a downstream stage filtering on `kpis__total_amount_sum`) needs to remain constructible. Instead, **strict resolution at binding time** catches the cases that are actually wrong: any bare name in a query measure / filter / dimension that doesn't resolve to a `Column` / `ModelMeasure` / custom aggregation / canonical agg alias / query-level alias on the source model raises `ReferenceError`. Typos like `customers__region` (against a model that has `customers` joined to `region`, but no virtual column with that flattened name) are surfaced at execution time, not at construction. `reject_user_dunder` in `slayer/core/refs.py` is retained as a helper for narrow contexts where `__` is unambiguously wrong (e.g. `SlayerQuery.name`, where `__` would clash with the SQL alias namespace) — it is not applied to free-form formula / filter strings. @@ -45,7 +45,7 @@ The `Column._validate_name` validator allows `__` inside `Column.name`. This is 3. **No predicate promotion.** A query filter that names a windowed `Column` raises with a suggestion to use a rank-family transform (`rank` / `percent_rank` / `dense_rank` / `ntile`) or a multi-stage `source_queries` model. The rank-family transforms cover top-N filtering in pure DSL. -4. **Single reference-resolution surface.** Identifier handling lives in `slayer/core/refs.py`; join walks live in `_walk_join_chain` in the engine. +4. **Single reference-resolution surface.** Identifier handling lives in `slayer/core/refs.py`; join walks live in the binder (`slayer/engine/binding.py`), which resolves each hop against the resolved source bundle. ## Examples — accepted and rejected diff --git a/docs/database-support.md b/docs/database-support.md index fa652472..03f7d7bb 100644 --- a/docs/database-support.md +++ b/docs/database-support.md @@ -220,7 +220,7 @@ plus `$GCP_PROJECT_ID` for billing). The `bigquery://` driver requires the rewrites `.` aliases (`orders._count`, `orders.products.category`) to `___` at emit time and reverses the mapping on result rows. The triple-underscore separator is - distinct from `__` (used by `_query_as_model` for cross-model leaf + distinct from `__` (used by the query-backed virtual-model wrap for cross-model leaf flattening), so the two encodings never collide. In `Column.sql`, fully-qualified table paths must be backticked per-segment (`` `project`.`dataset`.`table` ``) — a single backticked dotted path of diff --git a/docs/development.md b/docs/development.md index 60a1efb1..e0b1c6cd 100644 --- a/docs/development.md +++ b/docs/development.md @@ -27,6 +27,26 @@ poetry run pytest tests/integration/test_integration.py poetry run pytest tests/test_mcp_server.py -v ``` +### Scope-closure validation + +The test suite validates every generated SQL statement for **scope closure** — +that no SELECT scope references a table alias it does not bind, and no +cross-scope reference names a column an inner scope does not project. This runs +automatically under pytest (`SLAYER_VALIDATE_SCOPES=1` is set for the whole +suite), so a scope leak fails at generation time. + +To enable the same check at runtime for debugging (e.g. when a generated query +fails against a live database), set the environment variable: + +```bash +SLAYER_VALIDATE_SCOPES=1 poetry run slayer serve +``` + +A provable out-of-scope reference raises `ScopeLeakError` from +`slayer/sql/scope_check.py`. The validator is deliberately conservative (no +false positives): unqualified references and physical-table column names are +treated as unverifiable and never flagged. + ## Linting ```bash @@ -48,7 +68,8 @@ slayer/ engine/ query_engine.py # SlayerQueryEngine — central orchestrator ingestion.py # Auto-ingestion with rollup-style FK joins - enriched.py # EnrichedQuery — fully resolved query for SQL generation + stage_planner.py # SlayerQuery --> PlannedQuery (typed keys, slots, phases) + planned.py # PlannedQuery — fully resolved query for SQL generation storage/ base.py # StorageBackend ABC yaml_storage.py # YAML file storage diff --git a/docs/examples/02_sql_vs_dsl/sql_vs_dsl_nb.ipynb b/docs/examples/02_sql_vs_dsl/sql_vs_dsl_nb.ipynb index 34beafc2..fb254fa4 100644 --- a/docs/examples/02_sql_vs_dsl/sql_vs_dsl_nb.ipynb +++ b/docs/examples/02_sql_vs_dsl/sql_vs_dsl_nb.ipynb @@ -25,10 +25,10 @@ "id": "8785c43a", "metadata": { "execution": { - "iopub.execute_input": "2026-05-05T20:19:03.971500Z", - "iopub.status.busy": "2026-05-05T20:19:03.971386Z", - "iopub.status.idle": "2026-05-05T20:19:04.151515Z", - "shell.execute_reply": "2026-05-05T20:19:04.151069Z" + "iopub.execute_input": "2026-05-27T14:24:25.372595Z", + "iopub.status.busy": "2026-05-27T14:24:25.371226Z", + "iopub.status.idle": "2026-05-27T14:24:30.107987Z", + "shell.execute_reply": "2026-05-27T14:24:30.099565Z" } }, "outputs": [], @@ -62,10 +62,10 @@ "id": "4bbdb270", "metadata": { "execution": { - "iopub.execute_input": "2026-05-05T20:19:04.152763Z", - "iopub.status.busy": "2026-05-05T20:19:04.152695Z", - "iopub.status.idle": "2026-05-05T20:19:04.154781Z", - "shell.execute_reply": "2026-05-05T20:19:04.154539Z" + "iopub.execute_input": "2026-05-27T14:24:30.127282Z", + "iopub.status.busy": "2026-05-27T14:24:30.125235Z", + "iopub.status.idle": "2026-05-27T14:24:30.161721Z", + "shell.execute_reply": "2026-05-27T14:24:30.154353Z" } }, "outputs": [ @@ -76,13 +76,13 @@ "orders model has 7 columns:\n", "\n", "Columns (name -> SQL expression):\n", - " id -> sql: 'id' type: string [PK]\n", - " customer_id -> sql: 'customer_id' type: string\n", - " ordered_at -> sql: 'ordered_at' type: date\n", - " store_id -> sql: 'store_id' type: string\n", - " subtotal -> sql: 'subtotal' type: number\n", - " tax_paid -> sql: 'tax_paid' type: number\n", - " order_total -> sql: 'order_total' type: number\n" + " id -> sql: 'id' type: TEXT [PK]\n", + " customer_id -> sql: 'customer_id' type: TEXT\n", + " ordered_at -> sql: 'ordered_at' type: DATE\n", + " store_id -> sql: 'store_id' type: TEXT\n", + " subtotal -> sql: 'subtotal' type: DOUBLE\n", + " tax_paid -> sql: 'tax_paid' type: DOUBLE\n", + " order_total -> sql: 'order_total' type: DOUBLE\n" ] } ], @@ -132,10 +132,10 @@ "id": "b6a98979", "metadata": { "execution": { - "iopub.execute_input": "2026-05-05T20:19:04.155914Z", - "iopub.status.busy": "2026-05-05T20:19:04.155859Z", - "iopub.status.idle": "2026-05-05T20:19:04.210358Z", - "shell.execute_reply": "2026-05-05T20:19:04.209996Z" + "iopub.execute_input": "2026-05-27T14:24:30.176710Z", + "iopub.status.busy": "2026-05-27T14:24:30.175123Z", + "iopub.status.idle": "2026-05-27T14:24:32.225452Z", + "shell.execute_reply": "2026-05-27T14:24:32.216376Z" } }, "outputs": [ @@ -145,14 +145,19 @@ "text": [ "Month Revenue MoM Change\n", "--------------------------------------\n", - "2024-05 $ 118,106.62 $+17,512\n", - "2024-06 $ 154,699.94 $+36,593\n", - "2024-07 $ 168,960.69 $+14,261\n", - "2024-08 $ 180,589.12 $+11,628\n", - "2024-09 $ 177,773.88 $-2,815\n", - "2024-10 $ 187,103.17 $+9,329\n", + "2024-05 $ 114,871.03 $+16,044\n", + "2024-06 $ 134,270.36 $+19,399\n", + "2024-07 $ 166,394.63 $+32,124\n", + "2024-08 $ 180,528.01 $+14,133\n", + "2024-09 $ 178,288.75 $-2,239\n", + "2024-10 $ 187,832.31 $+9,544\n", "\n", - " SQL of the query WITH base AS (\n", + " SQL of the query SELECT\n", + " \"orders.ordered_at\",\n", + " \"orders.order_total_sum\",\n", + " \"orders.mom_change\"\n", + "FROM (\n", + "WITH base AS (\n", "SELECT\n", " DATE_TRUNC('MONTH', orders.ordered_at) AS \"orders.ordered_at\",\n", " SUM(orders.order_total) AS \"orders.order_total_sum\"\n", @@ -160,34 +165,35 @@ "GROUP BY\n", " DATE_TRUNC('MONTH', orders.ordered_at)\n", "),\n", - "shifted__ts_mom_change AS (\n", + "shifted__time_shift_inner AS (\n", "SELECT\n", - " DATE_TRUNC('MONTH', orders.ordered_at + INTERVAL '1' MONTH) AS \"orders.ordered_at\",\n", + " DATE_TRUNC('MONTH', orders.ordered_at + INTERVAL 1 MONTH) AS \"orders.ordered_at\",\n", " SUM(orders.order_total) AS \"orders.order_total_sum\"\n", "FROM orders AS orders\n", "GROUP BY\n", - " DATE_TRUNC('MONTH', orders.ordered_at + INTERVAL '1' MONTH)\n", + " DATE_TRUNC('MONTH', orders.ordered_at + INTERVAL 1 MONTH)\n", "),\n", - "sjoin__ts_mom_change AS (\n", - "SELECT base.\"orders.order_total_sum\", base.\"orders.ordered_at\", shifted__ts_mom_change.\"orders.order_total_sum\" AS \"orders._ts_mom_change\"\n", + "sjoin__time_shift_inner AS (\n", + "SELECT base.\"orders.order_total_sum\", base.\"orders.ordered_at\", shifted__time_shift_inner.\"orders.order_total_sum\" AS \"orders._time_shift_inner\"\n", "FROM base\n", - "LEFT JOIN shifted__ts_mom_change\n", - " ON base.\"orders.ordered_at\" = shifted__ts_mom_change.\"orders.ordered_at\"\n", + "LEFT JOIN shifted__time_shift_inner\n", + " ON base.\"orders.ordered_at\" = shifted__time_shift_inner.\"orders.ordered_at\"\n", "),\n", - "step2 AS (\n", + "step1 AS (\n", "SELECT\n", - " \"orders._ts_mom_change\",\n", + " \"orders._time_shift_inner\",\n", " \"orders.order_total_sum\",\n", " \"orders.ordered_at\",\n", - " \"orders.order_total_sum\" - \"orders._ts_mom_change\" AS \"orders.mom_change\"\n", - "FROM sjoin__ts_mom_change\n", + " \"orders.order_total_sum\" - \"orders._time_shift_inner\" AS \"orders.mom_change\"\n", + "FROM sjoin__time_shift_inner\n", ")\n", "SELECT\n", - " \"orders._ts_mom_change\",\n", + " \"orders._time_shift_inner\",\n", " \"orders.mom_change\",\n", " \"orders.order_total_sum\",\n", " \"orders.ordered_at\"\n", - "FROM step2\n", + "FROM step1\n", + ") AS _outer\n", "ORDER BY \"orders.ordered_at\" ASC\n", "LIMIT 6\n", "OFFSET 12\n" @@ -245,10 +251,10 @@ "id": "56baba38", "metadata": { "execution": { - "iopub.execute_input": "2026-05-05T20:19:04.211664Z", - "iopub.status.busy": "2026-05-05T20:19:04.211590Z", - "iopub.status.idle": "2026-05-05T20:19:04.250403Z", - "shell.execute_reply": "2026-05-05T20:19:04.250015Z" + "iopub.execute_input": "2026-05-27T14:24:32.250993Z", + "iopub.status.busy": "2026-05-27T14:24:32.246549Z", + "iopub.status.idle": "2026-05-27T14:24:33.078293Z", + "shell.execute_reply": "2026-05-27T14:24:33.071928Z" } }, "outputs": [ @@ -256,8 +262,8 @@ "name": "stdout", "output_type": "stream", "text": [ - " Brooklyn: 256,532 orders, $2,793,658.90\n", - " Philadelphia: 192,450 orders, $2,099,292.44\n", + " Brooklyn: 252,910 orders, $2,838,779.79\n", + " Philadelphia: 193,786 orders, $2,175,586.96\n", "\n", "The filter 'stores.name' resolved to SQL:\n", " WHERE\n", @@ -304,10 +310,10 @@ "id": "e3947bac", "metadata": { "execution": { - "iopub.execute_input": "2026-05-05T20:19:04.251470Z", - "iopub.status.busy": "2026-05-05T20:19:04.251398Z", - "iopub.status.idle": "2026-05-05T20:19:04.266120Z", - "shell.execute_reply": "2026-05-05T20:19:04.265743Z" + "iopub.execute_input": "2026-05-27T14:24:33.100876Z", + "iopub.status.busy": "2026-05-27T14:24:33.097649Z", + "iopub.status.idle": "2026-05-27T14:24:33.399629Z", + "shell.execute_reply": "2026-05-27T14:24:33.391348Z" } }, "outputs": [ @@ -316,11 +322,11 @@ "output_type": "stream", "text": [ "Orders where subtotal > 5x tax (raw SQL filter via ModelExtension):\n", - " Brooklyn: 256,532 orders, $2,793,658.90\n", - " Philadelphia: 192,450 orders, $2,099,292.44\n", - " Chicago: 106,290 orders, $1,183,765.61\n", - " San Francisco: 90,401 orders, $1,059,640.66\n", - " New Orleans: 15,473 orders, $180,782.69\n" + " Brooklyn: 252,910 orders, $2,838,779.79\n", + " Philadelphia: 193,786 orders, $2,175,586.96\n", + " Chicago: 106,715 orders, $1,146,106.66\n", + " San Francisco: 92,703 orders, $1,026,546.09\n", + " New Orleans: 15,398 orders, $170,325.62\n" ] } ], @@ -398,7 +404,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.11.14" + "version": "3.11.11" } }, "nbformat": 4, diff --git a/docs/examples/11_dbt_metricflow/setup_metricflow.py b/docs/examples/11_dbt_metricflow/setup_metricflow.py index e2eef1a2..5cc1f47a 100644 --- a/docs/examples/11_dbt_metricflow/setup_metricflow.py +++ b/docs/examples/11_dbt_metricflow/setup_metricflow.py @@ -24,6 +24,7 @@ import logging import shutil import subprocess +import time from pathlib import Path from typing import List @@ -61,6 +62,48 @@ class MetricFlowDemoError(RuntimeError): genuine conversion bug.""" +# Substrings (matched case-insensitively against git's stderr) that mark a fetch +# failure as a transient network/server hiccup — a 5xx/429 from GitHub, DNS, or a +# dropped connection — rather than a deterministic error (bad SHA, auth). These +# are worth retrying, and worth skipping (not failing) the integration test on. +_TRANSIENT_GIT_SIGNATURES = ( + "the requested url returned error: 500", + "the requested url returned error: 502", + "the requested url returned error: 503", + "the requested url returned error: 504", + "the requested url returned error: 429", + "error: 429", + "could not resolve host", + "failed to connect", + "connection reset", + "connection timed out", + "timed out", + "empty reply from server", + "recv failure", + "send failure", + "gnutls_handshake", + "ssl_read", + "early eof", + "rpc failed", + "remote end hung up", + "unexpectedly closed", +) + +# Retry knobs for the shallow fetch. Small, bounded backoff: enough to ride out a +# brief GitHub blip without slowing CI when the very first attempt succeeds. +_FETCH_RETRIES = 3 +_FETCH_BACKOFF_SECONDS = 2.0 + + +def _is_transient_git_error(text: str) -> bool: + """True if ``text`` (a git stderr string) looks like a transient network or + server failure rather than a deterministic one. Used both to decide whether a + failed fetch is worth retrying and whether the integration test should skip + instead of fail.""" + lowered = text.lower() + return any(sig in lowered for sig in _TRANSIENT_GIT_SIGNATURES) + + def _git(*args: str, cwd: Path) -> str: """Run a git command, returning stripped stdout. Raises on failure.""" result = subprocess.run( @@ -73,6 +116,28 @@ def _git(*args: str, cwd: Path) -> str: return result.stdout.strip() +def _fetch_pinned_commit(tmp: Path) -> None: + """Shallow-fetch the pinned SHA into ``tmp``, retrying on transient network + errors (GitHub 5xx/429, DNS, dropped connections). Deterministic failures + (bad SHA, missing ``git``) raise on the first attempt.""" + for attempt in range(1, _FETCH_RETRIES + 1): + try: + _git("fetch", "--depth", "1", "origin", DBT_PIN_SHA, cwd=tmp) + return + except subprocess.CalledProcessError as exc: + stderr = exc.stderr or "" + if attempt < _FETCH_RETRIES and _is_transient_git_error(stderr): + logger.warning( + "Transient git fetch failure (attempt %d/%d), retrying: %s", + attempt, + _FETCH_RETRIES, + stderr.strip(), + ) + time.sleep(_FETCH_BACKOFF_SECONDS * attempt) + continue + raise + + def _checkout_is_valid(checkout: Path) -> bool: """True iff ``checkout`` is a git repo whose HEAD is the pinned commit.""" if not (checkout / ".git").exists(): @@ -110,10 +175,11 @@ def clone_dbt_project() -> Path: try: # Shallow-fetch the exact pinned commit. GitHub allows fetching an - # unadvertised SHA, so we never depend on the branch tip. + # unadvertised SHA, so we never depend on the branch tip. The fetch is + # retried on transient network failures (GitHub 5xx/429, DNS, resets). _git("init", "-q", cwd=tmp) _git("remote", "add", "origin", DBT_REPO_URL, cwd=tmp) - _git("fetch", "--depth", "1", "origin", DBT_PIN_SHA, cwd=tmp) + _fetch_pinned_commit(tmp) _git("checkout", "-q", "FETCH_HEAD", cwd=tmp) head = _git("rev-parse", "HEAD", cwd=tmp) if head != DBT_PIN_SHA: diff --git a/docs/index.md b/docs/index.md index dca11ca9..e2ce53ab 100644 --- a/docs/index.md +++ b/docs/index.md @@ -75,7 +75,7 @@ Agent --> MCP / REST API / Python SDK | SlayerQueryEngine (resolves model definitions from storage) | - EnrichedQuery (resolved SQL expressions, model metadata) + PlannedQuery (typed value keys, slots, join paths, phases) | SQLGenerator (sqlglot AST --> dialect-aware SQL) | @@ -84,6 +84,6 @@ Agent --> MCP / REST API / Python SDK SlayerResponse (data, columns, sql) ``` -**SlayerQuery** is what the user sends — names and references, no SQL. **EnrichedQuery** is the engine-internal form where every measure and dimension carries its resolved SQL, aggregation, and model context. New datasource adapters only need to translate EnrichedQuery. +**SlayerQuery** is what the user sends — names and references, no SQL. **PlannedQuery** is the engine-internal form: every measure and dimension is a typed *value key* interned into a slot, carrying its resolved expression, aggregation, join path, and phase. New datasource adapters only need to translate PlannedQuery. Full concept docs: [Models](concepts/models.md) | [Queries](concepts/queries.md) | [Formulas](concepts/formulas.md) diff --git a/slayer/core/enums.py b/slayer/core/enums.py index df35f0c9..a1f9bcac 100644 --- a/slayer/core/enums.py +++ b/slayer/core/enums.py @@ -1,6 +1,7 @@ """Core enums for SLayer.""" import datetime # noqa: F401 (kept for downstream imports of TimeGranularity) +import difflib from enum import Enum from typing import Any @@ -211,6 +212,21 @@ def normalize_aggregation_name(name: str) -> str: return candidate if candidate in BUILTIN_AGGREGATIONS else name +def format_unknown_aggregation(name: str, known: "set[str] | frozenset[str]") -> str: + """DEV-1576: the shared 'Unknown aggregation' error message. + + Used by both the binder's aggregation gate (``slayer/engine/binding.py``) + and the typed binding gate (``slayer/engine/binding.py``) so the wording + stays byte-identical: an unknown aggregation name is distinguished from a + known-but-disallowed one, with a close-match suggestion and the model-wide + known list. ``known`` = ``BUILTIN_AGGREGATIONS`` unioned with the owning + model's custom aggregation names. + """ + suggestion = difflib.get_close_matches(word=name, possibilities=sorted(known), n=1) + hint = f" Did you mean '{suggestion[0]}'?" if suggestion else "" + return f"Unknown aggregation '{name}'.{hint} Known: {sorted(known)}." + + # Built-in aggregation SQL formulas (for aggregations that use a template). # {value} = measure's SQL expression; {param_name} = parameter values. # Note: percentile is dialect-dependent (no single template works on @@ -230,7 +246,7 @@ def normalize_aggregation_name(name: str) -> str: # Aggregations that only make sense on numeric-valued measures. Applying them # to a non-numeric measure (e.g. AVG on a VARCHAR column) is always invalid -# and is rejected during query enrichment rather than at SQL execution time. +# and is rejected during query binding rather than at SQL execution time. # min, max, count, count_distinct, first, last work on any type and are NOT # in this set. NUMERIC_ONLY_AGGREGATIONS: frozenset[str] = frozenset({ diff --git a/slayer/core/errors.py b/slayer/core/errors.py index c02f38c1..9882bd24 100644 --- a/slayer/core/errors.py +++ b/slayer/core/errors.py @@ -7,7 +7,7 @@ from __future__ import annotations -from typing import TYPE_CHECKING, Any +from typing import TYPE_CHECKING, Any, List, Tuple if TYPE_CHECKING: from slayer.engine.schema_drift import ToDeleteEntry # noqa: F401 @@ -116,6 +116,310 @@ def __init__(self, cycle: list[tuple[str, str]]) -> None: super().__init__(f"Circular column reference detected: {chain}") +# --------------------------------------------------------------------------- +# DEV-1450 stage-5 errors — typed, stable str() format. +# +# All error classes below build their message via ``_format_error_message`` +# so ``str(error)`` follows the documented snapshot-friendly shape:: +# +# : +# at +# scope: +# suggestion: +# +# Each indented line is optional. The first line ALWAYS starts with the +# class name so log greps and snapshot tests bind to a stable prefix. +# --------------------------------------------------------------------------- + + +def _format_error_message( + *, + cls_name: str, + summary: str, + location: str | None = None, + scope: str | None = None, + suggestion: str | None = None, + extras: List[Tuple[str, str]] | None = None, +) -> str: + """Build the stable error-message string used by stage-5 error classes. + + ``extras`` lets a class add bespoke key/value rows after the summary + while keeping the leading ``ClassName:`` token intact. + """ + lines = [f"{cls_name}: {summary}"] + if location: + lines.append(f" at {location}") + if scope: + lines.append(f" scope: {scope}") + for k, v in (extras or []): + lines.append(f" {k}: {v}") + if suggestion: + lines.append(f" suggestion: {suggestion}") + return "\n".join(lines) + + +class UnknownReferenceError(SlayerError, ValueError): + """A bare or dotted reference cannot be resolved in the current scope. + + Multi-inherits ``ValueError`` (like :class:`ColumnCycleError`) so the + pre-existing call sites and tests that catch ``ValueError`` for a failed + reference / model resolution keep working after the DEV-1450 cutover + replaced the legacy ``ValueError`` resolution paths with this typed error. + """ + + def __init__( + self, + name: str, + scope_kind: str, + scope_summary: str, + suggestion: str | None = None, + ) -> None: + self.name = name + self.scope_kind = scope_kind + self.scope_summary = scope_summary + self.suggestion = suggestion + super().__init__(_format_error_message( + cls_name=type(self).__name__, + summary=f"Cannot resolve reference {name!r}.", + scope=f"{scope_kind}: {scope_summary}", + suggestion=suggestion, + )) + + +class AmbiguousReferenceError(SlayerError, ValueError): + """A reference matches multiple candidates in scope and can't pick one. + + Multi-inherits ``ValueError`` for back-compat (see + :class:`UnknownReferenceError`). + """ + + def __init__(self, name: str, candidates: List[str]) -> None: + self.name = name + self.candidates = sorted(candidates) + super().__init__(_format_error_message( + cls_name=type(self).__name__, + summary=f"Reference {name!r} has multiple candidates.", + extras=[("candidates", repr(self.candidates))], + )) + + +class IllegalScopeReferenceError(SlayerError, ValueError): + """A reference is syntactically rejected by the current scope kind. + + Examples: ``__`` in a Mode-B ``ModelScope`` ref (use the dotted form); + a dotted ref against a ``StageSchema`` (downstream stages see a flat + namespace, no join syntax). + + Multi-inherits ``ValueError`` for back-compat (see + :class:`UnknownReferenceError`). + """ + + def __init__(self, name: str, scope_kind: str, reason: str) -> None: + self.name = name + self.scope_kind = scope_kind + self.reason = reason + super().__init__(_format_error_message( + cls_name=type(self).__name__, + summary=f"Reference {name!r} is not legal in this scope.", + scope=scope_kind, + extras=[("reason", reason)], + )) + + +class IllegalWindowInFilterError(SlayerError, ValueError): + """A filter contains a raw ``OVER(...)`` window expression, or refers + to a ``Column.sql`` whose body contains a window function (DEV-1369 / + DEV-1336 — predicate promotion was removed). Use a rank-family + transform instead. + + Multi-inherits ``ValueError`` (like :class:`UnknownReferenceError`) so + the pre-existing call sites and tests that catch ``ValueError`` for the + legacy windowed-filter rejection keep working after the cutover. + """ + + def __init__( + self, + filter_expr: str, + source: str, + suggestion: str = "use a rank-family transform (e.g. `rank() <= N`).", + ) -> None: + self.filter_expr = filter_expr + self.source = source + self.suggestion = suggestion + super().__init__(_format_error_message( + cls_name=type(self).__name__, + summary="Window expressions are not allowed in filters.", + extras=[ + ("expr", repr(filter_expr)), + ("source", source), + ], + suggestion=suggestion, + )) + + +class AggregationNotAllowedError(SlayerError, ValueError): + """An aggregation cannot apply to a column. + + Covers type-bucket violations (``sum`` on TEXT), primary-key + restrictions (only ``count`` / ``count_distinct``), and explicit + ``Column.allowed_aggregations`` whitelist violations. + + Subclasses ``ValueError`` (like the other resolution-time errors in + this module) so callers wrapping the engine in ``except ValueError`` + keep catching aggregation-gating failures — the legacy enrichment + pipeline raised a bare ``ValueError`` here. + """ + + def __init__(self, column: str, agg: str, reason: str) -> None: + self.column = column + self.agg = agg + self.reason = reason + super().__init__(_format_error_message( + cls_name=type(self).__name__, + summary=f"Aggregation {agg!r} is not allowed on column {column!r}.", + extras=[("reason", reason)], + )) + + +class UnknownFunctionError(SlayerError, ValueError): + """A function call in Mode B is not in the ``SCALAR_FUNCTIONS`` allowlist, + the transform registry, or the model's aggregation set (C12). + + Subclasses ``ValueError`` (like the other binding-time errors here) so + the REST ``ValueError -> 400`` mapping and ``except ValueError`` callers + keep catching it — the legacy enrichment pipeline raised a bare + ``ValueError`` for this case. + """ + + _DEFAULT_SUGGESTION = "move the call to a derived Column.sql (Mode A)." + + def __init__( + self, + name: str, + location: str, + suggestion: str | None = None, + ) -> None: + self.name = name + self.location = location + self.suggestion = suggestion or self._DEFAULT_SUGGESTION + super().__init__(_format_error_message( + cls_name=type(self).__name__, + summary=f"Function {name!r} is not allowed in Mode B.", + location=location, + suggestion=self.suggestion, + )) + + +class MeasureRecursionLimitError(SlayerError, ValueError): + """Named-measure expansion exceeded the configurable depth limit + (default 32; ``SLAYER_MEASURE_EXPANSION_DEPTH``). + + ValueError-derived for REST/caller parity with the other binding-time + errors (the legacy pipeline raised a bare ``ValueError``). + """ + + def __init__(self, chain: List[str], limit: int = 32) -> None: + self.chain = list(chain) + self.limit = limit + super().__init__(_format_error_message( + cls_name=type(self).__name__, + summary=f"Named-measure expansion exceeded depth (limit={limit}).", + extras=[("chain", " → ".join(self.chain))], + )) + + +class MeasureCycleError(SlayerError, ValueError): + """Named-measure expansion encountered a cycle. + + ValueError-derived for REST/caller parity with the other binding-time + errors (the legacy pipeline raised a bare ``ValueError``). + """ + + def __init__(self, chain: List[str]) -> None: + self.chain = list(chain) + super().__init__(_format_error_message( + cls_name=type(self).__name__, + summary="Cyclic reference in named-measure expansion.", + extras=[("chain", " → ".join(self.chain))], + )) + + +class DuplicateMeasureNameError(SlayerError, ValueError): + """Two measures in the same query declare the same explicit ``name`` + (DEV-1443). + + ValueError-derived for REST/caller parity with the other binding-time + errors (the legacy pipeline raised a bare ``ValueError``). + """ + + def __init__(self, name: str, occurrences: List[str]) -> None: + self.name = name + self.occurrences = list(occurrences) + super().__init__(_format_error_message( + cls_name=type(self).__name__, + summary=f"Measure name {name!r} is declared more than once.", + extras=[("occurrences", repr(self.occurrences))], + )) + + +class MeasureNameCollidesWithColumnError(SlayerError, ValueError): + """A declared measure ``name`` matches a source column on the model + (DEV-1443) — the alias-form filter would silently bind to the source + column instead of the aggregate. + + ValueError-derived for REST/caller parity with the other binding-time + errors (the legacy pipeline raised a bare ``ValueError``). + """ + + def __init__(self, name: str, model: str) -> None: + self.name = name + self.model = model + super().__init__(_format_error_message( + cls_name=type(self).__name__, + summary=( + f"Declared measure name {name!r} matches a source column on " + f"model {model!r}." + ), + )) + + +class CanonicalAliasShadowsColumnError(SlayerError, ValueError): + """A formula's canonical alias (e.g., ``amount_sum`` for ``amount:sum``) + shadows a source column on the same model (DEV-1443). + + ValueError-derived for REST/caller parity with the other binding-time + errors (the legacy pipeline raised a bare ``ValueError``). + """ + + def __init__(self, formula: str, canonical: str, model: str) -> None: + self.formula = formula + self.canonical = canonical + self.model = model + super().__init__(_format_error_message( + cls_name=type(self).__name__, + summary=( + f"Canonical alias {canonical!r} for formula {formula!r} " + f"shadows a source column on model {model!r}." + ), + )) + + +class UnreachableFilterDroppedWarning(UserWarning): + """A host filter referenced slots that aren't reachable from a + cross-model CTE's root, so the filter was dropped from the CTE. + The host query still applies the filter to its own rows; this is a + visibility/debug warning, not an error. + """ + + def __init__(self, filter_text: str, reason: str) -> None: + self.filter_text = filter_text + self.reason = reason + super().__init__( + f"Filter {filter_text!r} dropped from cross-model CTE " + f"(unreachable from CTE root): {reason}" + ) + + class IdCollisionError(SlayerError, ValueError): """Raised by filename-backed (YAML) storage when saving an entity whose id differs from an existing id only by letter case — such ids diff --git a/slayer/core/formula.py b/slayer/core/formula.py index d41dd5a2..aacaba9a 100644 --- a/slayer/core/formula.py +++ b/slayer/core/formula.py @@ -932,7 +932,6 @@ def _parse_transform_kwargs( # NOSONAR S3776 — straight-line whitelist + per- # --------------------------------------------------------------------------- # Internal filter functions (used after pre-processing operators like `like`) -FILTER_FUNCTIONS = {"__like__", "__notlike__"} class ParsedFilter(BaseModel): diff --git a/slayer/core/keys.py b/slayer/core/keys.py new file mode 100644 index 00000000..b07296cf --- /dev/null +++ b/slayer/core/keys.py @@ -0,0 +1,755 @@ +"""Stage 1 (DEV-1450) — typed identity primitives for the new resolution +pipeline. + +Identity is structural (P2 of the DEV-1450 spec). Two expression occurrences +with the same key intern to the same slot — whether the occurrence is a +declared measure, an inner reference inside a transform, or a filter +predicate. + +Rendering state (SQL text, public alias, projection position, hidden-ness) +does not live here. Those decisions belong to the planner and the SQL +generator. The keys carry only what's needed to decide "are these the same +slot?". + +Public types: ``ValueKey`` (Union alias), ``Phase`` (IntEnum), ``ColumnKey``, +``ColumnSqlKey``, ``StarKey``, ``SqlExprKey``, ``AggregateKey``, +``TransformKey``, ``ArithmeticKey``, ``ScalarCallKey``. Helpers: +``normalize_scalar``, ``SCALAR_FUNCTIONS``. + +These types are dormant in stage 1 — no engine code routes through them. +Stages 7a and 7b wire them up. +""" + +from __future__ import annotations + +from decimal import Decimal +from enum import IntEnum +from typing import Optional, Tuple, Union + +from pydantic import BaseModel, ConfigDict, field_validator + + +# --------------------------------------------------------------------------- +# Closed scalar-function allowlist (C12). +# --------------------------------------------------------------------------- + +# Anything outside this set in Mode B raises ``UnknownFunctionError`` at +# binding time. Lives here (not in formula.py) so the keys module is the +# single source of truth for what counts as a structurally-keyed scalar +# call. The binder (stage 7a) imports from here. +SCALAR_FUNCTIONS: frozenset[str] = frozenset({ + # Null handling + "nullif", "coalesce", "ifnull", + # Math + "ln", "log10", "log2", "log", "exp", "sqrt", "pow", "power", + "abs", "floor", "ceil", "round", + # String hygiene (was DEV-1378's STRING_HYGIENE_OPS) + "lower", "upper", "trim", "replace", "substr", "instr", "length", "concat", + # Pattern match — ``like(value, pattern)`` emits the SQL ``LIKE`` operator + # (sqlglot ``exp.Like``); see SQLGenerator scalar-call rendering. + "like", +}) + + +# --------------------------------------------------------------------------- +# Phase +# --------------------------------------------------------------------------- + + +class Phase(IntEnum): + """Resolution phase of a ValueKey (P8). + + Filters and arithmetic compose by taking the maximum phase of their + operands; the filter's phase then routes it to WHERE (ROW), HAVING + (AGGREGATE), or post-filter on the outer SELECT (POST). + """ + + ROW = 0 + AGGREGATE = 1 + POST = 2 + + +# --------------------------------------------------------------------------- +# Scalar +# --------------------------------------------------------------------------- + +Scalar = Union[Decimal, str, bool, None] + + +def normalize_scalar(value): + """Canonicalize a raw scalar before keying. + + - Booleans pass through unchanged (checked BEFORE int because bool + is-a int in Python). + - ``None`` passes through unchanged. + - ``Decimal`` passes through unchanged. + - ``int`` becomes ``Decimal(value)``. + - ``float`` becomes ``Decimal(str(value))`` — via ``str`` so floats + land on their displayed decimal form, not their binary + approximation (``Decimal(0.5)`` differs from ``Decimal("0.5")``). + - ``str`` passes through unchanged. + + Raises ``TypeError`` for anything else (lists, dicts, custom objects). + Caller-side conversion of identifiers to ``ColumnKey`` happens in the + binder; this helper does not touch ColumnKey. + """ + if isinstance(value, bool): + return value + if value is None: + return None + if isinstance(value, Decimal): + return value + if isinstance(value, int): + return Decimal(value) + if isinstance(value, float): + return Decimal(str(value)) + if isinstance(value, str): + return value + raise TypeError( + f"Cannot normalize scalar of type {type(value).__name__!r}: " + f"only int/float/Decimal/str/bool/None are accepted (got {value!r})." + ) + + +# --------------------------------------------------------------------------- +# Base +# --------------------------------------------------------------------------- + + +class _FrozenKey(BaseModel): + """Common config for the typed-key family: frozen (hashable, immutable).""" + + model_config = ConfigDict(frozen=True) + + +def _typed_leaf(v): + """Return a hash- and equality-friendly representation of a scalar + leaf that does NOT conflate numerically-equal values of different + types. + + Python collapses ``True == 1 == Decimal("1")`` (and the same for + ``False`` / ``0``), so a key built from ``args=(True,)`` would + intern with one built from ``args=(Decimal("1"),)`` if the + container's hash/eq blindly delegate to tuple-of-bare-values. + Wrapping the leaf in a ``(type_tag, value)`` pair at hash/eq time + restores the type distinction without changing the stored + representation users see via ``key.args[0]``. + + ``ValueKey`` leaves (ColumnKey, AggregateKey, ...) are themselves + frozen Pydantic models with value-based equality — they ride in the + generic ``("__key__", v)`` slot. Every branch returns a uniform + ``(tag, value)`` pair so callers never have to special-case the + container shape. + """ + if isinstance(v, bool): + return ("__bool__", v) + if v is None: + return ("__none__", None) + if isinstance(v, Decimal): + return ("__num__", v) + if isinstance(v, str): + return ("__str__", v) + return ("__key__", v) + + +def _typed_args(args): + return tuple(_typed_leaf(a) for a in args) + + +def _typed_kwargs(kwargs): + return tuple((k, _typed_leaf(v)) for k, v in kwargs) + + +# --------------------------------------------------------------------------- +# Row-phase keys +# --------------------------------------------------------------------------- + + +class ColumnKey(_FrozenKey): + """Row-level reference to a base column on a model. + + ``path`` is the join walk from the query's source model to the + terminal model — empty for local refs, non-empty for joined refs + (``("customers",)``, ``("customers", "regions")``, …). ``leaf`` is + the column name on the terminal model. + + Local and cross-model references share this shape (P3) — the only + difference is whether ``path`` is empty. The planner uses + ``path == ()`` to decide whether to materialize the value in the + base CTE or in a cross-model sub-query. + """ + + path: Tuple[str, ...] = () + leaf: str + + @property + def phase(self) -> Phase: + return Phase.ROW + + +class ColumnSqlKey(_FrozenKey): + """Reference to a derived column (one whose ``Column.sql`` is set). + + The expansion AST is recovered from the model definition at binding + time — the key only carries identity. Two references to the same + derived column on the same model intern to one slot. + + ``path`` is the join walk from the query's source model to the + model that owns the derived column — empty for local references, + non-empty for joined ones (``("customers",)``, + ``("customers", "regions")``, …). Cross-model planners use + ``path`` the same way they use ``ColumnKey.path``. + """ + + path: Tuple[str, ...] = () + model: str + column_name: str + + @property + def phase(self) -> Phase: + return Phase.ROW + + +class TimeTruncKey(_FrozenKey): + """Row-level reference to a time-truncated column (DEV-1450 stage 7b.3). + + Identifies a time dimension by ``(column, granularity)``. The + underlying column is recoverable via ``column`` so date-range filters + can bind against the raw column independently of the truncation. + + Identity is structural: two ``TimeTruncKey``s with the same + ``column`` and the same ``granularity`` intern to the same slot; + different granularities on the same column are distinct slots. This + lets the ``ValueRegistry`` keep month / day / raw uses of the same + column as separate materialised values without special-casing. + + ``column`` is a ``ColumnKey`` (base temporal column) or a + ``ColumnSqlKey`` (DEV-1450 follow-up #4a — a DERIVED temporal column + whose ``Column.sql`` is set). The SQL generator applies the + ``DATE_TRUNC`` over the bare identifier (``ColumnKey``) or over the + expanded derived expression (``ColumnSqlKey``). + + ``granularity`` is the string value of a ``TimeGranularity`` member + (``"day"`` / ``"month"`` / ...). Stored as ``str`` so the key stays + a pure-data frozen Pydantic model without an enum import here. + """ + + column: Union["ColumnKey", "ColumnSqlKey"] + granularity: str + + @property + def phase(self) -> Phase: + return Phase.ROW + + +def column_leaf(col: Union["ColumnKey", "ColumnSqlKey"]) -> str: + """The leaf column name of a ``TimeTruncKey.column`` regardless of kind. + + ``ColumnKey`` carries ``leaf``; ``ColumnSqlKey`` carries + ``column_name``. Using this helper everywhere a ``TimeTruncKey``'s + column is unwrapped avoids ``leaf`` / ``column_name`` drift. + """ + return getattr(col, "leaf", None) or getattr(col, "column_name") + + +def column_path(col: Union["ColumnKey", "ColumnSqlKey"]) -> Tuple[str, ...]: + """The join path of a ``TimeTruncKey.column`` regardless of kind. + + Both ``ColumnKey`` and ``ColumnSqlKey`` carry ``.path``. + """ + return col.path + + +class StarKey(_FrozenKey): + """Sentinel source for ``*:count`` aggregations. + + ``path`` is empty for the local star (``*:count`` over the host) and + non-empty for a cross-model star (``customers.*:count`` → + ``path=("customers",)``), mirroring ``ColumnKey.path`` so the + cross-model planner can route a star aggregate through the join graph + (P3). Two stars with the same path intern; the default empty path + keeps the local-star identity bit-identical to before. + """ + + path: Tuple[str, ...] = () + + @property + def phase(self) -> Phase: + return Phase.ROW + + +class LiteralKey(_FrozenKey): + """Identity for a literal value inside an expression tree. + + Used wherever an ``ArithmeticKey``, ``TransformKey``, or other + composite key needs a literal operand (``revenue:sum + 1`` — the + ``1`` is a ``LiteralKey``). Carries phase ROW so it doesn't + artificially elevate the phase of expressions it appears in. + + Scalar normalization (int → Decimal, float → Decimal via str) + happens at the call site via ``normalize_scalar`` so equality + is type-stable (``LiteralKey(Decimal(1))`` and + ``LiteralKey(True)`` are distinct). + """ + + value: Union[Decimal, str, bool, None] = None + + @property + def phase(self) -> Phase: + return Phase.ROW + + def __hash__(self) -> int: + return hash(("LiteralKey", _typed_leaf(self.value))) + + def __eq__(self, other: object) -> bool: + if not isinstance(other, LiteralKey): + return NotImplemented + return _typed_leaf(self.value) == _typed_leaf(other.value) + + +class SqlExprKey(_FrozenKey): + """Identity for a Mode-A SQL fragment. + + Currently used as ``AggregateKey.column_filter_key`` so a + ``Column.filter`` wired in at aggregation time becomes part of the + aggregate's structural identity. Two aggregates over the same column + differ when their attached ``Column.filter`` differs; same-filter + ones intern. + + ``canonical_sql`` is a sqlglot-normalized string (the binder is + responsible for normalization — the key trusts the form it receives). + + DEV-1503 — ``referenced_join_paths`` is the typed SET (semantically; + stored as an ordered tuple for hashability) of non-anchor join-path + prefixes the filter touches after derived-ref expansion. Computed + once at bind time via + ``slayer.engine.column_filter_paths.compute_column_filter_join_paths``; + the planner reads it to decide whether a filtered-local measure must + isolate (the DEV-1503 trigger predicate). ``()`` for same-model + filters; non-empty for cross-model column filters. The field + participates in structural identity — two filters with the same + canonical SQL but different referenced paths would be a bug, so + folding it into the key catches that invariant violation by + comparison. + + The ``before``-validator canonicalises the input to a sorted, + de-duplicated tuple of tuples — so callers can pass any iterable + (list, set, generator) and order doesn't affect identity (otherwise + two semantically-equal SqlExprKeys built with paths in different + order would intern as different keys; CodeRabbit nitpick). + """ + + canonical_sql: str + referenced_join_paths: Tuple[Tuple[str, ...], ...] = () + + @field_validator("referenced_join_paths", mode="before") + @classmethod + def _canonicalize_referenced_join_paths(cls, v): + if not v: + return () + return tuple(sorted({tuple(p) for p in v})) + + @property + def phase(self) -> Phase: + return Phase.ROW + + def __hash__(self) -> int: + return hash(("SqlExprKey", self.canonical_sql, self.referenced_join_paths)) + + def __eq__(self, other: object) -> bool: + if not isinstance(other, SqlExprKey): + return NotImplemented + return ( + self.canonical_sql == other.canonical_sql + and self.referenced_join_paths == other.referenced_join_paths + ) + + +# --------------------------------------------------------------------------- +# Aggregate / Transform / Arithmetic / ScalarCall +# --------------------------------------------------------------------------- + + +_AggregateSource = Union[ColumnKey, ColumnSqlKey, StarKey] +# Positional and keyword arg values accept the same union — both +# `last(created_at)` (positional ColumnKey time arg) and +# `weighted_avg(weight=qty)` (kwarg ColumnKey) bind to identifier columns +# via `_bind_agg_arg`. Reusing one alias for both keeps the surface tight. +_AggregateArgValue = Union[ColumnKey, ColumnSqlKey, Decimal, str, bool, None] +_AggregateKwargValue = _AggregateArgValue + + +def _sort_kwargs_tuple(v): + """Validator helper: canonicalize a kwargs tuple to sorted order by key.""" + if v is None: + return () + return tuple(sorted(v, key=lambda kv: kv[0])) + + +class AggregateKey(_FrozenKey): + """Identity for an aggregation slot (P3). + + Local and cross-model aggregates share this shape: ``source.path`` + is empty for local, non-empty for joined. The render strategy + (base CTE vs cross-model CTE) is decided downstream by the planner. + + ``args`` and ``kwargs`` carry the aggregation's parameters. Numeric + scalars must already be normalized to ``Decimal`` (use + ``normalize_scalar``). Identifier kwargs (``weighted_avg(weight=quantity)``) + arrive as ``ColumnKey`` / ``ColumnSqlKey``. ``kwargs`` is canonicalized + to sorted-by-key order by the validator so input order does not affect + identity. + + ``column_filter_key`` is the ``Column.filter`` attached to the + aggregated column, if any — pulled into the structural key so two + aggregates with different attached filters do not collide. + """ + + source: _AggregateSource + agg: str + args: Tuple[_AggregateArgValue, ...] = () + kwargs: Tuple[Tuple[str, _AggregateKwargValue], ...] = () + column_filter_key: Optional[SqlExprKey] = None + + @field_validator("kwargs", mode="before") + @classmethod + def _canonicalize_kwargs(cls, v): + return _sort_kwargs_tuple(v) + + @property + def phase(self) -> Phase: + return Phase.AGGREGATE + + def __hash__(self) -> int: + return hash(( + "AggregateKey", + self.source, + self.agg, + _typed_args(self.args), + _typed_kwargs(self.kwargs), + self.column_filter_key, + )) + + def __eq__(self, other: object) -> bool: + if not isinstance(other, AggregateKey): + return NotImplemented + return ( + self.source == other.source + and self.agg == other.agg + and _typed_args(self.args) == _typed_args(other.args) + and _typed_kwargs(self.kwargs) == _typed_kwargs(other.kwargs) + and self.column_filter_key == other.column_filter_key + ) + + +def _reroot_path_ref(ref, *, target_path: Tuple[str, ...]): + """Re-anchor a single embedded reference from the host coordinate system + into the target's local scope (DEV-1707). + + Prefix-strip with residual: if ``ref`` carries a join ``path`` that starts + with ``target_path``, drop that prefix and keep the residual hops + (``("customers", "regions")`` under target ``("customers",)`` → + ``("regions",)``; an exact match → ``()``). A ``path`` that does NOT start + with ``target_path`` — or a value with no ``path`` at all (a scalar + ``Decimal`` / ``str`` / ``bool`` / ``None``) — is returned unchanged. The + strip applies uniformly to ``ColumnKey``, ``ColumnSqlKey``, and + ``StarKey``; the non-``path`` fields (``leaf`` / ``model`` / + ``column_name``) ride along untouched via ``model_copy``. + """ + path = getattr(ref, "path", None) + if path is None: + return ref + path = tuple(path) + if path[: len(target_path)] != tuple(target_path): + return ref + residual = path[len(target_path):] + if residual == path: + return ref + return ref.model_copy(update={"path": residual}) + + +def reroot_aggregate_key( + key: "AggregateKey", *, target_path: Tuple[str, ...], +) -> "AggregateKey": + """Re-anchor EVERY embedded reference of a cross-model ``AggregateKey`` + from the host's coordinate system into its target's local scope + (DEV-1707 / DEV-1703 Stage 3). + + When a cross-model aggregate (``customers.revenue:sum``, + ``customers.amount:last(customers.signup_at)``) is rendered inside its + target-rooted CTE, its ``source``, positional ``args``, keyword ``kwargs`` + values, and — invariantly — its ``column_filter_key`` must all be + expressed relative to the target rather than the query root. This is the + single, symmetric replacement for the per-field strip logic that used to + live in ``slayer/engine/cross_model_planner.py`` (``_local_agg_formula`` / + ``_reroot_col_kwarg``) and ``slayer/sql/generator.py`` (the inline + ``_reroot_kwarg`` / ``local_args`` / ``_reroot_having`` blocks) with two + divergent semantics. + + ``target_path`` is the join path of the aggregate's source (the hops from + the query root to the target model). Semantics — prefix-strip with + residual, applied uniformly (see ``_reroot_path_ref``): + + * ``source``, each positional arg, and each kwarg VALUE whose ``path`` + starts with ``target_path`` drops that prefix (exact match → local); + * a ref whose ``path`` does not start with ``target_path`` is left + unchanged — the function is TOTAL and never raises; a genuinely + mis-pathed ref surfaces at the downstream binder / kwarg-path validator + exactly as before; + * scalar args / kwargs pass through untouched; kwarg NAMES are preserved + (and the ``AggregateKey`` validator keeps them canonically sorted); + * ``target_path == ()`` is the identity (the filtered-local case, where + the source is already host-local — the empty prefix strips zero hops). + + ``column_filter_key`` is copied UNCHANGED. Its ``canonical_sql`` and + ``referenced_join_paths`` are anchored at the OWNING MODEL of the source + column (stamped by ``slayer.engine.binding._resolve_column_filter_key`` + via ``compute_column_filter_join_paths`` with ``anchor_model`` = that + owning model). Rerooting only changes how that owner is REACHED from the + query root; it never moves the owner, so the filter's owner-relative SQL + and paths are invariant under reroot. After rerooting a filtered + cross-model aggregate the source reads local (``path == ()``) while + ``referenced_join_paths`` stays non-empty — precisely the DEV-1503 + filtered-local isolation trigger shape. + """ + target_path = tuple(target_path) + if not target_path: + return key + # Rebuild from the existing key so fields reroot does NOT own (``agg``, + # ``column_filter_key``, and any field added to ``AggregateKey`` later) + # ride through automatically rather than being silently dropped. Only + # ``source`` / ``args`` / ``kwargs`` carry rerootable paths. ``model_copy`` + # skips the ``_canonicalize_kwargs`` validator, which is a no-op here: + # reroot preserves kwarg names and order, so the input's already-canonical + # sort is unchanged (pinned by + # ``test_kwargs_canonical_sort_preserved_after_reroot``). + return key.model_copy(update={ + "source": _reroot_path_ref(key.source, target_path=target_path), + "args": tuple( + _reroot_path_ref(a, target_path=target_path) for a in key.args + ), + "kwargs": tuple( + (k, _reroot_path_ref(v, target_path=target_path)) + for k, v in key.kwargs + ), + }) + + +class TransformKey(_FrozenKey): + """Identity for a transform slot (window / temporal operator over a value). + + The ``input`` is the value the transform operates on — typically an + aggregate or another transform, occasionally a row-level column. + + ``partition_keys`` is a frozenset (order-independent); ``time_key`` is + addressed separately as the sort dimension for time-ordered transforms. + """ + + op: str + input: "ValueKey" + args: Tuple[Scalar, ...] = () + kwargs: Tuple[Tuple[str, Scalar], ...] = () + partition_keys: frozenset["ValueKey"] = frozenset() + time_key: Optional["ValueKey"] = None + + @field_validator("kwargs", mode="before") + @classmethod + def _canonicalize_kwargs(cls, v): + return _sort_kwargs_tuple(v) + + @property + def phase(self) -> Phase: + return Phase.POST + + def __hash__(self) -> int: + return hash(( + "TransformKey", + self.op, + self.input, + _typed_args(self.args), + _typed_kwargs(self.kwargs), + self.partition_keys, + self.time_key, + )) + + def __eq__(self, other: object) -> bool: + if not isinstance(other, TransformKey): + return NotImplemented + return ( + self.op == other.op + and self.input == other.input + and _typed_args(self.args) == _typed_args(other.args) + and _typed_kwargs(self.kwargs) == _typed_kwargs(other.kwargs) + and self.partition_keys == other.partition_keys + and self.time_key == other.time_key + ) + + +class ArithmeticKey(_FrozenKey): + """Identity for an arithmetic / comparison / boolean expression. + + ``op`` is the operator symbol (``+``, ``-``, ``*``, ``/``, ``<``, + ``<=``, ``and``, ``or``, …). Operand order matters — subtraction + and division are non-commutative, comparisons have a fixed LHS/RHS, + and even commutative ops keep their textual order for deterministic + SQL emission. + + Phase is the maximum of operand phases (P8). + """ + + op: str + operands: Tuple["ValueKey", ...] + + @property + def phase(self) -> Phase: + return max((o.phase for o in self.operands), default=Phase.ROW) + + +_ScalarCallArg = Union["ValueKey", Decimal, str, bool, None] + + +def _arg_phase(arg) -> Optional[Phase]: + """Return ``arg.phase`` for ValueKey args, ``None`` for pure scalars.""" + return getattr(arg, "phase", None) + + +class ScalarCallKey(_FrozenKey): + """Identity for a closed-allowlist scalar function call (C12). + + ``name`` must be a member of ``SCALAR_FUNCTIONS``. The key constructor + does NOT validate this — the binder rejects unknown names with + ``UnknownFunctionError``. Keeping validation out of the key keeps + identity construction cheap on the hot path. + + Phase is the maximum of arg phases over the args that carry a phase + (i.e., ``ValueKey``s); pure-scalar args contribute the ROW floor. + """ + + name: str + args: Tuple[_ScalarCallArg, ...] = () + + @property + def phase(self) -> Phase: + phases = [p for a in self.args if (p := _arg_phase(a)) is not None] + return max(phases) if phases else Phase.ROW + + def __hash__(self) -> int: + return hash(("ScalarCallKey", self.name, _typed_args(self.args))) + + def __eq__(self, other: object) -> bool: + if not isinstance(other, ScalarCallKey): + return NotImplemented + return ( + self.name == other.name + and _typed_args(self.args) == _typed_args(other.args) + ) + + +# --------------------------------------------------------------------------- +# BetweenKey — DEV-1450 stage 7b.9 +# --------------------------------------------------------------------------- + + +class BetweenKey(_FrozenKey): + """Typed identity for a ``col BETWEEN low AND high`` predicate. + + Closed-form Mode-A SQL constructs (``BETWEEN``) and equivalent + Mode-B compound forms (``col >= low and col <= high``) render to + different SQL text. The planner uses ``BetweenKey`` to mark the + spots where ``BETWEEN`` is the right legacy-parity rendering — today + only ``TimeDimension.date_range`` produces them. User-written DSL + filters never produce ``BetweenKey``: the syntax parser doesn't + have a ``between`` construct, and a user-written ``col >= a and + col <= b`` stays as ``ArithmeticKey(and, [GE, LE])`` so its parity + with the legacy generator (which keeps the AND form verbatim) is + preserved. + + Phase is always ROW — ``BetweenKey`` predicates filter row-level + columns. The renderer emits ``exp.Between``. + """ + + column: "ValueKey" + low: "ValueKey" + high: "ValueKey" + + @property + def phase(self) -> Phase: + return Phase.ROW + + +# --------------------------------------------------------------------------- +# InKey — DEV-1475 +# --------------------------------------------------------------------------- + + +class InKey(_FrozenKey): + """Typed identity for a ``col IN (lit, lit, …)`` / ``NOT IN`` predicate. + + Modelled on ``BetweenKey``: a closed-form SQL predicate with a column + LHS and a fixed tuple of literal-valued RHS operands. Two ``InKey``s + with the same column and the same set of values (in the same order) + intern; ``negated`` flips IN vs NOT IN without doubling the class + count. + + ``values`` is a tuple of ``LiteralKey`` (not bare scalars) so equality + rides through ``LiteralKey``'s type-stable ``_typed_leaf`` machinery + — ``InKey(values=(LiteralKey(True),))`` does not collide with + ``InKey(values=(LiteralKey(Decimal(1)),))``. + + Phase is always ROW; the renderer emits ``exp.In`` (wrapped in + ``exp.Not`` when ``negated``). + """ + + column: "ValueKey" + values: Tuple[LiteralKey, ...] + negated: bool = False + + @field_validator("values") + @classmethod + def _reject_empty_values( + cls, v: Tuple[LiteralKey, ...], + ) -> Tuple[LiteralKey, ...]: + # Defense in depth (Codex review): the parser's ``ast.Compare`` + # branch already rejects empty RHS, but direct construction can + # bypass it and reach the SQL generator, which would emit + # ``col IN ()`` — invalid in every supported dialect. + if not v: + raise ValueError( + "InKey requires a non-empty ``values`` tuple; ``col IN " + "()`` is invalid SQL across every supported dialect.", + ) + return v + + @property + def phase(self) -> Phase: + return Phase.ROW + + +# --------------------------------------------------------------------------- +# Union alias + rebuild for forward refs +# --------------------------------------------------------------------------- + + +ValueKey = Union[ + ColumnKey, + ColumnSqlKey, + TimeTruncKey, + StarKey, + LiteralKey, + AggregateKey, + TransformKey, + ArithmeticKey, + ScalarCallKey, + BetweenKey, + InKey, +] + + +# Resolve the recursive forward references on the keys that take ValueKey. +TransformKey.model_rebuild() +ArithmeticKey.model_rebuild() +ScalarCallKey.model_rebuild() +BetweenKey.model_rebuild() +InKey.model_rebuild() +# TimeTruncKey.column is a Union[ColumnKey, ColumnSqlKey] (DEV-1450 #4a). +TimeTruncKey.model_rebuild() diff --git a/slayer/core/models.py b/slayer/core/models.py index 293638d6..1da87281 100644 --- a/slayer/core/models.py +++ b/slayer/core/models.py @@ -24,9 +24,6 @@ logger = logging.getLogger(__name__) -_MULTIDOT_COLUMN_RE = re.compile(r'\b([a-zA-Z_]\w*(?:\.[a-zA-Z_]\w*){2,})\b') -_STRING_LITERAL_RE = re.compile(r"'[^']*'") - # Host-field normalization for the generic connection URL. ``URL.create`` # wants a raw host (IPv6 without brackets) plus a separate port, but the # pre-fix string branch tolerated the port being embedded in the host @@ -129,41 +126,6 @@ def _validate_column_name(name: str, context: str) -> str: return name -def _convert_multidot_ref(match: re.Match) -> str: - """Convert a multi-dot reference like ``a.b.c`` to ``a__b.c``.""" - ref = match.group(1) - parts = ref.split(".") - return "__".join(parts[:-1]) + "." + parts[-1] - - -def _fix_multidot_sql(sql: str, context: str) -> str: - """Auto-convert multi-dot references in a SQL snippet to __ alias syntax. - - Single-dot references (``table.column``) are left as-is. - Multi-dot references (``a.b.c``) are converted to ``a__b.c`` with a warning. - String literals are skipped. - """ - # Build a map of string-literal spans to skip - literal_spans = [m.span() for m in _STRING_LITERAL_RE.finditer(sql)] - - def _in_literal(start: int) -> bool: - return any(s <= start < e for s, e in literal_spans) - - result = sql - for match in list(_MULTIDOT_COLUMN_RE.finditer(sql)): - if _in_literal(match.start()): - continue - ref = match.group(1) - fixed = _convert_multidot_ref(match) - logger.warning( - "%s: auto-converting multi-dot reference '%s' to '%s'. " - "Use '__' for join paths in SQL snippets (e.g., '%s').", - context, ref, fixed, fixed, - ) - result = result.replace(ref, fixed) - return result - - class Column(BaseModel): """A row-level column on a model. @@ -216,18 +178,10 @@ def _coerce_legacy_type(cls, data: Any) -> Any: def _validate_name(cls, v: str) -> str: return _validate_column_name(v, "Column") - @field_validator("sql") - @classmethod - def _fix_multidot_sql(cls, v: str | None) -> str | None: - if v is not None: - v = _fix_multidot_sql(v, context="Column sql") - return v - @field_validator("filter") @classmethod - def _fix_multidot_filter(cls, v: str | None) -> str | None: + def _validate_filter_predicate(cls, v: Optional[str]) -> Optional[str]: if v is not None: - v = _fix_multidot_sql(v, context="Column filter") # DEV-1369: Column.filter is SQL-mode — validate at construction # time so DSL constructs (aggregation colon, transform calls) are # caught early. Result is discarded; we only care about the @@ -311,7 +265,7 @@ def _reject_raw_window_function(cls, v: str) -> str: # DEV-1369: ModelMeasure formulas may legitimately contain ``__`` — # virtual-model columns produced by ``_query_as_model`` flatten join # paths into names like ``kpis__total_amount_sum``, which downstream - # stages reference directly. Strict resolution at enrichment time + # stages reference directly. Strict resolution at binding time # catches typos like ``customers__region`` that don't resolve to any # Column on the model. @@ -418,7 +372,7 @@ class SourceModelOrigin(BaseModel): ``agg_column_names`` (Codex review on PR #137 round 9) records the flat names of columns on this stage that came from ``cross_model_measures`` or aggregated ``measures`` in the inner - query's enrichment — i.e. the columns the cross-stage intercept + query's binding — i.e. the columns the cross-stage intercept is safe to re-aggregate. Without this provenance, a user-defined dimension whose name happens to look like an aggregation canonical (e.g. ``customers__revenue_sum``) would be silently re-summed by @@ -537,23 +491,18 @@ def _require_data_source_unless_query_backed(self) -> "SlayerModel": @field_validator("filters") @classmethod - def _fix_multidot_filters(cls, v: list[str]) -> list[str]: - """Auto-convert multi-dot column references in model filters and - validate each entry as a SQL-mode predicate (DEV-1369). + def _validate_filter_predicates(cls, v: list[str]) -> list[str]: + """Validate each model filter as a SQL-mode predicate (DEV-1369). Model filters are SQL snippets: joined column references use the - ``__`` alias syntax (``customers__regions.name``), not the - multi-dot query syntax (``customers.regions.name``). Single-dot - references like ``customers.name`` (table.column) are left as-is. - - After the multi-dot rewrite each entry is parsed with - :func:`parse_sql_predicate` so DSL constructs (aggregation colon, - transform calls, raw OVER) are caught at construction time. + ``__`` alias syntax (``customers__regions.name``). Each entry is + parsed with :func:`parse_sql_predicate` so DSL constructs + (aggregation colon, transform calls, raw OVER) are caught at + construction time. """ - rewritten = [_fix_multidot_sql(f, context="Model filter") for f in v] - for f in rewritten: + for f in v: parse_sql_predicate(f) - return rewritten + return v @model_validator(mode="after") def _validate_column_measure_disjoint(self) -> "SlayerModel": diff --git a/slayer/core/query.py b/slayer/core/query.py index 2ddc4eba..19f3b456 100644 --- a/slayer/core/query.py +++ b/slayer/core/query.py @@ -1,8 +1,9 @@ """Query models for SLayer. SlayerQuery is the user-facing query object — minimal, just enough to express intent. -It is later converted into EnrichedQuery (see slayer/engine/enriched.py) which carries -fully resolved SQL expressions, model metadata, and is ready for SQL generation. +It is later planned into a ``PlannedQuery`` (see slayer/engine/planned.py), which +carries typed value keys interned into slots with their resolved expressions, join +paths and phases, and is ready for SQL generation. """ from __future__ import annotations @@ -38,9 +39,9 @@ def _validate_query_filter_string(formula: str) -> None: syntax. Raw SQL function calls (``json_extract``, ``coalesce``, …) and - unknown bare names are rejected at enrichment time by + unknown bare names are rejected at binding time by :func:`slayer.core.formula.parse_filter` and the strict-resolution - pass in :func:`slayer.engine.enrichment.resolve_filter_columns`. + pass in :func:`slayer.engine.binding.bind_expr`. """ if has_window_function(formula): raise ValueError(f"Filter '{formula}' {WINDOW_IN_FILTER_ERROR}") @@ -722,6 +723,47 @@ def _coerce_column_ref(v: Any) -> Any: _FUNCSTYLE_CALL_PATTERN = re.compile(r"^\w+\([^()]*\)$") +# DEV-1733: sentinel ``ColumnRef.name`` values meaning "this ORDER BY item is +# an EXPRESSION — resolve it from ``raw_formula``, not as a column reference". +# ``_funcstyle_pending`` marks an unrewritten function-style call (a custom +# aggregation or a transform); ``_expr_pending`` marks any other formula shape +# that is not expressible as a ``ColumnRef`` (composite arithmetic, a scalar +# call over an aggregation, arithmetic over a transform). Consumers MUST also +# require a non-empty ``raw_formula`` before treating a name as a sentinel, so +# a model that genuinely has a column of that name still resolves normally. +_FUNCSTYLE_PENDING = "_funcstyle_pending" +_EXPR_PENDING = "_expr_pending" +ORDER_PLACEHOLDER_NAMES = frozenset({_FUNCSTYLE_PENDING, _EXPR_PENDING}) + + +def _order_formula_candidate(v: str) -> str | None: + """The func-style-rewritten form of ``v`` when it carries a measure + expression (a colon aggregation, or a function-style call), else ``None``. + + Single source of truth for "this ORDER BY string is a formula, not a column + reference". Shared by :meth:`OrderItem._capture_raw_formula` (which + preserves the original text) and :func:`_coerce_order_column` (which emits + the placeholder ``ColumnRef``) so the two cannot drift — if only one of + them recognised a shape, the item would either lose its formula or bind a + meaningless placeholder name. + """ + from slayer.core.formula import _rewrite_funcstyle_aggregations + + rewritten = _rewrite_funcstyle_aggregations(v) + if ":" in rewritten or _FUNCSTYLE_CALL_PATTERN.match(rewritten): + return rewritten + return None + + +def _is_valid_column_ref_name(name: str) -> bool: + """Whether ``name`` parses as a ``ColumnRef`` (bare leaf or dotted path).""" + try: + ColumnRef.model_validate({"name": name}) + except Exception: + return False + return True + + def _coerce_order_column(v: Any) -> Any: """Coerce ORDER BY column, normalizing aggregation syntax. @@ -735,16 +777,28 @@ def _coerce_order_column(v: Any) -> Any: - "sum(revenue)" → "revenue_sum" - "revenue:last(ordered_at)" → "revenue_last" - "rolling_avg(revenue)" → placeholder, raw_formula carries the call so - enrichment can resolve it via ``extra_agg_names``. + binding can resolve it via ``extra_agg_names``. + - "revenue:sum / cnt:sum" → placeholder (DEV-1733), raw_formula carries the + composite so the planner binds it as an expression. + + DEV-1733: a composite that is NOT a formula candidate — ``"rev / cnt"``, + arithmetic over declared measure ALIASES — falls through to normal + ``ColumnRef`` validation and keeps its original error. Alias references + inside expressions are unsupported everywhere in SLayer, so failing fast at + construction is better than a deep binder error. """ if isinstance(v, str): from slayer.core.formula import _rewrite_funcstyle_aggregations - rewritten = _rewrite_funcstyle_aggregations(v) + candidate = _order_formula_candidate(v) + rewritten = ( + candidate if candidate is not None + else _rewrite_funcstyle_aggregations(v) + ) if _FUNCSTYLE_CALL_PATTERN.match(rewritten): # Unrewritten function-style call (custom aggregation). Enrichment # parses raw_formula with custom_agg_names and overwrites # column.name with the canonical alias, so a placeholder is fine. - return {"name": "_funcstyle_pending"} + return {"name": _FUNCSTYLE_PENDING} if ":" in rewritten: base, agg = rewritten.rsplit(":", 1) agg_name = agg.split("(", 1)[0] # strip arglist @@ -752,6 +806,11 @@ def _coerce_order_column(v: Any) -> Any: rewritten = f"_{agg_name}" else: rewritten = f"{base}_{agg_name}" + if candidate is not None and not _is_valid_column_ref_name(rewritten): + # A formula that does not canonicalise to a column reference — + # composite arithmetic, a scalar call over an aggregation, or + # arithmetic over a transform. ``raw_formula`` carries the original. + return {"name": _EXPR_PENDING} return {"name": rewritten} return v @@ -794,14 +853,17 @@ class OrderItem(BaseModel): @model_validator(mode="before") @classmethod def _capture_raw_formula(cls, data: Any) -> Any: - """Capture the raw column formula before coercion normalizes it.""" + """Capture the raw column formula before coercion normalizes it. + + Shares :func:`_order_formula_candidate` with ``_coerce_order_column`` + so a shape can never be recognised by one and not the other. + """ if isinstance(data, dict): col = data.get("column") if isinstance(col, str): - from slayer.core.formula import _rewrite_funcstyle_aggregations - rewritten = _rewrite_funcstyle_aggregations(col) - if ":" in rewritten or _FUNCSTYLE_CALL_PATTERN.match(rewritten): - data = {**data, "raw_formula": rewritten} + candidate = _order_formula_candidate(col) + if candidate is not None: + data = {**data, "raw_formula": candidate} return data @field_validator("direction") @@ -946,7 +1008,7 @@ class SlayerQuery(BaseModel): """User-facing query object. Specifies what data to retrieve from a model. This is intentionally minimal — just names and references, no SQL. - The query engine enriches it into an EnrichedQuery for execution. + The query engine plans it into a ``PlannedQuery`` for execution. Use ``measures`` for computed/aggregated values and ``filters`` for conditions:: @@ -1005,13 +1067,13 @@ def _validate_dsl_user_input(self) -> "SlayerQuery": Filter strings are pre-parsed in DSL mode so raw ``OVER (...)`` is caught at construction time with an actionable error message. Bare-name strict resolution and raw-SQL-function rejection happen - at enrichment, where the parser has full custom-aggregation and + at binding, where the parser has full custom-aggregation and named-measure context. Note: ``__`` is **not** rejected here. Virtual-model columns produced by ``_query_as_model`` flatten join paths into single identifiers like ``kpis__total_amount_sum``, which downstream - stages reference directly. Strict resolution at enrichment + stages reference directly. Strict resolution at binding catches typos that don't resolve to any column / measure. """ if self.filters: @@ -1030,7 +1092,7 @@ def _validate_distinct_dimension_values(self) -> None: * Both ``dimensions`` and ``time_dimensions`` empty — there are no projected columns to ``SELECT``. - Deep filter / order measure-reference checks happen at enrichment, + Deep filter / order measure-reference checks happen at binding, where named measures, custom aggregations, and post-substitution text are all available. Detecting them here would either reject valid ``{var}`` filters before substitution or miss model-defined diff --git a/slayer/core/refs.py b/slayer/core/refs.py index 9eeac803..1ac659a0 100644 --- a/slayer/core/refs.py +++ b/slayer/core/refs.py @@ -2,17 +2,24 @@ DEV-1369 consolidates the identifier-shape regexes and aggregation-suffix parsing that previously lived in four different files (``formula.py``, -``dbt/converter.py``, ``engine/enrichment.py``, ``memories/resolver.py``). +``dbt/converter.py``, ``engine/binding.py``, ``memories/resolver.py``). Keeping a single source of truth prevents the four copies from drifting out of sync. -This module is intentionally side-effect-free and depends on nothing from -``slayer.core.models`` / ``slayer.core.query`` so it can be imported from -those modules' validators without circular import risk. +This module is intentionally side-effect-free and depends only on +``slayer.core.keys`` (for the ``ColumnKey`` shape ``agg_kwarg_canonical_str`` +canonicalises). It does NOT import ``slayer.core.models`` / +``slayer.core.query`` so it can be imported from those modules' +validators without circular import risk. ``slayer.core.keys`` is itself +free of ``slayer`` imports. """ from __future__ import annotations import re +from decimal import Decimal +from typing import Any + +from slayer.core.keys import ColumnKey, ColumnSqlKey # --------------------------------------------------------------------------- # Identifier shapes @@ -84,6 +91,104 @@ def agg_signature_suffix( return "_" + "_".join(parts) if parts else "" +def _decimal_to_plain_str(value: Decimal) -> str: + """Return ``value`` as a plain-decimal string with no scientific notation. + + ``str(Decimal("1E-7"))`` yields ``"1E-7"``, which the generator's + ``_SAFE_AGG_PARAM_RE`` SQL-injection allowlist rejects. ``f"{x:f}"`` + forces plain notation but pads short fractional values with extra + zeros (``f"{Decimal('0.5'):f}"`` -> ``"0.5"`` is fine, but + ``f"{0.5:f}"`` on a float yields ``"0.500000"``). To preserve + short forms while expanding exponents, normalize via the Decimal + layer's own ``normalize()`` + a fix-up for the + ``Decimal('-0E+1')`` "-0" exponent quirk. + """ + # Trip the exponent down so ``Decimal("1E-7")`` becomes + # ``Decimal("0.0000001")`` (and ``Decimal("1.0E+3")`` becomes + # ``Decimal("1000")``). For sign normalization use the standard + # ``f"{value:f}"`` then trim trailing zeros after a decimal point. + s = f"{value:f}" + if "." in s: + s = s.rstrip("0").rstrip(".") + return s or "0" + + +def agg_kwarg_canonical_str(value: Any) -> str: + """Canonicalize an AggregateKey kwarg / arg value to SQL-string form. + + DEV-1450 stage 7b.13: ``EnrichedMeasure.agg_kwargs`` is typed as + ``Dict[str, str]`` and every value flows through + ``_validate_agg_param_value`` (``slayer/sql/generator.py:172``) which + accepts only identifiers, qualified names, or numeric literals. + Sites that build the synth ``EnrichedMeasure`` from a typed + ``AggregateKey`` -- AND the two canonical-alias renderers that + previously called ``str(v)`` directly (``slayer/sql/generator.py:3753`` + and ``slayer/engine/cross_model_planner.py:286``) -- route every + kwarg value through this helper instead, so a ``ColumnKey`` never + surfaces as Pydantic-repr noise. + + Conversion rules: + + * ``bool`` / ``None`` -> ``TypeError`` (legacy never accepted these; + ``AggregateKey``'s structural-key normalisation at + ``slayer/core/keys.py:139-142`` keeps them distinct from numerics + precisely so they fail loudly here). + * ``Decimal`` -> ``str(value)`` (Decimal's ``__str__`` matches + ``_SAFE_AGG_PARAM_RE`` for the planner's normalised forms; ``0.5`` + / ``0.95`` / ``100``). + * ``int`` / ``float`` -> ``str(value)`` (planner-side normalisation + already routes literals through ``Decimal``, but the helper stays + total for direct callers). + * ``str`` -> returned unchanged. Callers writing strings into the + key are responsible for safety; downstream validation catches + malformed input at the generator boundary. + * ``ColumnKey(path=(), leaf=L)`` -> ``L``. + * ``ColumnKey(path=P, leaf=L)`` -> ``".".join(P) + "." + L``. + * ``ColumnSqlKey`` (a derived-column arg/kwarg, e.g. + ``corr(other=derived_col)`` — DEV-1450 #4a/#4b) -> the same + ``[path.]column_name`` form so a parametric agg over a derived + column canonicalizes instead of raising. + + Anything else raises ``TypeError`` -- the AggregateKey key shape is + closed over these branches. + """ + if isinstance(value, bool): + # bool is-a int, must check first. + raise TypeError( + f"AggregateKey kwarg cannot be bool: {value!r}", + ) + if value is None: + raise TypeError("AggregateKey kwarg cannot be None") + if isinstance(value, Decimal): + # Route through ``_decimal_to_plain_str`` to force plain + # decimal notation: ``str(Decimal("1E-7"))`` yields ``"1E-7"``, + # which the generator's ``_SAFE_AGG_PARAM_RE`` rejects (no + # scientific notation in the SQL-injection allowlist). + return _decimal_to_plain_str(value) + if isinstance(value, int): + return str(value) + if isinstance(value, float): + # Route floats through Decimal(str(float)) so the + # human-readable decimal text is preserved (matches the + # planner's ``normalize_scalar`` recipe at + # ``slayer/core/keys.py:102``). + return _decimal_to_plain_str(Decimal(str(value))) + if isinstance(value, str): + return value + if isinstance(value, ColumnKey): + if value.path: + return ".".join(value.path) + "." + value.leaf + return value.leaf + if isinstance(value, ColumnSqlKey): + if value.path: + return ".".join(value.path) + "." + value.column_name + return value.column_name + raise TypeError( + f"AggregateKey kwarg value of type {type(value).__name__!r} " + f"is not supported: {value!r}", + ) + + def canonical_agg_name( measure_name: str, aggregation_name: str, diff --git a/slayer/core/scope.py b/slayer/core/scope.py new file mode 100644 index 00000000..cc25ad07 --- /dev/null +++ b/slayer/core/scope.py @@ -0,0 +1,117 @@ +"""Stage 2 (DEV-1450) — typed scope and stage-schema for the new pipeline. + +Two scope kinds, never confused (P5): + +- ``ModelScope``: joins exist; dotted refs walk the join graph rooted at + ``source_model``. ``__`` in a Mode-B ref is an error unless it exact- + matches a column literally named that way (legacy persisted query-backed + columns). +- ``StageSchema``: flat namespace; dots are not join syntax; + ``__``-bearing identifiers are flat names. + +``StageColumn`` is the typed projection element (P6): explicit ``name`` +(downstream bind name), ``sql_alias`` (emitted SQL identifier), +``public_alias`` (result-key piece), plus the per-column metadata that +downstream stages need. + +Per I2 of the DEV-1450 execution plan, ``ModelScope.source_model`` is +``Optional`` from day one so a future anchor-less mode is a type-additive +change. DEV-1450's binder will assert ``source_model is not None`` at +use sites — the type-level optionality is the extension point. +""" + +from __future__ import annotations + +from typing import Any, Dict, List, Optional + +from pydantic import BaseModel, ConfigDict + +from slayer.core.enums import DataType +from slayer.core.format import NumberFormat +from slayer.core.models import SlayerModel + + +class StageColumn(BaseModel): + """Typed projection element for one stage (P6). + + ``name`` is the downstream bind name — flat (e.g. + ``robot_details__modelseriesval`` or ``rev``). ``sql_alias`` is the + identifier emitted in the stage's SELECT projection (usually equal + to ``name``, but the typed split lets the planner reserve hidden + or alias-bearing forms without coupling them). ``public_alias`` is + the result-key piece returned to the user — set only for non-hidden + columns. + + ``format`` (DEV-1452 Stage B decision #8) is the typed ``NumberFormat`` + inherited from the source ``ModelMeasure`` / ``Column`` or computed + by ``_infer_aggregated_format``. ``description`` propagates the source + column's documentation through the typed plan. + """ + + model_config = ConfigDict(frozen=True) + + name: str + sql_alias: str + public_alias: Optional[str] = None + type: Optional[DataType] = None + label: Optional[str] = None + format: Optional[NumberFormat] = None + hidden: bool = False + description: Optional[str] = None + meta: Optional[Dict[str, Any]] = None + sampled: Optional[str] = None + provenance: Optional[str] = None + + +class StageSchema(BaseModel): + """The typed projection of one query stage (P6). + + Downstream stages bind against this as a flat namespace (P5). They + never re-walk the upstream join graph through a StageSchema — the + only legal refs are entries in ``columns``. + + ``relation_name`` is the SQL identifier used when this stage is + referenced from a downstream stage (CTE name or subquery alias). + ``sql`` is the emitted text of the stage's SELECT — populated by the + planner; left ``None`` until rendering. + """ + + model_config = ConfigDict(arbitrary_types_allowed=True) + + relation_name: str + sql: Optional[str] = None + columns: List[StageColumn] + + def __getitem__(self, name: str) -> StageColumn: + for c in self.columns: + if c.name == name: + return c + raise KeyError( + f"No column named {name!r} in stage {self.relation_name!r}." + ) + + def get(self, name: str) -> Optional[StageColumn]: + for c in self.columns: + if c.name == name: + return c + return None + + def __contains__(self, name: object) -> bool: + return isinstance(name, str) and self.get(name) is not None + + +class ModelScope(BaseModel): + """Scope for binding Mode-B refs against a model with joins (P5). + + Dotted refs walk the join graph rooted at ``source_model``; + ``__``-bearing refs are flat-only and reject unless they exact-match + a column literally named that way on the model. + + I2: ``source_model`` is ``Optional`` from day one. DEV-1450's binder + asserts ``source_model is not None`` at use sites so behavior is + unchanged. A future anchor-less mode uses ``source_model=None`` and + a different binder branch (DatasourceScope-style binding). Keeping + the type optional avoids a breaking change later. + """ + + source_model: Optional[SlayerModel] = None diff --git a/slayer/core/time_bounds.py b/slayer/core/time_bounds.py new file mode 100644 index 00000000..6f12ec43 --- /dev/null +++ b/slayer/core/time_bounds.py @@ -0,0 +1,150 @@ +"""Frame-bound predicate analysis for trailing-window / shifted CTEs (DEV-1732). + +Some CTEs must read rows from OUTSIDE the query's visible time frame: + +* a duration-windowed measure's ``_src`` subquery (``revenue:sum(window='90d')``) + — the trailing window reaches back before the earliest visible bucket, or that + bucket under-counts; +* a ``time_shift`` shifted CTE — the shifted value for the earliest visible + bucket comes from a bucket outside the frame. + +``TimeDimension.date_range`` has always been excluded from those CTEs for that +reason. This module generalises the exclusion from that one carrier to the +*semantic class* it belongs to, so the two spellings of one intent agree: + + A ROW-phase filter conjunct that is a relational bound, with a temporal + literal, on the raw column of one of the query's time dimensions is a FRAME + bound, not a population filter. Frame bounds constrain the visible buckets + only. Everything else is a population filter and is applied unchanged. + +Dependency-free by design (imports only :mod:`slayer.core.keys`) so both the +engine planner and the SQL generator can call it without either importing the +other — the same placement rationale as :mod:`slayer.core.window_duration`. +""" + +from __future__ import annotations + +from typing import AbstractSet, Optional + +from slayer.core.keys import ArithmeticKey, BetweenKey, LiteralKey, ValueKey + +__all__ = [ + "RELATIONAL_OPS", + "is_temporal_literal", + "is_frame_bound", + "strip_frame_bounds", +] + +#: Operators that can express a frame bound. ``==`` / ``!=`` / ``in`` / ``is`` +#: are deliberately absent: an equality on a raw timestamp means "this instant" +#: or "this set", never a range, and stripping one would sum the whole window +#: where a single instant was asked for. +RELATIONAL_OPS = frozenset({"<", "<=", ">", ">="}) + +_AND = "and" + + +def is_temporal_literal(key: object) -> bool: + """Is ``key`` a literal usable as a frame-bound endpoint? + + A **bare** ``LiteralKey`` holding a non-``None`` ``str`` — nothing else. + "Bare" means the operand IS the literal, not an expression tree that merely + contains one: ``ArithmeticKey('+', (LiteralKey(1), LiteralKey(2)))`` does not + qualify. (``isinstance`` is deliberate — ``LiteralKey`` has no subclasses, + and an exact ``type(...) is`` check would be unidiomatic here.) + + Deliberately a whitelist of one shape rather than "contains no column + reference", which would also admit dynamic expressions (a zero-argument + scalar call, say) and quietly treat them as frame bounds. + + Mirrors ``BetweenKey``, whose ``low``/``high`` are + ``LiteralKey(value=normalize_scalar(...))`` and are strings for dates — so + the explicit spelling is recognised on exactly the same terms as the + ``date_range`` one. + + Two cases the strictness matters for: + + * ``created_at < None`` binds to ``LiteralKey(value=None)``. ``col < NULL`` + matches nothing; stripping it would turn an empty result into the full + population. + * ``created_at >= 5`` binds to ``LiteralKey(value=Decimal(5))`` — a + type-invalid comparison, not a frame bound. + + ``bool`` is excluded for free: ``isinstance(True, str)`` is ``False``. + """ + return isinstance(key, LiteralKey) and isinstance(key.value, str) + + +def is_frame_bound(*, key: object, time_columns: AbstractSet[ValueKey]) -> bool: + """Is ``key`` a single frame bound on one of ``time_columns``? + + ``time_columns`` holds the RAW column keys (``ColumnKey`` / ``ColumnSqlKey``) + of the query's non-hidden time dimensions. Matching is by ValueKey identity, + so a derived (``Column.sql``) temporal column is covered without any special + casing, and a same-named column on a joined model cannot collide. + + Both operand orders count — ``'2024-06-01' <= created_at`` says the same + thing as ``created_at >= '2024-06-01'``. + """ + if isinstance(key, BetweenKey): + return ( + key.column in time_columns + and is_temporal_literal(key.low) + and is_temporal_literal(key.high) + ) + if not isinstance(key, ArithmeticKey) or key.op not in RELATIONAL_OPS: + return False + if len(key.operands) != 2: + return False + lhs, rhs = key.operands + if lhs in time_columns: + return is_temporal_literal(rhs) + if rhs in time_columns: + return is_temporal_literal(lhs) + return False + + +def strip_frame_bounds( + *, key: ValueKey, time_columns: AbstractSet[ValueKey], +) -> Optional[ValueKey]: + """Return ``key`` with its top-level frame bounds removed. + + * ``None`` — the whole predicate was a frame bound (or a conjunction of + them); the caller omits the filter from the CTE entirely. + * the **same object** — nothing was stripped; the caller can skip building a + rewrite entry, and the CTE renders the host's predicate verbatim. + * a new key — the residual population predicate. + + A top-level ``and`` is split: each operand is tested independently, frame + bounds are dropped, survivors are rebuilt in their original order (a lone + survivor replaces the conjunction). Nested ``and`` is recursed into. + + ``or`` and ``not`` are never descended into — no sound split exists under a + disjunction or a negation, and keeping the predicate whole preserves the + pre-DEV-1732 result, which is the safe direction to err in. + """ + if not time_columns: + return key + if is_frame_bound(key=key, time_columns=time_columns): + return None + if not (isinstance(key, ArithmeticKey) and key.op == _AND): + return key + + kept: list[ValueKey] = [] + changed = False + for operand in key.operands: + residual = strip_frame_bounds(key=operand, time_columns=time_columns) + if residual is None: + changed = True + continue + if residual is not operand: + changed = True + kept.append(residual) + + if not changed: + return key + if not kept: + return None + if len(kept) == 1: + return kept[0] + return ArithmeticKey(op=_AND, operands=tuple(kept)) diff --git a/slayer/core/warnings.py b/slayer/core/warnings.py new file mode 100644 index 00000000..0f5a1659 --- /dev/null +++ b/slayer/core/warnings.py @@ -0,0 +1,55 @@ +"""Stage 5 (DEV-1450) — slack-normalization warning types. + +The slack-normalization layer (stage 6) rewrites tolerant-but-unambiguous +agent input to canonical form before the typed pipeline sees it, and +emits one ``NormalizationWarning`` payload per rewrite. The payload is +surfaced two ways: + +- Emitted as a Python warning via ``warnings.warn(SlayerNormalizationWarning(payload), ...)`` + so callers using ``warnings.catch_warnings()`` see the rewrite. +- Appended to ``SlayerResponse.warnings: List[NormalizationWarning]`` so + REST/MCP/CLI consumers get the structured payload alongside the result. + +Living in ``slayer.core.warnings`` (not ``slayer.engine.normalization``) +lets memory/storage/REST schemas reference the Pydantic payload without +pulling in engine code. +""" + +from __future__ import annotations + +from typing import Optional + +from pydantic import BaseModel + + +class NormalizationWarning(BaseModel): + """Structured payload describing one slack-normalization rewrite. + + ``rule_id`` identifies the rule that fired (``FUNC_STYLE_AGG``, + ``DOT_PATH_IN_SQL``, ``MISPLACED_MEASURE``). ``location`` is a + human-readable pointer into the query input (e.g. + ``measures[2].formula``). ``rule_doc_url`` is an optional anchor + into ``docs/agent_input_slack.md``. + """ + + rule_id: str + original: str + normalized: str + location: str + rule_doc_url: Optional[str] = None + + +class SlayerNormalizationWarning(UserWarning): + """Carrier ``UserWarning`` for a ``NormalizationWarning`` payload. + + Lets callers route both via ``warnings.catch_warnings(...)`` and + via the structured ``SlayerResponse.warnings`` list — same data, + two surfaces, one source of truth. + """ + + def __init__(self, payload: NormalizationWarning) -> None: + self.payload = payload + super().__init__( + f"[{payload.rule_id}] {payload.original!s} → {payload.normalized!s} " + f"(at {payload.location})" + ) diff --git a/slayer/core/window_duration.py b/slayer/core/window_duration.py new file mode 100644 index 00000000..2e857b55 --- /dev/null +++ b/slayer/core/window_duration.py @@ -0,0 +1,49 @@ +"""Compact duration parsing for windowed measures (``window='90d'``). + +Shared by the plan-time windowed guard (engine layer, DEV-1714) and the SQL +generator's per-unit interval emission (sql layer). Lives in ``slayer.core`` and +is dependency-free so the engine planner can validate a window duration at plan +time WITHOUT importing the SQL layer. + +Compact syntax only — an integer immediately followed by a unit, repeated with +no separators: ``1y2m3w5d6h7min8s``. Units: ``y`` year, ``m`` month, ``w`` week, +``d`` day, ``h`` hour, ``min`` minute, ``s`` second. +""" + +from __future__ import annotations + +import re + +# ``min`` must precede the single-char alternation so ``7min`` parses the whole +# ``min`` unit rather than a bare ``m`` followed by a stray ``in``. +_WINDOW_DURATION_RE = re.compile(r"(?P\d+)(?Pmin|[ymwdhs])") + + +def parse_window_duration(value: str) -> list[tuple[int, str]]: + """Parse a compact duration like ``1y2m3w5d6h7min8s`` into ``(amount, unit)`` + parts, in written order. + + Raises ``ValueError`` on an empty string, a non-positive amount, or any + malformed / gapped input (e.g. ``'90x'``, ``'d90'``). The error messages are + a stable contract — the plan-time guard and its tests match on them. + """ + if not value: + raise ValueError("Window duration cannot be empty") + pos = 0 + parts: list[tuple[int, str]] = [] + for match in _WINDOW_DURATION_RE.finditer(value): + if match.start() != pos: + raise ValueError( + f"Invalid window duration '{value}'. Use syntax like '1y2m3w5d6h7min8s'." + ) + amount = int(match.group("num")) + unit = match.group("unit") + if amount <= 0: + raise ValueError(f"Window duration parts must be positive in '{value}'") + parts.append((amount, unit)) + pos = match.end() + if pos != len(value) or not parts: + raise ValueError( + f"Invalid window duration '{value}'. Use syntax like '1y2m3w5d6h7min8s'." + ) + return parts diff --git a/slayer/engine/agg_registry.py b/slayer/engine/agg_registry.py new file mode 100644 index 00000000..4b7c224a --- /dev/null +++ b/slayer/engine/agg_registry.py @@ -0,0 +1,152 @@ +"""Stage 4 (DEV-1450) — aggregation registry helpers. + +Lifts the agg-name collection BFS from ``enrichment.py`` and the +parameter-resolution helpers from ``sql/generator.py`` so the new binder +modules don't have to reach into those tangles. The helpers are pure: +given a model + a resolve_join_target callback, they produce structured +results without touching storage or spawning side maps. + +Public surface: +- ``collect_reachable_agg_names`` — BFS the join graph for custom + aggregation names. +- ``resolve_aggregation`` — find an ``Aggregation`` definition by name. +- ``is_known_aggregation_name`` — built-in or in the custom set. +- ``required_params_for`` — required built-in params (e.g., + ``weighted_avg`` requires ``weight``). +- ``merge_agg_params`` — defaults from the agg-def overridden by + query-time kwargs. + +These are dormant in stage 4 — the existing call sites still inline +their own logic. Stages 7a/7b switch them over. +""" + +from __future__ import annotations + +from typing import Any, Awaitable, Callable, Dict, FrozenSet, List, Optional, Tuple + +from slayer.core.enums import ( + BUILTIN_AGGREGATION_REQUIRED_PARAMS, + BUILTIN_AGGREGATIONS, +) +from slayer.core.models import Aggregation, SlayerModel + + +# --------------------------------------------------------------------------- +# Agg-name collection +# --------------------------------------------------------------------------- + + +ResolveJoinTarget = Callable[..., Awaitable[Optional[Tuple[Any, SlayerModel]]]] + + +async def collect_reachable_agg_names( + source_model: SlayerModel, + resolve_join_target: ResolveJoinTarget, + named_queries: Optional[Dict] = None, +) -> Optional[FrozenSet[str]]: + """Collect custom aggregation names from ``source_model`` and every + join-reachable model. + + BFS bounded only by the visited set (no fixed depth cap — dotted-path + resolution supports arbitrary depth, so the agg-name rewrite must too). + Returns ``None`` when no custom aggregations exist anywhere in the + reachable subgraph. + + ``resolve_join_target`` is the existing engine callback whose return + shape is ``(target_sql, target_model) | None``; this helper only uses + the ``target_model`` element. + """ + names: set[str] = set() + visited: set[str] = set() + queue: List[SlayerModel] = [source_model] + + while queue: + current = queue.pop(0) + if current.name in visited: + continue + visited.add(current.name) + + if current.aggregations: + names.update(a.name for a in current.aggregations) + + for join in current.joins: + if join.target_model in visited: + continue + target_info = await resolve_join_target( + target_model_name=join.target_model, + named_queries=named_queries or {}, + ) + if target_info: + _, target_model_obj = target_info + if target_model_obj is not None: + queue.append(target_model_obj) + + return frozenset(names) if names else None + + +# --------------------------------------------------------------------------- +# Name-based lookups +# --------------------------------------------------------------------------- + + +def is_known_aggregation_name( + name: str, + custom_names: Optional[FrozenSet[str]], +) -> bool: + """``True`` if ``name`` is a built-in aggregation or appears in the + model-collected custom set. + """ + if name in BUILTIN_AGGREGATIONS: + return True + return bool(custom_names) and name in custom_names + + +def resolve_aggregation( + name: str, + available_aggs: List[Aggregation], +) -> Optional[Aggregation]: + """Return the ``Aggregation`` definition for ``name`` if one is + declared in ``available_aggs``, else ``None``. + + A model-level entry whose ``name`` matches a built-in is treated as + an override and returned. ``None`` for a built-in name with no + override is the signal to use the default built-in formula. + """ + for agg in available_aggs: + if agg.name == name: + return agg + return None + + +# --------------------------------------------------------------------------- +# Parameter resolution +# --------------------------------------------------------------------------- + + +def required_params_for(agg_name: str) -> Tuple[str, ...]: + """Required parameter names for a built-in aggregation (e.g., + ``weighted_avg`` requires ``weight``). + + Custom aggregations declare their parameter shape via + ``Aggregation.params``; this helper only knows about the built-in + table in ``slayer.core.enums``. Returns an empty tuple for unknown + names so callers can branch on emptiness rather than ``KeyError``. + """ + return tuple(BUILTIN_AGGREGATION_REQUIRED_PARAMS.get(agg_name, [])) + + +def merge_agg_params( + agg_def: Optional[Aggregation], + query_kwargs: Dict[str, Any], +) -> Dict[str, Any]: + """Combine ``Aggregation.params`` defaults with query-time kwargs. + + Query-time kwargs override defaults. Kwargs not declared by the + ``agg_def`` pass through unchanged — validation of param names + (e.g., rejecting unknown ones) is the binder's responsibility, + not this helper's. + """ + if agg_def is None: + return dict(query_kwargs) + defaults = {p.name: p.sql for p in agg_def.params} + return {**defaults, **query_kwargs} diff --git a/slayer/engine/aggregate_input_paths.py b/slayer/engine/aggregate_input_paths.py new file mode 100644 index 00000000..fc669216 --- /dev/null +++ b/slayer/engine/aggregate_input_paths.py @@ -0,0 +1,184 @@ +"""DEV-1709 (Stage 5) — plan-time crossing-input discovery for aggregates. + +The widened Law-3 trigger isolates a LOCAL aggregate into a host-rooted CTE +when ANY of its explicit inputs crosses a join. This module answers "which +join paths do the aggregate's inputs cross?" for every input kind: + +* **source** — a structural ``source.path`` contributes as-is; a derived + ``ColumnSqlKey`` with ``path == ()`` has its ``Column.sql`` expanded and + scanned with the shared Law-1 scanner (via + ``compute_column_filter_join_paths``, the same parse → expand → walk + pipeline the ``Column.filter`` trigger half uses). +* **positional args** (covers the explicit first/last time arg) — same + structural + derived-sql treatment. +* **kwargs** — column-valued kwargs same as args; template-fragment STRING + kwargs (user-supplied values for custom-aggregation params) are parsed + with the dialect-fallback chain and scanned. Model-default + ``AggregationParam.sql`` fragments of the custom aggregation named by + ``key.agg`` are scanned too — but only for params NOT overridden by a + user kwarg (an overridden default never renders). +* **``column_filter_key`` is deliberately NOT re-scanned** — the trigger + reads its bind-time ``SqlExprKey.referenced_join_paths`` directly + (DEV-1503, unchanged). + +Defensive fallbacks mirror ``column_filter_paths.py``: an unparseable +fragment contributes nothing (parity with the ``Column.filter`` scan — +pre-Stage-5 behavior is preserved for fragments the dialect fallback chain +cannot parse; a documented D1 carve-out, not an endorsement), and scalar / +duration / literal kwarg values contribute nothing. +""" + +from __future__ import annotations + +from typing import List, Optional, Tuple, Union + +from slayer.core.keys import AggregateKey, ColumnKey, ColumnSqlKey, StarKey +from slayer.core.models import SlayerModel +from slayer.engine.column_filter_paths import compute_column_filter_join_paths +from slayer.engine.source_bundle import ResolvedSourceBundle + +_PathList = List[Tuple[str, ...]] +_StructuralRef = Union[ColumnKey, ColumnSqlKey, StarKey] + + +def _add_path_prefixes(path: Tuple[str, ...], out: _PathList) -> None: + """Emit every prefix of ``path`` once (``("a", "b")`` → ``("a",)`` AND + ``("a", "b")``) — same prefix semantics as the Law-1 scanner.""" + for i in range(1, len(path) + 1): + prefix = tuple(path[:i]) + if prefix not in out: + out.append(prefix) + + +def _scan_sql_fragment( + sql: str, + *, + anchor_model: SlayerModel, + anchor_relation: str, + bundle: ResolvedSourceBundle, + out: _PathList, +) -> None: + """Scan a free-SQL fragment (derived ``Column.sql`` or a template + fragment) for crossed join paths, reusing the filter-side pipeline + (dialect-fallback parse → anchor-derived expansion → root-scope walk). + Unparseable fragments contribute nothing.""" + for path in compute_column_filter_join_paths( + canonical_sql=sql, + anchor_model=anchor_model, + anchor_relation=anchor_relation, + bundle=bundle, + ): + if path not in out: + out.append(path) + + +def _collect_ref_paths( + ref: object, + *, + anchor_model: SlayerModel, + anchor_relation: str, + bundle: ResolvedSourceBundle, + out: _PathList, +) -> None: + """Crossed paths of one embedded reference (source / arg / kwarg value). + + Scalars (Decimal / int / float / None) contribute nothing; strings are + template fragments and get the free-SQL scan. + """ + if isinstance(ref, (ColumnKey, StarKey)): + _add_path_prefixes(tuple(getattr(ref, "path", ()) or ()), out) + return + if isinstance(ref, ColumnSqlKey): + if ref.path: + # Structural crossing; any FURTHER crossing inside the target's + # own Column.sql is the target-rooted CTE's concern (Stage 4). + _add_path_prefixes(tuple(ref.path), out) + return + col = next( + (c for c in anchor_model.columns if c.name == ref.column_name), + None, + ) + if col is not None and col.sql: + _scan_sql_fragment( + col.sql, + anchor_model=anchor_model, + anchor_relation=anchor_relation, + bundle=bundle, + out=out, + ) + return + if isinstance(ref, str): + _scan_sql_fragment( + ref, + anchor_model=anchor_model, + anchor_relation=anchor_relation, + bundle=bundle, + out=out, + ) + + +def _collect_default_fragment_paths( + key: AggregateKey, + *, + anchor_model: SlayerModel, + anchor_relation: str, + bundle: ResolvedSourceBundle, + out: _PathList, +) -> None: + """Scan the model-default ``AggregationParam.sql`` fragments of the + custom aggregation named by ``key.agg`` — skipping params a user kwarg + overrides (the default never renders for those).""" + agg_def = next( + (a for a in (anchor_model.aggregations or []) if a.name == key.agg), + None, + ) + if agg_def is None: + return + overridden = {name for name, _ in key.kwargs} + for param in agg_def.params or []: + param_sql: Optional[str] = getattr(param, "sql", None) + if param.name in overridden or not param_sql: + continue + _scan_sql_fragment( + param_sql, + anchor_model=anchor_model, + anchor_relation=anchor_relation, + bundle=bundle, + out=out, + ) + + +def compute_aggregate_input_join_paths( + *, + key: AggregateKey, + anchor_model: Optional[SlayerModel], + anchor_relation: str, + bundle: ResolvedSourceBundle, +) -> Tuple[Tuple[str, ...], ...]: + """Ordered, de-duplicated tuple of join-path prefixes crossed by the + aggregate's explicit inputs (source, positional args, kwargs, and + non-overridden custom-aggregation default fragments). + + ``()`` for a purely-local aggregate. ``column_filter_key`` crossing is + intentionally excluded — read ``referenced_join_paths`` on the key. + """ + if anchor_model is None: + return () + out: _PathList = [] + refs: List[object] = [key.source, *key.args, *(v for _, v in key.kwargs)] + for ref in refs: + _collect_ref_paths( + ref, + anchor_model=anchor_model, + anchor_relation=anchor_relation, + bundle=bundle, + out=out, + ) + _collect_default_fragment_paths( + key, + anchor_model=anchor_model, + anchor_relation=anchor_relation, + bundle=bundle, + out=out, + ) + return tuple(out) diff --git a/slayer/engine/binding.py b/slayer/engine/binding.py new file mode 100644 index 00000000..0a8eb811 --- /dev/null +++ b/slayer/engine/binding.py @@ -0,0 +1,1351 @@ +"""Stage 7a.5 (DEV-1450) — ExpressionBinder + FilterBinder. + +The binder consumes a ``ParsedExpr`` (from ``slayer/engine/syntax.py``) +plus a scope (``ModelScope`` or ``StageSchema``) and a +``ResolvedSourceBundle`` (for join resolution). It produces a typed +``BoundExpr`` whose leaves are resolved ``ValueKey``s. + +Public surface: + +* ``bind_expr(parsed, *, scope, bundle) -> BoundExpr`` +* ``bind_filter(parsed, *, scope, bundle) -> BoundFilter`` + +Two scope kinds (P5): + +* ``ModelScope``: joins exist; dotted refs walk the join graph rooted + at ``source_model``. ``__``-bearing refs raise + ``IllegalScopeReferenceError`` unless they exact-match a column on + the model. I2: ``source_model is not None`` is asserted. +* ``StageSchema``: flat namespace; dotted refs raise + ``IllegalScopeReferenceError``; flat names with ``__`` are legal. + +C14: same-model self-prefix in Mode-B (`orders.status` over an +``orders``-rooted query) is stripped before the join walk. + +FilterBinder layers on top: ``bind_expr`` + phase classification +(``Phase.ROW`` / ``AGGREGATE`` / ``POST`` = the max phase of any +referenced slot) + walk for the referenced ``ValueKey``s + reject +filters that touch a windowed ``Column.sql``. + +Dormant in 7a — no engine wiring. The planner (7a.6) is the first +consumer. +""" + +from __future__ import annotations + +from typing import Dict, List, Optional, Tuple, Union + +from pydantic import BaseModel, ConfigDict, Field + +from slayer.core.errors import ( + AggregationNotAllowedError, + IllegalScopeReferenceError, + IllegalWindowInFilterError, + UnknownFunctionError, + UnknownReferenceError, +) +from slayer.core.enums import ( + BUILTIN_AGGREGATIONS, + DEFAULT_AGGREGATIONS_BY_TYPE, + PRIMARY_KEY_AGGREGATIONS, + DataType, + format_unknown_aggregation, + normalize_aggregation_name, +) +from slayer.core.keys import ( + SCALAR_FUNCTIONS, + AggregateKey, + ArithmeticKey, + BetweenKey, + ColumnKey, + ColumnSqlKey, + InKey, + LiteralKey, + Phase, + ScalarCallKey, + SqlExprKey, + StarKey, + TimeTruncKey, + TransformKey, + ValueKey, + column_leaf, + column_path, + normalize_scalar, +) +from slayer.core.models import SlayerModel +from slayer.core.query import TimeDimension +from slayer.core.scope import ModelScope, StageSchema +from slayer.engine.column_filter_paths import compute_column_filter_join_paths +from slayer.engine.source_bundle import ResolvedSourceBundle +from slayer.engine.syntax import ( + AggCall, + Arith, + BoolOp, + Cmp, + DottedRef, + Literal, + ParsedExpr, + Ref, + ScalarCall, + StarSource, + TransformCall, + TupleLit, + UnaryOp, +) +from slayer.sql.sql_expr import has_window_function + +__all__ = [ + "BoundExpr", + "BoundFilter", + "bind_expr", + "bind_filter", + "bind_time_dimension", + "walk_value_keys", +] + + +_TEMPORAL_TYPES = frozenset({DataType.DATE, DataType.TIMESTAMP}) + + +# --------------------------------------------------------------------------- +# BoundExpr / BoundFilter +# --------------------------------------------------------------------------- + + +class BoundExpr(BaseModel): + """A bound expression — its leaves are resolved ``ValueKey``s. + + ``value_key`` is the structural identity of the entire expression. + ``phase`` is the property of ``value_key.phase`` (lifted for + convenience). + """ + + model_config = ConfigDict(arbitrary_types_allowed=True, frozen=True) + + value_key: ValueKey + + @property + def phase(self) -> Phase: + return self.value_key.phase + + +class BoundFilter(BaseModel): + """A bound filter predicate. + + The same ``value_key`` shape as ``BoundExpr`` (boolean ops and + comparisons are encoded as ``ArithmeticKey`` with the corresponding + op string), plus: + + * ``phase`` — the maximum phase any referenced slot reaches. + * ``referenced_keys`` — every ``ValueKey`` touched anywhere in the + bound tree (used by the cross-model planner's filter routing). + """ + + model_config = ConfigDict(arbitrary_types_allowed=True, frozen=True) + + value_key: ValueKey + phase: Phase + referenced_keys: Tuple[ValueKey, ...] = Field(default_factory=tuple) + + +# --------------------------------------------------------------------------- +# Public entry points +# --------------------------------------------------------------------------- + + +def bind_expr( + parsed: ParsedExpr, + *, + scope: Union[ModelScope, StageSchema], + bundle: ResolvedSourceBundle, +) -> BoundExpr: + """Bind a parsed expression against a scope. + + Returns a ``BoundExpr`` carrying the structural identity of the + entire expression. Raises ``UnknownReferenceError`` if a ref doesn't + resolve; ``IllegalScopeReferenceError`` if a dotted ref is used + against a ``StageSchema`` (or vice versa for ``__`` against a + ``ModelScope``). + """ + value_key = _bind(parsed, scope=scope, bundle=bundle, in_filter=False) + return BoundExpr(value_key=value_key) + + +def bind_time_dimension( + td: TimeDimension, + *, + scope: Union[ModelScope, StageSchema], + bundle: ResolvedSourceBundle, +) -> BoundExpr: + """Bind a ``TimeDimension`` into a ``BoundExpr`` carrying a + ``TimeTruncKey``. + + The underlying column is resolved against ``scope`` exactly like a + Mode-B identifier ref (local name or dotted-join path); the bound + column must be a plain ``ColumnKey`` whose ``Column.type`` is in the + temporal bucket (``DATE`` / ``TIMESTAMP``). + + Stage 7b.3b limitations: + + * Only ``ModelScope`` with a non-None ``source_model`` is accepted. + Downstream stages bind upstream-emitted truncated columns by flat + name through ``bind_expr``; they do not re-truncate at a different + grain through this entry point. Passing a ``StageSchema`` raises + ``IllegalScopeReferenceError``. + * Derived (``Column.sql`` is set) temporal columns route through + ``ColumnSqlKey`` rather than ``ColumnKey``, and ``TimeTruncKey`` + is typed as ``column: ColumnKey``. Rather than silently widen the + typed key, this stage rejects derived-TD columns with + ``NotImplementedError`` and a clear message. + """ + if isinstance(scope, StageSchema): + raise IllegalScopeReferenceError( + name=td.dimension.full_name, + scope_kind="StageSchema", + reason=( + "time dimensions only bind against a ModelScope; downstream " + "stages already see the truncated column as a flat name " + "from the upstream stage's schema." + ), + ) + + assert isinstance(scope, ModelScope) + if scope.source_model is None: + raise UnknownReferenceError( + name=td.dimension.full_name, + scope_kind="ModelScope", + scope_summary="(no source_model anchor; anchor-less mode not implemented)", + suggestion=None, + ) + + full = td.dimension.full_name + if "." in full: + parts = tuple(full.split(".")) + bound_col = _resolve_dotted(parts, scope=scope, bundle=bundle) + else: + bound_col = _resolve_ref(full, scope=scope, bundle=bundle) + + if not isinstance(bound_col, (ColumnKey, ColumnSqlKey)): + # Defensive — the binder should never produce a non-column key + # for an identifier ref against a ModelScope. + raise ValueError( + f"TimeDimension {full!r} did not resolve to a column " + f"reference (got {type(bound_col).__name__})." + ) + + # DEV-1450 follow-up #4a: a derived (Column.sql) temporal column routes + # through ColumnSqlKey; TimeTruncKey.column accepts both kinds, so the + # leaf / path are read via the kind-agnostic helpers. + terminal_model = _terminal_model_for_path( + path=column_path(bound_col), + scope=scope, + bundle=bundle, + ) + if terminal_model is None: + # Shouldn't be reachable: _resolve_ref / _resolve_dotted would + # already have raised. Defensive only. + raise UnknownReferenceError( + name=full, + scope_kind="ModelScope", + scope_summary=f"could not resolve terminal model for {full!r}", + suggestion=None, + ) + col = next( + (c for c in terminal_model.columns if c.name == column_leaf(bound_col)), + None, + ) + if col is None or col.type not in _TEMPORAL_TYPES: + observed = col.type if col is not None else "" + raise ValueError( + f"TimeDimension {full!r} must reference a temporal column " + f"(DATE / TIMESTAMP); got column type {observed!r}." + ) + + return BoundExpr( + value_key=TimeTruncKey( + column=bound_col, granularity=str(td.granularity.value), + ), + ) + + +def _terminal_model_for_path( + *, + path: Tuple[str, ...], + scope: ModelScope, + bundle: ResolvedSourceBundle, +) -> Optional[SlayerModel]: + """Walk ``path`` from ``scope.source_model`` and return the terminal + model. Returns the host when ``path`` is empty. + """ + current = scope.source_model + if current is None: + return None + for hop in path: + nxt = bundle.get_referenced_model(hop) + if nxt is None: + return None + current = nxt + return current + + +def bind_filter( + parsed: ParsedExpr, + *, + scope: Union[ModelScope, StageSchema], + bundle: ResolvedSourceBundle, + alias_map: Optional[Dict[str, "ValueKey"]] = None, +) -> BoundFilter: + """Bind a parsed filter predicate + classify its phase. + + Walks the bound tree to gather every referenced ``ValueKey`` and + raises ``IllegalWindowInFilterError`` if any referenced + ``Column.sql`` contains a window function (DEV-1369: no + auto-promotion). + + ``alias_map`` maps a stage's declared-measure names (user ``name``, + canonical alias, declared name) to their bound ``ValueKey`` so a + filter may reference a declared measure by alias (P4 / DEV-1445: + ``filters=["rev >= 100"]`` for a measure declared ``name="rev"``). + A bare ref that matches an alias interns onto that exact slot rather + than resolving against the model columns — so the colon form and the + alias form share one slot. + """ + value_key = _bind( + parsed, scope=scope, bundle=bundle, in_filter=True, alias_map=alias_map, + ) + refs = tuple(walk_value_keys(value_key)) + phase = max( + (k.phase for k in refs), + default=value_key.phase, + ) + _reject_windowed_column_sql(refs, scope=scope, bundle=bundle, parsed=parsed) + return BoundFilter( + value_key=value_key, phase=phase, referenced_keys=refs, + ) + + +# --------------------------------------------------------------------------- +# Walk helper +# --------------------------------------------------------------------------- + + +_VALUE_KEY_TYPES = ( + ColumnKey, ColumnSqlKey, StarKey, LiteralKey, + AggregateKey, TransformKey, ArithmeticKey, ScalarCallKey, + BetweenKey, InKey, TimeTruncKey, +) + + +def walk_value_keys(key: ValueKey): + """Yield every ``ValueKey`` reachable from ``key``, including ``key``.""" + yield key + if isinstance(key, AggregateKey): + if isinstance(key.source, _VALUE_KEY_TYPES): + yield from walk_value_keys(key.source) + for a in key.args: + if isinstance(a, _VALUE_KEY_TYPES): + yield from walk_value_keys(a) + for _, v in key.kwargs: + if isinstance(v, _VALUE_KEY_TYPES): + yield from walk_value_keys(v) + elif isinstance(key, TransformKey): + if isinstance(key.input, _VALUE_KEY_TYPES): + yield from walk_value_keys(key.input) + for a in key.args: + if isinstance(a, _VALUE_KEY_TYPES): + yield from walk_value_keys(a) + for _, v in key.kwargs: + if isinstance(v, _VALUE_KEY_TYPES): + yield from walk_value_keys(v) + for pk in key.partition_keys: + yield from walk_value_keys(pk) + if key.time_key is not None: + yield from walk_value_keys(key.time_key) + elif isinstance(key, ArithmeticKey): + for op in key.operands: + yield from walk_value_keys(op) + elif isinstance(key, ScalarCallKey): + for arg in key.args: + if isinstance(arg, _VALUE_KEY_TYPES): + yield from walk_value_keys(arg) + elif isinstance(key, BetweenKey): + yield from walk_value_keys(key.column) + yield from walk_value_keys(key.low) + yield from walk_value_keys(key.high) + elif isinstance(key, InKey): + # DEV-1475: walk the column LHS and every literal RHS so the + # cross-model filter router and the windowed-column rejection + # check both see InKey-rooted predicates the same way they see + # BetweenKey ones. + yield from walk_value_keys(key.column) + for v in key.values: + yield from walk_value_keys(v) + + +# --------------------------------------------------------------------------- +# Internals +# --------------------------------------------------------------------------- + + +def _bind( + parsed: ParsedExpr, + *, + scope: Union[ModelScope, StageSchema], + bundle: ResolvedSourceBundle, + in_filter: bool, + alias_map: Optional[Dict[str, "ValueKey"]] = None, +) -> ValueKey: + if isinstance(parsed, Literal): + return LiteralKey(value=normalize_scalar(parsed.value)) + + if isinstance(parsed, Ref): + return _resolve_ref( + parsed.name, scope=scope, bundle=bundle, alias_map=alias_map, + ) + + if isinstance(parsed, DottedRef): + return _resolve_dotted(parsed.parts, scope=scope, bundle=bundle) + + if isinstance(parsed, StarSource): + return StarKey() + + if isinstance(parsed, AggCall): + return _bind_agg(parsed, scope=scope, bundle=bundle) + + if isinstance(parsed, TransformCall): + return _bind_transform( + parsed, scope=scope, bundle=bundle, alias_map=alias_map, + ) + + if isinstance(parsed, ScalarCall): + return _bind_scalar( + parsed, scope=scope, bundle=bundle, in_filter=in_filter, + alias_map=alias_map, + ) + + if isinstance(parsed, Arith): + return ArithmeticKey( + op=parsed.op, + operands=( + _bind(parsed.left, scope=scope, bundle=bundle, in_filter=in_filter, alias_map=alias_map), + _bind(parsed.right, scope=scope, bundle=bundle, in_filter=in_filter, alias_map=alias_map), + ), + ) + + if isinstance(parsed, UnaryOp): + return ArithmeticKey( + op=parsed.op, + operands=(_bind(parsed.operand, scope=scope, bundle=bundle, in_filter=in_filter, alias_map=alias_map),), + ) + + if isinstance(parsed, Cmp): + # DEV-1475: ``IN`` / ``NOT IN`` predicates fold into a single + # ``InKey`` rather than an ``ArithmeticKey`` so the SQL generator + # has a structured handle on the column + literal-tuple shape. + # The parser already validated that ``parsed.right`` is a + # ``TupleLit`` of ``Literal`` elements for these ops. + if parsed.op in ("in", "not in"): + return _bind_in( + parsed, + scope=scope, bundle=bundle, in_filter=in_filter, + alias_map=alias_map, + ) + return ArithmeticKey( + op=parsed.op, + operands=( + _bind(parsed.left, scope=scope, bundle=bundle, in_filter=in_filter, alias_map=alias_map), + _bind(parsed.right, scope=scope, bundle=bundle, in_filter=in_filter, alias_map=alias_map), + ), + ) + + if isinstance(parsed, BoolOp): + operands = tuple( + _bind(v, scope=scope, bundle=bundle, in_filter=in_filter, alias_map=alias_map) + for v in parsed.operands + ) + return ArithmeticKey(op=parsed.op, operands=operands) + + raise ValueError( + f"Unsupported ParsedExpr node: {type(parsed).__name__}" + ) + + +def _bind_in( + parsed: Cmp, + *, + scope: Union[ModelScope, StageSchema], + bundle: ResolvedSourceBundle, + in_filter: bool, + alias_map: Optional[Dict[str, "ValueKey"]] = None, +) -> InKey: + """Bind an ``IN`` / ``NOT IN`` predicate into an ``InKey`` (DEV-1475). + + The LHS is bound through the normal column-resolution path + (``ColumnKey`` for a bare ref, ``ColumnKey`` with a non-empty + ``path`` for a dotted join ref, ``ColumnSqlKey`` for a derived + column, or an alias-map hit for a declared-measure name). + + The RHS is a ``TupleLit`` of ``Literal`` nodes (the parser already + enforced that shape); every element binds to a ``LiteralKey`` after + scalar normalization. + """ + if not isinstance(parsed.right, TupleLit): + # Defensive — the parser's ``ast.Compare`` branch guarantees a + # ``TupleLit`` on the RHS for ``in`` / ``not in``. Surface a + # clear runtime error if a future caller bypasses the parser. + raise ValueError( + f"_bind_in: expected TupleLit on RHS of {parsed.op!r}, got " + f"{type(parsed.right).__name__}." + ) + column = _bind( + parsed.left, + scope=scope, bundle=bundle, in_filter=in_filter, alias_map=alias_map, + ) + values = tuple( + LiteralKey(value=normalize_scalar(elt.value)) + for elt in parsed.right.elements + ) + return InKey( + column=column, + values=values, + negated=(parsed.op == "not in"), + ) + + +def _resolve_ref( + name: str, + *, + scope: Union[ModelScope, StageSchema], + bundle: ResolvedSourceBundle, + alias_map: Optional[Dict[str, "ValueKey"]] = None, +) -> ValueKey: + """Resolve a bare identifier against the scope. + + A name present in ``alias_map`` (a stage's declared-measure aliases, + supplied only on the filter/order path) interns onto that declared + slot's ``ValueKey`` before any column lookup — so a filter referencing + a measure by its user ``name`` shares the measure's slot (P4). + """ + if alias_map and name in alias_map: + return alias_map[name] + + if isinstance(scope, StageSchema): + col = scope.get(name) + if col is None: + raise UnknownReferenceError( + name=name, + scope_kind="StageSchema", + scope_summary=( + f"stage {scope.relation_name!r} columns: " + f"{[c.name for c in scope.columns]}" + ), + suggestion=None, + ) + return ColumnKey(path=(), leaf=name) + + assert isinstance(scope, ModelScope) + if scope.source_model is None: + raise UnknownReferenceError( + name=name, + scope_kind="ModelScope", + scope_summary="(no source_model anchor; anchor-less mode not implemented)", + suggestion=None, + ) + model = scope.source_model + + if "__" in name: + # The Mode-B parser already rejects `__` for user input; this + # branch is reached only via direct ParsedExpr.Ref construction + # (e.g., downstream binders for StageSchema flat columns). The + # `__` is legal iff it exact-matches a column literally named + # that way on the model (legacy persisted query-backed columns). + if any(c.name == name for c in model.columns): + return ColumnKey(path=(), leaf=name) + raise IllegalScopeReferenceError( + name=name, + scope_kind="ModelScope", + reason=( + "`__` is reserved for internal join-path aliases. " + "Use single-dot DSL paths in queries." + ), + ) + + col = next((c for c in model.columns if c.name == name), None) + if col is None: + # Try ModelMeasure as a fallback for bare measure refs. + mm = next((m for m in model.measures if m.name == name), None) + if mm is not None: + # ModelMeasure expansion lives in the planner; the binder + # raises here so callers know expansion is required. + raise UnknownReferenceError( + name=name, + scope_kind="ModelScope", + scope_summary=f"model {model.name!r}", + suggestion=( + f"{name!r} is a saved measure on {model.name!r}; " + f"expand via ModelMeasure expansion before binding." + ), + ) + raise UnknownReferenceError( + name=name, + scope_kind="ModelScope", + scope_summary=( + f"model {model.name!r} columns: " + f"{[c.name for c in model.columns]}" + ), + suggestion=None, + ) + + if col.sql is not None and col.sql.strip() != name: + return ColumnSqlKey(path=(), model=model.name, column_name=col.name) + return ColumnKey(path=(), leaf=col.name) + + +def _resolve_dotted( + parts: Tuple[str, ...], + *, + scope: Union[ModelScope, StageSchema], + bundle: ResolvedSourceBundle, +) -> ValueKey: + """Resolve a dotted ref against the scope.""" + if isinstance(scope, StageSchema): + raise IllegalScopeReferenceError( + name=".".join(parts), + scope_kind="StageSchema", + reason=( + "downstream stages see a flat schema — dotted refs are " + "not legal. Use the flat column name." + ), + ) + + assert isinstance(scope, ModelScope) + if scope.source_model is None: + raise UnknownReferenceError( + name=".".join(parts), + scope_kind="ModelScope", + scope_summary="(no source_model anchor; anchor-less mode not implemented)", + suggestion=None, + ) + + # C14: strip same-model self-prefix. + host = scope.source_model + if parts and parts[0] == host.name: + parts = parts[1:] + if not parts: + raise UnknownReferenceError( + name=host.name, + scope_kind="ModelScope", + scope_summary=f"model {host.name!r}", + suggestion="self-prefix only — expected a column or join target.", + ) + if len(parts) == 1: + return _resolve_ref(parts[0], scope=scope, bundle=bundle) + + # parts now has the join walk to perform. + if len(parts) == 1: + # Single-segment after possible stripping — already a local ref. + return _resolve_ref(parts[0], scope=scope, bundle=bundle) + + # Walk join chain. parts[:-1] are join targets; parts[-1] is the leaf column. + hop_path = parts[:-1] + leaf = parts[-1] + current = host + visited_models = {host.name} + for hop in hop_path: + join = next( + (j for j in current.joins if j.target_model == hop), None, + ) + if join is None: + raise UnknownReferenceError( + name=".".join(parts), + scope_kind="ModelScope", + scope_summary=( + f"model {current.name!r} joins: " + f"{[j.target_model for j in current.joins]}" + ), + suggestion=f"model {current.name!r} has no join to {hop!r}.", + ) + nxt = bundle.get_referenced_model(hop) + if nxt is None: + raise UnknownReferenceError( + name=".".join(parts), + scope_kind="ModelScope", + scope_summary=f"target {hop!r} not in source bundle", + suggestion=None, + ) + # A dotted path that walks back to an already-visited model is a + # circular join (e.g. ``a -> b -> a``); the leaf can never resolve + # and an unguarded walk would otherwise just fail confusingly on the + # leaf. Raise the legacy-compatible ``Circular join`` ValueError. + if nxt.name in visited_models: + raise ValueError( + f"Circular join detected resolving {'.'.join(parts)!r}: " + f"revisits model {nxt.name!r}." + ) + visited_models.add(nxt.name) + current = nxt + + # `current` is the terminal model; `leaf` is the column on it. + col = next((c for c in current.columns if c.name == leaf), None) + if col is None: + raise UnknownReferenceError( + name=".".join(parts), + scope_kind="ModelScope", + scope_summary=( + f"model {current.name!r} columns: " + f"{[c.name for c in current.columns]}" + ), + suggestion=None, + ) + + if col.sql is not None and col.sql.strip() != leaf: + # Derived column on a joined model. The path is part of the key + # so the cross-model planner can route via the join graph. + return ColumnSqlKey( + path=tuple(hop_path), model=current.name, column_name=leaf, + ) + return ColumnKey(path=tuple(hop_path), leaf=leaf) + + +def _resolve_dotted_star( + parts: Tuple[str, ...], + *, + scope: Union[ModelScope, StageSchema], + bundle: ResolvedSourceBundle, +) -> StarKey: + """Resolve a dotted star (``customers.*``, trailing ``*``) to a StarKey. + + Mirrors ``_resolve_dotted``'s self-prefix strip (C14) and join-chain + validation, but the leaf is ``*`` (no terminal column) so the result + is a ``StarKey`` whose ``path`` is the validated hop chain. An empty + path after stripping is the local star (``orders.*`` on ``orders``). + """ + assert parts and parts[-1] == "*" + if isinstance(scope, StageSchema): + raise IllegalScopeReferenceError( + name=".".join(parts), + scope_kind="StageSchema", + reason=( + "downstream stages see a flat schema — dotted refs are " + "not legal. Use the flat column name." + ), + ) + assert isinstance(scope, ModelScope) + host = scope.source_model + if host is None: + raise UnknownReferenceError( + name=".".join(parts), + scope_kind="ModelScope", + scope_summary="(no source_model anchor; anchor-less mode not implemented)", + suggestion=None, + ) + hop_path = parts[:-1] + # C14: strip same-model self-prefix (``orders.*`` on ``orders``). + if hop_path and hop_path[0] == host.name: + hop_path = hop_path[1:] + current = host + visited_models = {host.name} + for hop in hop_path: + join = next((j for j in current.joins if j.target_model == hop), None) + if join is None: + raise UnknownReferenceError( + name=".".join(parts), + scope_kind="ModelScope", + scope_summary=( + f"model {current.name!r} joins: " + f"{[j.target_model for j in current.joins]}" + ), + suggestion=f"no join from {current.name!r} to {hop!r}.", + ) + nxt = bundle.get_referenced_model(hop) + if nxt is None: + raise UnknownReferenceError( + name=".".join(parts), + scope_kind="ModelScope", + scope_summary=f"target {hop!r} not in source bundle", + suggestion=None, + ) + # A dotted star that revisits a model is a circular join (``a.b.a.*``) + # — reject it the same way ``_resolve_dotted`` rejects ``a.b.a.col`` + # so the two stay consistent (CR). + if nxt.name in visited_models: + raise ValueError( + f"Circular join detected resolving {'.'.join(parts)!r}: " + f"revisits model {nxt.name!r}." + ) + visited_models.add(nxt.name) + current = nxt + return StarKey(path=tuple(hop_path)) + + +def _bind_agg( + parsed: AggCall, *, + scope: Union[ModelScope, StageSchema], + bundle: ResolvedSourceBundle, +) -> AggregateKey: + if isinstance(parsed.source, StarSource): + source = StarKey() + elif ( + isinstance(parsed.source, DottedRef) + and parsed.source.parts + and parsed.source.parts[-1] == "*" + ): + # Cross-model star: ``customers.*:count`` → a StarKey carrying the + # join path so the cross-model planner routes COUNT(*) through the + # join graph, exactly like ``customers.revenue:sum`` (P3). Parity + # with the legacy dotted-star path. + source = _resolve_dotted_star( + parsed.source.parts, scope=scope, bundle=bundle, + ) + else: + bound_source = _bind( + parsed.source, scope=scope, bundle=bundle, in_filter=False, + ) + if not isinstance(bound_source, (ColumnKey, ColumnSqlKey, StarKey)): + raise ValueError( + f"Aggregation source must resolve to a column / star, " + f"got {type(bound_source).__name__}." + ) + source = bound_source + + # Bind args / kwargs. For aggregations, identifier args/kwargs become + # ColumnKey via the binder; scalars normalise. + args = tuple( + _bind_agg_arg(a, scope=scope, bundle=bundle) for a in parsed.args + ) + kwargs = tuple( + (k, _bind_agg_arg(v, scope=scope, bundle=bundle)) + for k, v in parsed.kwargs + ) + # DEV-1450 stage 7b.12: propagate ``Column.filter`` into the + # AggregateKey's structural identity. The resolved source's column + # may carry a Mode-A SQL fragment (``filter="status = 'paid'"``) + # that wraps the aggregate argument as ``SUM(CASE WHEN ... THEN col + # END)``. Two aggregates over the same column with different + # ``Column.filter`` therefore differ at the key level; same-filter + # ones intern (legacy CASE-WHEN-at-agg-time semantics, preserved by + # the spec's C5 + ``column_filter_key`` invariants). + column_filter_key = _resolve_column_filter_key( + source=source, bundle=bundle, + ) + # Codex review: enforce per-column aggregation eligibility gates + # the legacy enrichment site at ``enrichment.py:401-417`` enforced. + # Without this, ``id:sum`` (a PK) or ``status:avg`` (text) compile + # silently in the typed pipeline. The check is best-effort against + # the bundle — sources whose target model can't be resolved (e.g. + # an unreferenced join target) skip the check. + # DEV-1576 / DEV-1717: heal alias + gate, then store the EFFECTIVE + # (healed) name on the key so the generator resolves the canonical + # aggregation rather than the raw parser token. + effective_agg = _validate_agg_eligibility( + source=source, agg=parsed.agg, bundle=bundle, + ) + return AggregateKey( + source=source, + agg=effective_agg, + args=args, + kwargs=kwargs, + column_filter_key=column_filter_key, + ) + + +def _resolve_column_filter_key( + *, source, bundle: ResolvedSourceBundle, +) -> Optional[SqlExprKey]: + """Look up the resolved source's ``Column.filter`` and convert it + to a ``SqlExprKey``. + + Returns ``None`` for ``StarKey`` sources (``*:count`` has no column + to attach a filter to) and for any column whose ``filter`` is + unset. For ``ColumnKey`` / ``ColumnSqlKey`` sources the resolver + walks ``source.path`` through the bundle and reads the target + model's column entry. Models the planner doesn't have access to + (e.g. an unresolved join target) are tolerated — no exception is + raised; the key just stays ``None`` (the compile-time validator + in path resolution would have caught a genuinely missing model). + """ + if isinstance(source, StarKey): + return None + path = getattr(source, "path", ()) + leaf = getattr(source, "leaf", None) or getattr(source, "column_name", None) + if leaf is None: + return None + host = bundle.source_model + if host is None: + return None + current: SlayerModel = host + for hop in path: + nxt = bundle.get_referenced_model(hop) + if nxt is None: + return None + current = nxt + col = next((c for c in current.columns if c.name == leaf), None) + if col is None or not col.filter: + return None + # DEV-1503 — stamp the typed non-anchor join paths on the SqlExprKey so + # the planner's isolation trigger reads typed data, not parsed SQL. The + # anchor is the model the filter is bound against — the joined target for + # cross-model aggregates, the host for filtered-local. The anchor relation + # uses the ``__``-canonical path alias when the anchor is a joined model. + anchor_relation = "__".join(path) if path else current.name + paths = compute_column_filter_join_paths( + canonical_sql=col.filter, + anchor_model=current, + anchor_relation=anchor_relation, + bundle=bundle, + ) + return SqlExprKey(canonical_sql=col.filter, referenced_join_paths=paths) + + +def _resolve_gate_owner( + source, bundle: ResolvedSourceBundle, +) -> "Optional[tuple[SlayerModel, str]]": + """Resolve the ``(owning_model, leaf)`` an aggregation gate applies to. + + Returns ``None`` when the target can't be confirmed — a ``StarKey`` + (``*:count`` has no column), a source with no leaf, no host model, or an + unresolved join hop — so the caller best-effort skips the gate (the + compile-time path validator catches truly broken refs). + """ + if isinstance(source, StarKey): + return None + leaf = getattr(source, "leaf", None) or getattr(source, "column_name", None) + if leaf is None: + return None + host = bundle.source_model + if host is None: + return None + current: SlayerModel = host + for hop in tuple(getattr(source, "path", ())): + nxt = bundle.get_referenced_model(hop) + if nxt is None: + return None + current = nxt + return current, leaf + + +def _validate_agg_eligibility( + *, source, agg: str, bundle: ResolvedSourceBundle, +) -> str: + """Heal the aggregation name and enforce per-column eligibility gates. + + Returns the **effective** (alias-healed) aggregation name, which the + caller stores on ``AggregateKey.agg`` (DEV-1576 / DEV-1717) — the typed + colon parser (``syntax.py``) does not normalise, so healing must land + here, after the owning model is resolved, or the generator later fails on + the raw token at ``_resolve_aggregation_def``. + + Healing (:func:`normalize_aggregation_name`) is **skipped** when the raw + token exactly matches a custom aggregation registered on the owning model, + so a custom ``countd`` wins over the ``countd -> count_distinct`` alias. + + Gate order (mirrors the legacy ``enrichment.py`` v2 contract): + + 0. Unknown-name-first: a name that is neither a built-in nor a model + custom aggregation raises ``"Unknown aggregation ..."`` **before** the + PK / whitelist / type gates, so a misspelled agg on an otherwise + aggregatable column is not mislabelled as a type restriction. + 1. Primary-key columns are restricted to ``count`` / ``count_distinct``. + 2. An explicit ``Column.allowed_aggregations`` whitelist overrides + type defaults. + 3. Otherwise, built-in aggregations are gated by + ``DEFAULT_AGGREGATIONS_BY_TYPE``; model-custom aggregations are exempt. + + ``StarKey`` sources (``*:count`` / ``customers.*:count``) have no + column to attach a whitelist to and pass through. Cross-model and + derived (``ColumnSqlKey``) sources are best-effort: if the target + model can't be resolved through the bundle (an unresolved join + target) the gate is skipped — the compile-time path validator would + have raised earlier on a truly broken ref. + """ + owner = _resolve_gate_owner(source, bundle) + if owner is None: + return normalize_aggregation_name(agg) + current, leaf = owner + # DEV-1576 alias healing — custom aggregation named like an alias wins. + custom_names = {a.name for a in (current.aggregations or [])} + effective = agg if agg in custom_names else normalize_aggregation_name(agg) + # Gate 0: unknown-name-first (precedence over PK / whitelist / type). + known = BUILTIN_AGGREGATIONS | custom_names + if effective not in known: + raise ValueError(format_unknown_aggregation(effective, known)) + col = next((c for c in current.columns if c.name == leaf), None) + if col is None: + return effective + if col.primary_key: + if effective not in PRIMARY_KEY_AGGREGATIONS: + raise AggregationNotAllowedError( + column=leaf, + agg=effective, + reason=( + f"primary-key column {leaf!r} restricted to " + f"{sorted(PRIMARY_KEY_AGGREGATIONS)}; got {effective!r}." + ), + ) + return effective + if col.allowed_aggregations is not None: + if effective not in col.allowed_aggregations: + raise AggregationNotAllowedError( + column=leaf, + agg=effective, + reason=( + f"column {leaf!r} restricts allowed_aggregations to " + f"{sorted(col.allowed_aggregations)}; got {effective!r}." + ), + ) + return effective + # Model-custom aggregations are exempt from the type-default gate. + if effective in custom_names: + return effective + allowed = DEFAULT_AGGREGATIONS_BY_TYPE.get(col.type, frozenset()) + if effective not in allowed: + raise AggregationNotAllowedError( + column=leaf, + agg=effective, + reason=( + f"aggregation {effective!r} is not applicable to " + f"{col.type} column {leaf!r}; default aggregations are " + f"{sorted(allowed)}." + ), + ) + return effective + + +def _bind_agg_arg( + parsed: ParsedExpr, *, + scope: Union[ModelScope, StageSchema], + bundle: ResolvedSourceBundle, +): + """Bind one positional / kwarg argument of an aggregation. + + The AggregateKey shape stores Scalars inline (not as LiteralKey) + so identity matches the spec — see ``slayer/core/keys.py``. + Identifier args become ``ColumnKey`` / ``ColumnSqlKey``; literal + args normalise via ``normalize_scalar``. + """ + if isinstance(parsed, Literal): + return normalize_scalar(parsed.value) + if isinstance(parsed, (Ref, DottedRef)): + return _bind(parsed, scope=scope, bundle=bundle, in_filter=False) + raise ValueError( + f"Aggregation argument of kind {type(parsed).__name__} is not " + f"supported. Pass a column reference or a scalar." + ) + + +_NOT_SCALAR = object() # sentinel returned by _fold_to_scalar when the input isn't a literal-resolvable scalar + + +def _fold_to_scalar(parsed: ParsedExpr): + """Resolve a parsed expression to a scalar literal if possible. + + Folds ``Literal`` directly, and unary ``-`` over a numeric ``Literal`` + (the AST shape Python emits for ``periods=-1``) into the negated + literal value. Returns ``_NOT_SCALAR`` for anything that doesn't + reduce — transform kwargs are typed as ``Scalar``, so a non-scalar + expression is a binding error. + """ + if isinstance(parsed, Literal): + return normalize_scalar(parsed.value) + if ( + isinstance(parsed, UnaryOp) + and parsed.op == "-" + and isinstance(parsed.operand, Literal) + ): + from decimal import Decimal + + inner = parsed.operand.value + if isinstance(inner, bool): + # Reject explicitly — ``-True`` is nonsense and bool is an + # int subclass that would otherwise pass the next branch. + return _NOT_SCALAR + if isinstance(inner, (int, float, Decimal)): + return normalize_scalar(-inner) + return _NOT_SCALAR + + +# Per-op kwarg whitelist for the typed pipeline. Broader than the legacy +# ``slayer.core.formula._ALLOWED_TRANSFORM_KWARGS`` because the new +# pipeline allows ``partition_by`` on more than just the rank family +# (DEV-1450 C6: ``change(measure, partition_by=...)`` threads through to +# the desugared time_shift). Every transform also implicitly accepts +# ``partition_by`` — that branch is handled before the whitelist check. +_TRANSFORM_KWARG_RULES: dict = { + "cumsum": frozenset(), + "change": frozenset(), + "change_pct": frozenset(), + "first": frozenset(), + "last": frozenset(), + "time_shift": frozenset({"periods", "granularity"}), + "lag": frozenset({"periods"}), + "lead": frozenset({"periods"}), + "rank": frozenset(), + "percent_rank": frozenset(), + "dense_rank": frozenset(), + "ntile": frozenset({"n"}), + "consecutive_periods": frozenset({"period"}), +} + +# Positional-parameter signature (after the value) for the transforms whose +# documented DSL form accepts positional args: ``time_shift(x, periods, +# granularity)``, ``lag(x, periods)``, ``lead(x, periods)``. Each name maps the +# i-th positional onto the matching kwarg. Transforms absent here are +# keyword-only after the value. +_TRANSFORM_POSITIONAL_KWARGS: dict = { + "time_shift": ("periods", "granularity"), + "lag": ("periods",), + "lead": ("periods",), +} + + +def _bind_transform( + parsed: TransformCall, *, + scope: Union[ModelScope, StageSchema], + bundle: ResolvedSourceBundle, + alias_map: Optional[Dict[str, "ValueKey"]] = None, +) -> TransformKey: + # ``alias_map`` lets a transform input reference a declared-measure + # alias inside a filter (``change(rev) > 0``); partition_by must still + # be a real column, so it is bound without the alias map. + inp = _bind( + parsed.input, scope=scope, bundle=bundle, in_filter=False, + alias_map=alias_map, + ) + # The value to transform is the first positional (``parsed.input``). + # A few transforms accept further POSITIONAL params per the documented + # DSL surface (``time_shift(x, periods, granularity)``, + # ``lag(x, periods)``, ``lead(x, periods)``); map those onto their kwarg + # names. Every other transform (rank family, cumsum, change, + # consecutive_periods, ...) stays keyword-only after the value. + positional_pairs: List = [] + pos_names = _TRANSFORM_POSITIONAL_KWARGS.get(parsed.op) + if parsed.args: + if pos_names is None: + raise ValueError( + f"Transform {parsed.op!r} accepts exactly one positional " + f"argument (the value to transform); pass any offset, " + f"partition, or other settings as keyword arguments " + f"(e.g. ``{parsed.op}(value, partition_by=...)``)." + ) + if len(parsed.args) > len(pos_names): + raise ValueError( + f"Transform {parsed.op!r} accepts at most {len(pos_names)} " + f"positional argument(s) after the value " + f"({', '.join(pos_names)}); got {len(parsed.args)}." + ) + positional_pairs = list(zip(pos_names, parsed.args)) + args: List = [] + kwargs: List = [] + partition_keys: List = [] + allowed_kwargs = _TRANSFORM_KWARG_RULES.get(parsed.op, frozenset()) + seen_kwargs: set = set() + # Positional params first, then explicit kwargs; a name supplied BOTH + # ways is an error (ambiguous, e.g. ``time_shift(x, -1, periods=-2)``). + _explicit_kw_names = {k for k, _ in parsed.kwargs} + for k, _ in positional_pairs: + if k in _explicit_kw_names: + raise ValueError( + f"Transform {parsed.op!r} got {k!r} both positionally and " + f"as a keyword argument." + ) + for k, v in [*positional_pairs, *parsed.kwargs]: + if k == "partition_by": + # ``partition_by`` accepts a single column ref OR a tuple/list of + # them (Codex review): ``rank(x, partition_by=[region, channel])``. + # ``_convert_kwarg_value`` returns a Python tuple for the list + # form; bind each element independently and accumulate into + # ``partition_keys`` so the SQL gen emits a multi-column OVER + # (PARTITION BY ...). A single ref still flows through the + # scalar branch. + elements = v if isinstance(v, tuple) else (v,) + for elem in elements: + bound_elem = _bind( + elem, scope=scope, bundle=bundle, in_filter=False, + ) + if isinstance(bound_elem, (ColumnKey, ColumnSqlKey)): + partition_keys.append(bound_elem) + else: + raise ValueError( + f"transform {parsed.op!r} partition_by must resolve " + f"to a column reference; got " + f"{type(bound_elem).__name__}." + ) + continue + if k not in allowed_kwargs: + raise ValueError( + f"Transform {parsed.op!r} does not accept keyword " + f"argument {k!r}. Accepted: " + f"{sorted(allowed_kwargs | {'partition_by'})}." + ) + seen_kwargs.add(k) + scalar = _fold_to_scalar(v) + if scalar is _NOT_SCALAR: + raise ValueError( + f"Transform {parsed.op!r} keyword {k!r} must be a " + f"scalar literal; got expression of kind " + f"{type(v).__name__}." + ) + kwargs.append((k, scalar)) + # Per-op required-kwarg validation + defaults. + kwargs = _apply_transform_kwarg_defaults( + op=parsed.op, kwargs=kwargs, seen=seen_kwargs, + ) + return TransformKey( + op=parsed.op, + input=inp, + args=tuple(args), + kwargs=tuple(kwargs), + partition_keys=frozenset(partition_keys), + ) + + +def _apply_transform_kwarg_defaults( + *, op: str, kwargs: list, seen: set, +) -> list: + """Validate required kwargs and apply per-op defaults for the typed + TransformKey. + + Validation: + * ``ntile`` requires ``n``; ``n`` must be a positive integer + (``bool`` rejected — it's an ``int`` subclass in Python but a + boolean ``True``/``False`` is never a sensible bucket count). + * ``time_shift`` requires ``periods`` (integer; may be negative). + + Defaults: + * ``lag`` / ``lead`` default ``periods=1`` when missing so the + typed TransformKey carries the resolved kwarg list; the SQL + generator can render PARTITION/ORDER without re-applying defaults. + + ``normalize_scalar`` wraps numeric literals in ``Decimal``, so the + integer checks accept ``Decimal`` whose value is integral as well + as plain ``int``. + """ + from decimal import Decimal + + def _ensure_positive_integer(value: object, *, kw: str) -> None: + if isinstance(value, bool): + raise ValueError( + f"Transform {op!r} keyword {kw} must be a positive " + f"integer; got {value!r}." + ) + if isinstance(value, int): + ival = value + elif isinstance(value, Decimal): + if value != value.to_integral_value(): + raise ValueError( + f"Transform {op!r} keyword {kw} must be a positive " + f"integer; got {value!r}." + ) + ival = int(value) + else: + raise ValueError( + f"Transform {op!r} keyword {kw} must be a positive " + f"integer; got {value!r}." + ) + if ival <= 0: + raise ValueError( + f"Transform {op!r} keyword {kw} must be a positive " + f"integer; got {value!r}." + ) + + if op == "ntile": + if "n" not in seen: + raise ValueError( + "Transform 'ntile' requires keyword argument n (the " + "number of buckets, a positive integer)." + ) + n_value = next(v for k, v in kwargs if k == "n") + _ensure_positive_integer(n_value, kw="n") + if op == "time_shift" and "periods" not in seen: + raise ValueError( + "Transform 'time_shift' requires keyword argument periods " + "(the integer offset, negative for a backward shift)." + ) + if op in ("lag", "lead") and "periods" not in seen: + kwargs.append(("periods", normalize_scalar(1))) + return kwargs + + +def _bind_scalar( + parsed: ScalarCall, *, + scope: Union[ModelScope, StageSchema], + bundle: ResolvedSourceBundle, + in_filter: bool, + alias_map: Optional[Dict[str, "ValueKey"]] = None, +) -> ScalarCallKey: + if parsed.name not in SCALAR_FUNCTIONS: + # Defence in depth: the parser already enforces the allowlist, + # but direct ParsedExpr construction can bypass the parser. + # Re-check here so the typed key family is always sound. + raise UnknownFunctionError( + name=parsed.name, + location="(binder)", + suggestion=( + f"Mode-B scalar calls are restricted to " + f"{sorted(SCALAR_FUNCTIONS)}." + ), + ) + if parsed.name == "like" and len(parsed.args) != 2: + raise ValueError( + f"Scalar function 'like' takes exactly 2 arguments " + f"(value, pattern); got {len(parsed.args)}." + ) + args = tuple( + _bind(a, scope=scope, bundle=bundle, in_filter=in_filter, alias_map=alias_map) + for a in parsed.args + ) + return ScalarCallKey(name=parsed.name, args=args) + + +def _reject_windowed_column_sql( + refs: Tuple[ValueKey, ...], + *, + scope: Union[ModelScope, StageSchema], + bundle: ResolvedSourceBundle, + parsed: ParsedExpr, +) -> None: + """Raise ``IllegalWindowInFilterError`` if any referenced + ``ColumnSqlKey`` has a windowed ``Column.sql`` body. + + DEV-1369 removed predicate-promotion; filters touching a windowed + column SQL now raise. + """ + if isinstance(scope, StageSchema): + # StageSchema columns don't carry Column.sql in the bundle; + # window detection is handled when the upstream stage was bound. + return + for k in refs: + if not isinstance(k, ColumnSqlKey): + continue + model = _lookup_model(name=k.model, scope=scope, bundle=bundle) + if model is None: + continue + col = next((c for c in model.columns if c.name == k.column_name), None) + if col is None or col.sql is None: + continue + if has_window_function(col.sql): + raise IllegalWindowInFilterError( + filter_expr=str(parsed), + source=( + f"filter references column {k.column_name!r} on model " + f"{k.model!r} whose Column.sql contains a window " + f"function" + ), + suggestion=( + "use a rank-family transform (rank, percent_rank, " + "dense_rank, ntile) in the formula instead, or " + "compute the windowed value in an earlier stage." + ), + ) + + +def _lookup_model( + *, + name: str, + scope: Union[ModelScope, StageSchema], + bundle: ResolvedSourceBundle, +) -> Optional[SlayerModel]: + if isinstance(scope, ModelScope) and scope.source_model is not None: + if scope.source_model.name == name: + return scope.source_model + return bundle.get_referenced_model(name) diff --git a/slayer/engine/column_dependency.py b/slayer/engine/column_dependency.py index 3211c2f5..3008b505 100644 --- a/slayer/engine/column_dependency.py +++ b/slayer/engine/column_dependency.py @@ -48,7 +48,7 @@ def _resolve_target_for_ref( """Return the model that a column reference resolves to, or ``None``. Mirrors the runtime alias resolution in - :func:`slayer.engine.column_expansion._walk_path_to_target` so the + :func:`slayer.engine.column_expansion._walk_path_to_target_sync` so the save-time validator and the compile-time expander agree on which references count. ``table_alias`` may be: diff --git a/slayer/engine/column_expansion.py b/slayer/engine/column_expansion.py index 080b7b06..27b890e3 100644 --- a/slayer/engine/column_expansion.py +++ b/slayer/engine/column_expansion.py @@ -12,13 +12,13 @@ base-column references qualify to the canonical ``__``-delimited path alias. -The expansion runs in the enrichment phase, so the SQL generator never sees +The expansion runs during binding/planning, so the SQL generator never sees unresolved derived references. """ from __future__ import annotations -from typing import Any -from collections.abc import Awaitable, Callable +from collections.abc import Callable +from typing import List, Optional, Protocol, Set, Tuple import sqlglot from sqlglot import exp @@ -26,9 +26,7 @@ from slayer.core.errors import ColumnCycleError from slayer.core.models import Column, SlayerModel -from slayer.sql.reserved_keywords import prequote_reserved_identifiers -ResolveModel = Callable[..., Awaitable[SlayerModel | None]] def _is_trivial_base(*, column: Column) -> bool: @@ -105,37 +103,121 @@ def _root_scope_column_ids(*, parsed: exp.Expression) -> set[int]: return root_ids -async def _walk_path_to_target( +class _SyncBundle(Protocol): + """Minimal contract for ``collect_root_scope_joined_paths``'s bundle — + matches ``ResolvedSourceBundle.get_referenced_model``. Declared inline so + this helper stays import-free of the engine layer. + """ + + def get_referenced_model(self, name: str) -> Optional[SlayerModel]: ... + + +def _resolve_alias_to_join_segments( + *, + alias: str, + source_model: SlayerModel, + bundle: _SyncBundle, +) -> Optional[Tuple[str, ...]]: + """Walk the ``__``-segmented join alias against ``source_model``'s joins. + + Returns the segments tuple when EVERY hop resolves (so a prefix walk + can emit join-path tuples), or ``None`` when any hop fails — that + aborts the caller's emission for this column (CTE / subquery alias + or a spurious dotted ref). + """ + segments = tuple(alias.split("__")) + current = source_model + for seg in segments: + join = next( + (j for j in current.joins if j.target_model == seg), None, + ) + if join is None: + return None + nxt = bundle.get_referenced_model(seg) + if nxt is None: + return None + current = nxt + return segments + + +def collect_root_scope_joined_paths( + *, + parsed: exp.Expression, + source_model: SlayerModel, + source_relation: str, + bundle: _SyncBundle, +) -> List[Tuple[str, ...]]: + """Collect the ordered de-duplicated list of join-path prefixes a parsed + SQL fragment references in its root scope. + + Each ROOT-scope ``.`` whose ``alias`` is not the source + relation and fully resolves as a join walk on ``source_model`` + contributes its prefixes (``a__b`` yields ``("a",)`` AND ``("a", "b")``). + Aliases that don't resolve as a join walk (CTE / subquery aliases, + spurious dotted refs) are skipped, as are columns inside a nested + scope (subquery / set-op branch) — those belong to the inner rowset. + + Shared by the SQL generator (``SQLGenerator._joined_paths_in_sql``) and + the planner-side column filter discovery + (``slayer.engine.column_filter_paths._walk_root_scope_paths``) so the + two surfaces agree on what counts as "crosses a join." + """ + root_ids = _root_scope_column_ids(parsed=parsed) + seen: Set[Tuple[str, ...]] = set() + ordered: List[Tuple[str, ...]] = [] + anchor_aliases = (source_relation, source_model.name) + for col in parsed.find_all(exp.Column): + tbl = col.args.get("table") + if tbl is None or col.args.get("db") or col.args.get("catalog"): + continue + if id(col) not in root_ids: + continue + alias = tbl.name + if alias in anchor_aliases: + continue + segments = _resolve_alias_to_join_segments( + alias=alias, source_model=source_model, bundle=bundle, + ) + if segments is None: + continue + for i in range(1, len(segments) + 1): + prefix = segments[:i] + if prefix not in seen: + seen.add(prefix) + ordered.append(prefix) + return ordered + + +# --------------------------------------------------------------------------- +# Synchronous expansion (DEV-1450 typed pipeline) +# --------------------------------------------------------------------------- +# +# The generator runs synchronously over a ``ResolvedSourceBundle`` that has +# already loaded every referenced model (P11: storage consulted once, up +# front), so it expands derived refs through a *sync* model resolver. +# +# DEV-1485: the async twins (``_walk_path_to_target`` / ``_process_column_node`` +# / ``expand_derived_refs``) resolved join targets through ``storage.get_model`` +# for the legacy enrichment path and are deleted with it, as this comment always +# said they would be. Nothing awaits model resolution here any more. + +SyncResolveModel = Callable[[str], Optional[SlayerModel]] + + +def _walk_path_to_target_sync( *, source_model: SlayerModel, source_alias: str, table_alias: str, - resolve_model: ResolveModel, - named_queries: dict[str, Any], + resolve_model: SyncResolveModel, is_root: bool, -) -> tuple[SlayerModel | None, str | None]: - """Resolve a ``table_alias`` (e.g. ``B`` or ``B__C``) seen inside a - Column.sql to the terminal joined model and the canonical alias to use - in emitted SQL. - - The ``is_root`` flag captures whether ``source_model`` is the FROM root - of the outer query. When True, walked paths are emitted bare - (``"__".join(parts)``); when False, they are prefixed with - ``source_alias`` so a derived column on a joined model referencing a - further-joined model resolves to the right ``__``-delimited path - (e.g., walking ``C`` off source ``B`` reached from root via ``B`` → - canonical ``B__C``, not ``C``). Closes the alias-prefix bug raised on - PR #89. - - Returns ``(None, None)`` if the alias does not resolve as a join path — - in that case the caller should leave the reference untouched (it is - likely a CTE / sub-query alias the user wired up themselves). +) -> Tuple[Optional[SlayerModel], Optional[str]]: + """Walk a ``__``-delimited alias path to its terminal model. + + Returns ``(target_model, canonical_alias)``, or ``(None, None)`` when any + hop is unresolvable — an opaque alias (a CTE / subquery reference) is not + an error here, the caller leaves it untouched. """ - # DEV-1410: literal match against the host's FROM alias or model name - # comes FIRST, before any ``__`` splitting. ``alias_path`` is already a - # canonical ``__``-delimited path coming from the engine (e.g. - # ``"B__C"``) — splitting it would falsely treat it as a multi-hop - # walk and fail to resolve as the host. if table_alias == source_alias or table_alias == source_model.name: return source_model, source_alias parts = table_alias.split("__") if "__" in table_alias else [table_alias] @@ -144,7 +226,7 @@ async def _walk_path_to_target( join = next((j for j in current.joins if j.target_model == hop), None) if join is None: return None, None - nxt = await resolve_model(model_name=hop, named_queries=named_queries) + nxt = resolve_model(hop) if nxt is None: return None, None current = nxt @@ -153,169 +235,99 @@ async def _walk_path_to_target( return current, canonical -async def _process_column_node( +def _process_column_node_sync( *, col: exp.Column, model: SlayerModel, alias_path: str, - resolve_model: ResolveModel, - named_queries: dict[str, Any], + resolve_model: SyncResolveModel, dialect: str, - visited: tuple[tuple[str, str], ...], + visited: Tuple[Tuple[str, str], ...], is_root: bool, - root_scope_ids: set[int], + root_scope_ids: Set[int], ) -> None: - """Resolve one ``exp.Column`` node in the parsed AST, mutating it in - place. Encapsulates the multi-branch decision that drives expansion: - - - multi-part qualifier (``catalog.db.table.col``) → leave alone - - bare identifier → qualify to ``alias_path`` - - ``
.`` where the alias doesn't resolve as a join path - → leave alone (CTE / sub-query alias) - - ``
.`` where the target column is base → rewrite table - to the canonical alias - - ``
.`` where the target column is derived → recurse and - splice the expanded AST in (parenthesized for precedence safety) - - Cycle detection raises ``ValueError`` with the recursion chain. + """Expand one ``exp.Column`` node in place if it names a derived column. + + Returns ``True`` when the node was rewritten, ``False`` when it was left + alone (a physical column, or an unresolvable/opaque alias path). """ - # exp.Column may carry a multi-part qualifier (catalog.db.table.col). - # We treat anything beyond the immediate table identifier as outside - # SLayer's contract (the Column.sql convention is `.`). if col.args.get("db") or col.args.get("catalog"): return - table_id = col.args.get("table") col_name = col.name - - # DEV-1410: bare identifiers and qualified ``.`` refs flow - # through the SAME lookup. A bare ref is treated as if it had been - # written with the host alias — ``_walk_path_to_target`` returns - # ``(source_model, alias_path)`` for that single-part match, so the - # downstream derived-vs-base decision (and recursion) is shared. table_alias = table_id.name if table_id is not None else alias_path - target_model, canonical_alias = await _walk_path_to_target( + target_model, canonical_alias = _walk_path_to_target_sync( source_model=model, source_alias=alias_path, table_alias=table_alias, resolve_model=resolve_model, - named_queries=named_queries, is_root=is_root, ) if target_model is None or canonical_alias is None: - return # unknown alias — leave untouched - + return target_col = target_model.get_column(col_name) if target_col is None or _is_trivial_base(column=target_col): - # Base column or unknown identifier on a known target model: - # rewrite the table to the canonical alias and stop. col.set("table", exp.to_identifier(canonical_alias)) return - - # DEV-1410 scope guard: only inline derived-column bodies when the - # reference is in the ROOT scope of the parent fragment. Nested - # scopes (subqueries, set-op branches, VALUES, CTEs) can legitimately - # use the same identifier to mean an inner column of a different - # rowset — leave them alone. if id(col) not in root_scope_ids: return - - # Derived → recurse. Recursion stays "root" only when the target - # column lives on the same model (no alias change); a remote target - # descended via a path is by definition non-root, so its own walks - # must prefix the canonical alias. next_is_root = is_root and (target_model is model) key = (target_model.name, col_name) if key in visited: cycle_start = visited.index(key) cycle = (*visited[cycle_start:], key) raise ColumnCycleError(cycle=list(cycle)) - expanded_sql = await expand_derived_refs( + expanded_sql = expand_derived_refs_sync( sql=target_col.sql, model=target_model, alias_path=canonical_alias, resolve_model=resolve_model, - named_queries=named_queries, dialect=dialect, visited=(*visited, key), is_root=next_is_root, ) if expanded_sql is None: return - # Splice in, parenthesized so the surrounding expression's precedence - # is preserved. - # DEV-1686: quote bare reserved-word qualifiers/leaves (e.g. a derived - # column referencing a reserved joined model like ``grant.amount``) so the - # generated SQL parses. - expanded_ast = sqlglot.parse_one( - prequote_reserved_identifiers(sql=expanded_sql, dialect=dialect), dialect=dialect - ) + expanded_ast = sqlglot.parse_one(expanded_sql, dialect=dialect) col.replace(exp.Paren(this=expanded_ast)) -async def expand_derived_refs( +def expand_derived_refs_sync( *, - sql: str | None, + sql: Optional[str], model: SlayerModel, alias_path: str, - resolve_model: ResolveModel, - named_queries: dict[str, Any] | None = None, + resolve_model: SyncResolveModel, dialect: str, - visited: tuple[tuple[str, str], ...] | None = None, + visited: Optional[Tuple[Tuple[str, str], ...]] = None, is_root: bool = True, -) -> str | None: - """Recursively expand cross-model and local derived-column references - inside ``sql``. - - Args: - sql: The Column / measure SQL to expand. May be ``None`` — returned - unchanged. - model: The model whose join graph is the reference frame for - unprefixed and singly-prefixed identifiers in ``sql``. - alias_path: The alias prefix under which bare identifiers in ``sql`` - should be qualified — typically the FROM alias used for ``model`` - in the outer query (e.g., ``"orders"`` or ``"customers__regions"``). - resolve_model: Async callable ``(model_name=str, named_queries=...)`` - that returns a ``SlayerModel`` (or None). - named_queries: Pass-through context for ``resolve_model``. - dialect: sqlglot dialect for parse/emit. - visited: Ordered cycle-detection chain of ``(model_name, - column_name)`` tuples populated during recursion. Ordered - (not a set) so the cycle path in error messages reflects the - actual recursion order — frozenset iteration is randomized - via PYTHONHASHSEED. Callers leave as None. - - Raises: - ValueError: on a circular column-reference chain. +) -> Optional[str]: + """Inline every derived-column reference in ``sql`` to its definition. + + Recurses through chains (``A.ratio`` -> ``A.bar / B.foo_normalized`` -> + …) with cycle detection via ``visited``, raising ``ColumnCycleError`` on a + self-referential chain. + + ``resolve_model`` is a plain ``name -> Optional[SlayerModel]`` lookup + (typically ``bundle.get_referenced_model``); there is no + ``named_queries`` parameter because the bundle has already resolved the + full referenced-model set. """ if not sql: return sql visited = visited or () - named_queries = named_queries or {} - - # DEV-1686: prequote reserved-word qualifiers/leaves before parsing user - # ``Column.sql`` (may reference a reserved joined model, e.g. ``grant.x``). - parsed = sqlglot.parse_one( - prequote_reserved_identifiers(sql=sql, dialect=dialect), dialect=dialect - ) - # Materialize the columns first — we may mutate them in place via .replace(). + parsed = sqlglot.parse_one(sql, dialect=dialect) column_nodes = list(parsed.find_all(exp.Column)) - # DEV-1410: compute root-scope membership once. Derived-column inlining - # only applies to root-scope refs; nested scopes (subqueries, set ops, - # VALUES, CTEs) are left alone. root_scope_ids = _root_scope_column_ids(parsed=parsed) - for col in column_nodes: - await _process_column_node( + _process_column_node_sync( col=col, model=model, alias_path=alias_path, resolve_model=resolve_model, - named_queries=named_queries, dialect=dialect, visited=visited, is_root=is_root, root_scope_ids=root_scope_ids, ) - return parsed.sql(dialect=dialect) diff --git a/slayer/engine/column_filter_paths.py b/slayer/engine/column_filter_paths.py new file mode 100644 index 00000000..2aa2278e --- /dev/null +++ b/slayer/engine/column_filter_paths.py @@ -0,0 +1,286 @@ +"""DEV-1503 — planner-facing helper for ``Column.filter`` join-path discovery. + +When the binder constructs a ``SqlExprKey`` for an ``AggregateKey.column_filter_key``, +it computes the typed set of non-anchor join paths the filter touches and stamps +them on the key as ``SqlExprKey.referenced_join_paths``. The DEV-1503 trigger +predicate then reads this typed field instead of re-parsing SQL text at plan +time — preserving the typed pipeline's "no string rewriting after parse" P7 +boundary at the planner layer. + +The actual rewriting for RENDERING (inlining derived refs inside the +``SUM(CASE WHEN THEN col END)`` wrapper) still lives in +``slayer/sql/generator.py`` (DEV-1494's ``_expand_column_filter_sql``). The +helper here is a parallel structural-analysis pass that returns paths only, +shared by the binder. + +The discovery rules match the generator-side ones byte-for-byte: + +* Same-model bare refs ("status") qualify to the anchor relation — no path. +* Cross-model dotted refs ("loss_payment.has_flag") contribute ``("loss_payment",)``. +* Multi-hop ``__``-delimited refs ("loss_payment__claim.state") contribute + ``("loss_payment",)`` AND ``("loss_payment", "claim")``. +* Bare refs that name a non-trivial DERIVED column whose own ``Column.sql`` + crosses a join (``is_eu`` reaching ``customers.region``) expand to the + derived sql and contribute the expansion's paths. +* Refs inside nested scopes (subqueries, set-op branches) are ignored — + they belong to the inner rowset. +* Aliases that don't resolve as a join walk on the anchor model are + silently skipped — they may be CTE / subquery aliases out of scope. +""" + +from __future__ import annotations + +from typing import Optional, Tuple + +import sqlglot +from sqlglot import exp + +from slayer.core.models import Column, SlayerModel +from slayer.engine.column_expansion import ( + _is_trivial_base, + collect_root_scope_joined_paths, + expand_derived_refs_sync, +) +from slayer.engine.source_bundle import ResolvedSourceBundle + + +# Fallback dialect chain (Codex round 7) — the planner doesn't carry the +# datasource's dialect, so a user-configured backend with dialect-specific +# syntax in ``Column.filter`` (MySQL backticks, T-SQL square brackets, +# ClickHouse-specific functions) could fail the Postgres parse and +# silently return no referenced join paths. The DEV-1503 trigger would +# then miss and the generator would render the filter inline in ``_base``, +# pulling the cross-model join into the host rowset. Try each dialect in +# order; only return ``()`` if ALL fail. The path discovery itself is +# dialect-agnostic (AST walking, no SQL emission); the dialect-aware +# re-parse for emission still happens on the generator side. +_PLANNER_PARSE_DIALECT_CHAIN: Tuple[Optional[str], ...] = ( + "postgres", + None, # sqlglot's permissive default — accepts ANSI SQL broadly + "mysql", + "clickhouse", # ClickHouse-only constructs (countIf, distinct identifier escapes) — CR PR #153 r3350000228 + "bigquery", + "tsql", +) + + +def _parse_filter_sql_any_dialect(sql: str) -> Optional[exp.Expression]: + """Parse ``sql`` trying each dialect in the fallback chain. + + Returns the first successful parse, or ``None`` when every dialect + rejects the input — that fall-through preserves the catch-all the + original ``except Exception`` provided for malicious / unparseable + payloads (the generator's dialect-aware emission is the + authoritative gate). + """ + for dialect in _PLANNER_PARSE_DIALECT_CHAIN: + try: + return sqlglot.parse_one(sql, dialect=dialect) + except Exception: + continue + return None + + +def _expand_derived_refs_any_dialect( + *, + sql: str, + model: SlayerModel, + alias_path: str, + bundle: ResolvedSourceBundle, +) -> Optional[str]: + """Run ``expand_derived_refs_sync`` against each dialect in the chain. + + ``expand_derived_refs_sync`` parses ``sql`` (and the derived columns' + own ``sql`` fields) internally with the supplied dialect. If a + derived column whose ``Column.sql`` uses dialect-specific syntax + (MySQL backticks, BigQuery struct literals, etc.) tips over the + Postgres parser, the expansion would silently drop join paths the + DEV-1503 trigger needs (Codex round 9). Try the same chain + ``_parse_filter_sql_any_dialect`` uses; return the first + successful expansion or ``None`` if every dialect fails. + """ + for dialect in _PLANNER_PARSE_DIALECT_CHAIN: + if dialect is None: + # ``expand_derived_refs_sync`` requires a dialect string. + continue + try: + expanded = expand_derived_refs_sync( + sql=sql, + model=model, + alias_path=alias_path, + resolve_model=bundle.get_referenced_model, + dialect=dialect, + ) + except Exception: + continue + if expanded: + return expanded + return None + + +def _is_nontrivial_derived(model: SlayerModel, name: str) -> bool: + """True iff ``name`` is a column on ``model`` whose ``Column.sql`` is a + non-trivial expression (set, and not just a bare-identifier remap). + + Mirrors ``SQLGenerator._is_nontrivial_derived``; duplicated here to keep + the planner-facing helper free of generator imports. + """ + col: Optional[Column] = next( + (c for c in model.columns if c.name == name), None, + ) + return col is not None and col.sql is not None and not _is_trivial_base( + column=col, + ) + + +def _is_anchor_local_col_ref( + col: exp.Column, *, anchor_aliases: set, +) -> bool: + """A column ref counts as "on the anchor" when it is bare (no ``table``) + OR self-qualified to the anchor (``orders.is_eu`` where ``orders`` is + the anchor relation / model name). + """ + if not isinstance(col.this, exp.Identifier): + return False + tbl = col.args.get("table") + if tbl is None: + return True + return tbl.name in anchor_aliases + + +def _expand_filter_sql_if_anchor_derived( + *, + parsed: exp.Expression, + canonical_sql: str, + anchor_model: SlayerModel, + anchor_relation: str, + bundle: ResolvedSourceBundle, +) -> Optional[exp.Expression]: + """Expand any non-trivial derived anchor-local refs in ``parsed``, + re-parsing the expanded SQL. Returns the new AST, the original + ``parsed`` when no derived ref was present, or ``None`` when the + expansion produced unparseable output (caller should bail to ``()``). + + Mirrors the generator's ``_expand_column_filter_sql`` gate so the + planner and the renderer surface the same set of crossed joins. + """ + anchor_aliases = {anchor_relation, anchor_model.name} + anchor_local_names = { + col.this.name + for col in parsed.find_all(exp.Column) + if _is_anchor_local_col_ref(col, anchor_aliases=anchor_aliases) + } + has_derived = any( + _is_nontrivial_derived(anchor_model, n) for n in anchor_local_names + ) + if not has_derived: + return parsed + + # Degenerate: the whole predicate IS a single derived column ref + # (``filter="is_eu"`` or self-qualified ``filter="orders.is_eu"``). + # ``expand_derived_refs_sync`` rewrites refs via in-place + # ``col.replace`` which is a no-op on the AST root, so expand the + # column's sql directly. + if ( + isinstance(parsed, exp.Column) + and _is_anchor_local_col_ref(parsed, anchor_aliases=anchor_aliases) + and _is_nontrivial_derived(anchor_model, parsed.name) + ): + col = next( + (c for c in anchor_model.columns if c.name == parsed.name), None, + ) + if col is None or not col.sql: + return None + sql_to_expand = col.sql + else: + sql_to_expand = canonical_sql + + expanded = _expand_derived_refs_any_dialect( + sql=sql_to_expand, + model=anchor_model, + alias_path=anchor_relation, + bundle=bundle, + ) + if not expanded: + return parsed + return _parse_filter_sql_any_dialect(expanded) + + +def compute_column_filter_join_paths( + *, + canonical_sql: Optional[str], + anchor_model: SlayerModel, + anchor_relation: str, + bundle: ResolvedSourceBundle, +) -> Tuple[Tuple[str, ...], ...]: + """Return the ordered tuple of non-anchor join-path prefixes a + ``Column.filter`` predicate touches after derived-ref expansion. + + ``anchor_model`` is the model the filter is bound against — for a + filtered-local measure on the host (``AggregateKey.source.path == ()``), + the anchor is the host; for a cross-model aggregate on a target column + (``source.path == ("customers",)``), the anchor is ``customers``. + + Returns ``()`` for same-model filters (no cross-anchor joins), for an + empty / unparseable canonical_sql, and as a defensive fallback if join + resolution fails partway through. + + Multi-hop alias paths emit each prefix once (``loss_payment__claim`` + yields ``("loss_payment",)`` and ``("loss_payment", "claim")``). + """ + if not canonical_sql: + return () + parsed = _parse_filter_sql_any_dialect(canonical_sql) + if parsed is None: + return () + + expanded = _expand_filter_sql_if_anchor_derived( + parsed=parsed, + canonical_sql=canonical_sql, + anchor_model=anchor_model, + anchor_relation=anchor_relation, + bundle=bundle, + ) + if expanded is None: + return () + parsed = expanded + + # ``_walk_root_scope_paths`` exercises sqlglot's scope analyser, which + # can raise ``TypeError`` etc. on unusual / malicious payloads (SQL + # injection attempts like ``status = 'x' UNION SELECT * FROM users``). + # The dialect-aware generator path is the authoritative gate for those — + # it raises ``ParseError`` / ``ValueError`` at SQL emission. The planner + # discovery here is a best-effort structural pass; swallow any internal + # parser failure so it can't shadow the generator's rejection. + try: + return _walk_root_scope_paths( + parsed=parsed, + anchor_model=anchor_model, + anchor_relation=anchor_relation, + bundle=bundle, + ) + except Exception: + return () + + +def _walk_root_scope_paths( + *, + parsed: exp.Expression, + anchor_model: SlayerModel, + anchor_relation: str, + bundle: ResolvedSourceBundle, +) -> Tuple[Tuple[str, ...], ...]: + """Collect every root-scope ``.`` whose ``alias`` resolves as + a join walk on ``anchor_model``, returning the ordered tuple of path + prefixes (de-duplicated). + + Thin shim over the shared ``collect_root_scope_joined_paths`` helper so + the planner and ``SQLGenerator._joined_paths_in_sql`` agree on what + counts as "crosses a join." + """ + return tuple(collect_root_scope_joined_paths( + parsed=parsed, + source_model=anchor_model, + source_relation=anchor_relation, + bundle=bundle, + )) diff --git a/slayer/engine/cross_model_planner.py b/slayer/engine/cross_model_planner.py new file mode 100644 index 00000000..15ff9d8e --- /dev/null +++ b/slayer/engine/cross_model_planner.py @@ -0,0 +1,1204 @@ +"""Stage 7a.2 (DEV-1450) — CrossModelPlanner Protocol + IsolatedCte impl (I1). + +The cross-model aggregate strategy is a substitutable component (I1): +``CrossModelPlanner`` is a Protocol; ``IsolatedCteCrossModelPlanner`` is +the default impl encoding today's "one CTE per (target_model, +shared_grain)" pattern and the ``inherited_filter_policy`` decision +table from the DEV-1450 spec. + +The Protocol's ``plan(...)`` consumes: + +* the aggregate slot id + ``AggregateKey`` (whose ``source.path`` + identifies the cross-model target), +* a ``ResolvedSourceBundle`` (the eagerly-resolved model graph), +* ``host_slots`` (every ``ValueSlot`` on the host query — used to + classify filter routing and compute shared grain), +* ``host_filters`` as ``HostFilterRouting`` records (filter id + + phase + referenced slot ids). + +It produces a ``CrossModelAggregatePlan`` (in ``planned.py``) with +explicit ``where_filter_ids`` / ``having_filter_ids`` / +``target_model_filters`` routes so the SQL generator (stage 7b) doesn't +re-classify. + +Decision table (host filter routing only): + +| Filter references | Route | +| -------------------------------------------- | ---------------------- | +| Host-local row slot only | DROP_HOST_LOCAL | +| All on joined-target path (row) | PROPAGATE_WHERE | +| Cross-model agg-ref on same target | PROPAGATE_HAVING | +| Slots on a different joined branch | DROP_UNREACHABLE | +| Mixed reachable + unreachable | DROP_UNREACHABLE | +| Transform / POST phase | STAY_AT_HOST_POST | + +Target model's own ``SlayerModel.filters`` and ``Column.filter`` on the +aggregated column are intrinsic — they ride on the target / the +``AggregateKey`` itself and don't go through host-filter classification. + +Dormant in 7a — no engine code calls these yet. ProjectionPlanner +(stage 7a.6) is the first consumer. +""" + +from __future__ import annotations + +from enum import Enum +from typing import Callable, List, Optional, Protocol, Tuple + +from pydantic import BaseModel, ConfigDict, Field + +from slayer.core.enums import DataType +from slayer.core.errors import ( + AmbiguousReferenceError, + IllegalScopeReferenceError, + UnknownReferenceError, + UnreachableFilterDroppedWarning, +) +from slayer.core.keys import ( + AggregateKey, + ColumnKey, + ColumnSqlKey, + Phase, + StarKey, + TimeTruncKey, + ValueKey, + column_path, + reroot_aggregate_key, +) +from slayer.core.models import ModelMeasure, SlayerModel +from slayer.core.query import ColumnRef, SlayerQuery, TimeDimension +from slayer.core.refs import agg_kwarg_canonical_str, canonical_agg_name +from slayer.core.scope import ModelScope, StageColumn, StageSchema +from slayer.engine.aggregate_input_paths import ( + compute_aggregate_input_join_paths, +) +from slayer.engine.binding import ( + bind_expr, + bind_filter, + bind_time_dimension, + walk_value_keys, +) +from slayer.engine.planned import ( + BoundFilterId, + CrossModelAggregatePlan, + JoinRequirement, + PlannedQuery, + SlotId, + ValueSlot, +) +from slayer.engine.source_bundle import ResolvedSourceBundle +from slayer.engine.syntax import parse_expr, parse_filter_expr + + +# --------------------------------------------------------------------------- +# Public types +# --------------------------------------------------------------------------- + + +class FilterRoute(str, Enum): + """Routing decision for one host filter on a cross-model CTE.""" + + DROP_HOST_LOCAL = "drop_host_local" + PROPAGATE_WHERE = "propagate_where" + PROPAGATE_HAVING = "propagate_having" + DROP_UNREACHABLE = "drop_unreachable" + STAY_AT_HOST_POST = "stay_at_host_post" + + +class HostFilterRouting(BaseModel): + """A host filter + the slot ids it references. + + The planner consumes a list of these; each is classified per + ``classify_host_filter`` and routed into the resulting + ``CrossModelAggregatePlan``. + """ + + model_config = ConfigDict(arbitrary_types_allowed=True) + + filter_id: BoundFilterId + phase: Phase + referenced_slot_ids: List[SlotId] = Field(default_factory=list) + text: Optional[str] = None + + +# --------------------------------------------------------------------------- +# Classifier +# --------------------------------------------------------------------------- + + +def classify_host_filter( + *, + host_filter: HostFilterRouting, + host_slots: List[ValueSlot], + target_path: Tuple[str, ...], + host_model_name: Optional[str] = None, +) -> FilterRoute: + """Classify one host filter for cross-model CTE propagation. + + See the module docstring for the decision table. The classifier is + pure: same inputs → same output, no side effects. + + ``host_model_name`` is used to route ``ColumnSqlKey`` refs (derived + columns): the key carries its host model name but not a path, so we + compare it against the host model name and the path to decide + reachable / local / unreachable. When ``host_model_name`` is None, + ColumnSqlKey refs default to local — conservative for callers that + don't have the host model in scope. + """ + if host_filter.phase == Phase.POST: + return FilterRoute.STAY_AT_HOST_POST + if not host_filter.referenced_slot_ids: + # No referenced slots — nothing to route into the CTE. + return FilterRoute.STAY_AT_HOST_POST + + by_id = {s.id: s for s in host_slots} + + local_row: List[SlotId] = [] + reachable_path: List[SlotId] = [] + unreachable: List[SlotId] = [] + aggregate_on_target: List[SlotId] = [] + aggregate_other: List[SlotId] = [] + + for sid in host_filter.referenced_slot_ids: + s = by_id.get(sid) + if s is None: + # Unknown slot id — be conservative, treat as unreachable. + unreachable.append(sid) + continue + if isinstance(s.key, AggregateKey): + agg_source = s.key.source + agg_path = getattr(agg_source, "path", ()) + if agg_path == target_path: + aggregate_on_target.append(sid) + else: + aggregate_other.append(sid) + elif isinstance(s.key, ColumnKey): + if not s.key.path: + local_row.append(sid) + elif s.key.path == target_path[: len(s.key.path)]: + reachable_path.append(sid) + else: + unreachable.append(sid) + elif isinstance(s.key, ColumnSqlKey): + # Derived column. Route by its host model: on host → local; + # on any model in target_path → reachable; otherwise → + # unreachable. + cm = s.key.model + if host_model_name is not None and cm == host_model_name: + local_row.append(sid) + elif cm in target_path: + reachable_path.append(sid) + elif host_model_name is None: + # No host name to compare against — conservative default. + local_row.append(sid) + else: + unreachable.append(sid) + else: + # Transform / Arithmetic / ScalarCall: phase already decided. + # POST was checked above; ROW/AGGREGATE land here from + # arithmetic / scalar calls. Treat as local for routing. + local_row.append(sid) + + if unreachable or aggregate_other: + # Any unreachable ref → drop + warn (covers pure-unreachable AND + # mixed-with-reachable cases per decision table rows 6/7). + return FilterRoute.DROP_UNREACHABLE + if local_row and not (aggregate_on_target or reachable_path): + return FilterRoute.DROP_HOST_LOCAL + if local_row: + # Mixed local + (target-path / target-agg). The local refs can't + # be evaluated in the CTE, so the filter stays at host. + return FilterRoute.DROP_HOST_LOCAL + if aggregate_on_target: + return FilterRoute.PROPAGATE_HAVING + if reachable_path: + return FilterRoute.PROPAGATE_WHERE + return FilterRoute.STAY_AT_HOST_POST + + +# --------------------------------------------------------------------------- +# Protocol +# --------------------------------------------------------------------------- + + +class CrossModelPlanner(Protocol): + """Strategy for compiling one cross-model aggregate slot. + + DEV-1450 follow-up #2: re-rooting is owned by the strategy, not a + post-hoc mutation in ``plan_query``. When the host carries dimensions / + filters reachable from the target only by walking the TARGET's own join + graph (off the host→target forward path), the strategy may build a nested + re-rooted ``PlannedQuery`` and attach it to the returned plan. To do so it + needs the host query, its public projection, and a callback that compiles + a sub-query — all keyword-only and defaulting to ``None`` so direct + callers (and test doubles) that don't re-root keep working unchanged. + """ + + def plan( + self, + *, + aggregate_slot_id: SlotId, + aggregate_key: AggregateKey, + bundle: ResolvedSourceBundle, + host_slots: List[ValueSlot], + host_filters: List[HostFilterRouting], + public_alias: Optional[str] = None, + hidden: bool = False, + host_query: Optional[SlayerQuery] = None, + public_projection: Optional[List[SlotId]] = None, + subplan_builder: Optional[ + Callable[[SlayerQuery, ResolvedSourceBundle], PlannedQuery] + ] = None, + ) -> CrossModelAggregatePlan: + ... + + +# --------------------------------------------------------------------------- +# Default impl +# --------------------------------------------------------------------------- + + +def _walk_chain( + *, + host_model: SlayerModel, + hops: Tuple[str, ...], + bundle: ResolvedSourceBundle, +) -> Tuple[SlayerModel, List[JoinRequirement]]: + """Walk the join graph from ``host_model`` through ``hops``. + + Returns ``(terminal_model, [JoinRequirement, ...])``. Raises + ``ValueError`` if a hop has no matching join on the current model + or the referenced model isn't in ``bundle.referenced_models``. + + The walker is sync — the bundle holds eagerly-resolved models, so + no async I/O is needed (P11). + """ + current = host_model + chain: List[JoinRequirement] = [] + for hop in hops: + join = next( + (j for j in current.joins if j.target_model == hop), None, + ) + if join is None: + raise ValueError( + f"Model {current.name!r} has no join to {hop!r}. " + f"Available joins: {[j.target_model for j in current.joins]}" + ) + nxt = bundle.get_referenced_model(hop) + if nxt is None: + raise ValueError( + f"Join target {hop!r} from {current.name!r} not found in " + f"resolved source bundle." + ) + chain.append(JoinRequirement( + source_model=current.name, + target_model=hop, + join_pairs=[list(p) for p in join.join_pairs], + join_type=join.join_type, + )) + current = nxt + return current, chain + + +def _aggregate_alias(*, key: AggregateKey) -> str: + """Canonical alias for the aggregate's output column in the CTE. + + Mirrors the result-key contract: ``leaf`` + ``_`` + ``agg`` plus an + args/kwargs signature suffix that disambiguates parameterised + aggregates (``revenue:percentile(p=0.5)`` vs ``p=0.95``). The + ``*:count`` star form collapses to ``_count``. + + Built on ``slayer.core.refs.canonical_agg_name`` so the signature + suffix matches the rest of the engine (legacy enrichment, search, + DBT converter). + """ + # ColumnKey -> leaf, ColumnSqlKey (derived agg source) -> column_name, + # StarKey -> "*" (CR / Codex: a derived source must alias as + # ``net_sum``, not ``_sum``). + measure_name = ( + getattr(key.source, "leaf", None) + or getattr(key.source, "column_name", None) + or "*" + ) + # AggregateKey.args / kwargs are normalised tuples of scalars / + # ColumnKey-shaped values; convert to the (List[str], + # Dict[str, Any]) shape ``canonical_agg_name`` expects. DEV-1450 + # stage 7b.13: route through ``agg_kwarg_canonical_str`` so a + # ColumnKey kwarg renders as ``leaf`` (or ``path.leaf`` for joined + # paths) instead of Pydantic-repr noise from naive ``str(v)``. + # The kwarg suffix is preserved here -- the legacy enrichment at + # ``query_engine.py:2160`` drops it, causing two parametric aggs + # with different kwargs to collide on CTE alias (legacy bug). + # The 7b.5 fix added kwarg-aware aliases here as a correctness + # improvement -- ``test_cross_model_planner_wiring.py:: + # test_parameterized_aggregates_get_distinct_cte_aliases`` pins + # this. Parity with legacy for cross-model parametric aggs is + # not achievable on this axis. + args_list = [agg_kwarg_canonical_str(a) for a in key.args] + kwargs_dict = {k: agg_kwarg_canonical_str(v) for k, v in key.kwargs} + return canonical_agg_name( + measure_name=measure_name, + aggregation_name=key.agg, + agg_args=args_list or None, + agg_kwargs=kwargs_dict or None, + ) + + +def _make_cte_schema( + *, + aggregate_owner: SlayerModel, + join_back_target_owner: SlayerModel, + aggregate_key: AggregateKey, + join_back_pairs: List[Tuple], +) -> StageSchema: + """Build the typed projection schema for the CTE. + + The CTE walks the join chain inside its body but groups at the + FIRST hop's target grain — so the projection's join-back keys are + columns on ``join_back_target_owner`` (the first hop's target model), + while the aggregate output column's type comes from + ``aggregate_owner`` (the terminal/aggregated model). + + For single-hop plans the two owners are the same model. For multi- + hop (``orders → customers → regions``), ``aggregate_owner`` is + ``regions`` and ``join_back_target_owner`` is ``customers``. + + Stage 7b's SQL generator consumes the schema when emitting the CTE + body and the join-back ON clause. + """ + columns: List[StageColumn] = [] + agg_alias = _aggregate_alias(key=aggregate_key) + src_leaf = ( + getattr(aggregate_key.source, "leaf", None) + or getattr(aggregate_key.source, "column_name", None) + ) + agg_type: Optional[DataType] = None + if src_leaf and hasattr(aggregate_owner, "get_column"): + src_col = aggregate_owner.get_column(src_leaf) + if src_col is not None: + agg_type = src_col.type + columns.append(StageColumn( + name=agg_alias, + sql_alias=agg_alias, + public_alias=None, + hidden=True, + type=agg_type or DataType.DOUBLE, + provenance=f"agg:{aggregate_key.agg}", + )) + for _, target_key in join_back_pairs: + leaf = getattr(target_key, "leaf", None) + if leaf is None: + continue + if any(c.name == leaf for c in columns): + continue + target_col = ( + join_back_target_owner.get_column(leaf) + if hasattr(join_back_target_owner, "get_column") else None + ) + col_type = target_col.type if target_col is not None else None + columns.append(StageColumn( + name=leaf, + sql_alias=leaf, + public_alias=None, + hidden=True, + type=col_type, + provenance="join_back_key", + )) + return StageSchema( + relation_name=f"cm_{aggregate_owner.name}", + columns=columns, + ) + + +def _match_filtered_local_grain_pairs( + *, + host_slots: List[ValueSlot], + public_projection: List[SlotId], + sub_plan: PlannedQuery, +) -> List[Tuple[SlotId, SlotId]]: + """Pair each host dimension / time-dimension slot with the sub-plan's + corresponding row slot for the LEFT JOIN ON clause. + + Both plans bind against the SAME underlying column on the host model, + so slot identity (the ValueKey) matches across plans. + """ + sub_row_by_key = {s.key: s.id for s in sub_plan.row_slots} + grain_pairs: List[Tuple[SlotId, SlotId]] = [] + for host_sid in public_projection: + host_slot = next( + (s for s in host_slots if s.id == host_sid), None, + ) + if host_slot is None: + continue + sub_sid = sub_row_by_key.get(host_slot.key) + if sub_sid is not None: + grain_pairs.append((host_sid, sub_sid)) + return grain_pairs + + +def _find_filtered_local_sub_agg_slot( + *, + sub_plan: PlannedQuery, + formula: str, + host_model: SlayerModel, +) -> SlotId: + """Locate the sub-plan's single local aggregate slot. + + Recursion suppression guarantees no nested cross-model plans so the + sub-plan has exactly one local aggregate — the filtered measure being + isolated. + """ + for s in sub_plan.aggregate_slots: + if isinstance(s.key, AggregateKey) and not getattr( + s.key.source, "path", (), + ): + return s.id + raise ValueError( + "DEV-1503 sub-plan produced no local aggregate slot for " + f"{formula!r} on {host_model.name!r} — planner bug." + ) + + +def _build_filtered_local_cte_schema( + *, + aggregate_key: AggregateKey, + host_model: SlayerModel, +) -> StageSchema: + """Build the minimal CTE schema for a filtered-local plan. + + The actual CTE columns are derived from the sub-plan's stage_schema / + projection at render time; this entry exists so external consumers see + a schema shape that matches the existing CrossModelAggregatePlan + contract. + """ + agg_alias = _aggregate_alias(key=aggregate_key) + leaf = getattr(aggregate_key.source, "leaf", None) or getattr( + aggregate_key.source, "column_name", None, + ) + agg_type: Optional[DataType] = None + if leaf is not None and hasattr(host_model, "get_column"): + col = host_model.get_column(leaf) + if col is not None: + agg_type = col.type + return StageSchema( + relation_name=f"cm_{host_model.name}", + columns=[StageColumn( + name=agg_alias, + sql_alias=agg_alias, + public_alias=None, + hidden=True, + type=agg_type or DataType.DOUBLE, + provenance=f"agg:{aggregate_key.agg}", + )], + ) + + +def _classify_subplan_filters( + *, + host_filters: List[HostFilterRouting], +) -> Optional[List[str]]: + """Decide which host-query filters propagate into the DEV-1503 sub-plan. + + ROW: pass through — the sub-plan applies them to the aggregate's rowset + (otherwise a non-dim filter like ``status = 'active'`` has no effect on + the join-back aggregate value). + AGGREGATE (any slot ref): skip. Pushing such a filter into the sub-plan + as HAVING would drop CTE rows where the aggregate fails the test; the + outer LEFT JOIN then surfaces the host row with a NULL aggregate + instead of dropping it — wrong semantics. The generator's outer-WHERE + wrapper applies the filter on the joined-back column so the row is + actually dropped (DEV-1503 spec). + POST: skip — stays at the existing host post-transform wrapper. + + Consume ``routing.text`` directly — it carries the original user-filter + string for user-filter routings (None for date_range bounds) and is + populated by ``stage_planner`` from the deduped ``bound_filters`` list, + so it stays in lock-step with ``host_filters`` even after Mode-B + dedup-by-bound-key collapses two textually-different filter spellings + onto one routing (CR PR #153 thread r3350000254). Slicing + ``host_query.filters`` here would mis-pair phases when a ``date_range``- + bearing time_dimension is present (Codex review) OR when dedup drops + user-filter entries. + """ + sub_filter_texts: List[str] = [] + for routing in host_filters: + if routing.text is None: + # date_range bound — not a user filter, do not propagate. + continue + if routing.phase in (Phase.POST, Phase.AGGREGATE): + continue + # ROW phase — propagate. + sub_filter_texts.append(routing.text) + return sub_filter_texts or None + + +def _route_host_filters( + *, + host_filters: List[HostFilterRouting], + host_slots: List[ValueSlot], + target_path: Tuple[str, ...], + host_model: SlayerModel, + terminal_model: SlayerModel, +) -> Tuple[ + List[BoundFilterId], List[BoundFilterId], List[BoundFilterId], + List[UnreachableFilterDroppedWarning], +]: + """Classify each host filter via the ``inherited_filter_policy`` decision + table (``classify_host_filter``) into ``(applied, where_ids, having_ids, + dropped)`` — extracted from ``IsolatedCteCrossModelPlanner.plan`` (DEV-1708) + to keep that method focused. ``DROP_HOST_LOCAL`` / ``STAY_AT_HOST_POST`` are + neither propagated nor warned.""" + applied: List[BoundFilterId] = [] + where_ids: List[BoundFilterId] = [] + having_ids: List[BoundFilterId] = [] + dropped: List[UnreachableFilterDroppedWarning] = [] + for hf in host_filters: + route = classify_host_filter( + host_filter=hf, + host_slots=host_slots, + target_path=target_path, + host_model_name=host_model.name, + ) + if route is FilterRoute.PROPAGATE_WHERE: + where_ids.append(hf.filter_id) + applied.append(hf.filter_id) + elif route is FilterRoute.PROPAGATE_HAVING: + having_ids.append(hf.filter_id) + applied.append(hf.filter_id) + elif route is FilterRoute.DROP_UNREACHABLE: + dropped.append(UnreachableFilterDroppedWarning( + filter_text=hf.text or hf.filter_id, + reason=( + f"filter {hf.filter_id!r} references slot(s) outside " + f"the join path to {terminal_model.name!r}; " + f"unreachable filters are dropped." + ), + )) + return applied, where_ids, having_ids, dropped + + +def _compute_shared_grain_slots( + *, host_slots: List[ValueSlot], target_path: Tuple[str, ...], +) -> List[SlotId]: + """Host ROW slots (dimensions / time-dimensions) whose path lies on the + target's join chain flow through as the cross-model CTE's shared grain + (extracted from ``IsolatedCteCrossModelPlanner.plan`` — DEV-1708). Cross- + branch and aggregate/transform slots do not. + + A path-bearing **plain derived** (``ColumnSqlKey``) dimension on the target + path flows through identically to a base ``ColumnKey`` dim (DEV-1728): the + generator expands its ``Column.sql`` inside the ``_cm_*`` CTE, groups by it, + and joins back on the DOTTED host alias (the DEV-1708 raise is gone now that + DEV-1713 fixed the naming half). ``path == ()`` (a host-local derived dim) + still broadcasts by design — the generator's grain loop skips empty-path + slots — and a hidden filter-only derived ref is excluded there via the + ``base_projection_ids`` intersection, so no ``not s.hidden`` guard is needed. + """ + shared_grain: List[SlotId] = [] + for s in host_slots: + # Base and derived dims carry their path directly; a time dimension + # carries it on the wrapped column. One prefix test then serves all + # three kinds — the DEV-1728 merge of what were two identical branches. + if isinstance(s.key, (ColumnKey, ColumnSqlKey)): + p = s.key.path + elif isinstance(s.key, TimeTruncKey): + p = column_path(s.key.column) + else: + continue + if not p or p == target_path[: len(p)]: + shared_grain.append(s.id) + return shared_grain + + +class IsolatedCteCrossModelPlanner: + """Default impl — one CTE per (target_model, shared_grain) tuple. + + Encodes the ``inherited_filter_policy`` decision table from the + DEV-1450 spec via ``classify_host_filter`` for host filters; pulls + target ``SlayerModel.filters`` automatically. + """ + + def plan( + self, + *, + aggregate_slot_id: SlotId, + aggregate_key: AggregateKey, + bundle: ResolvedSourceBundle, + host_slots: List[ValueSlot], + host_filters: List[HostFilterRouting], + public_alias: Optional[str] = None, + hidden: bool = False, + host_query: Optional[SlayerQuery] = None, + public_projection: Optional[List[SlotId]] = None, + subplan_builder: Optional[ + Callable[[SlayerQuery, ResolvedSourceBundle], PlannedQuery] + ] = None, + ) -> CrossModelAggregatePlan: + host_model = bundle.source_model + if host_model is None: + raise ValueError( + "ResolvedSourceBundle.source_model is None — " + "IsolatedCteCrossModelPlanner needs a host model anchor " + "(I2 anchor-less mode is not yet implemented)." + ) + + agg_source = aggregate_key.source + path = getattr(agg_source, "path", ()) + if not path: + return self._dispatch_filtered_local( + aggregate_slot_id=aggregate_slot_id, + aggregate_key=aggregate_key, + bundle=bundle, + host_model=host_model, + host_slots=host_slots, + host_filters=host_filters, + public_alias=public_alias, + hidden=hidden, + host_query=host_query, + public_projection=public_projection, + subplan_builder=subplan_builder, + ) + + terminal_model, join_chain = _walk_chain( + host_model=host_model, hops=path, bundle=bundle, + ) + + # Build join_back_pairs from the FIRST hop's join_pairs. The CTE + # is grouped at the first hop's target columns; the host joins + # back on those. + join_back_pairs: List[Tuple] = [] + if join_chain: + first_hop = join_chain[0] + for pair in first_hop.join_pairs: + host_col, target_col = pair + join_back_pairs.append(( + ColumnKey(path=(), leaf=host_col), + ColumnKey(path=(), leaf=target_col), + )) + + target_path = path + applied, where_ids, having_ids, dropped = _route_host_filters( + host_filters=host_filters, + host_slots=host_slots, + target_path=target_path, + host_model=host_model, + terminal_model=terminal_model, + ) + + target_model_filters = list(terminal_model.filters or []) + + # Shared grain: host ROW dimensions / time-dimensions on the target's + # join chain flow through (a plain derived one on the target path + # raises — DEV-1708). Extracted to keep this method focused. + shared_grain = _compute_shared_grain_slots( + host_slots=host_slots, target_path=target_path, + ) + + first_hop = join_chain[0] + first_hop_target = ( + bundle.get_referenced_model(first_hop.target_model) + or terminal_model + ) + cte_schema = _make_cte_schema( + aggregate_owner=terminal_model, + join_back_target_owner=first_hop_target, + aggregate_key=aggregate_key, + join_back_pairs=join_back_pairs, + ) + + forward_plan = CrossModelAggregatePlan( + aggregate_slot_id=aggregate_slot_id, + target_model=terminal_model.name, + datasource=host_model.data_source, + join_chain=join_chain, + join_back_pairs=join_back_pairs, + cte_stage_schema=cte_schema, + shared_grain_slots=shared_grain, + applied_filter_ids=applied, + where_filter_ids=where_ids, + having_filter_ids=having_ids, + target_model_filters=target_model_filters, + dropped_filter_warnings=dropped, + hidden=hidden, + public_alias=public_alias, + ) + + # DEV-1450 #2: re-rooting is the strategy's call. When the caller + # supplies the host query + a sub-plan builder, decide forward-plan + # vs re-rooted-plan here; without them (direct ``plan(...)`` callers / + # test doubles) return the forward plan unchanged. + if subplan_builder is not None and host_query is not None: + return _maybe_reroot_cross_model_plan( + plan=forward_plan, + query=host_query, + agg_key=aggregate_key, + bundle=bundle, + host_model=host_model, + public_projection=public_projection or [], + subplan_builder=subplan_builder, + ) + return forward_plan + + # ---------------------------------------------------------------------- + # DEV-1503 — filtered-local isolation + # ---------------------------------------------------------------------- + + def _dispatch_filtered_local( + self, + *, + aggregate_slot_id: SlotId, + aggregate_key: AggregateKey, + bundle: ResolvedSourceBundle, + host_model: SlayerModel, + host_slots: List[ValueSlot], + host_filters: List[HostFilterRouting], + public_alias: Optional[str], + hidden: bool, + host_query: Optional[SlayerQuery], + public_projection: Optional[List[SlotId]], + subplan_builder: Optional[ + Callable[[SlayerQuery, ResolvedSourceBundle], PlannedQuery] + ], + ) -> CrossModelAggregatePlan: + """Validate the host-rooted trigger preconditions and dispatch + into ``_plan_filtered_local`` — the aggregate is on a HOST column + but at least one of its inputs crosses a join (``Column.filter`` + per DEV-1503; source ``Column.sql`` / positional args / kwargs per + DEV-1709), so a host-rooted nested sub-plan owns the aggregation + and the host base LEFT JOINs back. + """ + agg_source = aggregate_key.source + cfk = aggregate_key.column_filter_key + has_crossing_filter = cfk is not None and bool( + cfk.referenced_join_paths, + ) + has_crossing_input = has_crossing_filter or bool( + compute_aggregate_input_join_paths( + key=aggregate_key, + anchor_model=host_model, + anchor_relation=host_model.name, + bundle=bundle, + ), + ) + if not has_crossing_input: + raise ValueError( + f"AggregateKey on {agg_source!r} has empty source.path, " + f"no cross-model column_filter_key, AND no other crossing " + f"input — this is a plain local aggregate. The cross-model " + f"planner should not have been invoked." + ) + if subplan_builder is None or host_query is None: + # The DEV-1503 strategy requires a sub-plan builder + the host + # query for grain-pair matching. Direct callers without these + # (legacy test doubles) can't trigger filtered-local — raise + # loudly so the call site is fixed rather than emitting + # silently wrong SQL. + raise ValueError( + "DEV-1503 filtered-local isolation requires host_query " + "and subplan_builder; received None for one or both. " + "Confirm the stage_planner is wired to pass them." + ) + return self._plan_filtered_local( + aggregate_slot_id=aggregate_slot_id, + aggregate_key=aggregate_key, + bundle=bundle, + host_model=host_model, + host_slots=host_slots, + host_filters=host_filters, + host_query=host_query, + public_alias=public_alias, + public_projection=public_projection or [], + hidden=hidden, + subplan_builder=subplan_builder, + ) + + def _plan_filtered_local( + self, + *, + aggregate_slot_id: SlotId, + aggregate_key: AggregateKey, + bundle: ResolvedSourceBundle, + host_model: SlayerModel, + host_slots: List[ValueSlot], + host_filters: List[HostFilterRouting], + host_query: SlayerQuery, + public_alias: Optional[str], + public_projection: List[SlotId], + hidden: bool, + subplan_builder: Callable[ + [SlayerQuery, ResolvedSourceBundle], PlannedQuery, + ], + ) -> CrossModelAggregatePlan: + """Build a host-rooted nested sub-plan for a cross-model-FILTERED + local measure (DEV-1503). + + The sub-plan is a ``SlayerQuery`` rooted at the SAME host model with + ``measures=[]`` and the host's dimensions / + time_dimensions. The sub-plan's ``plan_query`` recursion handles + the filter-target join (its ``Column.filter`` will pull in the + joined table at the generator's inline path), the host model's own + ``SlayerModel.filters``, and the per-dimension GROUP BY — producing a + per-grain aggregate that the host base LEFT JOINs back. + + Host query filters are NOT propagated into the sub-plan here — the + host base CTE applies them. The generator's outer-WHERE wrapper + handles aggregate-referencing filters separately (DEV-1503 spec). + """ + # Reconstruct the local measure formula from the AggregateKey. The + # source.path is empty so ``_local_agg_formula`` emits a bare + # ``leaf:agg`` shape (plus any args / kwargs). Carry the user- + # supplied alias through so a host filter referencing the rename + # (``latest_pmt > 500`` for a measure named ``latest_pmt``) binds + # against the same alias in the sub-plan rather than the canonical + # ``latest_payment_last_updated_at`` form. + formula = _local_agg_formula(aggregate_key) + measure_name_for_subplan = public_alias + sub_filters = _classify_subplan_filters(host_filters=host_filters) + rerooted_query = SlayerQuery( + source_model=host_model.name, + measures=[ModelMeasure( + formula=formula, name=measure_name_for_subplan, + )], + dimensions=list(host_query.dimensions or []) or None, + time_dimensions=list(host_query.time_dimensions or []) or None, + filters=sub_filters, + ) + sub_plan = subplan_builder(rerooted_query, bundle) + + grain_pairs = _match_filtered_local_grain_pairs( + host_slots=host_slots, + public_projection=public_projection, + sub_plan=sub_plan, + ) + sub_agg_sid = _find_filtered_local_sub_agg_slot( + sub_plan=sub_plan, formula=formula, host_model=host_model, + ) + cte_schema = _build_filtered_local_cte_schema( + aggregate_key=aggregate_key, host_model=host_model, + ) + + return CrossModelAggregatePlan( + aggregate_slot_id=aggregate_slot_id, + # ``target_model`` is conventionally set to the host name for + # filtered-local; ``cte_root_model`` is the disambiguator the + # renderer reads. + target_model=host_model.name, + cte_root_model=host_model.name, + datasource=host_model.data_source, + join_chain=[], + join_back_pairs=[], + cte_stage_schema=cte_schema, + shared_grain_slots=[host_sid for host_sid, _ in grain_pairs], + applied_filter_ids=[], + where_filter_ids=[], + having_filter_ids=[], + target_model_filters=[], + dropped_filter_warnings=[], + hidden=hidden, + public_alias=public_alias, + rerooted_plan=sub_plan, + rerooted_grain_pairs=grain_pairs, + rerooted_agg_slot_id=sub_agg_sid, + ) + + +# --------------------------------------------------------------------------- +# Cross-model re-rooting (DEV-1450 stage 7b.15e, C1; relocated here in #2) +# --------------------------------------------------------------------------- +# +# When a cross-model aggregate (``policy_amount.total:sum``) is queried with +# host dimensions that are reachable from the TARGET by walking the target's +# own join graph (``policy_amount -> policy -> policy_number``), the +# forward-path CTE ("FROM bare target, GROUP BY forward-path dims only") +# collapses the host dimension to a scalar CROSS JOIN -- every host row gets +# the global aggregate. +# +# The fix mirrors legacy ``_build_rerooted_enriched``: build a full nested +# ``SlayerQuery`` rooted at the target (so all of the target's joins are in +# scope for dimensions AND filters), compile it via ``subplan_builder``, and +# attach the sub-plan to the ``CrossModelAggregatePlan``. The generator +# renders the sub-plan as the ``_cm_*`` CTE and joins it back to the host base +# on the (re-rooted) dimension. Dimensions / filters that don't resolve from +# the target are dropped -- matching legacy's drop-unreachable behaviour. +# +# DEV-1450 #2: this used to be a post-hoc pass in ``stage_planner.plan_query``; +# it now lives behind ``IsolatedCteCrossModelPlanner.plan`` so the +# render-strategy decision (forward vs re-rooted) is owned by the strategy. +# The recursive ``plan_query`` call is injected as ``subplan_builder`` so this +# module does not import ``stage_planner`` (no cycle). + + +def _reroot_ref( + *, model_prefix: Optional[str], name: str, host_model_name: str, + target_model_name: str, +) -> str: + """Re-root one Mode-B ref from the host's perspective to the target's. + + Mirrors legacy ``_build_rerooted_enriched``: + + * host-local (``model_prefix is None``) -> ``.`` (now a + cross-model dim from the target's view), + * on the target itself -> bare ```` (local on target), + * a path through the target -> strip the target prefix, + * any other dotted ref -> kept as-is (resolved via the target's joins). + """ + if model_prefix is None: + return f"{host_model_name}.{name}" + if model_prefix == target_model_name: + return name + if model_prefix.startswith(target_model_name + "."): + return f"{model_prefix[len(target_model_name) + 1:]}.{name}" + return f"{model_prefix}.{name}" + + +def _host_ref_path(model_prefix: Optional[str]) -> Tuple[str, ...]: + """The join path a host ColumnRef / TimeDimension prefix denotes.""" + if not model_prefix: + return () + return tuple(model_prefix.split(".")) + + +def _scalar_formula_literal(value) -> str: + """Render a normalized scalar back into formula text.""" + if isinstance(value, bool): + return "True" if value else "False" + if value is None: + return "None" + if isinstance(value, str): + return repr(value) + return str(value) + + +def _filter_ref_paths(value_key: ValueKey) -> List[Tuple[str, ...]]: + """Join paths of every column-like leaf a (bound) filter references.""" + paths: List[Tuple[str, ...]] = [] + for k in walk_value_keys(value_key): + if isinstance(k, (ColumnKey, ColumnSqlKey, StarKey)): + paths.append(tuple(k.path)) + elif isinstance(k, TimeTruncKey): + paths.append(tuple(column_path(k.column))) + return paths + + +def _render_ref_formula(ref) -> str: + """Render one already-rerooted embedded reference back into formula text. + + Column-like refs dot-join their (residual) path with the leaf; scalars + fall through to ``_scalar_formula_literal``. Contains NO path-stripping + decisions — the reroot has already happened (DEV-1707). + """ + if isinstance(ref, ColumnSqlKey): + return ".".join((*ref.path, ref.column_name)) + if isinstance(ref, ColumnKey): + return ".".join((*ref.path, ref.leaf)) + return _scalar_formula_literal(ref) + + +def _local_agg_formula(key: AggregateKey) -> str: + """Reconstruct the LOCAL colon-formula for a cross-model aggregate + (``customers.revenue:sum`` -> ``revenue:sum``) so it can be re-planned + against the target model as a plain local measure. + + Every embedded reference — source, positional args, and column-valued + kwargs — is re-anchored symmetrically via the unified + ``reroot_aggregate_key`` (DEV-1707), then rendered by the path-free + ``_render_ref_formula``. A kwarg / arg one hop past the target keeps its + residual path (``other=regions.code``); an exact match becomes local + (``other=region_id``). The public string contract is unchanged — the + strip logic simply no longer lives here. + """ + local = reroot_aggregate_key( + key, target_path=tuple(getattr(key.source, "path", ())), + ) + src = local.source + if isinstance(src, StarKey): + base = "*" + elif isinstance(src, ColumnSqlKey): + base = ".".join((*src.path, src.column_name)) + else: # ColumnKey + base = ".".join((*src.path, src.leaf)) + + formula = f"{base}:{local.agg}" + parts: List[str] = [_render_ref_formula(a) for a in local.args] + parts += [f"{k}={_render_ref_formula(v)}" for k, v in local.kwargs] + if parts: + formula += "(" + ", ".join(parts) + ")" + return formula + + +_REROOT_BIND_ERRORS = ( + UnknownReferenceError, + AmbiguousReferenceError, + IllegalScopeReferenceError, + ValueError, + NotImplementedError, +) + + +def _maybe_reroot_cross_model_plan( + *, + plan, + query: SlayerQuery, + agg_key: AggregateKey, + bundle: ResolvedSourceBundle, + host_model: SlayerModel, + public_projection: List[str], + subplan_builder: Callable[[SlayerQuery, ResolvedSourceBundle], PlannedQuery], +): + """Attach a re-rooted sub-``PlannedQuery`` to ``plan`` when the host + query carries dimensions reachable from the target by re-rooting through + the target's join graph. Returns ``plan`` unchanged when re-rooting is + unnecessary (only forward-path or genuinely unreachable dims).""" + target_model_name = plan.target_model + target_model = bundle.get_referenced_model(target_model_name) + if target_model is None: + return plan + target_path = tuple(getattr(agg_key.source, "path", ())) + rerooted_bundle = bundle.model_copy(update={"source_model": target_model}) + target_scope = ModelScope(source_model=target_model) + + def _resolvable_ref(ref_str: str) -> Optional[ValueKey]: + try: + return bind_expr( + parse_expr(ref_str), + scope=target_scope, + bundle=rerooted_bundle, + ).value_key + except _REROOT_BIND_ERRORS: + return None + + def _is_forward(path: Tuple[str, ...]) -> bool: + # On the host->target path (handled by the forward-path CTE already). + return bool(path) and path == target_path[: len(path)] + + n_dims = len(query.dimensions or []) + rerooted_dims: List[ColumnRef] = [] + rerooted_tds: List[TimeDimension] = [] + grain_host_sids: List[str] = [] + grain_rerooted_keys: List[ValueKey] = [] + needs_reroot = False + + for i, dim in enumerate(query.dimensions or []): + host_sid = public_projection[i] if i < len(public_projection) else None + host_path = _host_ref_path(dim.model) + rr = _reroot_ref( + model_prefix=dim.model, name=dim.name, + host_model_name=host_model.name, target_model_name=target_model_name, + ) + rr_key = _resolvable_ref(rr) + if rr_key is None: + continue # unreachable from target -> drop + if not _is_forward(host_path): + needs_reroot = True + if host_sid is None: + continue + rerooted_dims.append(ColumnRef(name=rr, label=dim.label)) + grain_host_sids.append(host_sid) + grain_rerooted_keys.append(rr_key) + + for j, td in enumerate(query.time_dimensions or []): + idx = n_dims + j + host_sid = public_projection[idx] if idx < len(public_projection) else None + host_path = _host_ref_path(td.dimension.model) + rr = _reroot_ref( + model_prefix=td.dimension.model, name=td.dimension.name, + host_model_name=host_model.name, target_model_name=target_model_name, + ) + rr_td = TimeDimension( + dimension=ColumnRef(name=rr), + granularity=td.granularity, + date_range=td.date_range, + label=td.label, + ) + try: + rr_key = bind_time_dimension( + rr_td, scope=target_scope, bundle=rerooted_bundle, + ).value_key + except _REROOT_BIND_ERRORS: + continue + if not _is_forward(host_path): + needs_reroot = True + if host_sid is None: + continue + rerooted_tds.append(rr_td) + grain_host_sids.append(host_sid) + grain_rerooted_keys.append(rr_key) + + # Filters. A purely host-local filter (every ref on the host's own + # columns) filters host rows -- it stays at the host base; the join-back + # propagates the cardinality reduction, so adding it to the CTE would risk + # binding a bare name to a same-named TARGET column. A join-traversing + # filter affects the aggregate value and rides into the re-rooted CTE; one + # that reaches OFF the host->target forward path is exactly what the + # forward-path classifier drops, so it also triggers re-rooting (covers a + # cross-model agg filtered through the target's graph with no dimensions). + host_scope = ModelScope(source_model=host_model) + rerooted_filters: List[str] = [] + for f in (query.filters or []): + try: + host_bound = bind_filter( + parse_filter_expr(f), scope=host_scope, bundle=bundle, + ) + except _REROOT_BIND_ERRORS: + continue + host_paths = _filter_ref_paths(host_bound.value_key) + if all(p == () for p in host_paths): + continue # host-local -> applied at the host base only + # The binder strips a same-model self-prefix (C14), so a + # ``.col`` ref binds locally against the target scope without + # any string surgery -- pass the filter through verbatim. + try: + bind_filter( + parse_filter_expr(f), scope=target_scope, bundle=rerooted_bundle, + ) + except _REROOT_BIND_ERRORS: + continue + rerooted_filters.append(f) + if any(p != target_path[: len(p)] for p in host_paths if p): + needs_reroot = True + + if not needs_reroot or not ( + rerooted_dims or rerooted_tds or rerooted_filters + ): + return plan + + rerooted_query = SlayerQuery( + source_model=target_model_name, + measures=[ModelMeasure(formula=_local_agg_formula(agg_key))], + dimensions=rerooted_dims or None, + time_dimensions=rerooted_tds or None, + filters=rerooted_filters or None, + ) + sub_plan = subplan_builder(rerooted_query, rerooted_bundle) + + sub_row_by_key = {s.key: s.id for s in sub_plan.row_slots} + grain_pairs: List[Tuple[str, str]] = [] + for host_sid, rr_key in zip(grain_host_sids, grain_rerooted_keys): + sub_sid = sub_row_by_key.get(rr_key) + if sub_sid is not None: + grain_pairs.append((host_sid, sub_sid)) + + sub_agg_sid = None + for s in sub_plan.aggregate_slots: + if isinstance(s.key, AggregateKey) and not getattr( + s.key.source, "path", (), + ): + sub_agg_sid = s.id + break + if sub_agg_sid is None: + return plan + + return plan.model_copy(update={ + "rerooted_plan": sub_plan, + "rerooted_grain_pairs": grain_pairs, + "rerooted_agg_slot_id": sub_agg_sid, + # The forward-path classifier marked these host filters + # DROP_UNREACHABLE, but the re-rooted CTE re-applies every + # target-reachable filter (and the host base keeps the rest for + # cardinality), so nothing is silently dropped -- clear the now-stale + # warnings and forward-only routing ids. + "dropped_filter_warnings": [], + "where_filter_ids": [], + "having_filter_ids": [], + "applied_filter_ids": [], + }) diff --git a/slayer/engine/enriched.py b/slayer/engine/enriched.py deleted file mode 100644 index a6ef17c1..00000000 --- a/slayer/engine/enriched.py +++ /dev/null @@ -1,316 +0,0 @@ -"""EnrichedQuery — fully resolved query ready for SQL generation. - -Architecture: - SlayerQuery (user-facing) → EnrichedQuery (engine-internal) → SQL - -SlayerQuery is what the user/agent provides — just names and references, -no SQL expressions or model details. It's intentionally minimal. - -EnrichedQuery is what the query engine produces after resolving SlayerQuery -against model definitions. Every measure and dimension carries its fully -resolved SQL expression, aggregation type, and model context. The SQL generator -works exclusively with EnrichedQuery — it never needs to look up model definitions. - -This separation means: -- New datasource clients only need to translate EnrichedQuery, not understand model resolution -- Validation happens at enrichment time, not during SQL generation -- The query engine controls resolution logic (placeholder expansion, join resolution) -""" - -from typing import Optional - -from pydantic import BaseModel, Field - -from slayer.core.enums import DataType, TimeGranularity -from slayer.core.format import NumberFormat -from slayer.core.formula import ParsedFilter -from slayer.core.models import Aggregation -from slayer.core.query import OrderItem - - -class EnrichedDimension(BaseModel): - """A dimension with its SQL expression fully resolved.""" - - name: str - sql: str | None - type: DataType - alias: str = Field(description="Result column name, e.g. 'orders.status'") - model_name: str - label: str | None = Field(default=None, description="Human-readable label") - format: NumberFormat | None = Field(default=None, description="Number format from the source dimension") - - -class EnrichedMeasure(BaseModel): - """A measure with its SQL expression and aggregation fully resolved.""" - - name: str - sql: str | None = Field(description="SQL expression; None for *:count (COUNT(*))") - aggregation: str = Field(description="Aggregation name: sum, avg, count, weighted_avg, etc.") - alias: str = Field(description="Result column name, e.g. 'orders.revenue_sum'") - user_declared: bool = Field( - default=False, - description=( - "DEV-1444: True iff this entry corresponds to an item the user " - "wrote in `query.measures`. False for auto-extracted hidden " - "entries (order-by aggregates, filter-extracted transforms, " - "window-arg hoists)." - ), - ) - model_name: str - aggregation_def: Aggregation | None = Field(default=None, description="Full aggregation definition (formula, params)") - agg_kwargs: dict[str, str] = Field(default_factory=dict, description="Query-time aggregation param overrides") - window: str | None = Field(default=None, description="Trailing time window for windowed sum/avg aggregations") - window_time_alias: str | None = Field(default=None, description="Time dimension alias used for windowed aggregations") - label: str | None = Field(default=None, description="Human-readable label") - time_column: str | None = Field(default=None, description="Explicit time col for first/last (overrides query default)") - source_measure_name: str | None = Field(default=None, description="Original measure name before canonicalization") - filter_sql: str | None = Field(default=None, description="Resolved SQL condition for filtered measures (CASE WHEN)") - filter_columns: list[str] = Field( - default_factory=list, - description="Resolved (qualified) column names referenced by the filter, for join planning", - ) - type: DataType | None = Field( - default=None, - description=( - "DEV-1361: declared result type of the aggregation. When set, " - "the SQL generator wraps the final agg expression in CAST AS . " - "Inherits from ModelMeasure.type at enrichment time." - ), - ) - column_type: DataType | None = Field( - default=None, - description=( - "DEV-1361: source column's declared type — wraps the inner " - "(pre-aggregation) expression in CAST when the column.sql is a " - "non-bare expression like json_extract(...). Distinct from " - "``type`` which wraps the outer aggregation result." - ), - ) - from_cross_model_intercept: bool = Field( - default=False, - description=( - "DEV-1449 / Codex round 10: True iff this EnrichedMeasure " - "was produced by the cross-stage intercept " - "(`_try_intercept_cross_model_as_local`) as a re-aggregation " - "of an inner stage's cross-model projection. Downstream " - "stages treat its column as a safe re-aggregation source " - "(equivalent to a CrossModelMeasure projection) when " - "computing ``SourceModelOrigin.agg_column_names``." - ), - ) - - -class EnrichedTimeDimension(BaseModel): - """A time dimension with resolved SQL and granularity.""" - - name: str - sql: str | None - granularity: TimeGranularity - date_range: list[str] | None - alias: str - model_name: str - label: str | None = None - - -class EnrichedExpression(BaseModel): - """An arithmetic expression computed from measure aliases. - - The sql references measure aliases from the base query (e.g., "revenue / count"). - Generated as an outer SELECT over a CTE containing the base query. - """ - - name: str - sql: str = Field(description="Expression referencing measure aliases") - alias: str = Field(description="Result column name") - user_declared: bool = Field( - default=False, - description=( - "DEV-1444: True iff this entry corresponds to a user-written " - "scalar formula in `query.measures`. False for auto-extracted " - "expressions (e.g. desugared change/change_pct arithmetic)." - ), - ) - label: str | None = None - type: DataType | None = Field( - default=None, - description=( - "DEV-1361: declared result type — when set, the outer SELECT " - "wraps the expression in CAST AS ." - ), - ) - - -class EnrichedTransform(BaseModel): - """A window-function or subquery transform applied to a measure. - - Most transforms generate window functions in an outer SELECT. - time_shift generates a self-join CTE. - change and change_pct are desugared at enrichment time into - a hidden time_shift transform + an EnrichedExpression for the arithmetic. - """ - - name: str - user_declared: bool = Field( - default=False, - description=( - "DEV-1444: True iff this entry corresponds to a user-written " - "window-transform measure in `query.measures`. False for the " - "internal time_shift transforms generated by change/change_pct " - "desugaring and for window-arg hoists." - ), - ) - transform: str = Field(description="Transform name: cumsum, lag, lead, rank, percent_rank, dense_rank, ntile, time_shift, first, last, consecutive_periods") - measure_alias: str = Field(description="Alias of the measure in the base CTE to transform") - alias: str = Field(description="Result column name") - offset: int = Field(description="For time_shift: number of rows or calendar units") - granularity: str | None = Field(default=None, description="For time_shift: year, month, quarter, etc.") - time_alias: str | None = Field(default=None, description="Alias of the time dimension column for ORDER BY") - partition_aliases: list[str] = Field(default_factory=list, description="Dimension aliases to PARTITION BY") - n: int | None = Field(default=None, description="Bucket count for ntile(measure, n=...)") - predicate_is_boolean: bool = Field( - default=False, - description="True when the transform's measure_alias points at a boolean expression " - "(e.g. consecutive_periods(revenue:sum > 0)). Drives portable CASE WHEN emission: " - "Postgres rejects 'boolean <> integer' so the numeric `IS NOT NULL AND <> 0` " - "predicate cannot be used.", - ) - label: str | None = None - type: DataType | None = Field( - default=None, - description=( - "DEV-1361: declared result type — when set, the window-layer " - "emitter wraps the transform expression in CAST AS ." - ), - ) - - -class EnrichedQuery(BaseModel): - """Fully resolved query — everything needed to generate SQL. - - Constructed by SlayerQueryEngine._enrich() from a SlayerQuery + SlayerModel. - Passed to SQLGenerator.generate() for SQL generation. - """ - - model_name: str - sql_table: str | None = None - sql: str | None = None - - resolved_joins: list[tuple] = Field(default_factory=list, description="[(target_table_sql, target_alias, join_condition, join_type), ...]") - - dimensions: list[EnrichedDimension] = Field(default_factory=list) - measures: list[EnrichedMeasure] = Field(default_factory=list) - time_dimensions: list[EnrichedTimeDimension] = Field(default_factory=list) - - expressions: list[EnrichedExpression] = Field(default_factory=list) - transforms: list[EnrichedTransform] = Field(default_factory=list) - - cross_model_measures: list["CrossModelMeasure"] = Field(default_factory=list) - - last_agg_time_column: str | None = Field(default=None, description="Time column for first/last aggregation (ORDER BY for ROW_NUMBER)") - - filters: list[ParsedFilter] = Field(default_factory=list) - order: list[OrderItem] | None = None - limit: int | None = None - offset: int | None = None - - field_name_aliases: dict[str, str] = Field(default_factory=dict, description="Custom field name → enriched alias mapping (for ORDER BY resolution)") - - user_projection: list[str] = Field( - default_factory=list, - description=( - "DEV-1444: ordered list of result-column aliases that the user " - "explicitly declared, in `query.dimensions + query.time_dimensions " - "+ query.measures` order. Drives the outer SELECT projection in " - "outer render mode and the public-attribute filter." - ), - ) - - distinct_dimension_values: bool = Field( - default=True, - description=( - "DEV-1543: pass-through of ``SlayerQuery.distinct_dimension_values``. " - "When ``False``, the SQL generator skips the dim-only-dedup " - "``GROUP BY`` clause and emits raw rows." - ), - ) - - -class CrossModelMeasure(BaseModel): - """A measure from a joined model, computed as a separate sub-query. - - The sub-query aggregates the measure from the target model scoped to - the shared dimensions, then the result is LEFT JOINed to the main query. - """ - - name: str - user_declared: bool = Field( - default=False, - description=( - "DEV-1444: True iff this entry corresponds to a user-written " - "cross-model measure reference in `query.measures` " - "(``customers.revenue:sum``). False for auto-extracted hidden " - "cross-model aggregates (e.g. used to satisfy an ORDER BY)." - ), - ) - alias: str = Field(description="Result column name, e.g. 'orders.customers__avg_score'") - target_model_name: str = Field(description="The joined model name") - target_model_sql_table: str | None - target_model_sql: str | None - measure: EnrichedMeasure = Field(description="The measure to aggregate") - join_pairs: list[list[str]] = Field(description="[[source_dim, target_dim], ...] from ModelJoin") - shared_dimensions: list[EnrichedDimension] = Field(description="Dimensions shared between main and target") - shared_time_dimensions: list[EnrichedTimeDimension] = Field(description="Time dims shared between main and target") - source_model_name: str = Field(description="The main query's model name") - source_sql_table: str | None = Field(description="Main model's table") - source_sql: str | None = Field(description="Main model's SQL") - join_type: str = Field(default="left", description="'left' or 'inner'") - label: str | None = None - format: NumberFormat | None = Field(default=None, description="Inferred format for this cross-model measure") - rerooted_enriched: Optional["EnrichedQuery"] = Field(default=None, description="Re-rooted subquery with target as source") - - -# Rebuild models with forward references -EnrichedQuery.model_rebuild() -CrossModelMeasure.model_rebuild() - - -def public_projection_aliases(enriched: EnrichedQuery) -> list[str]: - """Return the ordered list of public-projection aliases for ``enriched``. - - DEV-1444: the outer rendered SELECT projects exactly the - user-declared ``dimensions + time_dimensions + measures`` of the final - stage. The trim wrapper, ``SlayerResponse.attributes`` filter, and - ``expected_columns`` all consume this helper as the single source of - truth. - - Order is determined by ``enriched.user_projection`` when populated; - otherwise the helper falls back to the legacy bucket-union (every - entry whose ``name`` doesn't start with an internal prefix). The - fallback intentionally does NOT filter by ``user_declared`` — a - caller that constructs ``EnrichedQuery`` directly (e.g. tests - exercising specific rendered SQL shapes) has no obligation to flip - the flag, and filtering by ``user_declared=False`` (the default) - would drop every measure from the projection. - """ - if enriched.user_projection: - return list(enriched.user_projection) - # Fallback: declared-order bucket union, filtering ONLY entries whose - # names begin with one of the engine's internal-hoist prefixes - # (``_inner_*`` from nested-transform inner-arg hoists, ``_ft*`` from - # filter-transform extraction, ``_ts*`` from change/change_pct - # desugaring). Matches the pre-DEV-1444 ``expected_columns`` rule - # in ``query_engine.py``. - internal_prefixes = ("_inner_", "_ft", "_ts") - out: list[str] = [d.alias for d in enriched.dimensions] - out.extend(td.alias for td in enriched.time_dimensions) - out.extend( - m.alias for m in enriched.measures - if not m.name.startswith(internal_prefixes) - ) - out.extend(e.alias for e in enriched.expressions) - out.extend( - t.alias for t in enriched.transforms - if not t.name.startswith(internal_prefixes) - ) - out.extend(cm.alias for cm in enriched.cross_model_measures) - return out diff --git a/slayer/engine/enrichment.py b/slayer/engine/enrichment.py deleted file mode 100644 index 59ec3d47..00000000 --- a/slayer/engine/enrichment.py +++ /dev/null @@ -1,3370 +0,0 @@ -"""Query enrichment — resolves a SlayerQuery into an EnrichedQuery. - -Converts user-facing name-based references (e.g., field="count") into fully -resolved SQL expressions, aggregation types, and model context. The result -is an EnrichedQuery ready for SQL generation. - -Separated from query_engine.py for clarity — this is the largest single -transformation step in the query pipeline. -""" - -import difflib -import re -from typing import Any -from collections.abc import Mapping - -import sqlglot -from sqlglot import exp - -from slayer.core.enums import ( - BUILTIN_AGGREGATIONS, - DEFAULT_AGGREGATIONS_BY_TYPE, - DataType, - PRIMARY_KEY_AGGREGATIONS, -) -from slayer.core.formula import ( - canonical_agg_name, - ALL_TRANSFORMS, - AggregatedMeasureRef, - ArithmeticField, - MixedArithmeticField, - ParsedFilter, - RANK_FAMILY_TRANSFORMS, - TIME_TRANSFORMS, - TransformField, - _preprocess_like, - _rewrite_funcstyle_aggregations, - parse_filter, - parse_formula, -) -from slayer.core.models import Column, SlayerModel -from slayer.core.query import OrderItem, SlayerQuery, substitute_variables -from slayer.core.refs import DOTTED_IDENT_REF_RE as _DOTTED_IDENT_REF_RE -from slayer.engine.column_expansion import _is_trivial_base, expand_derived_refs -from slayer.engine.enriched import ( - CrossModelMeasure, - EnrichedDimension, - EnrichedExpression, - EnrichedMeasure, - EnrichedQuery, - EnrichedTimeDimension, - EnrichedTransform, -) -from slayer.sql.reserved_keywords import prequote_reserved_identifiers -from slayer.sql.sql_predicate import parse_sql_predicate -from slayer.sql.window_detect import WINDOW_IN_FILTER_ERROR, has_window_function - -_SELF_JOIN_TRANSFORMS = {"time_shift"} -# DEV-1686: quote-tolerant so join-path discovery matches a reserved qualifier -# that RESERVED_KEYWORDS emits quoted in expanded derived-column SQL. Tolerates -# every dialect's identifier quote char — ANSI ``"grant"``, MySQL/BigQuery -# `` `grant` ``, T-SQL ``[grant]`` — as well as bare refs and ``__``-path -# aliases (strict superset of the old bare ``word.word`` form). group(1) is -# still the unquoted qualifier name. -_TABLE_COL_RE = re.compile( - r'(? str: - """Strip one layer of single/double quotes from a query parameter value.""" - if len(value) >= 2 and value[0] == value[-1] and value[0] in ("'", '"'): - return value[1:-1] - return value - - -_canonical_agg_name = canonical_agg_name # Module-internal alias for the shared helper - - -def canonical_expression_key(node: Any) -> tuple[Any, ...]: # NOSONAR(S8495) — variable-length tuple shape IS the discriminator; type signature already declares Tuple[Any, ...] - """DEV-1444: build an alias-independent, structural hash key for a - parsed formula AST node. - - Two formulas that are *structurally equal* — same aggregation, same - column, same transform stack, same args / kwargs (order-insensitive) — - yield identical keys regardless of any user ``name`` override. - - Consumed by the provenance-merge step in enrichment: when an - auto-extracted entry (order-by aggregate, filter-extracted hidden - field, window-arg hoist) shares a key with an already-user-declared - entry, the entries collapse to the declared one — preventing phantom - ``orders.revenue_sum`` columns alongside a user-declared - ``{"formula":"revenue:sum","name":"total"}``. - """ - if isinstance(node, AggregatedMeasureRef): - return ( - "agg", - node.measure_name, - node.aggregation_name, - tuple(node.agg_args), - tuple(sorted(node.agg_kwargs.items())), - ) - if isinstance(node, TransformField): - return ( - "transform", - node.transform, - canonical_expression_key(node.inner), - tuple(node.args), - tuple(sorted((k, str(v)) for k, v in node.kwargs.items())), - ) - if isinstance(node, (ArithmeticField, MixedArithmeticField)): - # Arithmetic structural keys: the preprocessed SQL string with - # placeholders, plus the sorted set of inner agg-ref keys so - # ``a+b`` and ``b+a`` produce the same key when the underlying - # placeholders are interchangeable. - agg_keys = tuple(sorted( - canonical_expression_key(ref) for ref in node.agg_refs.values() - )) - return ("arith", node.sql, agg_keys) - # Fallback: stringify so downstream callers always get a hashable. - return ("raw", repr(node)) - - -async def _collect_reachable_agg_names( - model: SlayerModel, - resolve_join_target, - named_queries: dict, -) -> frozenset[str] | None: - """Collect custom aggregation names from the source model and all reachable joined models. - - Walks the full reachable join graph via BFS, bounded only by the ``visited`` - cycle guard (no fixed depth cap). Dotted-path resolution supports arbitrary - depth, so the rewrite must too. Returns ``None`` when no custom aggregations - exist anywhere. - """ - names: set[str] = set() - visited: set[str] = set() - queue: list[SlayerModel] = [model] - - while queue: - current = queue.pop(0) - if current.name in visited: - continue - visited.add(current.name) - - if current.aggregations: - names.update(a.name for a in current.aggregations) - - for join in current.joins: - if join.target_model not in visited: - target_info = await resolve_join_target( - target_model_name=join.target_model, - named_queries=named_queries, - ) - if target_info: - _, target_model_obj = target_info - if target_model_obj: - queue.append(target_model_obj) - - return frozenset(names) if names else None - - -def _public_field_name(qfield: Any) -> str: - """The public name a query measure surfaces under. - - An explicit ``name`` wins; otherwise the formula is mangled into an - identifier. Shared by the main measure loop and the reserved-name set - that hidden-transform allocation checks against, so the two can't drift. - """ - return qfield.name or qfield.formula.replace(" ", "_").replace("/", "_div_").replace(":", "_").replace( - "*", "" - ) - - -async def enrich_query( - query: SlayerQuery, - model: SlayerModel, - named_queries: dict[str, SlayerQuery] | None = None, - *, - resolve_dimension_via_joins, - resolve_cross_model_measure, - resolve_join_target, - resolve_model=None, - dialect: str = "postgres", - drop_unreachable_filters: bool = False, -) -> EnrichedQuery: - """Resolve a SlayerQuery against model definitions into an EnrichedQuery. - - Args: - query: The user-facing query. - model: The resolved model definition. - named_queries: Named sub-queries (for query lists). - resolve_dimension_via_joins: Callback(model, parts, named_queries) -> - (Column, SlayerModel) | None — returns the resolved column AND - the terminal model so the SQL expander can recurse into derived - references in ``Column.sql``. (Legacy single-value callbacks - that return just a Column are also accepted; in that case the - engine falls back to ``model`` as the terminal, which is fine - for tests that pass ``_noop_async``.) - resolve_cross_model_measure: Callback for cross-model measure refs. - resolve_join_target: Callback(target_model_name, named_queries) -> (table_sql, model)|None - resolve_model: Async callback ``(model_name, named_queries)`` -> - ``SlayerModel | None``, used by the column-SQL expander to - recursively walk join paths inside derived ``Column.sql`` - expressions. May be None in tests that don't exercise the - expansion path. - dialect: sqlglot dialect for parsing/emitting expanded SQL. - """ - named_queries = named_queries or {} - model_name_str = query.source_model if isinstance(query.source_model, str) else model.name - - # Custom aggregation names from source + all reachable joined models - custom_agg_names = await _collect_reachable_agg_names( - model=model, - resolve_join_target=resolve_join_target, - named_queries=named_queries, - ) - - # Saved-formula library for bare-name resolution. Only the source model's - # named measures are in scope here; cross-model references (`other.aov`) - # remain handled by the cross-model resolver. - named_measures: dict[str, str] = {} - for m in model.measures: - if not m.name: - continue - if m.name in named_measures: - raise ValueError( - f"Duplicate saved measure name '{m.name}' in model " - f"'{model.name}'. Saved measure names must be unique." - ) - named_measures[m.name] = m.formula - - # A bare named-measure reference like ``measures=["companies_count"]`` is - # an implicit rename: the user already chose ``companies_count`` as the - # surfacing name when they declared the ModelMeasure. Promote it to an - # explicit ``qf.name`` so DEV-1443's surface / collision / canonical - # machinery (which keys on ``qf.name``) treats it identically to an - # explicit ``{"formula": "...", "name": "..."}`` rename. Without this, - # the SELECT alias canonicalises off the expanded formula - # (``company_name_count_distinct``) while ORDER BY / result-key lookups - # still use the declared measure name (``companies_count``) — they - # diverge and the emitted SQL references a column that isn't projected. - for qf in (query.measures or []): - if qf.name is None and qf.formula.strip() in named_measures: - qf.name = qf.formula.strip() - - # --- Dimensions --- - dimensions = await _resolve_dimensions( - query=query, - model=model, - model_name_str=model_name_str, - named_queries=named_queries, - resolve_dimension_via_joins=resolve_dimension_via_joins, - resolve_model=resolve_model, - dialect=dialect, - ) - - # --- Measures (populated from fields below) --- - measures: list[EnrichedMeasure] = [] - - # --- Time dimensions --- - time_dimensions = await _resolve_time_dimensions( - query=query, - model=model, - model_name_str=model_name_str, - named_queries=named_queries, - resolve_dimension_via_joins=resolve_dimension_via_joins, - resolve_model=resolve_model, - dialect=dialect, - ) - - # DEV-1444: a query that lists the same column as both a regular - # dimension and a time dimension produces ambiguous aliases (same - # ``.`` key in both EnrichedDimension and - # EnrichedTimeDimension). Reject up-front with a clear error rather - # than silently picking one. - dim_alias_set = {d.alias for d in dimensions} - for td in time_dimensions: - if td.alias in dim_alias_set: - raise ValueError( - f"Column {td.alias!r} appears in both `dimensions` and " - f"`time_dimensions` — ambiguous projection. Use one or the other." - ) - - # --- Time resolution for transforms --- - resolved_time_alias = _resolve_time_alias( - time_dimensions=time_dimensions, - query=query, - model=model, - ) - - # --- Time column for type=last aggregation --- - last_agg_time_column = _resolve_last_agg_time( - query=query, - model=model, - dimensions=dimensions, - time_dimensions=time_dimensions, - ) - - # --- Process fields --- - enriched_expressions: list[EnrichedExpression] = [] - enriched_transforms: list[EnrichedTransform] = [] - cross_model_measures: list[CrossModelMeasure] = [] - known_aliases: dict[str, str] = {} - field_name_aliases: dict[str, str] = {} - # DEV-1443: canonical-agg alias → user-supplied measure name. Populated - # when a query measure renames the canonical (``{"formula": "col:agg", - # "name": "alias"}``). Consumed by the filter pre-pass (so a filter - # written as ``col:agg N`` resolves to the user alias and HAVINGs - # correctly) and by the ORDER BY enrichment (same shape). - canonical_to_user_name: dict[str, str] = {} - # Cached source-column name set for the remap eligibility guard - # (Codex Finding 1 — skip remap when the canonical alias also literally - # names a source column on the model, since the regex sub would then - # clobber the literal source-column reference). - _source_column_names: set[str] = {c.name for c in model.columns} - - # DEV-1444 provenance-merge index: canonical_expression_key → - # surfaced alias. Populated when an EnrichedMeasure is created; - # consulted by ``_ensure_aggregated_measure`` so an auto-extracted - # ref whose canonical form matches an already-declared measure - # (e.g. order-by ``revenue:sum`` matching a user-declared - # ``{"formula":"revenue:sum","name":"total"}``) reuses the existing - # alias instead of materialising a phantom ``orders.revenue_sum``. - measure_canonical_key_to_alias: dict[tuple[Any, ...], str] = {} - - def _mark_user_declared(alias: str) -> bool: - """DEV-1444: flip ``user_declared=True`` on the enriched entry that - owns ``alias``. The entry could live in any of measures, expressions, - transforms, or cross_model_measures. Returns True iff an entry was - found — callers use that to detect missing wiring. - """ - for m in measures: - if m.alias == alias: - m.user_declared = True - return True - for e in enriched_expressions: - if e.alias == alias: - e.user_declared = True - return True - for t in enriched_transforms: - if t.alias == alias: - t.user_declared = True - return True - for cm in cross_model_measures: - if cm.alias == alias: - cm.user_declared = True - return True - return False - - async def _ensure_aggregated_measure( - alias_key: str, - measure_name: str, - aggregation_name: str, - agg_args: list | None = None, - agg_kwargs: dict | None = None, - ): - """Create an EnrichedMeasure for an aggregated measure ref. - - Args: - alias_key: Key to use in known_aliases (placeholder ID or canonical name). - measure_name: Measure name ("revenue") or "*" for COUNT(*). - aggregation_name: Aggregation name ("sum", "weighted_avg", etc.). - agg_args: Positional args from colon syntax (e.g., time col for last/first). - agg_kwargs: Keyword args from colon syntax (e.g., weight override). - """ - agg_args = agg_args or [] - agg_kwargs = {k: _strip_string_literal(v) for k, v in (agg_kwargs or {}).items()} - - window = agg_kwargs.pop("window", None) - window_time_alias = None - if window is not None: - if aggregation_name not in ("sum", "avg"): - raise ValueError( - f"Aggregation parameter 'window' is only supported for sum and avg, " - f"not '{aggregation_name}'." - ) - if resolved_time_alias is None: - raise ValueError( - f"Windowed aggregation '{measure_name}:{aggregation_name}' requires an " - f"unambiguous time dimension. Add a single time_dimensions entry, or set " - f"main_time_dimension to select among multiple time dimensions." - ) - window_time_alias = resolved_time_alias - - # Canonical name for the result column (colon → underscore). Includes a - # signature suffix when args/kwargs are present so that parameterized - # variants (e.g. percentile(p=0.5) vs percentile(p=0.95)) don't collide. - canonical_name = _canonical_agg_name( - measure_name=measure_name, - aggregation_name=aggregation_name, - agg_args=agg_args, - agg_kwargs={**agg_kwargs, **({"window": window} if window is not None else {})}, - ) - - # DEV-1444 provenance merge: structural key collapsed across user - # ``name`` overrides. If a previous call already created an - # EnrichedMeasure for the same canonical form (e.g. a - # user-declared ``{"formula":"revenue:sum","name":"total"}``), - # reuse its surfaced alias and skip the duplicate hoist. - merged_kwargs = { - **agg_kwargs, - **({"window": window} if window is not None else {}), - } - canon_key = ( - "agg", - measure_name, - aggregation_name, - tuple(agg_args), - tuple(sorted(merged_kwargs.items())), - ) - existing_alias = measure_canonical_key_to_alias.get(canon_key) - if existing_alias is not None: - known_aliases[alias_key] = existing_alias - return - - # Skip if already ensured with this alias_key - alias = f"{model_name_str}.{canonical_name}" - if any(m.alias == alias for m in measures): - known_aliases[alias_key] = alias - measure_canonical_key_to_alias[canon_key] = alias - return - - # Resolve column SQL - measure_def = None - if measure_name == "*": - if aggregation_name != "count": - raise ValueError( - f"Aggregation '{aggregation_name}' not allowed with measure '*' — use '*:count' for COUNT(*)" - ) - sql = None - else: - measure_def = model.get_column(measure_name) - if measure_def is None: - raise ValueError( - f"Column '{measure_name}' not found in model '{model.name}'" - ) - # DEV-1576 §3: distinguish "unknown aggregation name" from "known - # but not allowed for this column type". The name check runs BEFORE - # the PK / whitelist / type gates so an unknown name never gets - # mislabelled as a type restriction (a perfectly aggregatable DOUBLE - # column with a misspelled agg should say "Unknown aggregation", - # not "not applicable to DOUBLE column"). - known_aggregations = BUILTIN_AGGREGATIONS | { - a.name for a in model.aggregations - } - if aggregation_name not in known_aggregations: - suggestion = difflib.get_close_matches( - word=aggregation_name, - possibilities=sorted(known_aggregations), - n=1, - ) - hint = f" Did you mean '{suggestion[0]}'?" if suggestion else "" - raise ValueError( - f"Unknown aggregation '{aggregation_name}'.{hint} " - f"Known: {sorted(known_aggregations)}." - ) - # Apply aggregation eligibility gates per the v2 contract: - # 1. Primary-key columns are always restricted to count/count_distinct - # (regardless of type or any explicit whitelist). - # 2. An explicit allowed_aggregations whitelist on a non-PK column - # overrides type defaults. - # 3. Otherwise, built-in aggregations are gated by type defaults; - # custom model-level aggregations are allowed without further - # type restriction. - if measure_def.primary_key: - if aggregation_name not in PRIMARY_KEY_AGGREGATIONS: - raise ValueError( - f"Aggregation '{aggregation_name}' not allowed for " - f"primary-key column '{measure_name}'. " - f"Allowed: {sorted(PRIMARY_KEY_AGGREGATIONS)}" - ) - elif measure_def.allowed_aggregations is not None: - if aggregation_name not in measure_def.allowed_aggregations: - raise ValueError( - f"Aggregation '{aggregation_name}' not allowed for column " - f"'{measure_name}'. Allowed: {measure_def.allowed_aggregations}" - ) - else: - is_custom_agg = model.get_aggregation(aggregation_name) is not None - if not is_custom_agg: - allowed = DEFAULT_AGGREGATIONS_BY_TYPE.get( - measure_def.type, frozenset() - ) - if aggregation_name not in allowed: - raise ValueError( - f"Aggregation '{aggregation_name}' is not applicable to " - f"{measure_def.type} column '{measure_name}' in model " - f"'{model.name}'. Default aggregations: {sorted(allowed)}" - ) - sql = measure_def.sql or measure_name - if measure_def.sql and resolve_model is not None: - expanded_sql = await expand_derived_refs( - sql=measure_def.sql, - model=model, - alias_path=model_name_str, - resolve_model=resolve_model, - named_queries=named_queries, - dialect=dialect, - ) - if expanded_sql is not None: - sql = expanded_sql - - # Validate aggregation exists - aggregation_def = model.get_aggregation(aggregation_name) - if aggregation_name not in BUILTIN_AGGREGATIONS and aggregation_def is None: - raise ValueError( - f"Aggregation '{aggregation_name}' is not a built-in aggregation " - f"and is not defined in model '{model.name}'." - ) - - # For first/last with explicit time dimension arg, store on the measure - explicit_time_col = None - if aggregation_name in ("first", "last") and agg_args: - explicit_time_col = agg_args[0] - if "." not in explicit_time_col: - explicit_time_col = f"{model.name}.{explicit_time_col}" - - # Resolve measure-level filter. ``Column.filter`` is Mode A SQL - # (DEV-1369 / DEV-1378): arbitrary SQL function calls - # (``json_extract``, ``coalesce``, ``CASE WHEN``, dialect-specific - # operators) flow through; DSL constructs (aggregation colon - # syntax, transform calls, ``OVER``) were rejected at construction - # by ``parse_sql_predicate``. - filter_sql = None - filter_columns: list[str] = [] - if measure_def and measure_def.filter: - parsed = parse_sql_predicate(measure_def.filter) - resolved = await resolve_filter_columns( - parsed_filters=[parsed], - model=model, - model_name=model_name_str, - resolve_join_target=resolve_join_target, - named_queries=named_queries, - resolve_model=resolve_model, - dialect=dialect, - strict=False, - ) - filter_sql = resolved[0].sql - filter_columns = list(resolved[0].columns) - - # DEV-1361: pull through the source Column's declared type so the - # generator can wrap the pre-aggregation expression in CAST when the - # column's sql is non-bare (e.g. json_extract). - column_type = ( - measure_def.type - if measure_def is not None and isinstance(measure_def.type, DataType) - else None - ) - measures.append( - EnrichedMeasure( - name=canonical_name, - sql=sql, - aggregation=aggregation_name, - alias=alias, - model_name=model_name_str, - aggregation_def=aggregation_def, - agg_kwargs=agg_kwargs, - window=window, - window_time_alias=window_time_alias, - label=measure_def.label if measure_def else None, - time_column=explicit_time_col, - source_measure_name=measure_name, - filter_sql=filter_sql, - filter_columns=filter_columns, - column_type=column_type, - ) - ) - known_aliases[alias_key] = alias - # DEV-1444: record the canonical key so later refs to the same - # canonical form collapse onto this entry's alias (and onto any - # subsequent user-name rename of it). - measure_canonical_key_to_alias[canon_key] = alias - - def _resolve_sql(sql: str) -> str: - resolved = sql - for name, alias in sorted(known_aliases.items(), key=lambda x: -len(x[0])): - # Negative lookbehind for . and " prevents matching inside - # already-quoted identifiers (e.g., _count inside "orders._count") - resolved = re.sub(rf'(? list[str]: - """Resolve partition_by= column references to base-CTE aliases. - - partition_by entries must reference query dimensions or time dimensions — - otherwise the column wouldn't be in the base CTE. Match by bare name - (e.g. 'customer_id' against EnrichedDimension.name) or by qualified - alias (e.g. 'orders.customer_id'). Cross-model dotted paths - ('customers.region') match via the dimension alias as built by - _resolve_dimensions. - """ - by_name = {d.name: d.alias for d in dimensions} - by_alias = {d.alias: d.alias for d in dimensions} - for td in time_dimensions: - by_name.setdefault(td.name, td.alias) - by_alias.setdefault(td.alias, td.alias) - - resolved: list[str] = [] - for col in partition_by: - if col in by_alias: - resolved.append(by_alias[col]) - elif col in by_name: - resolved.append(by_name[col]) - else: - available = sorted(set(by_name) | set(by_alias)) - raise ValueError( - f"Transform '{transform}': partition_by column '{col}' is not " - f"a query dimension. Add it to dimensions/time_dimensions, or " - f"choose one of: {', '.join(available) or '(none)'}." - ) - return resolved - - # DEV-1692: names for transforms hoisted out of an arithmetic formula are - # derived from the owning measure's field_name, which keeps them distinct - # from one another — but nothing stops a user from *also* selecting a - # measure or dimension literally named ``_t0_growth``. Both would then - # claim the alias ``._t0_growth``: the self-join CTE projects two - # columns under that name and the hoisted reference silently resolves to - # the user's column (no error, wrong numbers). Allocate against every - # projected name instead of trusting field_name uniqueness on its own. - # Dimensions and time dimensions alias as ``.`` just like - # hoisted transforms do, so they are reserved by bare name too. - _reserved_public_names: set[str] = ( - {_public_field_name(qf) for qf in (query.measures or [])} - | {d.name for d in dimensions} - | {td.name for td in time_dimensions} - ) - _hidden_names: set[str] = set() - - def _allocate_hidden_name(preferred: str) -> str: - candidate = preferred - suffix = 2 - while ( - candidate in _reserved_public_names - or candidate in _hidden_names - or candidate in known_aliases - ): - candidate = f"{preferred}_{suffix}" - suffix += 1 - _hidden_names.add(candidate) - return candidate - - def _add_transform( - name: str, - transform: str, - measure_alias: str, - offset: int = 1, - granularity: str = None, - predicate_is_boolean: bool = False, - kwargs: dict[str, Any] | None = None, - ): - needs_time = transform in TIME_TRANSFORMS - if needs_time and resolved_time_alias is None: - raise ValueError( - f"Field '{name}' ({transform}) requires an unambiguous time dimension. " - f"Add a single time_dimensions entry, or set main_time_dimension to " - f"select among multiple time dimensions." - ) - alias = f"{model_name_str}.{name}" - kwargs = kwargs or {} - - # Rank-family transforms default to no partition (rank across the entire - # result set) and accept an explicit partition_by= override. Other - # transforms (cumsum, lag, lead, first, last, time_shift, - # consecutive_periods) partition by all query dimensions, matching the - # invariant that adding a measure must not change cardinality. - if transform in RANK_FAMILY_TRANSFORMS: - partition_by = kwargs.get("partition_by") - partition_aliases = ( - _resolve_rank_partition(transform, partition_by) if partition_by else [] - ) - else: - partition_aliases = [d.alias for d in dimensions] - - enriched_transforms.append( - EnrichedTransform( - name=name, - transform=transform, - measure_alias=measure_alias, - alias=alias, - offset=offset, - granularity=granularity, - time_alias=resolved_time_alias if needs_time else None, - partition_aliases=partition_aliases, - predicate_is_boolean=predicate_is_boolean, - n=kwargs.get("n"), - ) - ) - known_aliases[name] = alias - - # DEV-1449: aggregations whose group-wise results can be re-aggregated - # to an equivalent overall result. `sum`/`min`/`max` are distributive - # (re-aggregating with the same op is exact). `count` is additive — the - # outer must use `sum` over the inner per-group count, not `count` of - # stage rows (which would just count groups). Everything else (avg, - # count_distinct, median, percentile, stddev, ...) is non-distributive - # and silently changes semantics under re-aggregation, so the intercept - # falls through to the cross-model CTE path for them. - _DISTRIBUTIVE_AGGS = frozenset({"sum", "min", "max"}) - - def _intercept_candidate_for_cross_model(ref) -> "tuple[str, str] | None": - """DEV-1449: return ``(flat_with_agg, outer_agg)`` if the - intercept would resolve a virtual-stage cross-model agg ref to a - local re-aggregation on a flat column, or ``None`` if the - intercept doesn't apply. - - Pure / side-effect-free: callers use this first to build a - dup-guard key on the *resolved* flat name (so two refs that - differ only in source-prefix and resolve to the same underlying - column collide in the guard), then call - ``_try_intercept_cross_model_as_local`` to actually apply. - - Lookup candidates: try ancestor-stripped flat first, then full - flat, mirroring ``resolve_via_stage_origin``'s Candidate A/B. - Semantics gate: only ``sum``/``min``/``max``/``count`` are - distributive enough to re-aggregate. Parameterized aggs are - skipped — ``resolve_cross_model_measure`` canonicalizes with - no args/kwargs participation, so the inner flat name we'd look - for doesn't account for params either. - """ - if model.source_model_origin is None: - return None - if ref.agg_args or ref.agg_kwargs: - return None - if ref.aggregation_name in _DISTRIBUTIVE_AGGS: - outer_agg = ref.aggregation_name - elif ref.aggregation_name == "count": - outer_agg = "sum" - else: - return None - leaf = ref.measure_name.rsplit(".", 1)[-1] - canonical_leaf_agg = ( - f"_{ref.aggregation_name}" if leaf == "*" - else f"{leaf}_{ref.aggregation_name}" - ) - hop_parts = ref.measure_name.split(".") - ancestor_names: set[str] = set() - cursor = model.source_model_origin - while cursor is not None: - ancestor_names.add(cursor.name) - cursor = cursor.parent - # Codex review on PR #137 round 9: gate the candidate on the - # column being an AGGREGATION projection from the inner stage - # (not a dim that coincidentally matches the canonical-flat - # shape). ``agg_column_names`` is populated by - # ``_query_as_model`` from the inner enriched query's measures - # / cross_model_measures / transforms / expressions. - agg_names = model.source_model_origin.agg_column_names - # Candidate A — strip a leading ancestor name from the hop path. - if hop_parts and hop_parts[0] in ancestor_names and len(hop_parts) >= 2: - stripped = hop_parts[1:-1] + [canonical_leaf_agg] - candidate = "__".join(stripped) - if candidate in agg_names and model.get_column(candidate) is not None: - return candidate, outer_agg - # Candidate B — full flat. - if hop_parts: - candidate = "__".join(hop_parts[:-1] + [canonical_leaf_agg]) - if candidate in agg_names and model.get_column(candidate) is not None: - return candidate, outer_agg - return None - - async def _try_intercept_cross_model_as_local( - ref, field_name: str, - ) -> str | None: - """Apply the intercept (computes candidate + builds the - EnrichedMeasure). Returns the full enriched alias the caller - can use, or ``None`` if no candidate.""" - candidate = _intercept_candidate_for_cross_model(ref=ref) - if candidate is None: - return None - flat_with_agg, outer_agg = candidate - await _ensure_aggregated_measure( - alias_key=field_name, - measure_name=flat_with_agg, - aggregation_name=outer_agg, - agg_args=ref.agg_args, - agg_kwargs=ref.agg_kwargs, - ) - local_alias = known_aliases[field_name] - # Codex round 10: mark the created-or-reused EnrichedMeasure - # as intercept-produced so `_query_as_model` includes its - # downstream short in `agg_column_names`. Downstream stages - # then recognise it as a safe cross-model re-aggregation - # source on equal footing with auto-derived CMM canonicals. - for em in measures: - if em.alias == local_alias: - em.from_cross_model_intercept = True - break - # Codex round 11: register the dotted cross-model canonical - # in `field_name_aliases` so `generator._resolve_order_column`'s - # qualified-match branch finds it. This is what allows - # `order=[{"column":"customers.revenue:sum"}]` to resolve when - # the user didn't also declare the measure as a query measure - # (which would go through the qfield-site path that already - # registers the alias). The intercept-via-`_flatten_spec` / - # `_ensure_measure_from_spec` paths reach here. - ref_canonical = _canonical_agg_name( - measure_name=ref.measure_name, - aggregation_name=ref.aggregation_name, - agg_args=ref.agg_args, - agg_kwargs=ref.agg_kwargs, - ) - field_name_aliases[ref_canonical] = local_alias - return local_alias - - async def _ensure_measure_from_spec(mname: str, agg_refs: dict | None = None): - """Ensure a measure is resolved — handles agg refs only.""" - agg_refs = agg_refs or {} - if mname in agg_refs: - ref = agg_refs[mname] - if "." in ref.measure_name and ref.measure_name != "*": - # DEV-1449: cross-model agg ref against a virtual stage - # whose inner stage already projected the flat alias → - # emit a local re-aggregated measure instead of a CTE. - local_alias = await _try_intercept_cross_model_as_local( - ref=ref, field_name=mname, - ) - if local_alias is not None: - return - # Cross-model aggregated measure inside an expression — - # resolve as a CrossModelMeasure (gets its own CTE). - cm = await resolve_cross_model_measure( - spec_name=ref.measure_name, - field_name=mname, - model=model, - query=query, - dimensions=dimensions, - time_dimensions=time_dimensions, - named_queries=named_queries, - aggregation_name=ref.aggregation_name, - agg_kwargs=ref.agg_kwargs, - ) - cross_model_measures.append(cm) - known_aliases[mname] = cm.alias - return - await _ensure_aggregated_measure( - alias_key=mname, - measure_name=ref.measure_name, - aggregation_name=ref.aggregation_name, - agg_args=ref.agg_args, - agg_kwargs=ref.agg_kwargs, - ) - else: - raise ValueError(f"Bare measure name '{mname}' in expression is not valid. Use colon syntax.") - - async def _resolve_inner_alias(inner_spec, fallback_name: str) -> str: - """Flatten a transform's inner spec to a measure alias. - - ``AggregatedMeasureRef`` inners reuse their canonical alias - (e.g. ``revenue:sum`` → ``revenue_sum``) so the hidden inner - measure shares the same column key as a sibling-level reference; - every other shape falls back to ``fallback_name``. - """ - if isinstance(inner_spec, AggregatedMeasureRef): - canonical = _canonical_agg_name( - measure_name=inner_spec.measure_name, - aggregation_name=inner_spec.aggregation_name, - agg_args=inner_spec.agg_args, - agg_kwargs=inner_spec.agg_kwargs, - ) - return await _flatten_spec(inner_spec, canonical) - return await _flatten_spec(inner_spec, fallback_name) - - async def _flatten_spec(spec, field_name: str) -> str: - if isinstance(spec, AggregatedMeasureRef): - if "." in spec.measure_name and spec.measure_name != "*": - # DEV-1449: cross-model agg ref against a virtual stage - # whose inner stage already projected the flat alias. - local_alias = await _try_intercept_cross_model_as_local( - ref=spec, field_name=field_name, - ) - if local_alias is not None: - return local_alias - # Cross-model aggregated measure - cm = await resolve_cross_model_measure( - spec_name=spec.measure_name, - field_name=field_name, - model=model, - query=query, - dimensions=dimensions, - time_dimensions=time_dimensions, - named_queries=named_queries, - aggregation_name=spec.aggregation_name, - agg_kwargs=spec.agg_kwargs, - ) - cross_model_measures.append(cm) - known_aliases[field_name] = cm.alias - return cm.alias - - canonical_name = _canonical_agg_name( - measure_name=spec.measure_name, - aggregation_name=spec.aggregation_name, - agg_args=spec.agg_args, - agg_kwargs=spec.agg_kwargs, - ) - await _ensure_aggregated_measure( - alias_key=canonical_name, - measure_name=spec.measure_name, - aggregation_name=spec.aggregation_name, - agg_args=spec.agg_args, - agg_kwargs=spec.agg_kwargs, - ) - # DEV-1444: after provenance-merge the canonical alias may - # point at a previously declared user measure (e.g. when the - # user renamed ``revenue:sum`` to ``total``). Consult - # known_aliases so downstream callers receive the surfaced - # alias rather than synthesising the unrenamed canonical form. - return known_aliases.get( - canonical_name, f"{model_name_str}.{canonical_name}" - ) - - elif isinstance(spec, ArithmeticField): - for mname in spec.measure_names: - await _ensure_measure_from_spec(mname, spec.agg_refs) - alias = f"{model_name_str}.{field_name}" - enriched_expressions.append( - EnrichedExpression( - name=field_name, - sql=_resolve_sql(spec.sql), - alias=alias, - ) - ) - known_aliases[field_name] = alias - return alias - - elif isinstance(spec, MixedArithmeticField): - for mname in spec.measure_names: - await _ensure_measure_from_spec(mname, spec.agg_refs) - # DEV-1692: the ``_t{n}`` placeholder counter restarts on every - # formula parse, so two measures that each wrap a transform in - # arithmetic would both flatten under the name ``_t0`` — colliding - # on the self-join CTE names (``shifted__t0``) and, worse, on the - # expression alias, silently making the second measure read the - # first one's value. Qualify with the owning measure's field_name - # and run it through _allocate_hidden_name so the result can't - # collide with a user measure that happens to share the shape. - placeholder_aliases: list[tuple[str, str]] = [] - for placeholder, sub_transform in spec.sub_transforms: - hidden_name = _allocate_hidden_name(f"{placeholder}_{field_name}") - sub_alias = await _flatten_spec( - spec=sub_transform, field_name=hidden_name - ) - placeholder_aliases.append((placeholder, sub_alias)) - # Bind the placeholders just long enough for _resolve_sql to rewrite - # this formula's references, then restore. A user measure may itself - # be named `_t0`, so a shadowed binding is put back rather than - # dropped. - shadowed: list[tuple[str, str | None]] = [ - (placeholder, known_aliases.get(placeholder)) - for placeholder, _ in placeholder_aliases - ] - for placeholder, sub_alias in placeholder_aliases: - known_aliases[placeholder] = sub_alias - resolved_sql = _resolve_sql(spec.sql) - for placeholder, prior in shadowed: - if prior is None: - del known_aliases[placeholder] - else: - known_aliases[placeholder] = prior - alias = f"{model_name_str}.{field_name}" - enriched_expressions.append( - EnrichedExpression( - name=field_name, - sql=resolved_sql, - alias=alias, - ) - ) - known_aliases[field_name] = alias - return alias - - elif isinstance(spec, TransformField): - if spec.transform in ("change", "change_pct"): - # Desugar: change(a) → a - time_shift(a, offset) - # change_pct(a) → CASE WHEN ts != 0 THEN (a - ts) / ts END - if ( - isinstance(spec.inner, TransformField) - and spec.inner.transform in (*_SELF_JOIN_TRANSFORMS, "change", "change_pct") - ): - raise ValueError( - f"Nesting '{spec.transform}' around '{spec.inner.transform}' is not supported. " - f"Both use self-join CTEs. Try wrapping with a window function instead " - f"(e.g., cumsum, lag)." - ) - - # Flatten the inner spec to get the measure alias - inner_alias = await _resolve_inner_alias( - spec.inner, f"_inner_{field_name}" - ) - - # Determine offset and granularity - offset = -1 - granularity = None - if spec.args: - offset = spec.args[0] if isinstance(spec.args[0], int) else -1 - if len(spec.args) >= 2: - granularity = str(spec.args[1]) - - # Create hidden time_shift transform - ts_name = f"_ts_{field_name}" - _add_transform( - name=ts_name, - transform="time_shift", - measure_alias=inner_alias, - offset=offset, - granularity=granularity, - ) - # Find the known_aliases key for the inner measure - inner_key = next(k for k, v in known_aliases.items() if v == inner_alias) - - # Build expression - if spec.transform == "change": - expr_sql = _resolve_sql(f"{inner_key} - {ts_name}") - else: # change_pct - expr_sql = _resolve_sql( - f"CASE WHEN {ts_name} != 0 " - f"THEN ({inner_key} - {ts_name}) * 1.0 / {ts_name} END" - ) - - alias = f"{model_name_str}.{field_name}" - enriched_expressions.append( - EnrichedExpression(name=field_name, sql=expr_sql, alias=alias) - ) - known_aliases[field_name] = alias - return alias - - # Non-change transforms (time_shift, cumsum, lag, lead, rank, last) - if ( - spec.transform in _SELF_JOIN_TRANSFORMS - and isinstance(spec.inner, TransformField) - and spec.inner.transform in _SELF_JOIN_TRANSFORMS - ): - raise ValueError( - f"Nesting '{spec.transform}' around '{spec.inner.transform}' is not supported. " - f"Both use self-join CTEs. Try wrapping with a window function instead " - f"(e.g., cumsum, lag)." - ) - inner_alias = await _resolve_inner_alias( - spec.inner, f"_inner_{field_name}" - ) - - offset = 1 - granularity = None - if spec.args: - offset = spec.args[0] if isinstance(spec.args[0], int) else 1 - if len(spec.args) >= 2: - granularity = str(spec.args[1]) - - # consecutive_periods (and any other transform that wraps a - # predicate) needs to know whether the inner expression renders - # as boolean — Postgres rejects `boolean <> integer` so the - # numeric form ` IS NOT NULL AND <> 0` cannot be - # used for boolean inputs. - inner_is_predicate = ( - isinstance(spec.inner, (ArithmeticField, MixedArithmeticField)) - and spec.inner.is_predicate - ) - _add_transform( - name=field_name, - transform=spec.transform, - measure_alias=inner_alias, - offset=offset, - granularity=granularity, - predicate_is_boolean=inner_is_predicate, - kwargs=spec.kwargs, - ) - return f"{model_name_str}.{field_name}" - - raise ValueError(f"Unsupported field spec: {spec!r}") - - # DEV-1444: track aliases in declared order so EnrichedQuery.user_projection - # can be populated at the end. Dims and time-dims come first. - user_projection: list[str] = [d.alias for d in dimensions] - user_projection.extend(td.alias for td in time_dimensions) - - # DEV-1444 (Codex review on PR #134): the provenance-merge index in - # ``_ensure_aggregated_measure`` would silently collapse two - # user-declared measures that share a canonical key — e.g. - # ``{"formula":"amount:sum","name":"revenue1"}`` followed by - # ``{"formula":"amount:sum","name":"revenue2"}`` — onto whichever - # surfaced alias was claimed first, leaving the second qfield's - # alias in ``user_projection`` with no matching EnrichedMeasure. The - # outer trim would then project a column the inner SELECT doesn't - # expose. Track the set of canonical keys already owned by a - # user-declared qfield and refuse the duplicate. - user_declared_canon_keys: dict[tuple[Any, ...], str] = {} - - # DEV-1443 (CodeRabbit thread + Codex round 4 on PR #133): the - # duplicate-explicit-name check must run for every query measure - # kind, not just inside the local AggregatedMeasureRef rename branch. - # Cross-model aggregates ``continue`` before reaching that branch and - # arithmetic/transform measures fall through to ``_flatten_spec``; in - # both cases a duplicate ``name`` would silently collapse two - # measures onto a single alias. Run the pairwise check once up front. - _seen_explicit_names: dict[str, str] = {} - for qf in (query.measures or []): - if not qf.name: - continue - if qf.name in _seen_explicit_names: - raise ValueError( - f"Measure '{qf.formula}' and measure " - f"'{_seen_explicit_names[qf.name]}' both declare name " - f"'{qf.name}'. Two distinct aggregates would otherwise be " - f"silently merged into one column. Pick a different `name` " - f"for one of them." - ) - _seen_explicit_names[qf.name] = qf.formula - - # DEV-1448: lift the canonical-collision guard out of the local-rename - # branch so it runs symmetrically for local AND cross-model renames. A - # query measure whose surfaced public alias OR downstream short name - # equals another sibling's would otherwise let - # ``_ensure_aggregated_measure``'s alias-keyed dedup (or the virtual- - # model column-name dedup in ``_query_as_model``) silently merge two - # distinct aggregates into one column. Compute the would-be public - # alias + downstream short for each ``AggregatedMeasureRef`` qfield - # and check for pairwise collisions on either axis. - # - # Codex review round 2 on PR #136: the prior version compared only - # ``qfield.name`` against the sibling's full canonical name, so - # ``customers.revenue:sum`` renamed to ``"id_count_distinct"`` alongside - # an unrenamed ``customers.id:count_distinct`` slipped through because - # ``"id_count_distinct"`` != ``"customers.id_count_distinct"`` — yet - # both surface at ``orders.customers.id_count_distinct``. - def _surfaces_for(qf, sp): - canonical = _canonical_agg_name( - measure_name=sp.measure_name, - aggregation_name=sp.aggregation_name, - agg_args=sp.agg_args, - agg_kwargs=sp.agg_kwargs, - ) - is_cross_model = ( - "." in sp.measure_name and sp.measure_name != "*" - ) - renamed = bool(qf.name) and qf.name != canonical - if is_cross_model: - # Codex review round 3 on PR #136: mirror the actual canonical - # construction in ``_resolve_cross_model_measure`` (the leaf- - # only form, with ``*`` collapsing to ``_``) rather than - # the full-name ``_canonical_agg_name`` form. The two differ - # for ``.*:`` (the resolver emits ``_``; the - # full canonical would emit ``*_``) — and the public - # alias we need to compare against is the one the resolver - # actually produces. - hop, leaf = sp.measure_name.rsplit(".", 1) - cm_leaf = ( - f"_{sp.aggregation_name}" if leaf == "*" - else f"{leaf}_{sp.aggregation_name}" - ) - if renamed: - public = f"{model_name_str}.{hop}.{qf.name}" - short = qf.name - else: - public = f"{model_name_str}.{hop}.{cm_leaf}" - # _query_as_model derives the downstream short from - # _alias_to_short(cm.alias) for unrenamed cross-model: - # the source-model prefix is stripped, then dots are - # converted to ``__``. Mirror that here. - short = f"{hop}.{cm_leaf}".replace(".", "__") - else: - if renamed: - public = f"{model_name_str}.{qf.name}" - short = qf.name - else: - public = f"{model_name_str}.{canonical}" - short = canonical - return public, short - - # CodeRabbit review round 3 on PR #136: seed the collision set with - # the already-enriched dimension + time-dimension aliases so a renamed - # cross-model measure whose alias collides with a dim/time-dim alias - # (e.g. ``dimensions=[customers.region_id]`` + ``measures=[{"formula": - # "customers.revenue:sum", "name": "region_id"}]`` both producing - # ``orders.customers.region_id``) is caught. The source-column guard - # at lines 871-878 only catches collisions with columns on the OUTER - # source model — not with columns surfaced via joined dims. - # - # Codex review round 4 on PR #136: also track each dim/time-dim's - # DOWNSTREAM SHORT (the ``_alias_to_short``-flattened form used as - # the virtual-model column name in ``_query_as_model``). A renamed - # measure with a matching downstream short would surface as a - # duplicate column on the virtual model even when the public - # aliases differ. ``_alias_to_short`` strips the source-model - # prefix (``model_name_str.`` portion) and converts remaining dots - # to ``__``. - def _alias_to_short_local(alias: str) -> str: - stripped = alias.split(".", 1)[-1] if "." in alias else alias - return stripped.replace(".", "__") - - _occupied_aliases: dict[str, str] = {} - _occupied_shorts: dict[str, str] = {} - for _d in dimensions: - _occupied_aliases[_d.alias] = f"dimension '{_d.name}'" - _occupied_shorts[_alias_to_short_local(_d.alias)] = ( - f"dimension '{_d.name}'" - ) - for _td in time_dimensions: - _occupied_aliases[_td.alias] = f"time dimension '{_td.name}'" - _occupied_shorts[_alias_to_short_local(_td.alias)] = ( - f"time dimension '{_td.name}'" - ) - - # Codex review round 6 on PR #136: the pre-pass previously skipped - # non-``AggregatedMeasureRef`` qfields (arithmetic / transform / mixed - # formulas), but those measures also surface in ``_query_as_model`` - # via their ``field_name`` (= ``qf.name`` or the mangled formula). - # A renamed cross-model measure whose ``name`` matches another - # measure's mangled ``field_name`` would still emit two columns with - # the same short in the virtual model. Compute (public, short) for - # every qfield kind so the collision checks below cover all - # combinations. ``canonical_pre`` is only meaningful for aggregated - # refs (used by the logical canonical-name check); other kinds get - # an empty string which never matches a real canonical. - def _mangled_formula(formula: str) -> str: - # Mirror the field-name mangling at the top of the per-qfield - # loop (line ~879) so the pre-pass sees the same ``field_name`` - # ``_flatten_spec`` will emit for non-renamed arithmetic / - # transform measures. - return ( - formula.replace(" ", "_") - .replace("/", "_div_") - .replace(":", "_") - .replace("*", "") - ) - - _surfaces: list = [] - for qf_pre in (query.measures or []): - sp_pre = parse_formula( - qf_pre.formula, - extra_agg_names=custom_agg_names, - named_measures=named_measures, - ) - if isinstance(sp_pre, AggregatedMeasureRef): - public_pre, short_pre = _surfaces_for(qf_pre, sp_pre) - canonical_pre = _canonical_agg_name( - measure_name=sp_pre.measure_name, - aggregation_name=sp_pre.aggregation_name, - agg_args=sp_pre.agg_args, - agg_kwargs=sp_pre.agg_kwargs, - ) - else: - # Arithmetic / transform / mixed: surfaced via _flatten_spec. - field_name_pre = qf_pre.name or _mangled_formula(qf_pre.formula) - public_pre = f"{model_name_str}.{field_name_pre}" - short_pre = field_name_pre - canonical_pre = "" # no meaningful canonical for this kind - # CodeRabbit round 3: catch measure-vs-(dim|time-dim) public-alias - # collisions. - if public_pre in _occupied_aliases: - owner = _occupied_aliases[public_pre] - raise ValueError( - f"Measure '{qf_pre.formula}' surfaces as '{public_pre}', " - f"which collides with the {owner} on the same query — the " - f"outer projection key would be duplicated and the result " - f"shape could silently merge values. Pick a different " - f"`name`, or remove the duplicate dimension." - ) - # Codex round 4: catch measure-vs-(dim|time-dim) DOWNSTREAM-short - # collisions. The public aliases may differ but the virtual-model - # column emitted by ``_query_as_model`` would still duplicate. - if short_pre in _occupied_shorts: - owner = _occupied_shorts[short_pre] - raise ValueError( - f"Measure '{qf_pre.formula}' produces the downstream " - f"short name '{short_pre}', which collides with the " - f"{owner} on the same query — a nested-DAG stage's " - f"virtual model would have two columns with the same " - f"alias. Pick a different `name`, or remove the " - f"duplicate dimension." - ) - for qf_other, sp_other, public_other, short_other, canonical_other in _surfaces: - if public_pre == public_other: - raise ValueError( - f"Measure '{qf_pre.formula}' and measure " - f"'{qf_other.formula}' both surface as " - f"'{public_pre}'. Two distinct aggregates would " - f"otherwise be silently merged into one column. " - f"Pick a different `name`, or rename the other " - f"measure too." - ) - if short_pre == short_other: - raise ValueError( - f"Measure '{qf_pre.formula}' and measure " - f"'{qf_other.formula}' both produce the downstream " - f"short name '{short_pre}' — the alias a nested-DAG " - f"stage would use to reference the value. Two " - f"distinct aggregates would otherwise collide in " - f"the downstream stage's virtual model. Pick a " - f"different `name`, or rename the other measure too." - ) - # DEV-1443 logical-canonical guard (retained alongside the - # alias / short collision checks above): a rename whose target - # equals another measure's canonical alias is rejected even - # when the constructed public aliases differ. The original - # rationale was that ``_ensure_aggregated_measure``'s alias- - # keyed dedup would still collapse the two aggregates under - # subtle processing-order conditions. Keep both directions of - # the comparison so the guard runs symmetrically. Only - # meaningful when both sides are ``AggregatedMeasureRef`` - # — non-Agg measures have ``canonical = ""`` (empty sentinel) - # which never matches a real ``qf.name``. - if qf_pre.name and canonical_other and qf_pre.name == canonical_other: - raise ValueError( - f"Measure '{qf_pre.formula}' renamed to " - f"'{qf_pre.name}', but that name collides with the " - f"canonical alias of another query measure " - f"'{qf_other.formula}' (also canonicalises to " - f"'{qf_pre.name}'). Two distinct aggregates would " - f"otherwise be silently merged into one column. " - f"Pick a different `name`, or rename the other " - f"measure too." - ) - if qf_other.name and canonical_pre and qf_other.name == canonical_pre: - raise ValueError( - f"Measure '{qf_other.formula}' renamed to " - f"'{qf_other.name}', but that name collides with the " - f"canonical alias of another query measure " - f"'{qf_pre.formula}' (also canonicalises to " - f"'{qf_other.name}'). Two distinct aggregates would " - f"otherwise be silently merged into one column. " - f"Pick a different `name`, or rename the other " - f"measure too." - ) - _surfaces.append((qf_pre, sp_pre, public_pre, short_pre, canonical_pre)) - - # Process each query field - for qfield in query.measures or []: - spec = parse_formula( - qfield.formula, - extra_agg_names=custom_agg_names, - named_measures=named_measures, - ) - # DEV-1443 (Codex Finding 2): block the latent bug where an - # alias-form filter against a renamed measure silently resolves to - # the source column instead of the HAVING aggregate. Reject up - # front rather than letting strict resolution misfire downstream. - if qfield.name and qfield.name in _source_column_names: - raise ValueError( - f"Query measure name '{qfield.name}' collides with a source " - f"column on model '{model.name}'. Pick a different name " - f"(or omit `name` to use the canonical alias). Filters and " - f"ORDER BY would otherwise bind to the source column " - f"instead of the renamed aggregate." - ) - field_name = _public_field_name(qfield) - - if isinstance(spec, AggregatedMeasureRef): - # New colon syntax: "revenue:sum", "*:count", etc. - canonical_name = _canonical_agg_name( - measure_name=spec.measure_name, - aggregation_name=spec.aggregation_name, - agg_args=spec.agg_args, - agg_kwargs=spec.agg_kwargs, - ) - if ( - field_name == qfield.formula.replace(" ", "_").replace("/", "_div_").replace(":", "_").replace("*", "") - and qfield.formula.strip() not in named_measures - ): - field_name = canonical_name - - if "." in spec.measure_name and spec.measure_name != "*": - # DEV-1449: cross-model agg ref against a virtual stage - # whose inner stage already projected the flat alias → - # emit a local re-aggregated measure instead of a CTE. - # - # Codex review on PR #137 (rounds 2+4): key the - # duplicate-canonical guard on the RESOLVED flat column, - # not the raw `spec.measure_name`. Otherwise two qfields - # like `orders.customers.revenue:sum` (Candidate A - # strips `orders`) and `customers.revenue:sum` - # (Candidate B) — which both land on - # `customers__revenue_sum` — slip past the guard and - # corrupt the projection. - intercept_candidate = _intercept_candidate_for_cross_model( - ref=spec, - ) - if intercept_candidate is not None: - flat_with_agg, outer_agg = intercept_candidate - cross_canon_key = ( - "agg-intercept", flat_with_agg, outer_agg, - ) - if cross_canon_key in user_declared_canon_keys: - prior_name = user_declared_canon_keys[cross_canon_key] - this_name = qfield.name or canonical_name - if prior_name != this_name: - raise ValueError( - f"Measure '{qfield.formula}' (surfacing as " - f"'{this_name}') canonicalises to the same " - f"cross-stage aggregation as an earlier query " - f"measure (surfacing as '{prior_name}'). Two " - f"user-declared measures with the same " - f"canonical aggregation would otherwise " - f"collapse into one column, leaving the " - f"second name with no backing aggregate. " - f"Pick a single name, or drop the duplicate." - ) - # CodeRabbit review on PR #137 round 4: refuse a - # rename target that collides with another query - # measure's canonical alias. `_ensure_aggregated_measure`'s - # alias-keyed dedup would otherwise silently collapse - # two distinct aggregates onto the renamed first one. - # Mirrors the guard the standard local-agg branch - # runs below. - if qfield.name and qfield.name != canonical_name: - for qf_other in (query.measures or []): - if qf_other is qfield: - continue - spec_other = parse_formula( - qf_other.formula, - extra_agg_names=custom_agg_names, - named_measures=named_measures, - ) - if not isinstance(spec_other, AggregatedMeasureRef): - continue - other_canonical = _canonical_agg_name( - measure_name=spec_other.measure_name, - aggregation_name=spec_other.aggregation_name, - agg_args=spec_other.agg_args, - agg_kwargs=spec_other.agg_kwargs, - ) - if other_canonical == qfield.name: - raise ValueError( - f"Measure '{qfield.formula}' renamed to " - f"'{qfield.name}', but that name " - f"collides with the canonical alias of " - f"another query measure " - f"'{qf_other.formula}' (also " - f"canonicalises to '{qfield.name}'). " - f"Two distinct aggregates would " - f"otherwise be silently merged into " - f"one column. Pick a different `name`, " - f"or rename the other measure too." - ) - await _ensure_aggregated_measure( - alias_key=field_name, - measure_name=flat_with_agg, - aggregation_name=outer_agg, - agg_args=spec.agg_args, - agg_kwargs=spec.agg_kwargs, - ) - local_alias = known_aliases[field_name] - # Codex round 10: mark the EnrichedMeasure as - # intercept-produced (see - # `_try_intercept_cross_model_as_local`). - for em in measures: - if em.alias == local_alias: - em.from_cross_model_intercept = True - break - user_declared_canon_keys[cross_canon_key] = ( - qfield.name or canonical_name - ) - # Codex review round 3 on PR #137: the intercept - # builds the EnrichedMeasure against the flat stage - # column (e.g. `customers__revenue_sum`), so - # `_ensure_aggregated_measure` produces an internal - # alias like `s1.customers__revenue_sum_sum`. The - # cross-model CTE path (the non-intercept fallback) - # produces `s1.customers.revenue_sum` instead, and - # that's the alias users expect for colon-form - # filter / ORDER BY refs and as the public result - # key. Unify by ALWAYS renaming the intercepted - # measure to the cross-model canonical alias (or the - # user-supplied `qfield.name` when set). - target_name = qfield.name or canonical_name - target_alias = f"{model_name_str}.{target_name}" - if target_alias != local_alias: - prev_alias = local_alias - for em in measures: - if em.alias == prev_alias: - # Codex review on PR #137 (rounds 5+6): - # `em.name` becomes the wrapped virtual - # model's `Column.name` when this stage - # is the inner of a downstream stage — - # `Column.name` forbids dots, and a - # third stage's intercept looks up the - # flat form a single-stage cross-model - # query would produce (e.g. - # `customers__revenue_sum`). So in the - # unrenamed case, derive `em.name` from - # the dotted cross-model canonical by - # replacing dots with `__` (matching - # `_alias_to_short`'s convention), - # NOT keep the doubled-sum internal - # form `_ensure_aggregated_measure` - # produced. The dotted form lives only - # on `em.alias` (public result key, - # filter / ORDER BY remap). - if qfield.name and qfield.name != canonical_name: - em.name = qfield.name - else: - em.name = canonical_name.replace(".", "__") - em.alias = target_alias - break - known_aliases[target_name] = target_alias - known_aliases[canonical_name] = target_alias - # DEV-1444 provenance merge: any canonical key - # currently pointing at the pre-rename alias must - # follow the rename. - for k, v in list(measure_canonical_key_to_alias.items()): - if v == prev_alias: - measure_canonical_key_to_alias[k] = target_alias - # canonical_to_user_name only fires when the - # user explicitly renamed via qfield.name; the - # auto-rename to cross-model canonical doesn't - # change the user-visible name. - if qfield.name and qfield.name != canonical_name: - canonical_to_user_name[canonical_name] = qfield.name - # Codex review on PR #137 round 8: register the - # dotted canonical name as a field-name alias so - # ORDER BY's qualified-match branch - # (generator._resolve_order_column) can resolve - # ``order=[{"column":"customers.revenue:sum"}]`` - # to the projection alias instead of falling - # through to a non-existent - # ``customers.revenue_sum`` bare column. - field_name_aliases[canonical_name] = target_alias - surfaced_alias = target_alias - # Propagate qfield metadata onto the - # created-or-reused EnrichedMeasure (matches the - # standard local-agg branch handling). - for em in measures: - if em.alias == surfaced_alias: - if qfield.label is not None: - em.label = qfield.label - if qfield.type is not None: - em.type = qfield.type - break - _mark_user_declared(surfaced_alias) - user_projection.append(surfaced_alias) - continue - # Cross-model aggregated measure - cm = await resolve_cross_model_measure( - spec_name=spec.measure_name, - field_name=field_name, - model=model, - query=query, - dimensions=dimensions, - time_dimensions=time_dimensions, - label=qfield.label, - named_queries=named_queries, - aggregation_name=spec.aggregation_name, - agg_kwargs=spec.agg_kwargs, - ) - # DEV-1361: propagate declared result type into the inner - # EnrichedMeasure so _build_combined wraps the agg in CAST. - if qfield.type is not None: - cm.measure.type = qfield.type - # DEV-1444: this CrossModelMeasure corresponds to a user- - # declared qfield, so mark it as such and surface its alias - # in the public projection. - cm.user_declared = True - cross_model_measures.append(cm) - # DEV-1448: when the user supplies an explicit ``name`` on a - # cross-model measure spec, surface it as the - # CrossModelMeasure's outer handle so the public projection - # and downstream nested stages emit the user's chosen alias - # instead of the canonical ``.._`` - # form. Only the **leaf** of the dotted path is swapped to - # the user name; the hop path is preserved (e.g. - # ``customers.regions.population:sum`` + ``name="region_pop"`` - # surfaces as ``orders.customers.regions.region_pop``). This - # matches the dot-syntax convention every other multi-hop - # caller-facing key in SLayer uses. Downstream-stage virtual - # models then use the bare ``cm.name`` (no ``__`` flattening) - # via a special-case in ``_query_as_model`` so callers can - # reference the user's chosen name directly. Cross-model - # canonicals always contain dots and ``ModelMeasure.name`` - # rejects dots, so the ``!= canonical_name`` guard is - # structurally true when ``qfield.name`` is supplied; we - # keep the explicit check for forward-compat. Filter / - # ORDER BY remap of the colon-form ``.:`` - # is intentionally NOT wired up here (DEV-1445 territory); - # ``known_aliases[qfield.name]`` only registers the user - # alias so user-alias-form filters / ORDER BY resolve via - # the existing alias-lookup path. - if qfield.name and qfield.name != canonical_name: - hop_path = spec.measure_name.rsplit(".", 1)[0] - user_alias = f"{model_name_str}.{hop_path}.{qfield.name}" - cm.alias = user_alias - cm.name = qfield.name - known_aliases[qfield.name] = user_alias - user_projection.append(cm.alias) - continue - - # DEV-1444 (Codex review on PR #134): refuse two user- - # declared qfields with the same canonical aggregation. The - # provenance-merge index would otherwise reuse the first - # alias for the second qfield, surfacing ``user_projection`` - # entries that no EnrichedMeasure backs. - qfield_canon_key = ( - "agg", - spec.measure_name, - spec.aggregation_name, - tuple(spec.agg_args), - tuple(sorted(spec.agg_kwargs.items())), - ) - if qfield_canon_key in user_declared_canon_keys: - prior_name = user_declared_canon_keys[qfield_canon_key] - this_name = qfield.name or canonical_name - if prior_name != this_name: - raise ValueError( - f"Measure '{qfield.formula}' (surfacing as " - f"'{this_name}') canonicalises to the same " - f"aggregation as an earlier query measure " - f"(surfacing as '{prior_name}'). Two user-declared " - f"measures with the same canonical aggregation " - f"would otherwise collapse into one column, " - f"leaving the second name with no backing " - f"aggregate. Pick a single name, or drop the " - f"duplicate." - ) - user_declared_canon_keys[qfield_canon_key] = ( - qfield.name or canonical_name - ) - - await _ensure_aggregated_measure( - alias_key=canonical_name, - measure_name=spec.measure_name, - aggregation_name=spec.aggregation_name, - agg_args=spec.agg_args, - agg_kwargs=spec.agg_kwargs, - ) - # When the user supplies an explicit ``name`` on the measure spec, - # surface it as the EnrichedMeasure's name/alias so downstream - # stages (and the wrap subquery) emit the user's chosen alias - # instead of the canonical ``col_agg`` form. The canonical alias - # remains resolvable via known_aliases for inline references. - if qfield.name and qfield.name != canonical_name: - # DEV-1443 (Codex review on PR #133): if the canonical alias - # itself shadows a source ``Column`` on the model, the colon- - # form filter ``col:agg N`` is ambiguous — the remap - # would resolve to the user alias, while strict resolution - # would resolve a literal ``col_agg`` reference to the source - # column. Refuse the query at construction time rather than - # silently picking one and producing surprising SQL. - if canonical_name in _source_column_names: - raise ValueError( - f"Measure '{qfield.formula}' renamed to '{qfield.name}', " - f"but model '{model.name}' has a source column named " - f"'{canonical_name}' that shadows the canonical alias of " - f"this aggregation. Filters / ORDER BY using " - f"'{qfield.formula}' would be ambiguous. Pick a different " - f"`name` (so the canonical alias is unused), rename the " - f"source column, or reference the measure by its user " - f"alias '{qfield.name}'." - ) - # DEV-1448: the rename-vs-other-canonical collision guard - # previously inlined here was lifted into the pre-pass at - # the top of this function so it runs symmetrically for - # local AND cross-model renames. See the pre-pass next to - # ``_seen_explicit_names``. - user_alias = f"{model_name_str}.{qfield.name}" - prev_alias = f"{model_name_str}.{canonical_name}" - for m in measures: - if m.alias == prev_alias: - m.name = qfield.name - m.alias = user_alias - break - known_aliases[qfield.name] = user_alias - known_aliases[canonical_name] = user_alias - # DEV-1444 provenance merge: any canonical key currently - # pointing at the pre-rename alias must follow the rename - # so later auto-extracted refs collapse onto the new alias. - for k, v in list(measure_canonical_key_to_alias.items()): - if v == prev_alias: - measure_canonical_key_to_alias[k] = user_alias - # DEV-1443: record the canonical → user-name mapping so - # query filters and ORDER BY items referencing the raw - # ``col:agg`` formula can be remapped to the user alias - # before resolution. - canonical_to_user_name[canonical_name] = qfield.name - # Register custom field name so ORDER BY can resolve it - if field_name != canonical_name and canonical_name in known_aliases: - field_name_aliases[field_name] = known_aliases[canonical_name] - - if spec.aggregation_name in ("first", "last") and last_agg_time_column is None: - raise ValueError( - f"Aggregation '{spec.aggregation_name}' on measure '{spec.measure_name}' " - f"requires a time column. Add a time dimension, use an explicit arg " - f"(e.g., '{spec.measure_name}:{spec.aggregation_name}(time_col)'), " - f"or set default_time_dimension on the model." - ) - if qfield.label: - target_name = qfield.name if (qfield.name and qfield.name != canonical_name) else canonical_name - for m in measures: - if m.name == target_name: - m.label = qfield.label - # DEV-1361: declared result type → wrap aggregation in CAST. - if qfield.type is not None: - target_name = qfield.name if (qfield.name and qfield.name != canonical_name) else canonical_name - for m in measures: - if m.name == target_name: - m.type = qfield.type - # DEV-1444: mark the surfaced EnrichedMeasure as user-declared - # and append its alias to the projection. - surfaced_alias = ( - f"{model_name_str}.{qfield.name}" - if (qfield.name and qfield.name != canonical_name) - else f"{model_name_str}.{canonical_name}" - ) - _mark_user_declared(surfaced_alias) - user_projection.append(surfaced_alias) - - else: - await _flatten_spec(spec, field_name) - if qfield.label: - alias = f"{model_name_str}.{field_name}" - for e in enriched_expressions: - if e.alias == alias: - e.label = qfield.label - for t in enriched_transforms: - if t.alias == alias: - t.label = qfield.label - # DEV-1361: declared result type → wrap arithmetic / transform - # expression in CAST at the outer SELECT. - if qfield.type is not None: - alias = f"{model_name_str}.{field_name}" - for e in enriched_expressions: - if e.alias == alias: - e.type = qfield.type - # Pure-transform measures (lag/lead/cumsum/...) end up in - # ``enriched_transforms``, not ``enriched_expressions``; - # propagate the declared type there too so the window-layer - # emitter can wrap in CAST. - for t in enriched_transforms: - if t.alias == alias: - t.type = qfield.type - # DEV-1444: mark the surfaced EnrichedExpression / EnrichedTransform - # (whichever the formula landed in) as user-declared. The inner - # hoisted measures created by _flatten_spec remain user_declared=False. - surfaced_alias = f"{model_name_str}.{field_name}" - _mark_user_declared(surfaced_alias) - user_projection.append(surfaced_alias) - - # --- Enrich ORDER BY formulas as hidden fields --- - for item in query.order or []: - if not item.raw_formula: - continue - spec = parse_formula( - item.raw_formula, - extra_agg_names=custom_agg_names, - named_measures=named_measures, - ) - if isinstance(spec, AggregatedMeasureRef): - canonical = _canonical_agg_name( - measure_name=spec.measure_name, - aggregation_name=spec.aggregation_name, - agg_args=spec.agg_args, - agg_kwargs=spec.agg_kwargs, - ) - else: - canonical = item.raw_formula.replace(" ", "_").replace("/", "_div_").replace( - ":", "_" - ).replace("*", "").replace("(", "_").replace(")", "").replace(",", "_") - # Only enrich if not already present from fields - if canonical not in known_aliases: - await _flatten_spec(spec, canonical) - # DEV-1443: when the canonical points at a measure renamed by the - # query, the user alias is the real column key in the projection. - # Setting the order item's column name to the canonical would send - # the generator's ``_resolve_order_by_column`` down the fallback - # branch (``{model_prefix}.{canonical}``), producing a reference - # to a column that does not exist. DEV-1444's provenance-merge - # also ensures any auto-extracted canonical-form ref resolves to - # the surfaced user alias through this map. - item.column.name = canonical_to_user_name.get(canonical, canonical) - - # --- Validate model filters --- - # DEV-1378: Mode A model filters get parsed via ``parse_sql_predicate`` - # (the SQL-mode validator) so arbitrary SQL function calls - # (``json_extract``, ``coalesce``, ``CASE WHEN``, dialect-specific - # operators) flow through unchanged. The construction-time validator - # at ``slayer/core/models.py:412`` already rejected DSL constructs. - measure_names_set = {m.name for m in measures} - parsed_model_filters: list[ParsedFilter] = [] - for mf in model.filters: - parsed_mf = parse_sql_predicate(mf) - for col in parsed_mf.columns: - if col in measure_names_set: - raise ValueError( - f"Model filter '{mf}' references measure '{col}'. " - f"Model filters can only reference table columns (WHERE). " - f"Use query-level filters for measure conditions." - ) - parsed_model_filters.append(parsed_mf) - - # --- Process filters --- - # Apply variable substitution to query-level (Mode-B) filters. Model-level - # Mode-A surfaces (SlayerModel.sql / .filters / Column.sql / .filter) are - # substituted upstream in the engine (``_substitute_model_sql_surfaces``, - # DEV-1625) before this function sees the model. - query_filters = list(query.filters or []) - if query.variables and query_filters: - # Mode-B filters are parsed by the Python-AST formula parser → escape - # string values Python-style so quotes/backslashes round-trip. - query_filters = [ - substitute_variables(filter_str=f, variables=query.variables, escape="python") - for f in query_filters - ] - - # DEV-1543: distinct_dimension_values=False rejects any measure - # reference in filters / order. This pass runs AFTER variable - # substitution (so a ``{var}`` revealing an aggregation is caught) - # and BEFORE ``extract_filter_transforms`` lifts transforms into - # hidden fields (so the original measure-reference shape is still - # visible). Pre-empts the construction-time check which is structural - # only. - if not query.distinct_dimension_values: - _reject_measure_references_for_raw_rows( - query=query, - query_filters=query_filters, - custom_agg_names=custom_agg_names, - named_measures=named_measures, - ) - - # Transform extraction runs only on Mode B (DSL) query filters. Model - # filters are SQL mode — they don't carry SLayer transforms (rejected - # at construction by ``parse_sql_predicate``) and don't go through - # ``_preprocess_like`` / ``_preprocess_agg_refs``. - processed_query_filters: list[str] = [] - ft_counter = [0] - for f_str in query_filters: - rewritten, extra_fields = extract_filter_transforms( - f_str, counter=ft_counter, extra_agg_names=custom_agg_names, - named_measures=named_measures, - ) - for name, formula in extra_fields: - spec = parse_formula( - formula, - extra_agg_names=custom_agg_names, - named_measures=named_measures, - ) - await _flatten_spec(spec, name) - processed_query_filters.append(rewritten) - - # Mode-tagged filter list, in WHERE order: model filters first, then - # query filters. Used by the windowed-column scan, ``_resolve_joins`` / - # ``_collect_needed_paths``, and the ordering of the final - # ``EnrichedQuery.filters`` list. - processed_filters_with_mode: list[tuple[str, str]] = ( - [(mf, "sql") for mf in model.filters] - + [(qf, "dsl") for qf in processed_query_filters] - ) - - has_first_or_last = any(m.aggregation in ("first", "last") for m in measures) - - # DEV-1369: a query filter that names a Column whose `sql` contains a - # window function used to auto-promote to a post-aggregation outer - # WHERE. The escape hatch is removed — the rank-family transforms - # (`rank` / `percent_rank` / `dense_rank` / `ntile`) cover top-N - # filtering in pure DSL. Applied to both modes — neither standard SQL - # nor SLayer DSL allows window functions in WHERE. - _windowed_columns: dict[str, str] = { - c.name: c.sql for c in model.columns if c.sql and has_window_function(c.sql) - } - if _windowed_columns: - for f, _mode in processed_filters_with_mode: - for col_name in _windowed_columns: - if re.search(rf"(?) <= N`, " - f"`percent_rank(...)`, `dense_rank(...)`, `ntile(n=4, ...)`) " - f"or factor the column into a multi-stage source_queries " - f"model. The filter was: {f!r}" - ) - - # --- Resolve JOINs --- - resolved_joins = await _resolve_joins( - model=model, - model_name_str=model_name_str, - dimensions=dimensions, - time_dimensions=time_dimensions, - measures=measures, - cross_model_measures=cross_model_measures, - processed_filters=processed_filters_with_mode, - named_queries=named_queries, - resolve_join_target=resolve_join_target, - extra_agg_names=custom_agg_names, - dialect=dialect, - ) - - # Names that resolve at the query level (named measures, transforms, - # expressions) — pass through as legitimate filter targets even though - # they are not Columns / ModelMeasures on the source model. - _query_aliases: set[str] = set() - _query_aliases.update(m.name for m in measures if m.name) - _query_aliases.update(t.name for t in enriched_transforms if t.name) - _query_aliases.update(e.name for e in enriched_expressions if e.name) - # DEV-1448 (Codex review on PR #136): we previously added cross-model - # measure names to ``_query_aliases`` so a same-stage filter - # ``"cust_rev > 100"`` would pass strict resolution. That admission was - # half-baked — the SQL generator has no path to route the bare name to - # the cross-model CTE's output column ``"orders.customers.cust_rev"``, - # so it qualified the bare alias as ``orders.cust_rev`` (a column that - # doesn't exist on the base table) and shipped invalid SQL. Until the - # full cross-model filter remap lands in DEV-1445, the bare user alias - # is NOT a valid filter / ORDER BY target on a renamed cross-model - # measure — strict resolution must reject it cleanly rather than - # silently produce broken SQL. The rename remains useful for the - # projection alias and the downstream-stage virtual model column, - # which is the ticket's actual repro shape. - # DEV-1378: model filters and query filters resolve under different - # strictness rules. Model filters are SQL-mode and may reference any - # column on the underlying table even when not declared as a - # ``Column`` (``strict=False`` — see the comment block at - # ``resolve_filter_columns`` lines ~1652-1657). Query filters are - # DSL-mode and must strictly resolve to a Column / ModelMeasure / - # custom aggregation / canonical agg alias / query-level alias - # (``strict=True``). Run the resolver twice and concatenate. - resolved_model_filters = await resolve_filter_columns( - parsed_filters=parsed_model_filters, - model=model, - model_name=model_name_str, - resolve_join_target=resolve_join_target, - named_queries=named_queries, - resolve_model=resolve_model, - dialect=dialect, - strict=False, - drop_if_unresolved=False, - query_aliases=set(), - ) - # DEV-1443: pre-pass remap of canonical agg aliases → user aliases for - # query filters. Applied here (and ONLY here) so model filters and - # ``Column.filter`` predicates — which never carry colon-syntax - # synthesized aliases anyway — are left untouched. - parsed_query_filters_pre = [ - parse_filter(f, extra_agg_names=custom_agg_names) - for f in processed_query_filters - ] - for pf in parsed_query_filters_pre: - _remap_renamed_aliases_in_filter( - pf=pf, - canonical_to_user_name=canonical_to_user_name, - ) - resolved_query_filters = await resolve_filter_columns( - parsed_filters=parsed_query_filters_pre, - model=model, - model_name=model_name_str, - resolve_join_target=resolve_join_target, - named_queries=named_queries, - resolve_model=resolve_model, - dialect=dialect, - # Strict resolution for DSL query filters — bare names AND dotted - # paths must resolve. Rerooted CTEs may drop unresolved filters - # (DEV-1367) via ``drop_if_unresolved``. - strict=True, - drop_if_unresolved=drop_unreachable_filters, - query_aliases=_query_aliases, - ) - parsed_filters = list(resolved_model_filters) + list(resolved_query_filters) - - return EnrichedQuery( - model_name=model_name_str, - sql_table=model.sql_table, - sql=model.sql, - resolved_joins=resolved_joins, - dimensions=dimensions, - measures=measures, - time_dimensions=time_dimensions, - expressions=enriched_expressions, - transforms=enriched_transforms, - cross_model_measures=cross_model_measures, - last_agg_time_column=last_agg_time_column if has_first_or_last else None, - filters=classify_filters( - filters=parsed_filters, - measure_names={m.name for m in measures}, - computed_names=( - {t.name for t in enriched_transforms} - | {e.name for e in enriched_expressions} - ), - groupby_names={d.name for d in dimensions} | {td.name for td in time_dimensions}, - windowed_measure_names={m.name for m in measures if m.window}, - ), - order=query.order, - limit=query.limit, - offset=query.offset, - field_name_aliases=field_name_aliases, - user_projection=user_projection, - distinct_dimension_values=query.distinct_dimension_values, - ) - - -# --------------------------------------------------------------------------- -# Dimension / time resolution helpers -# --------------------------------------------------------------------------- - - -def _unpack_dim_resolution(result): - """Accept either ``Column`` or ``(Column, SlayerModel)`` from - ``resolve_dimension_via_joins`` so legacy test callbacks (which return a - plain Column or None) keep working alongside the engine's tuple form. - """ - if result is None: - return None, None - if isinstance(result, tuple): - return result[0], result[1] - return result, None - - -async def _maybe_expand( - *, - sql: str | None, - terminal_model: SlayerModel | None, - fallback_model: SlayerModel, - alias_path: str, - resolve_model, - named_queries: dict, - dialect: str, - is_root: bool = True, -) -> str | None: - """Run the column-SQL expander when we have what we need; otherwise - return ``sql`` unchanged. Lets tests that don't supply ``resolve_model`` - keep getting the legacy unexpanded behavior — production always supplies - it via the engine. - - ``is_root=False`` for cross-model dims/measures: the source has been - reached via a join path, so any further walks inside its derived - Column.sql must prefix the alias path (closes PR #89 alias-prefix bug). - """ - if not sql or resolve_model is None: - return sql - return await expand_derived_refs( - sql=sql, - model=terminal_model or fallback_model, - alias_path=alias_path, - resolve_model=resolve_model, - named_queries=named_queries, - dialect=dialect, - is_root=is_root, - ) - - -def resolve_via_stage_origin( - *, model: SlayerModel, parts: list[str], -) -> Column | None: - """DEV-1449: Resolve a dotted reference against a virtual stage - model produced by ``_query_as_model``. - - Returns the matching ``Column`` from ``model.columns``, or ``None`` - if ``model`` is not a virtual stage model OR no flat candidate - matches. The shared callee for the four cross-stage resolution - paths (dimensions, time dimensions, cross-model measures, filters) - when the standard join-walk doesn't apply. - - Lookup procedure (both candidates are first-class — neither is a - "fallback" semantically; the order is just deterministic precedence - for the rare collision case): - - Candidate A — ancestor-stripped flat: - If ``parts[0]`` matches the ``name`` of any ancestor in the - ``source_model_origin`` chain, drop it and ``__``-join the rest. - Candidate B — full flat: - ``__``-join all ``parts`` verbatim. - - Try A first; if no match, try B. Returns the first match. - - Why both: ``_alias_to_short`` (query_engine.py) strips only the - immediate inner-model prefix at each ``_query_as_model`` call. At - depth 1, the original source-model name IS the immediate prefix, so - A matches. At depth >= 2 with a source-prefixed user ref, the - ancestor lives inside the flat column name; only B matches. - """ - if model.source_model_origin is None: - return None - ancestor_names: set[str] = set() - cursor = model.source_model_origin - while cursor is not None: - ancestor_names.add(cursor.name) - cursor = cursor.parent - # Candidate A — ancestor-stripped. - if parts and parts[0] in ancestor_names: - stripped = parts[1:] - if stripped: - col = model.get_column("__".join(stripped)) - if col is not None: - return col - # Candidate B — full-flat. - if parts: - col = model.get_column("__".join(parts)) - if col is not None: - return col - return None - - -async def _resolve_dotted_dim_with_stage_fallback( - *, - dim_ref_model: str, - dim_ref_name: str, - model: SlayerModel, - model_name_str: str, - named_queries: dict, - resolve_dimension_via_joins, -) -> "tuple[Column | None, SlayerModel | None, str]": - """Resolve a dotted dim / time-dim reference for one query field. - - Shared by ``_resolve_dimensions`` and ``_resolve_time_dimensions`` - (DEV-1449 / Sonar S3776). Tries the standard join-walk first; if - that returns nothing AND ``model`` is a virtual stage produced by - ``_query_as_model``, tries the stage-origin resolver. On stage-origin - miss, falls through to today's lenient behavior (returns ``None`` - dim_def + the join-walk's `__`-flattened effective_model); cross-model - CTE re-rooting depends on that fall-through. - """ - parts = dim_ref_model.split(".") + [dim_ref_name] - raw = await resolve_dimension_via_joins( - model=model, - parts=parts, - named_queries=named_queries, - ) - dim_def, terminal_model = _unpack_dim_resolution(raw) - effective_model = "__".join(dim_ref_model.split(".")) - if dim_def is None and model.source_model_origin is not None: - stage_col = resolve_via_stage_origin(model=model, parts=parts) - if stage_col is not None: - dim_def = stage_col - terminal_model = model - effective_model = model_name_str # local to the virtual stage - return dim_def, terminal_model, effective_model - - -async def _resolve_dimensions( - query: SlayerQuery, - model: SlayerModel, - model_name_str: str, - named_queries: dict, - resolve_dimension_via_joins, - resolve_model=None, - dialect: str = "postgres", -) -> list[EnrichedDimension]: - dimensions = [] - for dim_ref in query.dimensions or []: - terminal_model: SlayerModel | None = None - is_local = dim_ref.model is None - if is_local: - dim_def = model.get_column(dim_ref.name) - effective_model = model_name_str - terminal_model = model - else: - dim_def, terminal_model, effective_model = ( - await _resolve_dotted_dim_with_stage_fallback( - dim_ref_model=dim_ref.model, - dim_ref_name=dim_ref.name, - model=model, - model_name_str=model_name_str, - named_queries=named_queries, - resolve_dimension_via_joins=resolve_dimension_via_joins, - ) - ) - # Grouping by an opaque column emits SQL the database rejects (no - # equality operator), so fail here with an actionable message instead - # of surfacing a raw driver error. Projecting such a column is fine — - # only its use as a GROUP BY / DISTINCT key is refused. - if dim_def is not None and dim_def.type.is_opaque: - db_type = getattr(dim_def, "db_type", None) - described = f" (database type {db_type!r})" if db_type else "" - raise ValueError( - f"Column '{dim_ref.full_name}'{described} cannot be used as a " - f"dimension: its type does not support the grouping this query " - f"requires. Define a derived column that extracts a comparable " - f"value instead, e.g. sql=\"payload->>'status'\" with type TEXT." - ) - expanded_sql = await _maybe_expand( - sql=dim_def.sql if dim_def else None, - terminal_model=terminal_model, - fallback_model=model, - alias_path=effective_model, - resolve_model=resolve_model, - named_queries=named_queries, - dialect=dialect, - is_root=is_local, - ) - dimensions.append( - EnrichedDimension( - name=dim_ref.name, - sql=expanded_sql, - type=dim_def.type if dim_def else DataType.TEXT, - alias=f"{model_name_str}.{dim_ref.full_name}", - model_name=effective_model, - label=dim_ref.label or (dim_def.label if dim_def else None), - format=dim_def.format if dim_def else None, - ) - ) - return dimensions - - -async def _resolve_time_dimensions( - query: SlayerQuery, - model: SlayerModel, - model_name_str: str, - named_queries: dict, - resolve_dimension_via_joins, - resolve_model=None, - dialect: str = "postgres", -) -> list[EnrichedTimeDimension]: - time_dimensions = [] - for td in query.time_dimensions or []: - terminal_model: SlayerModel | None = None - is_local = td.dimension.model is None - if is_local: - dim_def = model.get_column(td.dimension.name) - td_model_name = model_name_str - terminal_model = model - else: - dim_def, terminal_model, td_model_name = ( - await _resolve_dotted_dim_with_stage_fallback( - dim_ref_model=td.dimension.model, - dim_ref_name=td.dimension.name, - model=model, - model_name_str=model_name_str, - named_queries=named_queries, - resolve_dimension_via_joins=resolve_dimension_via_joins, - ) - ) - expanded_sql = await _maybe_expand( - sql=dim_def.sql if dim_def else None, - terminal_model=terminal_model, - fallback_model=model, - alias_path=td_model_name, - resolve_model=resolve_model, - named_queries=named_queries, - dialect=dialect, - is_root=is_local, - ) - time_dimensions.append( - EnrichedTimeDimension( - name=td.dimension.name, - sql=expanded_sql, - granularity=td.granularity, - date_range=td.date_range, - alias=f"{model_name_str}.{td.dimension.full_name}", - model_name=td_model_name, - label=td.label or (dim_def.label if dim_def else None), - ) - ) - return time_dimensions - - -def _resolve_time_alias( - time_dimensions: list[EnrichedTimeDimension], - query: SlayerQuery, - model: SlayerModel, -) -> str | None: - if len(time_dimensions) == 1: - return time_dimensions[0].alias - elif len(time_dimensions) > 1: - if query.main_time_dimension: - return f"{model.name}.{query.main_time_dimension}" - elif model.default_time_dimension: - td_names = {td.name for td in time_dimensions} - if model.default_time_dimension in td_names: - return f"{model.name}.{model.default_time_dimension}" - # No fallback to default_time_dimension without explicit time_dimensions — - # transforms require a time_dimensions entry so the column is in the base CTE. - return None - - -def _resolve_last_agg_time( - query: SlayerQuery, - model: SlayerModel, - dimensions: list[EnrichedDimension], - time_dimensions: list[EnrichedTimeDimension], -) -> str | None: - if query.main_time_dimension: - mtd = query.main_time_dimension - if "." not in mtd: - mtd = f"{model.name}.{mtd}" - return mtd - - def _qualified(model_name: str, sql: str | None, name: str) -> str: - # Once derived-ref expansion has run, `sql` may already be qualified - # (e.g. ``orders.created_at`` instead of bare ``created_at``); don't - # double-prefix in that case. - expr = sql or name - if "." in expr: - return expr - return f"{model_name}.{expr}" - - for d in dimensions: - if d.type in (DataType.TIMESTAMP, DataType.DATE): - return _qualified(d.model_name, d.sql, d.name) - if time_dimensions: - td = time_dimensions[0] - return _qualified(td.model_name, td.sql, td.name) - if query.filters: - time_dim_names = {c.name for c in model.columns if c.type in (DataType.TIMESTAMP, DataType.DATE)} - for f_str in query.filters or []: - for td_name in time_dim_names: - if td_name in f_str: - return f"{model.name}.{td_name}" - if model.default_time_dimension: - return f"{model.name}.{model.default_time_dimension}" - return None - - -# --------------------------------------------------------------------------- -# JOIN resolution -# --------------------------------------------------------------------------- - - -def _add_with_prefixes(segments: list[str], paths: set[tuple[str, ...]]) -> None: - """Add ``segments[:1], segments[:2], …, segments`` to ``paths``.""" - for i in range(1, len(segments) + 1): - paths.add(tuple(segments[:i])) - - -def _raise_column_cycle( - visited: tuple[tuple[str, str], ...], key: tuple[str, str], -) -> None: - """Raise a deterministic ``Circular column reference`` error matching - the chain format used by ``expand_derived_refs``. - """ - cycle_start = visited.index(key) - cycle = (*visited[cycle_start:], key) - chain = " → ".join(f"{m}.{c}" for m, c in cycle) - raise ValueError(f"Circular column reference detected: {chain}") - - -def _scan_sql_table_refs(*, sql: str, model_name: str, paths: set[tuple[str, ...]]) -> None: - """Regex-fallback scan: pick out ``
.`` shapes and add the - table prefix paths (skipping references to ``model_name`` itself). - """ - for match in _TABLE_COL_RE.finditer(sql): - segments = match.group(1).split("__") - if segments and segments[0] != model_name: - _add_with_prefixes(segments, paths) - - -def _process_node_for_paths( - *, - node: exp.Column, - model: SlayerModel, - paths: set[tuple[str, ...]], - visited: tuple[tuple[str, str], ...], - dialect: str | None = None, -) -> None: - """Resolve one ``exp.Column`` node into either a recursion into a - local derived column or a join-path-prefix add. - - Branches: - - multi-part qualifier (catalog/db) → ignore (outside SLayer's contract) - - bare identifier → recurse into a possibly-derived local column - - ``.`` → self-qualified local ref, recurse - - ``
.`` (table not the source model) → add the prefix path - """ - if node.args.get("db") or node.args.get("catalog"): - return - table_id = node.args.get("table") - if table_id is None: - _collect_paths_from_local_column_chain( - model=model, col_name=node.name, paths=paths, - visited=visited, dialect=dialect, - ) - return - segments = table_id.name.split("__") - if not segments: - return - if segments[0] == model.name: - _collect_paths_from_local_column_chain( - model=model, col_name=node.name, paths=paths, - visited=visited, dialect=dialect, - ) - return - _add_with_prefixes(segments, paths) - - -def _collect_paths_from_local_column_chain( - *, - model: SlayerModel, - col_name: str, - paths: set[tuple[str, ...]], - visited: tuple[tuple[str, str], ...] = (), - dialect: str | None = None, -) -> None: - """Walk the SQL of a *local* derived column on ``model`` to discover - the join paths its expression implies — recursing through references - to other derived columns on the same model. - - Closes DEV-1334. ``_collect_needed_paths`` previously only saw cross- - table aliases that already appeared verbatim in the *parsed-out filter - columns* (dotted refs like ``customers.region``). When a filter - referenced a *bare-named* derived column (e.g. ``is_eu = 1`` where - ``is_eu.sql`` references ``customers.region``), the chain was never - walked and the join was silently dropped. This helper closes that - gap by inspecting the column's SQL body. - - ``dialect`` is the active sqlglot dialect — passed to ``parse_one`` so - dialect-specific syntax in derived ``Column.sql`` parses correctly - (PR #96 review). - """ - col = model.get_column(col_name) - if col is None or _is_trivial_base(column=col): - return - sql = col.sql or "" - if not sql: - return - key = (model.name, col_name) - if key in visited: - _raise_column_cycle(visited, key) - next_visited = (*visited, key) - - try: - # DEV-1686: quote bare reserved-word qualifiers/leaves (a derived - # column referencing a reserved joined model, e.g. ``grant.amount``) - # so join-path discovery finds the ref instead of silently falling - # back to a ref-less ``Command`` parse (which would drop the JOIN). - parsed = sqlglot.parse_one( - prequote_reserved_identifiers(sql=sql, dialect=dialect), dialect=dialect - ) - except Exception: - _scan_sql_table_refs(sql=sql, model_name=model.name, paths=paths) - return - - for node in parsed.find_all(exp.Column): - _process_node_for_paths( - node=node, model=model, paths=paths, - visited=next_visited, dialect=dialect, - ) - - -def _collect_needed_paths( - model: SlayerModel, - dimensions: list[EnrichedDimension], - time_dimensions: list[EnrichedTimeDimension], - measures: list[EnrichedMeasure], - cross_model_measures: list, - processed_filters: list[tuple[str, str]], - extra_agg_names: frozenset | None = None, - dialect: str | None = None, -) -> set[tuple[str, ...]]: - """Extract ordered join-path tuples the query needs (including all prefixes). - - ``processed_filters`` is a list of ``(filter_text, mode)`` tuples - where ``mode`` is ``"sql"`` for Mode A (model-side) filters and - ``"dsl"`` for Mode B (query-side) filters; each is parsed by the - matching parser so model filters with arbitrary SQL functions - don't trip the DSL allowlist (DEV-1378). - """ - paths: set[tuple[str, ...]] = set() - - for d in dimensions: - if d.model_name != model.name: - _add_with_prefixes(d.model_name.split("__"), paths) - for td in time_dimensions: - if td.model_name != model.name: - _add_with_prefixes(td.model_name.split("__"), paths) - for cm in cross_model_measures: - paths.add((cm.target_model_name,)) - - # Scan SQL expressions for __-delimited table references - sql_refs = [d.sql for d in dimensions] + [td.sql for td in time_dimensions] + [m.sql for m in measures] - for sql_expr in sql_refs: - if sql_expr and "." in sql_expr: - for match in _TABLE_COL_RE.finditer(sql_expr): - _add_with_prefixes(match.group(1).split("__"), paths) - - # Scan filters for column references — dotted refs add their join - # path directly; bare-name refs to derived local columns trigger a - # walk of the column's SQL chain (DEV-1334). - for f_str, mode in processed_filters: - if mode == "sql": - parsed_f = parse_sql_predicate(f_str) - else: - parsed_f = parse_filter(f_str, extra_agg_names=extra_agg_names) - for col in parsed_f.columns: - _scan_filter_column_ref(model=model, col=col, paths=paths, dialect=dialect) - - # Scan measure filter columns. For column-level ``filter=`` attributes - # ``resolve_filter_columns`` may store the fully-expanded SQL fragment - # rather than the original column name (when the filter references a - # bare-named derived column whose own sql is non-trivial — DEV-1334). - # ``_scan_filter_column_ref`` distinguishes the three shapes (bare name, - # dotted ref, expanded SQL) and routes each accordingly. - for m in measures: - for col in m.filter_columns: - _scan_filter_column_ref(model=model, col=col, paths=paths, dialect=dialect) - - return paths - - -def _scan_filter_column_ref( - *, - model: SlayerModel, - col: str, - paths: set[tuple[str, ...]], - dialect: str | None = None, -) -> None: - """Route one entry from a parsed filter's column list to the right - path-discovery branch. - - Three shapes occur: - - **bare name** (``"is_eu"``): a reference to a local column. Walk - the column's SQL chain to find any cross-table refs it implies. - - **identifier dotted ref** (``"customers.region"``, - ``"customers.regions.name"``): a join-path-qualified reference — - add the prefix path directly. - - **expanded SQL fragment** (``"CASE WHEN customers.region = 'EU' …"``): - arises when ``resolve_filter_columns`` stores the inlined SQL of a - bare-name reference to a derived column. Scan via the same - ``_TABLE_COL_RE`` regex that handles ``EnrichedMeasure.sql``. - """ - if "." not in col: - _collect_paths_from_local_column_chain( - model=model, col_name=col, paths=paths, dialect=dialect, - ) - return - if _looks_like_dotted_identifier_ref(col): - parts = col.split(".") - expanded: list[str] = [] - for part in parts[:-1]: - # Model filters convert dots to __; expand both forms. - expanded.extend(part.split("__")) - if expanded: - _add_with_prefixes(expanded, paths) - return - # Expanded SQL fragment. - for match in _TABLE_COL_RE.finditer(col): - table_alias = match.group(1) - segments = table_alias.split("__") - if segments and segments[0] != model.name: - _add_with_prefixes(segments, paths) - - -def _looks_like_dotted_identifier_ref(value: str) -> bool: - """True iff ``value`` is a chain of ``.``-joined identifiers — e.g. - ``customers.region``, ``customers.regions.name``. False for SQL - fragments containing parens, spaces, operators, or quotes. - """ - return bool(_DOTTED_IDENT_REF_RE.match(value)) - - -async def _resolve_joins( - model: SlayerModel, - model_name_str: str, - dimensions: list[EnrichedDimension], - time_dimensions: list[EnrichedTimeDimension], - measures: list[EnrichedMeasure], - cross_model_measures: list, - processed_filters: list[tuple[str, str]], - named_queries: dict, - resolve_join_target, - extra_agg_names: frozenset | None = None, - dialect: str | None = None, -) -> list[tuple]: - """Resolve only the JOINs the query actually needs by walking the join graph. - - Instead of relying on baked-in multi-hop joins, this walks each intermediate - model's own direct joins hop-by-hop to build the complete chain. - - ``dialect`` is the active sqlglot dialect; it propagates into the - derived-column SQL parser used to discover join paths (PR #96 review). - """ - needed_paths = _collect_needed_paths( - model=model, - dimensions=dimensions, - time_dimensions=time_dimensions, - measures=measures, - cross_model_measures=cross_model_measures, - processed_filters=processed_filters, - extra_agg_names=extra_agg_names, - dialect=dialect, - ) - if not needed_paths: - return [] - - # Sort shorter paths first so prefixes are resolved before extensions - sorted_paths = sorted(needed_paths, key=len) - - resolved_joins: dict[str, tuple] = {} # alias -> (table_sql, alias, condition) - resolved_models: dict[str, SlayerModel] = {} # model_name -> SlayerModel - - for path in sorted_paths: - alias = "__".join(path) - if alias in resolved_joins: - continue - - current_model = model - current_alias = model_name_str - - for i, segment in enumerate(path): - hop_alias = "__".join(path[: i + 1]) - if hop_alias in resolved_joins: - # Already resolved from a previous path prefix — advance - if segment in resolved_models: - current_model = resolved_models[segment] - current_alias = hop_alias - continue - - # Find a direct join on the current model - join = None - for j in current_model.joins: - if j.target_model == segment: - join = j - break - - if join is None: - break # No join found — remaining hops unresolvable - - # Resolve the target model - target_info = await resolve_join_target( - target_model_name=segment, - named_queries=named_queries, - ) - if target_info: - target_table, target_model_obj = target_info - else: - target_table = segment - target_model_obj = None - - if target_model_obj: - resolved_models[segment] = target_model_obj - - # Build join condition - join_conds = [] - for src_col, tgt_col in join.join_pairs: - join_conds.append(f"{current_alias}.{src_col} = {hop_alias}.{tgt_col}") - - resolved_joins[hop_alias] = (target_table, hop_alias, " AND ".join(join_conds), str(join.join_type)) - - # Advance to the resolved model for the next hop - if target_model_obj: - current_model = target_model_obj - current_alias = hop_alias - - return list(resolved_joins.values()) - - - - -# --------------------------------------------------------------------------- -# Filter processing -# --------------------------------------------------------------------------- - - -def _remap_renamed_aliases_in_filter( - *, - pf: ParsedFilter, - canonical_to_user_name: dict[str, str], -) -> None: - """DEV-1443: rewrite canonical-agg aliases in a parsed query filter - to the user-supplied alias when the same node renamed the measure. - - Eligibility: ``c in pf.synthesized_aliases`` — only remap names the - parser saw as colon-syntax in *this* filter. A literal column reference - (no colon syntax) is left alone since the parser would not have - synthesized it. - - Mutates ``pf.sql`` and ``pf.columns`` in place. ``synthesized_aliases`` - and ``agg_refs`` are left intact (they're parser provenance, not the - rendered SQL). - - Note: the case where ``canonical_name`` is also the name of a source - ``Column`` on the model is rejected up front at measure enrichment - (DEV-1443 Codex-review on PR #133). By the time this helper runs the - mapping is guaranteed not to alias a source column, so any - ``\\bcanonical\\b`` occurrence in ``pf.sql`` came from colon syntax - and is safe to rewrite — outside of quoted string literals. - - DEV-1443 (CodeRabbit thread on PR #133): the regex sub must not touch - string-literal contents. Mask single-quoted spans with placeholders - before applying the rewrite, then restore them. Otherwise - ``country:first = 'country_first'`` with the measure renamed to - ``primary_country`` becomes ``primary_country = 'primary_country'`` - and the filter compares the column to itself. - """ - if not canonical_to_user_name: - return - eligible = { - c: u for c, u in canonical_to_user_name.items() - if c in pf.synthesized_aliases - } - if not eligible: - return - # Mask single-quoted spans so the identifier sub below can't reach - # into string literals. ``_STRING_LITERAL_RE`` is the same pattern - # used elsewhere in the codebase for this purpose - # (slayer/core/formula.py). - literal_re = re.compile(r"'(?:[^'\\]|\\.)*'") - literals = literal_re.findall(pf.sql) - masked = literal_re.sub("\x00LIT\x00", pf.sql) - for canonical, user_name in eligible.items(): - # Word-boundary regex mirroring ``_resolve_sql`` (line ~393) so - # canonical names embedded inside dotted paths or already-quoted - # identifiers are not rewritten. - masked = re.sub( - rf'(? None: - """DEV-1543: walk a single (substituted) query filter for measure - references and raise ``DistinctDimensionValuesError`` on any match. - Co-defined with :func:`_reject_measure_ref_in_order_item` below; - both are dispatched by :func:`_reject_measure_references_for_raw_rows`. - """ - from slayer.core.errors import DistinctDimensionValuesError - from slayer.core.formula import parse_filter - - masked = _RAW_ROW_STR_LIT_RE.sub("''", raw_filter) - - # Transform-call detection FIRST. ``extract_filter_transforms`` is - # robust against filter shapes ``parse_filter`` rejects (e.g. - # ``rank(amount:sum) <= 5``). - try: - _, lifted = extract_filter_transforms( - masked, - counter=[0], - extra_agg_names=custom_agg_names, - named_measures=named_measures, - ) - except Exception: - lifted = [] - if lifted: - raise DistinctDimensionValuesError( - f"distinct_dimension_values=False rejects measure references. " - f"Filter {raw_filter!r} contains a transform call " - f"({lifted[0][1]}). {_RAW_ROW_FIX_HINT}" - ) - - # Colon-aggregation + saved-measure detection via ``parse_filter``. - try: - parsed = parse_filter(masked, extra_agg_names=custom_agg_names) - except Exception: - # The real parser raises downstream with a useful message tied - # to the original filter text; don't pre-empt. - return - if parsed.agg_refs: - ref = parsed.agg_refs[0] - raise DistinctDimensionValuesError( - f"distinct_dimension_values=False rejects measure references. " - f"Filter {raw_filter!r} contains an aggregation " - f"({ref.measure_name}:{ref.aggregation_name}). " - f"{_RAW_ROW_FIX_HINT}" - ) - for col in parsed.columns: - if col in named_measures: - raise DistinctDimensionValuesError( - f"distinct_dimension_values=False rejects measure references. " - f"Filter {raw_filter!r} references saved ModelMeasure " - f"{col!r}. {_RAW_ROW_FIX_HINT}" - ) - - -def _reject_measure_ref_in_order_item( - *, - item: OrderItem, - custom_agg_names: frozenset, - named_measures: dict[str, str], -) -> None: - """DEV-1543: walk a single ``OrderItem`` for measure references. - - Covers ``raw_formula`` shapes (``AggregatedMeasureRef``, - ``TransformField``, ``MixedArithmeticField``, and ``ArithmeticField`` - when its ``agg_refs`` is non-empty — i.e. arithmetic OVER - aggregations like ``revenue:sum / *:count``; a scalar arithmetic - formula like ``amount + 1`` is fine in raw-row mode) plus the - bare-name case where ``item.column.name`` resolves to a saved - ``ModelMeasure``. - """ - from slayer.core.errors import DistinctDimensionValuesError - from slayer.core.formula import ( - AggregatedMeasureRef as _AggRef, - ArithmeticField as _ArithField, - MixedArithmeticField as _MixedField, - TransformField as _TransformField, - parse_formula, - ) - - if item.raw_formula is not None: - try: - spec = parse_formula( - item.raw_formula, - extra_agg_names=custom_agg_names, - named_measures=named_measures, - ) - except Exception: - spec = None - # Direct aggregation / transform / mixed forms are always rejected. - is_agg_form = isinstance(spec, (_AggRef, _TransformField, _MixedField)) - # ArithmeticField is only an aggregation form when it actually - # carries aggregate refs (``revenue:sum / *:count``); a pure - # scalar arithmetic like ``amount + 1`` is fine in raw-row mode. - is_agg_arith = isinstance(spec, _ArithField) and bool(spec.agg_refs) - if is_agg_form or is_agg_arith: - raise DistinctDimensionValuesError( - f"distinct_dimension_values=False rejects measure references. " - f"Order item raw_formula={item.raw_formula!r} contains a " - f"measure / transform reference. {_RAW_ROW_FIX_HINT}" - ) - - # Bare-name resolution: OrderItem.column.name matches a saved measure. - col_name = item.column.name if item.column else None - if col_name and col_name in named_measures: - raise DistinctDimensionValuesError( - f"distinct_dimension_values=False rejects measure references. " - f"Order item column={col_name!r} resolves to a saved " - f"ModelMeasure on the source model. {_RAW_ROW_FIX_HINT}" - ) - - -def _reject_measure_references_for_raw_rows( - *, - query: SlayerQuery, - query_filters: list[str], - custom_agg_names: frozenset, - named_measures: dict[str, str], -) -> None: - """DEV-1543: when ``query.distinct_dimension_values is False``, reject - any measure reference in ``query.filters`` or ``query.order``. - - Hooks AFTER variable substitution and BEFORE - ``extract_filter_transforms`` / order-formula hoisting, so the - original measure-reference shape is still visible. The construction- - time check in ``SlayerQuery._validate_distinct_dimension_values`` is - structural only (``measures`` non-empty, dims+tds both empty); this - pass is the authoritative measure-reference catch. - - Dispatches per-filter and per-order-item to the focused helpers - above so each unit stays cognitively simple. - """ - for raw_filter in query_filters: - _reject_measure_ref_in_filter( - raw_filter=raw_filter, - custom_agg_names=custom_agg_names, - named_measures=named_measures, - ) - for item in query.order or []: - _reject_measure_ref_in_order_item( - item=item, - custom_agg_names=custom_agg_names, - named_measures=named_measures, - ) - - -def extract_filter_transforms( - filter_str: str, - counter: list[int] | None = None, - extra_agg_names: frozenset[str] | None = None, - named_measures: Mapping[str, str] | None = None, -) -> tuple: - """Extract transform function calls from a filter string. - - Returns (rewritten_filter, [(name, formula), ...]) where transform - calls are replaced with generated field names. - - Bare references to ``named_measures`` keys are inline-expanded before - transform extraction so that filters like ``change(aov) > 0`` work when - ``aov`` is a saved formula. - """ - import ast as _ast - - from slayer.core.formula import ( - _expand_named_measures, - _preprocess_agg_refs, - _preprocess_concat, - ) - - if counter is None: - counter = [0] - - if named_measures: - filter_str = _expand_named_measures(filter_str, named_measures) - # DEV-1336: reject raw window-function syntax (`OVER (...)`) before AST parsing. - # Without this, the AST parser fails on `over` and falls through to a - # confusing "Invalid filter syntax" error from parse_filter; here we surface - # a helpful error that points at SLayer's transforms / Column.sql. - if has_window_function(filter_str): - raise ValueError(f"Filter '{filter_str}' {WINDOW_IN_FILTER_ERROR}") - preprocessed = _rewrite_funcstyle_aggregations(filter_str, extra_agg_names) - funcstyle_rewritten = preprocessed # capture after funcstyle rewrite, before further preprocessing - # DEV-1378: rewrite SQL `||` to `<<` so AST parsing accepts the filter. - preprocessed = _preprocess_concat(preprocessed) - preprocessed = _preprocess_like(preprocessed) - # Preprocess colon syntax (e.g., "order_total:sum") into ast-safe placeholders - preprocessed, agg_refs = _preprocess_agg_refs( - formula=preprocessed, custom_agg_names=extra_agg_names or frozenset() - ) - # Build reverse map: placeholder → original colon form - _agg_reverse = { - ph: ( - f"{ref.measure_name}:{ref.aggregation_name}" - if not ref.agg_args and not ref.agg_kwargs - else f"{ref.measure_name}:{ref.aggregation_name}({', '.join(ref.agg_args + [f'{k}={v}' for k, v in ref.agg_kwargs.items()])})" - ) - for ph, ref in agg_refs.items() - } - - try: - tree = _ast.parse(preprocessed, mode="eval") - except SyntaxError: - return filter_str, [] - - transforms: list[tuple] = [] - - def _unmangle(s: str) -> str: - """Restore colon syntax from placeholders in unparsed formulas.""" - for ph, orig in _agg_reverse.items(): - s = s.replace(ph, orig) - return s - - def _replace(node): - if isinstance(node, _ast.Call) and isinstance(node.func, _ast.Name) and node.func.id in ALL_TRANSFORMS: - name = f"_ft{counter[0]}" - counter[0] += 1 - formula = _unmangle(_ast.unparse(node)) - transforms.append((name, formula)) - return _ast.Name(id=name, ctx=_ast.Load()) - if isinstance(node, _ast.BinOp): - node.left = _replace(node.left) - node.right = _replace(node.right) - elif isinstance(node, _ast.UnaryOp): - node.operand = _replace(node.operand) - elif isinstance(node, _ast.Compare): - node.left = _replace(node.left) - node.comparators = [_replace(c) for c in node.comparators] - elif isinstance(node, _ast.BoolOp): - node.values = [_replace(v) for v in node.values] - return node - - modified = _replace(tree.body) - if not transforms: - return funcstyle_rewritten, [] - return _unmangle(_ast.unparse(modified)), transforms - - -# DEV-1539: compound AST shapes that ALWAYS need an outer ``(...)`` -# wrap when their SQL is substituted into a filter context with a -# surrounding comparator. Checked **before** the atomic list below -# because in sqlglot 30.4.3 ``exp.And`` and ``exp.Or`` inherit from -# ``exp.Func`` — a single inverse-atomic check would mis-classify -# ``a AND b`` as atomic and skip the wrap (Codex finding). -_COMPOUND_FILTER_INLINE_TYPES: tuple = ( - exp.Binary, # arith / comparison - exp.Connector, # AND / OR - exp.Unary, # NOT, -x - exp.Predicate, # BETWEEN, IN, LIKE, IS, … -) - -# DEV-1539: AST shapes whose precedence is already unambiguous when -# substituted into a filter context. A ``Column.sql`` body whose root -# is one of these does NOT need an outer paren wrap. Used as the -# fallback after the compound-types check. -_ATOMIC_FILTER_INLINE_TYPES: tuple = ( - exp.Column, - exp.Literal, - exp.Func, # covers function calls, CAST, CASE, Anonymous, … - exp.Paren, # already self-wrapped - exp.Boolean, # TRUE / FALSE - exp.Null, -) - - -def _filter_inline_needs_paren_wrap(*, sql: str, dialect: str) -> bool: - """DEV-1539: decide whether an inlined ``Column.sql`` body needs an - outer ``(...)`` wrap when substituted into a filter's text. - - The wrap matters when the body is a multi-term / predicate - expression whose precedence is ambiguous against the surrounding - comparator. Atomic shapes — bare columns, literals, single - function calls, single CASE / CAST — are already unambiguous; - wrapping them adds noise without changing meaning. **Anything - else** (BinOp, BoolOp, ``NOT``, ``BETWEEN``, ``IN``, ``LIKE``, - ``IS``, …) needs wrapping. - - Parses ``sql`` once via sqlglot to determine the root AST shape. - The compound-type check fires first because in sqlglot 30.4.3 - ``exp.And`` / ``exp.Or`` inherit from ``exp.Func``; a single - inverse-atomic check would mis-classify ``a AND b`` as atomic. - Conservative on parse failure: returns ``True`` so the caller wraps - (errs on the side of correctness over noise). - """ - try: - # DEV-1686: prequote reserved qualifiers so a filter over a reserved - # joined model classifies correctly instead of conservatively wrapping. - tree = sqlglot.parse_one( - prequote_reserved_identifiers(sql=sql, dialect=dialect), dialect=dialect - ) - except Exception: # noqa: BLE001 — sqlglot raises a variety of error types - return True - if isinstance(tree, _COMPOUND_FILTER_INLINE_TYPES): - return True - return not isinstance(tree, _ATOMIC_FILTER_INLINE_TYPES) - - -async def resolve_filter_columns( - parsed_filters: list, - model: SlayerModel, - model_name: str, - resolve_join_target=None, - named_queries: dict = None, - resolve_model=None, - dialect: str = "postgres", - *, - strict: bool = False, - drop_if_unresolved: bool = False, - query_aliases: set[str] | None = None, -) -> list: - """Resolve filter column references through model dimensions/measures. - - When ``resolve_model`` is supplied, derived ``Column.sql`` expressions - are recursively expanded so chained derivations (cross-model or local) - yield fully-qualified physical-table SQL inside WHERE clauses. - - With ``strict=True`` (DSL-mode callers — query-level filters) any - bare name that doesn't resolve to a Column / ModelMeasure / custom - aggregation / canonical agg alias / query-level alias raises - ``ValueError``; the same applies on the dotted-path branch when the - head segment names no join target on the source model. With - ``strict=False`` (SQL-mode callers — ``Column.filter``, model-level - filters) unknown bare names pass through as references to - underlying-table columns. - - With ``strict=True`` and ``drop_if_unresolved=True`` (DEV-1367 — - used by the cross-model-measure rerooting path in - ``query_engine._build_rerooted_enriched``), unresolved bare names and - dotted paths cause the **entire filter** to be dropped from the - output rather than raising. The rerooting machinery inherits the - outer query's filter list; only the subset reachable from the - rerooted source applies, so this turns the would-raise into a clean - drop. With ``drop_if_unresolved=False`` (the default), unresolved - strict-mode references raise ``ValueError`` as documented above. - """ - import re as _re - - async def _expanded_sql_expr(*, sql_expr: str, owning_model: SlayerModel, - alias_path: str, is_root: bool) -> str: - """Expand derived references inside a filter's resolved SQL fragment.""" - if resolve_model is None: - return sql_expr - expanded = await expand_derived_refs( - sql=sql_expr, - model=owning_model, - alias_path=alias_path, - resolve_model=resolve_model, - named_queries=named_queries or {}, - dialect=dialect, - is_root=is_root, - ) - return expanded if expanded is not None else sql_expr - - out_filters: list = [] - for f in parsed_filters: - resolved_sql = f.sql - resolved_columns = [] - # DEV-1369: precise allowlist of synthesised aliases this filter - # introduced via colon syntax (e.g. ``revenue:sum`` → ``revenue_sum``, - # ``*:count`` → ``_count``). Strict-resolution checks against this - # set, replacing the prior permissive regex that matched any - # ``*_sum``-shaped name and let typos like ``made_up_sum`` through. - filter_synthesized_aliases = set(getattr(f, "synthesized_aliases", [])) - drop_this_filter = False - for col_name in dict.fromkeys(f.columns): - if "." not in col_name: - dim = model.get_column(col_name) - if dim: - sql_expr = dim.sql or col_name - if sql_expr.isidentifier(): - qualified = f"{model_name}.{sql_expr}" - else: - qualified = await _expanded_sql_expr( - sql_expr=sql_expr, - owning_model=model, - alias_path=model_name, - is_root=True, - ) - # DEV-1539: wrap the inlined non-bare Column.sql - # in outer parens so the precedence of any - # surrounding comparator is explicit. Mirrors the - # ``exp.Paren`` wrap that ``expand_derived_refs`` - # already applies to spliced derived bodies. Skip - # the wrap when the inlined body is already a - # single atomic expression (literal / column / - # function call) — its precedence is unambiguous - # and the wrap would only add noise. - if _filter_inline_needs_paren_wrap(sql=qualified, dialect=dialect): - qualified = f"({qualified})" - # DEV-1539: lambda replacement so backslashes inside - # ``qualified`` aren't interpreted as ``re`` escape - # sequences (which would silently halve ``\\`` or - # raise on a ``\1`` backref). - resolved_sql = _re.sub( - rf"(?.`` and skip the strict-error - # branches. - # - # Codex review on PR #137 round 7: this fallback was - # designed for DIMENSION refs (e.g. `customers.regions.name`) - # cross-stage. If the leaf looks like an aggregated - # canonical (``_`` / ``_``), the user - # is filtering on a re-aggregated MEASURE and the - # right SQL placement is HAVING over the projection - # alias, not WHERE on the inner flat column. The - # intercept-as-local path leaves cross-model measure - # filters in DEV-1445 territory (not yet - # auto-resolved); skip the fallback for those leaves - # so the standard strict-error fires rather than - # silently emitting a wrong WHERE. - # - # Lenient (`strict=False`) callers never see virtual - # stages because `_query_as_model` does not propagate - # inner-model `filters` to the wrapped model — so the - # resolver lives inside `if strict:` only. - # Codex review on PR #137 round 9: use parser - # provenance (`filter_synthesized_aliases`) to - # distinguish a colon-syntax-synthesized aggregate - # alias from a user-typed literal dim ref. The - # earlier suffix heuristic falsely blocked real - # dims whose leaf happened to end with an - # aggregation suffix (e.g. a dim literally named - # `customers.revenue_sum`). - is_synthesized_agg_alias = col_name in filter_synthesized_aliases - if ( - model.source_model_origin is not None - and not is_synthesized_agg_alias - ): - stage_col = resolve_via_stage_origin( - model=model, parts=path_parts + [dim_name], - ) - if stage_col is not None: - qualified = f"{model.name}.{stage_col.name}" - resolved_sql = _re.sub( - rf"(? None: - """Mutate one ParsedFilter to set is_post_filter / is_having flags. - - Order matters: post-filter classifications take precedence (computed - columns can only be referenced after the base aggregate is built); then - windowed-measure refs (their value lives in a downstream CTE so they - can't be HAVING); then plain non-windowed measure HAVING. - """ - if any(col in computed_names for col in f.columns): - f.is_post_filter = True - return - if any(col in windowed_measure_names for col in f.columns): - f.is_post_filter = True - return - if any(col in measure_names for col in f.columns): - f.is_having = True - for col in f.columns: - if col not in measure_names and col not in groupby_names: - raise ValueError( - f"Filter '{f.sql}' references measure and dimension '{col}', " - f"but '{col}' is not in the query's dimensions or time_dimensions. " - f"Add it to dimensions/time_dimensions or split into separate filters." - ) - - -def classify_filters( - filters: list, - measure_names: set, - computed_names: set | None = None, - groupby_names: set | None = None, - windowed_measure_names: set | None = None, -) -> list: - """Classify filters as WHERE, HAVING, or post-filter. - - Delegates per-filter classification to `_classify_one_filter` so this - function stays a flat for-loop. - """ - computed_names = computed_names or set() - groupby_names = groupby_names or set() - windowed_measure_names = windowed_measure_names or set() - for f in filters: - _classify_one_filter( - f, - measure_names=measure_names, - computed_names=computed_names, - groupby_names=groupby_names, - windowed_measure_names=windowed_measure_names, - ) - return filters diff --git a/slayer/engine/join_graph.py b/slayer/engine/join_graph.py index 8f628962..dcb09b53 100644 --- a/slayer/engine/join_graph.py +++ b/slayer/engine/join_graph.py @@ -7,7 +7,8 @@ INNER joins are kept symmetric by the storage layer (``slayer/storage/join_sync.py`` materialises a reverse ``B→A`` edge for every ``A→B`` INNER join) — the same invariant the query engine's own -``_walk_join_chain`` relies on. So a symmetric INNER pair appears here as +the query-time join walk in ``binding.py`` relies on. So a symmetric INNER +pair appears here as two directed edges and is therefore traversable in both directions, and every path this primitive emits is walkable by the engine at query time. diff --git a/slayer/engine/measure_expansion.py b/slayer/engine/measure_expansion.py new file mode 100644 index 00000000..fc370780 --- /dev/null +++ b/slayer/engine/measure_expansion.py @@ -0,0 +1,353 @@ +"""Stage 7b.2 (DEV-1450) — pre-bind ModelMeasure expansion. + +Pre-bind AST -> AST rewrite of a ``ParsedExpr`` tree: every ``Ref(name=X)`` +whose ``X`` resolves to a ``ModelMeasure`` (on the model or in +``extra_measures``) is replaced with the recursively-expanded +``ParsedExpr`` produced by ``parse_expr(measure.formula)``. + +Why pre-bind: the binder (``slayer.engine.binding``) raises +``UnknownReferenceError`` for bare measure names because measures are not +columns; running expansion before the binder sees the tree turns those +refs into binder-resolvable column / aggregation nodes. + +Eligibility matrix: + +* Eligible positions: root; ``Arith`` / ``UnaryOp`` / ``Cmp`` / ``BoolOp`` + operands; ``ScalarCall.args``; ``TransformCall.input`` / args / + kwarg values. +* Not eligible: ``DottedRef`` (cross-model dotted paths resolve through + the join graph, not through measure expansion); ``AggCall`` in any + position (``source`` / ``args`` / ``kwargs`` are column-level by + contract); function-name slots on ``TransformCall.op`` / + ``ScalarCall.name`` (those are strings, not ``Ref`` nodes, and would + never match the dispatch even if a measure with the same name + existed); ``Literal`` / ``StarSource``; ``SlayerQuery.order`` entries + (the caller does not pass order entries through this function — order + resolves declared slot names only at the planner layer). + +Recursion controls: + +* Depth limit configurable via ``SLAYER_MEASURE_EXPANSION_DEPTH`` env + var (default ``32``). An explicit ``depth_limit=`` kwarg wins. Exceeded + -> :class:`MeasureRecursionLimitError`. +* Per-chain cycle detection. A measure transitively referencing itself + raises :class:`MeasureCycleError` with the offending chain attached. + +Purity: input ``ParsedExpr`` nodes are frozen Pydantic models; the +function returns a fresh tree. + +Dormant in this commit. Stage 7b.6 (BoundExpr unification) and stage +7b.15 (engine cutover) wire this into the engine pipeline. +""" + +from __future__ import annotations + +import os +from typing import Any, Dict, Optional, Sequence, Tuple, get_args + +from slayer.core.errors import MeasureCycleError, MeasureRecursionLimitError +from slayer.core.models import ModelMeasure, SlayerModel +from slayer.engine.syntax import ( + AggCall, + Arith, + BoolOp, + Cmp, + DottedRef, + Literal, + ParsedExpr, + Ref, + ScalarCall, + StarSource, + TransformCall, + UnaryOp, + parse_expr, +) + +_DEFAULT_DEPTH = 32 +_DEPTH_ENV_VAR = "SLAYER_MEASURE_EXPANSION_DEPTH" + +# Authoritative ParsedExpr node-type tuple, derived from the union in +# slayer.engine.syntax so new node types added there are automatically +# walked by `_maybe_walk` without a silent skip. +_PARSED_EXPR_TYPES: Tuple[type, ...] = get_args(ParsedExpr) + + +def expand_model_measures( + *, + expr: ParsedExpr, + model: SlayerModel, + extra_measures: Sequence[ModelMeasure] = (), + depth_limit: Optional[int] = None, +) -> ParsedExpr: + """Walk ``expr`` and replace bare measure refs with their formula AST. + + See module docstring for the eligibility matrix and recursion rules. + Raises ``ValueError`` if ``depth_limit`` is not a positive integer. + """ + if depth_limit is not None and depth_limit < 1: + raise ValueError( + f"depth_limit must be a positive integer, got {depth_limit!r}." + ) + measures = _collect_named_measures(model=model, extras=extra_measures) + limit = depth_limit if depth_limit is not None else _env_depth_limit() + parse_cache: Dict[str, ParsedExpr] = {} + return _walk( + expr, + measures=measures, + depth_limit=limit, + chain=(), + parse_cache=parse_cache, + ) + + +def _collect_named_measures( + *, + model: SlayerModel, + extras: Sequence[ModelMeasure], +) -> Dict[str, ModelMeasure]: + """Build a ``name -> ModelMeasure`` map. ``extras`` shadow model + measures with the same name. Unnamed measures are not addressable + via bare ref and are skipped. + """ + out: Dict[str, ModelMeasure] = {} + for m in model.measures: + if m.name: + out[m.name] = m + for m in extras: + if m.name: + out[m.name] = m + return out + + +def _env_depth_limit() -> int: + raw = os.environ.get(_DEPTH_ENV_VAR) + if raw is None: + return _DEFAULT_DEPTH + try: + return max(1, int(raw)) + except ValueError: + return _DEFAULT_DEPTH + + +def _walk( + node: ParsedExpr, + *, + measures: Dict[str, ModelMeasure], + depth_limit: int, + chain: Tuple[str, ...], + parse_cache: Dict[str, ParsedExpr], +) -> ParsedExpr: + if isinstance(node, Ref): + return _expand_ref( + node=node, + measures=measures, + depth_limit=depth_limit, + chain=chain, + parse_cache=parse_cache, + ) + if isinstance(node, (DottedRef, StarSource, Literal, AggCall)): + return node + if isinstance(node, TransformCall): + return _walk_transform_call( + node=node, + measures=measures, + depth_limit=depth_limit, + chain=chain, + parse_cache=parse_cache, + ) + if isinstance(node, ScalarCall): + return _walk_scalar_call( + node=node, + measures=measures, + depth_limit=depth_limit, + chain=chain, + parse_cache=parse_cache, + ) + if isinstance(node, Arith): + return node.model_copy( + update={ + "left": _maybe_walk( + node.left, + measures=measures, + depth_limit=depth_limit, + chain=chain, + parse_cache=parse_cache, + ), + "right": _maybe_walk( + node.right, + measures=measures, + depth_limit=depth_limit, + chain=chain, + parse_cache=parse_cache, + ), + } + ) + if isinstance(node, UnaryOp): + return node.model_copy( + update={ + "operand": _maybe_walk( + node.operand, + measures=measures, + depth_limit=depth_limit, + chain=chain, + parse_cache=parse_cache, + ), + } + ) + if isinstance(node, Cmp): + return node.model_copy( + update={ + "left": _maybe_walk( + node.left, + measures=measures, + depth_limit=depth_limit, + chain=chain, + parse_cache=parse_cache, + ), + "right": _maybe_walk( + node.right, + measures=measures, + depth_limit=depth_limit, + chain=chain, + parse_cache=parse_cache, + ), + } + ) + if isinstance(node, BoolOp): + return node.model_copy( + update={ + "operands": tuple( + _maybe_walk( + o, + measures=measures, + depth_limit=depth_limit, + chain=chain, + parse_cache=parse_cache, + ) + for o in node.operands + ) + } + ) + # Unknown node type: leave alone. This branch is unreachable for + # well-formed ParsedExpr but keeps the helper total. + return node + + +def _expand_ref( + *, + node: Ref, + measures: Dict[str, ModelMeasure], + depth_limit: int, + chain: Tuple[str, ...], + parse_cache: Dict[str, ParsedExpr], +) -> ParsedExpr: + if node.name not in measures: + return node + if node.name in chain: + raise MeasureCycleError(chain=list(chain) + [node.name]) + new_chain = chain + (node.name,) + if len(new_chain) > depth_limit: + raise MeasureRecursionLimitError( + chain=list(new_chain), limit=depth_limit + ) + cached = parse_cache.get(node.name) + if cached is None: + cached = parse_expr(measures[node.name].formula) + parse_cache[node.name] = cached + return _walk( + cached, + measures=measures, + depth_limit=depth_limit, + chain=new_chain, + parse_cache=parse_cache, + ) + + +def _walk_transform_call( + *, + node: TransformCall, + measures: Dict[str, ModelMeasure], + depth_limit: int, + chain: Tuple[str, ...], + parse_cache: Dict[str, ParsedExpr], +) -> TransformCall: + new_input = _maybe_walk( + node.input, + measures=measures, + depth_limit=depth_limit, + chain=chain, + parse_cache=parse_cache, + ) + new_args = tuple( + _maybe_walk( + a, + measures=measures, + depth_limit=depth_limit, + chain=chain, + parse_cache=parse_cache, + ) + for a in node.args + ) + new_kwargs = tuple( + ( + k, + _maybe_walk( + v, + measures=measures, + depth_limit=depth_limit, + chain=chain, + parse_cache=parse_cache, + ), + ) + for k, v in node.kwargs + ) + return node.model_copy( + update={"input": new_input, "args": new_args, "kwargs": new_kwargs} + ) + + +def _walk_scalar_call( + *, + node: ScalarCall, + measures: Dict[str, ModelMeasure], + depth_limit: int, + chain: Tuple[str, ...], + parse_cache: Dict[str, ParsedExpr], +) -> ScalarCall: + new_args = tuple( + _maybe_walk( + a, + measures=measures, + depth_limit=depth_limit, + chain=chain, + parse_cache=parse_cache, + ) + for a in node.args + ) + return node.model_copy(update={"args": new_args}) + + +def _maybe_walk( + v: Any, + *, + measures: Dict[str, ModelMeasure], + depth_limit: int, + chain: Tuple[str, ...], + parse_cache: Dict[str, ParsedExpr], +) -> Any: + """Walk ``v`` only if it is a ParsedExpr node; pass scalars through + unchanged. AggCall args/kwargs and the like contain scalars + (Decimal / str / bool) that should not be touched. + """ + if isinstance(v, _PARSED_EXPR_TYPES): + return _walk( + v, + measures=measures, + depth_limit=depth_limit, + chain=chain, + parse_cache=parse_cache, + ) + return v + + +__all__ = ["expand_model_measures"] diff --git a/slayer/engine/normalization.py b/slayer/engine/normalization.py new file mode 100644 index 00000000..f31b1ea4 --- /dev/null +++ b/slayer/engine/normalization.py @@ -0,0 +1,688 @@ +"""Stage 6 (DEV-1450) — slack normalization layer. + +Rewrites tolerant-but-unambiguous agent input to canonical form before the +typed pipeline sees it, returning every rewrite as a typed +``NormalizationWarning`` (P0). Downstream stages never see the slack form. + +Three seed rules: + +- ``FUNC_STYLE_AGG`` (Mode B only): ``sum(revenue)`` / ``count(*)`` / + ``percentile(amount, p=0.5)`` → colon syntax. Rewrites Mode-B fields + (``ModelMeasure.formula``, ``SlayerQuery.measures[].formula``, + ``SlayerQuery.filters`` entries). + +- ``MISPLACED_MEASURE`` (query shape): bare column-looking entries in + ``SlayerQuery.measures`` that resolve as a column (not a named + ``ModelMeasure``) move to ``SlayerQuery.dimensions``. Mirrors the + existing ``_auto_move_fields_to_dimensions`` heuristic but emits a + structured warning. + +- ``DOT_PATH_IN_SQL`` (Mode A only): sqlglot-AST ``Column`` node in root + scope whose dotted path's leading segment matches a known join target + on the host model → ``__`` alias form (``customers.regions.name`` → + ``customers__regions.name``). Scope-aware via lexical-ancestor walking + so refs inside subqueries / CTE bodies / set-op branches are left + alone. First-segment shadow detection covers CTE names, explicit + ``AS`` aliases, Subquery/CTE FROM sources, and schema/catalog + qualifiers on FROM tables (``FROM customers.regions`` → ``customers`` + shadows). Shadowed cases emit an ambiguity warning without rewriting. + Wired into ``normalize_model`` over ``Column.sql``, ``Column.filter``, + and ``SlayerModel.filters``. + +Each rule emits a ``SlayerNormalizationWarning`` via ``warnings.warn(...)`` +AND appends a ``NormalizationWarning`` payload to the returned result, +so REST / MCP / CLI consumers see the rewrite alongside the response and +``warnings.catch_warnings()`` callers see it via the standard channel. +""" + +from __future__ import annotations + +import re +import warnings as _warnings_module +from typing import List, Optional, Set, Tuple + +import sqlglot +from sqlglot import exp +from sqlglot.optimizer.scope import ScopeType, traverse_scope +from pydantic import BaseModel, ConfigDict, Field + +from slayer.core.enums import BUILTIN_AGGREGATIONS +from slayer.core.models import SlayerModel +from slayer.core.query import SlayerQuery +from slayer.core.refs import IDENT_OR_PATH_RE +from slayer.core.warnings import NormalizationWarning, SlayerNormalizationWarning +from slayer.engine.column_expansion import _root_scope_column_ids + + +# --------------------------------------------------------------------------- +# Result type +# --------------------------------------------------------------------------- + + +class NormalizationResult(BaseModel): + """Output of a normalization pass. + + ``query`` and ``model`` are either the same object the caller passed + in (if no rewrite fired) or a new instance with the slack form + rewritten. ``warnings`` lists one ``NormalizationWarning`` per + rewrite — empty when the input was already canonical. + """ + + model_config = ConfigDict(arbitrary_types_allowed=True) + + query: Optional[SlayerQuery] = None + model: Optional[SlayerModel] = None + warnings: List[NormalizationWarning] = Field(default_factory=list) + + +# --------------------------------------------------------------------------- +# Rule: FUNC_STYLE_AGG +# --------------------------------------------------------------------------- + + +# Aggregation names that are also transform names — the rewrite only fires +# when the inner is a bare identifier, not when it's a colon-form aggregate. +_AMBIGUOUS_AGG_TRANSFORMS = frozenset({"first", "last"}) + +_STRING_LITERAL_RE = re.compile(r"'(?:[^']|'')*'|\"(?:[^\"]|\"\")*\"") + + +def _find_balanced_close(s: str, open_idx: int) -> int: + depth = 0 + in_string = False + string_ch = "" + i = open_idx + while i < len(s): + ch = s[i] + if in_string: + if ch == string_ch: + # Handle '' / "" escapes. + if i + 1 < len(s) and s[i + 1] == string_ch: + i += 2 + continue + in_string = False + elif ch in ("'", '"'): + in_string = True + string_ch = ch + elif ch == "(": + depth += 1 + elif ch == ")": + depth -= 1 + if depth == 0: + return i + i += 1 + return -1 + + +def _split_args(s: str) -> List[str]: + parts: List[str] = [] + depth = 0 + in_string = False + string_ch = "" + current: List[str] = [] + for ch in s: + if in_string: + current.append(ch) + if ch == string_ch: + in_string = False + continue + if ch in ("'", '"'): + in_string = True + string_ch = ch + current.append(ch) + continue + if ch == "(": + depth += 1 + current.append(ch) + continue + if ch == ")": + depth -= 1 + current.append(ch) + continue + if ch == "," and depth == 0: + parts.append("".join(current)) + current = [] + continue + current.append(ch) + if current: + parts.append("".join(current)) + return [p.strip() for p in parts if p.strip()] + + +def _apply_func_style_agg( + formula: str, + *, + location: str, + custom_agg_names: Optional[frozenset[str]] = None, +) -> tuple[str, List[NormalizationWarning]]: + """Rewrite function-style aggregations in ``formula`` to colon syntax. + + Returns ``(rewritten_formula, warnings)`` — ``warnings`` is empty when + nothing changed. + """ + agg_names = BUILTIN_AGGREGATIONS | (custom_agg_names or frozenset()) + sorted_names = sorted(agg_names, key=len, reverse=True) + pattern = re.compile( + r"(? str: + """Rewrite function-style aggregations (``sum(x)`` → ``x:sum``, + ``count(*)`` → ``*:count``) to colon syntax, returning only the rewritten + string. + + Quiet variant of the ``FUNC_STYLE_AGG`` slack rule for read-only, + best-effort consumers (schema-drift cascade attribution, memory entity + tagging) that inspect formulas with the typed Mode-B parser but must NOT + re-surface slack advice to the user — the pipeline path + (``normalize_query`` / ``normalize_model``) is the one that emits + ``SlayerNormalizationWarning``. Returns the formula unchanged when nothing + matches. + """ + with _warnings_module.catch_warnings(): + _warnings_module.simplefilter("ignore", SlayerNormalizationWarning) + rewritten, _ = _apply_func_style_agg( + formula, location="(inspect)", custom_agg_names=custom_agg_names, + ) + return rewritten + + +# --------------------------------------------------------------------------- +# Rule: MISPLACED_MEASURE +# --------------------------------------------------------------------------- + + +def _apply_misplaced_measure( + query: SlayerQuery, + *, + model: Optional[SlayerModel], +) -> tuple[SlayerQuery, List[NormalizationWarning]]: + """Move bare (no-colon, no-function) entries from ``query.measures`` to + ``query.dimensions`` when they name a column on the model that isn't + a ``ModelMeasure`` formula. + + Mirrors the existing ``_auto_move_fields_to_dimensions`` heuristic but + emits a structured warning. When ``model`` is None we can't classify, + so the rule is a no-op. + """ + if not query.measures or model is None: + return query, [] + + measure_formula_names = {m.name for m in model.measures} + column_names = {c.name for c in model.columns} + + new_measures = list(query.measures) + moved_dim_strings: List[str] = [] + emitted: List[NormalizationWarning] = [] + + kept: List = [] + for i, m in enumerate(new_measures): + formula = getattr(m, "formula", None) + if not isinstance(formula, str): + kept.append(m) + continue + if ":" in formula or "(" in formula: + kept.append(m) + continue + # Bare token. If it names a known ModelMeasure formula, keep it as + # a measure. If it names a column on the model, move to dimensions. + bare = formula.strip() + if bare in measure_formula_names: + kept.append(m) + continue + if bare in column_names: + moved_dim_strings.append(bare) + emitted.append(NormalizationWarning( + rule_id="MISPLACED_MEASURE", + original=bare, + normalized=f"dimensions += {bare!r}", + location=f"measures[{i}].formula", + rule_doc_url="docs/agent_input_slack.md#misplaced-measure", + )) + _warnings_module.warn( + SlayerNormalizationWarning(emitted[-1]), stacklevel=2, + ) + continue + # Unknown bare token — leave for downstream resolver to error on. + kept.append(m) + + if not emitted: + return query, [] + + existing_dims = list(query.dimensions or []) + # Append each moved bare column name as a dimension entry. We add as + # plain strings since SlayerQuery.dimensions accepts string entries + # alongside ColumnRefs (the pydantic union validators handle the + # coercion). + new_dimensions = existing_dims + moved_dim_strings + return ( + query.model_copy(update={"measures": kept, "dimensions": new_dimensions}), + emitted, + ) + + +# --------------------------------------------------------------------------- +# Rule: DOT_PATH_IN_SQL (stub for stage 6 — full implementation deferred) +# --------------------------------------------------------------------------- + + +# Node types that have a "natural" root scope sqlglot can analyse directly. +# Any other parsed input (a scalar expression like ``lower(a.b.c)``) gets +# wrapped in a synthetic ``SELECT ... AS _`` for scope traversal — mirrors +# the precedent in ``column_expansion._root_scope_column_ids``. +_STATEMENT_TYPES: Tuple[type, ...] = ( + exp.Select, exp.Union, exp.Intersect, exp.Except, +) + + +def _dot_path_root_scope_analysis( + *, parsed: exp.Expression, +) -> Tuple[Set[int], Set[str]]: + """Return ``(root_col_ids, shadow_names)`` for ``parsed``. + + ``root_col_ids``: ids of ``exp.Column`` nodes whose innermost + scope-defining ancestor is parsed's root scope. Walks lexical + ancestors rather than trusting ``Scope.columns`` (which can include + correlated refs from inner subqueries). + + ``shadow_names``: identifiers defined at the same root scope as: + - CTE definitions, + - FROM/JOIN sources introduced by an explicit ``AS`` alias OR by a + Subquery/CTE source (always alias-like), + - schema/catalog parts of qualified FROM tables (``mydb.foo`` → + ``mydb`` shadows; ``a.b.foo`` → both ``a`` and ``b`` shadow). + Unaliased plain ``FROM customers`` does NOT shadow per spec wording + ("AS alias, CTE name, or schema name"). + """ + if isinstance(parsed, _STATEMENT_TYPES): + scope_id_to_type: dict[int, ScopeType] = {} + for scope in traverse_scope(parsed): + scope_id_to_type[id(scope.expression)] = scope.scope_type + root_scope_node_id = next( + (sid for sid, st in scope_id_to_type.items() if st == ScopeType.ROOT), + None, + ) + if root_scope_node_id is None: + return set(), set() + + root_col_ids: Set[int] = set() + for col in parsed.find_all(exp.Column): + node: Optional[exp.Expression] = col.parent + while node is not None: + if id(node) in scope_id_to_type: + if id(node) == root_scope_node_id: + root_col_ids.add(id(col)) + break + node = node.parent + + shadow_names: Set[str] = set() + for scope in traverse_scope(parsed): + if scope.scope_type != ScopeType.ROOT: + continue + shadow_names |= set(scope.cte_sources) + for src_name, source in scope.sources.items(): + if isinstance(source, exp.Table): + # Explicit AS alias. + if source.alias: + shadow_names.add(src_name) + # Schema / catalog qualifiers on the FROM table. + for part_key in ("db", "catalog"): + part = source.args.get(part_key) + if part is not None: + shadow_names.add(part.name) + else: + # Subquery / CTE-referenced source — always alias-like. + shadow_names.add(src_name) + break + return root_col_ids, shadow_names + return _root_scope_column_ids(parsed=parsed), set() + + +def _apply_dot_path_in_sql( + sql_text: Optional[str], *, location: str, model: Optional[SlayerModel], +) -> Tuple[Optional[str], List[NormalizationWarning]]: + """AST-based, scope-aware DOT_PATH_IN_SQL rewrite. + + Rewrites root-scope dotted refs (``customers.regions.name``) to the + ``__`` alias form (``customers__regions.name``) when the leading + segment is a known join target on ``model``. Refs in subqueries / + CTE-local scopes / set-op branches are left alone (scope-aware). + Refs whose leading segment matches a join target AND is also a + CTE / FROM-alias in the same scope are flagged ambiguous: no + rewrite, one warning carrying ``normalized="(ambiguous: ...)"``. + + Intermediate hops are not validated at normalize-time (no storage + access here); the contract is "first segment matches a join target + on the host model". Downstream join resolution catches an invalid + intermediate the same way it would for the canonical form. + """ + if not sql_text or model is None or not model.joins: + return sql_text, [] + + try: + statements = sqlglot.parse(sql_text) + except Exception: + return sql_text, [] + statements = [s for s in statements if s is not None] + if len(statements) != 1: + # Multi-statement (or empty) — slack input is contractually a single + # scalar expression / predicate. Leave alone. + return sql_text, [] + parsed = statements[0] + + join_target_names = {j.target_model for j in model.joins} + root_col_ids, shadow_names = _dot_path_root_scope_analysis(parsed=parsed) + + emitted: List[NormalizationWarning] = [] + changed = False + + for col in parsed.find_all(exp.Column): + if id(col) not in root_col_ids: + continue + parts = [p.name for p in col.parts] + if len(parts) < 3: + continue + first = parts[0] + if first not in join_target_names: + continue + original = ".".join(parts) + + if first in shadow_names: + payload = NormalizationWarning( + rule_id="DOT_PATH_IN_SQL", + original=original, + normalized="(ambiguous: shadowed by local alias or CTE — not rewritten)", + location=location, + rule_doc_url="docs/agent_input_slack.md#dot-path-in-sql", + ) + emitted.append(payload) + _warnings_module.warn( + SlayerNormalizationWarning(payload), stacklevel=2, + ) + continue + + new_table_name = "__".join(parts[:-1]) + leaf_name = parts[-1] + normalized = f"{new_table_name}.{leaf_name}" + col.set("catalog", None) + col.set("db", None) + col.set("table", exp.to_identifier(new_table_name)) + + payload = NormalizationWarning( + rule_id="DOT_PATH_IN_SQL", + original=original, + normalized=normalized, + location=location, + rule_doc_url="docs/agent_input_slack.md#dot-path-in-sql", + ) + emitted.append(payload) + _warnings_module.warn( + SlayerNormalizationWarning(payload), stacklevel=2, + ) + changed = True + + if not changed: + return sql_text, emitted + return parsed.sql(), emitted + + +# --------------------------------------------------------------------------- +# Top-level entry points +# --------------------------------------------------------------------------- + + +def normalize_query( + query: SlayerQuery, + *, + model: Optional[SlayerModel] = None, + custom_agg_names: Optional[frozenset[str]] = None, +) -> NormalizationResult: + """Apply all enabled slack rules to a ``SlayerQuery``. + + Returns the (possibly rewritten) query and the structured warnings. + Existing in-tree rewriters (notably + ``slayer.core.formula._rewrite_funcstyle_aggregations``) continue to + run during binding; in stage 6 they see canonical input and + silently no-op for any input this layer already rewrote. + """ + all_warnings: List[NormalizationWarning] = [] + + # Rule 1: FUNC_STYLE_AGG over Mode-B fields. + new_measures = [] + for i, m in enumerate(query.measures or []): + formula = getattr(m, "formula", None) + if isinstance(formula, str): + rewritten, ws = _apply_func_style_agg( + formula, + location=f"measures[{i}].formula", + custom_agg_names=custom_agg_names, + ) + all_warnings.extend(ws) + if rewritten != formula: + m = m.model_copy(update={"formula": rewritten}) + new_measures.append(m) + + new_filters: List[str] = [] + for i, f in enumerate(query.filters or []): + if isinstance(f, str): + rewritten, ws = _apply_func_style_agg( + f, + location=f"filters[{i}]", + custom_agg_names=custom_agg_names, + ) + all_warnings.extend(ws) + new_filters.append(rewritten) + else: + new_filters.append(f) + + query = query.model_copy(update={ + "measures": new_measures, + "filters": new_filters, + }) + + # Rule 2: MISPLACED_MEASURE. + query, ws = _apply_misplaced_measure(query, model=model) + all_warnings.extend(ws) + + # Rule 3: DOT_PATH_IN_SQL (stub in stage 6). + # Mode-A fields on the query itself are rare — most Mode-A lives on + # the model. Wiring is preserved so future activations need no + # plumbing changes. + + return NormalizationResult(query=query, warnings=all_warnings) + + +def _normalize_model_measures( + model: SlayerModel, *, custom_agg_names: Optional[frozenset[str]], +) -> Tuple[SlayerModel, List[NormalizationWarning]]: + """FUNC_STYLE_AGG over ``model.measures`` (Mode-B). See + :func:`normalize_model` for the ``custom_agg_names`` contract. + """ + if not model.measures: + return model, [] + if custom_agg_names is not None: + custom_names = custom_agg_names + else: + custom_names = frozenset(a.name for a in (model.aggregations or [])) + warnings: List[NormalizationWarning] = [] + new_measures = [] + for i, mm in enumerate(model.measures): + formula = mm.formula + rewritten, ws = _apply_func_style_agg( + formula, + location=f"measures[{i}].formula", + custom_agg_names=custom_names, + ) + warnings.extend(ws) + if rewritten != formula: + mm = mm.model_copy(update={"formula": rewritten}) + new_measures.append(mm) + return model.model_copy(update={"measures": new_measures}), warnings + + +def _normalize_column_dot_paths( + model: SlayerModel, +) -> Tuple[SlayerModel, List[NormalizationWarning]]: + """DOT_PATH_IN_SQL over ``Column.sql`` / ``Column.filter`` (Mode-A).""" + warnings: List[NormalizationWarning] = [] + new_columns = [] + changed = False + for i, c in enumerate(model.columns): + updates: dict = {} + if c.sql is not None: + rewritten_sql, ws = _apply_dot_path_in_sql( + c.sql, location=f"columns[{i}].sql", model=model, + ) + warnings.extend(ws) + if rewritten_sql != c.sql: + updates["sql"] = rewritten_sql + if c.filter is not None: + rewritten_filter, ws = _apply_dot_path_in_sql( + c.filter, location=f"columns[{i}].filter", model=model, + ) + warnings.extend(ws) + if rewritten_filter != c.filter: + updates["filter"] = rewritten_filter + if updates: + c = c.model_copy(update=updates) + changed = True + new_columns.append(c) + if changed: + model = model.model_copy(update={"columns": new_columns}) + return model, warnings + + +def _normalize_model_filter_dot_paths( + model: SlayerModel, +) -> Tuple[SlayerModel, List[NormalizationWarning]]: + """DOT_PATH_IN_SQL over ``SlayerModel.filters`` (Mode-A).""" + if not model.filters: + return model, [] + warnings: List[NormalizationWarning] = [] + new_filters = [] + changed = False + for i, f in enumerate(model.filters): + rewritten, ws = _apply_dot_path_in_sql( + f, location=f"filters[{i}]", model=model, + ) + warnings.extend(ws) + if rewritten != f: + changed = True + new_filters.append(rewritten if rewritten is not None else f) + if changed: + model = model.model_copy(update={"filters": new_filters}) + return model, warnings + + +def normalize_model( + model: SlayerModel, + *, + custom_agg_names: Optional[frozenset[str]] = None, +) -> NormalizationResult: + """Apply slack rules to a ``SlayerModel`` before persistence. + + Mode-A rewrites (``DOT_PATH_IN_SQL``) target ``Column.sql``, + ``Column.filter``, and ``SlayerModel.filters``. Mode-B rewrites + (``FUNC_STYLE_AGG``) target ``ModelMeasure.formula``. The rewrite + semantics match ``normalize_query``. + + ``custom_agg_names`` lets the caller supply the full reachable + aggregation set (model's own aggregations PLUS any defined on joined + models the caller has resolved through storage) so a funcstyle measure + over a joined-model custom aggregation gets rewritten — mirrors + ``normalize_query``'s param. Sharp edges: + + * ``custom_agg_names=None`` (default) → fall back to the model's own + ``aggregations`` (backward-compatible; matches the pre-DEV-1500 + behaviour for direct callers and tests that don't resolve joins). + * ``custom_agg_names=frozenset()`` → empty set is honoured AS-IS: the + model's-own fallback is suppressed. Pass an explicit empty frozenset + only when you want builtins-only recognition. + """ + all_warnings: List[NormalizationWarning] = [] + model, ws = _normalize_model_measures( + model, custom_agg_names=custom_agg_names, + ) + all_warnings.extend(ws) + if model.joins: + model, ws = _normalize_column_dot_paths(model) + all_warnings.extend(ws) + model, ws = _normalize_model_filter_dot_paths(model) + all_warnings.extend(ws) + return NormalizationResult(model=model, warnings=all_warnings) diff --git a/slayer/engine/planned.py b/slayer/engine/planned.py new file mode 100644 index 00000000..a55b48a6 --- /dev/null +++ b/slayer/engine/planned.py @@ -0,0 +1,465 @@ +"""Stage 7a.1 (DEV-1450) — typed plan shapes consumed by the SQL generator. + +A ``PlannedQuery`` is the final, fully resolved plan that the SQL +generator (stage 7b) compiles to SQL. The plan carries everything a +renderer needs: row slots, aggregate slots, cross-model aggregate +sub-plans, transform layers, filter routing, projection / order / +limit, and an emitted ``StageSchema`` for downstream stages to bind +against. + +Identity-bearing structure is in ``slayer/core/keys.py`` (the +``ValueKey`` family); the planner here associates each key with a +``SlotId`` and the rendering metadata (alias, hidden, label). + +The planning logic that produces a ``PlannedQuery`` lives in other +7a substages — ``planning.py`` (ValueRegistry, TransformLowerer, +ProjectionPlanner), ``cross_model_planner.py`` (I1 strategy), +``stage_planner.py`` (multi-stage DAG). This file is the typed +target. + +These types are dormant in stage 7a — no engine code consumes them +yet. Stage 7b's engine cutover routes through them. +""" + +from __future__ import annotations + +from typing import List, Optional, Tuple + +from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator + +from slayer.core.enums import DataType, JoinType +from slayer.core.errors import UnreachableFilterDroppedWarning +from slayer.core.format import NumberFormat +from slayer.core.keys import Phase, ValueKey +from slayer.core.models import SlayerModel +from slayer.core.scope import StageSchema +from slayer.engine.binding import BoundExpr # re-exported below + + +# Opaque identifier types — kept as plain ``str`` for now. SlotId is +# allocated by the planner's ValueRegistry; BoundFilterId by the +# FilterBinder. The string form keeps tracebacks readable and lets +# tests assert on them without exotic comparisons. +SlotId = str +BoundFilterId = str + + +# --------------------------------------------------------------------------- +# BoundExpr — re-exported from slayer.engine.binding (DEV-1450 stage 7b.6). +# --------------------------------------------------------------------------- +# +# Until stage 7b.6 the planned-side BoundExpr was a separate scaffold +# Pydantic class with an optional ``sql_text`` cache. The binder +# produced its own ``BoundExpr`` shape, so ``ValueSlot.expression`` and +# ``FilterPhase.expression`` could not store binder output directly +# without type unification (Codex HIGH F2 in the earlier round). 7b.6 +# folds the two: the binder's ``BoundExpr(value_key=ValueKey)`` is the +# canonical shape. The render artifact ``sql_text`` is dropped — the +# generator renders from the typed ``value_key`` against the slot +# registry, not a cached string. +__all__ = [ + "BoundExpr", + "BoundFilterId", + "CrossModelAggregatePlan", + "FilterPhase", + "JoinRequirement", + "OrderEntry", + "PlannedQuery", + "SlotId", + "TransformLayer", + "ValueSlot", + "WindowedAggregatePlan", +] + + +# --------------------------------------------------------------------------- +# ValueSlot +# --------------------------------------------------------------------------- + + +class ValueSlot(BaseModel): + """One materialised slot in a ``PlannedQuery`` (P6). + + Identity comes from ``key`` (a ``ValueKey`` from + ``slayer.core.keys``). Two structurally equal keys share one slot. + Rendering metadata (alias, hidden, label, type) lives here, not on + the key. + + ``declared_name`` is either the user-supplied ``name`` or the + canonical form derived from the formula. ``public_name`` is the + user-facing alias when the slot is part of the public projection + (None for hidden slots). ``public_aliases`` carries multiple + aliases when the same structural key was declared with multiple + explicit names (P4 / C13). + """ + + model_config = ConfigDict(arbitrary_types_allowed=True) + + id: SlotId + key: ValueKey + declared_name: str + public_name: Optional[str] = None + public_aliases: List[str] = Field(default_factory=list) + hidden: bool = False + phase: Phase + label: Optional[str] = None + type: Optional[DataType] = None + expression: Optional[BoundExpr] = None + # DEV-1452 Stage B decision #8 — typed format / description propagated + # by the planner from the source ``ModelMeasure`` / ``Column`` and + # ``_infer_aggregated_format``. Consumed by the migrated + # ``_expand_query_backed_model`` (via the public ``StageSchema``) so + # query-backed virtual-model columns carry the same display metadata + # the legacy enrichment pipeline produced. + format: Optional[NumberFormat] = None + description: Optional[str] = None + + @model_validator(mode="after") + def _hidden_invariant(self) -> "ValueSlot": + # Hidden slots are materialised but never surfaced — they must + # not carry a public_name or public_aliases, otherwise the + # generator would emit them in the public projection. + if self.hidden and (self.public_name is not None or self.public_aliases): + raise ValueError( + f"ValueSlot(id={self.id!r}) is hidden but carries " + f"public_name={self.public_name!r} / " + f"public_aliases={self.public_aliases!r}; hidden slots " + f"must have public_name=None and public_aliases=[]." + ) + return self + + +# --------------------------------------------------------------------------- +# JoinRequirement +# --------------------------------------------------------------------------- + + +class JoinRequirement(BaseModel): + """One hop in a cross-model join chain. + + Mirrors the shape of ``slayer.core.models.ModelJoin`` but is + rooted on the typed-plan side — the planner builds these from + resolved bundle models so the SQL generator never re-walks the + model graph. + """ + + source_model: str + target_model: str + join_pairs: List[List[str]] + join_type: JoinType = JoinType.LEFT + + @field_validator("join_pairs") + @classmethod + def _non_empty(cls, v: List[List[str]]) -> List[List[str]]: + if not v: + raise ValueError("join_pairs must be non-empty") + for i, pair in enumerate(v): + if len(pair) != 2 or not all(isinstance(s, str) and s for s in pair): + raise ValueError( + f"join_pairs[{i}] must be [source_dim, target_dim] " + f"with non-empty strings, got {pair!r}" + ) + return v + + +# --------------------------------------------------------------------------- +# CrossModelAggregatePlan +# --------------------------------------------------------------------------- + + +class CrossModelAggregatePlan(BaseModel): + """Plan for one cross-model aggregate slot (P3 / I1). + + The strategy that populated this plan (today's isolated-CTE form + or a future alternative — see ``cross_model_planner.py``) lives + outside this struct; this is the typed result, not the algorithm. + + Filter routing is route-explicit so the SQL generator (stage 7b) + can render each route without re-classifying: + - ``where_filter_ids`` — host filters propagated to the CTE's WHERE + (decision-table rows: host-local-but-targeted, joined-target-path). + - ``having_filter_ids`` — host filters propagated as HAVING (decision- + table row: cross-model agg-ref on the same target). + - ``target_model_filters`` — the target model's own + ``SlayerModel.filters`` (always-applied WHERE). + ``applied_filter_ids`` is the audit union of where + having for + backward compatibility with the spec's external surface. + + ``hidden=True`` is used for order-only / filter-only refs whose + aggregate value is materialised but not surfaced in the public + projection; ``public_alias`` is ``None`` in that case. + """ + + model_config = ConfigDict(arbitrary_types_allowed=True) + + aggregate_slot_id: SlotId + target_model: str + datasource: str + join_chain: List[JoinRequirement] + join_back_pairs: List[Tuple[ValueKey, ValueKey]] = Field(default_factory=list) + cte_stage_schema: StageSchema + shared_grain_slots: List[SlotId] + applied_filter_ids: List[BoundFilterId] = Field(default_factory=list) + where_filter_ids: List[BoundFilterId] = Field(default_factory=list) + having_filter_ids: List[BoundFilterId] = Field(default_factory=list) + target_model_filters: List[str] = Field(default_factory=list) + dropped_filter_warnings: List[UnreachableFilterDroppedWarning] = Field(default_factory=list) + hidden: bool = False + public_alias: Optional[str] = None + + # DEV-1450 stage 7b.15e (C1) — re-rooted sub-plan. When the host query + # carries dimensions that are reachable from the target by re-rooting + # through the target's join graph (the legacy ``_build_rerooted_enriched`` + # case), the cross-model CTE is rendered as a full nested ``PlannedQuery`` + # rooted at the target (FROM target + joins), preserving the host + # dimension grain instead of collapsing to a scalar CROSS JOIN. ``None`` + # keeps the forward-path "FROM bare target" rendering. + # + # ``rerooted_grain_pairs`` maps (host_dim_slot_id, rerooted_dim_slot_id) + # for the combined LEFT JOIN ON; the generator resolves each side's SQL + # alias independently (host alias vs sub-plan alias need not match). + # ``rerooted_agg_slot_id`` is the sub-plan slot id of the local aggregate; + # the combined SELECT projects it ``AS`` the canonical / public alias. + rerooted_plan: Optional["PlannedQuery"] = None + rerooted_grain_pairs: List[Tuple[SlotId, SlotId]] = Field(default_factory=list) + rerooted_agg_slot_id: Optional[SlotId] = None + + # DEV-1503 — host-rooted CTE for a cross-model-FILTERED local measure. + # The cross-model planner has two distinct cases that produce a nested + # ``rerooted_plan``: + # + # * Cross-model aggregate re-rooting (``target_model`` is the join target, + # the sub-plan is rooted at the target so the target's own join graph + # reaches host dims that the forward-path CTE collapses): ``cte_root_model`` + # stays ``None`` and the renderer uses ``target_model`` for the FROM / + # joins (existing pre-DEV-1503 behaviour). + # * Filtered-local isolation (``AggregateKey.source.path`` is empty but the + # measure's ``Column.filter`` crosses a join, so the aggregate must + # evaluate in its own CTE rooted at the HOST + the filter-target join): + # ``cte_root_model`` is set to the HOST model name and the renderer uses + # the sub-plan's own ``source_relation`` / ``render_source_model`` for + # the FROM. ``target_model`` is conventionally set to the host name in + # this case but the renderer reads ``cte_root_model`` to disambiguate. + cte_root_model: Optional[str] = None + + +# --------------------------------------------------------------------------- +# WindowedAggregatePlan +# --------------------------------------------------------------------------- + + +class WindowedAggregatePlan(BaseModel): + """Plan for one duration-windowed aggregate slot (DEV-1714 Stage 10). + + A windowed measure (``revenue:sum(window='90d')``) is a trailing rolling + aggregate: for each output bucket, SLayer sums source rows in the trailing + ``window`` interval ending at that bucket's end. It renders as a HOST-ROOTED + ``_wm___`` CTE — an inner ``_src`` self-join subquery joined + to ``_base`` on the query grain with an ``INTERVAL`` range predicate — then + LEFT-JOINed back to ``_base`` on the shared grain (same join-back machinery + as the cross-model ``_cm_*`` CTEs, so adding a windowed measure never + changes host cardinality). + + The renderer looks up the aggregate ``ValueSlot`` (source column, ``agg``, + result ``type``, ``column_filter_key``) and the grain ``ValueSlot``s by id + from the owning ``PlannedQuery``; this plan carries the window duration, the + resolved window time-dimension slot + its granularity, the per-role grain + slot partition, and the WHERE-phase filter ids inherited into ``_src``. + + Frame bounds are excluded from ``where_filter_ids`` so the trailing window + reaches rows before the visible frame starts (DEV-1732). A filter that is + only PARTLY a frame bound (``created_at >= X and status = 'paid'``) stays in + ``where_filter_ids`` and gets a ``src_filter_rewrites`` entry carrying the + residual — the population half — which the renderer substitutes for the + host's predicate. + + Scope for Stage 10 is ``sum``/``avg`` local measures only; cross-model, + transform-combined, composite, hidden, and mixed-filter windowed shapes are + guarded loudly at plan time (DEV-1504 lifts those guards). + """ + + model_config = ConfigDict(arbitrary_types_allowed=True) + + aggregate_slot_id: SlotId + agg: str + window_raw: str + window_parts: List[Tuple[int, str]] + window_time_dimension_slot_id: SlotId + window_granularity: str + dimension_slot_ids: List[SlotId] = Field(default_factory=list) + other_time_dimension_slot_ids: List[SlotId] = Field(default_factory=list) + grain_slot_ids: List[SlotId] = Field(default_factory=list) + where_filter_ids: List[BoundFilterId] = Field(default_factory=list) + src_filter_rewrites: List["SrcFilterRewrite"] = Field(default_factory=list) + public_alias: Optional[str] = None + hidden: bool = False + + +# --------------------------------------------------------------------------- +# SrcFilterRewrite — DEV-1732 +# --------------------------------------------------------------------------- + + +class SrcFilterRewrite(BaseModel): + """A ROW filter whose CTE-local form differs from the host's (DEV-1732). + + Emitted when a filter is only PARTLY a frame bound, so it must still apply + inside the CTE but with its frame-bound conjuncts removed: + ``created_at >= '2024-06-01' and status = 'paid'`` becomes ``status = + 'paid'``. + + A filter that is ENTIRELY a frame bound needs no rewrite — the planner just + leaves its id out of ``where_filter_ids``. + """ + + model_config = ConfigDict(arbitrary_types_allowed=True) + + filter_id: BoundFilterId + expression: BoundExpr + + +# --------------------------------------------------------------------------- +# TransformLayer +# --------------------------------------------------------------------------- + + +class TransformLayer(BaseModel): + """One transform layer in the planned query. + + Window / temporal transforms (``cumsum``, ``time_shift``, + ``rank``, ``lag``, ``lead``, ...) are grouped into layers so the + SQL generator can emit them in the right order (window functions + in an inner SELECT, time_shift as a self-join CTE, etc.). The + layer carries the slot ids that belong to it; rendering details + are decided by the generator per ``op``. + """ + + op: str + slot_ids: List[SlotId] + + +# --------------------------------------------------------------------------- +# FilterPhase +# --------------------------------------------------------------------------- + + +class FilterPhase(BaseModel): + """A bound filter expression routed to its phase (P8). + + ``phase`` is the maximum phase of the slots the filter + references: ROW → WHERE, AGGREGATE → HAVING, POST → post-filter + on the outer SELECT. + + Two carrier modes, mutually exclusive in practice: + + * ``expression`` is a typed ``BoundExpr`` — used for the Mode-B + DSL filters bound by ``bind_filter`` and the planner-emitted + ``BetweenKey`` for ``TimeDimension.date_range``. The renderer + walks the typed value-key tree. + * ``text`` is a Mode-A SQL fragment — used for + ``SlayerModel.filters`` (always-applied WHERE). The renderer + qualifies bare-identifier column refs in ``text_columns`` with + the source-relation alias and emits the result verbatim + (matching legacy ``_build_where_and_having`` qualification). + """ + + id: BoundFilterId + phase: Phase + text: Optional[str] = None + text_columns: Tuple[str, ...] = () + expression: Optional[BoundExpr] = None + + +# --------------------------------------------------------------------------- +# OrderEntry +# --------------------------------------------------------------------------- + + +class OrderEntry(BaseModel): + """One entry in the ORDER BY of a planned query.""" + + slot_id: SlotId + direction: str # "asc" or "desc" + + @field_validator("direction") + @classmethod + def _validate_direction(cls, v: str) -> str: + if v not in ("asc", "desc"): + raise ValueError( + f"OrderEntry.direction must be 'asc' or 'desc', got {v!r}" + ) + return v + + +# --------------------------------------------------------------------------- +# PlannedQuery +# --------------------------------------------------------------------------- + + +class PlannedQuery(BaseModel): + """The fully typed plan for one query stage (P7). + + Consumed by the SQL generator (stage 7b). Carries everything + needed to emit SQL without re-walking the model graph. + + ``stage_schema`` is the projection emitted by this stage — + downstream stages bind against it (P6). Top-level queries that + aren't part of a multi-stage DAG can leave ``stage_schema`` as + ``None``. + """ + + model_config = ConfigDict(arbitrary_types_allowed=True) + + source_relation: str + join_plan: List[JoinRequirement] = Field(default_factory=list) + row_slots: List[ValueSlot] = Field(default_factory=list) + aggregate_slots: List[ValueSlot] = Field(default_factory=list) + cross_model_aggregate_plans: List[CrossModelAggregatePlan] = Field(default_factory=list) + windowed_aggregate_plans: List["WindowedAggregatePlan"] = Field(default_factory=list) + combined_expression_slots: List[ValueSlot] = Field(default_factory=list) + transform_layers: List[TransformLayer] = Field(default_factory=list) + filters_by_phase: List[FilterPhase] = Field(default_factory=list) + projection: List[SlotId] = Field(default_factory=list) + order: List[OrderEntry] = Field(default_factory=list) + limit: Optional[int] = None + offset: Optional[int] = None + stage_schema: Optional[StageSchema] = None + # Stage 7b.10 — the slot id of the active TD (resolved via + # ``_resolve_main_time_dimension``). ``None`` when the stage has no + # time dimension. Time-needing transforms (cumsum / lag / lead / + # first / last / time_shift / consecutive_periods) carry this slot's + # key in ``TransformKey.time_key``; the generator uses it for the + # ``ORDER BY`` clause of the OVER expression. + active_time_dimension_slot_id: Optional[SlotId] = None + # DEV-1450 stage 7b.15d — the concrete ``SlayerModel`` this stage renders + # against, carried from the planner so the generator binds the stage's + # FROM / joins against the SAME model the binder used. For a multi-stage + # DAG this is the stage's OWN source (e.g. ``orders`` for a stage the root + # never reads from), a ModelExtension overlay, or a synthetic model over a + # sibling stage's CTE. ``None`` for a StageSchema-scoped chain stage (the + # generator builds a synthetic model from the upstream schema) and for a + # plain single-model query (the generator uses ``bundle.source_model``). + render_source_model: Optional[SlayerModel] = None + # DEV-1543 — pass-through of ``SlayerQuery.distinct_dimension_values``. When + # ``False`` the generator skips the dim-only dedup GROUP BY and emits raw + # rows for a measure-less dimension query. + distinct_dimension_values: bool = True + # DEV-1732 — raw column keys of this stage's NON-HIDDEN time dimensions: the + # set of columns on which an explicit relational bound counts as a FRAME + # bound rather than a population filter. Computed once here so the windowed + # ``_src`` path (planner) and the ``time_shift`` shifted-CTE path + # (generator) cannot drift apart. + # + # Hidden ``TimeTruncKey`` slots are excluded deliberately: they are not + # equality-joined into ``_src`` (``_build_windowed_plans`` skips them), so + # stripping a bound on one would leave that axis unconstrained. + frame_bound_columns: List[ValueKey] = Field(default_factory=list) + + +# ``CrossModelAggregatePlan.rerooted_plan`` is a forward reference to +# ``PlannedQuery`` (defined above only after the CMA plan). Resolve it now +# that both classes exist (DEV-1450 stage 7b.15e, C1). +CrossModelAggregatePlan.model_rebuild() +# ``WindowedAggregatePlan.src_filter_rewrites`` forward-references +# ``SrcFilterRewrite``, declared just after it (DEV-1732). +WindowedAggregatePlan.model_rebuild() diff --git a/slayer/engine/planning.py b/slayer/engine/planning.py new file mode 100644 index 00000000..604996d9 --- /dev/null +++ b/slayer/engine/planning.py @@ -0,0 +1,882 @@ +"""Stage 7a.6 (DEV-1450) — ValueRegistry, TransformLowerer, ProjectionPlanner. + +Three composable concerns: + +* ``ValueRegistry`` interns ``ValueKey``s by structural identity. Two + structurally-equal keys share one ``ValueSlot`` (P2). The same key + declared with multiple ``name``s accumulates multiple + ``public_aliases`` on a single slot (P4 / C13). Alias collisions + with source columns / duplicate names are rejected per DEV-1443. + +* ``desugar_change`` / ``desugar_change_pct`` lower sugar transforms + into their underlying form. The inner operand keeps the same + structural identity across all occurrences (DEV-1446) so the + ValueRegistry interns it once. ``partition_by`` threads through to + the underlying ``time_shift`` (C6). + +* ``ProjectionPlanner`` allocates slots for declared measures and + creates hidden slots for refs that appear ONLY in order/filter. + Hidden slots are materialised but trimmed from the public projection. + +Dormant in 7a — no engine wiring. Stage 7a.7's ``stage_planner.py`` +composes these with the cross-model planner to build a +``PlannedQuery``. +""" + +from __future__ import annotations + +from typing import Callable, Dict, FrozenSet, List, Optional + +from pydantic import BaseModel, ConfigDict, Field + +from slayer.core.enums import DataType +from slayer.core.format import NumberFormat +from slayer.core.errors import ( + CanonicalAliasShadowsColumnError, + DuplicateMeasureNameError, + MeasureNameCollidesWithColumnError, +) +from slayer.core.keys import ( + AggregateKey, + ArithmeticKey, + BetweenKey, + ColumnKey, + ColumnSqlKey, + InKey, + LiteralKey, + Phase, + ScalarCallKey, + StarKey, + TimeTruncKey, + TransformKey, + ValueKey, + column_leaf, + column_path, + normalize_scalar, +) +from slayer.core.formula import RANK_FAMILY_TRANSFORMS +from slayer.core.refs import agg_kwarg_canonical_str, canonical_agg_name +from slayer.engine.binding import BoundExpr, BoundFilter +from slayer.engine.planned import SlotId, ValueSlot + +__all__ = [ + "DeclaredMeasure", + "OrderSpec", + "ProjectionPlan", + "ProjectionPlanner", + "ValueRegistry", + "desugar_change", + "desugar_change_pct", + "filter_referenced_slot_ids", + "lower_sugar_transforms", +] + + +# --------------------------------------------------------------------------- +# ValueRegistry +# --------------------------------------------------------------------------- + + +def _fill_missing_metadata( + *, + slot: ValueSlot, + updates: Dict, + label: Optional[str] = None, + type: Optional[DataType] = None, + format: Optional[NumberFormat] = None, + description: Optional[str] = None, +) -> None: + """Populate ``updates`` with each per-field value that is set on the + incoming intern call AND missing on the existing slot (DEV-1452 round-2 + refactor of the previous if-chain to keep + :meth:`ValueRegistry._merge_into_existing` under the S3776 complexity + cap). Mutates ``updates`` in place; never overrides an already-set + field on the slot. + """ + for field_name, new_value in ( + ("label", label), + ("type", type), + ("format", format), + ("description", description), + ): + if getattr(slot, field_name) is None and new_value is not None: + updates[field_name] = new_value + + +class ValueRegistry: + """Interns ``ValueKey``s by structural identity into ``ValueSlot``s. + + Constructor takes ``source_column_names``: the set of column names + on the host model used for the alias-collision validations + (``MeasureNameCollidesWithColumnError``, + ``CanonicalAliasShadowsColumnError``). Pass an empty set when the + host doesn't expose column names (or when the validations should + skip — e.g., in unit tests for the registry in isolation). + """ + + def __init__( + self, + *, + source_column_names: Optional[FrozenSet[str]] = None, + host_model_name: str = "(host)", + ) -> None: + self._source_columns: FrozenSet[str] = ( + source_column_names or frozenset() + ) + self._host_model_name = host_model_name + self._slots: Dict[SlotId, ValueSlot] = {} + self._by_key: Dict[ValueKey, SlotId] = {} + self._declared_names: Dict[str, SlotId] = {} + self._counter = 0 + # DEV-1733: every alias name already spoken for — user-declared public + # names (reserved up front, see ``reserve_public_names``) plus the + # resolved ``declared_name`` of each interned slot. HIDDEN slots are + # uniquified against this set at intern time so no renderer can emit + # two different expressions under one alias. + self._taken_names: set = set() + + def _next_id(self) -> SlotId: + self._counter += 1 + return f"s{self._counter}" + + def reserve_public_names(self, names) -> None: + """Claim user-declared names BEFORE any hidden slot is interned. + + DEV-1733: hidden-name uniquification must not depend on intern order. + Measures are interned public-slot-then-hidden-deps, so measure *i*'s + hidden dependency would otherwise be able to claim a name that measure + *j > i* later declares publicly — and public names are never renamed, + so the collision would come straight back. Reserving every declared + name first makes the outcome order-independent. + """ + for name in names: + if name: + self._taken_names.add(name) + + def _unique_hidden_name(self, declared_name: str) -> str: + """``declared_name``, suffixed ``_2`` / ``_3`` / … if already taken. + + Hidden canonical names are structural, not user-facing + (``_cumsum_inner``, ``_arith_/``, ``_scalar_abs``), so two distinct + keys of the same shape collide by construction: + ``cumsum(a:sum) + cumsum(b:sum)`` interned two hidden slots both named + ``_cumsum_inner``, the step CTE projected two columns under that one + alias, and the composite silently evaluated ``cumsum(a) + cumsum(a)``. + DEV-1692 fixed this inside the ``time_shift`` / ``consecutive_periods`` + emitters only; owning it here covers every renderer at once. + """ + if declared_name not in self._taken_names: + return declared_name + n = 2 + while f"{declared_name}_{n}" in self._taken_names: + n += 1 + return f"{declared_name}_{n}" + + def _validate_alias_collisions( + self, + *, + key: ValueKey, + declared_name: str, + public_name: Optional[str], + canonical_alias: Optional[str], + ) -> None: + """Alias-collision validations (P4 / DEV-1443). + + Split out of :meth:`intern` so the interning path reads as + validate → merge-or-create. Raises + ``MeasureNameCollidesWithColumnError`` when a public name shadows a + source column, and ``CanonicalAliasShadowsColumnError`` when a + renamed measure's canonical alias does. + + Exemption: a dimension whose public name IS its own column name + (``ColumnKey(path=(), leaf=X)`` declared as ``X``) is the column, not a + rename of it — collision check skipped. Same exemption for a local + ``TimeTruncKey`` over that same column, since a time dimension on + ``created_at`` projects the (truncated) ``created_at`` column rather + than introducing a new alias. DEV-1450 stage 7b.13: also exempt + ``ColumnSqlKey(model=..., column_name=X)`` declared as ``X`` — a + derived column (``Column.sql`` set) selected as a dimension projects + the column unchanged, identical to the plain-column case. + """ + is_self_named_dimension = ( + isinstance(key, ColumnKey) + and key.path == () + and public_name == key.leaf + ) or ( + isinstance(key, ColumnSqlKey) + and key.path == () + and public_name == key.column_name + ) or ( + isinstance(key, TimeTruncKey) + and column_path(key.column) == () + and public_name == column_leaf(key.column) + ) + # An UNNAMED ``*:`` (StarKey source) re-aggregation is exempt: its + # canonical alias (``_count``) is a structural marker, not a column + # reference — ``COUNT(*)`` reads no column, so it can't be ambiguous + # with a same-named source column. This is the chain re-count case + # (``*:count`` over a stage that already projects ``_count``). An + # EXPLICIT user name that collides still raises (canonical_alias is set + # only on a rename, so it stays None here for the unnamed form). + is_unnamed_star_agg = ( + isinstance(key, AggregateKey) + and isinstance(getattr(key, "source", None), StarKey) + and canonical_alias is None + ) + if ( + public_name is not None + and public_name in self._source_columns + and not is_self_named_dimension + and not is_unnamed_star_agg + ): + raise MeasureNameCollidesWithColumnError( + name=public_name, model=self._host_model_name, + ) + if ( + canonical_alias is not None + and canonical_alias in self._source_columns + ): + raise CanonicalAliasShadowsColumnError( + formula=declared_name, + canonical=canonical_alias, + model=self._host_model_name, + ) + + def intern( + self, + *, + key: ValueKey, + declared_name: str, + phase: Phase, + public_name: Optional[str] = None, + canonical_alias: Optional[str] = None, + hidden: bool = False, + label: Optional[str] = None, + type: Optional[DataType] = None, + expression: Optional["BoundExpr"] = None, + format: Optional[NumberFormat] = None, + description: Optional[str] = None, + ) -> SlotId: + self._validate_alias_collisions( + key=key, + declared_name=declared_name, + public_name=public_name, + canonical_alias=canonical_alias, + ) + + existing_sid = self._by_key.get(key) + if existing_sid is not None: + return self._merge_into_existing( + existing_sid=existing_sid, + public_name=public_name, + declared_name=declared_name, + hidden=hidden, + label=label, + type=type, + format=format, + description=description, + ) + + # Fresh slot. Check declared_name collision against a different key. + if public_name is not None: + owner = self._declared_names.get(public_name) + if owner is not None: + raise DuplicateMeasureNameError( + name=public_name, + occurrences=[ + self._slots[owner].declared_name, + declared_name, + ], + ) + + sid = self._next_id() + # DEV-1733: uniquify HIDDEN slot names only. A public name is the + # user's result-key contract and is never rewritten — a genuine + # duplicate there already raised ``DuplicateMeasureNameError`` above. + if hidden: + declared_name = self._unique_hidden_name(declared_name) + self._taken_names.add(declared_name) + if public_name is not None: + self._taken_names.add(public_name) + public_aliases = [public_name] if public_name is not None else [] + slot = ValueSlot( + id=sid, + key=key, + declared_name=declared_name, + public_name=public_name, + public_aliases=public_aliases, + hidden=hidden, + phase=phase, + label=label, + type=type, + expression=expression if expression is not None else BoundExpr(value_key=key), + format=format, + description=description, + ) + self._slots[sid] = slot + self._by_key[key] = sid + if public_name is not None: + self._declared_names[public_name] = sid + return sid + + def _merge_into_existing( + self, + *, + existing_sid: SlotId, + public_name: Optional[str], + declared_name: str, + hidden: bool, + label: Optional[str] = None, + type: Optional[DataType] = None, + format: Optional[NumberFormat] = None, + description: Optional[str] = None, + ) -> SlotId: + slot = self._slots[existing_sid] + updates: Dict = {} + if public_name is not None and public_name not in slot.public_aliases: + owner = self._declared_names.get(public_name) + if owner is not None and owner != existing_sid: + raise DuplicateMeasureNameError( + name=public_name, + occurrences=[ + self._slots[owner].declared_name, + declared_name, + ], + ) + updates["public_aliases"] = list(slot.public_aliases) + [public_name] + if slot.hidden: + updates["hidden"] = False + updates["public_name"] = public_name + self._declared_names[public_name] = existing_sid + elif not hidden and slot.hidden and public_name is None: + # Re-intern as non-hidden — promote to public. + updates["hidden"] = False + # Codex: when a hidden slot is promoted to public, carry the + # display metadata supplied by the public re-intern. Only fill + # missing fields — never overwrite metadata the first intern + # already supplied. + _fill_missing_metadata( + slot=slot, + updates=updates, + label=label, + type=type, + format=format, + description=description, + ) + if updates: + new_slot = slot.model_copy(update=updates) + self._slots[existing_sid] = new_slot + return existing_sid + + def get(self, slot_id: SlotId) -> ValueSlot: + return self._slots[slot_id] + + def find_by_key(self, key: ValueKey) -> Optional[SlotId]: + return self._by_key.get(key) + + @property + def slots(self) -> List[ValueSlot]: + return list(self._slots.values()) + + +# --------------------------------------------------------------------------- +# TransformLowerer +# --------------------------------------------------------------------------- + + +def desugar_change(key: TransformKey) -> ArithmeticKey: + """``change(x)`` → ``x - time_shift(x, periods=-1, [partition_by=…])``. + + The inner ``x`` is identity-preserving — the ``ArithmeticKey`` and + the ``TransformKey`` use the SAME ``ValueKey`` instance, so a + downstream ValueRegistry interns it as one slot (DEV-1446). + + ``partition_by`` (the binder put it on ``key.partition_keys``) + threads through to the underlying ``time_shift`` (C6). ``periods`` + is fixed at ``-1`` (one period back) because ``change`` has no + user-tunable offset. + """ + if key.op != "change": + raise ValueError( + f"desugar_change expected op='change', got {key.op!r}." + ) + inner = key.input + shifted = TransformKey( + op="time_shift", + input=inner, + kwargs=(("periods", normalize_scalar(-1)),), + partition_keys=key.partition_keys, + time_key=key.time_key, + ) + return ArithmeticKey(op="-", operands=(inner, shifted)) + + +def lower_sugar_transforms(key: ValueKey) -> ValueKey: + """Recursively lower ``change`` / ``change_pct`` TransformKeys to + their desugared arithmetic form, preserving the inner aggregate's + structural identity (DEV-1446). Other ValueKey shapes are walked + but otherwise unchanged. + + The desugar functions preserve ``partition_keys`` / ``time_key`` on + the resulting ``time_shift`` TransformKey (DEV-1450 C6), so + ``change(amount:sum, partition_by=region)`` lowers to + ``amount:sum - time_shift(amount:sum, partition_by=region)``. + """ + if isinstance(key, TransformKey): + new_input = lower_sugar_transforms(key.input) + if new_input is not key.input: + key = key.model_copy(update={"input": new_input}) + if key.op == "change": + return desugar_change(key) + if key.op == "change_pct": + return desugar_change_pct(key) + return key + if isinstance(key, ArithmeticKey): + new_ops = tuple(lower_sugar_transforms(op) for op in key.operands) + if all(a is b for a, b in zip(new_ops, key.operands)): + return key + return ArithmeticKey(op=key.op, operands=new_ops) + if isinstance(key, ScalarCallKey): + new_args = tuple( + lower_sugar_transforms(a) + if isinstance( + a, _SLOTTABLE_KIND + (ArithmeticKey, ScalarCallKey, BetweenKey), + ) + else a + for a in key.args + ) + if all(a is b for a, b in zip(new_args, key.args)): + return key + return ScalarCallKey(name=key.name, args=new_args) + if isinstance(key, BetweenKey): + new_col = lower_sugar_transforms(key.column) + new_low = lower_sugar_transforms(key.low) + new_high = lower_sugar_transforms(key.high) + if ( + new_col is key.column + and new_low is key.low + and new_high is key.high + ): + return key + return BetweenKey(column=new_col, low=new_low, high=new_high) + if isinstance(key, InKey): + # DEV-1475: ``InKey.values`` is a literal-only tuple, so it + # carries no sugar to lower; only the LHS column can host a + # rewritable transform. Rebuild only if the column changed. + new_col = lower_sugar_transforms(key.column) + if new_col is key.column: + return key + return InKey(column=new_col, values=key.values, negated=key.negated) + return key + + +def rewrite_rank_partition_keys( # NOSONAR(S3776) — sequential isinstance dispatch over the closed ValueKey union; each branch is the per-type identity-preserving rebuild contract, mirroring lower_sugar_transforms. Extracting per-type helpers would scatter the contract across the module. + key: ValueKey, *, rewrite_fn: Callable[[TransformKey], FrozenSet], +) -> ValueKey: + """Walk ``key``; for every rank-family ``TransformKey`` carrying an + explicit ``partition_by`` (non-empty ``partition_keys``), replace those + keys with ``rewrite_fn(transform_key)`` (DEV-1497). + + ``rewrite_fn`` receives the whole ``TransformKey`` and returns a new + ``frozenset`` of partition keys — validating that each resolves to a query + dimension / time-dimension and rewriting a time-dimension source column to + its ``TimeTruncKey`` bucket. It may raise ``ValueError`` for a partition + column that is not a query dimension. + + Identity-preserving (mirrors :func:`lower_sugar_transforms`): parents are + rebuilt only where a child changed, so this runs BEFORE interning without + churning unrelated slots. Reaches rank transforms nested in composite + measures (``ArithmeticKey`` / ``ScalarCallKey``) and in filter predicates + (comparisons are ``ArithmeticKey``). + """ + def _rec(k: ValueKey) -> ValueKey: + return rewrite_rank_partition_keys(key=k, rewrite_fn=rewrite_fn) + + if isinstance(key, TransformKey): + new_input = _rec(key.input) + new_pk = key.partition_keys + if key.op in RANK_FAMILY_TRANSFORMS and key.partition_keys: + new_pk = rewrite_fn(key) + if new_input is key.input and new_pk == key.partition_keys: + return key + return key.model_copy(update={"input": new_input, "partition_keys": new_pk}) + if isinstance(key, ArithmeticKey): + new_ops = tuple(_rec(op) for op in key.operands) + unchanged = all(a is b for a, b in zip(new_ops, key.operands)) + return key if unchanged else ArithmeticKey(op=key.op, operands=new_ops) + if isinstance(key, ScalarCallKey): + rewritable = _SLOTTABLE_KIND + (ArithmeticKey, ScalarCallKey, BetweenKey) + new_args = tuple( + _rec(a) if isinstance(a, rewritable) else a for a in key.args + ) + unchanged = all(a is b for a, b in zip(new_args, key.args)) + return key if unchanged else ScalarCallKey(name=key.name, args=new_args) + if isinstance(key, BetweenKey): + new_col, new_low, new_high = _rec(key.column), _rec(key.low), _rec(key.high) + unchanged = ( + new_col is key.column and new_low is key.low and new_high is key.high + ) + return key if unchanged else BetweenKey( + column=new_col, low=new_low, high=new_high, + ) + if isinstance(key, InKey): + new_col = _rec(key.column) + return key if new_col is key.column else InKey( + column=new_col, values=key.values, negated=key.negated, + ) + return key + + +def desugar_change_pct(key: TransformKey) -> ArithmeticKey: + """``change_pct(x)`` → ``(x - time_shift(x, periods=-1)) / + NULLIF(time_shift(x, periods=-1), 0)``. + + The divisor is wrapped in ``NULLIF(..., 0)`` so a zero prior-period + value yields NULL instead of a divide-by-zero error / Inf. Same + identity-preservation as ``desugar_change`` — numerator and divisor + share the one ``shifted`` ValueKey instance. + """ + if key.op != "change_pct": + raise ValueError( + f"desugar_change_pct expected op='change_pct', got {key.op!r}." + ) + inner = key.input + shifted = TransformKey( + op="time_shift", + input=inner, + kwargs=(("periods", normalize_scalar(-1)),), + partition_keys=key.partition_keys, + time_key=key.time_key, + ) + numerator = ArithmeticKey(op="-", operands=(inner, shifted)) + guarded_divisor = ScalarCallKey( + name="nullif", args=(shifted, normalize_scalar(0)), + ) + return ArithmeticKey(op="/", operands=(numerator, guarded_divisor)) + + +# --------------------------------------------------------------------------- +# ProjectionPlanner +# --------------------------------------------------------------------------- + + +class DeclaredMeasure(BaseModel): + """One declared measure on a query. + + ``bound`` is the binder's output. ``declared_name`` is the canonical + or user-supplied name. ``public_name`` is the user-facing alias — + set when the user supplied an explicit ``name`` on the measure spec. + + DEV-1452 Stage B decisions #2 + #8: ``type``, ``format``, and + ``description`` carry typed display + slot metadata from the source + ``ModelMeasure`` / ``Column`` so the public slot retains the same + contract the legacy enrichment pipeline produced. ``type`` mirrors + the legacy ``EnrichedMeasure.type`` (count → INT, avg → DOUBLE, + sum/min/max → source column type). + """ + + model_config = ConfigDict(arbitrary_types_allowed=True) + + bound: BoundExpr + declared_name: str + public_name: Optional[str] = None + label: Optional[str] = None + canonical_alias: Optional[str] = None + type: Optional[DataType] = None + format: Optional[NumberFormat] = None + description: Optional[str] = None + + +class OrderSpec(BaseModel): + """One ORDER BY entry on a query.""" + + model_config = ConfigDict(arbitrary_types_allowed=True) + + bound: BoundExpr + direction: str = "asc" + + +class ProjectionPlan(BaseModel): + """ProjectionPlanner output: registry + projection order + filters / order.""" + + model_config = ConfigDict(arbitrary_types_allowed=True) + + registry: "ValueRegistry" + public_projection: List[SlotId] = Field(default_factory=list) + filters: List[BoundFilter] = Field(default_factory=list) + order: List["OrderSpec"] = Field(default_factory=list) + + +_SLOTTABLE_KIND = ( + ColumnKey, ColumnSqlKey, AggregateKey, TransformKey, TimeTruncKey, +) + + +def _iter_slot_deps(key: ValueKey): + """Yield only ``ValueKey``s that need a materialised slot. + + Skips composite-only nodes that the SQL generator inlines: + ``ArithmeticKey`` (operators), ``ScalarCallKey`` (function calls + inlined into SELECT / WHERE), ``LiteralKey``, ``StarKey``. Stops + at ``AggregateKey`` (its inner ``source`` ColumnKey is materialised + inside the aggregate, not as a separate slot). Recurses into + ``TransformKey.input`` so a nested aggregate inside a transform + gets its own hidden slot. + + ``TimeTruncKey`` is itself the materialised slot (the generator + emits the DATE_TRUNC at SELECT time); the inner ColumnKey is not + yielded as a separate dependency — adding a time dimension must + not auto-add the raw column as an output (matches legacy). + """ + if isinstance(key, AggregateKey): + yield key + return + if isinstance(key, TransformKey): + yield key + yield from _iter_slot_deps(key.input) + # Transform aux deps: partition_keys and time_key must be + # materialised as their own slots so the SQL generator (slice + # 7b.10 / 7b.11) can render PARTITION BY / ORDER BY against + # named SELECT projections instead of re-walking the model + # graph. + for pk in key.partition_keys: + yield from _iter_slot_deps(pk) + if key.time_key is not None: + yield from _iter_slot_deps(key.time_key) + return + if isinstance(key, (ColumnKey, ColumnSqlKey, TimeTruncKey)): + yield key + return + if isinstance(key, ArithmeticKey): + for op in key.operands: + yield from _iter_slot_deps(op) + return + if isinstance(key, ScalarCallKey): + for arg in key.args: + if isinstance( + arg, + _SLOTTABLE_KIND + (ArithmeticKey, ScalarCallKey, BetweenKey), + ): + yield from _iter_slot_deps(arg) + return + if isinstance(key, BetweenKey): + # BetweenKey is not itself a slot — the generator inlines it + # into WHERE. Recurse into the column / low / high so the + # underlying ColumnKey shows up as a referenced slot for the + # cross-model routing / hidden-slot pass (Codex F4). + yield from _iter_slot_deps(key.column) + yield from _iter_slot_deps(key.low) + yield from _iter_slot_deps(key.high) + if isinstance(key, InKey): + # DEV-1475: InKey, like BetweenKey, is inlined into WHERE by the + # generator (no public slot of its own). Surface its LHS column + # for cross-model routing / hidden-slot collection; the literal + # RHS values are never slottable. + yield from _iter_slot_deps(key.column) + # StarKey, LiteralKey — never slottable on their own. + + +class ProjectionPlanner: + """Allocate slots for declared measures + hidden slots for refs only + used in order/filter.""" + + @staticmethod + def _intern_hidden(registry: "ValueRegistry", key: ValueKey) -> None: + """Intern ``key`` as a hidden slot unless it already has one. + + The single rule every hidden-slot site shares — measure aux deps, + filter operands, order operands, and (DEV-1733) an order target that + is itself a composite. Keeping it in one place is what stops the four + call sites drifting on ``declared_name`` / ``phase``. + """ + if registry.find_by_key(key) is None: + registry.intern( + key=key, + declared_name=_canonical_name(key), + hidden=True, + phase=key.phase, + ) + + def plan( + self, + *, + measures: List[DeclaredMeasure], + filters: List[BoundFilter], + order: List[OrderSpec], + source_column_names: Optional[FrozenSet[str]] = None, + host_model_name: str = "(host)", + ) -> ProjectionPlan: + registry = ValueRegistry( + source_column_names=source_column_names, + host_model_name=host_model_name, + ) + # DEV-1733: claim every user-declared name before the first intern, so + # hidden-name uniquification is independent of intern order (a hidden + # dependency of measure 1 must not be able to take a name measure 2 + # declares publicly — public names are never renamed). + registry.reserve_public_names( + name + for m in measures + for name in (m.declared_name, m.public_name, m.canonical_alias) + ) + public_projection: List[SlotId] = [] + for m in measures: + sid = registry.intern( + key=m.bound.value_key, + declared_name=m.declared_name, + public_name=m.public_name, + canonical_alias=m.canonical_alias, + phase=m.bound.phase, + label=m.label, + type=m.type, + format=m.format, + description=m.description, + ) + public_projection.append(sid) + # Materialise any auxiliary slot-worthy deps of the measure + # as hidden slots (e.g. the inner AggregateKey of a transform, + # the partition columns, the time_key column). These are + # rendered by the generator into the inner SELECT but not + # surfaced in the public projection. + for dep in _iter_slot_deps(m.bound.value_key): + if dep != m.bound.value_key: + self._intern_hidden(registry, dep) + + # Filter and order share the same dependency-selection rule: walk + # the bound expression, intern each slot-worthy key as a hidden + # slot if not already present. + for f in filters: + for dep in _iter_slot_deps(f.value_key): + self._intern_hidden(registry, dep) + + for o in order: + for dep in _iter_slot_deps(o.bound.value_key): + self._intern_hidden(registry, dep) + # DEV-1733: ``_iter_slot_deps`` yields a composite's OPERANDS but + # never the composite itself (the generator normally inlines those + # nodes). An ORDER BY target that IS a composite therefore had no + # slot of its own, so ``plan_query``'s ``find_by_key`` lookup + # returned None and the order entry was SILENTLY DROPPED — the + # ``change`` / ``change_pct`` / scalar-call ORDER BY bug. Intern the + # top-level key so it gets a hidden slot the generator can + # materialise and the outer wrap can order on. + # + # ORDER ONLY: filters keep the operands-only walk. A filter's + # top-level composite is rendered inline into WHERE / HAVING, and + # giving it a slot would change that emission. + if isinstance(o.bound.value_key, (ArithmeticKey, ScalarCallKey)): + self._intern_hidden(registry, o.bound.value_key) + + return ProjectionPlan( + registry=registry, + public_projection=public_projection, + filters=filters, + order=order, + ) + + +def _canonical_name(key: ValueKey) -> str: # NOSONAR(S3776) — sequential isinstance dispatch over the closed ValueKey union; each branch is the per-type canonical-name contract. Extracting per-type helpers would scatter the contract. + """Best-effort canonical name for a hidden slot. + + Mirrors the public-alias canonical form used by the engine + elsewhere: ``revenue:sum`` → ``revenue_sum``; ``*:count`` → + ``_count``; ``customers.regions.name`` → flattened ``customers__regions__name``. + """ + if isinstance(key, ColumnKey): + return "__".join(key.path + (key.leaf,)) + if isinstance(key, ColumnSqlKey): + prefix = "__".join(key.path) + "__" if key.path else "" + return f"{prefix}{key.column_name}" + if isinstance(key, TimeTruncKey): + # Legacy alias contract: granularity is encoded in the SQL + # DATE_TRUNC, not in the alias. + return _canonical_name(key.column) + if isinstance(key, AggregateKey): + # DEV-1501: include args/kwargs so parametric aggregates over the + # same value column (``revenue:last(created_at)`` vs + # ``revenue:last(updated_at)``, ``revenue:percentile(p=0.5)`` vs + # ``revenue:percentile(p=0.95)``) get DISTINCT declared names — + # mirrors the cross-model parametric P10 exception. Without this, + # two hidden parametric aggregates collide on a single base-CTE + # alias when materialised. + if isinstance(key.source, StarKey): + measure_name = "*" + else: + leaf = getattr(key.source, "leaf", None) or getattr( + key.source, "column_name", None, + ) + if leaf is None: + return f"_agg_{key.agg}" + measure_name = leaf + agg_args = ( + [agg_kwarg_canonical_str(a) for a in key.args] + if key.args + else None + ) + agg_kwargs = ( + {k: agg_kwarg_canonical_str(v) for k, v in key.kwargs} + if key.kwargs + else None + ) + return canonical_agg_name( + measure_name=measure_name, + aggregation_name=key.agg, + agg_args=agg_args, + agg_kwargs=agg_kwargs, + ) + if isinstance(key, TransformKey): + return f"_{key.op}_inner" + if isinstance(key, ArithmeticKey): + return f"_arith_{key.op}" + if isinstance(key, ScalarCallKey): + return f"_scalar_{key.name}" + if isinstance(key, LiteralKey): + return f"_lit_{key.value}" + if isinstance(key, StarKey): + return "_star" + if isinstance(key, BetweenKey): + # Defensive — BetweenKey shouldn't materialise as a public slot + # in 7b.9; it's always inlined into WHERE by the renderer. + return f"_between_{_canonical_name(key.column)}" + if isinstance(key, InKey): + # Defensive — InKey (DEV-1475) is always inlined into WHERE + # like BetweenKey; never materialises as a public slot. + return f"_in_{_canonical_name(key.column)}" + return "_hidden" + + +ProjectionPlan.model_rebuild() + + +# --------------------------------------------------------------------------- +# Stage 7b.5 — filter → slot id mapping for cross-model planner routing +# --------------------------------------------------------------------------- + + +def filter_referenced_slot_ids( + bound_filter: "BoundFilter", + registry: "ValueRegistry", +) -> "set": + """Return the set of ``SlotId``s that ``bound_filter``'s predicate + references through interned slots. + + Walks the predicate's ``ValueKey`` tree via ``_iter_slot_deps`` — + yielding only slot-worthy keys (``ColumnKey`` / ``ColumnSqlKey`` / + ``AggregateKey`` / ``TransformKey`` / ``TimeTruncKey``) and skipping + composite-only nodes (``ArithmeticKey``, ``ScalarCallKey``, + ``LiteralKey``, ``StarKey``). Each slot-worthy key is looked up in + the registry; keys without an interned slot are silently skipped + (filter literals, hidden registry misses). + + Codex HIGH #3/#4 for DEV-1450: this helper exists so the + cross-model planner gets ``set[SlotId]`` instead of having to + classify ``BoundFilter.referenced_keys`` (which are + pre-interning ``ValueKey``s, not slot ids) or naively walking only + the top-level key (which misses composite-predicate leaves). + """ + result: set = set() + for dep in _iter_slot_deps(bound_filter.value_key): + sid = registry.find_by_key(dep) + if sid is not None: + result.add(sid) + return result diff --git a/slayer/engine/profiling.py b/slayer/engine/profiling.py index ecc39f12..2e5f908c 100644 --- a/slayer/engine/profiling.py +++ b/slayer/engine/profiling.py @@ -224,9 +224,14 @@ async def _profile_numeric_temporal_columns( """Profile every numeric/temporal column in a single batched min/max query.""" if not columns: return {} + # Deliberately omit ``type`` on the ext columns: DEV-1361's CAST wrap + # on the aggregation expression (``CAST(MIN(ordered_at) AS TIMESTAMP)``) + # is harmful on SQLite, which has no TIMESTAMP type and falls back to + # NUMERIC affinity — coercing ``'2025-01-15'`` to the int ``2025``. The + # profile query only needs the raw min/max value, so we keep the column + # untyped and let the backend return whatever native shape it stores. ext_columns = [ - {"name": f"_slayer_range_{c.name}", "sql": c.sql if c.sql else c.name, - "type": str(c.type)} + {"name": f"_slayer_range_{c.name}", "sql": c.sql if c.sql else c.name} for c in columns ] measures_payload: list[dict[str, str]] = [] diff --git a/slayer/engine/query_engine.py b/slayer/engine/query_engine.py index 8f2c0fc9..9ebfbbee 100644 --- a/slayer/engine/query_engine.py +++ b/slayer/engine/query_engine.py @@ -1,47 +1,42 @@ """Query engine — central orchestrator for SLayer queries. -Flow: SlayerQuery → _enrich() → EnrichedQuery → SQLGenerator → SQL → execute +Flow: SlayerQuery → plan_query() → PlannedQuery → SQLGenerator → SQL → execute """ import copy import decimal import logging import re -from contextvars import ContextVar -from typing import Any, Callable +from collections.abc import Callable +from typing import Any, Dict, List, Optional +import sqlalchemy as sa from pydantic import ( BaseModel, ConfigDict as PydanticConfigDict, Field as PydanticField, model_validator, ) -import sqlalchemy as sa -from sqlglot import exp from slayer.core.enums import DEFAULT_AGGREGATIONS_BY_TYPE, DataType from slayer.core.errors import AmbiguousModelError, ForcedFilterError from slayer.core.policy import JoinFilterRuleset, SessionPolicy -from slayer.core.format import NumberFormat, NumberFormatType, format_number +from slayer.core.format import format_number from slayer.core.models import ( Column, DatasourceConfig, - ModelJoin, ModelMeasure, SlayerModel, - SourceModelOrigin, ) from slayer.core.query import ( - ColumnRef, SlayerQuery, - TimeDimension, _contains_block_delimiter, coerce_declared_list_variables, declares_variables, - extract_placeholder_names, list_valued_variable_names, substitute_variables, ) +from slayer.core.warnings import NormalizationWarning from slayer.core.recommend import ( CandidateCoverage, ItemPath, @@ -56,16 +51,24 @@ RefreshResult, _CacheEntry, ) -from slayer.engine.join_graph import JoinGraph, min_hops_root -from slayer.engine.enriched import ( - CrossModelMeasure, - EnrichedMeasure, - EnrichedQuery, - public_projection_aliases, +from slayer.engine.agg_registry import collect_reachable_agg_names +from slayer.engine.normalization import normalize_model, normalize_query +from slayer.engine.planned import PlannedQuery +from slayer.engine.response_meta import ( + FieldMetadata as FieldMetadata, # re-export for slayer_client / tests + ResponseAttributes, + build_response_metadata, +) +from slayer.engine.source_bundle import ( + ResolvedSourceBundle, + build_resolved_source_bundle, + expand_query_backed_models_in_bundle, ) -from slayer.engine.enrichment import enrich_query +from slayer.engine.stage_ordering import topologically_order_stages +from slayer.engine.stage_planner import plan_stages +from slayer.engine.variables import apply_variables_to_query from slayer.engine.introspect_utils import _safe_get_columns -from slayer.engine import timing +from slayer.engine.join_graph import JoinGraph, min_hops_root from slayer.memories.resolver import ( _all_models_in_datasource, resolve_entity, @@ -74,25 +77,17 @@ from slayer.sql.dialects import SqlDialect, dialect_for_ds_type, get_dialect from slayer.sql import engine_factory from slayer.sql.engine_factory import _runtime_fingerprint -from slayer.sql.generator import SQLGenerator -from slayer.sql.reserved_keywords import SLAYER_RESERVED_KEYWORDS +from slayer.sql.generator import generate_planned_stages from slayer.sql.session_policy import ScopedTable, apply_session_policy +from slayer.sql.stage_wrapper import build_flat_rename_wrapper from slayer.storage.base import StorageBackend logger = logging.getLogger(__name__) -def _sql_client_cache_key(datasource: DatasourceConfig) -> tuple[str, str]: - """Cache key for ``SlayerQueryEngine._sql_clients``. - - Mirrors ``engine_factory``'s cache key so two datasources differing - in (e.g.) Snowflake ``warehouse`` get distinct ``SlayerSQLClient`` - instances (and therefore distinct factory-cached engines with the - correct per-connection ``USE`` listener). - """ - return (datasource.get_connection_string(), _runtime_fingerprint(datasource)) - - +# ------------------------------------------------------------------ +# recommend_root_model (DEV-1626) — ported from origin/main (DEV-1717) +# ------------------------------------------------------------------ class _ResolvedItem(BaseModel): """A recommend_root_model input item after resolution/validation.""" @@ -209,53 +204,26 @@ def dominates(a: tuple, b: tuple) -> bool: entries.sort(key=lambda e: (-len(e[0].reachable_items), e[1], e[0].model_name)) return [e[0] for e in entries] +_PLACEHOLDER_FILL_VALUE = "0" -# Per-task in-flight join-target names. Used by _resolve_join_target to break -# loops when a query-backed target's own join graph references it back. Lives -# in a ContextVar (not on the engine) so concurrent requests through the same -# engine don't see each other's in-flight state — each asyncio task gets its -# own copy of the context. The default=None + lazy-init pattern below means -# only tasks that actually hit a query-backed join target allocate a set. -_join_target_resolving_var: ContextVar[set | None] = ContextVar( - "_join_target_resolving", default=None -) - - -# Per-task "forbidden sibling stage names" — names that exist in the enclosing -# source_queries list but are NOT visible from the stage currently being -# resolved (i.e. forward references and self references). Used by -# _resolve_model_inner to differentiate forward/self refs from genuine -# misspellings, so the user gets a clear error instead of "Model 'X' not found". -# Each entry maps a forbidden target name to the stage that tried to reach it. -_forbidden_sibling_refs_var: ContextVar[dict[str, str] | None] = ContextVar( - "_forbidden_sibling_refs", default=None -) - - -class _NoJoinError(Exception): - """Internal sentinel raised by ``_walk_join_chain`` when - ``strict_missing_join=False`` and a hop has no matching join. Lets - callers like ``_resolve_dimension_with_terminal`` map a missing - join to a ``None`` return without re-walking the path.""" - def __init__(self, hop_name: str) -> None: - super().__init__(f"no join target named {hop_name!r}") - self.hop_name = hop_name - - -# EXPLAIN prefix/postfix moved to per-dialect classes in -# ``slayer/sql/dialects/`` (DEV-1542). ``_build_explain_sql`` below -# delegates via ``get_dialect``. +def _sql_client_cache_key(datasource: DatasourceConfig) -> tuple[str, str]: + """Cache key for ``SlayerQueryEngine._sql_clients``. -_PLACEHOLDER_FILL_VALUE = "0" + Mirrors ``engine_factory``'s cache key so two datasources differing + in (e.g.) Snowflake ``warehouse`` get distinct ``SlayerSQLClient`` + instances (and therefore distinct factory-cached engines with the + correct per-connection ``USE`` listener) — DEV-1551. + """ + return (datasource.get_connection_string(), _runtime_fingerprint(datasource)) def _merge_query_variables( *, - outer: dict[str, Any] | None, - stage: dict[str, Any] | None, - runtime: dict[str, Any] | None, -) -> dict[str, Any]: + outer: Optional[Dict[str, Any]], + stage: Optional[Dict[str, Any]], + runtime: Optional[Dict[str, Any]], +) -> Dict[str, Any]: """Merge variable layers per spec precedence: ``runtime > stage > outer``. Model-level defaults are folded into ``outer`` by the caller before @@ -382,67 +350,32 @@ def _render_probe_model(model: SlayerModel, *, dialect: SqlDialect) -> SlayerMod return model -def _apply_placeholder_fill( - query: SlayerQuery, effective: dict[str, Any] -) -> dict[str, Any]: - """Add ``{var: '0'}`` for any unresolved ``{var}`` placeholder in - ``query.filters`` so save-time dry-run SQL generation can proceed even - when a runtime variable has no default. - - Existing values in ``effective`` are preserved. - """ - placeholders = extract_placeholder_names(query) - missing = {p: _PLACEHOLDER_FILL_VALUE for p in placeholders if p not in effective} - if not missing: - return effective - return {**missing, **effective} def _build_explain_sql(dialect: str, sql: str) -> str: """Build a dialect-appropriate EXPLAIN statement. - Delegates to the dialect strategy registered in - ``slayer.sql.dialects`` (DEV-1542). Both branches that previously - raised ``ValueError`` (unsupported dialect via the None-prefix sentinel, - AND today's ``_EXPLAIN_PREFIX.get(...)`` miss for typo-ed dialect - names) preserve that exception type — the strict ``KeyError`` from - ``get_dialect`` is converted to ``ValueError`` to match. + DEV-1716: delegates to the dialect strategy's ``build_explain_sql`` hook + (raises ``ValueError`` for dialects without SQL-level EXPLAIN, e.g. + BigQuery) instead of an inline prefix/postfix map. """ - try: - active = get_dialect(dialect) - except KeyError: - raise ValueError( - f"EXPLAIN is not supported for dialect '{dialect}'. " - "Use dry_run=True to inspect the generated SQL instead." - ) from None - return active.build_explain_sql(sql) - - -class FieldMetadata(BaseModel): - """Metadata for a single field in the query response.""" - - label: str | None = None - format: NumberFormat | None = None - - -class ResponseAttributes(BaseModel): - """Field metadata for a query response, split by type.""" - - dimensions: dict[str, FieldMetadata] = PydanticField(default_factory=dict) - measures: dict[str, FieldMetadata] = PydanticField(default_factory=dict) - - def get(self, column: str) -> FieldMetadata | None: - """Look up metadata for a column across both dicts.""" - return self.dimensions.get(column) or self.measures.get(column) + return get_dialect(dialect).build_explain_sql(sql) class SlayerResponse(BaseModel): """Response from a SLayer query.""" - data: list[dict[str, Any]] - columns: list[str] = PydanticField(default_factory=list) - sql: str | None = None + data: List[Dict[str, Any]] + columns: List[str] = PydanticField(default_factory=list) + sql: Optional[str] = None attributes: ResponseAttributes = PydanticField(default_factory=ResponseAttributes) + # DEV-1450 stage 6 — slack-normalization warnings. + # Structured payload for each slack rewrite the normalization layer + # performed on the input (function-style aggs, misplaced measures, + # AST-resolvable dotted refs in raw SQL). Empty for queries that + # arrived in canonical form. Surfaced alongside the result so + # REST / MCP / CLI consumers can echo the rewrites back to authors. + warnings: List[NormalizationWarning] = PydanticField(default_factory=list) @model_validator(mode="after") def _populate_columns(self) -> "SlayerResponse": @@ -476,64 +409,64 @@ def to_markdown(self) -> str: return "\n".join([header, separator] + body_lines) -class _Prepared(BaseModel): - """DB-free result of the resolve→enrich→SQL-gen pipeline. - - Everything ``execute()`` needs to run (or cache-key) a query without a - database round-trip. Produced by :meth:`SlayerQueryEngine._prepare_pipeline` - and shared by the execute path, ``evict`` (key only), and ``refresh`` - re-execution. ``ds_fingerprint`` is ``"|".join(_sql_client_cache_key(ds))`` - — the cache-key datasource identity. - """ - - model_config = PydanticConfigDict(arbitrary_types_allowed=True) - - model: SlayerModel - enriched: Any - datasource: DatasourceConfig - dialect: str - sql: str - attributes: "ResponseAttributes" - expected_columns: list[str] - ds_key: tuple[str, str] - ds_fingerprint: str - - -def _infer_aggregated_format( +def _normalize_source_query_stages( model: SlayerModel, - measure_name: str, - aggregation: str, -) -> NumberFormat | None: - """Infer NumberFormat for an aggregated measure based on aggregation type and source measure format. - - Rules: - - count, count_distinct: always INTEGER - - avg, weighted_avg, median: always FLOAT - - sum, min, max, first, last: inherit from source measure - - *:count (measure_name="*"): INTEGER + *, + custom_aggs: Optional[frozenset[str]], +) -> SlayerModel: + """For a query-backed model, run ``normalize_query`` on every stage so + funcstyle aggregations over joined-model custom aggs (DEV-1500) land in + canonical form at save time — ``normalize_model`` itself only walks + ``model.measures``, never ``source_queries``. No-op for table-backed + models. CR PR #153 thread r3330620881. """ - if measure_name == "*": - return NumberFormat(type=NumberFormatType.INTEGER) + if not model.source_queries: + return model + new_stages = [ + normalize_query(stage, model=None, custom_agg_names=custom_aggs).query + for stage in model.source_queries + ] + return model.model_copy(update={"source_queries": new_stages}) - if aggregation in ("count", "count_distinct"): - return NumberFormat(type=NumberFormatType.INTEGER) - if aggregation in ("avg", "weighted_avg", "median"): - return NumberFormat(type=NumberFormatType.FLOAT) +class _Prepared(BaseModel): + """DB-free product of ``_prepare_pipeline`` (DEV-1715). + + Everything the execute / cache-hook / evict / refresh paths need after + resolve→enrich→plan→SQL-gen→policy but BEFORE any SQL-client construction. + ``sql`` is the FINAL, policy-rewritten SQL that is actually executed, so a + cache key computed from it (``make_key(sql, ds_fingerprint)``) always + matches the executed statement. ``resolved_data_source`` is + ``datasource.name`` — the authoritative datasource used to run the query + (not ``model.data_source``, which is less authoritative for inline / + expanded query-backed models). + + The ds-key / ds-fingerprint are intentionally NOT eager fields: computing + them calls ``datasource.get_connection_string()``, which some dialects + (e.g. Snowflake) reject without credentials. A ``dry_run`` must still + render SQL for a credential-less datasource, so the fingerprint is computed + lazily via ``_ds_fingerprint`` only on the cache / execute paths. + """ - # sum, min, max, first, last: inherit from source column's format - source_col = model.get_column(measure_name) - if source_col and source_col.format: - return source_col.format + model_config = PydanticConfigDict(arbitrary_types_allowed=True) - return None + sql: str + dialect: str + datasource: DatasourceConfig + resolved_data_source: Optional[str] = None + attributes: Any + expected_columns: List[str] + touched: set + model: SlayerModel + slack_warnings: List[Any] = PydanticField(default_factory=list) class SlayerQueryEngine: """Central orchestrator: resolves queries via storage, generates SQL, executes. - The engine enriches a SlayerQuery (user-facing, just names) into an - EnrichedQuery (fully resolved SQL expressions), then passes it to the + The engine resolves a SlayerQuery (user-facing, just names) into a + PlannedQuery (typed value keys interned into slots, each carrying its + resolved expression, join path and phase), then passes it to the SQLGenerator for SQL generation. """ @@ -541,45 +474,49 @@ def __init__( self, storage: StorageBackend, *, - cache_config: CacheConfig | None = None, - policy: SessionPolicy | None = None, + policy: Optional[SessionPolicy] = None, + cache_config: Optional[CacheConfig] = None, ): self.storage = storage + # DEV-1587 / DEV-1715: per-engine, in-memory, opt-in query result cache. + # ``cache_config`` defaults to an empty ``CacheConfig()`` (caches + # indefinitely with no auto-staleness). The cache lives on the engine + # instance, so two engines with different connection settings / policy + # keep separate caches. + self._cache = QueryCache(config=cache_config or CacheConfig()) # Cache key: (connection_string, runtime_fingerprint) — matches # ``engine_factory``'s cache so Snowflake datasources sharing a - # connection_name but differing in warehouse/role/database/schema - # get distinct clients (DEV-1551). + # connection_name but differing in warehouse/role/database/schema get + # distinct clients (DEV-1551). self._sql_clients: dict[tuple[str, str], SlayerSQLClient] = {} # DEV-1578: immutable, engine-global forced-filter policy. When set, # every generated SQL is rewritten to scope each physical table to the - # configured tenant before execution / dry-run / explain / profiling. + # configured tenant before execution / dry-run / explain. self.policy = policy # Cache of confirmed column-presence facts keyed by - # (ds_key, schema, table, column). Only ``True``/``False`` are cached; - # an unconfirmable ``None`` is re-probed so a transient introspection - # failure self-heals once the datasource recovers. + # (ds_key, catalog, schema, table, column). Only ``True``/``False`` are + # cached; an unconfirmable ``None`` is re-probed so a transient + # introspection failure self-heals once the datasource recovers. self._column_presence_cache: dict[tuple, bool] = {} # DEV-1627: cached ClickHouse ``(major, minor)`` server version per - # datasource, for the correlated-subquery join-rule gate. ``None`` (or - # a missing entry) fails closed. Populated by + # datasource, for the correlated-subquery join-rule gate. ``None`` (or a + # missing entry) fails closed. Populated by # ``_preflight_clickhouse_correlated`` before the policy rewrite. self._ch_version_cache: dict[tuple[str, str], tuple[int, int] | None] = {} - # DEV-1587: per-engine, in-memory, opt-in query result cache. The - # cache is local to this engine instance so two engines with - # different RLS / connection settings keep separate caches. - self._cache = QueryCache(config=cache_config or CacheConfig()) + + # ---- query cache management (DEV-1587 / DEV-1715) ---------------------- @property def cache_config(self) -> CacheConfig: + """The active :class:`CacheConfig` (read-only view of ``_cache``).""" return self._cache.config @cache_config.setter - def cache_config(self, value: CacheConfig) -> None: - # Reassigning the config drops all cached entries — the new TTL / - # refresh-key policy shouldn't retroactively apply to entries whose - # baselines were captured under the old policy. - self._cache.config = value - self._cache.clear() + def cache_config(self, config: CacheConfig) -> None: + """Reassign the cache policy. This **clears the cache** — stale entries + must not survive under a new TTL / refresh-key set. The existing clock + is preserved so an injected test clock survives the reassignment.""" + self._cache = QueryCache(config=config, clock=self._cache._clock) @property def cache_size(self) -> int: @@ -587,7 +524,7 @@ def cache_size(self) -> int: return self._cache.size() def clear_cache(self) -> None: - """Drop all cached entries.""" + """Drop every cached entry.""" self._cache.clear() def _apply_policy( @@ -767,261 +704,55 @@ def _column_present( self._column_presence_cache[key] = present return present - def _get_join_target_resolving(self) -> set: - """Return the per-task in-flight join-target name set, allocating one - on first access in this asyncio context. See ``_join_target_resolving_var``. - """ - s = _join_target_resolving_var.get() - if s is None: - s = set() - _join_target_resolving_var.set(s) - return s - - @staticmethod - def _scope_named_queries_to_prior( - named_queries: dict[str, "SlayerQuery"], stage_name: str | None - ) -> dict[str, "SlayerQuery"]: - """Slice an insertion-ordered named-queries dict to entries that - come strictly before ``stage_name``. - - When a non-final stage of a ``source_queries`` list is being - resolved, only its *prior* siblings are visible to it. This keeps - the DAG acyclic. Runtime query lists pre-sort via - :meth:`_topologically_order_queries` so the insertion order here - is already a valid topological order; ``SlayerModel.source_queries`` - retains strict-order semantics and relies on this slice plus the - forward-reference error in ``_resolve_model_inner`` to catch - out-of-order references. - - Returns ``named_queries`` unchanged when ``stage_name`` is None or - absent from the dict (e.g. the final stage, or an externally-named - stored model). - """ - if not stage_name or stage_name not in named_queries: - return named_queries - out: dict[str, "SlayerQuery"] = {} - for k, v in named_queries.items(): - if k == stage_name: - return out - out[k] = v - return out - - @staticmethod - def _extract_sibling_refs(query: "SlayerQuery", against: set) -> set: - """Names from ``query.source_model`` / inline joins that match ``against``. - - Walks the three shapes ``source_model`` can take — plain string, - dict (``ModelExtension`` or inline ``SlayerModel``), or typed - instance — and collects every name that resolves against the - ``against`` set. Used by both the dependency-graph builder and - the self-reference / root-as-sink validators. - """ - out: set = set() - sm = query.source_model - if isinstance(sm, str): - if sm in against: - out.add(sm) - return out - # Dict shape — disambiguate ModelExtension vs inline SlayerModel - # by presence of ``source_name``. - if isinstance(sm, dict): - src = sm.get("source_name") - if isinstance(src, str) and src in against: - out.add(src) - for j in sm.get("joins") or []: - tgt = j.get("target_model") if isinstance(j, dict) else getattr(j, "target_model", None) - if isinstance(tgt, str) and tgt in against: - out.add(tgt) - return out - # Typed ModelExtension / SlayerModel: source_name lives on - # ModelExtension only; SlayerModel.name is the inline model's - # own identifier, not a reference. - src = getattr(sm, "source_name", None) - if isinstance(src, str) and src in against: - out.add(src) - for j in getattr(sm, "joins", None) or []: - tgt = getattr(j, "target_model", None) - if isinstance(tgt, str) and tgt in against: - out.add(tgt) - return out - - @staticmethod - def _index_query_list_by_name( - rest: list["SlayerQuery"], root: "SlayerQuery", - ) -> dict[str, "SlayerQuery"]: - """Build ``{name: query}`` for non-final entries, validating that - every non-final entry has a unique name and that the root's - name (if any) doesn't collide. - """ - rest_by_name: dict[str, "SlayerQuery"] = {} - for q in rest: - if not q.name: - raise ValueError( - "Every non-final entry in a query list must have a " - "'name' (siblings reference each other by name)." - ) - if q.name in rest_by_name: - raise ValueError(f"Duplicate stage name '{q.name}' in query list.") - rest_by_name[q.name] = q - if root.name and root.name in rest_by_name: - raise ValueError( - f"Stage name '{root.name}' is duplicated: the final entry " - f"shares a name with an earlier entry." - ) - return rest_by_name - - @classmethod - def _validate_query_list_invariants( - cls, - queries: list["SlayerQuery"], - rest: list["SlayerQuery"], - root: "SlayerQuery", - sibling_names: set, - ) -> None: - """Reject self-references and any sibling that depends on the root. - - Self-references are caught for every entry (including the root). - Root-as-sink: no non-final stage may reference the root by name. - """ - for q in queries: - if q.name and q.name in cls._extract_sibling_refs(q, {q.name} | sibling_names): - raise ValueError( - f"Stage '{q.name}' references itself — self-references " - f"are not allowed." - ) - if root.name: - referrers = sorted( - q.name for q in rest if root.name in cls._extract_sibling_refs(q, {root.name}) - ) - if referrers: - raise ValueError( - f"The final entry '{root.name}' is the DAG root and must " - f"not be referenced by other stages. Referenced by: " - f"{referrers}." - ) - @classmethod - def _build_dependency_graph( + def _topologically_order_queries( cls, - rest_by_name: dict[str, "SlayerQuery"], - sibling_names: set, - ) -> tuple: - """Build the (in_degree, dependents) adjacency for Kahn's. - - Edge direction: prerequisite → dependent. ``in_degree[X]`` is - the count of siblings ``X`` depends on; ``dependents[X]`` is the - list of siblings that depend on ``X``. + queries: List["SlayerQuery"], + ) -> List["SlayerQuery"]: + """Thin classmethod shim — delegates to + :func:`slayer.engine.stage_ordering.topologically_order_stages` + (DEV-1452 Stage B decision #1). Existing callers continue to use + the engine-method surface; the migrated paths import the helper + directly. """ - in_degree: dict[str, int] = dict.fromkeys(rest_by_name, 0) - dependents: dict[str, list[str]] = {name: [] for name in rest_by_name} - for name, q in rest_by_name.items(): - for prereq in cls._extract_sibling_refs(q, sibling_names): - dependents[prereq].append(name) - in_degree[name] += 1 - return in_degree, dependents + return topologically_order_stages(queries) - @staticmethod - def _kahn_sort( - in_degree: dict[str, int], - dependents: dict[str, list[str]], - ) -> list[str]: - """Topologically sort by Kahn's algorithm. Cycle → ValueError. - - Mutates ``in_degree`` in place; callers shouldn't reuse it. - The frontier is kept sorted for deterministic output order across - runs. - """ - frontier: list[str] = sorted(n for n, d in in_degree.items() if d == 0) - sorted_names: list[str] = [] - while frontier: - n = frontier.pop(0) - sorted_names.append(n) - unlocked: list[str] = [] - for dep in dependents[n]: - in_degree[dep] -= 1 - if in_degree[dep] == 0: - unlocked.append(dep) - frontier.extend(sorted(unlocked)) - if len(sorted_names) < len(in_degree): - cycle = sorted(set(in_degree) - set(sorted_names)) - raise ValueError( - f"Cycle in query list: stages {cycle} form a cyclic " - f"dependency. The reference graph must be acyclic." - ) - return sorted_names + async def aclose(self) -> None: + """Dispose every cached client's async engine; keep the clients themselves. - @classmethod - def _topologically_order_queries( - cls, - queries: list["SlayerQuery"], - ) -> list["SlayerQuery"]: - """Re-order a runtime query list so every stage appears after the - siblings it references via ``source_model`` or - ``joins.target_model``. Lets callers submit a DAG in any order — - cycles and self-references are rejected; the input order itself - no longer needs to be a valid topological order. - - The last entry of the input is the entry point / DAG root: its - result is what ``execute`` returns. It stays last; only the - non-final entries are reordered. Stages that aren't reachable - from the root are accepted as utility sub-queries — they flow - through the sort like any other node and remain in the - ``named_queries`` dict (the SQL generator emits them only if - something references them). - - Hand-rolled Kahn's algorithm; no ``graphlib`` dependency. The - actual work is delegated to four single-purpose helpers - (:meth:`_index_query_list_by_name`, - :meth:`_validate_query_list_invariants`, - :meth:`_build_dependency_graph`, :meth:`_kahn_sort`) so this - orchestrator stays under the cognitive-complexity gate. - - Raises ``ValueError`` on: missing ``name`` on any non-final - entry; duplicate stage names; self-references; the root being - depended on by any other stage; or a cycle among non-final - stages. + Per-instance async engines bind their asyncpg/aiomysql pool to the loop + that first opened a connection; closing that loop without disposing + leaks the server-side connections (asyncpg.Connection.close needs a + live loop). Clients are kept so ``_sync_engine`` survives — important + for ``:memory:`` SQLite, whose StaticPool pins the connection holding + all data. """ - if len(queries) <= 1: - return list(queries) - rest = list(queries[:-1]) - root = queries[-1] - rest_by_name = cls._index_query_list_by_name(rest, root) - sibling_names: set = set(rest_by_name) - cls._validate_query_list_invariants(queries, rest, root, sibling_names) - in_degree, dependents = cls._build_dependency_graph(rest_by_name, sibling_names) - sorted_names = cls._kahn_sort(in_degree, dependents) - return [rest_by_name[n] for n in sorted_names] + [root] + for client in self._sql_clients.values(): + await client.aclose() async def execute( self, query: "SlayerQuery | dict | list[SlayerQuery | dict] | str", - variables: dict[str, Any] | None = None, + variables: Optional[Dict[str, Any]] = None, *, dry_run: bool = False, explain: bool = False, - data_source: str | None = None, + data_source: Optional[str] = None, cache: bool = False, ) -> SlayerResponse: - """Resolve, enrich, generate SQL, and execute ``query``. - - Accepts a ``SlayerQuery`` / dict, a multi-stage DAG list, or a - run-by-name string. DEV-1587: pass ``cache=True`` to serve the result - from (and store it in) this engine's per-instance result cache; - ignored (no caching, no error) when ``dry_run`` or ``explain`` is set. - """ - query_obj, named_queries, runtime_kwarg, prefer = await self._normalize_input( - query, variables=variables, data_source=data_source, + runtime_kwarg = variables or {} + main_query, named_queries, prefer_data_source = await self._normalize_input( + query, runtime_kwarg=runtime_kwarg, prefer_data_source=data_source ) return await self._execute_pipeline( - query=query_obj, + query=main_query, named_queries=named_queries, runtime_kwarg=runtime_kwarg, dry_run=dry_run, explain=explain, - prefer_data_source=prefer, + prefer_data_source=prefer_data_source, cache=cache, original_input=query, - original_variables=variables, original_data_source=data_source, ) @@ -1029,23 +760,25 @@ async def _normalize_input( # NOSONAR S3776 — public dispatch over str/dict/l self, query: "SlayerQuery | dict | list[SlayerQuery | dict] | str", *, - variables: dict[str, Any] | None = None, - data_source: str | None = None, - ) -> "tuple[SlayerQuery, dict[str, SlayerQuery], dict[str, Any], str | None]": - """Normalize the ``execute`` / ``evict`` input union into - ``(query, named_queries, runtime_kwarg, prefer_data_source)``. - - Folds the run-by-name lookup, list topo-sort, dict validation, and - ``variables=`` merge. Shared by the execute path and ``evict`` so a - cached entry's key is computed identically on both. + runtime_kwarg: Dict[str, Any], + prefer_data_source: Optional[str], + ) -> "tuple[SlayerQuery, Dict[str, SlayerQuery], Optional[str]]": + """Resolve the user input union into ``(main_query, named_queries, + prefer_data_source)`` shared by ``execute()``, ``evict()``, and + ``refresh()`` re-exec (DEV-1715). + + Async because the ``str`` (run-by-name) branch reads storage — which is + exactly what lets ``refresh()`` re-prep pick up ``source_queries`` + edits and pin the model lookup to the entry's originally-resolved + datasource (``prefer_data_source``). """ - runtime_kwarg = variables or {} - - # Run-by-name dispatch: ``execute("model_name", variables=...)`` runs - # the backing query of a query-backed model. + # Run-by-name dispatch: ``execute("model_name", ...)`` runs the backing + # query of a query-backed model. if isinstance(query, str): return await self._normalize_by_name( - name=query, runtime_kwarg=runtime_kwarg, data_source=data_source, + name=query, + runtime_kwarg=runtime_kwarg, + prefer_data_source=prefer_data_source, ) # Accept dicts and validate them into SlayerQuery objects @@ -1060,32 +793,39 @@ async def _normalize_input( # NOSONAR S3776 — public dispatch over str/dict/l # last entry stays last as the entry point. Validates names, # duplicates, self-refs, root-as-sink, and cycles up front. queries = self._topologically_order_queries(queries) - query = queries[-1] + main_query = queries[-1] named_queries = {q.name: q for q in queries[:-1] if q.name} else: if isinstance(query, dict): query = SlayerQuery.model_validate(query) + main_query = query named_queries = {} # Merge ``variables=`` kwarg into query.variables so filter # substitution and downstream resolution see the merged set. # ``runtime_kwarg`` always wins (per spec precedence). if runtime_kwarg: - merged_top = {**(query.variables or {}), **runtime_kwarg} - if merged_top != (query.variables or {}): - query = query.model_copy(update={"variables": merged_top}) + merged_top = {**(main_query.variables or {}), **runtime_kwarg} + if merged_top != (main_query.variables or {}): + main_query = main_query.model_copy(update={"variables": merged_top}) - return query, named_queries, runtime_kwarg, data_source + return main_query, named_queries, prefer_data_source async def _normalize_by_name( self, *, name: str, - runtime_kwarg: dict[str, Any], - data_source: str | None, - ) -> "tuple[SlayerQuery, dict[str, SlayerQuery], dict[str, Any], str | None]": - """Resolve a run-by-name input into the normalized tuple.""" - model = await self.storage.get_model(name, data_source=data_source) + runtime_kwarg: Dict[str, Any], + prefer_data_source: Optional[str], + ) -> "tuple[SlayerQuery, Dict[str, SlayerQuery], Optional[str]]": + """Normalize a run-by-name input into the shared prepare tuple. + + ``prefer_data_source`` pins the ``storage.get_model`` lookup: on a + ``refresh()`` re-exec it is the entry's originally-resolved datasource, + so a datasource-priority flip after caching cannot re-read a different + query-backed model (Codex #3). + """ + model = await self.storage.get_model(name, data_source=prefer_data_source) if model is None: raise ValueError(f"Model '{name}' not found") if not model.source_queries: @@ -1094,9 +834,15 @@ async def _normalize_by_name( f"with source_model='{name}'." ) - stages = list(model.source_queries) + # Codex: stored ``source_queries`` may be in non-topological order + # for ``joins[].target_model`` deps (the save path's + # ``_expand_query_backed_model`` calls ``topologically_order_stages`` + # so it accepts that; ``plan_stages._topo_sort`` only handles + # ``source_model`` deps). Re-use the engine-wide topo-sort here so + # run-by-name matches save-time semantics. + stages = topologically_order_stages(list(model.source_queries)) main_query = stages[-1] - named_queries: dict[str, SlayerQuery] = {} + named_queries: Dict[str, SlayerQuery] = {} for q in stages[:-1]: if q.name: if q.name in named_queries: @@ -1118,29 +864,29 @@ async def _normalize_by_name( if merged != (main_query.variables or {}): main_query = main_query.model_copy(update={"variables": merged}) - return main_query, named_queries, runtime_kwarg, model.data_source or data_source + return main_query, named_queries, model.data_source or prefer_data_source - async def _prepare_pipeline( + async def _prepare_pipeline( # NOSONAR S3776 — linear pipeline (resolve→enrich→generate→policy); breaking it up obscures the order of operations self, query: SlayerQuery, - named_queries: dict[str, SlayerQuery], - runtime_kwarg: dict[str, Any], + named_queries: Dict[str, SlayerQuery], + runtime_kwarg: Dict[str, Any], *, - prefer_data_source: str | None = None, + prefer_data_source: Optional[str] = None, + override_datasource: Optional[DatasourceConfig] = None, ) -> _Prepared: - """Resolve → enrich → generate SQL (+ forced-filter policy rewrite) → - build response metadata → compute the datasource cache identity. - - No ``SlayerSQLClient`` is instantiated and no data query runs, so with - no forced-filter policy configured this is fully DB-free — the basis - for ``evict`` recomputing a cache key without executing. When a policy - IS set, ``_apply_policy`` probes column presence via a SQLAlchemy - ``Inspector`` (a lightweight, cached metadata connection), so under RLS - the key computation touches the database that far — and must, since the - cached SQL is itself policy-rewritten and the key has to match. - - Shared by ``_execute_pipeline`` (execute + cache miss), ``evict`` (key - only), and ``refresh`` re-execution. + """DB-free-ish prepare portion shared by execute / evict / refresh + (DEV-1715): resolve→enrich→normalize→plan→SQL-gen→ClickHouse-preflight→ + policy-rewrite→response-metadata. Produces the FINAL executed SQL and + the datasource fingerprint but constructs **no** SQL client on the + common (no-policy) path — so ``evict()`` recomputes a cache key without + connecting. When a policy IS configured, producing the correct SQL may + introspect column presence / preflight ClickHouse; that is inherent to + the rewrite. + + Assumes ``query.variables`` already reflects the resolved variable + context for the top of the chain (kwarg merged in by ``_normalize_ + input``). """ # Pre-processing: strip redundant source model name prefixes from all references query = query.strip_source_model_prefix() @@ -1149,169 +895,220 @@ async def _prepare_pipeline( for name, q in named_queries.items() } + # Preprocessing if query.whole_periods_only: query = query.snap_to_whole_periods() - resolving: set = set() - _t = timing.start() - model = await self._resolve_query_model( - query_model=query.source_model, + # P11 — build the resolved source bundle once. Storage is consulted + # here and only here; the binder then reads from the bundle purely. + bundle = await build_resolved_source_bundle( + query=query, + storage=self.storage, + data_source=prefer_data_source, + runtime_variables=runtime_kwarg, named_queries=named_queries, - _resolving=resolving, + ) + + # Expand every query-backed model in the bundle (source + referenced + # + stage_source) and re-apply root inline_extensions. Shared with + # the migrated ``_expand_query_backed_model`` path (DEV-1452 Stage B + # decision F) so both surfaces consume the identical expansion + # contract — divergence between them silently broke nested + # query-backed targets in the legacy stack. + original_source_model = bundle.source_model + bundle = await expand_query_backed_models_in_bundle( + bundle=bundle, outer_vars=query.variables, runtime_kwarg=runtime_kwarg, - prefer_data_source=prefer_data_source, + dry_run_placeholders=False, + expander=self._expand_query_backed_model, ) - timing.record("resolve_model", _t) - - # Auto-correct: move bare field names to dimensions if they match - query = await self._auto_move_fields_to_dimensions(query, model, named_queries) - - _t = timing.start() - datasource = await self._resolve_datasource(model=model) - timing.record("resolve_datasource", _t) - - # DEV-1625: substitute {var} into the direct source model's Mode-A + # ``build_resolved_source_bundle`` raises if the source model can't be + # resolved, so ``source_model`` is always populated here. + model = bundle.source_model + assert model is not None + + # ``override_datasource`` pins the connection identity (used by + # refresh() re-exec): the query is re-run against the EXACT datasource + # the entry was cached under (its ds_key), not whatever the model's + # ``data_source`` name resolves to now — so a same-name repoint can't + # migrate the entry or store rows from a different database. + # + # Resolved HERE (before Mode-A substitution) because DEV-1727 escaping + # is dialect-aware and needs it. Safe to hoist: substitution rewrites + # only the four raw-SQL surfaces, never ``model.data_source``. + datasource = override_datasource or await self._resolve_datasource(model=model) + + # DEV-1625: substitute {var} into the DIRECT source model's Mode-A # raw-SQL surfaces (sql / filters / Column.sql / Column.filter) before - # enrichment, so the substituted body is what sqlglot parses AND what - # cross-model re-rooting (which reuses this same ``model`` object) sees. - # Guarded to template models — a rendered virtual model - # (``source_model_origin`` set, from a query-backed source) is skipped; - # its stages / join targets / cross-model targets are DEV-1678 scope. - # ``query.variables`` already merges runtime > stage > outer; the - # model's own defaults are the lowest layer. + # anything parses them. The typed planner, binder and cross-model + # renderers all read models from the bundle, so the substituted copy + # replaces the model both as ``source_model`` and under its name in + # ``referenced_models``. A rendered virtual model + # (``source_model_origin`` set, from a query-backed source) is + # skipped; nested stages / join targets / cross-model targets are + # DEV-1678 scope. ``bundle.query_variables`` already merges + # runtime > stage > outer > model defaults. + # + # DEV-1730: called UNCONDITIONALLY (no ``and bundle.query_variables`` + # guard) — the helper is a no-op for a variable-free, block-free model + # (preserving DEV-1625 raw-brace-literal protection), but a + # block-bearing model must still run so its ``{? ?}`` blocks collapse + # to ``(1=1)`` even on a zero-variable call. DEV-1727: escaping is + # dialect-aware, so the resolved datasource's dialect is threaded in — + # backslash dialects (MySQL/ClickHouse/…) get the hardened regime, + # standard dialects the ``''`` doubling. if model.source_model_origin is None: - effective_vars = {**(model.query_variables or {}), **(query.variables or {})} - # Always call: the helper is a no-op for a variable-free, block-free - # model (preserving DEV-1625 raw-brace-literal protection), but a - # block-bearing model must still run so its {? ?} blocks collapse to - # (1=1) even on a zero-variable call (DEV-1730). DEV-1727: escaping is - # dialect-aware — pass the resolved datasource's dialect so backslash - # dialects (MySQL/ClickHouse/…) get the hardened regime, standard - # dialects the '' doubling. - model = _substitute_model_sql_surfaces( + substituted = _substitute_model_sql_surfaces( model=model, - variables=effective_vars, + variables=bundle.query_variables, dialect=dialect_for_ds_type(datasource.type), ) + bundle = bundle.model_copy( + update={ + "source_model": substituted, + "referenced_models": [ + substituted if m.name == model.name else m + for m in bundle.referenced_models + ], + } + ) + model = substituted + + # P0 — slack-normalization pass. Rewrites slack-but-unambiguous agent + # input (function-style aggs, misplaced bare measures) to canonical + # form before the typed parser sees it. Each stage normalizes against + # its own resolved model; warnings surface on the response. + sibling_names = set(named_queries) + query, slack_warnings = self._normalize_stage( + query=query, bundle=bundle, sibling_names=sibling_names, + ) + normed_named: Dict[str, SlayerQuery] = {} + for nm, nq in named_queries.items(): + nq2, nq_warnings = self._normalize_stage( + query=nq, bundle=bundle, sibling_names=sibling_names, + ) + normed_named[nm] = nq2 + slack_warnings.extend(nq_warnings) + + # Variable substitution into filters (the only field legacy + # substituted). Root uses the bundle's merged variables; each sibling + # re-merges with its own stage layer (precedence runtime > stage > + # outer > model defaults). + query = apply_variables_to_query( + query=query, variables=bundle.query_variables, + ) + root_vars = query.variables + normed_named = { + nm: apply_variables_to_query( + query=nq, + variables={ + # Lowest layer: the stage's OWN source-model defaults (a + # sibling-sourced stage has no resolved model here, so fall + # back to the root model's defaults). + **( + ( + bundle.stage_source_models[nm].query_variables + if nm in bundle.stage_source_models + else (model.query_variables if model else None) + ) + or {} + ), + **(root_vars or {}), + **(nq.variables or {}), + **(runtime_kwarg or {}), + }, + ) + for nm, nq in normed_named.items() + } - # Enrich: SlayerQuery + model → EnrichedQuery - _t = timing.start() - enriched = await self._enrich(query=query, model=model, named_queries=named_queries) - timing.record("enrich", _t) + # Plan the DAG (root last) and render the whole chain to one SQL string. + stages = [*normed_named.values(), query] + planned_list = plan_stages(queries=stages, bundle=bundle) + root_planned = planned_list[-1] - # Generate SQL from EnrichedQuery. DEV-1444: pin ``outer`` mode so the - # projection is trimmed to public_projection_aliases(enriched). dialect = self._dialect_for_type(datasource.type) - generator = SQLGenerator(dialect=dialect) - _t = timing.start() - sql = generator.generate(enriched=enriched, render_mode="outer") - timing.record("generate_sql", _t) - # DEV-1578: forced-filter rewrite. Applied here — before the dry_run - # return and before execution — so dry_run, explain, real execution, - # and profiling all see the same tenant-scoped SQL. - # DEV-1627: probe/cache the ClickHouse server version first (no-op for - # non-ClickHouse dialects and for policies without join rules) so the - # correlated-subquery guard has a version to gate on. + sql = generate_planned_stages( + planned_list, bundle=bundle, dialect=dialect, + ) + # DEV-1578: forced-filter (RLS) rewrite — scope each physical table to + # the configured tenant. Applied to the rendered SQL before dry-run / + # explain / execute so all three surfaces (and the cache key) see the + # policy-rewritten SQL. Zero overhead (returns unchanged) when no + # policy is configured. await self._preflight_clickhouse_correlated( dialect=dialect, datasource=datasource ) sql = self._apply_policy(sql=sql, dialect=dialect, datasource=datasource) logger.debug("Generated SQL:\n%s", sql) - attributes, expected_columns = self._build_response_metadata( - enriched=enriched, model=model, + # Response metadata (attributes + expected_columns) from the typed + # plan + rendered SQL. expected_columns reads the outer SELECT's + # result keys straight from the SQL; attributes classify each public + # slot dimension-vs-measure with its label / format. + attributes, expected_columns = build_response_metadata( + root_planned=root_planned, bundle=bundle, sql=sql, dialect=dialect, ) - # DEV-1587: datasource cache identity, computed from config only - # (no client instantiation). DEV-1551: includes Snowflake runtime - # overrides so a warehouse/role change re-keys. - ds_key = _sql_client_cache_key(datasource) - ds_fingerprint = "|".join(ds_key) + # Models whose live schema a query-time DBAPI error could be attributed + # to (the typed-plan equivalent of the legacy enriched-derived set). + touched = self._touched_models_for_plan( + bundle=bundle, + planned_list=planned_list, + original_source_model=original_source_model, + ) return _Prepared( - model=model, - enriched=enriched, - datasource=datasource, - dialect=dialect, sql=sql, + dialect=dialect, + datasource=datasource, + resolved_data_source=datasource.name, attributes=attributes, - expected_columns=expected_columns, - ds_key=ds_key, - ds_fingerprint=ds_fingerprint, + expected_columns=list(expected_columns), + touched=touched, + model=model, + slack_warnings=slack_warnings, ) - def _build_response_metadata( # NOSONAR S3776 — linear per-bucket alias/format collection (dims/tds/measures/exprs/transforms/cross-model); splitting the flat loops adds indirection without reducing the branch count - self, *, enriched: "EnrichedQuery", model: SlayerModel, - ) -> "tuple[ResponseAttributes, list[str]]": - """Build the response ``attributes`` (label/format per public alias) - and ``expected_columns`` from an enriched query — mirroring the - trimmed outer projection (DEV-1444). Hidden transforms / ORDER-BY - aggregates / window-arg hoists are dropped.""" - public_aliases = set(public_projection_aliases(enriched)) - - dim_meta: dict[str, FieldMetadata] = {} - measure_meta: dict[str, FieldMetadata] = {} - for d in enriched.dimensions: - if d.alias in public_aliases and (d.label or d.format): - dim_meta[d.alias] = FieldMetadata(label=d.label, format=d.format) - for td in enriched.time_dimensions: - if td.alias in public_aliases and td.label: - dim_meta[td.alias] = FieldMetadata(label=td.label) - for m in enriched.measures: - if m.alias not in public_aliases: - continue - measure_fmt = _infer_aggregated_format( - model=model, - measure_name=m.source_measure_name or m.name, - aggregation=m.aggregation, - ) - if m.label or measure_fmt: - measure_meta[m.alias] = FieldMetadata(label=m.label, format=measure_fmt) - for e in enriched.expressions: - if e.alias not in public_aliases: - continue - measure_meta[e.alias] = FieldMetadata( - label=e.label, - format=NumberFormat(type=NumberFormatType.FLOAT), - ) - for t in enriched.transforms: - if t.alias not in public_aliases: - continue - measure_meta[t.alias] = FieldMetadata( - label=t.label, - format=NumberFormat(type=NumberFormatType.FLOAT), - ) - for cm in enriched.cross_model_measures: - if cm.alias in public_aliases and (cm.label or cm.format): - measure_meta[cm.alias] = FieldMetadata(label=cm.label, format=cm.format) - attributes = ResponseAttributes(dimensions=dim_meta, measures=measure_meta) - - expected_columns = list(public_projection_aliases(enriched)) or ( - [d.alias for d in enriched.dimensions] - + [td.alias for td in enriched.time_dimensions] - + [m.alias for m in enriched.measures if not m.name.startswith(("_inner_", "_ft"))] - + [e.alias for e in enriched.expressions] - + [t.alias for t in enriched.transforms if not t.name.startswith(("_inner_", "_ft"))] - + [cm.alias for cm in enriched.cross_model_measures] - ) - return attributes, expected_columns + @staticmethod + def _ds_fingerprint(datasource: DatasourceConfig) -> str: + """The datasource identity fingerprint used in the cache key — + ``connection_string|runtime_fingerprint``. Computed lazily (only on the + cache / execute paths) because ``get_connection_string`` rejects some + credential-less dialects that a ``dry_run`` must still render.""" + return "|".join(_sql_client_cache_key(datasource)) + + def _client_for(self, datasource: DatasourceConfig) -> SlayerSQLClient: + """Reuse (or lazily construct) the cached SQL client for a datasource. + + Keyed by ``_sql_client_cache_key`` so two datasources differing only in + (e.g.) Snowflake warehouse get distinct clients (DEV-1551). + """ + ds_key = _sql_client_cache_key(datasource) + if ds_key not in self._sql_clients: + self._sql_clients[ds_key] = SlayerSQLClient(datasource=datasource) + return self._sql_clients[ds_key] async def _execute_pipeline( self, query: SlayerQuery, - named_queries: dict[str, SlayerQuery], - runtime_kwarg: dict[str, Any], + named_queries: Dict[str, SlayerQuery], + runtime_kwarg: Dict[str, Any], *, dry_run: bool = False, explain: bool = False, - prefer_data_source: str | None = None, + prefer_data_source: Optional[str] = None, cache: bool = False, original_input: Any = None, - original_variables: dict[str, Any] | None = None, - original_data_source: str | None = None, + original_data_source: Optional[str] = None, ) -> SlayerResponse: - """Prepare (DB-free) then run: dry_run / explain / cached / plain.""" + """Prepare (DB-free-ish) then dry-run / explain / cache-hook / execute. + + The cache hook sits at the one seam between ``_prepare_pipeline`` (which + produces the final policy-rewritten SQL + ds fingerprint) and SQL-client + construction: a hit skips only the DB execute + decode; a miss scans + refresh-key baselines BEFORE the data query and stores a deep copy. + """ prepared = await self._prepare_pipeline( query=query, named_queries=named_queries, @@ -1319,452 +1116,489 @@ async def _execute_pipeline( prefer_data_source=prefer_data_source, ) - # dry_run: return SQL without executing (never cached). + # dry_run: return SQL without executing. NEVER cached. if dry_run: return SlayerResponse( - data=[], - columns=prepared.expected_columns, - sql=prepared.sql, - attributes=prepared.attributes, - ) - - client = self._get_client(prepared.datasource, prepared.ds_key) - - # explain: dialect-appropriate EXPLAIN (never cached). + data=[], columns=prepared.expected_columns, sql=prepared.sql, + attributes=prepared.attributes, warnings=prepared.slack_warnings, + ) + + use_cache = cache and not dry_run and not explain + # Bind the cache instance ONCE for the whole cached path. A concurrent + # ``cache_config`` reassignment (the setter swaps in a fresh QueryCache) + # during the DB awaits below must not split the read (get) and write + # (put) across two caches — that would land an entry whose applicable / + # baselines were computed under the old refresh-key set into the new + # cache, defeating the "reassigning cache_config clears stale entries" + # contract on CacheConfig. + cache_obj = self._cache + key: Optional[str] = None + if use_cache: + key = QueryCache.make_key(prepared.sql, self._ds_fingerprint(prepared.datasource)) + entry = await cache_obj.get(key) + if entry is not None: + # Hit: return an independent deep copy so caller mutation can't + # poison the cached response. + return entry.response.model_copy(deep=True) + + # Miss (or cache=False) → a SQL client is required. + client = self._client_for(prepared.datasource) + + # explain: run dialect-appropriate EXPLAIN on the query. NEVER cached. if explain: explain_sql = _build_explain_sql(dialect=prepared.dialect, sql=prepared.sql) try: rows = await client.execute(sql=explain_sql) except Exception as exc: await self._maybe_raise_schema_drift( - err=exc, model=prepared.model, enriched=prepared.enriched + err=exc, model=prepared.model, touched_models=prepared.touched ) raise return SlayerResponse( - data=rows, sql=prepared.sql, attributes=prepared.attributes + data=rows, sql=prepared.sql, attributes=prepared.attributes, + warnings=prepared.slack_warnings, + ) + + # On a cache miss, capture refresh-key baselines BEFORE the data query + # (so the cached data reflects a state >= the baseline). A write-time + # baseline-scan failure PROPAGATES — nothing is stored. + applicable: list[tuple[str, str]] = [] + refresh_key_values: list[RefreshKeyValue] = [] + if use_cache: + applicable, refresh_key_values = await self._scan_refresh_key_baselines( + prepared=prepared, client=client, cache=cache_obj ) - # DEV-1587 cache hook: only for plain data queries. - if cache: - return await self._execute_cached( + rows = await self._run_data_query(prepared=prepared, client=client) + columns = prepared.expected_columns if not rows else [] # [] triggers auto-derive + response = SlayerResponse( + data=rows, columns=columns, sql=prepared.sql, + attributes=prepared.attributes, warnings=prepared.slack_warnings, + ) + + if use_cache: + entry = self._build_cache_entry( prepared=prepared, - client=client, + response=response, original_input=original_input, - original_variables=original_variables, - original_data_source=original_data_source, + variables=runtime_kwarg, + data_source=original_data_source, + created_at=cache_obj.now(), + applicable=applicable, + refresh_key_values=refresh_key_values, ) + await cache_obj.put(key, entry) + # Return the original response; the stored copy is a defensive deep copy. + return response - return await self._run_and_build(prepared=prepared, client=client) - - def _get_client( - self, datasource: DatasourceConfig, ds_key: tuple[str, str] - ) -> SlayerSQLClient: - """Reuse (or open) the SQL client + connection pool for a datasource. + async def _run_data_query( + self, *, prepared: _Prepared, client: SlayerSQLClient + ) -> "list[dict]": + """Run the prepared data query with schema-drift attribution on error. - DEV-1551: keyed by ``(connection_string, runtime_fingerprint)`` so two - datasources sharing a ``connection_name`` but differing in warehouse / - role / database / schema get distinct clients. + DEV-1716: applies the dialect's read-side ``decode_result_keys`` hook so + BigQuery / T-SQL alias-mangled result keys are reversed back to SLayer's + universal dotted shape (identity for every other dialect / on empty + rows). Shared by the execute miss path and the refresh() re-exec. """ - if ds_key not in self._sql_clients: - self._sql_clients[ds_key] = SlayerSQLClient(datasource=datasource) - return self._sql_clients[ds_key] - - async def _run_and_build( - self, *, prepared: _Prepared, client: SlayerSQLClient - ) -> SlayerResponse: - """Execute the prepared SQL, apply schema-drift attribution + the - dialect read-side decode, and build the response.""" - _t = timing.start() try: rows = await client.execute(sql=prepared.sql) except Exception as exc: await self._maybe_raise_schema_drift( - err=exc, model=prepared.model, enriched=prepared.enriched + err=exc, model=prepared.model, touched_models=prepared.touched ) raise - timing.record("execute", _t) - # Dialect-driven read-side decode: BigQuery reverses its alias - # mangling here so the response keys match SLayer's universal - # dotted shape. Default hook is identity for every other dialect. - rows = get_dialect(prepared.dialect).decode_result_keys(rows) - columns = prepared.expected_columns if not rows else [] # [] auto-derives - return SlayerResponse( - data=rows, - columns=columns, - sql=prepared.sql, - attributes=prepared.attributes, - ) - - # -- DEV-1587 query cache ------------------------------------------------- + return get_dialect(prepared.dialect).decode_result_keys(rows) - async def _execute_cached( + async def _scan_one_table_values( self, *, - prepared: _Prepared, + table: str, + exprs: "list[str]", + dialect: str, + datasource: DatasourceConfig, client: SlayerSQLClient, - original_input: Any, - original_variables: dict[str, Any] | None, - original_data_source: str | None, - ) -> SlayerResponse: - """Serve from cache or run-and-store. On both hit and miss the caller - receives a deep copy so it can't mutate the cached response.""" - key = QueryCache.make_key(prepared.sql, prepared.ds_fingerprint) - hit = await self._cache.get(key) - if hit is not None: - return hit.response.model_copy(deep=True) - response, entry = await self._build_fresh_entry( - prepared=prepared, - client=client, - original_input=original_input, - original_variables=original_variables, - original_data_source=original_data_source, - ) - await self._cache.put(key, entry) - return response.model_copy(deep=True) + ) -> "dict[str, Any]": + """Run one batched refresh-key scan for a table and return + ``{expression: value}`` (read from the ``slayer_rk_`` aliases). + + The scan SQL is policy-rewritten identically to the data query so a + tenant-scoped query's baseline can't be masked by a global MAX/COUNT. + """ + scan_sql = self._cache.build_refresh_key_sql(table, exprs, dialect) + scan_sql = self._apply_policy(sql=scan_sql, dialect=dialect, datasource=datasource) + rows = await client.execute(sql=scan_sql) + row0 = rows[0] if rows else {} + return {e: row0.get(self._cache.rk_alias(i)) for i, e in enumerate(exprs)} + + async def _scan_refresh_key_baselines( + self, *, prepared: _Prepared, client: SlayerSQLClient, cache: QueryCache + ) -> "tuple[list[tuple[str, str]], list[RefreshKeyValue]]": + """Capture the write-time refresh-key baselines for a cache entry. + + Returns ``(applicable, refresh_key_values)`` where ``applicable`` is the + entry's applicable ``(table, expression)`` refresh keys (order + dups + preserved) and ``refresh_key_values`` are the scanned baselines in the + same order. Scan failures propagate (write-time contract). + """ + applicable = cache.applicable_keys(prepared.sql, prepared.dialect) + if not applicable: + return [], [] + by_table = self._group_expressions_by_table(applicable) + scanned: dict[str, dict[str, Any]] = {} + for table, exprs in by_table.items(): + scanned[table] = await self._scan_one_table_values( + table=table, exprs=exprs, dialect=prepared.dialect, + datasource=prepared.datasource, client=client, + ) + values = [ + RefreshKeyValue(table=t, expression=e, value=scanned[t][e]) + for (t, e) in applicable + ] + return applicable, values - async def _build_fresh_entry( + @staticmethod + def _group_expressions_by_table( + applicable: "list[tuple[str, str]]", + ) -> "dict[str, list[str]]": + """Collate ``(table, expression)`` pairs into ``{table: [exprs]}``, + preserving first-occurrence order and de-duplicating identical + expressions (so the ``slayer_rk_`` alias order is stable).""" + by_table: dict[str, list[str]] = {} + for table, expr in applicable: + exprs = by_table.setdefault(table, []) + if expr not in exprs: + exprs.append(expr) + return by_table + + def _build_cache_entry( self, *, prepared: _Prepared, - client: SlayerSQLClient, + response: SlayerResponse, original_input: Any, - original_variables: dict[str, Any] | None, - original_data_source: str | None, - ) -> "tuple[SlayerResponse, _CacheEntry]": - """Run a cache miss: scan refresh-key baselines BEFORE the data query - (so cached data reflects a state >= the baseline — Codex finding #1), - execute, and build an entry holding a deep copy of the response. - - A write-time baseline-scan failure PROPAGATES (the - ``execute(cache=True)`` call fails); ``refresh()`` instead treats the - same failure as continue-on-error. - """ - applicable = self._cache.applicable_keys(prepared.sql, prepared.dialect) - refresh_key_values = await self._scan_refresh_keys( - applicable=applicable, - client=client, - dialect=prepared.dialect, - datasource=prepared.datasource, - ) - response = await self._run_and_build(prepared=prepared, client=client) - # Deep-copy the replay inputs (as we do the response): ``refresh()`` - # re-prepares from ``original_input`` / ``variables``, so a caller - # mutating their SlayerQuery / dict / list / variables dict after the - # write must not change what a later refresh re-executes. - entry = _CacheEntry( + variables: Optional[Dict[str, Any]], + data_source: Optional[str], + created_at: float, + applicable: "list[tuple[str, str]]", + refresh_key_values: "list[RefreshKeyValue]", + ) -> _CacheEntry: + """Build a ``_CacheEntry`` holding a defensive deep copy of the response + and the original user input (so later caller mutation can't change a + cached hit or a refresh replay).""" + ds_key = _sql_client_cache_key(prepared.datasource) + return _CacheEntry( response=response.model_copy(deep=True), sql=prepared.sql, - ds_fingerprint=prepared.ds_fingerprint, + ds_fingerprint="|".join(ds_key), dialect=prepared.dialect, - ds_key=prepared.ds_key, - resolved_data_source=prepared.datasource.name, + ds_key=ds_key, + resolved_data_source=prepared.resolved_data_source, original_input=copy.deepcopy(original_input), - variables=copy.deepcopy(original_variables), - data_source=original_data_source, - created_at=self._cache.now(), - applicable=applicable, - refresh_key_values=refresh_key_values, - ) - return response, entry - - async def _scan_refresh_keys( - self, - *, - applicable: list[tuple[str, str]], - client: SlayerSQLClient, - dialect: str, - datasource: DatasourceConfig, - ) -> list[RefreshKeyValue]: - """Evaluate every applicable refresh key, batched one scan per table. - Raises on a malformed / multi-row scan (the caller decides propagate - vs continue-on-error).""" - by_table: dict[str, list[str]] = {} - for table, expr in applicable: - by_table.setdefault(table, []).append(expr) - out: list[RefreshKeyValue] = [] - for table, exprs in by_table.items(): - row = await self._scan_one_table( - table=table, exprs=exprs, client=client, - dialect=dialect, datasource=datasource, - ) - for i, expr in enumerate(exprs): - out.append( - RefreshKeyValue( - table=table, - expression=expr, - value=row[self._cache.rk_alias(i)], - ) - ) - return out - - async def _scan_one_table( - self, - *, - table: str, - exprs: list[str], - client: SlayerSQLClient, - dialect: str, - datasource: DatasourceConfig, - ) -> dict[str, Any]: - scan_sql = self._cache.build_refresh_key_sql(table, exprs, dialect) - # DEV-1587 × DEV-1578: apply the forced-filter policy to the scan too, - # so the refresh-key baseline is computed over the SAME tenant-scoped - # rows as the cached data query. Without this, a global MAX/COUNT could - # mask a tenant-local change (or over-refresh on another tenant's). - # No-op (zero overhead) when no policy is configured. - scan_sql = self._apply_policy( - sql=scan_sql, dialect=dialect, datasource=datasource + # Deep copy (not a shallow ``dict(...)``) so nested list/dict values + # can't be mutated by the caller after execute and leak into a + # refresh() replay — matching the ``original_input`` snapshot above. + variables=copy.deepcopy(dict(variables)) if variables else None, + data_source=data_source, + created_at=created_at, + applicable=list(applicable), + refresh_key_values=list(refresh_key_values), ) - rows = await client.execute(sql=scan_sql) - if len(rows) != 1: - raise ValueError( - f"refresh-key scan for table '{table}' returned {len(rows)} " - f"rows (expected exactly 1)." - ) - return rows[0] async def evict( self, query: "SlayerQuery | dict | list[SlayerQuery | dict] | str", + variables: Optional[Dict[str, Any]] = None, *, - variables: dict[str, Any] | None = None, - data_source: str | None = None, + data_source: Optional[str] = None, ) -> bool: - """Remove the cache entry for ``query`` (same input union as - ``execute``). Recomputes the SQL + datasource key via the prepare - pipeline — no data query and no ``SlayerSQLClient`` (a forced-filter - policy may still introspect column presence; see ``_prepare_pipeline``). - Returns ``True`` if an entry was present.""" - query_obj, named_queries, runtime_kwarg, prefer = await self._normalize_input( - query, variables=variables, data_source=data_source, + """Remove one cached entry, recomputing its key DB-free (resolve→enrich + →SQL-gen→policy). Returns ``True`` if an entry was present. Never + constructs a SQL client on the no-policy path.""" + runtime_kwarg = variables or {} + main_query, named_queries, prefer_ds = await self._normalize_input( + query, runtime_kwarg=runtime_kwarg, prefer_data_source=data_source ) prepared = await self._prepare_pipeline( - query=query_obj, + query=main_query, named_queries=named_queries, runtime_kwarg=runtime_kwarg, - prefer_data_source=prefer, + prefer_data_source=prefer_ds, ) - key = QueryCache.make_key(prepared.sql, prepared.ds_fingerprint) + key = QueryCache.make_key(prepared.sql, self._ds_fingerprint(prepared.datasource)) return await self._cache.delete(key) def evict_sync( self, query: "SlayerQuery | dict | list[SlayerQuery | dict] | str", + variables: Optional[Dict[str, Any]] = None, *, - variables: dict[str, Any] | None = None, - data_source: str | None = None, + data_source: Optional[str] = None, ) -> bool: """Synchronous wrapper for :meth:`evict`.""" from slayer.async_utils import run_sync - return run_sync( - self.evict(query, variables=variables, data_source=data_source) - ) + async def _run() -> bool: + try: + return await self.evict(query, variables=variables, data_source=data_source) + finally: + await self.aclose() - async def refresh(self) -> RefreshResult: - """Re-scan refresh keys + TTL for every cached entry and re-execute - the stale ones (Cube-style manual refresh). Continue-on-failure. + return run_sync(_run()) - All DB awaits happen OUTSIDE the cache lock; the entry set is - snapshotted before awaiting, and each re-execution commit is - identity-guarded so a concurrent ``evict`` / ``clear_cache`` / - newer ``execute`` is never clobbered or resurrected. + async def _reexecute_entry(self, entry: _CacheEntry, now: float) -> _CacheEntry: + """Re-prepare a stale entry from its ORIGINAL input and re-execute it, + returning a fresh entry (``created_at=now`` → TTL reset). + + Replays through the full ``_normalize_input`` / ``_prepare_pipeline`` + pipeline — so model / ``source_queries`` edits and ``whole_periods_only`` + re-snapping are picked up — but the connection identity is PINNED to the + entry's ``ds_key`` via ``override_datasource`` (the datasource carried by + the client cached at write time). So neither a datasource-priority flip + NOR a same-name config edit can migrate the entry or re-execute against a + different database; a new identity is a new cache entry via ``execute``, + never a refresh migration. If the write-time client is gone the re-exec + can't be pinned faithfully → raise, and ``refresh()`` records the + ``re_execute`` error and keeps the stale entry. Any other error (e.g. the + re-exec baseline scan hitting a dropped table) propagates the same way. """ - snapshot = await self._cache.snapshot() - result = RefreshResult() + client = self._sql_clients.get(entry.ds_key) + if client is None: + raise RuntimeError( + f"no cached SQL client for datasource fingerprint {entry.ds_key!r}; " + "cannot pin re-execution to the entry's connection identity" + ) + main_query, named_queries, prefer_ds = await self._normalize_input( + entry.original_input, + runtime_kwarg=entry.variables or {}, + prefer_data_source=entry.resolved_data_source, + ) + prepared = await self._prepare_pipeline( + query=main_query, + named_queries=named_queries, + runtime_kwarg=entry.variables or {}, + prefer_data_source=prefer_ds, + override_datasource=client.datasource, + ) + applicable, refresh_key_values = await self._scan_refresh_key_baselines( + prepared=prepared, client=client, cache=self._cache + ) + rows = await self._run_data_query(prepared=prepared, client=client) + columns = prepared.expected_columns if not rows else [] + response = SlayerResponse( + data=rows, columns=columns, sql=prepared.sql, + attributes=prepared.attributes, warnings=prepared.slack_warnings, + ) + return self._build_cache_entry( + prepared=prepared, + response=response, + original_input=entry.original_input, + variables=entry.variables, + data_source=entry.data_source, + created_at=now, + applicable=applicable, + refresh_key_values=refresh_key_values, + ) + + async def refresh(self) -> RefreshResult: # NOSONAR S3776 — Cube-style refresh: snapshot → collate scans → per-entry TTL/refresh-key decision + """Cube-style explicit refresh over all cached entries. + + Snapshots the cache, runs one batched refresh-key scan per + ``(datasource-fingerprint, table)`` — through the SQL client cached at + write time, so each entry is scanned against the exact connection + identity (``ds_key``) it was cached under. Keying by fingerprint (not + the bare datasource name) mirrors the cache key: a same-name config + edit or a datasource-priority flip cannot migrate the scan to a + different database. Then per entry: TTL-expired ⇒ re-exec + (``expired_refreshed``); an applicable table whose scan failed ⇒ keep + ``unchanged``; any applicable refresh-key value moved ⇒ re-exec + (``refreshed``); else ``unchanged``. Continue-on-failure: scan / + re-exec errors become :class:`RefreshError` and keep the stale entry. + """ + result = RefreshResult() + snapshot = await self._cache.snapshot() if not snapshot: return result - fresh_values, failed_tables = await self._refresh_scan_all(snapshot, result) + + # Collate {ds_key: {table: ordered exprs}} across entries — keyed by the + # SQL-client fingerprint (connection_string|runtime_fingerprint), NOT + # the bare datasource name. An entry cached under one connection + # identity is thus scanned against THAT identity even if the datasource + # was later edited under the same name (the cache key is fingerprint- + # scoped too, so such entries coexist). + collate: dict[tuple[str, str], dict[str, list[str]]] = {} + for entry in snapshot.values(): + if not entry.applicable: + continue + tables = collate.setdefault(entry.ds_key, {}) + for table, expr in entry.applicable: + exprs = tables.setdefault(table, []) + if expr not in exprs: + exprs.append(expr) + + # One batched scan per (ds_key, table), continue-on-error per table. + # The scan runs through the client cached at write time (which carries + # its exact DatasourceConfig) — no name re-resolution, so a priority + # flip or same-name config edit can't migrate the scan. + scanned: dict[tuple[tuple[str, str], str], dict[str, Any]] = {} + failed: set[tuple[tuple[str, str], str]] = set() + for ds_key, tables in collate.items(): + client = self._sql_clients.get(ds_key) + for table, exprs in tables.items(): + if client is None: + # The write-time client is gone (should not happen — clients + # live for the engine's lifetime). Fail-soft: keep the entry + # rather than re-resolve the name to a possibly-different + # fingerprint and scan the wrong database. + failed.add((ds_key, table)) + result.errors.append(RefreshError( + key=table, phase="refresh_key_scan", + message=f"no cached SQL client for datasource fingerprint {ds_key!r}", + )) + continue + try: + datasource = client.datasource + dialect = self._dialect_for_type(datasource.type) + # Codex #5: warm the ClickHouse correlated-subquery version + # cache before policy-applying the standalone scan SQL, so + # a join-policy refresh scan matches normal execution. + await self._preflight_clickhouse_correlated( + dialect=dialect, datasource=datasource + ) + scanned[(ds_key, table)] = await self._scan_one_table_values( + table=table, exprs=exprs, dialect=dialect, + datasource=datasource, client=client, + ) + except Exception as exc: + failed.add((ds_key, table)) + result.errors.append(RefreshError( + key=table, phase="refresh_key_scan", message=str(exc), + )) + for key, entry in snapshot.items(): - await self._refresh_one( - key=key, - entry=entry, - fresh_values=fresh_values, - failed_tables=failed_tables, - result=result, + now = self._cache.now() + ttl = self._cache.config.ttl_seconds + ttl_expired = ttl is not None and (now - entry.created_at) > ttl + if ttl_expired: + bucket = result.expired_refreshed + else: + dk = entry.ds_key + if any((dk, t) in failed for (t, _e) in entry.applicable): + result.unchanged.append(key) + continue + moved = any( + QueryCache.values_differ( + scanned.get((dk, rkv.table), {}).get(rkv.expression), + rkv.value, + ) + for rkv in entry.refresh_key_values + ) + if not moved: + result.unchanged.append(key) + continue + bucket = result.refreshed + + # Re-exec. Bucket only after a SUCCESSFUL re-exec AND a landed + # commit: a re-exec failure records a re_execute error and keeps the + # stale entry; a commit skipped by the identity guard (the entry was + # concurrently evicted / cleared / superseded) must NOT be reported + # as refreshed, since the cache was not actually updated. + try: + new_entry = await self._reexecute_entry(entry, now) + except Exception as exc: + result.errors.append(RefreshError( + key=key, phase="re_execute", message=str(exc), + )) + continue + new_key = QueryCache.make_key(new_entry.sql, new_entry.ds_fingerprint) + replaced = await self._cache.commit_replace( + old_key=key, expected=entry, new_key=new_key, new_entry=new_entry, ) + if replaced: + bucket.append(key) + return result def refresh_sync(self) -> RefreshResult: - """Synchronous wrapper for :meth:`refresh`. - - Like ``execute_sync``, disposes per-call async engines in ``finally`` - (refresh runs refresh-key scans + re-executions) so they don't outlive - their owning loop. - """ + """Synchronous wrapper for :meth:`refresh`.""" from slayer.async_utils import run_sync - async def _run_and_cleanup() -> RefreshResult: + async def _run() -> RefreshResult: try: return await self.refresh() finally: await self.aclose() - return run_sync(_run_and_cleanup()) - - async def _refresh_scan_all( - self, snapshot: "dict[str, _CacheEntry]", result: RefreshResult, - ) -> "tuple[dict[tuple[tuple[str, str], str, str], Any], set]": - """Collate applicable ``(ds_key, table)`` scan targets across all - entries and run ONE batched scan per pair. Scan failures are recorded - as continue-on-error ``RefreshError(phase="refresh_key_scan")`` and the - table is marked failed so its dependent entries are left unchanged.""" - targets: dict[tuple[tuple[str, str], str], set] = {} - dialects: dict[tuple[str, str], str] = {} - clients: dict[tuple[str, str], SlayerSQLClient] = {} - for entry in snapshot.values(): - ds_key = tuple(entry.ds_key) - dialects[ds_key] = entry.dialect - for table, expr in entry.applicable: - targets.setdefault((ds_key, table), set()).add(expr) - - fresh: dict[tuple[tuple[str, str], str, str], Any] = {} - failed: set = set() - for (ds_key, table), exprs in targets.items(): - expr_list = sorted(exprs) - try: - client = await self._client_for_refresh(ds_key, snapshot, clients) - row = await self._scan_one_table( - table=table, exprs=expr_list, client=client, - dialect=dialects[ds_key], datasource=client.datasource, - ) - for i, expr in enumerate(expr_list): - fresh[(ds_key, table, expr)] = row[self._cache.rk_alias(i)] - except Exception as exc: # noqa: BLE001 — continue-on-failure per table - failed.add((ds_key, table)) - result.errors.append( - RefreshError(key=table, phase="refresh_key_scan", message=str(exc)) - ) - return fresh, failed - - async def _client_for_refresh( - self, - ds_key: tuple[str, str], - snapshot: "dict[str, _CacheEntry]", - clients: dict[tuple[str, str], SlayerSQLClient], - ) -> SlayerSQLClient: - """Get the cached client for ``ds_key``, reopening it from an entry's - recorded resolved datasource name if it isn't cached.""" - if ds_key in clients: - return clients[ds_key] - client = self._sql_clients.get(ds_key) - if client is None: - ds_name = next( - ( - e.resolved_data_source - for e in snapshot.values() - if tuple(e.ds_key) == ds_key and e.resolved_data_source - ), - None, - ) - ds = await self.storage.get_datasource(ds_name) if ds_name else None - if ds is None: - raise ValueError( - f"cannot resolve datasource for refresh (ds_key={ds_key})" - ) - client = self._get_client(ds, ds_key) - clients[ds_key] = client - return client + return run_sync(_run()) - async def _refresh_one( + def _normalize_stage( self, *, - key: str, - entry: _CacheEntry, - fresh_values: "dict[tuple[tuple[str, str], str, str], Any]", - failed_tables: set, - result: RefreshResult, - ) -> None: - """Decide + apply the per-entry refresh action: TTL-expired ⇒ re-exec - (``expired_refreshed``); else a scan failure on an applicable table ⇒ - keep unchanged; else any applicable refresh-key value moved ⇒ re-exec - (``refreshed``); else unchanged.""" - ds_key = tuple(entry.ds_key) - ttl = self._cache.config.ttl_seconds - if ttl is not None and (self._cache.now() - entry.created_at) > ttl: - await self._refresh_reexec( - key=key, entry=entry, bucket="expired_refreshed", result=result - ) - return - - entry_tables = {t for t, _ in entry.applicable} - if any((ds_key, t) in failed_tables for t in entry_tables): - result.unchanged.append(key) # scan failed → keep the stale entry - return - - moved = any( - QueryCache.values_differ( - rkv.value, fresh_values.get((ds_key, rkv.table, rkv.expression)) - ) - for rkv in entry.refresh_key_values - ) - if moved: - await self._refresh_reexec( - key=key, entry=entry, bucket="refreshed", result=result - ) + query: SlayerQuery, + bundle: ResolvedSourceBundle, + sibling_names: "set[str]", + ) -> "tuple[SlayerQuery, list[NormalizationWarning]]": + """Slack-normalize one stage against its resolved model (P0). + + Resolves the stage's source model from the bundle so MISPLACED_MEASURE + and custom-aggregation-aware FUNC_STYLE_AGG see the right column / + aggregation names. A stage sourced from a sibling (a flat StageSchema) + has no ``SlayerModel`` — it normalizes with ``model=None`` (FUNC_STYLE_ + AGG still applies; MISPLACED_MEASURE is a no-op without column names). + """ + sm = query.source_model + model: Optional[SlayerModel] = None + if query.name and query.name in bundle.stage_source_models: + # A named non-root stage resolves to its OWN source model + # (string-concrete or inline) — normalize against that, not the + # root, so MISPLACED_MEASURE / custom-agg rewrites see the right + # columns/aggregations (CR). Sibling-sourced stages are absent + # from stage_source_models, so they fall through to model=None. + model = bundle.stage_source_models[query.name] + elif isinstance(sm, str): + if sm not in sibling_names: + model = bundle.get_referenced_model(sm) + if model is None and ( + bundle.source_model is not None + and bundle.source_model.name == sm + ): + model = bundle.source_model else: - result.unchanged.append(key) + model = bundle.source_model + custom_aggs: Optional[frozenset[str]] = None + if model is not None: + # DEV-1500: scoped per-stage BFS over the pre-resolved bundle, + # so funcstyle calls over joined-model custom aggregations + # (``rolling_avg(customers.score)`` where ``rolling_avg`` lives + # on ``customers``) are recognised by the slack rewrite. + custom_aggs = bundle.reachable_aggregation_names(start=model) + norm = normalize_query(query, model=model, custom_agg_names=custom_aggs) + out = norm.query if norm.query is not None else query + return out, list(norm.warnings) + + def _touched_models_for_plan( + self, + *, + bundle: ResolvedSourceBundle, + planned_list: "list[PlannedQuery]", + original_source_model: Optional[SlayerModel], + ) -> "set[str]": + """Names of every model this query touched, for schema-drift attribution. - async def _refresh_reexec( - self, *, key: str, entry: _CacheEntry, bucket: str, result: RefreshResult, - ) -> None: - """Re-execute a stale entry and identity-guarded-commit the result. - On re-exec failure the stale entry is kept and the error recorded.""" - try: - new_key, new_entry = await self._reexecute_entry(entry, self._cache.now()) - except Exception as exc: # noqa: BLE001 — keep the stale entry on failure - result.errors.append( - RefreshError(key=key, phase="re_execute", message=str(exc)) - ) - return - committed = await self._cache.commit_replace( - old_key=key, expected=entry, new_key=new_key, new_entry=new_entry, - ) - if committed: - getattr(result, bucket).append(key) - - async def _reexecute_entry( - self, entry: _CacheEntry, now: float - ) -> "tuple[str, _CacheEntry]": - """Replay a stale entry from its ORIGINAL input through the full - ``execute`` normalization (so run-by-name picks up ``source_queries`` - edits and lists re-topo-sort), re-scanning baselines before the data - query. Returns the (possibly re-keyed) key and a fresh entry with a - new ``created_at`` (TTL reset — Codex finding #4). - - ``now`` is the refresh clock reading, retained for signature stability - (the fresh entry stamps ``created_at`` from the same injectable clock). + The bundle's ``referenced_models`` already hold the transitive join + walk plus every sibling-stage base; cross-model aggregate targets come + off each planned stage; query-backed base names are recovered from the + pre-expansion source model. ``_maybe_raise_schema_drift`` widens this + further via ``_expand_join_graph``. """ - del now # created_at is stamped inside _build_fresh_entry via the clock - # Pin the replay to the datasource this entry was ORIGINALLY resolved - # against (falling back to it when the caller passed no explicit - # data_source). A cache entry is bound to a resolved datasource - # identity (it's in the key), and refresh() already scanned this - # entry's refresh keys against that datasource — so re-executing must - # stay on it, not silently migrate to a different datasource if the - # priority list changed after the entry was cached. - replay_data_source = entry.data_source or entry.resolved_data_source - query_obj, named_queries, runtime_kwarg, prefer = await self._normalize_input( - entry.original_input, - variables=entry.variables, - data_source=replay_data_source, - ) - prepared = await self._prepare_pipeline( - query=query_obj, - named_queries=named_queries, - runtime_kwarg=runtime_kwarg, - prefer_data_source=prefer, - ) - client = self._get_client(prepared.datasource, prepared.ds_key) - _, new_entry = await self._build_fresh_entry( - prepared=prepared, - client=client, - original_input=entry.original_input, - original_variables=entry.variables, - original_data_source=entry.data_source, - ) - new_key = QueryCache.make_key(prepared.sql, prepared.ds_fingerprint) - return new_key, new_entry + touched: set[str] = {m.name for m in bundle.referenced_models} + for pq in planned_list: + for cmp in pq.cross_model_aggregate_plans: + touched.add(cmp.target_model) + if original_source_model is not None and original_source_model.source_queries: + touched.add(original_source_model.name) + touched |= self._collect_query_backed_base_names(original_source_model) + return touched @staticmethod def _collect_query_backed_base_names(model: SlayerModel) -> "set[str]": @@ -1796,103 +1630,64 @@ def _collect_query_backed_base_names(model: SlayerModel) -> "set[str]": out.add(target) return out - async def _collect_models_touched( - self, *, model: SlayerModel, enriched: "EnrichedQuery" - ) -> "set[str]": - """Compute the set of model names that participated in this query. - - Includes the source model, every cross-model measure root, every - query-backed base name (resolved from storage when ``model`` is a - virtual stage produced by ``_query_as_model``), and (transitively) - every join target reachable through the join graph. - """ - touched: set[str] = {model.name} - for cm in enriched.cross_model_measures: - touched.add(cm.target_model_name) - touched.add(cm.source_model_name) - touched |= self._collect_query_backed_base_names(model) - # The resolved ``model`` may be a virtual stage from - # _query_as_model() — its ``source_queries`` is already expanded, - # so the base-name walk above turns up nothing. Fall back to the - # persisted record under ``model.name`` (if any) so query-backed - # drift attribution still names the real persisted base models. - if model.data_source: - try: - persisted = await self.storage.get_model( - model.name, data_source=model.data_source - ) - except Exception: - persisted = None - if persisted is not None and persisted.source_queries: - touched |= self._collect_query_backed_base_names(persisted) - await self._expand_join_graph( - touched=touched, data_source=model.data_source or None - ) - return touched - - async def _load_candidate_models( - self, *, data_source: str | None - ) -> list[SlayerModel]: - """Load the candidate model set for join-graph routing. Scoped to - one datasource when given; otherwise best-effort over every stored - model (the ``data_source is None`` path is rare — only models with - an empty ``data_source`` reach it).""" - if data_source: - return await _all_models_in_datasource(self.storage, data_source) - out: list[SlayerModel] = [] - for ds, name in await self.storage._list_all_model_identities(): - try: - m = await self.storage.get_model(name, data_source=ds) - except Exception: - m = None - if m is not None: - out.append(m) - return out - async def _expand_join_graph( - self, *, touched: "set[str]", data_source: str | None + self, *, touched: "set[str]", data_source: Optional[str] ) -> None: """Follow each touched model's joins transitively, adding reachable - target_model names to ``touched``. - - Directed reachability is delegated to :class:`JoinGraph` so the - engine has a single reachability implementation shared with - ``recommend_root_model`` (DEV-1626). Diamond / cyclic graphs are - visited-guarded by ``JoinGraph.reachable_from``. + target_model names to ``touched``. Visited-set guarded to avoid + infinite loops on diamond / cyclic join graphs. """ - graph = JoinGraph.build_from_models( - await self._load_candidate_models(data_source=data_source) - ) - expanded: set[str] = set() - for seed in touched: - expanded |= graph.reachable_from(seed) - touched |= expanded + frontier = list(touched) + visited: set[str] = set() + while frontier: + name = frontier.pop() + if name in visited: + continue + visited.add(name) + try: + m = await self.storage.get_model(name, data_source=data_source) + except Exception: + m = None + if m is None: + continue + for j in m.joins: + if j.target_model not in touched: + touched.add(j.target_model) + frontier.append(j.target_model) async def _maybe_raise_schema_drift( self, *, err: BaseException, model: SlayerModel, - enriched: "EnrichedQuery", + touched_models: "set[str]", ) -> None: """Attribute a query-time exception to schema drift via ``validate_models``. If drift is found in the touched models, raise ``SchemaDriftError`` (with ``err`` as ``__cause__``); otherwise return so the caller re-raises the original exception untouched. + ``touched_models`` is computed from the resolved bundle / plan; the + join graph is widened via ``_expand_join_graph`` before attribution. + (DEV-1485 Stage D removed the alternative ``enriched=`` mode, which + derived the same set from a legacy ``EnrichedQuery``.) + Any error from ``validate_models`` itself is swallowed so the original exception is never masked. """ from slayer.core.errors import SchemaDriftError try: - touched = await self._collect_models_touched(model=model, enriched=enriched) + touched = set(touched_models) + await self._expand_join_graph( + touched=touched, data_source=model.data_source or None + ) # Cross-model measure source models share the parent's DS in # validated queries (cross-DS joins are rejected at resolve # time), so attribution only needs the parent's data_source. data_sources: set[str] = {model.data_source} if model.data_source else set() - collected: list[Any] = [] + collected: List[Any] = [] for ds_name in data_sources or {None}: try: entries = await self.validate_models(data_source=ds_name) @@ -1930,7 +1725,7 @@ def _build_type_probe_query(self, model: SlayerModel) -> SlayerQuery: types) and falls back to the first allowed aggregation otherwise. Skips primary-key columns (they're identifiers, not values to probe). """ - measures: list[ModelMeasure] = [] + measures: List[ModelMeasure] = [] for c in model.columns: if c.hidden or c.primary_key: continue @@ -1944,11 +1739,11 @@ def _build_type_probe_query(self, model: SlayerModel) -> SlayerQuery: measures.append(ModelMeasure(formula=f"{c.name}:{agg}")) return SlayerQuery(source_model=model.name, measures=measures) - async def get_column_types( + async def get_column_types( # NOSONAR(S3776) — linear probe pipeline: query-backed prelude → bundle → expand-nested → plan → render → execute → result-key map-back. Splitting hides the order; each step is its own try/except + early-return so flatness is the easier read. self, model_name: str, - data_source: str | None = None, - ) -> dict[str, str]: + data_source: Optional[str] = None, + ) -> Dict[str, str]: """Infer column types for a model's columns via a type-probe query. Builds a real query through the engine's enrich+generate pipeline @@ -2001,6 +1796,13 @@ async def get_column_types( client = self._sql_clients[ds_key] probe_query = self._build_type_probe_query(model=model) + # If the model was expanded by the prelude above (query-backed + # case), it's a virtual sql-mode model — pass it INLINE so the + # bundle resolves against the expanded shape rather than re- + # consulting storage (which still holds the un-expanded source + # plus a potentially stale ``data_source``). + if not model.source_queries: + probe_query = probe_query.model_copy(update={"source_model": model}) try: # DEV-1625: a template source model's Mode-A {var} surfaces must be # rendered (from its own query_variables defaults) before probe SQL @@ -2008,94 +1810,113 @@ async def get_column_types( # via the surrounding except, so partially-defaulted models stay safe. # DEV-1727: pass the resolved datasource's dialect so probe-SQL # escaping matches the backend that parses it. - enriched = await self._enrich( - query=probe_query, - model=_render_probe_model( - model, dialect=dialect_for_ds_type(datasource.type) - ), + model = _render_probe_model( + model, dialect=dialect_for_ds_type(datasource.type) ) + if not model.source_queries: + probe_query = probe_query.model_copy(update={"source_model": model}) + bundle = await build_resolved_source_bundle( + query=probe_query, + storage=self.storage, + data_source=model.data_source or None, + runtime_variables={}, + named_queries={}, + ) + # Expand any nested query-backed models (join targets, etc.) + # so the planner / generator sees ``sql``-mode shapes. + bundle = await expand_query_backed_models_in_bundle( + bundle=bundle, + outer_vars=None, + runtime_kwarg=None, + dry_run_placeholders=True, + expander=self._expand_query_backed_model, + ) + planned = plan_stages(queries=[probe_query], bundle=bundle) + root = planned[-1] dialect = self._dialect_for_type(datasource.type) - generator = SQLGenerator(dialect=dialect) - # DEV-1444: type probing is a user-visible call site; pin - # ``outer`` mode explicitly so a future default-change cannot - # silently shift type-probe behaviour. - sql = generator.generate(enriched=enriched, render_mode="outer") + sql = generate_planned_stages(planned, bundle=bundle, dialect=dialect) # DEV-1578: type probing is a user-visible execution path, so it - # must honour the forced-filter policy too. A policy failure + # honours the forced-filter policy too — a policy failure # (block / fail-closed) degrades to {} via this try/except rather - # than leaking an unscoped probe. - # DEV-1627: preflight the ClickHouse version before the rewrite so - # the correlated-subquery guard can gate (no-op otherwise). + # than leaking an unscoped probe. DEV-1627: preflight the ClickHouse + # version before the rewrite so the correlated-subquery guard can + # gate (no-op otherwise, and no-op entirely when no policy is set). await self._preflight_clickhouse_correlated( dialect=dialect, datasource=datasource ) sql = self._apply_policy(sql=sql, dialect=dialect, datasource=datasource) except Exception: - logger.warning("get_column_types enrich/generate failed for model '%s'", model_name) + logger.warning( + "get_column_types plan/generate failed for model '%s'", + model_name, + ) return {} try: raw_types = await client.get_column_types(sql=sql) except Exception: - logger.warning("get_column_types probe failed for model '%s'", model_name) + logger.warning( + "get_column_types probe failed for model '%s'", model_name, + ) return {} - # Map qualified aliases (e.g., "orders.revenue_max") back to bare measure names - result: dict[str, str] = {} - for em in enriched.measures: - if em.alias in raw_types: - result[em.source_measure_name or em.name] = raw_types[em.alias] + # DEV-1716: on BigQuery / T-SQL the probe SQL is alias-mangled (it has to + # be, to execute), so the cursor returns mangled keys like + # ``orders___revenue_max``. Decode them back to the canonical dotted form + # the ``full`` lookups below use. Identity for every non-mangling dialect. + raw_types = get_dialect(dialect).decode_result_keys([raw_types])[0] + + # Map qualified aliases (e.g., "orders.revenue_max") back to bare + # measure names. Probe sources can be ColumnKey (.leaf) or + # ColumnSqlKey (.column_name) per DEV-1369 derived columns. + result: Dict[str, str] = {} + source_relation = root.source_relation + for slot in root.aggregate_slots: + if slot.hidden: + continue + src = getattr(slot.key, "source", None) + bare = ( + getattr(src, "leaf", None) + or getattr(src, "column_name", None) + ) + if bare is None: + continue + public = slot.public_name or slot.declared_name + full = f"{source_relation}.{public}" + if full in raw_types: + result[bare] = raw_types[full] return result - async def aclose(self) -> None: - """Dispose every cached client's async engine; keep the clients themselves. - - Per-instance async engines bind their asyncpg/aiomysql pool to the loop - that first opened a connection; closing that loop without disposing - leaks the server-side connections (asyncpg.Connection.close needs a - live loop). Clients are kept so ``_sync_engine`` survives — important - for ``:memory:`` SQLite, whose StaticPool pins the connection holding - all data. - """ - for client in self._sql_clients.values(): - await client.aclose() - def execute_sync( self, query: "SlayerQuery | dict | list[SlayerQuery | dict] | str", - variables: dict[str, Any] | None = None, + variables: Optional[Dict[str, Any]] = None, *, dry_run: bool = False, explain: bool = False, - data_source: str | None = None, + data_source: Optional[str] = None, cache: bool = False, ) -> SlayerResponse: """Synchronous wrapper for execute(). For CLI, notebooks, and scripts. - DEV-1587: forwards ``cache`` and ``data_source`` (the latter closes the - pre-existing sync ``data_source`` gap so sync cache / evict reach the - datasource-override paths). Disposes per-call async engines in - ``finally`` so they don't outlive their owning loop — see ``aclose``. + Forwards ``cache`` and ``data_source`` (the sync surface previously + lacked ``data_source``; DEV-1715 closes that gap). Disposes per-call + async engines in ``finally`` so they don't outlive their owning loop — + see ``aclose`` (DEV-1656). """ from slayer.async_utils import run_sync async def _run_and_cleanup() -> SlayerResponse: try: return await self.execute( - query, - variables=variables, - dry_run=dry_run, - explain=explain, - data_source=data_source, - cache=cache, + query, variables=variables, dry_run=dry_run, + explain=explain, data_source=data_source, cache=cache, ) finally: await self.aclose() return run_sync(_run_and_cleanup()) - # ------------------------------------------------------------------ - # recommend_root_model (DEV-1626) # ------------------------------------------------------------------ async def _scope_bare_name_to_datasource( self, *, raw: str, name: str, data_source: str @@ -2195,7 +2016,7 @@ async def _resolve_recommend_item( return item, warnings async def _recommend_resolve_items( - self, items: list[str], data_source: str | None + self, *, items: list[str], data_source: str | None ) -> "tuple[list[_ResolvedItem], list[str]]": """Resolve every input item, then enforce single-datasource + dedup. @@ -2263,7 +2084,9 @@ async def recommend_root_model( resolved *after* the datasource is determined from ``items`` / ``data_source``, so it cannot influence datasource selection. """ - resolved, base_warnings = await self._recommend_resolve_items(items, data_source) + resolved, base_warnings = await self._recommend_resolve_items( + items=items, data_source=data_source + ) ds = resolved[0].data_source models = await _all_models_in_datasource(self.storage, ds) graph = JoinGraph.build_from_models(models) @@ -2347,16 +2170,17 @@ async def _run() -> RootModelRecommendation: return run_sync(_run()) + async def edit_model_remove( self, *, model_name: str, - data_source: str | None, - remove_columns: list[str] | None = None, - remove_measures: list[str] | None = None, - remove_aggregations: list[str] | None = None, - remove_joins: list[str] | None = None, - remove_filters: list[str] | None = None, + data_source: Optional[str], + remove_columns: Optional[List[str]] = None, + remove_measures: Optional[List[str]] = None, + remove_aggregations: Optional[List[str]] = None, + remove_joins: Optional[List[str]] = None, + remove_filters: Optional[List[str]] = None, ) -> SlayerModel: """Apply surgical removals to a persisted model. @@ -2423,13 +2247,13 @@ async def edit_model_remove( return updated async def delete_model_by_name( - self, *, model_name: str, data_source: str | None + self, *, model_name: str, data_source: Optional[str] ) -> bool: """Delete a persisted model by name. Returns True if the model existed.""" return await self.storage.delete_model(model_name, data_source=data_source) async def apply_drift_deletes( - self, deletes: "list[Any]" + self, deletes: "List[Any]" ) -> "Any": """Apply each ``ToDeleteEntry`` via the engine helpers and return the combined ``ApplyDriftResult`` (applied, errors, residual). @@ -2446,8 +2270,8 @@ async def apply_drift_deletes( ApplyError, ) - applied: list[AppliedEntry] = [] - errors: list[ApplyError] = [] + applied: List[AppliedEntry] = [] + errors: List[ApplyError] = [] touched_ds: set[str] = set() for entry in deletes: @@ -2490,7 +2314,7 @@ async def apply_drift_deletes( ) # Re-validate the touched datasources to compute residual drift. - residual: list[Any] = [] + residual: List[Any] = [] for ds_name in touched_ds: try: residual.extend(await self.validate_models(data_source=ds_name)) @@ -2507,8 +2331,8 @@ async def apply_drift_deletes( ) async def validate_models( - self, data_source: str | None = None - ) -> "list[Any]": + self, data_source: Optional[str] = None + ) -> "List[Any]": """Diff persisted models against live database schemas. Returns the minimal list of deletes needed for SQL generation to @@ -2529,7 +2353,7 @@ async def validate_models( return [] identities = await self.storage._list_all_model_identities() ds_model_names = [n for d, n in identities if d == data_source] - models: list[SlayerModel] = [] + models: List[SlayerModel] = [] for name in ds_model_names: m = await self.storage.get_model(name, data_source=data_source) if m is not None: @@ -2544,13 +2368,13 @@ async def validate_models( if not ds_names: return [] - async def _validate_one(name: str) -> "list[ToDeleteEntry]": + async def _validate_one(name: str) -> "List[ToDeleteEntry]": return await self.validate_models(data_source=name) results = await _asyncio.gather( *(_validate_one(n) for n in ds_names), return_exceptions=True ) - out: list = [] + out: List = [] for r in results: if isinstance(r, BaseException): logger.warning("validate_models: per-DS validation failed: %s", r) @@ -2562,8 +2386,8 @@ def create_model_from_query_sync( self, query: "SlayerQuery | list[SlayerQuery] | dict | list[dict]", name: str, - description: str | None = None, - variables: dict[str, Any] | None = None, + description: Optional[str] = None, + variables: Optional[Dict[str, Any]] = None, save: bool = True, ) -> SlayerModel: """Synchronous wrapper for create_model_from_query().""" @@ -2579,135 +2403,207 @@ def create_model_from_query_sync( ) ) - async def _expand_query_backed_model( + async def _expand_query_backed_model( # NOSONAR S3776 — linear render pipeline (topo-sort → bundle → expand-nested → normalize → variables → plan → render → wrap); splitting hides the order of operations self, model: SlayerModel, - outer_vars: dict[str, Any] | None, - runtime_kwarg: dict[str, Any] | None, + outer_vars: Optional[Dict[str, Any]], + runtime_kwarg: Optional[Dict[str, Any]], dry_run_placeholders: bool, - _resolving: set | None, + _resolving: Optional[set], ) -> SlayerModel: - """If ``model`` is query-backed, expand its ``source_queries`` into a - virtual model (with rendered SQL). Otherwise return ``model`` unchanged. - - Read-only — never writes to storage. The persisted cache - (``columns`` / ``backing_query_sql`` / ``data_source``) is populated - only by ``engine.save_model`` / ``create_model_from_query(save=True)``. + """Expand a query-backed ``model`` into a virtual ``sql``-mode model + through the typed pipeline (DEV-1452 Stage B). + + Mirrors ``_execute_pipeline``'s mid-section (bundle → expand-nested + → normalize → variables → ``plan_stages`` → ``generate_planned_stages``) + and wraps the rendered backing SQL in a flat-rename SELECT so the + virtual model exposes downstream-bindable flat columns. Read-only — + never writes to storage; the persisted cache (``columns`` / + ``backing_query_sql`` / ``data_source``) is populated only by + ``engine.save_model`` / ``create_model_from_query(save=True)``. + + ``_resolving`` is preserved for caller signature parity but the + recursion guard moves out of the migrated path. Forward / self / + cycle references in stored ``source_queries`` are caught by + ``topologically_order_stages`` up front; nested query-backed + targets / stage sources are handled by + ``expand_query_backed_models_in_bundle`` (which re-enters this + method recursively via the ``expander`` callback). The two legacy + ContextVars (``_join_target_resolving_var`` / + ``_forbidden_sibling_refs_var``) are not set or read on this path. """ if not model.source_queries: return model - stages = list(model.source_queries) - merged_outer = {**model.query_variables, **(outer_vars or {})} + + # 1. Topo-sort + validate (root-as-sink, joins.target_model + # awareness, inline-nested ``SlayerModel.source_queries`` + # recursion per decision E). + stages = topologically_order_stages(list(model.source_queries)) + final_stage = stages[-1] named_q = {q.name: q for q in stages[:-1] if q.name} - return await self._query_as_model( - inner_query=stages[-1], + + # 2. Build resolved source bundle for the final stage. Stage B + # mirrors the legacy ``_query_as_model`` data_source behaviour: the + # bundle is built WITHOUT a DS hint, so the inner resolution falls + # back to the unique-match / priority-list resolver. This lets + # ``get_column_types`` recover from a stale persisted + # ``model.data_source`` (the cache populator may not have refreshed + # yet); join-target lookups still scope to whichever DS the + # resolved base model actually lives in. + bundle = await build_resolved_source_bundle( + query=final_stage, + storage=self.storage, + data_source=None, + runtime_variables=runtime_kwarg, + outer_variables={**model.query_variables, **(outer_vars or {})}, named_queries=named_q, - override_name=model.name, - _resolving=_resolving, - outer_vars=merged_outer, + ) + + # 3. Expand every query-backed model in the bundle and re-apply + # root inline_extensions. Shared with ``_execute_pipeline``. The + # ``_resolving`` set propagates the in-flight expansion names so + # a query-backed join target that transitively references its + # parent short-circuits via cached ``backing_query_sql`` rather + # than recursing forever. + # + # Codex: pass the bundle's MERGED variables (which already include + # ``{model.query_variables, outer_vars, stage, runtime}`` per + # precedence) rather than the bare stage-level ``final_stage. + # variables``. Otherwise nested expansions / sibling stages lose + # the outer model's ``query_variables`` layer and substitution + # diverges between execute and save-time dry-run. + bundle = await expand_query_backed_models_in_bundle( + bundle=bundle, + outer_vars=bundle.query_variables, runtime_kwarg=runtime_kwarg, dry_run_placeholders=dry_run_placeholders, + expander=self._expand_query_backed_model, + _resolving=(_resolving or set()) | {model.name}, ) - async def _resolve_query_model( # NOSONAR S3776 — type-dispatch on str/SlayerModel/ModelExtension/dict; flat is clearer than per-shape helpers here - self, - query_model, - named_queries: dict = None, - _resolving: set = None, - outer_vars: dict[str, Any] | None = None, - runtime_kwarg: dict[str, Any] | None = None, - dry_run_placeholders: bool = False, - prefer_data_source: str | None = None, - ) -> SlayerModel: - """Resolve query.source_model — handles str, SlayerModel, and ModelExtension.""" - from slayer.core.query import ModelExtension - - named_queries = named_queries or {} - - if isinstance(query_model, str): - return await self._resolve_model( - model_name=query_model, - named_queries=named_queries, - _resolving=_resolving, - outer_vars=outer_vars, - runtime_kwarg=runtime_kwarg, - dry_run_placeholders=dry_run_placeholders, - prefer_data_source=prefer_data_source, - ) - elif isinstance(query_model, SlayerModel): - # Inline SlayerModel may itself be query-backed; expand its - # source_queries the same way storage-backed models do, otherwise - # the outer enrichment can't see the virtual columns. - return await self._expand_query_backed_model( - model=query_model, - outer_vars=outer_vars, - runtime_kwarg=runtime_kwarg, - dry_run_placeholders=dry_run_placeholders, - _resolving=_resolving, - ) - elif isinstance(query_model, ModelExtension): - base = await self._resolve_model( - model_name=query_model.source_name, - named_queries=named_queries, - _resolving=_resolving, - outer_vars=outer_vars, - runtime_kwarg=runtime_kwarg, + # 4. Per-stage normalize + variable substitution. Mirrors + # ``_execute_pipeline:486-535``. + sibling_names = set(named_q) + final_stage, _slack = self._normalize_stage( + query=final_stage, bundle=bundle, sibling_names=sibling_names, + ) + normed_named: Dict[str, SlayerQuery] = {} + for nm, nq in named_q.items(): + nq2, _ = self._normalize_stage( + query=nq, bundle=bundle, sibling_names=sibling_names, + ) + normed_named[nm] = nq2 + final_stage = apply_variables_to_query( + query=final_stage, + variables=bundle.query_variables, + dry_run_placeholders=dry_run_placeholders, + ) + # Codex: ``final_stage.variables`` is the user-supplied stage-level + # dict; ``apply_variables_to_query`` substitutes into filters but + # does not promote merged layers onto ``.variables``. Sibling + # substitution therefore needs ``bundle.query_variables`` (the + # merged ``{runtime > final_stage.variables > model.query_variables + # > outer_vars > source_model_defaults}`` set) — using the bare + # stage dict drops ``model.query_variables`` and dry-run save + # fills sibling filters' ``{var}`` placeholders with ``0``. + normed_named = { + nm: apply_variables_to_query( + query=nq, + variables={ + **( + ( + bundle.stage_source_models[nm].query_variables + if nm in bundle.stage_source_models + else ( + bundle.source_model.query_variables + if bundle.source_model else None + ) + ) + or {} + ), + **(bundle.query_variables or {}), + **(nq.variables or {}), + **(runtime_kwarg or {}), + }, dry_run_placeholders=dry_run_placeholders, - prefer_data_source=prefer_data_source, ) - # Extend the base model with extra columns/measures/joins - # ModelJoin already imported at the top of the file. + for nm, nq in normed_named.items() + } - extra_cols = [ - Column.model_validate(c) if isinstance(c, dict) else c for c in (query_model.columns or []) - ] - extra_measures = [ - ModelMeasure.model_validate(m) if isinstance(m, dict) else m for m in (query_model.measures or []) - ] - extra_joins = [ModelJoin.model_validate(j) if isinstance(j, dict) else j for j in (query_model.joins or [])] - return base.model_copy( - update={ - "columns": list(base.columns) + extra_cols, - "measures": list(base.measures) + extra_measures, - "joins": list(base.joins) + extra_joins, - } + # 5. Plan + render the DAG. + plan_input = [*normed_named.values(), final_stage] + planned_list = plan_stages(queries=plan_input, bundle=bundle) + root_planned = planned_list[-1] + inner_source_model = bundle.source_model + assert inner_source_model is not None + datasource = await self._resolve_datasource(model=inner_source_model) + dialect = self._dialect_for_type(datasource.type) + rendered = generate_planned_stages( + planned_list, bundle=bundle, dialect=dialect, + ) + + # 6. Wrap with flat-renamed SELECT. Public StageColumn entries + # only — hoisted hidden slots are planner-synthesized + # intermediates the user never declared and must not surface as + # virtual-model columns (P4 closure / decision #3). + public_cols = [ + c for c in ( + root_planned.stage_schema.columns + if root_planned.stage_schema is not None else [] ) - elif isinstance(query_model, dict): - # Dict — could be ModelExtension or SlayerModel - if "source_name" in query_model: - ext = ModelExtension.model_validate(query_model) - return await self._resolve_query_model( - ext, - named_queries, - _resolving=_resolving, - outer_vars=outer_vars, - runtime_kwarg=runtime_kwarg, - dry_run_placeholders=dry_run_placeholders, - ) - else: - model = SlayerModel.model_validate(query_model) - return await self._expand_query_backed_model( - model=model, - outer_vars=outer_vars, - runtime_kwarg=runtime_kwarg, - dry_run_placeholders=dry_run_placeholders, - _resolving=_resolving, - ) - else: - raise ValueError(f"Invalid query.source_model type: {type(query_model)}") + if c.public_alias is not None + ] + expected = [c.name for c in public_cols] + wrapped_ast = build_flat_rename_wrapper( + source_relation=root_planned.source_relation, + stage_sql=rendered, + expected_columns=expected, + dialect=dialect, + ) + wrapped_sql = wrapped_ast.sql(dialect=dialect, pretty=True) + + # 7. Build virtual model from public StageColumn entries. + # Slot types drive Column.type (decision #2) so ``*:count`` → + # ``INT``, declared ``ModelMeasure.type`` is honored, and source + # column types propagate through ``sum`` / ``min`` / ``max``. + cols = [ + Column( + name=sc.name, + sql=sc.name, + type=sc.type or DataType.DOUBLE, + label=sc.label, + description=sc.description, + format=sc.format, + ) + for sc in public_cols + ] + return SlayerModel( + name=model.name, + sql=wrapped_sql, + data_source=inner_source_model.data_source, + columns=cols, + default_time_dimension=inner_source_model.default_time_dimension, + # source_model_origin intentionally NOT set (decision D): + # the typed pipeline answers DEV-1449 through the flat + # ``StageSchema`` namespace, not via the legacy lineage walk. + ) async def _resolve_model( self, model_name: str, - named_queries: dict[str, SlayerQuery] = None, _resolving: set = None, - outer_vars: dict[str, Any] | None = None, - runtime_kwarg: dict[str, Any] | None = None, + outer_vars: Optional[Dict[str, Any]] = None, + runtime_kwarg: Optional[Dict[str, Any]] = None, dry_run_placeholders: bool = False, - prefer_data_source: str | None = None, + prefer_data_source: Optional[str] = None, ) -> SlayerModel: - """Resolve a model by name — checks named queries first, then storage.""" - named_queries = named_queries or {} + """Resolve a model by name from storage, expanding a query-backed one. + + Sibling stages are resolved by the typed pipeline in ``source_bundle`` + (via ``_follow_sibling_chain``), never here — DEV-1485 Stage D deleted + the named-query branch along with the rest of the legacy stack, and + with it the ``named_queries`` parameter this used to thread through. + """ _resolving = _resolving if _resolving is not None else set() # Circular reference protection (per-call set, safe for concurrent requests) @@ -2720,7 +2616,6 @@ async def _resolve_model( try: return await self._resolve_model_inner( model_name, - named_queries, _resolving=_resolving, outer_vars=outer_vars, runtime_kwarg=runtime_kwarg, @@ -2733,23 +2628,12 @@ async def _resolve_model( async def _resolve_model_inner( self, model_name: str, - named_queries: dict[str, SlayerQuery], _resolving: set = None, - outer_vars: dict[str, Any] | None = None, - runtime_kwarg: dict[str, Any] | None = None, + outer_vars: Optional[Dict[str, Any]] = None, + runtime_kwarg: Optional[Dict[str, Any]] = None, dry_run_placeholders: bool = False, - prefer_data_source: str | None = None, + prefer_data_source: Optional[str] = None, ) -> SlayerModel: - # Named query overrides stored model - if model_name in named_queries: - return await self._query_as_model( - inner_query=named_queries[model_name], - named_queries=named_queries, - _resolving=_resolving, - outer_vars=outer_vars, - runtime_kwarg=runtime_kwarg, - dry_run_placeholders=dry_run_placeholders, - ) # v4 (DEV-1330): bare-name lookups consult the priority list (via # storage.get_model's None branch) and the ``prefer_data_source`` @@ -2768,22 +2652,6 @@ async def _resolve_model_inner( f"Model '{model_name}' not found in data_source " f"'{prefer_data_source}'." ) - forbidden = _forbidden_sibling_refs_var.get() - if forbidden and model_name in forbidden: - offender = forbidden[model_name] - if offender == model_name: - raise ValueError( - f"Stage '{offender}' cannot reference itself via " - f"'joins.target_model' (or as 'source_model'); a " - f"stage may only resolve to prior named stages in " - f"the same source_queries list." - ) - raise ValueError( - f"Stage '{offender}' cannot reference stage " - f"'{model_name}': forward references are not allowed. " - f"A stage may only resolve to prior named stages in " - f"the same source_queries list." - ) raise ValueError(f"Model '{model_name}' not found") # If model has source_queries, re-enrich from stored queries. @@ -2801,8 +2669,8 @@ async def create_model_from_query( self, query: "SlayerQuery | list[SlayerQuery] | dict | list[dict]", name: str, - description: str | None = None, - variables: dict[str, Any] | None = None, + description: Optional[str] = None, + variables: Optional[Dict[str, Any]] = None, save: bool = True, ) -> SlayerModel: """Create a query-backed model from a query (or list of stages). @@ -2840,18 +2708,126 @@ async def create_model_from_query( # can use the returned model directly. return await self._validate_and_populate_cache(model) + async def _reachable_aggs_for_save( + self, model: SlayerModel, + ) -> Optional[frozenset[str]]: + """Best-effort storage BFS over the join graph collecting custom + aggregation names (DEV-1500). Returns ``None`` when ``model`` has + nothing to normalise (no top-level measures AND no source_queries) + or when the walk yields nothing; absent join targets and storage + errors (incl. ``AmbiguousModelError``) are swallowed so a save + never aborts on a flaky join-target lookup. + + For query-backed models the walk unions the reachable custom-agg + names across every stage's ``source_model`` so the save-time + FUNC_STYLE_AGG rewrite on each stage's measures sees the full + joined-model agg surface (CR PR #153 thread r3330620881). + """ + if not model.measures and not model.source_queries: + return None + + resolver = self._make_join_target_resolver(model.data_source) + collected: set[str] = set(await self._walk_reachable_aggs(model, resolver)) + for stage in model.source_queries or []: + stage_model = await self._resolve_stage_source_model( + stage, data_source=model.data_source, + ) + if stage_model is not None: + collected.update( + await self._walk_reachable_aggs(stage_model, resolver) + ) + return frozenset(collected) if collected else None + + def _make_join_target_resolver( + self, data_source: Optional[str], + ): + """Build the best-effort `resolve_join_target` closure that + ``collect_reachable_agg_names`` consumes — swallows absent targets + and storage errors (incl. ``AmbiguousModelError``) so a save never + aborts on a flaky join-target lookup.""" + storage = self.storage + + async def _resolver( + target_model_name: str, named_queries, # noqa: ARG001 + ): + if not storage: + return None + try: + if data_source: + target = await storage.get_model( + target_model_name, data_source=data_source, + ) + else: + target = await storage.get_model(target_model_name) + except Exception: # noqa: BLE001 — best-effort; AmbiguousModelError + misc + return None + return (None, target) if target is not None else None + + return _resolver + + @staticmethod + async def _walk_reachable_aggs( + walk_model: SlayerModel, resolver, + ) -> frozenset[str]: + """One swallow-all BFS over ``walk_model``'s join graph collecting + the reachable custom-aggregation names.""" + try: + got = await collect_reachable_agg_names( + source_model=walk_model, + resolve_join_target=resolver, + named_queries={}, + ) + except Exception: # noqa: BLE001 — never let the walk abort a save + return frozenset() + return got or frozenset() + + async def _resolve_stage_source_model( + self, stage: SlayerQuery, *, data_source: Optional[str], + ) -> Optional[SlayerModel]: + """For a query-backed model's stage, resolve ``stage.source_model`` + to a concrete ``SlayerModel`` (inline → use as-is; string → look up + in storage; ModelExtension / dict → skip best-effort and fall back + to execute-time normalization).""" + src = stage.source_model + if isinstance(src, SlayerModel): + return src + if not isinstance(src, str): + return None + try: + if data_source: + return await self.storage.get_model(src, data_source=data_source) + return await self.storage.get_model(src) + except Exception: # noqa: BLE001 — best-effort + return None + async def save_model(self, model: SlayerModel) -> SlayerModel: """Persist a SlayerModel through the engine. For query-backed models, rejects user-supplied cache fields and runs save-time dry-run validation before populating the cache. For non- query-backed models, persists as-is. + + DEV-1450 stage 6 — runs the slack-normalization layer over the + incoming model so persisted formulas land in canonical form. The + (Before DEV-1485 the legacy in-tree rewriters also fired during + enrichment for callers that loaded and re-executed persisted models; + normalization at save time is now the only such rewrite.) + + DEV-1500 — the FUNC_STYLE_AGG rewrite recognises custom aggregations + defined on joined models via ``_reachable_aggs_for_save`` (best-effort + storage walk; swallowed on missing target / AmbiguousModelError / + storage hiccup). """ + custom_aggs = await self._reachable_aggs_for_save(model) + norm_model = normalize_model(model, custom_agg_names=custom_aggs) + if norm_model.model is not None: + model = norm_model.model + model = _normalize_source_query_stages(model, custom_aggs=custom_aggs) # Capture the *previous* data_source for this name so we can clean # up the old storage entry when a query-backed model's resolved # data_source changes (e.g. its backing query now points at a # different upstream datasource). - prior_data_source: str | None = None + prior_data_source: Optional[str] = None if model.source_queries: try: identity = await self.storage.resolve_model_identity(model.name) @@ -2891,18 +2867,20 @@ async def _validate_and_populate_cache(self, model: SlayerModel) -> SlayerModel: """Run save-time dry-run validation on a query-backed model and return a copy with ``columns``, ``backing_query_sql``, and ``data_source`` populated from the virtual model. + + DEV-1452 Stage B — pure delegate to the migrated + ``_expand_query_backed_model`` with ``dry_run_placeholders=True`` + so any required-but-undefaulted ``{var}`` placeholder is filled + with the legacy ``"0"`` sentinel rather than raising at SQL-gen. """ - stages = list(model.source_queries or []) - if not stages: + if not (model.source_queries or []): return model - virtual = await self._query_as_model( - inner_query=stages[-1], - named_queries={q.name: q for q in stages[:-1] if q.name}, - override_name=model.name, - _resolving=set(), + virtual = await self._expand_query_backed_model( + model=model, outer_vars=dict(model.query_variables), runtime_kwarg={}, dry_run_placeholders=True, + _resolving=set(), ) return model.model_copy(update={ "columns": list(virtual.columns), @@ -2915,928 +2893,6 @@ async def _validate_and_populate_cache(self, model: SlayerModel) -> SlayerModel: "data_source": virtual.data_source, }) - async def _enrich( # NOSONAR S3776 — orchestrates resolve-callback closures + cross-model post-processing; splitting into helpers obscures the closure variables threaded through enrich_query - self, - query: SlayerQuery, - model: SlayerModel, - named_queries: dict[str, SlayerQuery] = None, - dialect: str | None = None, - *, - drop_unreachable_filters: bool = False, - ) -> EnrichedQuery: - """Resolve a SlayerQuery against model definitions into an EnrichedQuery. - - Delegates to enrich_query() in enrichment.py, passing engine callbacks - for model resolution (joins, cross-model measures, join targets). - - ``dialect`` controls how Column.sql is parsed during derived-reference - expansion. Falls back to the model's resolved datasource type, then to - ``"postgres"`` if neither is available (e.g., in unit tests with a - fake data_source name). - """ - - if dialect is None: - dialect = "postgres" - try: - if model.data_source and self.storage: - ds = await self.storage.get_datasource(model.data_source) - if ds is not None: - dialect = self._dialect_for_type(ds.type) - except Exception: # noqa: BLE001 — diagnostics only; never block enrichment - pass - - async def _resolve_join_target(target_model_name, named_queries): - nq = named_queries or {} - if target_model_name in nq: - # Named-query stages inherit the variable context of the query - # being enriched (its filter substitutions) so nested query- - # backed model resolution works through joins as well. - target = await self._query_as_model( - inner_query=nq[target_model_name], - named_queries=nq, - outer_vars=query.variables, - ) - elif self.storage: - # v4 (DEV-1330): joins must stay inside the parent model's - # logical database (cross-datasource joins aren't executable). - # When parent has a ``data_source``, do a *strict* lookup — - # no bare-name fallback that could silently pick the same - # name from another datasource. Only fall through to the - # priority/unique-match resolver when the parent has no - # datasource hint to give. - if model.data_source: - target = await self.storage.get_model( - target_model_name, data_source=model.data_source - ) - else: - target = await self.storage.get_model(target_model_name) - if target is None: - # When the lookup misses, distinguish a forward / self - # reference (sibling stage that's not in this stage's - # scope) from a genuinely-missing storage model so the - # caller gets a clear error instead of a generic "not - # found" — same logic as ``_resolve_model_inner`` - # (DEV-1340). - forbidden = _forbidden_sibling_refs_var.get() - if forbidden and target_model_name in forbidden: - offender = forbidden[target_model_name] - if offender == target_model_name: - raise ValueError( - f"Stage '{offender}' cannot reference itself " - f"via 'joins.target_model'; a stage may only " - f"resolve to prior named stages in the same " - f"source_queries list." - ) - raise ValueError( - f"Stage '{offender}' cannot reference stage " - f"'{target_model_name}': forward references are " - f"not allowed. A stage may only resolve to prior " - f"named stages in the same source_queries list." - ) - if target and target.source_queries: - target = await self._render_query_backed_join_target( - target=target, - outer_query_variables=query.variables, - ) - else: - target = None - if target and target.sql_table: - return target.sql_table, target - elif target and target.sql: - return f"({target.sql})", target - return None - - async def _resolve_model_for_expansion(model_name, named_queries): - """Adapter for column_expansion: returns ``SlayerModel`` or None. - Catches lookup errors so unknown alias paths don't blow up the - whole enrichment — the expander treats them as opaque. - - v4: pass the *outer* model's ``data_source`` as the hint so - ``B.col`` references inside ``A``'s derived columns resolve - within ``A.data_source``, never across the join graph into a - sibling datasource. - """ - try: - return await self._resolve_model( - model_name=model_name, - named_queries=named_queries or {}, - prefer_data_source=model.data_source or None, - ) - except Exception: # noqa: BLE001 — opaque alias is expected for CTE/sub-query refs - return None - - enriched = await enrich_query( - query=query, - model=model, - named_queries=named_queries, - resolve_dimension_via_joins=self._resolve_dimension_with_terminal, - resolve_cross_model_measure=self._resolve_cross_model_measure, - resolve_join_target=_resolve_join_target, - resolve_model=_resolve_model_for_expansion, - dialect=dialect, - drop_unreachable_filters=drop_unreachable_filters, - ) - - # Post-process: build re-rooted enriched queries for cross-model measures - for cm in enriched.cross_model_measures: - cm.rerooted_enriched = await self._build_rerooted_enriched( - cm=cm, query=query, model=model, - named_queries=named_queries or {}, - ) - - return enriched - - async def _render_query_backed_join_target( - self, - target: SlayerModel, - outer_query_variables: dict[str, Any] | None, - ) -> SlayerModel: - """Resolve a query-backed model used as a JOIN target. - - Threads the enclosing query's variables into the target's stage filter - substitution so a target with ``filters=["amount > {threshold}"]`` sees - the runtime value, not the cached/default fill. - - Recursion guard: ``self._join_target_resolving`` blocks re-entry on the - same target name. The call stack crosses ``_enrich`` invocations - (target's source_queries → target's own joins → _resolve_join_target - again), so this guard lives on the engine instance, not on a closure. - Re-entry returns the cached SQL if available, else returns the raw - target unchanged so enrichment fails with a clear "no sql" error - instead of looping. - """ - resolving = self._get_join_target_resolving() - if target.name in resolving: - if target.backing_query_sql: - return target.model_copy(update={"sql": target.backing_query_sql}) - return target - # When the enclosing query has no variables AND a canonical cache - # exists, prefer the cached SQL (avoids the second render). - if not outer_query_variables and target.backing_query_sql: - return target.model_copy(update={"sql": target.backing_query_sql}) - # Otherwise render fresh with merged variables (target defaults + - # enclosing query's vars; enclosing wins). - stages = list(target.source_queries or []) - if not stages: - return target - merged = {**dict(target.query_variables), **(outer_query_variables or {})} - resolving.add(target.name) - try: - return await self._query_as_model( - inner_query=stages[-1], - named_queries={q.name: q for q in stages[:-1] if q.name}, - override_name=target.name, - outer_vars=merged, - runtime_kwarg=outer_query_variables or None, - ) - finally: - resolving.discard(target.name) - - async def _query_as_model( # NOSONAR S3776 — variable-precedence + enrich + SQL-gen + virtual-model assembly is a single conceptual unit - self, - inner_query: SlayerQuery, - named_queries: dict[str, SlayerQuery] = None, - override_name: str = None, - _resolving: set = None, - outer_vars: dict[str, Any] | None = None, - runtime_kwarg: dict[str, Any] | None = None, - dry_run_placeholders: bool = False, - ) -> SlayerModel: - """Build a virtual SlayerModel from a nested query's result. - - Enriches and generates SQL for the inner query, then creates a model - whose `sql` is the inner query's SQL and whose dimensions/measures - are derived from the inner query's enriched columns. - - ``outer_vars``, ``runtime_kwarg``, and ``dry_run_placeholders`` thread - the variable-precedence machinery through nested query-backed model - resolution; see ``_merge_query_variables`` and - ``_apply_placeholder_fill``. - """ - named_queries = named_queries or {} - - # Compute effective variables for this stage and stamp them onto a - # copy of the inner query so substitution at enrichment time uses - # the merged set. - effective = _merge_query_variables( - outer=outer_vars, - stage=inner_query.variables, - runtime=runtime_kwarg, - ) - if dry_run_placeholders: - effective = _apply_placeholder_fill(inner_query, effective) - if effective != (inner_query.variables or {}): - inner_query = inner_query.model_copy(update={"variables": effective}) - - # Scope ``named_queries`` to the prior siblings of this stage. A - # non-final stage may only resolve names that come BEFORE it in the - # source_queries list; forward references and self references fall - # out of scope here and surface a clear error from - # ``_resolve_model_inner``. (For top-level stages — final stage, - # un-named query-backed wrapper, or stored-model lookup — the scope - # is unchanged.) - scoped = self._scope_named_queries_to_prior( - named_queries, inner_query.name - ) - forbidden_now: dict[str, str] = {} - if scoped is not named_queries and inner_query.name: - for k in named_queries: - if k not in scoped: - forbidden_now[k] = inner_query.name - - # Stack the new forbidden refs on top of any from an enclosing - # stage; restore on the way out so concurrent / sibling resolutions - # don't see this frame's bans. - prev_forbidden = _forbidden_sibling_refs_var.get() - if forbidden_now: - merged_forbidden = dict(prev_forbidden) if prev_forbidden else {} - # Outer frames win on the same key (a closer ancestor's ban is - # the more specific one), but in practice keys don't overlap - # because each frame names a distinct stage. - for k, v in forbidden_now.items(): - merged_forbidden.setdefault(k, v) - token = _forbidden_sibling_refs_var.set(merged_forbidden) - else: - token = None - - try: - # Resolve the inner model (handles str, SlayerModel, ModelExtension). - # Pass ``effective`` as the next layer's outer_vars so nested - # query-backed models inherit this stage's resolved context. - inner_model = await self._resolve_query_model( - query_model=inner_query.source_model, - named_queries=scoped, - _resolving=_resolving, - outer_vars=effective, - runtime_kwarg=runtime_kwarg, - dry_run_placeholders=dry_run_placeholders, - ) - - # Enrich the inner query — pass scoped named_queries so any - # ``joins.target_model`` referencing a prior named sibling is - # resolvable here too (DEV-1340). - enriched = await self._enrich( - query=inner_query, model=inner_model, named_queries=scoped - ) - finally: - if token is not None: - _forbidden_sibling_refs_var.reset(token) - - # Generate SQL - datasource = await self._resolve_datasource(model=inner_model) - dialect = self._dialect_for_type(datasource.type) - generator = SQLGenerator(dialect=dialect) - # DEV-1444: _query_as_model wraps the inner query as a virtual model; - # downstream references reach EVERY hoisted alias, so the inner SQL - # must keep its full projection rather than getting trimmed. - inner_sql = generator.generate(enriched=enriched, render_mode="wrapped") - - # Build virtual model from enriched columns. - # Inner query columns have aliases like "orders.count" (with dots). - # We wrap the inner SQL in a renaming subquery so the virtual model - # has clean column names that work naturally in JOINs and references. - virtual_name = override_name or inner_query.name or f"_subquery_{inner_model.name}" - - # Build lookups for labels/descriptions from the source model. - # In v2 there is no dim/measure split — every column carries both - # potential roles, so a single map per attribute is sufficient. - source_label = {c.name: c.label for c in inner_model.columns if c.label} - source_desc = {c.name: c.description for c in inner_model.columns if c.description} - - # Collect all inner aliases and their short names. - # Short names must be valid SQL identifiers (no dots). We derive them - # from the alias by stripping the source model prefix and replacing - # dots with underscores. - def _alias_to_short(alias: str) -> str: - """Convert result alias to a flat column name for the virtual model. - - The query result is a self-contained table without the joins the - source model may have had, so dot syntax (join paths) is not - applicable. We use ``__`` to preserve the path information: - - 'orders.customers.regions.name' → 'customers__regions__name' - 'orders.count' → 'count' - """ - # Strip source model prefix - stripped = alias.split(".", 1)[-1] if "." in alias else alias - # Replace remaining dots with __ to encode the original join path - return stripped.replace(".", "__") - - # (inner_alias, short_name, data_type, label, description, format) - column_map = [] - for d in enriched.dimensions: - short = _alias_to_short(d.alias) - label = d.label or source_label.get(d.name) - desc = source_desc.get(d.name) - column_map.append((d.alias, short, d.type, label, desc, d.format)) - for td in enriched.time_dimensions: - short = _alias_to_short(td.alias) - label = td.label or source_label.get(td.name) - desc = source_desc.get(td.name) - column_map.append((td.alias, short, DataType.TIMESTAMP, label, desc, None)) - for m in enriched.measures: - src_name = m.source_measure_name or m.name - label = m.label or source_label.get(src_name) - desc = source_desc.get(src_name) - fmt = _infer_aggregated_format( - model=inner_model, - measure_name=src_name, - aggregation=m.aggregation, - ) - column_map.append((m.alias, m.name, DataType.DOUBLE, label, desc, fmt)) - for t in enriched.transforms: - column_map.append( - (t.alias, t.name, DataType.DOUBLE, t.label, None, NumberFormat(type=NumberFormatType.FLOAT)) - ) - for e in enriched.expressions: - column_map.append( - (e.alias, e.name, DataType.DOUBLE, e.label, None, NumberFormat(type=NumberFormatType.FLOAT)) - ) - for cm in enriched.cross_model_measures: - # DEV-1448: when the user supplied an explicit ``name``, cm.name is - # a bare identifier (ModelMeasure.name forbids dots). Use it - # directly as the downstream short form so callers reference the - # user's chosen name without learning the ``__``-flattened - # encoding. Auto-derived names always contain a dot (e.g. - # ``customers.revenue_sum``) so they fall through to the legacy - # ``_alias_to_short`` flatten path. - # - # Codex review round 3 on PR #136: gate the short-circuit on - # ``cm.user_declared`` — hidden cross-model measures auto- - # extracted from arithmetic / transform formulas (in enrichment.py - # ``_ensure_measure_from_spec`` / ``_flatten_spec``) have bare - # internal placeholder names (e.g. ``__agg0__``) that must NOT - # leak into the virtual model's column set. Only user-declared - # renames qualify for the bare-name short. - if cm.user_declared and cm.name and "." not in cm.name: - short = cm.name - else: - short = _alias_to_short(cm.alias) - column_map.append((cm.alias, short, DataType.DOUBLE, cm.label, None, cm.format)) - - # Wrap inner SQL: SELECT AS , ... FROM (inner) AS _inner - # DEV-1571 Bug 3 follow-up: identifier quoting must match the - # dialect ``inner_sql`` was generated for. On MySQL the inner CTEs - # use backticks; on T-SQL the inner CTEs use brackets AND have - # their dotted aliases mangled. Hardcoded ANSI double quotes - # would either fail to parse (MySQL) or reference an alias the - # mangled inner subquery doesn't expose (T-SQL). - # DEV-1686: the inner ``alias`` is always dialect-quoted; the ``short`` - # output alias must also be quoted when it is a reserved word (a - # user-declared cross-model rename like ``order``, or an - # ``_alias_to_short`` that yields one), else ``AS order`` is bare and - # the wrapped SQL fails to parse/execute. - def _short_sql(short: str) -> str: - if short.lower() in SLAYER_RESERVED_KEYWORDS: - return exp.Identifier(this=short, quoted=True).sql(dialect=dialect) - return short - - rename_parts = [ - f'{exp.Identifier(this=alias, quoted=True).sql(dialect=dialect)} AS {_short_sql(short)}' - for alias, short, _, _, _, _ in column_map - ] - wrapped_sql = f"SELECT {', '.join(rename_parts)} FROM ({inner_sql}) AS _inner" - # DEV-1571 Bug 2: apply the dialect's emitted-SQL rewrite (e.g. - # T-SQL bracket-mangling) so the rename clause's inner-alias - # references match what the inner subquery actually projects. - wrapped_sql = get_dialect(dialect).rewrite_emitted_sql(wrapped_sql) - - # One Column per result column — each is potentially both a dimension - # (group-by) or measure (with colon-aggregation) at query time. - cols: list[Column] = [] - for _, short, dtype, label, desc, fmt in column_map: - cols.append(Column(name=short, sql=short, type=dtype, label=label, description=desc, format=fmt)) - - # DEV-1449 / Codex round 10: record only columns that are - # reliably the same cross-model aggregate the outer-stage - # intercept would resolve a `customers.revenue:sum` reference - # to. Includes: - # * Auto-derived cross-model canonical-flats (`_alias_to_short(cm.alias)`). - # * Intercept-produced EnrichedMeasures (from a downstream - # stage re-using the intercepted projection). - # Excludes: - # * User-renamed CMM shorts: a user-supplied `name` could - # coincidentally match a different aggregate's canonical-flat. - # * Plain measures / transforms / expressions: their names are - # user-supplied and could collide with cross-model canonicals - # by coincidence. - agg_shorts = set() - for cm in enriched.cross_model_measures: - if not (cm.user_declared and cm.name and "." not in cm.name): - agg_shorts.add(_alias_to_short(cm.alias)) - for m in enriched.measures: - if m.from_cross_model_intercept: - agg_shorts.add(m.name) - - # DEV-1449: record the lineage breadcrumb so outer-stage dotted-ref - # lookup can strip the right ancestor prefix and find the flat - # column in this wrapped projection. ``parent`` carries any - # existing chain on ``inner_model``, so chained nested-DAGs - # build a linked list down to the original table-backed root. - return SlayerModel( - name=virtual_name, - sql=wrapped_sql, - data_source=inner_model.data_source, - columns=cols, - default_time_dimension=inner_model.default_time_dimension, - source_model_origin=SourceModelOrigin( - name=inner_model.name, - data_source=inner_model.data_source, - parent=inner_model.source_model_origin, - agg_column_names=frozenset(agg_shorts), - ), - ) - - async def _resolve_dimension_via_joins( - self, - model: SlayerModel, - parts: list[str], - named_queries: dict = None, - ) -> "Column | None": - """Walk the join graph to resolve a multi-hop column reference. - - For "customers.regions.name", walks: model → customers → regions, - then looks up "name" on the regions model. - """ - result = await self._resolve_dimension_with_terminal( - model=model, parts=parts, named_queries=named_queries, - ) - return result[0] if result is not None else None - - async def _resolve_dimension_with_terminal( - self, - model: SlayerModel, - parts: list[str], - named_queries: dict = None, - ) -> "tuple[Column, SlayerModel] | None": - """Like ``_resolve_dimension_via_joins`` but also returns the - terminal model so callers (column-SQL expansion) can recurse into - the resolved column's own ``sql``. - """ - try: - terminal_model, _first_join = await self._walk_join_chain( - source_model=model, - hop_names=parts[:-1], - named_queries=named_queries, - strict_missing_join=False, - ) - except _NoJoinError: - return None - - col = terminal_model.get_column(parts[-1]) - if col is None: - return None - return col, terminal_model - - async def _walk_join_chain( - self, - *, - source_model: SlayerModel, - hop_names: list[str], - named_queries: dict = None, - strict_missing_join: bool = True, - ) -> "tuple[SlayerModel, ModelJoin | None]": - """Walk the join graph from ``source_model`` through ``hop_names``, - returning ``(terminal_model, first_join)``. Single source of - truth for both dimension and cross-model-measure resolution - (DEV-1369 — consolidates two prior near-duplicate walkers). - - Cycle detection: a hop name that already appears on the visited - stack (including ``source_model.name``) raises ``ValueError`` with - the offending path. - - Missing-join behaviour: - - * ``strict_missing_join=True`` (cross-model-measure callers) — - raise ``ValueError`` listing the available joins. - * ``strict_missing_join=False`` (dimension callers) — raise the - internal :class:`_NoJoinError` sentinel so the caller can map - to a ``None`` return. - """ - current_model = source_model - visited = {source_model.name} - first_join: "ModelJoin | None" = None - for i, hop_name in enumerate(hop_names): - if hop_name in visited: - raise ValueError( - f"Circular join detected while resolving " - f"'{'.'.join(hop_names)}': '{hop_name}' already visited " - f"({' → '.join(visited)} → {hop_name})" - ) - join = next( - (j for j in current_model.joins if j.target_model == hop_name), - None, - ) - if join is None: - if strict_missing_join: - raise ValueError( - f"Model '{current_model.name}' has no join to " - f"'{hop_name}'. Available joins: " - f"{[j.target_model for j in current_model.joins]}" - ) - raise _NoJoinError(hop_name) - if i == 0: - first_join = join - current_model = await self._resolve_model( - model_name=hop_name, - named_queries=named_queries or {}, - prefer_data_source=current_model.data_source or None, - ) - visited.add(hop_name) - return current_model, first_join - - async def _auto_move_fields_to_dimensions( - self, - query: SlayerQuery, - model: SlayerModel, - named_queries: dict, - ) -> SlayerQuery: - """Move bare (no-colon) measure-formula entries to dimensions when they - name a column that isn't a (named) ModelMeasure formula. - - LLMs frequently place column names in ``measures`` instead of - ``dimensions``. When an entry has no colon (no aggregation) and - resolves as a column but NOT as a model-level ModelMeasure formula, - silently move it to ``dimensions`` with a warning. - """ - if not query.measures: - return query - - kept: list = [] - extra_dims = list(query.dimensions or []) - moved = False - - for f in query.measures: - formula = f.formula.strip() - # Only consider bare names (no colon, no operators, no parens) - if ":" not in formula and not any(c in formula for c in "+-*/()"): - if "." not in formula: - # Local reference - is_col = model.get_column(formula) is not None - is_named_measure = model.get_measure(formula) is not None - if is_col and not is_named_measure: - logger.warning( - "Auto-moved '%s' from measures to dimensions (not a named measure formula)", - formula, - ) - extra_dims.append(ColumnRef(name=formula)) - moved = True - continue - else: - # Cross-model reference — walk the full join path - parts = formula.split(".") - try: - col_def = await self._resolve_dimension_via_joins( - model=model, parts=parts, named_queries=named_queries, - ) - except ValueError: - col_def = None # Circular join — leave in measures - if col_def is not None: - # parts[-2] is the terminal model containing the column at parts[-1] - terminal_model_name = parts[-2] - try: - terminal_model = await self._resolve_model( - model_name=terminal_model_name, - named_queries=named_queries or {}, - prefer_data_source=model.data_source or None, - ) - except ValueError: - terminal_model = None - is_named_measure = ( - terminal_model.get_measure(parts[-1]) is not None - if terminal_model else False - ) - if not is_named_measure: - logger.warning( - "Auto-moved '%s' from measures to dimensions (not a named measure formula)", - formula, - ) - extra_dims.append(ColumnRef(name=formula)) - moved = True - continue - kept.append(f) - - if not moved: - return query - return query.model_copy(update={"measures": kept or None, "dimensions": extra_dims}) - - async def _resolve_cross_model_measure( - self, - spec_name: str, - field_name: str, - model: SlayerModel, - query, - dimensions: list, - time_dimensions: list, - label: str = None, - named_queries: dict = None, - aggregation_name: str = None, - agg_kwargs: dict = None, - ) -> CrossModelMeasure: - """Resolve a cross-model measure reference like 'customers.avg_score'. - - Supports multi-hop paths: 'claim_coverage.claim_amount.total_claim_amount' - walks the join graph hop-by-hop to reach the final model. - - Looks up the join from the source model, loads the target model - (checking named queries first), finds shared dimensions, and returns - a CrossModelMeasure for SQL generation. - """ - parts = spec_name.split(".") - if len(parts) < 2: - raise ValueError(f"Invalid cross-model measure reference: '{spec_name}'") - measure_name = parts[-1] - hop_names = parts[:-1] # e.g. ["claim_coverage", "claim_amount"] - - # Walk the join chain to find the final target model. v4 (DEV-1330): - # ``_walk_join_chain`` keeps each hop scoped to the source model's - # datasource, so ``customers.revenue:sum`` against ``orders@db_a`` - # never silently pulls ``customers@db_b``. - target_model, first_join = await self._walk_join_chain( - source_model=model, - hop_names=hop_names, - named_queries=named_queries, - strict_missing_join=True, - ) - - target_model_name = hop_names[-1] - join = first_join # For join_pairs: source model → first hop - - # Find the column in the target model - if measure_name == "*": - measure_def = Column(name="*", sql=None) - else: - from slayer.core.enums import NUMERIC_ONLY_AGGREGATIONS - - col_def = target_model.get_column(measure_name) - if col_def is None: - raise ValueError( - f"Column '{measure_name}' not found in model '{target_model_name}'. " - f"Available columns: {[c.name for c in target_model.columns]}" - ) - if ( - aggregation_name - and aggregation_name in NUMERIC_ONLY_AGGREGATIONS - and str(col_def.type) == "string" - ): - raise ValueError( - f"Aggregation '{aggregation_name}' is not applicable to " - f"string column '{measure_name}' in model '{target_model_name}'." - ) - measure_def = col_def - - # The cross-model sub-query starts FROM the source table with JOIN to - # the target, so all source dimensions are available for grouping. - # Use all query dimensions and time dimensions as the grouping context. - shared_dims = list(dimensions) - shared_time_dims = list(time_dimensions) - - query_model_name = query.source_model if isinstance(query.source_model, str) else model.name - - # Resolve aggregation: explicit colon syntax required - if aggregation_name: - agg = aggregation_name - canonical = f"_{aggregation_name}" if measure_name == "*" else f"{measure_name}_{aggregation_name}" - else: - raise ValueError( - f"Cross-model measure '{spec_name}' must include an aggregation (e.g., '{spec_name}:sum')." - ) - - hop_path = ".".join(hop_names) - alias = f"{query_model_name}.{hop_path}.{canonical}" - aggregation_def = target_model.get_aggregation(agg) - - # Infer format from the target model's measure and aggregation - cm_format = _infer_aggregated_format( - model=target_model, - measure_name=measure_name, - aggregation=agg, - ) - - # Expand derived references inside the target column's sql so that - # cross-model measures over chained derivations work. measure_def.sql - # is None for ``*:count``; nothing to expand there. - from slayer.engine.column_expansion import expand_derived_refs - - expanded_measure_sql = measure_def.sql - if measure_def.sql: - try: - ds = await self.storage.get_datasource(target_model.data_source) \ - if self.storage and target_model.data_source else None - except Exception: # noqa: BLE001 - ds = None - cross_dialect = self._dialect_for_type(ds.type) if ds else "postgres" - - async def _resolve_for_cross(model_name, named_queries): - try: - return await self._resolve_model( - model_name=model_name, - named_queries=named_queries or {}, - prefer_data_source=target_model.data_source or None, - ) - except Exception: # noqa: BLE001 - return None - - expanded = await expand_derived_refs( - sql=measure_def.sql, - model=target_model, - alias_path=target_model_name, - resolve_model=_resolve_for_cross, - named_queries=named_queries or {}, - dialect=cross_dialect, - ) - if expanded is not None: - expanded_measure_sql = expanded - - return CrossModelMeasure( - name=field_name, - alias=alias, - target_model_name=target_model_name, - target_model_sql_table=target_model.sql_table, - target_model_sql=target_model.sql, - measure=EnrichedMeasure( - name=canonical, - sql=expanded_measure_sql, - aggregation=agg, - alias=f"{target_model_name}.{canonical}", - model_name=target_model_name, - aggregation_def=aggregation_def, - agg_kwargs=agg_kwargs or {}, - source_measure_name=measure_name, - ), - join_pairs=join.join_pairs, - join_type=str(join.join_type), - shared_dimensions=shared_dims, - shared_time_dimensions=shared_time_dims, - source_model_name=model.name, - source_sql_table=model.sql_table, - source_sql=model.sql, - label=label, - format=cm_format, - ) - - async def _build_rerooted_enriched( - self, - cm: CrossModelMeasure, - query: SlayerQuery, - model: SlayerModel, - named_queries: dict, - ) -> EnrichedQuery: - """Build a re-rooted EnrichedQuery for a cross-model measure. - - Instead of the minimal source→target CTE, this constructs a full query - with the target model as source. All of the target model's joins are - available, so filters on related tables (e.g., premium.has_premium) - are applied correctly. - - Dimensions and filters referencing models not reachable from the - target are dropped. - """ - import re - - from slayer.core.formula import parse_filter - - target_model = await self._resolve_model( - model_name=cm.target_model_name, - named_queries=named_queries, - prefer_data_source=model.data_source or None, - ) - - source_model_name = model.name - target_model_name = cm.target_model_name - - # --- Build re-rooted field (measure becomes local) --- - measure_name = cm.measure.source_measure_name or cm.measure.name - aggregation = cm.measure.aggregation - if cm.measure.agg_kwargs: - kwargs_str = ", ".join(f"{k}={v}" for k, v in cm.measure.agg_kwargs.items()) - field_formula = f"{measure_name}:{aggregation}({kwargs_str})" - else: - field_formula = f"{measure_name}:{aggregation}" - - # --- Remap dimensions --- - rerooted_dims = [] - for dim in (query.dimensions or []): - if dim.model is None: - # Source-local dimension → cross-model from target's perspective - rerooted_dims.append(ColumnRef(name=f"{source_model_name}.{dim.name}")) - elif dim.model == target_model_name: - # Dimension on target model → now local - rerooted_dims.append(ColumnRef(name=dim.name)) - elif dim.model.startswith(target_model_name + "."): - # Path through target → strip target prefix - new_model = dim.model[len(target_model_name) + 1:] - rerooted_dims.append(ColumnRef(name=f"{new_model}.{dim.name}")) - else: - # Other cross-model dim → keep as-is (enrichment resolves via target's joins) - rerooted_dims.append(ColumnRef(name=dim.full_name)) - - # --- Remap time dimensions --- - rerooted_time_dims = [] - for td in (query.time_dimensions or []): - dim_ref = td.dimension - if dim_ref.model is None: - new_ref = ColumnRef(name=f"{source_model_name}.{dim_ref.name}") - elif dim_ref.model == target_model_name: - new_ref = ColumnRef(name=dim_ref.name) - elif dim_ref.model.startswith(target_model_name + "."): - new_model = dim_ref.model[len(target_model_name) + 1:] - new_ref = ColumnRef(name=f"{new_model}.{dim_ref.name}") - else: - new_ref = ColumnRef(name=dim_ref.full_name) - rerooted_time_dims.append(TimeDimension( - dimension=new_ref, - granularity=td.granularity, - date_range=td.date_range, - label=td.label, - )) - - # --- Remap filters --- - rerooted_filters = [] - target_prefix = target_model_name + "." - _custom_agg_names = frozenset( - a.name for m in (model, target_model) - for a in m.aggregations - ) or None - for f_str in (query.filters or []) + list(model.filters): - remapped = f_str - # Strip target model prefix from dotted references - # e.g., "policy_amount.premium.has_premium = '1'" → "premium.has_premium = '1'" - if target_prefix in remapped: - remapped = remapped.replace(target_prefix, "") - # For unqualified column references that are source model dimensions, - # prepend source model name (they're now on a joined table) - parsed = parse_filter(remapped, extra_agg_names=_custom_agg_names) - for col in parsed.columns: - if "." not in col: - src_col = model.get_column(col) - if src_col: - remapped = re.sub( - rf"(? DatasourceConfig: ds_name = model.data_source @@ -3851,12 +2907,11 @@ async def _resolve_datasource(self, model: SlayerModel) -> DatasourceConfig: return ds @staticmethod - def _dialect_for_type(ds_type: str | None) -> str: - """Map a datasource-config ``type`` string to a sqlglot dialect name. + def _dialect_for_type(ds_type: Optional[str]) -> str: + """Map a datasource ``type`` to its sqlglot dialect name. - Delegates to ``dialect_for_ds_type`` in ``slayer.sql.dialects`` - (DEV-1542). Returns a string for back-compat with the 24+ test - sites that assert ``_dialect_for_type("postgres") == "postgres"``. - Unknown / ``None`` / empty ds-types fall back to ``"postgres"``. + DEV-1716: delegates to the DEV-1542 registry (``dialect_for_ds_type``) + — single source of truth for the ds-type → dialect mapping — instead + of an inline duplicate map. Lenient (unknown / None → ``postgres``). """ return dialect_for_ds_type(ds_type).sqlglot_name diff --git a/slayer/engine/response_meta.py b/slayer/engine/response_meta.py new file mode 100644 index 00000000..7a5ed70a --- /dev/null +++ b/slayer/engine/response_meta.py @@ -0,0 +1,321 @@ +"""DEV-1450 stage 7b.15d — response metadata from the typed plan. + +The legacy engine derived ``SlayerResponse.attributes`` and +``expected_columns`` from an ``EnrichedQuery``. The typed pipeline has no +``EnrichedQuery``; this module rebuilds the same two artefacts from the root +``PlannedQuery`` plus the final rendered SQL. + +* ``expected_columns`` comes from the final SQL's ``named_selects`` — the + literal result-key columns the rows come back keyed by. Deriving them from + the SQL (rather than re-walking slots) is bulletproof: it is exactly the + outer SELECT projection the generator emitted. +* ``attributes`` (``ResponseAttributes.dimensions`` / ``.measures``) come from + the root ``PlannedQuery``'s public ``ValueSlot``s, mirroring the + ``_full_alias_for_slot`` result-key derivation in ``slayer/sql/generator.py`` + so the keys line up with the rendered projection. + +``FieldMetadata`` / ``ResponseAttributes`` / ``_infer_aggregated_format`` live +here (not in ``query_engine``) so this module imports nothing from the engine — +``query_engine`` re-exports them, keeping the dependency one-directional and +the public import path (``from slayer.engine.query_engine import FieldMetadata``) +unchanged. +""" + +from __future__ import annotations + +from typing import Dict, List, Optional, Tuple + +import sqlglot +from pydantic import BaseModel, Field as PydanticField + +from slayer.core.format import NumberFormat, NumberFormatType +from slayer.core.keys import ( + AggregateKey, + ColumnKey, + ColumnSqlKey, + Phase, + StarKey, + TimeTruncKey, + column_leaf, + column_path, +) +from slayer.core.models import Column, SlayerModel +from slayer.engine.planned import PlannedQuery, ValueSlot +from slayer.engine.source_bundle import ResolvedSourceBundle +from slayer.sql.dialects import get_dialect +from slayer.sql.naming import result_key, result_key_from_alias + + +# --------------------------------------------------------------------------- +# Response metadata types (moved here from query_engine for import hygiene). +# --------------------------------------------------------------------------- + + +class FieldMetadata(BaseModel): + """Metadata for a single field in the query response.""" + + label: Optional[str] = None + format: Optional[NumberFormat] = None + + +class ResponseAttributes(BaseModel): + """Field metadata for a query response, split by type.""" + + dimensions: Dict[str, FieldMetadata] = PydanticField(default_factory=dict) + measures: Dict[str, FieldMetadata] = PydanticField(default_factory=dict) + + def get(self, column: str) -> Optional[FieldMetadata]: + """Look up metadata for a column across both dicts.""" + return self.dimensions.get(column) or self.measures.get(column) + + +def _infer_aggregated_format( + model: SlayerModel, + measure_name: str, + aggregation: str, +) -> Optional[NumberFormat]: + """Infer NumberFormat for an aggregated measure based on aggregation type and source measure format. + + Rules: + - count, count_distinct, count_distinct_approx: always INTEGER + - avg, weighted_avg, median: always FLOAT + - sum, min, max, first, last: inherit from source measure + - *:count (measure_name="*"): INTEGER + """ + if measure_name == "*": + return NumberFormat(type=NumberFormatType.INTEGER) + + if aggregation in ("count", "count_distinct", "count_distinct_approx"): + return NumberFormat(type=NumberFormatType.INTEGER) + + if aggregation in ("avg", "weighted_avg", "median"): + return NumberFormat(type=NumberFormatType.FLOAT) + + # sum, min, max, first, last: inherit from source column's format + source_col = model.get_column(measure_name) + if source_col and source_col.format: + return source_col.format + + return None + + +# --------------------------------------------------------------------------- +# expected_columns / attributes from the typed plan +# --------------------------------------------------------------------------- + + +def expected_columns_from_sql(*, sql: str, dialect: str) -> List[str]: + """The outer SELECT's result-key columns, read from the rendered SQL. + + ``named_selects`` returns each projected column's alias (``orders.status``, + ``orders.revenue_sum``, ...) — the exact keys execution returns rows under. + """ + parsed = sqlglot.parse_one(sql, dialect=dialect) + return list(parsed.named_selects) + + +def _model_for_path( + *, bundle: ResolvedSourceBundle, path: Tuple[str, ...] +) -> Optional[SlayerModel]: + """The model a dotted join ``path`` lands on (best-effort). + + Empty path → the host source model. Otherwise the last path segment is + the join target model name; resolve it from the bundle's referenced + models, falling back to the host when absent. + """ + if not path: + return bundle.source_model + return bundle.get_referenced_model(path[-1]) or bundle.source_model + + +def _slot_result_keys(*, slot: ValueSlot, source_relation: str) -> List[str]: + """The public result-key alias(es) for ``slot``. + + Mirrors ``SQLGenerator._full_alias_for_slot`` via the SAME naming builders + (``slayer.sql.naming.result_key`` / ``result_key_from_alias``) so the SQL + alias and the response result key cannot drift. Joined ROW slots — base + ``ColumnKey``, derived ``ColumnSqlKey`` (DEV-1713 D3 / DEV-1495 bug 1), and + ``TimeTruncKey`` over either — emit the full dotted path + (``orders.customers.region``); everything else uses the slot's public + alias(es) — multiple for a C13 multi-name interned slot. + """ + key = slot.key + if slot.phase == Phase.ROW: + if isinstance(key, ColumnKey) and key.path: + return [result_key( + source_relation=source_relation, path=key.path, leaf=key.leaf, + )] + if isinstance(key, ColumnSqlKey) and key.path: + return [result_key( + source_relation=source_relation, + path=key.path, + leaf=key.column_name, + )] + if isinstance(key, TimeTruncKey) and column_path(key.column): + return [result_key( + source_relation=source_relation, + path=column_path(key.column), + leaf=column_leaf(key.column), + )] + aliases = slot.public_aliases or [slot.declared_name] + return [ + result_key_from_alias(source_relation=source_relation, alias=a) + for a in aliases + ] + + +def _column_for_row_slot( + *, slot: ValueSlot, bundle: ResolvedSourceBundle +) -> Optional[Column]: + """The source ``Column`` backing a ROW slot, for label / format lookup.""" + key = slot.key + if isinstance(key, TimeTruncKey): + key = key.column + if isinstance(key, ColumnKey): + model = _model_for_path(bundle=bundle, path=key.path) + leaf = key.leaf + elif isinstance(key, ColumnSqlKey): + model = bundle.get_referenced_model(key.model) or bundle.source_model + leaf = key.column_name + else: + return None + if model is None: + return None + return model.get_column(leaf) + + +def _owning_model_for_agg_source(*, src, bundle: ResolvedSourceBundle): + """The model that owns an aggregate's source column. + + A ``ColumnSqlKey`` (derived column) carries its owning model name in + ``src.model`` — resolve through that (DEV-1450 #4a/#4b), mirroring + ``_column_for_row_slot``. A ``ColumnKey`` / ``StarKey`` is resolved by + walking ``src.path`` from the host. + """ + if isinstance(src, ColumnSqlKey): + return bundle.get_referenced_model(src.model) or bundle.source_model + return _model_for_path(bundle=bundle, path=getattr(src, "path", ())) + + +def _measure_format( + *, slot: ValueSlot, bundle: ResolvedSourceBundle +) -> Optional[NumberFormat]: + """Number format for a measure slot. + + Aggregate slots inherit via ``_infer_aggregated_format`` (INTEGER for + count(-distinct) / star, FLOAT for avg-family, source-column format for + sum/min/max). Transform / arithmetic / scalar-call slots default to FLOAT, + matching the legacy ``EnrichedQuery`` expression/transform handling. + """ + key = slot.key + if isinstance(key, AggregateKey): + src = key.source + if isinstance(src, StarKey): + measure_name: Optional[str] = "*" + else: + measure_name = getattr(src, "leaf", None) or getattr( + src, "column_name", None + ) + model = _owning_model_for_agg_source(src=src, bundle=bundle) + if measure_name is None or model is None: + return NumberFormat(type=NumberFormatType.FLOAT) + return _infer_aggregated_format( + model=model, measure_name=measure_name, aggregation=key.agg + ) + return NumberFormat(type=NumberFormatType.FLOAT) + + +def _measure_label( + *, slot: ValueSlot, bundle: ResolvedSourceBundle +) -> Optional[str]: + """Label for a measure slot. + + A query measure (``labeled_rev:sum``) inherits its source column's label + when the measure spec carried none — mirroring the legacy enrichment that + propagated ``Column.label`` onto the aggregated field. Star aggregates and + transform / arithmetic slots have no single source column, so they fall + back to the slot's own label (usually ``None``). + """ + if slot.label: + return slot.label + key = slot.key + if isinstance(key, AggregateKey): + src = key.source + if isinstance(src, (ColumnKey, ColumnSqlKey)): + model = _owning_model_for_agg_source(src=src, bundle=bundle) + leaf = getattr(src, "leaf", None) or getattr( + src, "column_name", None, + ) + if model is not None and leaf is not None: + col = model.get_column(leaf) + if col is not None: + return col.label + return None + + +def build_response_metadata( # NOSONAR(S3776) — flat per-slot metadata classification (dimension vs measure, TimeTruncKey, label/format lookup) over one candidate-slot loop; complexity is inherent to the projection-to-metadata mapping and pre-dates this change. Splitting the loop body out would scatter the shared public_keys / source_relation state without improving readability. + *, + root_planned: PlannedQuery, + bundle: ResolvedSourceBundle, + sql: str, + dialect: str, +) -> Tuple[ResponseAttributes, List[str]]: + """Build ``(attributes, expected_columns)`` for one executed query. + + ``expected_columns`` is read from the rendered SQL (bulletproof); + ``attributes`` maps each public result key to its ``FieldMetadata``, + classified dimension (ROW-phase slots) vs measure (everything else). + Only keys that actually appear in the rendered projection are surfaced — + a guard against any divergence between this derivation and the generator. + """ + expected_columns = expected_columns_from_sql(sql=sql, dialect=dialect) + # DEV-1716: on BigQuery / T-SQL the rendered SQL carries alias-mangled + # projection names (``orders___status``); decode them back to the canonical + # dotted form so ``expected_columns`` and the attribute-matching below + # operate in the same space as the plan's slot result keys. Reuses the + # dialect read-side hook (identity for every non-mangling dialect) via a + # synthetic-row wrap — no ``decode_columns`` method needed. + if expected_columns: + expected_columns = list( + get_dialect(dialect).decode_result_keys([dict.fromkeys(expected_columns)])[0] + ) + public_keys = set(expected_columns) + source_relation = root_planned.source_relation + + dim_meta: Dict[str, FieldMetadata] = {} + measure_meta: Dict[str, FieldMetadata] = {} + + projection_ids = set(root_planned.projection) + candidate_slots = ( + list(root_planned.row_slots) + + list(root_planned.aggregate_slots) + + list(root_planned.combined_expression_slots) + ) + for slot in candidate_slots: + if slot.hidden or slot.id not in projection_ids: + continue + is_dim = slot.phase == Phase.ROW + for rk in _slot_result_keys(slot=slot, source_relation=source_relation): + if rk not in public_keys: + continue + if is_dim: + # Label falls back to the model Column's label when the query + # ColumnRef carried none (legacy ``dim_ref.label or + # dim_def.label``). + col = _column_for_row_slot(slot=slot, bundle=bundle) + label = slot.label or (col.label if col else None) + if isinstance(slot.key, TimeTruncKey): + # Time dimensions carry a label only (legacy parity). + if label: + dim_meta[rk] = FieldMetadata(label=label) + continue + fmt = col.format if col else None + if label or fmt: + dim_meta[rk] = FieldMetadata(label=label, format=fmt) + else: + fmt = _measure_format(slot=slot, bundle=bundle) + label = _measure_label(slot=slot, bundle=bundle) + if label or fmt: + measure_meta[rk] = FieldMetadata(label=label, format=fmt) + + return ResponseAttributes(dimensions=dim_meta, measures=measure_meta), expected_columns diff --git a/slayer/engine/schema_drift.py b/slayer/engine/schema_drift.py index 5997e846..0dff9a04 100644 --- a/slayer/engine/schema_drift.py +++ b/slayer/engine/schema_drift.py @@ -17,8 +17,11 @@ import logging from typing import ( Annotated, - Any, + List, Literal, + Optional, + Set, + Union, ) import sqlalchemy as sa @@ -27,14 +30,7 @@ from sqlglot import exp from slayer.core.enums import DataType -from slayer.core.formula import ( - AggregatedMeasureRef, - ArithmeticField, - MixedArithmeticField, - TransformField, - parse_filter, - parse_formula, -) +from slayer.core.formula import parse_filter from slayer.core.models import ( Column, DatasourceConfig, @@ -48,6 +44,15 @@ _sa_type_is_float, _sa_type_to_data_type, ) +from slayer.engine.normalization import func_style_agg_to_colon +from slayer.engine.syntax import ( + AggCall, + DottedRef, + Ref, + StarSource, + parse_expr, + walk_parsed_refs, +) from slayer.sql.client import SlayerSQLClient logger = logging.getLogger(__name__) @@ -507,70 +512,63 @@ def _extract_column_refs_from_sql(sql: str) -> list[tuple[str | None, str]]: return refs -def _agg_ref_names(agg_refs: dict[str, AggregatedMeasureRef]) -> set[str]: - """Names from a ``measure:agg`` placeholder map, excluding ``*``.""" - return {ref.measure_name for ref in agg_refs.values() if ref.measure_name != "*"} - - -def _bare_measure_names( - measure_names: list[str], - agg_refs: dict[str, AggregatedMeasureRef], - *, - skip_placeholder_prefix: str | None = None, -) -> set[str]: - """Filter raw ``measure_names`` to the ones that are not colon-syntax - placeholders, optionally also stripping sub-transform placeholders. - """ - out: set[str] = set() - for n in measure_names: - if n in agg_refs: - continue - if skip_placeholder_prefix and n.startswith(skip_placeholder_prefix): - continue - out.add(n) - return out - +def _parsed_ref_name(node: Union[Ref, DottedRef, AggCall]) -> Optional[str]: + """Textual name of a reference-bearing parse node. -def _walk_field_spec_measure_refs(spec: Any) -> set[str]: - """Walk a ``FieldSpec`` (parse_formula output) and return the set of - measure_name strings (which may be dotted: ``"customers.revenue"``). + ``AggCall`` collapses to its aggregated source name — the agg itself is + not a column reference, and ``*:count`` (``StarSource`` source) yields + ``None`` because ``*`` is not a real column. Args / kwargs of the + aggregation are opaque (legacy parity). Bare ``Ref`` / ``DottedRef`` + surface as their dotted textual form. """ - if isinstance(spec, AggregatedMeasureRef): - return _agg_ref_names({"_": spec}) - if isinstance(spec, ArithmeticField): - return _agg_ref_names(spec.agg_refs) | _bare_measure_names( - spec.measure_names, spec.agg_refs - ) - if isinstance(spec, MixedArithmeticField): - out = _agg_ref_names(spec.agg_refs) | _bare_measure_names( - spec.measure_names, spec.agg_refs, skip_placeholder_prefix="_t" - ) - for _, t in spec.sub_transforms: - out.update(_walk_field_spec_measure_refs(t)) - return out - if isinstance(spec, TransformField): - return _walk_field_spec_measure_refs(spec.inner) - return set() + if isinstance(node, AggCall): + source = node.source + if isinstance(source, StarSource): + return None + node = source + if isinstance(node, Ref): + return node.name + return ".".join(node.parts) def _measure_formula_refs( - formula: str, - *, - named_measures: dict[str, str] | None = None, -) -> set[str]: - """Best-effort: parse ``formula`` and return the set of column / measure - names it references. Returns the empty set on any parse failure. - - ``named_measures`` is the map ``{measure_name: formula_text}`` for the - enclosing model — required for bare measure references like - ``aov / *:count`` (where ``aov`` is itself a saved measure on the - model) to parse cleanly. + formula: str, *, custom_agg_names: Optional[Set[str]] = None, +) -> Set[str]: + """Best-effort: parse ``formula`` (Mode-B DSL) and return the set of + column / measure names it references (dotted for cross-model refs, e.g. + ``"customers.revenue"``). Returns the empty set on any parse failure. + + Textual extraction only — no scope binding. The cascade attribution + checks each returned name against the dropped-column / dropped-measure + sets itself, so bare named-measure refs surface by name (``aov``) rather + than being inline-expanded; the cascade reaches the underlying column + through the dropped-measure set in a later fixed-point pass. + + Function-style aggregations on legacy / un-normalized persisted formulas + (``sum(amount)``) are rewritten to colon syntax first via the quiet + ``FUNC_STYLE_AGG`` slack helper — matching the legacy ``parse_formula`` + path. ``custom_agg_names`` lets model-level custom aggregations + (``weighted_avg(amount, weight=qty)``) rewrite too (CR); without them + the call parses as an unknown function and the refs are lost, leaving + the drift cascade incomplete. """ try: - spec = parse_formula(formula, named_measures=named_measures) + parsed = parse_expr( + func_style_agg_to_colon( + formula, + custom_agg_names=( + frozenset(custom_agg_names) if custom_agg_names else None + ), + ) + ) except Exception: return set() - return _walk_field_spec_measure_refs(spec) + out: Set[str] = set() + for node in walk_parsed_refs(parsed): + name = _parsed_ref_name(node) + if name is not None: + out.add(name) + return out def _filter_refs(filter_str: str) -> list[str]: @@ -890,13 +888,25 @@ def _attribute_ref_to_base( def _measure_refs_on_base( stage: SlayerQuery, base_name: str, graph: _StageGraph -) -> set[str]: - out: set[str] = set() +) -> Set[str]: + out: Set[str] = set() + # Custom aggregations on models REACHABLE from the stage source so + # function-style custom aggs in a stage measure rewrite to colon form + # before ref extraction. Scoped to ``graph.reachable`` (not every model + # in the registry) so unrelated custom-agg names can't normalize a + # coincidental function call and produce a false cascade hit (CR). + custom_agg_names: Set[str] = set() + for model_name in graph.reachable: + model = graph.models_by_name.get(model_name) + if model is not None: + custom_agg_names.update(a.name for a in (model.aggregations or [])) for m in stage.measures or []: formula = getattr(m, "formula", None) if not formula: continue - for ref in _measure_formula_refs(formula): + for ref in _measure_formula_refs( + formula, custom_agg_names=custom_agg_names, + ): attributed = _attribute_ref_to_base( ref=ref, base_name=base_name, graph=graph ) @@ -1328,17 +1338,46 @@ def _first_dropped_cause( return None +def _reachable_agg_names_from_state( + *, start: SlayerModel, state: "_CascadeState", +) -> Set[str]: + """Sync BFS over ``state.models_by_name`` collecting custom aggregation + names reachable from ``start`` via the join graph. DEV-1500 — lets the + measure-cascade rule recognise function-style references to custom + aggregations defined on joined models (``rolling_avg(customers.score)`` + where ``rolling_avg`` lives on the joined ``customers``). Visited-guarded, + unbounded depth; absent targets are skipped (best-effort). + """ + names: Set[str] = set() + visited: Set[str] = set() + queue: List[SlayerModel] = [start] + while queue: + current = queue.pop(0) + if current.name in visited: + continue + visited.add(current.name) + if current.aggregations: + names.update(a.name for a in current.aggregations) + for join in current.joins: + if join.target_model in visited: + continue + nxt = state.models_by_name.get(join.target_model) + if nxt is not None: + queue.append(nxt) + return names + + def _cascade_measures(*, model: SlayerModel, state: _CascadeState) -> bool: """Rule 2: ``ModelMeasure.formula`` referencing a dropped column or dropped measure.""" changed = False - named_measures = {m.name: m.formula for m in model.measures if m.name} dropped_set = state.dropped_measures.get(model.name, set()) + custom_agg_names = _reachable_agg_names_from_state(start=model, state=state) for measure in model.measures: if measure.name is None or measure.name in dropped_set: continue refs = _measure_formula_refs( - measure.formula, named_measures=named_measures + measure.formula, custom_agg_names=custom_agg_names, ) cause = _first_dropped_cause(refs=refs, model=model, state=state) if cause is None: diff --git a/slayer/engine/source_bundle.py b/slayer/engine/source_bundle.py new file mode 100644 index 00000000..5df227f8 --- /dev/null +++ b/slayer/engine/source_bundle.py @@ -0,0 +1,558 @@ +"""Stage 2 (DEV-1450) — ResolvedSourceBundle: eagerly resolved query inputs (P11). + +The orchestrator builds this once at the top of execute; the binder reads +from it purely. No ContextVar machinery, no callback re-resolution — the +binder is provably scope-only because everything it needs is in the bundle. + +Contents (per DEV-1450 spec): +- Source model (the host of the query). +- All other referenced models (joined targets, sibling stage hosts). +- Inline ``ModelExtension`` overlays (extra columns / measures / joins). +- Named query siblings (raw ``SlayerQuery``s; the stage planner compiles + each to its own ``StageSchema`` as siblings are traversed in + topological order). +- ``query_variables`` (merged precedence: runtime > stage > outer > model). +- Datasource hint (the ``data_source=`` kwarg that wins over the priority + list). + +Per I2 of the DEV-1450 execution plan, ``source_model`` is ``Optional`` +from day one. DEV-1450's binder asserts ``source_model is not None``; +the type-level optionality is the extension point for a future +anchor-less mode. +""" + +from __future__ import annotations + +import logging +from typing import TYPE_CHECKING, Any, Dict, List, Optional, Union + +from pydantic import BaseModel, ConfigDict, Field + +from slayer.core.enums import DataType +from slayer.core.models import Column, ModelJoin, ModelMeasure, SlayerModel +from slayer.core.query import ModelExtension, SlayerQuery +from slayer.engine.variables import merge_query_variables + +if TYPE_CHECKING: + from slayer.core.scope import StageSchema + from slayer.storage.base import StorageBackend + +logger = logging.getLogger(__name__) + + +class ResolvedSourceBundle(BaseModel): + """Eagerly resolved inputs to one query execution (P11).""" + + model_config = ConfigDict(arbitrary_types_allowed=True) + + source_model: Optional[SlayerModel] = None + referenced_models: List[SlayerModel] = Field(default_factory=list) + inline_extensions: List[ModelExtension] = Field(default_factory=list) + named_queries: Dict[str, SlayerQuery] = Field(default_factory=dict) + # DEV-1450 stage 7b.15d — per-named-stage resolved source model, keyed by + # stage name. Populated for siblings whose source resolves to a concrete + # model (a stored model, an inline ``SlayerModel``, or a ``ModelExtension`` + # over a stored base). Siblings sourced FROM another sibling (chain or a + # ``ModelExtension`` over a sibling) are omitted — the planner resolves + # those against the upstream ``StageSchema`` at plan time. Lets each stage + # in a heterogeneous DAG bind against its OWN source rather than the root's. + stage_source_models: Dict[str, SlayerModel] = Field(default_factory=dict) + query_variables: Dict[str, Any] = Field(default_factory=dict) + datasource_hint: Optional[str] = None + + def get_referenced_model(self, name: str) -> Optional[SlayerModel]: + """Linear lookup by name. The list is small (handful of joined + models per query), so the O(n) scan is fine. + """ + for m in self.referenced_models: + if m.name == name: + return m + return None + + def reachable_aggregation_names( + self, *, start: SlayerModel, + ) -> Optional[frozenset[str]]: + """Custom aggregation names reachable from ``start`` via the join + graph, resolved against this bundle's pre-loaded + ``referenced_models``. Sync mirror of + ``agg_registry.collect_reachable_agg_names`` — used by the slack + FUNC_STYLE_AGG normalizer so a custom aggregation defined on a + joined model (e.g. ``rolling_avg(customers.score)``) is recognised + and rewritten to colon form. + + BFS, visited-guarded by model name, unbounded depth. Join targets + absent from ``referenced_models`` are skipped (best-effort, matches + the rest of the bundle). Returns ``None`` when nothing reachable + carries any custom aggregation; the contract is "None when empty", + not an empty frozenset. + + Scoping is per call: a stage normalises against its own source + model, so a stage only sees aggregations reachable from the model + it actually queries — not the union across every sibling stage. + """ + names: set[str] = set() + visited: set[str] = set() + queue: list[SlayerModel] = [start] + while queue: + current = queue.pop(0) + if current.name in visited: + continue + visited.add(current.name) + if current.aggregations: + names.update(a.name for a in current.aggregations) + for join in current.joins: + if join.target_model in visited: + continue + nxt = self.get_referenced_model(join.target_model) + if nxt is not None: + queue.append(nxt) + return frozenset(names) if names else None + + +# Anything accepted as ``SlayerQuery.source_model``. +SourceSpec = Union[str, SlayerModel, ModelExtension, Dict[str, Any]] + + +def _apply_extension_overlay( + base: SlayerModel, ext: ModelExtension +) -> SlayerModel: + """Extend ``base`` with the extra columns / measures / joins of ``ext``. + + Mirrors the ``ModelExtension`` branch of + ``SlayerQueryEngine._resolve_query_model`` so the typed pipeline sees the + same overlaid model the legacy path produced. + """ + extra_cols = [ + Column.model_validate(c) if isinstance(c, dict) else c + for c in (ext.columns or []) + ] + extra_measures = [ + ModelMeasure.model_validate(m) if isinstance(m, dict) else m + for m in (ext.measures or []) + ] + extra_joins = [ + ModelJoin.model_validate(j) if isinstance(j, dict) else j + for j in (ext.joins or []) + ] + return base.model_copy( + update={ + "columns": list(base.columns) + extra_cols, + "measures": list(base.measures) + extra_measures, + "joins": list(base.joins) + extra_joins, + } + ) + + +def _source_name_if_sibling( + spec: SourceSpec, sibling_names: "set[str] | Dict[str, Any]" +) -> Optional[str]: + """Return the sibling stage name a ``source_model`` spec reads from, if any. + + Covers the bare-string form (``source_model="kpis"``) AND the + ``ModelExtension`` / dict-with-``source_name`` form + (``source_model={"source_name": "kpis", ...}``) — both reference a sibling + when the name is in ``sibling_names``. Returns ``None`` otherwise. + """ + if isinstance(spec, str): + return spec if spec in sibling_names else None + if isinstance(spec, ModelExtension): + return spec.source_name if spec.source_name in sibling_names else None + if isinstance(spec, dict) and isinstance(spec.get("source_name"), str): + nm = spec["source_name"] + return nm if nm in sibling_names else None + return None + + +def _follow_sibling_chain( + spec: SourceSpec, named_queries: Dict[str, SlayerQuery] +) -> SourceSpec: + """Resolve a ``source_model`` that points at a named sibling stage down + to the real base spec it ultimately reads from. + + The bundle's ``source_model`` must be the real base the root chain bottoms + out at — not a sibling name — so a query-backed datasource lookup and the + single-model binding path resolve correctly. Follows both the bare-string + sibling form and the ``ModelExtension`` / dict-over-sibling form down to + the first spec that does NOT read from a sibling. A cycle raises + ``ValueError`` (mirrors the legacy ``_resolve_model`` circular-reference + guard). + """ + seen: List[str] = [] + while True: + sib = _source_name_if_sibling(spec, named_queries) + if sib is None: + return spec + if sib in seen: + chain = " -> ".join([*seen, sib]) + raise ValueError( + f"Circular reference detected in source_queries DAG: {chain}" + ) + seen.append(sib) + spec = named_queries[sib].source_model + + +async def _resolve_source_spec( + spec: SourceSpec, + *, + storage: "StorageBackend", + data_source: Optional[str], +) -> SlayerModel: + """Resolve any ``source_model`` spec to a concrete ``SlayerModel``. + + Storage-only and read-only (P11). Handles the four input shapes the + public API accepts: stored-model name, inline ``SlayerModel``, + ``ModelExtension`` overlay, and the dict forms of both. + """ + if isinstance(spec, SlayerModel): + return spec + if isinstance(spec, ModelExtension): + base = await storage.get_model(spec.source_name, data_source=data_source) + if base is None: + raise ValueError(f"Model '{spec.source_name}' not found") + return _apply_extension_overlay(base, spec) + if isinstance(spec, str): + model = await storage.get_model(spec, data_source=data_source) + if model is None: + raise ValueError(f"Model '{spec}' not found") + return model + if isinstance(spec, dict): + if "source_name" in spec: + ext = ModelExtension.model_validate(spec) + return await _resolve_source_spec( + ext, storage=storage, data_source=data_source + ) + return SlayerModel.model_validate(spec) + raise ValueError(f"Invalid source_model type: {type(spec)!r}") + + +async def build_resolved_source_bundle( + *, + query: SlayerQuery, + storage: "StorageBackend", + data_source: Optional[str] = None, + runtime_variables: Optional[Dict[str, Any]] = None, + outer_variables: Optional[Dict[str, Any]] = None, + named_queries: Optional[Dict[str, SlayerQuery]] = None, +) -> ResolvedSourceBundle: + """Eagerly assemble the :class:`ResolvedSourceBundle` for one execution (P11). + + Resolves the query's source model (every input shape), walks the join + graph transitively to collect every model the binder may hop through, + threads the named-query sibling map, merges the variable layers, and + records the datasource hint. Storage is consulted here and only here; + the binder then reads from the bundle purely. + + Variable precedence (highest first): runtime > query (stage) > outer > + source-model defaults. ``outer_variables`` is the enclosing-query layer + for a query-backed model resolved as a nested source — left ``None`` for + plain top-level execution. + """ + named_queries = named_queries or {} + sibling_names = set(named_queries) + + # The bundle's source_model is the real base the root chain bottoms out + # at — follow the sibling chain past any named-stage indirection. When the + # ROOT source is a ``ModelExtension`` over a NON-sibling base, the overlay + # is recorded in ``inline_extensions`` so the engine can re-apply it AFTER + # a query-backed base expands (expansion derives columns from the backing + # query and would otherwise drop the overlay's extra columns). + root_spec = _follow_sibling_chain(query.source_model, named_queries) + inline_extensions: List[ModelExtension] = [] + if _source_name_if_sibling(root_spec, sibling_names) is None: + ext = _as_extension_over_nonsibling(root_spec, sibling_names) + if ext is not None: + inline_extensions.append(ext) + source_model = await _resolve_source_spec( + root_spec, storage=storage, data_source=data_source + ) + + # Joins never cross datasource boundaries: scope the graph walk by the + # source model's own data_source (matches engine._expand_join_graph), + # falling back to the execution hint only when the model carries none. + walk_ds = source_model.data_source or data_source or None + + referenced_models = await _collect_referenced_models( + source_model=source_model, + named_queries=named_queries, + storage=storage, + data_source=walk_ds, + ) + + # Per-named-stage source models — each non-sibling-sourced sibling resolves + # to its OWN concrete model so heterogeneous DAGs (stage A over ``orders``, + # stage B over ``customers``) bind each stage against the right host. + stage_source_models: Dict[str, SlayerModel] = {} + for nm, nq in named_queries.items(): + if _source_name_if_sibling(nq.source_model, sibling_names) is not None: + continue # sibling-sourced: planner resolves via upstream StageSchema + # A non-sibling-sourced stage's source MUST resolve to a concrete model; + # a failure here (typoed / missing model) is a genuine error, not a + # best-effort skip — swallowing it would silently fall back to the root + # source and emit wrong SQL when column names overlap. + stage_source_models[nm] = await _resolve_source_spec( + nq.source_model, storage=storage, data_source=walk_ds or data_source + ) + + query_variables = merge_query_variables( + runtime=runtime_variables, + stage=query.variables, + outer=outer_variables, + model_defaults=source_model.query_variables, + ) + + return ResolvedSourceBundle( + source_model=source_model, + referenced_models=referenced_models, + inline_extensions=inline_extensions, + named_queries=dict(named_queries), + stage_source_models=stage_source_models, + query_variables=query_variables, + datasource_hint=data_source, + ) + + +async def _collect_referenced_models( + *, + source_model: SlayerModel, + named_queries: Dict[str, SlayerQuery], + storage: "StorageBackend", + data_source: Optional[str], +) -> List[SlayerModel]: + """Transitive join-graph walk (Kahn-free BFS), best-effort. + + Seeds: the source model, plus the real base model of every named sibling + stage (so a sibling's own ``plan_query`` can hop through targets the root + stage never touches). Follows each model's ``joins[].target_model`` within + ``data_source``; absent targets are skipped silently (mirrors + ``SlayerQueryEngine._expand_join_graph``). The source model is returned + first so ``get_referenced_model`` finds the host before any same-named + join target. + """ + # Models we already hold concretely (host + each sibling's real base, + # overlay-resolved), keyed by name. Resolving siblings through + # ``_resolve_source_spec`` means an extension-added join on a sibling is + # walked too. Best-effort: a sibling whose base is absent is skipped. + preseeded: Dict[str, SlayerModel] = {source_model.name: source_model} + for sib in named_queries.values(): + spec = _follow_sibling_chain(sib.source_model, named_queries) + try: + sib_model = await _resolve_source_spec( + spec, storage=storage, data_source=data_source + ) + except ValueError as exc: + logger.debug("sibling source resolution failed for %r: %s", spec, exc) + continue + preseeded.setdefault(sib_model.name, sib_model) + + collected: Dict[str, SlayerModel] = {} + visited: set[str] = set() + frontier: List[str] = list(preseeded) + while frontier: + name = frontier.pop() + if name in visited: + continue + visited.add(name) + model = preseeded.get(name) + if model is None: + try: + model = await storage.get_model(name, data_source=data_source) + except Exception as exc: # best-effort; absent target is fine + logger.debug("join-target lookup failed for %r: %s", name, exc) + model = None + if model is None: + continue + collected.setdefault(name, model) + for join in model.joins: + if join.target_model not in visited: + frontier.append(join.target_model) + + ordered = [source_model] + ordered.extend(m for n, m in collected.items() if n != source_model.name) + return ordered + + +def _as_extension_over_nonsibling( + spec: SourceSpec, sibling_names: "set[str]" +) -> Optional[ModelExtension]: + """Return the ``ModelExtension`` if ``spec`` overlays a NON-sibling base. + + Used to record the root overlay so the engine can re-apply it after a + query-backed base expands. Returns ``None`` for plain strings, inline + models, and overlays over a sibling (those are handled by the planner). + """ + if isinstance(spec, ModelExtension): + ext = spec + elif isinstance(spec, dict) and isinstance(spec.get("source_name"), str): + ext = ModelExtension.model_validate(spec) + else: + return None + if ext.source_name in sibling_names: + return None + return ext + + +def synthetic_model_from_stage_schema( + *, name: str, schema: "StageSchema", data_source: str +) -> SlayerModel: + """A stand-in ``SlayerModel`` whose ``sql_table`` is a stage's CTE name and + whose columns are that stage's flat output columns. + + Lets the binder / cross-model planner resolve a join (or cross-model ref) + targeting a sibling stage, and the generator emit ``FROM AS `` / + ``LEFT JOIN ...`` — the stage is materialised as a CTE elsewhere + (``generate_planned_stages``); this is the rendering vehicle for that CTE + relation. ``StageColumn.name`` is already the ``__``-flattened downstream + bind name, so the synthetic column names match how downstream refs bind. + """ + return SlayerModel( + name=name, + data_source=data_source or "_stage", + sql_table=name, + columns=[ + Column(name=c.name, type=c.type or DataType.DOUBLE) + for c in schema.columns + ], + ) + + +def stage_bundle_with_siblings( + *, + bundle: ResolvedSourceBundle, + source_model: SlayerModel, + sibling_schemas: Dict[str, "StageSchema"], + data_source: str, +) -> ResolvedSourceBundle: + """Per-stage bundle: ``source_model`` is the stage's own host; synthetic + sibling models (one per already-emitted ``StageSchema``) are threaded into + ``referenced_models`` so a join / cross-model ref to a sibling resolves. + + The host comes first (``get_referenced_model`` finds it before any same- + named join target), then the synthetic siblings, then the original bundle's + referenced models (minus any shadowed by the host or a synthetic sibling). + """ + synths = [ + synthetic_model_from_stage_schema( + name=n, schema=s, data_source=data_source + ) + for n, s in sibling_schemas.items() + ] + shadow = {source_model.name} | {s.name for s in synths} + referenced = ( + [source_model] + + synths + + [m for m in bundle.referenced_models if m.name not in shadow] + ) + return bundle.model_copy( + update={"source_model": source_model, "referenced_models": referenced} + ) + + +async def expand_query_backed_models_in_bundle( # NOSONAR(S3776) — three sequential expansion blocks (source + referenced + stage_source) that mutate the bundle in series. Each block guards on its own condition; splitting forces ``bundle`` to ping-pong through helpers without simplifying anything. The inner recursion guard is the only shared state. + *, + bundle: ResolvedSourceBundle, + outer_vars: Optional[Dict[str, Any]], + runtime_kwarg: Optional[Dict[str, Any]], + dry_run_placeholders: bool, + expander, + _resolving: Optional["set[str]"] = None, +) -> ResolvedSourceBundle: + """Expand every query-backed model in the bundle and re-apply any root + ``ModelExtension`` overlay (DEV-1452 Stage B decision F). + + Three expansion blocks, mirroring ``_execute_pipeline`` exactly: + + 1. Source model — if ``bundle.source_model.source_queries`` is set, + expand to ``sql``-mode and re-apply every ``bundle.inline_extensions`` + overlay (expansion derives columns from the backing query and would + otherwise drop the overlay's extra columns). + 2. Referenced models — every join / cross-model target with + ``source_queries`` set expands so the generator renders it as a + backing-SQL subquery rather than a bare table. + 3. Stage source models — a non-root stage whose own source is a stored + query-backed model likewise expands to ``sql``-mode before the + planner binds it. + + ``expander`` is the callback that performs the actual per-model + expansion. ``_execute_pipeline`` and the migrated + ``_expand_query_backed_model`` both pass + ``self._expand_query_backed_model``; the callback signature is + ``async (model, *, outer_vars, runtime_kwarg, dry_run_placeholders, + _resolving) -> SlayerModel``. + + Returns a fresh ``ResolvedSourceBundle`` with the expanded models; + ``inline_extensions`` is preserved as-is for traceability but the + overlay has already been folded into ``source_model``. + + ``_resolving`` is the recursion guard: a set of model names already + being expanded in this asyncio task. A query-backed join target that + transitively references its parent (or itself) is short-circuited + with the cached ``backing_query_sql`` if available, otherwise left + unchanged — mirrors the legacy ``_render_query_backed_join_target`` + contract. + """ + resolving: "set[str]" = set(_resolving) if _resolving is not None else set() + + async def _expand_or_short_circuit(model: SlayerModel) -> SlayerModel: + if model.name in resolving: + # Re-entry: use cached backing SQL if available, otherwise + # return unchanged (the binder / generator will surface a + # clear error when no sql_table / sql is set). + if model.backing_query_sql: + return model.model_copy( + update={"sql": model.backing_query_sql}, + ) + return model + resolving.add(model.name) + try: + return await expander( + model=model, + outer_vars=outer_vars, + runtime_kwarg=runtime_kwarg, + dry_run_placeholders=dry_run_placeholders, + _resolving=resolving, + ) + finally: + resolving.discard(model.name) + + # 1. Source model + inline_extensions re-apply. + if bundle.source_model is not None and bundle.source_model.source_queries: + expanded = await _expand_or_short_circuit(bundle.source_model) + for ext in bundle.inline_extensions: + expanded = _apply_extension_overlay(expanded, ext) + bundle = bundle.model_copy( + update={ + "source_model": expanded, + "referenced_models": [expanded] + + [ + m + for m in bundle.referenced_models + if m.name != expanded.name + ], + } + ) + source_model = bundle.source_model + + # 2. Referenced models (skip the source model itself — handled above). + if source_model is not None and any( + rm.name != source_model.name and rm.source_queries + for rm in bundle.referenced_models + ): + expanded_refs: List[SlayerModel] = [] + for rm in bundle.referenced_models: + if rm.name != source_model.name and rm.source_queries: + rm = await _expand_or_short_circuit(rm) + expanded_refs.append(rm) + bundle = bundle.model_copy(update={"referenced_models": expanded_refs}) + + # 3. Stage source models. + if any(m.source_queries for m in bundle.stage_source_models.values()): + expanded_stage_sources: Dict[str, SlayerModel] = {} + for nm, sm in bundle.stage_source_models.items(): + if sm.source_queries: + sm = await _expand_or_short_circuit(sm) + expanded_stage_sources[nm] = sm + bundle = bundle.model_copy( + update={"stage_source_models": expanded_stage_sources} + ) + + return bundle diff --git a/slayer/engine/stage_ordering.py b/slayer/engine/stage_ordering.py new file mode 100644 index 00000000..4aacbc96 --- /dev/null +++ b/slayer/engine/stage_ordering.py @@ -0,0 +1,238 @@ +"""DEV-1452 Stage B — Kahn topo-sort for stored / runtime ``source_queries`` +stage lists. + +Extracted from ``SlayerQueryEngine._topologically_order_queries`` so the +migrated ``_expand_query_backed_model`` / ``_validate_and_populate_cache`` +can validate stored ``source_queries`` with the same fault-tolerance +contract the runtime ``execute(query=list[...])`` path uses. Decisions +#1 + E of the Stage B plan: + +* Last stage stays root / sink. Cycles, self-references, duplicate + names, and root referenced by another stage all raise ``ValueError`` + with messages that name the offending stage. Forward references — + a stage that names a sibling appearing later in the input list — are + reordered, not rejected: that is the whole point of the topo-sort, + and the user-facing contract is that stored / runtime stage lists + may be supplied in any topologically valid order. +* Sibling refs are walked recursively through inline ``SlayerModel`` + (typed or dict), ``ModelExtension`` (typed or dict), and ``ModelJoin`` + shapes inside ``joins[].target_model``. An inline-nested + ``source_queries`` list contributes edges from the enclosing stage to + any sibling referenced inside. + +The classmethod shim on ``SlayerQueryEngine`` delegates here so existing +call sites (``execute(query=list[...])`` at query_engine.py:469) remain +unchanged. +""" +from __future__ import annotations + +from typing import Any, Dict, List, Set + + +def _extract_sibling_refs(query: Any, against: Set[str]) -> Set[str]: + """Collect every sibling name referenced by ``query`` that appears in + ``against``. Walks ``source_model`` (string, typed ``SlayerModel`` / + ``ModelExtension``, or dict) plus any ``joins[].target_model``. + + When ``source_model`` is an inline ``SlayerModel`` carrying its own + ``source_queries``, recurses into each inner stage so a sibling name + hidden inside a nested stage still surfaces as an edge from the + enclosing stage. Same recursion through ``ModelExtension`` shapes. + + The traversal is purely structural — it never touches storage and + never raises on a missing-sibling reference (the caller validates + edges against ``against``). + """ + out: Set[str] = set() + _walk_spec(query.source_model, against, out) + return out + + +def _walk_spec(spec: Any, against: Set[str], out: Set[str]) -> None: # NOSONAR(S3776) — recursive walker over five ``source_model`` shapes (str / dict-ModelExtension / dict-inline-SlayerModel / typed ModelExtension / typed SlayerModel) plus nested ``source_queries`` recursion. Splitting fragments the per-shape contract; the recursion + isinstance dispatch IS the function. + """Recursively collect sibling refs from a ``source_model`` spec. + + Handled shapes: + * ``str`` — bare sibling name. + * ``dict`` — disambiguates by key shape: + - ``source_name`` present → ``ModelExtension`` form. + - ``source_queries`` present → inline ``SlayerModel`` form. + - otherwise treated as inline ``SlayerModel``. + * Typed ``SlayerModel`` — walk ``joins[].target_model`` AND every + inner stage's ``source_model`` in ``source_queries``. + * Typed ``ModelExtension`` — walk ``source_name`` and ``joins``. + """ + if spec is None: + return + if isinstance(spec, str): + if spec in against: + out.add(spec) + return + if isinstance(spec, dict): + if "source_name" in spec: + src = spec.get("source_name") + if isinstance(src, str) and src in against: + out.add(src) + for j in spec.get("joins") or []: + tgt = ( + j.get("target_model") if isinstance(j, dict) + else getattr(j, "target_model", None) + ) + if isinstance(tgt, str) and tgt in against: + out.add(tgt) + return + # Inline SlayerModel-as-dict (presence of ``source_queries`` is + # the discriminator from a typed-shape dict; also handles the + # legitimate no-source_queries inline-model dict by inspecting + # ``joins`` directly). + for j in spec.get("joins") or []: + tgt = ( + j.get("target_model") if isinstance(j, dict) + else getattr(j, "target_model", None) + ) + if isinstance(tgt, str) and tgt in against: + out.add(tgt) + for inner_q in spec.get("source_queries") or []: + inner_spec = ( + inner_q.get("source_model") + if isinstance(inner_q, dict) + else getattr(inner_q, "source_model", None) + ) + _walk_spec(inner_spec, against, out) + return + # Typed shapes — ModelExtension vs SlayerModel discriminated by the + # presence of ``source_name`` (only ModelExtension has it). + src = getattr(spec, "source_name", None) + if isinstance(src, str) and src in against: + out.add(src) + for j in getattr(spec, "joins", None) or []: + tgt = getattr(j, "target_model", None) + if isinstance(tgt, str) and tgt in against: + out.add(tgt) + # Inline SlayerModel may itself carry ``source_queries``; recurse so + # references hidden inside any nested stage's ``source_model`` + # surface as edges from the enclosing stage. + for inner_q in getattr(spec, "source_queries", None) or []: + inner_spec = getattr(inner_q, "source_model", None) + _walk_spec(inner_spec, against, out) + + +def _index_query_list_by_name(rest: List[Any], root: Any) -> Dict[str, Any]: + """Build ``{name: query}`` for the non-final entries. Validates that + every non-final stage has a unique non-empty name and that the + root's name (if any) doesn't collide with a sibling. + """ + rest_by_name: Dict[str, Any] = {} + for q in rest: + if not q.name: + raise ValueError( + "Every non-final entry in a query list must have a " + "'name' (siblings reference each other by name)." + ) + if q.name in rest_by_name: + raise ValueError(f"Duplicate stage name '{q.name}' in query list.") + rest_by_name[q.name] = q + if root.name and root.name in rest_by_name: + raise ValueError( + f"Stage name '{root.name}' is duplicated: the final entry " + f"shares a name with an earlier entry." + ) + return rest_by_name + + +def _validate_query_list_invariants( + queries: List[Any], + rest: List[Any], + root: Any, + sibling_names: Set[str], +) -> None: + """Reject self-references and any sibling that depends on the root. + + Self-references are caught for every entry (including the root). + Root-as-sink: no non-final stage may reference the root by name. + """ + for q in queries: + if q.name and q.name in _extract_sibling_refs(q, {q.name} | sibling_names): + raise ValueError( + f"Stage '{q.name}' references itself — self-references " + f"are not allowed." + ) + if root.name: + referrers = sorted( + q.name for q in rest if root.name in _extract_sibling_refs(q, {root.name}) + ) + if referrers: + raise ValueError( + f"The final entry '{root.name}' is the DAG root and must " + f"not be referenced by other stages. Referenced by: " + f"{referrers}." + ) + + +def _build_dependency_graph( + rest_by_name: Dict[str, Any], + sibling_names: Set[str], +) -> "tuple[Dict[str, int], Dict[str, List[str]]]": + """Build the (in_degree, dependents) adjacency for Kahn's algorithm.""" + in_degree: Dict[str, int] = dict.fromkeys(rest_by_name, 0) + dependents: Dict[str, List[str]] = {name: [] for name in rest_by_name} + for name, q in rest_by_name.items(): + for prereq in _extract_sibling_refs(q, sibling_names): + dependents[prereq].append(name) + in_degree[name] += 1 + return in_degree, dependents + + +def _kahn_sort( + in_degree: Dict[str, int], + dependents: Dict[str, List[str]], +) -> List[str]: + """Topologically sort by Kahn's algorithm. Cycle → ``ValueError``. + + The frontier is kept sorted for deterministic output order across runs. + """ + frontier: List[str] = sorted(n for n, d in in_degree.items() if d == 0) + sorted_names: List[str] = [] + while frontier: + n = frontier.pop(0) + sorted_names.append(n) + unlocked: List[str] = [] + for dep in dependents[n]: + in_degree[dep] -= 1 + if in_degree[dep] == 0: + unlocked.append(dep) + frontier.extend(sorted(unlocked)) + if len(sorted_names) < len(in_degree): + cycle = sorted(set(in_degree) - set(sorted_names)) + raise ValueError( + f"Cycle in query list: stages {cycle} form a cyclic " + f"dependency. The reference graph must be acyclic." + ) + return sorted_names + + +def topologically_order_stages(queries: List[Any]) -> List[Any]: + """Re-order a query list so every stage appears after the siblings it + references via ``source_model`` or ``joins[].target_model``. + + The final entry is the entry point / DAG root: it stays last. Only + the non-final entries are reordered. Stages that aren't reachable + from the root are accepted as utility sub-queries — they flow + through the sort like any other node. + + Raises ``ValueError`` on: missing ``name`` on a non-final entry; + duplicate stage names; self-references; the root being depended on + by any other stage; or a cycle among non-final stages. + """ + if len(queries) <= 1: + return list(queries) + rest = list(queries[:-1]) + root = queries[-1] + rest_by_name = _index_query_list_by_name(rest, root) + sibling_names: Set[str] = set(rest_by_name) + _validate_query_list_invariants(queries, rest, root, sibling_names) + in_degree, dependents = _build_dependency_graph(rest_by_name, sibling_names) + sorted_names = _kahn_sort(in_degree, dependents) + return [rest_by_name[n] for n in sorted_names] + [root] + + +__all__ = ["topologically_order_stages"] diff --git a/slayer/engine/stage_planner.py b/slayer/engine/stage_planner.py new file mode 100644 index 00000000..10228061 --- /dev/null +++ b/slayer/engine/stage_planner.py @@ -0,0 +1,2575 @@ +"""Stage 7a.7 (DEV-1450) — multi-stage source_queries planner. + +Orchestrates a list of ``SlayerQuery`` stages into a list of +``PlannedQuery``s, the typed input the SQL generator (stage 7b) will +consume. + +Per-stage pipeline: + + raw SlayerQuery → parse (per measure / filter / order) → bind → + ProjectionPlanner → PlannedQuery (+ emitted StageSchema) + +Multi-stage: + +* Stages are topologically sorted so each stage appears after the + siblings it references via ``source_model``. +* Downstream stages bind against the upstream ``StageSchema`` (P6) — + flat namespace, no dotted-join walking. ``IllegalScopeReferenceError`` + on dotted refs (DEV-1449). +* Each stage's ``StageSchema`` columns use the user-supplied ``name`` + (or canonical alias) as the column ``name`` (DEV-1448). + +Dormant in 7a — no engine wiring. Stage 7b's engine cutover flips +``engine.execute`` / ``engine.save_model`` over to ``plan_stages``. +""" + +from __future__ import annotations + +from typing import Dict, FrozenSet, List, Optional, Tuple, Union + +from slayer.core.enums import DataType +from slayer.core.format import NumberFormat +from slayer.core.errors import ( + AmbiguousReferenceError, + DistinctDimensionValuesError, + UnknownReferenceError, + UnresolvableOrderColumnError, +) +from slayer.core.keys import ( + AggregateKey, + ArithmeticKey, + BetweenKey, + ColumnKey, + ColumnSqlKey, + InKey, + LiteralKey, + Phase, + ScalarCallKey, + StarKey, + TimeTruncKey, + TransformKey, + ValueKey, + normalize_scalar, +) +from slayer.core.models import SlayerModel +from slayer.core.query import ( + ORDER_PLACEHOLDER_NAMES, + ModelExtension, + SlayerQuery, + TimeDimension, +) +from slayer.core.refs import agg_kwarg_canonical_str, canonical_agg_name +from slayer.core.time_bounds import strip_frame_bounds +from slayer.core.window_duration import parse_window_duration +from slayer.core.scope import ModelScope, StageColumn, StageSchema +from slayer.engine.aggregate_input_paths import ( + compute_aggregate_input_join_paths, +) +from slayer.engine.binding import ( + BoundExpr as BinderBoundExpr, + BoundFilter, + bind_expr, + bind_filter, + bind_time_dimension, + walk_value_keys, +) +from slayer.engine.cross_model_planner import ( + CrossModelPlanner, + HostFilterRouting, + IsolatedCteCrossModelPlanner, +) +from slayer.engine.measure_expansion import expand_model_measures +from slayer.engine.response_meta import _infer_aggregated_format +from slayer.engine.planned import ( + BoundExpr as PlannedBoundExpr, + FilterPhase, + OrderEntry, + PlannedQuery, + SrcFilterRewrite, + TransformLayer, + ValueSlot, + WindowedAggregatePlan, +) +from slayer.engine.planning import ( + DeclaredMeasure, + OrderSpec, + ProjectionPlanner, + _canonical_name, + _iter_slot_deps, + filter_referenced_slot_ids, + lower_sugar_transforms, + rewrite_rank_partition_keys, +) +from slayer.engine.source_bundle import ( + ResolvedSourceBundle, + _apply_extension_overlay, + _source_name_if_sibling, + stage_bundle_with_siblings, + synthetic_model_from_stage_schema, +) +from slayer.engine.normalization import func_style_agg_to_colon +from slayer.engine.syntax import parse_expr, parse_filter_expr +from slayer.sql.naming import flat_name +from slayer.sql.sql_expr import has_window_function +from slayer.sql.sql_predicate import parse_sql_predicate + + +__all__ = ["plan_query", "plan_stages"] + + +# Stage 7b.10 — TIME_NEEDING transform ops that require a resolvable +# time dimension to render their OVER ``ORDER BY``. Mirrors the legacy +# ``TIME_TRANSFORMS`` set at ``slayer/core/formula.py:33``. +_TIME_NEEDING_TRANSFORM_OPS = frozenset({ + "cumsum", + "change", + "change_pct", + "time_shift", + "first", + "last", + "lag", + "lead", + "consecutive_periods", +}) + + +def _attach_time_keys( + key: ValueKey, *, td_key: TimeTruncKey, +) -> ValueKey: + """Walk ``key``; for every ``TransformKey`` whose op needs a time + dimension and whose ``time_key`` is ``None``, return a copy with + ``time_key=td_key``. Identity-preserving when nothing changes. + + Mirrors ``lower_sugar_transforms``' walker shape so identity + semantics line up: nested TransformKey/ArithmeticKey/ScalarCallKey/ + BetweenKey trees are rebuilt only on the path containing a patch. + """ + if isinstance(key, TransformKey): + new_input = _attach_time_keys(key.input, td_key=td_key) + out = key + if new_input is not key.input: + out = out.model_copy(update={"input": new_input}) + if out.op in _TIME_NEEDING_TRANSFORM_OPS and out.time_key is None: + out = out.model_copy(update={"time_key": td_key}) + return out + if isinstance(key, ArithmeticKey): + new_ops = tuple( + _attach_time_keys(o, td_key=td_key) for o in key.operands + ) + if all(a is b for a, b in zip(new_ops, key.operands)): + return key + return ArithmeticKey(op=key.op, operands=new_ops) + if isinstance(key, ScalarCallKey): + new_args = tuple( + _attach_time_keys(a, td_key=td_key) + if isinstance( + a, (TransformKey, ArithmeticKey, ScalarCallKey, BetweenKey), + ) + else a + for a in key.args + ) + if all(a is b for a, b in zip(new_args, key.args)): + return key + return ScalarCallKey(name=key.name, args=new_args) + if isinstance(key, BetweenKey): + nc = _attach_time_keys(key.column, td_key=td_key) + nl = _attach_time_keys(key.low, td_key=td_key) + nh = _attach_time_keys(key.high, td_key=td_key) + if nc is key.column and nl is key.low and nh is key.high: + return key + return BetweenKey(column=nc, low=nl, high=nh) + if isinstance(key, InKey): + # DEV-1475: ``InKey.values`` is a literal-only tuple — no + # transforms to attach a time key to. Only the LHS column path + # can carry a transform; rebuild only if it changed. + nc = _attach_time_keys(key.column, td_key=td_key) + if nc is key.column: + return key + return InKey(column=nc, values=key.values, negated=key.negated) + return key + + +def _partition_key_display(pk: ValueKey) -> str: + """Human-readable name of a rank ``partition_by`` key for error messages + (DEV-1497). Local refs surface as the bare leaf; joined refs keep the + dotted path.""" + if isinstance(pk, ColumnKey): + return ".".join([*pk.path, pk.leaf]) + if isinstance(pk, ColumnSqlKey): + return ".".join([*pk.path, pk.column_name]) + if isinstance(pk, TimeTruncKey): + return _partition_key_display(pk.column) + return str(pk) + + +def _row_key_path(key: ValueKey) -> tuple: + """Join path of a ROW value key (``()`` for local, non-empty for joined). + Unwraps a ``TimeTruncKey`` to its underlying column.""" + if isinstance(key, TimeTruncKey): + return _row_key_path(key.column) + return tuple(getattr(key, "path", ())) + + +def _find_unresolved_time_needing_op(key: ValueKey) -> Optional[str]: + """Return the op name of the first time-needing TransformKey reached + that has ``time_key is None``, or ``None`` if every time-needing + transform in the tree is resolved. + """ + if isinstance(key, TransformKey): + if key.op in _TIME_NEEDING_TRANSFORM_OPS and key.time_key is None: + return key.op + return _find_unresolved_time_needing_op(key.input) + if isinstance(key, ArithmeticKey): + for o in key.operands: + found = _find_unresolved_time_needing_op(o) + if found: + return found + return None + if isinstance(key, ScalarCallKey): + for a in key.args: + if isinstance( + a, (TransformKey, ArithmeticKey, ScalarCallKey, BetweenKey), + ): + found = _find_unresolved_time_needing_op(a) + if found: + return found + return None + if isinstance(key, BetweenKey): + for k in (key.column, key.low, key.high): + found = _find_unresolved_time_needing_op(k) + if found: + return found + return None + if isinstance(key, InKey): + # DEV-1475: only the LHS column can host a time-needing + # transform; the RHS values are literals. + return _find_unresolved_time_needing_op(key.column) + return None + + +# --------------------------------------------------------------------------- +# DEV-1714 Stage 10 — duration-windowed measures (``window='90d'``). +# --------------------------------------------------------------------------- + + +def _window_kwarg_of(key: ValueKey): + """The ``window`` kwarg value of an ``AggregateKey``, or ``None``. + + ``window`` is a globally reserved aggregation kwarg name (legacy parity — + the legacy enrichment pipeline popped it unconditionally before dispatch), so its + presence marks a windowed measure regardless of the aggregation. + """ + if isinstance(key, AggregateKey): + for k, v in key.kwargs: + if k == "window": + return v + return None + + +def _windowed_agg_keys(vk: ValueKey) -> list: + """Every windowed ``AggregateKey`` in ``vk``'s value-key tree.""" + return [k for k in walk_value_keys(vk) if _window_kwarg_of(k) is not None] + + +def _reject_unsupported_windowed_key(key: AggregateKey) -> None: + """Per-key guards shared by selected and filter/order-referenced windowed + aggregates: sum/avg-only (G1), string duration + compact syntax (G8), and + no cross-model source (G3). Raises with the pinned-message contract.""" + if key.agg not in ("sum", "avg"): + raise ValueError( + f"Aggregation parameter 'window' is only supported for sum and avg, " + f"not '{key.agg}'." + ) + window_val = _window_kwarg_of(key) + if not isinstance(window_val, str): + raise ValueError( + f"Window duration must be a compact duration string like '90d', got " + f"{window_val!r}. Use syntax like '1y2m3w5d6h7min8s'." + ) + parse_window_duration(window_val) # G8 — raises on empty / malformed + if getattr(key.source, "path", ()): # G3 + raise NotImplementedError( + "Windowed cross-model aggregates (e.g. customers.revenue:sum(" + "window='90d')) are not yet supported (DEV-1504)." + ) + + +def _guard_windowed_measures( # NOSONAR(S3776) — one cohesive ordered guard pass (G1→G8→G3→G4→G5→G7→G6→G2) over the original value-key trees; each branch is a distinct unsupported-shape rejection sharing the windowed-key scan, and splitting would scatter the precedence contract. + *, + measure_vks: list, + filter_vks: list, + order_vks: list, + active_td_key, +) -> dict: + """Reject unsupported windowed-measure shapes at plan time and return the + cleanly-SELECTED windowed ``AggregateKey``s (the ones that get a + ``WindowedAggregatePlan``) as an insertion-ordered mapping in measure + declaration order — so the emitted ``_wm_`` CTEs and combined-SELECT columns + are DETERMINISTIC (a set made the SQL output order vary across runs, which + breaks the SQL-text cache key of DEV-1587). + + Runs on the ORIGINAL declared-measure / filter / order value-key trees — + before projection interning would hide a transform / composite dependency + slot — so the transform (G4) and composite (G5) guards win over the + hidden-slot guard (G6). Precedence: G1 → G8 → G3 → G4 → G5 → G7 → G6 → G2. + """ + all_vks = [*measure_vks, *filter_vks, *order_vks] + if not any(_windowed_agg_keys(vk) for vk in all_vks): + return {} + + # G1 / G8 / G3 — per-key validation runs FIRST (documented precedence), so a + # windowed key with an invalid aggregation / malformed duration / cross-model + # source reports its specific error even when it is also wrapped in a + # transform (G4) or composite (G5). + for vk in all_vks: + for key in _windowed_agg_keys(vk): + _reject_unsupported_windowed_key(key) + + # G4 — a windowed measure cannot coexist with (or be the input of) any + # transform. Checked before the hidden-slot guard so + # ``cumsum(revenue:sum(window='90d'))`` reports 'transform', never 'selected'. + if any(isinstance(k, TransformKey) for vk in all_vks for k in walk_value_keys(vk)): + raise NotImplementedError( + "Windowed measures (window='…') combined with transforms are not yet " + "supported (DEV-1504). Compute the windowed measure in a separate " + "query stage." + ) + + # G5 — a top-level declared measure that IS a windowed AggregateKey is + # cleanly selected; a windowed key nested in an arithmetic / scalar composite + # measure is rejected. ``dict`` (not ``set``) preserves measure order; the + # value is the slot's ``hidden`` flag (DEV-1733) — False for a declared + # measure, True for an order-only target. + selected_windowed: dict = {} + for vk in measure_vks: + if not _windowed_agg_keys(vk): + continue + if _window_kwarg_of(vk) is not None: + selected_windowed.setdefault(vk, False) + else: + raise NotImplementedError( # G5 + "Windowed measures (window='…') inside arithmetic / composite / " + "scalar expressions are not yet supported (DEV-1504)." + ) + + # G7 (mixed) then G6 (hidden) for filter-referenced windowed measures. + for vk in filter_vks: + wkeys = _windowed_agg_keys(vk) + if not wkeys: + continue + has_plain_agg = any( + isinstance(k, AggregateKey) and _window_kwarg_of(k) is None + for k in walk_value_keys(vk) + ) + if has_plain_agg: + raise NotImplementedError( # G7 + "A single filter that mixes a windowed measure (window='…') with " + "a plain aggregate is not yet supported (DEV-1504)." + ) + if any(k not in selected_windowed for k in wkeys): + raise NotImplementedError( # G6 + "Filtering on a windowed measure (window='…') requires that " + "measure to also be selected (DEV-1504)." + ) + # DEV-1733 — order-only windowed targets. This IS a reachable shape (the + # pre-DEV-1733 comment here claimed otherwise): ``OrderItem`` canonicalises + # ``revenue:sum(window='90d')`` to the column name ``revenue_sum``, but + # ``OrderItem.raw_formula`` preserves the original text and the planner + # binds from it whenever the canonical name matches no declared measure. It + # used to fall through with no ``WindowedAggregatePlan``, materialising a + # PLAIN ``SUM`` in the base and ordering by it — the window silently gone. + # + # A windowed key referenced ONLY by ORDER BY is registered here as a HIDDEN + # plan (S-a top-level, S-b nested in a composite). Registering it after the + # measure loop means an also-declared key keeps ``hidden=False``. + for vk in order_vks: + for key in _windowed_agg_keys(vk): + selected_windowed.setdefault(key, True) + + # G2 — a windowed measure needs a resolvable time dimension. + if active_td_key is None: + raise ValueError( + "Windowed measure could not resolve its time dimension. Add a single " + "time_dimensions entry, or set main_time_dimension to select among " + "multiple time dimensions." + ) + return selected_windowed + + +def _windowed_grain_partition( + *, + row_slots: list, + active_td_slot_id, +) -> Tuple[list, list, list]: + """Split the PROJECTED (non-hidden) ROW slots into the ``_wm_`` grain roles. + + Returns ``(dim_slot_ids, other_td_slot_ids, grain_slot_ids)`` — plain + dimensions render as ``_w_dim_`` in the ``_src`` subquery, non-window + time dimensions as ``_w_td_``, and ``grain_slot_ids`` is the join-back + key order (dims, then the window TD, then the other TDs). + + HIDDEN row slots are excluded by design: the window buckets at the grain + the query actually projects, so an order-only (hidden) target must not + widen or narrow it. + """ + dim_slot_ids: list = [] + other_td_slot_ids: list = [] + for rs in row_slots: + if rs.hidden: + continue + if isinstance(rs.key, TimeTruncKey): + if rs.id != active_td_slot_id: + other_td_slot_ids.append(rs.id) + else: + dim_slot_ids.append(rs.id) + grain_slot_ids = ( + dim_slot_ids + + ([active_td_slot_id] if active_td_slot_id is not None else []) + + other_td_slot_ids + ) + return dim_slot_ids, other_td_slot_ids, grain_slot_ids + + +def _build_windowed_plans( + *, + selected_windowed: dict, + registry, + row_slots: list, + active_td_key, + active_td_slot_id, +) -> Tuple[list, set]: + """Build one ``WindowedAggregatePlan`` per selected windowed measure and + return ``(plans, windowed_slot_ids)``. The window time dimension is the + query's resolved main/active TD (same one first/last/time_shift use).""" + plans: list = [] + windowed_slot_ids: set = set() + if not selected_windowed: + return plans, windowed_slot_ids + + # CR#3 / G2 (post-projection): the window time dimension must be a SELECTED + # query time dimension (interned as a row slot so it becomes part of the + # bucket grain). A model ``default_time_dimension`` the query does not select + # resolves ``active_td_key`` (so the pre-projection G2 passes) but is never + # interned — ``active_td_slot_id`` is then None. Raise the G2 message rather + # than crash on the required ``window_time_dimension_slot_id: SlotId`` field. + if active_td_slot_id is None: + raise ValueError( + "Windowed measure could not resolve its time dimension. Add a single " + "time_dimensions entry, or set main_time_dimension to select among " + "multiple time dimensions." + ) + + dim_slot_ids, other_td_slot_ids, grain_slot_ids = _windowed_grain_partition( + row_slots=row_slots, active_td_slot_id=active_td_slot_id, + ) + + for key, is_hidden in selected_windowed.items(): + sid = registry.find_by_key(key) + if sid is None: + # CR#4: the guard pass already proved this is a cleanly-selected + # top-level windowed measure, so a missing slot is planner/projection + # drift — fail loudly rather than let the measure degrade to a plain + # (non-windowed) aggregate in the base (the silent-wrong-results mode + # the guards exist to prevent). + raise RuntimeError( + f"Windowed measure {key!r} was selected but has no projection " + f"slot; planner/projection drift (DEV-1714).", + ) + window_raw = _window_kwarg_of(key) + # DEV-1733: the slot is the authority on visibility — a key registered + # hidden by the order pass may still have been promoted to public by a + # declared measure sharing it (same key -> one slot). + slot = registry.get(sid) + hidden = bool(is_hidden and slot.hidden) + plans.append(WindowedAggregatePlan( + aggregate_slot_id=sid, + agg=key.agg, + window_raw=window_raw, + window_parts=parse_window_duration(window_raw), + window_time_dimension_slot_id=active_td_slot_id, + window_granularity=active_td_key.granularity, + dimension_slot_ids=dim_slot_ids, + other_time_dimension_slot_ids=other_td_slot_ids, + grain_slot_ids=grain_slot_ids, + hidden=hidden, + public_alias=None if hidden else slot.public_name, + )) + windowed_slot_ids.add(sid) + return plans, windowed_slot_ids + + +_RAW_ROW_FIX_HINT = ( + "Either remove the measure reference, or set " + "distinct_dimension_values=True (the default) to keep the " + "auto-aggregating behaviour." +) + + +def _iter_expr_children(node): + """Yield the child nodes of a parsed Mode-B expression node. + + Attribute-driven rather than type-driven so one walker covers every node + shape the typed parser emits. ``str`` / ``bool`` scalars are skipped — + they are leaf payloads (an operator name, a flag), never child nodes. + """ + for attr in ("input", "left", "right", "this", "operand"): + child = getattr(node, attr, None) + if child is not None and not isinstance(child, (str, bool)): + yield child + for attr in ("args", "operands", "kwargs"): + for item in getattr(node, attr, None) or (): + # kwargs come through as (name, value) pairs; take the value. + yield item[1] if isinstance(item, tuple) and len(item) == 2 else item + + +def _expr_has_measure_ref(node, *, measure_names: FrozenSet[str]) -> bool: + """True if a parsed Mode-B expression references an aggregation, a + transform, or a saved ``ModelMeasure`` by bare name. + + Structural walk over the typed parser's AST — the legacy check re-parsed + the raw text with the enrichment parsers, which no longer exist. + """ + from slayer.engine.syntax import AggCall, Ref, TransformCall + + if node is None: + return False + if isinstance(node, (AggCall, TransformCall)): + return True + if isinstance(node, Ref) and node.name in measure_names: + return True + return any( + _expr_has_measure_ref(child, measure_names=measure_names) + for child in _iter_expr_children(node) + ) + + +def _reject_measure_refs_for_raw_rows(*, query: SlayerQuery, scope) -> None: + """DEV-1543: with ``distinct_dimension_values=False`` the caller asked for + RAW ROWS, so no measure reference may appear in ``filters`` or ``order``. + + ``SlayerQuery``'s validator already rejects the model-free cases (a + non-empty ``measures``, no dimensions at all). This is the half that needs + the resolved model: a bare ``aov`` is only a measure reference if ``aov`` + is a saved ``ModelMeasure``. Unparseable text is left alone — the binder + raises on it downstream with a message tied to the original string. + """ + src = getattr(scope, "source_model", None) + measure_names: FrozenSet[str] = frozenset( + m.name for m in (getattr(src, "measures", None) or []) if m.name + ) + custom_agg_names: FrozenSet[str] = frozenset( + a.name for a in (getattr(src, "aggregations", None) or []) if a.name + ) + _reject_measure_refs_in_filters(query=query, measure_names=measure_names) + _reject_measure_refs_in_order( + query=query, + measure_names=measure_names, + custom_agg_names=custom_agg_names, + source_name=getattr(src, "name", None), + ) + + +def _reject_measure_refs_in_filters( + *, query: SlayerQuery, measure_names: FrozenSet[str], +) -> None: + """Filter half of :func:`_reject_measure_refs_for_raw_rows`.""" + for f in (query.filters or []): + if not isinstance(f, str): + continue + try: + parsed = parse_filter_expr(f) + except Exception: # noqa: BLE001 — binder reports parse errors properly + continue + if _expr_has_measure_ref(parsed, measure_names=measure_names): + raise DistinctDimensionValuesError( + f"distinct_dimension_values=False rejects measure references, " + f"but filter {f!r} contains one. {_RAW_ROW_FIX_HINT}" + ) + + +def _parse_order_formula(raw: str, *, custom_agg_names: FrozenSet[str]): + """Parse an ``OrderItem.raw_formula`` to a Mode-B AST, or ``None``. + + Function-style aggregations (``sum(amount)``) are not valid Mode-B; the + slack layer rewrites them to colon form, so do the same here — otherwise + this check would miss the aggregation and let the binder report a generic + "function not allowed" instead. + """ + try: + text = func_style_agg_to_colon(raw, custom_agg_names=custom_agg_names) + except Exception: # noqa: BLE001 — fall back to the original text + text = raw + try: + return parse_expr(text) + except Exception: # noqa: BLE001 — binder reports parse errors properly + return None + + +def _reject_measure_refs_in_order( + *, + query: SlayerQuery, + measure_names: FrozenSet[str], + custom_agg_names: FrozenSet[str], + source_name: Optional[str], +) -> None: + """ORDER BY half of :func:`_reject_measure_refs_for_raw_rows`.""" + for item in (query.order or []): + raw = getattr(item, "raw_formula", None) + if raw: + parsed = _parse_order_formula(raw, custom_agg_names=custom_agg_names) + if parsed is not None and _expr_has_measure_ref( + parsed, measure_names=measure_names, + ): + raise DistinctDimensionValuesError( + f"distinct_dimension_values=False rejects measure " + f"references, but order item {raw!r} contains one. " + f"{_RAW_ROW_FIX_HINT}" + ) + name = getattr(getattr(item, "column", None), "name", None) + if name and name in measure_names: + raise DistinctDimensionValuesError( + f"distinct_dimension_values=False rejects measure references, " + f"but order item {name!r} resolves to a saved measure on " + f"{source_name or 'the source model'!r}. " + f"{_RAW_ROW_FIX_HINT}" + ) + + +def plan_query( # NOSONAR(S3776) — planner entry-point dispatcher. The DEV-1503 addition is a small trigger-predicate branch + a kwarg pass-through; the function's pre-existing complexity is owned by the multi-stage scope / bundle / projection / filter-routing wiring it orchestrates and is tracked as a separate refactor. + *, + query: SlayerQuery, + bundle: ResolvedSourceBundle, + scope: Optional[Union[ModelScope, StageSchema]] = None, + cross_model_planner: Optional[CrossModelPlanner] = None, + stage_schemas: Optional[Dict[str, StageSchema]] = None, + disable_host_rooted_isolation: bool = False, +) -> PlannedQuery: + """Compile one ``SlayerQuery`` into a typed ``PlannedQuery``. + + ``scope`` defaults to a ``ModelScope`` over ``bundle.source_model``; + pass an explicit ``StageSchema`` to bind against an upstream stage. + ``stage_schemas`` is a name → StageSchema map used by + ``plan_stages`` to wire multi-stage references. + + ``disable_host_rooted_isolation`` (DEV-1503; renamed and widened by + DEV-1709) suppresses the HOST-ROOTED half of the Law-3 trigger — the + isolation of a LOCAL aggregate whose ``Column.filter`` or any other + input (source ``Column.sql``, positional args, kwargs) crosses a join. + It never affects target-rooted isolation (``source.path`` non-empty). + Threaded through ``subplan_builder`` whenever a cross-model strategy + recurses for a host-rooted (or target-rooted) nested sub-plan, so the + sub-plan's same crossing measure is rendered inline (not infinitely + re-isolated) and so re-rooting of a genuine cross-model aggregate + doesn't redundantly isolate the target's own filter joins. + """ + stage_schemas = stage_schemas or {} + cross_model_planner = ( + cross_model_planner or IsolatedCteCrossModelPlanner() + ) + + if scope is None: + source = query.source_model + if isinstance(source, str) and source in stage_schemas: + scope = stage_schemas[source] + else: + scope = ModelScope(source_model=bundle.source_model) + + # The generator must render this stage's FROM / joins against the SAME + # model the binder used. For a ModelScope that's the (possibly overlaid / + # synthetic) host; for a StageSchema chain stage it's None (the generator + # builds a synthetic model from the upstream schema). + render_source_model = ( + scope.source_model if isinstance(scope, ModelScope) else None + ) + + # DEV-1543: raw-rows mode rejects measure references in filters / order. + # Runs BEFORE binding so the targeted, actionable error wins over the + # binder's generic "cannot resolve reference" / "function not allowed" + # for a saved-measure or function-style-aggregate reference. + if query.distinct_dimension_values is False: + _reject_measure_refs_for_raw_rows(query=query, scope=scope) + + # Downstream stages bind against a flat StageSchema — ``__`` is legal + # in their refs (the upstream's flattened multi-hop aliases); model- + # scoped stages keep the P1 rejection. + flat_scope = isinstance(scope, StageSchema) + + declared_measures = _declared_measures_from_query( + query=query, scope=scope, bundle=bundle, + ) + + # DEV-1450 stage 7b.8 — alias lookup for ORDER BY resolution. + # A user-supplied order column may reference the declared measure + # by its public name (user-supplied ``name``), declared name + # (canonical OR user), or canonical alias. The order pass below + # checks this map BEFORE falling back to ``bind_expr`` so refs to + # aggregate aliases like ``amount_sum`` resolve through the + # projection registry rather than against model scope (where they + # don't exist as columns). + declared_alias_to_bound: Dict[str, BinderBoundExpr] = {} + for dm in declared_measures: + for alias in (dm.public_name, dm.declared_name, dm.canonical_alias): + if alias is not None: + declared_alias_to_bound.setdefault(alias, dm.bound) + + # DEV-1450 stage 7b.15 (DEV-1445, C5): declared-MEASURE aliases a + # filter may reference by name. A filter ``rev >= 100`` for a measure + # declared ``{"formula": "customers.revenue:sum", "name": "rev"}`` + # interns ``rev`` onto the cross-model aggregate slot rather than + # failing to resolve against the model columns; the dotted/colon form + # already interns structurally, so both forms share one slot (P2/P4). + # + # Only MEASURE aliases enter this map — never dimension / time- + # dimension names. A time dimension's declared name IS its raw column + # (e.g. ``created_at``), so a WHERE filter ``created_at <= '...'`` + # (such as the one ``snap_to_whole_periods`` injects) must resolve to + # the raw column, not to the truncated dimension slot. ``declared_ + # measures`` is built in dim → time-dim → measure order, so the + # measure entries are the tail past the dim/time-dim prefix. + n_dims = len(query.dimensions or []) + n_tds = len(query.time_dimensions or []) + filter_alias_map: Dict[str, ValueKey] = {} + for dm in declared_measures[n_dims + n_tds:]: + for alias in (dm.public_name, dm.declared_name, dm.canonical_alias): + if alias is not None: + filter_alias_map.setdefault(alias, dm.bound.value_key) + + # DEV-1450 stage 7b.9 — filter list construction in legacy WHERE + # order: date_range filters first, then SlayerModel.filters + # (Mode-A SQL), then user query filters (Mode-B DSL). The legacy + # generator emits date_range BEFORE iterating ``enriched.filters`` + # (slayer/sql/generator.py:2527 vs :2540), and ``enriched.filters`` + # itself is model filters then query filters (enrichment.py:1192). + # + # ``bound_filters`` carries the typed-BoundFilter entries (date_range + # + query filters) for the cross-model routing and projection + # planner passes. Model filters bypass ``bound_filters`` since + # they're Mode-A SQL text without a typed value-key — they're + # appended directly to ``filters_by_phase`` between the two + # bound-filter buckets. + bound_filters: List[BoundFilter] = [] + # Parallel to bound_filters — original query-filter text for user + # filters (None for date_range bounds, which are synthesized from + # TimeDimension.date_range and have no caller-visible source string). + # Wired into HostFilterRouting.text below so cross_model_planner can + # recover ROW-phase user-filter text WITHOUT slicing host_query.filters + # (which has not been deduped, unlike bound_filters) — CR PR #153 + # thread r3350000254. + bound_filter_texts: List[Optional[str]] = [] + text_filter_entries: List[FilterPhase] = [] + + # 1. date_range filters (one per TD with a 2-element date_range) + for td in (query.time_dimensions or []): + if not td.date_range or len(td.date_range) != 2: + continue + if not isinstance(scope, ModelScope): + continue + bf = _build_date_range_filter(td=td, scope=scope, bundle=bundle) + bound_filters.append(bf) + bound_filter_texts.append(None) + n_date_range = len(bound_filters) + + # 2. SlayerModel.filters — Mode-A SQL, always-applied WHERE. + if isinstance(scope, ModelScope) and scope.source_model is not None: + for j, mf in enumerate(scope.source_model.filters or []): + text_filter_entries.append(_validate_model_filter( + mf=mf, idx=j, model=scope.source_model, + )) + + # 3. user query filters (Mode-B DSL). + # + # DEV-1450 stage 7b.15 (DEV-1445): two filter strings that bind to the + # same structural ``ValueKey`` are one predicate (P2). The alias and + # dotted/colon forms of a renamed cross-model aggregate ref + # (``rev >= 100`` and ``customers.revenue:sum >= 100``) intern onto the + # same slot, so emitting both would duplicate the HAVING clause — + # dedupe by bound key, keeping first occurrence. + for f in (query.filters or []): + if not isinstance(f, str): + continue + bf = bind_filter( + parsed=parse_filter_expr(f, allow_dunder=flat_scope), + scope=scope, + bundle=bundle, + alias_map=filter_alias_map, + ) + if any(existing.value_key == bf.value_key for existing in bound_filters): + continue + bound_filters.append(bf) + bound_filter_texts.append(f) + + order_specs = [] + # Host identity for the qualifier check below — the source model for a + # ``ModelScope``, the stage relation name (``s1``) for a downstream + # ``StageSchema`` (so a self-qualified ``s1.metric`` order stays host-local, + # Codex). Same resolution ``_host_model_name`` uses everywhere else. + _order_host_name = _host_model_name(scope) + for o in (query.order or []): + col_name = o.column.name + full_name = o.column.full_name + # DEV-1733: a placeholder ColumnRef means the item is an EXPRESSION, + # not a column reference — bind ``raw_formula`` and skip the + # declared-alias lookups below (which could otherwise match a real + # model column that happens to share the sentinel's name). BOTH the + # sentinel AND a captured ``raw_formula`` are required, so a model with + # a genuine ``_expr_pending`` column, or a hand-built / deserialized + # ``OrderItem``, still resolves through the normal path. + if col_name in ORDER_PLACEHOLDER_NAMES and o.raw_formula: + order_specs.append(OrderSpec( + bound=bind_expr( + parsed=parse_expr(o.raw_formula, allow_dunder=flat_scope), + scope=scope, + bundle=bundle, + ), + direction=o.direction, + )) + continue + # An order ref qualified with a FOREIGN model (``owners.status`` when + # the host is ``orders``) must not resolve to a same-named local column + # via the bare-leaf shortcut — otherwise a joined sort key silently + # binds to the local column and sorts by the wrong field (Codex). The + # bare-name lookups below apply only to unqualified refs or refs + # qualified with the host itself; a foreign-qualified ref falls through + # to the dotted/flattened/`bind_expr` paths, where a truly-joined ref + # is then rejected by the plan-time order validation. + _order_qualifier = getattr(o.column, "model", None) + _order_host_local = ( + _order_qualifier is None or _order_qualifier == _order_host_name + ) + # Prefer declared-measure alias resolution over model-scope + # binding (DEV-1450 stage 7b.8 — gap fix): aggregate canonical + # aliases like ``amount_sum`` are not columns on the model, so + # ``bind_expr`` would raise. The alias map covers user-supplied + # ``name``, canonical alias, and the declared name itself. + # + # DEV-1450 stage 7b.15 (DEV-1443/1445): a cross-model order key + # written ``customers.revenue:sum`` is coerced by ``OrderItem`` + # to ColumnRef(model="customers", name="revenue_sum"), so the + # leaf alone (``col_name``) never matches the declared canonical + # ``customers.revenue_sum``. Try the full dotted form too, then + # fall back to binding the preserved colon/path ``raw_formula`` + # so the order key interns onto the same cross-model aggregate + # slot (P2/P4) rather than raising. + if _order_host_local and col_name in declared_alias_to_bound: + bo = declared_alias_to_bound[col_name] + elif full_name in declared_alias_to_bound: + bo = declared_alias_to_bound[full_name] + elif _flatten_dotted(full_name) in declared_alias_to_bound: + # A joined dimension / time dimension is declared under its + # flattened ``__`` form (``stores.opened_at`` → + # ``stores__opened_at``; DEV-1449 / C4). An ORDER BY entry + # written in dotted form must intern onto that same declared + # slot rather than binding the raw column as a fresh slot. + bo = declared_alias_to_bound[_flatten_dotted(full_name)] + elif _order_host_local and f"_{col_name}" in declared_alias_to_bound: + # ``*:count`` surfaces as the alias ``_count`` (the ``*`` is + # dropped, the leading ``_`` kept as a marker); users naturally + # order by the bare ``count``. Mirror the legacy + # ``_resolve_order_column`` ``_name`` fallback. + bo = declared_alias_to_bound[f"_{col_name}"] + elif o.raw_formula: + bo = bind_expr( + parsed=parse_expr(o.raw_formula, allow_dunder=flat_scope), + scope=scope, + bundle=bundle, + ) + else: + # Bind the FULL reference (``customers.region``), not just the + # leaf — otherwise a structured dotted ORDER ColumnRef without a + # raw_formula rebinds as ``region`` and hits the wrong host + # column or fails as ambiguous (CR). + bo = bind_expr( + parsed=parse_expr(full_name, allow_dunder=flat_scope), + scope=scope, + bundle=bundle, + ) + order_specs.append(OrderSpec(bound=bo, direction=o.direction)) + + # Stage 7b.10 — attach the active TD as ``time_key`` on every + # time-needing TransformKey (cumsum / lag / lead / first / last / + # time_shift / consecutive_periods / change / change_pct) whose + # binder-output left ``time_key`` as ``None``. Closes the 7b.4 + # carry-over gap: ``_bind_transform`` does not have query / scope + # context to resolve the TD, so the planner does it here after all + # binding completes. Validation mirrors legacy + # ``enrichment.py:564-569`` -- any time-needing transform with no + # resolvable TD raises with the legacy phrase. + active_td_key: Optional[TimeTruncKey] = None + if isinstance(scope, ModelScope) and scope.source_model is not None: + active_td = _resolve_main_time_dimension( + query=query, model=scope.source_model, + ) + if active_td is not None: + active_td_bound = bind_time_dimension( + td=active_td, scope=scope, bundle=bundle, + ) + atd_key = active_td_bound.value_key + assert isinstance(atd_key, TimeTruncKey) + active_td_key = atd_key + + if active_td_key is not None: + declared_measures = [ + DeclaredMeasure( + bound=BinderBoundExpr( + value_key=_attach_time_keys( + dm.bound.value_key, td_key=active_td_key, + ), + ), + declared_name=dm.declared_name, + public_name=dm.public_name, + label=dm.label, + canonical_alias=dm.canonical_alias, + type=dm.type, + format=dm.format, + description=dm.description, + ) + for dm in declared_measures + ] + bound_filters = [ + BoundFilter( + value_key=_attach_time_keys( + bf.value_key, td_key=active_td_key, + ), + phase=bf.phase, + referenced_keys=tuple( + walk_value_keys( + _attach_time_keys( + bf.value_key, td_key=active_td_key, + ), + ), + ), + ) + for bf in bound_filters + ] + order_specs = [ + OrderSpec( + bound=BinderBoundExpr( + value_key=_attach_time_keys( + spec.bound.value_key, td_key=active_td_key, + ), + ), + direction=spec.direction, + ) + for spec in order_specs + ] + + # Validation: any time-needing transform that still has + # ``time_key=None`` after patching means there was no resolvable TD. + for bucket in ( + [dm.bound.value_key for dm in declared_measures], + [bf.value_key for bf in bound_filters], + [spec.bound.value_key for spec in order_specs], + ): + for vk in bucket: + op = _find_unresolved_time_needing_op(vk) + if op is not None: + raise ValueError( + f"Transform '{op}' requires an unambiguous time " + f"dimension. Add a single time_dimensions entry, or " + f"set main_time_dimension to select among multiple " + f"time dimensions." + ) + + # Sugar lowering for ``change`` / ``change_pct`` runs AFTER the + # patching pass so the desugared ``time_shift`` inherits the patched + # ``time_key`` (DEV-1446 identity preservation still holds — the + # inner AggregateKey instance is not rebuilt by lowering). + declared_measures = [ + DeclaredMeasure( + bound=BinderBoundExpr( + value_key=lower_sugar_transforms(dm.bound.value_key), + ), + declared_name=dm.declared_name, + public_name=dm.public_name, + label=dm.label, + canonical_alias=dm.canonical_alias, + type=dm.type, + format=dm.format, + description=dm.description, + ) + for dm in declared_measures + ] + bound_filters = [ + BoundFilter( + value_key=lower_sugar_transforms(bf.value_key), + phase=bf.phase, + referenced_keys=tuple( + walk_value_keys(lower_sugar_transforms(bf.value_key)), + ), + ) + for bf in bound_filters + ] + order_specs = [ + OrderSpec( + bound=BinderBoundExpr( + value_key=lower_sugar_transforms(spec.bound.value_key), + ), + direction=spec.direction, + ) + for spec in order_specs + ] + + # DEV-1497: validate that every rank-family ``partition_by`` column resolves + # to a query dimension / time-dimension, and rewrite a time-dimension source + # column to its truncated-bucket ``TimeTruncKey`` (partition by the bucket, + # not the raw timestamp — which would silently widen the grain). Runs BEFORE + # interning so a rewritten key never leaves a stale slot behind (identity is + # only touched on the rewritten rank transform). + _dim_dms = declared_measures[:n_dims] + _td_dms = declared_measures[n_dims:n_dims + n_tds] + _dim_key_set = {dm.bound.value_key for dm in _dim_dms} + # A source column carrying two time-dimension granularities (``created_at`` + # at both month and day) maps to two distinct ``TimeTruncKey`` buckets — a + # bare ``partition_by=created_at`` is then ambiguous, so track those columns + # and reject rather than silently pick whichever bucket comes last. + _td_by_source: Dict[ValueKey, TimeTruncKey] = {} + _td_ambiguous_sources: set = set() + for dm in _td_dms: + vk = dm.bound.value_key + if not isinstance(vk, TimeTruncKey): + continue + # Ambiguous only when the SAME source column already mapped to a + # DIFFERENT bucket (a different granularity) — two identical + # ``created_at:month`` declarations resolve to one bucket, not a clash. + if vk.column in _td_by_source and _td_by_source[vk.column] != vk: + _td_ambiguous_sources.add(vk.column) + _td_by_source[vk.column] = vk + _td_key_set = set(_td_by_source.values()) + _available_dims = [dm.declared_name for dm in (*_dim_dms, *_td_dms)] + + def _validate_partition_keys(tk: TransformKey) -> frozenset: + new_pks = [] + for pk in tk.partition_keys: + if pk in _dim_key_set or pk in _td_key_set: + new_pks.append(pk) # already a query dim / td bucket + elif pk in _td_ambiguous_sources: + raise ValueError( + f"Transform '{tk.op}': partition_by column " + f"'{_partition_key_display(pk)}' is ambiguous — it is a " + f"time dimension at multiple granularities. Partition by a " + f"single query dimension instead." + ) + elif pk in _td_by_source: + new_pks.append(_td_by_source[pk]) # td source col -> bucket + else: + raise ValueError( + f"Transform '{tk.op}': partition_by column " + f"'{_partition_key_display(pk)}' is not a query dimension. " + f"Add it to dimensions/time_dimensions, or choose one of: " + f"{', '.join(_available_dims) or '(none)'}." + ) + return frozenset(new_pks) + + def _rw(vk: ValueKey) -> ValueKey: + return rewrite_rank_partition_keys(vk, rewrite_fn=_validate_partition_keys) + + declared_measures = [ + DeclaredMeasure( + bound=BinderBoundExpr(value_key=_rw(dm.bound.value_key)), + declared_name=dm.declared_name, + public_name=dm.public_name, + label=dm.label, + canonical_alias=dm.canonical_alias, + type=dm.type, + format=dm.format, + description=dm.description, + ) + for dm in declared_measures + ] + _rewritten_filters = [] + for bf in bound_filters: + bf_vk = _rw(bf.value_key) + _rewritten_filters.append(BoundFilter( + value_key=bf_vk, + phase=bf.phase, + referenced_keys=tuple(walk_value_keys(bf_vk)), + )) + bound_filters = _rewritten_filters + order_specs = [ + OrderSpec( + bound=BinderBoundExpr(value_key=_rw(spec.bound.value_key)), + direction=spec.direction, + ) + for spec in order_specs + ] + + source_col_names = _source_column_names(scope) + host_model_name = _host_model_name(scope) + + # DEV-1714 Stage 10 — windowed-measure guards on the ORIGINAL (pre- + # projection) value-key trees. Raises on unsupported shapes (non-sum/avg, + # no time dim, cross-model, transform, composite, hidden, mixed, malformed + # duration); returns the set of cleanly-selected windowed AggregateKeys. + selected_windowed = _guard_windowed_measures( + measure_vks=[dm.bound.value_key for dm in declared_measures], + filter_vks=[bf.value_key for bf in bound_filters], + order_vks=[sp.bound.value_key for sp in order_specs], + active_td_key=active_td_key, + ) + + projection = ProjectionPlanner().plan( + measures=declared_measures, + filters=bound_filters, + order=order_specs, + source_column_names=source_col_names, + host_model_name=host_model_name, + ) + + row_slots, agg_slots, combined_slots = _bucket_slots( + projection.registry.slots, + ) + + # DEV-1543: ``distinct_dimension_values=False`` asks for RAW ROWS, so no + # measure reference may appear anywhere in the query. ``SlayerQuery``'s + # validator already rejects the cheap structural cases (a non-empty + # ``measures``, or no dimensions at all) without needing a model; the + # remaining references hide in ``filters`` and ``order``, which only + # resolve once bound. + # + # The typed pipeline checks this structurally instead of re-parsing the + # filter/order TEXT the way the legacy stack did: after binding, ANY + # aggregate-phase slot in a raw-rows query can only have come from a + # filter or an order item, since measures were rejected upstream. Without + # this the query silently aggregates — a ``filters=["amount:sum > 100"]`` + # raw-rows query materialised a hidden aggregate and emitted + # ``GROUP BY ... HAVING ...``, the exact auto-aggregation the flag exists + # to turn off. + if query.distinct_dimension_values is False and agg_slots: + offender = _canonical_name(agg_slots[0].key) + raise DistinctDimensionValuesError( + f"distinct_dimension_values=False rejects measure references, but " + f"this query references the aggregation {offender!r} in its " + f"filters or order. Either remove the measure reference, or set " + f"distinct_dimension_values=True (the default) to keep the " + f"auto-aggregating behaviour." + ) + + # DEV-1714 Stage 10 — build one WindowedAggregatePlan per selected windowed + # measure. The window time dimension is the query's resolved active TD. + active_td_slot_id = ( + projection.registry.find_by_key(active_td_key) + if active_td_key is not None + else None + ) + windowed_plans, windowed_slot_ids = _build_windowed_plans( + selected_windowed=selected_windowed, + registry=projection.registry, + row_slots=row_slots, + active_td_key=active_td_key, + active_td_slot_id=active_td_slot_id, + ) + + # DEV-1714 (Codex round 5): a filter that references a windowed measure is + # reclassified WHOLE to Phase.POST (outer WHERE on the joined-back column). + # It must therefore reference ONLY windowed measures (+ literals). Mixing a + # windowed predicate with a row column or a plain aggregate in ONE filter + # can't be cleanly split — the windowed part is POST while the row part is a + # pre-aggregation WHERE — and would emit an outer-WHERE reference to an + # unprojected ``_base`` column. Reject it (the pre-projection G7 already + # catches the windowed+plain-aggregate half with a specific message; this + # also covers windowed+row-column, which G7's aggregate-only scan misses). + if windowed_slot_ids: + for bf in bound_filters: + refs = filter_referenced_slot_ids(bf, projection.registry) + if (refs & windowed_slot_ids) and (refs - windowed_slot_ids): + raise NotImplementedError( + "A single filter that mixes a windowed measure (window='…') " + "with another predicate (a row column or a plain aggregate) " + "is not yet supported (DEV-1504). Put them in separate " + "filters." + ) + + # DEV-1712 (Law 2) / DEV-1703 Phase 1: plan-time classification of every + # ORDER BY target that is not a declared/public slot. + # + # ONE RULE: an order-only ref resolves exactly like a filter ref. Law 1 + # pulls whatever joins it crosses into the scope that owns its rows — even + # when ORDER BY is the only thing referencing it — and the ref is then + # emitted in whatever form that scope makes legal: + # * order-only AGGREGATE (local or cross-model) -> hidden materialised + # slot, ordered by its alias (unchanged); + # * row column, UNGROUPED query -> split ``orders.created_at`` / + # ``customers__regions.name`` emission in the generator's + # ``_apply_order_limit_from_planned`` (the row IS the grain, so the + # bare reference is legal); + # * LOCAL row column, GROUPED query -> the bare reference is NOT legal + # (the column is not in GROUP BY), so it materialises as a hidden + # ``:max`` aggregate slot and the order entry is repointed at it. + # MAX is order-preserving per group and portable across every Tier-1 + # dialect; + # * JOINED row column, GROUPED query -> still + # ``UnresolvableOrderColumnError``. A host-rooted MAX over a joined + # column is not expressible today: an ``AggregateKey`` with a + # non-empty ``source.path`` always routes to a TARGET-rooted CTE + # (Law 3), which for a host-grain sort key degenerates to a scalar + # CROSS JOIN — every group would get the same global value and the + # sort would silently do nothing. Failing loudly beats sorting by a + # constant. Full support (host-rooted crossing MAX) is DEV-1735; + # * transform / composite -> materialised as a hidden slot and ordered at + # the outer wrap (DEV-1733), same Law-2 discipline as aggregates. + # + # The hidden MAX is interned post-bind, so the bind-time aggregation gate + # (PK columns, ``allowed_aggregations``, per-type defaults) deliberately + # does not apply: the caller asked to SORT by a column, not to aggregate + # it, and a sort must not fail because ``max`` is not whitelisted on the + # column being sorted. + _has_grouping = bool(agg_slots) or ( + bool(query.dimensions or query.time_dimensions) + and query.distinct_dimension_values + ) + # ORDER BY targets rewritten to a hidden aggregate: original key -> MAX key. + order_key_remap: Dict[ValueKey, ValueKey] = {} + for spec in order_specs: + okey = spec.bound.value_key + osid = projection.registry.find_by_key(okey) + if osid is not None and not projection.registry.get(osid).hidden: + continue # declared / projected output — orders on a real column + if isinstance(okey, AggregateKey): + continue # hidden aggregate (local base or cross-model CTE) + if isinstance(okey, (ColumnKey, ColumnSqlKey, TimeTruncKey)): + if not _has_grouping: + continue # raw-rows query -> split emission, no wrap needed + path = _row_key_path(okey) + if path: + # ``UnresolvableOrderColumnError`` formats ``qualifier.column``; + # pass the bare leaf as ``column`` and the joined path as the + # qualifier so the message reads ``customers.regions.name`` and + # not a duplicated ``customers.customers.regions.name``. + disp = _partition_key_display(okey) + raise UnresolvableOrderColumnError( + column=disp.rsplit(".", 1)[-1], qualifier=".".join(path), + ) + # A TimeTruncKey is not a legal aggregate source; wrap its + # underlying column instead. DATE_TRUNC is monotonic + # non-decreasing, so MAX(col) and MAX(trunc(col)) sort a group + # identically. + src = okey.column if isinstance(okey, TimeTruncKey) else okey + max_key = AggregateKey(source=src, agg="max") + if projection.registry.find_by_key(max_key) is None: + projection.registry.intern( + key=max_key, + declared_name=_canonical_name(max_key), + hidden=True, + phase=max_key.phase, + ) + order_key_remap[okey] = max_key + # DEV-1733: TransformKey / ArithmeticKey / ScalarCallKey — an inline + # transform or composite expression referenced only in ORDER BY. These + # materialise as hidden slots (a step CTE on the transform path, a + # trimmed base-SELECT column on the no-transform path, an inline + # combined-SELECT term when an operand is cross-model or windowed) and + # order at the outer wrap. Stage 8 rejected them here; nothing left to + # reject. + + # Re-bucket: the pass above may have interned hidden ``:max`` order slots, + # which must reach the PlannedQuery's aggregate bucket (and hence the + # generator's slot maps) like any other aggregate. + if order_key_remap: + row_slots, agg_slots, combined_slots = _bucket_slots( + projection.registry.slots, + ) + + # Build filters_by_phase in legacy WHERE order: + # 1. date_range bound filters (bound_filters[:n_date_range]) + # 2. model.filters (text_filter_entries) + # 3. user query bound filters (bound_filters[n_date_range:]) + # bound_filter_ids preserves the mapping back to bound_filters for + # the cross-model routing pass that follows (text_filter_entries + # are excluded — model filters never feed cross-model routing). + # DEV-1714 Stage 10 — a filter referencing a windowed slot is reclassified + # to Phase.POST: the windowed value is computed in the ``_wm_`` CTE and + # joined back, so the predicate must apply on the combined SELECT (outer + # WHERE), never as a HAVING on the plain base aggregate. + def _windowed_phase(bf: BoundFilter) -> Phase: + if windowed_slot_ids and ( + filter_referenced_slot_ids(bf, projection.registry) & windowed_slot_ids + ): + return Phase.POST + return bf.phase + + filters_by_phase: List[FilterPhase] = [] + bound_filter_ids: List[str] = [] + for i, bf in enumerate(bound_filters[:n_date_range]): + fid = f"f{i}" + filters_by_phase.append( + FilterPhase( + id=fid, phase=_windowed_phase(bf), text=None, + expression=PlannedBoundExpr(value_key=bf.value_key), + ), + ) + bound_filter_ids.append(fid) + filters_by_phase.extend(text_filter_entries) + for i, bf in enumerate(bound_filters[n_date_range:], start=n_date_range): + fid = f"f{i}" + filters_by_phase.append( + FilterPhase( + id=fid, phase=_windowed_phase(bf), text=None, + expression=PlannedBoundExpr(value_key=bf.value_key), + ), + ) + bound_filter_ids.append(fid) + # Stage 7b.5 — cross-model planner wiring. For every aggregate slot + # whose source carries a non-empty join path (cross-model agg-ref + # like ``customers.revenue:sum``), invoke the cross_model_planner + # to produce a CrossModelAggregatePlan with explicit WHERE/HAVING/ + # target_model_filters routes. HostFilterRouting records carry the + # post-projection slot ids each filter references (via + # filter_referenced_slot_ids — Codex HIGH #3/#4 fold-in). + # host_filter_routings only carries entries that have a typed + # BoundFilter (date_range + user filters). Model.filters (text-only) + # are always row-phase host-local WHERE and never need to be routed + # to a cross-model CTE — they're skipped here. + host_filter_routings: List[HostFilterRouting] = [] + for fid, bf, ftext in zip( + bound_filter_ids, bound_filters, bound_filter_texts, + ): + host_filter_routings.append(HostFilterRouting( + filter_id=fid, + phase=bf.phase, + referenced_slot_ids=sorted(filter_referenced_slot_ids( + bf, projection.registry, + )), + text=ftext, + )) + + cross_model_plans = [] + host_slots_for_classifier = projection.registry.slots + for slot in agg_slots: + # DEV-1714 Stage 10 — a windowed slot renders via its own ``_wm_`` CTE + # (host-rooted range join), never a cross-model ``_cm_`` CTE, even when + # its ``Column.filter`` crosses a join (which would otherwise trip the + # host-rooted isolation trigger below). + if slot.id in windowed_slot_ids: + continue + key = slot.key + if not isinstance(key, AggregateKey): + continue + agg_path = getattr(key.source, "path", ()) + # DEV-1503 / DEV-1709 — Law-3 trigger predicate. Invoke the + # cross-model planner when the aggregate's source carries a + # non-empty join path (target-rooted, existing behaviour) OR when + # ANY other input of a LOCAL aggregate crosses a join (host-rooted + # isolation): ``Column.filter`` (typed ``referenced_join_paths`` + # from binder time — DEV-1503), source ``Column.sql``, positional + # args incl. the explicit first/last time arg, kwargs (column + # refs, user template fragments, and non-overridden model-default + # ``AggregationParam`` fragments) — DEV-1709's widened trigger, + # computed plan-time by ``compute_aggregate_input_join_paths``. + has_crossing_filter = ( + key.column_filter_key is not None + and bool(key.column_filter_key.referenced_join_paths) + ) + has_crossing_input = ( + not disable_host_rooted_isolation + and not agg_path + and ( + has_crossing_filter + or bool(compute_aggregate_input_join_paths( + key=key, + anchor_model=bundle.source_model, + anchor_relation=( + bundle.source_model.name + if bundle.source_model is not None else "" + ), + bundle=bundle, + )) + ) + ) + if not agg_path and not has_crossing_input: + continue + # DEV-1450 #2: re-rooting (C1) is owned by the strategy. We hand it + # the host query, the public projection, and a sub-plan builder so it + # can compile a nested re-rooted PlannedQuery when the host carries + # dimensions / filters reachable only through the TARGET's join graph; + # otherwise it returns the forward plan unchanged. The builder is the + # same ``plan_query`` recursion the post-hoc pass used, injected here + # so cross_model_planner.py needn't import stage_planner. + # + # DEV-1503 / DEV-1709 — the subplan_builder ALWAYS suppresses + # host-rooted isolation: the host-rooted sub-plan contains the same + # crossing measure and would otherwise recurse infinitely; for the + # existing cross-model re-rooting case, the sub-plan's target-rooted + # local aggregate would redundantly isolate its own crossing inputs + # (already handled by the surrounding cross-model CTE). Inside the + # sub-plan, crossing inputs render INLINE (base-pull) — legal there + # because the CTE is the aggregate's own scope. + reroot_enabled = ( + isinstance(scope, ModelScope) and scope.source_model is not None + ) + plan = cross_model_planner.plan( + aggregate_slot_id=slot.id, + aggregate_key=key, + bundle=bundle, + host_slots=host_slots_for_classifier, + host_filters=host_filter_routings, + public_alias=slot.public_name, + hidden=slot.hidden, + host_query=query if reroot_enabled else None, + public_projection=( + projection.public_projection if reroot_enabled else None + ), + subplan_builder=( + (lambda q, b: plan_query( + query=q, bundle=b, cross_model_planner=cross_model_planner, + disable_host_rooted_isolation=True, + )) + if reroot_enabled else None + ), + ) + cross_model_plans.append(plan) + + order_entries = [] + for spec in order_specs: + # A grouped row-column sort key was rewritten to a hidden ``:max`` + # aggregate above; order on that slot, not the bare row key. + okey = order_key_remap.get(spec.bound.value_key, spec.bound.value_key) + sid = projection.registry.find_by_key(okey) + if sid is None: + # DEV-1733: an order target that reached here without a slot would + # be SILENTLY DROPPED — the query runs unsorted and returns wrong + # rows with no error. That was the original `change(...)` / + # scalar-call bug, and the entry-point relaxation makes new key + # shapes reachable (e.g. a top-level `IN` / `BETWEEN` predicate, + # which `_iter_slot_deps` treats as WHERE-inlined and never slots). + # Fail loudly for ANY unslotted shape rather than enumerating them, + # so this whole bug class cannot come back. + raise ValueError( + f"ORDER BY expression is not supported: " + f"{type(spec.bound.value_key).__name__} has no materialisable " + f"slot. Order by an aggregate, a transform, a composite " + f"arithmetic / scalar expression, a dimension, or declare the " + f"expression as a measure and order by its name." + ) + order_entries.append( + OrderEntry(slot_id=sid, direction=spec.direction), + ) + + transform_layers = _emit_transform_layers(slots=projection.registry.slots) + stage_schema = _emit_stage_schema( + query=query, projection=projection, + ) + source_relation = ( + query.source_model + if isinstance(query.source_model, str) + else host_model_name + ) + + # Stage 7b.10 — the active TD's slot id (``active_td_slot_id``) is resolved + # right after projection above so the windowed-plan builder can use it. + + # DEV-1732 — the frame-bound column set: raw columns of this stage's + # NON-HIDDEN time dimensions. Computed once and carried on the plan so the + # windowed ``_src`` path (below) and the generator's ``time_shift`` + # shifted-CTE path read the SAME set. + frame_bound_columns = _frame_bound_columns(row_slots=row_slots) + + # DEV-1714 Stage 10 / DEV-1732 — the ``_wm_`` ``_src`` scope inherits + # WHERE-phase row filters (model + user) MINUS their frame bounds: the + # trailing window must reach rows before the visible frame starts. + # POST-reclassified windowed-measure filters are already excluded + # (phase != ROW). + if windowed_plans: + date_range_fids = {f"f{i}" for i in range(n_date_range)} + src_where_ids, src_rewrites = _plan_src_row_filters( + filters_by_phase=filters_by_phase, + date_range_fids=date_range_fids, + frame_bound_columns=frame_bound_columns, + ) + for wp in windowed_plans: + wp.where_filter_ids = src_where_ids + wp.src_filter_rewrites = src_rewrites + + return PlannedQuery( + source_relation=source_relation, + row_slots=row_slots, + aggregate_slots=agg_slots, + cross_model_aggregate_plans=cross_model_plans, + windowed_aggregate_plans=windowed_plans, + combined_expression_slots=combined_slots, + transform_layers=transform_layers, + filters_by_phase=filters_by_phase, + projection=projection.public_projection, + order=order_entries, + limit=query.limit, + offset=query.offset, + stage_schema=stage_schema, + active_time_dimension_slot_id=active_td_slot_id, + render_source_model=render_source_model, + distinct_dimension_values=query.distinct_dimension_values, + frame_bound_columns=frame_bound_columns, + ) + + +def _frame_bound_columns(*, row_slots: list) -> List[ValueKey]: + """Raw column keys of the stage's NON-HIDDEN time dimensions (DEV-1732). + + An explicit relational bound on one of these is a FRAME bound — the + caller restating what ``TimeDimension.date_range`` expresses — and is + stripped from CTEs that must read outside the frame. A bound on any other + column, temporal or not, is a population filter and is left alone. + + Hidden ``TimeTruncKey`` slots are excluded deliberately, and the exclusion + is load-bearing rather than cosmetic: ``_build_windowed_plans`` skips hidden + row slots when building ``other_time_dimension_slot_ids``, so a hidden time + axis is never equality-joined into ``_src``. Stripping a bound on one would + leave that axis wholly unconstrained — an unbounded over-count — where + keeping it merely preserves the pre-DEV-1732 result. + + Order-stable and de-duplicated: the same column carried at two + granularities contributes one entry. + """ + out: List[ValueKey] = [] + seen: set = set() + for rs in row_slots: + if rs.hidden or not isinstance(rs.key, TimeTruncKey): + continue + col = rs.key.column + if col in seen: + continue + seen.add(col) + out.append(col) + return out + + +def _plan_src_row_filters( + *, + filters_by_phase: list, + date_range_fids: set, + frame_bound_columns: List[ValueKey], +) -> "Tuple[List[str], List[SrcFilterRewrite]]": + """Partition ROW-phase filters for a windowed measure's ``_src`` scope. + + Returns ``(where_filter_ids, src_filter_rewrites)``: + + * a filter that is ENTIRELY a frame bound is omitted from the ids; + * a filter that is PARTLY one keeps its id and gains a rewrite carrying the + residual population predicate; + * everything else keeps its id with no rewrite. + + Mode-A model filters (``FilterPhase.text``, no typed expression) are exempt + by design — a model filter defines which rows EXIST rather than which frame + the query looks at, there is no ``date_range`` spelling at model level, and + analysing raw dialect SQL would make a silent mis-strip possible. + + ``date_range_fids`` is skipped up front. That is redundant with + ``strip_frame_bounds`` (which recognises ``BetweenKey`` too) and kept + deliberately: it makes a Stage-10 regression structurally impossible even if + a ``date_range`` ever binds to a shape the helper does not match. + """ + time_cols = frozenset(frame_bound_columns) + where_ids: List[str] = [] + rewrites: List[SrcFilterRewrite] = [] + for fp in filters_by_phase: + if fp.phase != Phase.ROW or fp.id in date_range_fids: + continue + if fp.expression is None: + where_ids.append(fp.id) # Mode-A model filter — exempt. + continue + residual = strip_frame_bounds( + key=fp.expression.value_key, time_columns=time_cols, + ) + if residual is None: + continue # wholly a frame bound + where_ids.append(fp.id) + if residual is not fp.expression.value_key: + rewrites.append(SrcFilterRewrite( + filter_id=fp.id, expression=PlannedBoundExpr(value_key=residual), + )) + return where_ids, rewrites + + +def _coerce_extension(spec) -> ModelExtension: + """Coerce a ``ModelExtension`` / dict-with-``source_name`` to a typed + ``ModelExtension`` (for overlaying onto a synthetic sibling model).""" + if isinstance(spec, ModelExtension): + return spec + return ModelExtension.model_validate(spec) + + +def _stage_scope_and_bundle( + *, + query: SlayerQuery, + bundle: ResolvedSourceBundle, + stage_schemas: Dict[str, StageSchema], + data_source: str, + is_root: bool, +) -> "Tuple[Union[ModelScope, StageSchema], ResolvedSourceBundle]": + """Resolve one DAG stage's ``(scope, per-stage bundle)``. + + Each stage binds against its OWN source — not the root's — so a + heterogeneous DAG (stage A over ``orders``, stage B over ``customers``) + resolves each host correctly. Synthetic models for already-planned sibling + stages are threaded into the per-stage bundle so a join / cross-model ref + that targets a sibling resolves against the sibling's flat output columns. + """ + src = query.source_model + sibling_names = set(stage_schemas) + sib = _source_name_if_sibling(src, sibling_names) + + # 1. ``ModelExtension`` / dict OVER a sibling stage: overlay the extra + # columns / measures / joins onto a synthetic model of the sibling CTE + # and bind ModelScope-style (so derived overlay columns resolve). + if sib is not None and not isinstance(src, str): + base = synthetic_model_from_stage_schema( + name=sib, schema=stage_schemas[sib], data_source=data_source, + ) + overlaid = _apply_extension_overlay(base, _coerce_extension(src)) + others = {n: s for n, s in stage_schemas.items() if n != sib} + sb = stage_bundle_with_siblings( + bundle=bundle, source_model=overlaid, + sibling_schemas=others, data_source=data_source, + ) + return ModelScope(source_model=overlaid), sb + + # 2. Bare-string sibling source (chain): bind against the upstream flat + # StageSchema (P6 / DEV-1449). The synthetic upstream model is the + # per-stage host for any cross-model planning / generation consistency. + if isinstance(src, str) and src in stage_schemas: + synth = synthetic_model_from_stage_schema( + name=src, schema=stage_schemas[src], data_source=data_source, + ) + others = {n: s for n, s in stage_schemas.items() if n != src} + sb = stage_bundle_with_siblings( + bundle=bundle, source_model=synth, + sibling_schemas=others, data_source=data_source, + ) + return stage_schemas[src], sb + + # 3. Model-scoped: the stage's own resolved source model. The root uses the + # bundle's source_model (the chain bottoms out at the root's source); + # a named sibling uses its pre-resolved per-stage model. + if is_root: + stage_model = bundle.source_model + else: + stage_model = bundle.stage_source_models.get(query.name) or bundle.source_model + sb = stage_bundle_with_siblings( + bundle=bundle, source_model=stage_model, + sibling_schemas=stage_schemas, data_source=data_source, + ) + return ModelScope(source_model=stage_model), sb + + +def plan_stages( + *, + queries: List[SlayerQuery], + bundle: ResolvedSourceBundle, + cross_model_planner: Optional[CrossModelPlanner] = None, +) -> List[PlannedQuery]: + """Plan a multi-stage DAG. Topo sort, then plan each stage against its own + resolved source + the synthetic models of its already-planned siblings.""" + if len(queries) == 1: + return [plan_query( + query=queries[0], + bundle=bundle, + cross_model_planner=cross_model_planner, + )] + ordered = _topo_sort(queries) + root = ordered[-1] + data_source = ( + (bundle.source_model.data_source if bundle.source_model else None) + or "_stage" + ) + stage_schemas: Dict[str, StageSchema] = {} + results: List[PlannedQuery] = [] + for q in ordered: + scope, stage_bundle = _stage_scope_and_bundle( + query=q, + bundle=bundle, + stage_schemas=stage_schemas, + data_source=data_source, + is_root=q is root, + ) + planned = plan_query( + query=q, + bundle=stage_bundle, + scope=scope, + cross_model_planner=cross_model_planner, + stage_schemas=stage_schemas, + ) + results.append(planned) + if q.name and planned.stage_schema is not None: + stage_schemas[q.name] = planned.stage_schema + return results + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + + +def _format_description_for_dimension( + *, scope: Union[ModelScope, StageSchema], full_name: str, +) -> Tuple[Optional[NumberFormat], Optional[str]]: + """Lift ``format`` / ``description`` for a plain dimension off the + source ``Column``. Returns ``(None, None)`` when the ref can't be + resolved (joined / time-truncated / stage-scoped refs) — those + paths surface their metadata through ``response_meta`` instead. + + DEV-1452 Stage B decision #8 — the planner threads these into the + public slot so the migrated query-backed virtual model carries the + same display contract the legacy enrichment pipeline did. + """ + if not isinstance(scope, ModelScope) or scope.source_model is None: + return None, None + if "." in full_name: + return None, None + col = scope.source_model.get_column(full_name) + if col is None: + return None, None + return col.format, col.description + + +_COUNT_AGGREGATIONS: FrozenSet[str] = frozenset( + {"count", "count_distinct", "count_distinct_approx"} +) +_FLOAT_AGGREGATIONS: FrozenSet[str] = frozenset({ + "avg", "weighted_avg", "median", + "stddev_samp", "stddev_pop", "var_samp", "var_pop", + "corr", "covar_samp", "covar_pop", "percentile", +}) + + +def _infer_aggregated_type( + *, + model: SlayerModel, + measure_name: Optional[str], + aggregation: str, +) -> Optional[DataType]: + """Type for an aggregated measure slot. Mirrors + ``_infer_aggregated_format`` (decision #2 of the Stage B plan): + + * ``*:count`` (measure_name=``"*"``) → ``INT`` + * ``count`` / ``count_distinct`` / ``count_distinct_approx`` → ``INT`` + * ``avg`` / ``weighted_avg`` / ``median`` / parametric / stat aggs → + ``DOUBLE`` + * ``sum`` / ``min`` / ``max`` / ``first`` / ``last`` → inherit from + source column type (DOUBLE if absent). + """ + if measure_name == "*": + return DataType.INT + if aggregation in _COUNT_AGGREGATIONS: + return DataType.INT + if aggregation in _FLOAT_AGGREGATIONS: + return DataType.DOUBLE + # sum / min / max / first / last — preserve source column type. + if measure_name is None: + return None + col = model.get_column(measure_name) + if col is not None and col.type is not None: + return col.type + return None + + +def _format_description_for_measure_formula( + *, scope: Union[ModelScope, StageSchema], bound, +) -> Tuple[Optional[NumberFormat], Optional[str]]: + """Lift ``format`` / ``description`` for a measure formula. The + aggregation-aware format comes from ``_infer_aggregated_format`` when + the bound expression is a bare local aggregate; description follows + the source ``Column`` (sum / min / max preserve documentation). + """ + if not isinstance(scope, ModelScope) or scope.source_model is None: + return None, None + if not isinstance(bound.value_key, AggregateKey): + return None, None + src = bound.value_key.source + if isinstance(src, StarKey): + # ``*:count`` — INTEGER format inferred by helper; no description. + return ( + _infer_aggregated_format( + model=scope.source_model, + measure_name="*", + aggregation=bound.value_key.agg, + ), + None, + ) + if not isinstance(src, (ColumnKey, ColumnSqlKey)): + return None, None + if getattr(src, "path", ()): # cross-model — handled by response_meta + return None, None + bare = getattr(src, "leaf", None) or getattr(src, "column_name", None) + if bare is None: + return None, None + fmt = _infer_aggregated_format( + model=scope.source_model, + measure_name=bare, + aggregation=bound.value_key.agg, + ) + col = scope.source_model.get_column(bare) + desc = col.description if col is not None else None + return fmt, desc + + +def _type_for_measure_formula( + *, scope: Union[ModelScope, StageSchema], bound, +) -> Optional[DataType]: + """Lift ``type`` for a measure-formula slot. + + Mirrors ``_format_description_for_measure_formula`` — sources the + type from ``_infer_aggregated_type`` for local aggregates so the + migrated query-backed virtual model carries ``*:count → INT``, + ``avg → DOUBLE``, ``sum → source column type``. Declared + ``ModelMeasure.type`` overrides — that flow runs through + ``expand_model_measures`` so the bound key already carries the + declared type via the source column lookup. + """ + if not isinstance(scope, ModelScope) or scope.source_model is None: + return None + if not isinstance(bound.value_key, AggregateKey): + return None + src = bound.value_key.source + if isinstance(src, StarKey): + return _infer_aggregated_type( + model=scope.source_model, + measure_name="*", + aggregation=bound.value_key.agg, + ) + if not isinstance(src, (ColumnKey, ColumnSqlKey)): + return None + if getattr(src, "path", ()): # cross-model — handled elsewhere + return None + bare = getattr(src, "leaf", None) or getattr(src, "column_name", None) + if bare is None: + return None + return _infer_aggregated_type( + model=scope.source_model, + measure_name=bare, + aggregation=bound.value_key.agg, + ) + + +def _joined_column_type( + *, source_model: SlayerModel, full_name: str, bundle: ResolvedSourceBundle, +) -> Optional[DataType]: + """Best-effort type of a dotted (joined) dimension by walking the join + chain — mirrors ``binding._resolve_dotted`` (``parts[:-1]`` are join + hops matched on ``target_model``, ``parts[-1]`` is the leaf column), + but returns ``None`` on any miss instead of raising. The binder has + already validated the ref, so this is a guard rather than a primary + check. + """ + parts = full_name.split(".") + if parts and parts[0] == source_model.name: # C14 self-prefix strip + parts = parts[1:] + if not parts: + return None + *hops, leaf = parts + current = source_model + visited = {current.name} + for hop in hops: + if not any(j.target_model == hop for j in current.joins): + return None + nxt = bundle.get_referenced_model(hop) + if nxt is None or nxt.name in visited: + return None + visited.add(nxt.name) + current = nxt + col = current.get_column(leaf) + return col.type if col is not None else None + + +def _type_for_dimension( + *, + scope: Union[ModelScope, StageSchema], + full_name: str, + bundle: ResolvedSourceBundle, +) -> Optional[DataType]: + """Lift ``type`` for a dimension. Local refs read the source column; + joined (dotted) refs walk the join chain to the terminal column's + type. Returning ``None`` for joined refs (the old behaviour) made the + query-backed virtual-model wrap coerce them to ``DOUBLE`` (its + ``sc.type or DataType.DOUBLE`` fallback), mistyping joined string / + temporal dimensions on the persisted virtual model. + """ + if not isinstance(scope, ModelScope) or scope.source_model is None: + return None + if "." in full_name: + return _joined_column_type( + source_model=scope.source_model, full_name=full_name, bundle=bundle, + ) + col = scope.source_model.get_column(full_name) + return col.type if col is not None else None + + +def _opaque_dim_type( + *, + scope: Union[ModelScope, StageSchema], + full_name: str, + bundle: ResolvedSourceBundle, +) -> Optional[DataType]: + """Declared type of a query dimension, for the opaque-grouping guard only. + + Resolves BOTH a ``ModelScope`` origin (via ``_type_for_dimension``) AND a + downstream ``StageSchema`` (via its typed ``columns``). ``_type_for_dimension`` + deliberately returns ``None`` for a StageSchema — that ``None`` is load-bearing + for downstream typing (the virtual-model ``DOUBLE`` coercion) and must not + change — so + the guard needs its own resolver to catch an opaque column projected in one + stage and grouped in the next. + """ + if isinstance(scope, StageSchema): + col = scope.get(full_name) + return col.type if col is not None else None + return _type_for_dimension(scope=scope, full_name=full_name, bundle=bundle) + + +def _reject_opaque_grouping_dim( + *, + query: SlayerQuery, + scope: Union[ModelScope, StageSchema], + full_name: str, + bundle: ResolvedSourceBundle, +) -> None: + """Raise if ``full_name`` is an opaque dimension this query will GROUP BY. + + Grouping by an opaque column (``DataType.UNKNOWN`` — e.g. a PostGIS ``point`` + or any type with no equality operator) emits SQL the database rejects, so we + fail with an actionable message instead of a raw driver error. Only an + *actually grouped* dimension is refused: aggregating queries and dim-only + DISTINCT queries group, but raw-row mode (``distinct_dimension_values=False`` + with no measures, DEV-1543) projects dimensions without a top-level GROUP BY, + so an opaque column is legal there — and a downstream stage that groups such a + projected value is still caught via the StageSchema (see ``_opaque_dim_type``). + Checked on the declared type *before* ``bind_expr`` expands the column's + ``sql``, so an opaque *derived* column is caught by its type rather than + tripping the DEV-1410 cycle check first. (PR #259 "Unknown type" main-parity: + the legacy guard lived in ``enrichment._resolve_dimensions``, which the typed + pipeline bypasses.) + """ + if not (bool(query.measures) or query.distinct_dimension_values): + return + dim_type = _opaque_dim_type(scope=scope, full_name=full_name, bundle=bundle) + if dim_type is not None and dim_type.is_opaque: + raise ValueError( + f"Column '{full_name}' cannot be used as a dimension: its type does " + f"not support the GROUP BY / DISTINCT this query requires. Define a " + f"derived column that extracts a comparable value instead, e.g. " + f"sql=\"payload->>'status'\" with type TEXT." + ) + + +def _saved_model_measure_type( + *, scope: Union[ModelScope, StageSchema], formula: str, +) -> Optional[DataType]: + """Lift the explicit ``type`` from a saved ``ModelMeasure`` when the + query formula is a bare reference to one. + + ``expand_model_measures`` rewrites ``adjusted_total`` to the saved + measure's underlying formula AST but doesn't surface the measure's + explicit ``type=`` to downstream consumers — so an explicit + ``ModelMeasure(formula="amount:sum * 1.0", type=DataType.DOUBLE)`` + on a reusable named measure would otherwise be lost unless + ``_type_for_measure_formula`` happens to infer the same value. This + helper rescues it by re-looking-up the saved measure here. + + Only fires when the formula text is itself a bare identifier + matching a ``ModelMeasure.name`` on the source model; arithmetic / + function-call / colon-suffix formulas always fall through to + inference (the saved-measure type only applies when the user + references the saved measure by name directly). + """ + if not isinstance(scope, ModelScope) or scope.source_model is None: + return None + bare = formula.strip() + if not bare.isidentifier(): + return None + saved = scope.source_model.get_measure(bare) + return saved.type if saved is not None else None + + +def _bare_saved_measure_name( + *, scope: Union[ModelScope, StageSchema], formula: str, +) -> Optional[str]: + """The saved ``ModelMeasure.name`` when the query formula is a BARE + reference to one (DEV-1713 / DEV-1495 bare-named-measure aliasing). + + ``expand_model_measures`` rewrites ``rev_total`` to the saved measure's + underlying formula AST, so without this the measure would surface under + the formula-derived canonical (``revenue_sum``) instead of the name the + user referenced (``rev_total``). Fires ONLY for a bare identifier matching + a ``ModelMeasure.name`` on the source model — the same gate as + :func:`_saved_model_measure_type`; qualified / arithmetic / colon-suffix + formulas fall through (name-preservation is scoped to the bare form). + """ + if not isinstance(scope, ModelScope) or scope.source_model is None: + return None + bare = formula.strip() + if not bare.isidentifier(): + return None + saved = scope.source_model.get_measure(bare) + return saved.name if saved is not None else None + + +def _declared_measures_from_query( + *, + query: SlayerQuery, + scope: Union[ModelScope, StageSchema], + bundle: ResolvedSourceBundle, +) -> List[DeclaredMeasure]: + # Downstream stages bind against a flat StageSchema whose columns ARE + # the ``__``-flattened multi-hop aliases of the upstream stage, so + # ``__`` is legal in their refs (P5 / DEV-1449). Model-scoped stages + # keep the P1 rejection. + flat_scope = isinstance(scope, StageSchema) + declared: List[DeclaredMeasure] = [] + for d in (query.dimensions or []): + full = d.full_name + _reject_opaque_grouping_dim( + query=query, scope=scope, full_name=full, bundle=bundle, + ) + bound = bind_expr( + parsed=parse_expr(full, allow_dunder=flat_scope), + scope=scope, + bundle=bundle, + ) + flat_name = _flatten_dotted(full) + fmt, desc = _format_description_for_dimension( + scope=scope, full_name=full, + ) + dim_type = _type_for_dimension( + scope=scope, full_name=full, bundle=bundle, + ) + declared.append(DeclaredMeasure( + bound=bound, + declared_name=flat_name, + public_name=flat_name, + label=d.label, + type=dim_type, + format=fmt, + description=desc, + )) + # Time dimensions follow dimensions in the public projection — matches + # the legacy ``user_projection`` order (dims, then time dims, then + # measures). + for td in (query.time_dimensions or []): + full = td.dimension.full_name + bound = bind_time_dimension(td=td, scope=scope, bundle=bundle) + flat_name = _flatten_dotted(full) + declared.append(DeclaredMeasure( + bound=bound, + declared_name=flat_name, + public_name=flat_name, + label=td.label, + type=DataType.TIMESTAMP, + )) + for m in (query.measures or []): + formula = m.formula + explicit_name = m.name + parsed = parse_expr(formula, allow_dunder=flat_scope) + # DEV-1450 stage 7b.8 — pre-bind ModelMeasure expansion. A bare + # ``Ref`` whose name matches a saved ``ModelMeasure`` on the + # host model is rewritten to the measure's formula AST so the + # binder resolves the underlying columns. Only applies against + # ModelScope (downstream stages bind against StageSchema and + # don't expose saved measures). + if isinstance(scope, ModelScope) and scope.source_model is not None: + parsed = expand_model_measures( + expr=parsed, + model=scope.source_model, + ) + bound = bind_expr(parsed=parsed, scope=scope, bundle=bundle) + # Stage 7b.10: sugar-lowering of ``change`` / ``change_pct`` now + # runs in ``plan_query`` AFTER time-key patching, so the inner + # ``time_shift`` inherits a patched ``time_key`` instead of + # ``None``. Identity-preservation for the inner aggregate slot + # (DEV-1446) still holds — ``lower_sugar_transforms`` keeps the + # inner ``AggregateKey`` instance unchanged. + canonical = _canonical_alias_for_formula(formula, bound=bound) + # DEV-1713: a bare reference to a saved ModelMeasure surfaces under the + # measure NAME, not the formula-derived canonical. Explicit query + # ``name`` still wins; the saved name is an implicit ``name``. + saved_name = _bare_saved_measure_name(scope=scope, formula=formula) + alias_name = explicit_name or saved_name + declared_name = alias_name or canonical + public_name = alias_name or canonical + fmt, desc = _format_description_for_measure_formula( + scope=scope, bound=bound, + ) + # Codex: type-priority chain (highest wins): + # 1. ``m.type`` — user-supplied override on the query measure. + # 2. Saved ``ModelMeasure.type`` — when the query formula is a + # bare reference to a reusable saved measure on the source + # model, that measure's explicit type wins over inference. + # ``expand_model_measures`` rewrites the AST but drops the + # source measure's type metadata; re-look-up here. + # 3. ``_type_for_measure_formula`` — aggregation-aware inference. + # Mirrors how the legacy ``EnrichedMeasure.type`` honored an + # explicit type before falling back to inference. + m_type = ( + m.type + or _saved_model_measure_type(scope=scope, formula=formula) + or _type_for_measure_formula(scope=scope, bound=bound) + ) + declared.append(DeclaredMeasure( + bound=bound, + declared_name=declared_name, + public_name=public_name, + label=m.label, + # DEV-1443: keep the canonical alias whenever the surfaced name + # differs from it (explicit ``name`` OR an implicit saved-measure + # name) so a colon-form filter / ORDER BY still resolves. + canonical_alias=canonical if alias_name else None, + type=m_type, + format=fmt, + description=desc, + )) + return declared + + +def _topo_sort(queries: List[SlayerQuery]) -> List[SlayerQuery]: + """Kahn's algorithm: order stages so each appears after its + siblings it references via ``source_model``. + + Raises ``ValueError`` on: + * duplicate stage names, + * a cycle in the dependency graph. + + Stages without a ``name`` (typically the final / root) are appended + last in input order. + """ + if len(queries) <= 1: + return list(queries) + named = [q for q in queries if q.name] + names = [q.name for q in named] + duplicates = sorted({n for n in names if names.count(n) > 1}) + if duplicates: + raise ValueError( + f"Duplicate stage names in source_queries DAG: {duplicates}" + ) + by_name = {q.name: q for q in named} + in_degree = {q.name: 0 for q in named} + edges: Dict[str, List[str]] = {q.name: [] for q in named} + for q in named: + # A stage depends on a sibling when its ``source_model`` reads from it — + # either the bare-string form OR a ``ModelExtension`` / dict over the + # sibling. Capturing both keeps the topo order + cycle detection correct + # for extension-over-sibling stages (not just join-target deps, which + # the engine's runtime list sorter handles upstream). + dep = _source_name_if_sibling(q.source_model, by_name) + if dep is not None and dep != q.name: + in_degree[q.name] += 1 + edges[dep].append(q.name) + sorted_names: List[str] = [] + queue = [n for n, d in in_degree.items() if d == 0] + while queue: + n = queue.pop(0) + sorted_names.append(n) + for dep in edges[n]: + in_degree[dep] -= 1 + if in_degree[dep] == 0: + queue.append(dep) + if len(sorted_names) != len(in_degree): + remaining = sorted(set(in_degree) - set(sorted_names)) + raise ValueError( + f"Cycle detected in source_queries DAG involving stages: " + f"{remaining}" + ) + sorted_named = [by_name[n] for n in sorted_names] + unnamed = [q for q in queries if q.name is None] + return sorted_named + unnamed + + +def _flatten_dotted(name: str) -> str: + # DEV-1713: the ``__``-flatten is owned by the naming module. + return flat_name(name) + + +def _canonical_alias_for_formula( + formula: str, + *, + bound: Optional[BinderBoundExpr] = None, +) -> str: + """Compute the canonical public alias for a measure formula. + + Mirrors ``canonical_agg_name`` for any formula whose bound root is + an ``AggregateKey`` (covers bare ``revenue:sum`` AND parametric + forms like ``revenue:percentile(p=0.5)`` / ``corr(other=quantity)``). + Pre-binding text-shape recognition is used only as a fallback when + no bound expression is supplied. For arbitrary formulas + (transforms, arithmetic), sanitise the formula text so the alias + remains a valid identifier. + + DEV-1450 stage 7b.13: parametric aggregations route through + ``canonical_agg_name`` so kwargs are sanitised consistently with the + legacy enrichment path (``p=0.5`` -> ``_p_0_5``). Without this, the + naive text-replace fallback below leaks the ``=`` literally into the + alias (``amount_percentile_p=0_5_``), breaking parity. + """ + if bound is not None and isinstance(bound.value_key, AggregateKey): + key = bound.value_key + if isinstance(key.source, StarKey): + measure_name: Optional[str] = "*" + # Cross-model star (``customers.*:count``) carries its join + # path so the canonical alias keeps the ``customers.`` prefix + # (result key ``orders.customers._count``). + path: Tuple[str, ...] = key.source.path + else: + # ColumnKey exposes ``.leaf``; ColumnSqlKey exposes + # ``.column_name``. Both shapes can appear as aggregate + # sources (the synth adapter rejects ``ColumnSqlKey`` with + # a typed deferral; the planner still needs to derive an + # alias before the generator runs). Mirror ``_canonical_name`` + # at ``planning.py:540-545``. + measure_name = ( + getattr(key.source, "leaf", None) + or getattr(key.source, "column_name", None) + ) + path = getattr(key.source, "path", ()) + if measure_name is not None: + prefix = ".".join(path) + "." if path else "" + # Local aggregates retain the kwarg suffix to match legacy + # ``enrichment.py:349`` (``percentile(p=0.5)`` -> + # ``_p_0_5``). Cross-model aggregates ALSO retain it -- the + # legacy ``query_engine.py:2160`` drops it, causing CTE alias + # collision on two parametric variants, which the 7b.5 fix + # corrected at the planner layer. Result-key shape for + # cross-model parametric aggs therefore diverges from + # legacy in this slice (no parity tests for that combination; + # structural correctness over bit-identical legacy output). + return prefix + canonical_agg_name( + measure_name=measure_name, + aggregation_name=key.agg, + agg_args=[agg_kwarg_canonical_str(a) for a in key.args] or None, + agg_kwargs={ + k: agg_kwarg_canonical_str(v) for k, v in key.kwargs + } or None, + ) + # Fall through to text-based path -- AggregateKey source is + # neither StarKey, ColumnKey, nor ColumnSqlKey (shouldn't be + # reachable in practice; the binder restricts sources to + # those three shapes). + text = formula.strip() + if ":" in text and "(" not in text: + base, agg = text.rsplit(":", 1) + return canonical_agg_name( + measure_name=base, aggregation_name=agg, + ) + return ( + text.replace(".", "_").replace(":", "_").replace(" ", "_") + .replace("(", "_").replace(")", "_").replace(",", "_") + ) + + +def _source_column_names( + scope: Union[ModelScope, StageSchema], +) -> FrozenSet[str]: + if isinstance(scope, ModelScope) and scope.source_model is not None: + return frozenset(c.name for c in scope.source_model.columns) + if isinstance(scope, StageSchema): + return frozenset(c.name for c in scope.columns) + return frozenset() + + +def _host_model_name( + scope: Union[ModelScope, StageSchema], +) -> str: + if isinstance(scope, ModelScope) and scope.source_model is not None: + return scope.source_model.name + if isinstance(scope, StageSchema): + return scope.relation_name + return "(stage)" + + +def _bucket_slots(slots: List[ValueSlot]): + row: List[ValueSlot] = [] + agg: List[ValueSlot] = [] + combined: List[ValueSlot] = [] + for s in slots: + if s.phase == Phase.ROW: + row.append(s) + elif s.phase == Phase.AGGREGATE: + agg.append(s) + else: + combined.append(s) + return row, agg, combined + + +def _emit_stage_schema( + *, + query: SlayerQuery, + projection, +) -> StageSchema: + """Build the StageSchema from the projection plan. + + Only public slots appear (hidden slots are trimmed). One column per + occurrence in ``public_projection`` so multi-alias declarations + (same key with two ``name``s) emit one column per alias rather + than two copies of ``public_aliases[0]``. + """ + columns: List[StageColumn] = [] + alias_idx: Dict[str, int] = {} + for sid in projection.public_projection: + slot = projection.registry.get(sid) + if slot.hidden: + continue + idx = alias_idx.setdefault(sid, 0) + if idx < len(slot.public_aliases): + alias = slot.public_aliases[idx] + else: + alias = slot.declared_name + alias_idx[sid] = idx + 1 + # The downstream bind name + CTE column name are the ``__``-flattened + # form so a later stage can reference a cross-model aggregate + # (``customers.revenue_sum`` → ``customers__revenue_sum``), matching + # how dimensions already flatten and how the legacy virtual-model + # rename exposed these columns (P5/DEV-1449). ``public_alias`` keeps + # the dotted result-key form. Dimensions / local / user-named + # measures have no dot, so flattening is a no-op for them. + flat = _flatten_dotted(alias) + # Two distinct public columns that flatten to the same downstream + # name (e.g. a joined ``customers.region`` and a literal model column + # ``customers__region`` via the C11 carve-out) would make the stage's + # CTE column ambiguous. Surface it instead of silently binding the + # first match downstream. + if any(c.name == flat for c in columns): + raise ValueError( + f"Stage column name collision on {flat!r}: two projected " + f"columns flatten to the same downstream name. Give one an " + f"explicit measure `name` to disambiguate." + ) + columns.append(StageColumn( + name=flat, + sql_alias=flat, + public_alias=alias, + type=slot.type, + label=slot.label, + hidden=False, + format=slot.format, + description=slot.description, + )) + relation_name = query.name or "(unnamed_stage)" + return StageSchema(relation_name=relation_name, columns=columns) + + +def _emit_transform_layers(*, slots: List[ValueSlot]) -> List[TransformLayer]: + """One TransformLayer per ``TransformKey`` slot, emitted in + dependency order (innermost transform first). + + Nested transforms (``cumsum(change(amount:sum))``) require + per-slot layers so the generator can render the inner window / + self-join before the outer one consumes it. Repeated ops at + different nesting levels stay in separate layers; collapsing by + op would lose the ordering invariant. + + Per-slot transform metadata (partition_keys, time_key, args, + kwargs) lives on the slot's ``key`` (TransformKey); the generator + slices read it from there. + """ + transform_slots = [ + s for s in slots if isinstance(s.key, TransformKey) + ] + # Topological order: a slot whose TransformKey.input references + # another slot's key must come AFTER that other slot. Walk + # `_iter_slot_deps` to discover dependencies among transform slots. + slot_by_key = {s.key: s for s in transform_slots} + in_degree = {s.id: 0 for s in transform_slots} + deps_of: Dict[str, List[str]] = {s.id: [] for s in transform_slots} + for s in transform_slots: + # The slot's transform depends on whatever transform slots + # appear inside its ValueKey tree (e.g. cumsum(change(...))'s + # cumsum slot depends on the change/time_shift slot). + for dep in _iter_slot_deps(s.key): + if dep is s.key or not isinstance(dep, TransformKey): + continue + dep_slot = slot_by_key.get(dep) + if dep_slot is None: + continue + deps_of[dep_slot.id].append(s.id) + in_degree[s.id] += 1 + # Kahn's algorithm: start from independent layers. + ready = [s.id for s in transform_slots if in_degree[s.id] == 0] + ordered_ids: List[str] = [] + while ready: + nxt = ready.pop(0) + ordered_ids.append(nxt) + for child in deps_of[nxt]: + in_degree[child] -= 1 + if in_degree[child] == 0: + ready.append(child) + # Fallback: any remaining slots (shouldn't happen with the typed + # pipeline's identity-via-key, but guard) get appended in input order. + seen = set(ordered_ids) + for s in transform_slots: + if s.id not in seen: + ordered_ids.append(s.id) + by_id = {s.id: s for s in transform_slots} + return [ + TransformLayer(op=by_id[sid].key.op, slot_ids=[sid]) + for sid in ordered_ids + ] + + +# --------------------------------------------------------------------------- +# Stage 7b.3c — date_range → filter + main-TD disambiguation +# --------------------------------------------------------------------------- + + +def _validate_model_filter( + *, + mf: str, + idx: int, + model: SlayerModel, +) -> FilterPhase: + """Validate a ``SlayerModel.filters`` entry and emit a text-only + ``FilterPhase`` for it. + + Replicates legacy validation (``slayer/engine/enrichment.py:1138-1219``): + + * ``parse_sql_predicate`` rejects DSL constructs (colon aggregation, + transform calls) and raw ``OVER(...)`` window functions. + * Reject references to a ``ModelMeasure`` declared on the same + model — model filters are WHERE-clause SQL, can't reference + aggregates (legacy ``enrichment.py:1147-1153``). + * Reject references to a column whose ``Column.sql`` contains a + window function (legacy ``enrichment.py:1205-1219``). + * DEV-1450 follow-up #4b: references to a NON-windowed derived + ``Column.sql`` column are now accepted — the generator inlines the + column's expanded SQL at render time + (``SQLGenerator._render_model_filter_sql``) and pulls any joins the + expansion crosses into the FROM, matching legacy + ``resolve_filter_columns``. + """ + parsed = parse_sql_predicate(mf) + measure_names = {m.name for m in (model.measures or [])} + windowed_columns = { + c.name for c in model.columns + if c.sql and has_window_function(c.sql) + } + for col in parsed.columns: + if col in measure_names: + raise ValueError( + f"Model filter {mf!r} references measure {col!r}. " + f"Model filters can only reference table columns (WHERE). " + f"Use query-level filters for measure conditions." + ) + if col in windowed_columns: + raise ValueError( + f"Model filter {mf!r} references column {col!r} whose " + f"SQL contains a window function. Factor it into a " + f"multi-stage source_queries model or use a rank-family " + f"transform at query time." + ) + return FilterPhase( + id=f"mf{idx}", + phase=Phase.ROW, + text=mf, + text_columns=tuple(parsed.columns), + expression=None, + ) + + +def _build_date_range_filter( + *, + td: TimeDimension, + scope: ModelScope, + bundle: ResolvedSourceBundle, +) -> BoundFilter: + """Build a row-phase ``BoundFilter`` from a ``TimeDimension``'s + ``date_range``. + + The predicate binds against the bare underlying ``ColumnKey`` + (not the ``TimeTruncKey``) so generator slice 7b.11 can apply the + filter to the outer projection while the shifted self-join CTE + reads raw data. Shape: + + BetweenKey(column=col, low=start, high=end) + + Inclusive on both sides — matches legacy ``column BETWEEN start + AND end``. The typed BetweenKey lets the SQL generator emit + ``exp.Between`` rather than ``col >= start AND col <= end``, + closing the syntactic parity gap with the legacy generator + (DEV-1450 stage 7b.9). + + Bound literals are normalised via ``normalize_scalar``; strings + pass through unchanged. + """ + full = td.dimension.full_name + parsed = parse_expr(full) + bound_col_expr = bind_expr(parsed=parsed, scope=scope, bundle=bundle) + col_key = bound_col_expr.value_key + # DEV-1450 #4a: a derived (Column.sql) temporal column binds to a + # ColumnSqlKey; the BetweenKey accepts both kinds and the generator + # renders a ColumnSqlKey by expanding (`` BETWEEN ...``). + if not isinstance(col_key, (ColumnKey, ColumnSqlKey)): + raise ValueError( + f"date_range filter for TimeDimension {full!r} expected a " + f"column reference; got {type(col_key).__name__}." + ) + + start, end = td.date_range[0], td.date_range[1] + predicate = BetweenKey( + column=col_key, + low=LiteralKey(value=normalize_scalar(start)), + high=LiteralKey(value=normalize_scalar(end)), + ) + refs = tuple(walk_value_keys(predicate)) + phase = max((k.phase for k in refs), default=predicate.phase) + return BoundFilter( + value_key=predicate, phase=phase, referenced_keys=refs, + ) + + +def _resolve_main_time_dimension( + *, + query: SlayerQuery, + model: SlayerModel, +) -> Optional[TimeDimension]: + """Resolve the active time dimension for transform / windowing. + + * 0 TDs → ``None``. + * 1 TD → that TD (``query.main_time_dimension`` is ignored — + matches legacy semantics). + * 2+ TDs: + * ``query.main_time_dimension`` set → match by ``full_name`` + first, then by ``leaf``; raise ``UnknownReferenceError`` if + neither matches. + * Else ``model.default_time_dimension`` set → match by leaf; + return ``None`` if it doesn't match a TD in this query + (legacy graceful no-op — the default points at a column the + user didn't include in this query's time_dimensions). + * Else → ``None``. + """ + tds = list(query.time_dimensions or []) + if not tds: + return None + if len(tds) == 1: + return tds[0] + + if query.main_time_dimension: + target = query.main_time_dimension + # Prefer full-name (more specific) over leaf match. + for td in tds: + if td.dimension.full_name == target: + return td + leaf_matches = [td for td in tds if td.dimension.name == target] + if len(leaf_matches) == 1: + return leaf_matches[0] + if len(leaf_matches) > 1: + # Ambiguous: multiple TDs share the same leaf (e.g. + # ``customers.created_at`` and ``payments.created_at``). + # Force the user to disambiguate via full_name. + raise AmbiguousReferenceError( + name=target, + candidates=[td.dimension.full_name for td in leaf_matches], + ) + raise UnknownReferenceError( + name=target, + scope_kind="TimeDimension", + scope_summary=( + f"time_dimensions: " + f"{[td.dimension.full_name for td in tds]}" + ), + suggestion=None, + ) + + default = model.default_time_dimension + if default: + # Legacy ``_resolve_time_alias`` returns + # ``f"{model.name}.{default_time_dimension}"``, which only points + # at the host model — never at a joined TD. Preserve that: prefer + # a host-local TD (``td.dimension.model is None``) over any + # joined TD that happens to share the leaf name. + for td in tds: + if td.dimension.model is None and td.dimension.name == default: + return td + return None diff --git a/slayer/engine/syntax.py b/slayer/engine/syntax.py new file mode 100644 index 00000000..cea3b500 --- /dev/null +++ b/slayer/engine/syntax.py @@ -0,0 +1,985 @@ +"""Stage 7a.3 (DEV-1450) — Mode-B Python-AST parser. + +Public entry point: ``parse_expr(text: str) -> ParsedExpr``. + +The parser consumes a Mode-B expression string (the SLayer DSL used in +``ModelMeasure.formula``, ``SlayerQuery.measures``, +``SlayerQuery.filters``, …) and emits a typed ``ParsedExpr`` tree. It +is PURE syntax — no scope resolution, no named-measure expansion, no +function-style aggregation rewriting (those are upstream concerns: the +slack normalization layer does function-style → colon; the binder +handles scope and named-measure expansion). + +Pipeline order (per query / model save): + + raw → slack normalize → parse_expr → bind → plan → SQL + +Mode-B grammar: + +* bare identifier (``revenue``) +* dotted path (``customers.regions.name``) +* colon aggregation (``revenue:sum``, ``*:count``, + ``price:weighted_avg(weight=qty)``, ``revenue:last(ordered_at)``) +* transform call (``cumsum``, ``lag``, ``rank``, ``time_shift``, …) +* scalar function (closed allowlist from ``SCALAR_FUNCTIONS``) +* arithmetic / comparison / boolean / unary +* parenthesised grouping + +Rejections (per DEV-1450 spec): + +* Function calls not in SCALAR_FUNCTIONS / transforms / aggregations → + ``UnknownFunctionError``. +* Raw ``OVER(...)`` clauses → ``IllegalWindowInFilterError``. +* ``__`` in any user-supplied identifier → ``ValueError`` (reserved for + internal join-path aliases on the SQL side). +* Chained comparisons (``1 < x < 10``) → ``ValueError``; the user + splits as ``1 < x and x < 10``. + +ParsedExpr family: ``Ref`` / ``DottedRef`` / ``StarSource`` / +``Literal`` / ``AggCall`` / ``TransformCall`` / ``ScalarCall`` / +``Arith`` / ``UnaryOp`` / ``Cmp`` / ``BoolOp``. All are frozen +Pydantic models with value-based equality so tests assert via ``==``. + +Dormant in stage 7a — no engine code calls ``parse_expr`` yet. The +binder (stage 7a.5) is the first consumer. +""" + +from __future__ import annotations + +import ast +import re +from decimal import Decimal +from typing import Any, Dict, Iterator, List, Optional, Tuple, Union + +from pydantic import BaseModel, ConfigDict + +from slayer.core.errors import IllegalWindowInFilterError, UnknownFunctionError +from slayer.core.formula import ALL_TRANSFORMS +from slayer.core.keys import SCALAR_FUNCTIONS + + +# --------------------------------------------------------------------------- +# ParsedExpr family +# --------------------------------------------------------------------------- + + +class _BaseNode(BaseModel): + model_config = ConfigDict(frozen=True, arbitrary_types_allowed=True) + + +class Ref(_BaseNode): + name: str + + +class DottedRef(_BaseNode): + parts: Tuple[str, ...] + + +class StarSource(_BaseNode): + pass + + +class Literal(_BaseNode): + value: Union[Decimal, str, bool, None] = None + + +class TupleLit(_BaseNode): + """A literal-only tuple/list RHS for ``IN`` / ``NOT IN`` filters (DEV-1475). + + Only emitted on the right-hand side of a ``Cmp`` whose op is ``in`` or + ``not in``. Every ``elements`` entry is a ``Literal`` — references and + expressions on the RHS are rejected at parse time so the binder can + fold the predicate into a single ``InKey`` with a tuple of + ``LiteralKey``. Empty tuples are rejected too (an empty IN is a SQL + quirk that varies by dialect; reject early with a clear message). + """ + + elements: Tuple[Literal, ...] + + +class AggCall(_BaseNode): + source: Union[Ref, DottedRef, StarSource] + agg: str + args: Tuple[Any, ...] = () + kwargs: Tuple[Tuple[str, Any], ...] = () + + +class TransformCall(_BaseNode): + op: str + input: Any + args: Tuple[Any, ...] = () + kwargs: Tuple[Tuple[str, Any], ...] = () + + +class ScalarCall(_BaseNode): + name: str + args: Tuple[Any, ...] = () + + +class Arith(_BaseNode): + op: str + left: Any + right: Any + + +class UnaryOp(_BaseNode): + op: str + operand: Any + + +class Cmp(_BaseNode): + op: str + left: Any + right: Any + + +class BoolOp(_BaseNode): + op: str + operands: Tuple[Any, ...] + + +ParsedExpr = Union[ + Ref, DottedRef, StarSource, Literal, TupleLit, + AggCall, TransformCall, ScalarCall, + Arith, UnaryOp, Cmp, BoolOp, +] + + +# --------------------------------------------------------------------------- +# Constants +# --------------------------------------------------------------------------- + + +_PLACEHOLDER_PREFIX = "__slayer_agg_" +_PLACEHOLDER_RE = re.compile(rf"^{_PLACEHOLDER_PREFIX}(\d+)__$") +_OVER_RE = re.compile(r"\bOVER\s*\(", re.IGNORECASE) +# Python-string-literal matcher (handles backslash escapes). Mode-B expressions +# use Python string syntax, so a SQL-style ('' / "") doubling matcher would +# both miss ``"x \" OVER("`` and false-positive on the ``OVER(`` inside it +# (Codex). Used by ``_normalize_sql_filter_operators``, ``_preprocess_colons``, +# and the raw-``OVER(`` pre-scan. +_PY_STRING_LITERAL_RE = re.compile(r"'(?:\\.|[^'\\])*'|\"(?:\\.|[^\"\\])*\"") +# SQL ``LIKE`` / ``NOT LIKE`` operator → the ``like(col, pattern)`` scalar the +# Mode-B DSL already accepts (DEV-1484 emits it as SQL ``LIKE``). Mirrors +# ``formula._preprocess_like`` so the typed filter parser accepts the same LIKE +# spelling as the documented Mode-B filter grammar — a pg-facade WHERE +# ``col LIKE 'p%'`` (or ``NOT LIKE``) lands here as a verbatim filter (DEV-1704). +# LHS is a bare/dotted identifier or a single scalar call; RHS a quoted pattern. +# Applied to the whole expression (the pattern is a string literal), same as +# ``formula._preprocess_like``. +_SQL_LIKE_RE = re.compile( + r"\b(\w+\([^()]*\)|(?:\w+\.)*\w+)\s+(not\s+)?like\s+('[^']*')", + flags=re.IGNORECASE, +) + + +def _rewrite_sql_like(text: str) -> str: + """``col LIKE 'p%'`` → ``like(col, 'p%')``; ``col NOT LIKE 'p%'`` → + ``not like(col, 'p%')`` — outside/inside handling matches + ``formula._preprocess_like``.""" + + def _sub(m: "re.Match[str]") -> str: + lhs, neg, pat = m.group(1), m.group(2), m.group(3) + call = f"like({lhs}, {pat})" + return f"not {call}" if neg else call + + return _SQL_LIKE_RE.sub(_sub, text) +_COLON_AGG_RE = re.compile( + r"(\*|[a-zA-Z_]\w*(?:\.[a-zA-Z_]\w*)*(?:\.\*)?)" # source: * / ident / dotted + r":" + r"([a-zA-Z_]\w*)" + # No (args) consumption — Python's AST handles that. +) + +_FILTER_KEYWORDS = frozenset({"and", "or", "not", "in", "is"}) +_SCAN_TOKEN_RE = re.compile( + r"(?P\s+)" + r"|(?P'(?:\\.|[^'\\])*'|\"(?:\\.|[^\"\\])*\")" + r"|(?P[A-Za-z_]\w*)" + r"|(?P\d+(?:\.\d*)?|\.\d+)" + r"|(?P==|<=|>=|!=)" + r"|(?P=)" + r"|(?P\()" + r"|(?P\))" + r"|(?P\[)" + r"|(?P\])" + r"|(?P,)" + r"|(?P.)", + re.DOTALL, +) + + +_BIN_OP_MAP: Dict[type, str] = { + ast.Add: "+", ast.Sub: "-", ast.Mult: "*", ast.Div: "/", + ast.Mod: "%", ast.Pow: "**", ast.FloorDiv: "//", +} +_CMP_OP_MAP: Dict[type, str] = { + ast.Eq: "==", ast.NotEq: "!=", + ast.Lt: "<", ast.LtE: "<=", + ast.Gt: ">", ast.GtE: ">=", + # ``IS`` / ``IS NOT`` (Codex review): the filter normalizer lowers SQL + # ``IS NULL`` / ``IS NOT NULL`` to Python ``is None`` / ``is not None``; + # without these entries the AST converter raised on ``ast.Is`` / + # ``ast.IsNot`` and any DSL filter using the SQL-style spelling failed + # to plan. The downstream SQL generator renders ``is`` / ``is not`` + # against a ``None`` literal as ``IS NULL`` / ``IS NOT NULL``. + ast.Is: "is", ast.IsNot: "is not", + # DEV-1475: SQL-style ``IN`` / ``NOT IN`` with a literal-tuple RHS. + # ``_normalize_sql_filter_operators`` already lowercases ``IN`` / + # ``NOT IN`` to the Python keywords; the AST then carries ``ast.In`` + # / ``ast.NotIn`` here. The ``ast.Compare`` branch enforces the + # tuple/list-only RHS shape and validates that every element is a + # literal. + ast.In: "in", ast.NotIn: "not in", +} + + +# --------------------------------------------------------------------------- +# Public entry point +# --------------------------------------------------------------------------- + + +def parse_expr(text: str, *, allow_dunder: bool = False) -> ParsedExpr: + """Parse a Mode-B expression string into a ``ParsedExpr``. + + ``allow_dunder`` permits ``__`` in identifiers. It defaults to + ``False`` (P1: Mode-B user input rejects ``__``; use single-dot DSL + paths). The stage planner sets it ``True`` only when binding a + downstream stage against a flat ``StageSchema`` (P5/DEV-1449), whose + columns ARE the ``__``-flattened multi-hop aliases of the upstream + stage (``customers__region``). Legality there is the binder's + concern (the column must exist in the upstream schema). + + Raises: + ValueError: empty input, syntax error, unsupported AST node, + chained comparison, or ``__`` in a user identifier (unless + ``allow_dunder``). + UnknownFunctionError: function call not in + ``SCALAR_FUNCTIONS`` / ``ALL_TRANSFORMS``. + IllegalWindowInFilterError: raw ``OVER(...)`` clause anywhere + in ``text``. + """ + if not text or not text.strip(): + raise ValueError("Empty Mode-B expression.") + + # Scan for a raw window clause AFTER blanking string literals (Python + # syntax, so escapes count), so a value like ``status == 'OVER('`` or + # ``status == "x \" OVER("`` isn't mistaken for window usage (CR / Codex). + if _OVER_RE.search(_PY_STRING_LITERAL_RE.sub("", text)): + raise IllegalWindowInFilterError( + filter_expr=text, + source="raw OVER(...) is not allowed in Mode-B DSL", + suggestion=( + "use a transform instead (rank, percent_rank, dense_rank, " + "ntile, cumsum, lag, lead, time_shift, …)." + ), + ) + + preprocessed, agg_map = _preprocess_colons(text) + + try: + py_ast = ast.parse(preprocessed, mode="eval").body + except SyntaxError as e: + raise ValueError( + f"Invalid Mode-B expression {text!r}: {e}" + ) + + if not allow_dunder: + _reject_dunder_in_ast(py_ast, original=text) + + return _convert(py_ast, agg_map=agg_map, original=text) + + +def parse_filter_expr(text: str, *, allow_dunder: bool = False) -> ParsedExpr: + """Parse a Mode-B *filter* string, accepting SQL operator spellings. + + Filters historically accepted SQL-style operators (``=``, ``<>``, ``NULL``, + and the keyword forms ``AND`` / ``OR`` / ``NOT`` / ``IS`` / ``IN``) + alongside the Python spellings. This wrapper normalizes those to their + Python equivalents (string-literal-aware, so quoted contents are + untouched) and then delegates to :func:`parse_expr`. Measures / order use + ``parse_expr`` directly — only filters get the SQL-operator leniency, + matching the legacy ``parse_filter`` contract. + """ + return parse_expr(_normalize_sql_filter_operators(text), allow_dunder=allow_dunder) + + +def _normalize_sql_filter_operators(text: str) -> str: + """Rewrite SQL operator spellings to Python ones outside string literals. + + ``NULL`` → ``None``; ``IS`` / ``NOT`` / ``AND`` / ``OR`` / ``IN`` → + lowercase; standalone ``=`` → ``==``; ``<>`` → ``!=``; ``col [NOT] LIKE + 'p%'`` → ``[not ]like(col, 'p%')``. Replicated from the legacy + ``slayer.core.formula._preprocess_sql_operators`` / ``_preprocess_like`` so + the typed pipeline doesn't depend on the module DEV-1452 deletes. + """ + # LIKE runs first, on the whole string: its pattern is a quoted literal, so + # it can't be rewritten per-non-literal-part like the other operators. + text = _rewrite_sql_like(text) + # CR review: use the escape-aware Python-string matcher so backslash- + # escaped quotes don't leak ``IS`` / ``IN`` / ``AND`` rewrites into + # the string body (``"x \" IN ("``). + parts = _PY_STRING_LITERAL_RE.split(text) + literals = _PY_STRING_LITERAL_RE.findall(text) + result: List[str] = [] + for i, part in enumerate(parts): + part = re.sub(r"\bNULL\b", "None", part, flags=re.IGNORECASE) + for kw in ("IS", "NOT", "AND", "OR", "IN"): + part = re.sub(rf"\b{kw}\b", kw.lower(), part, flags=re.IGNORECASE) + part = part.replace("<>", "!=") + # SQL ``||`` concat → Python ``|`` (BitOr), reinterpreted as a + # ``concat(...)`` ScalarCall in ``_convert``. ``|`` binds tighter + # than comparisons in Python just as ``||`` does in SQL, so + # ``a || b = 'x'`` and ``a | b == 'x'`` group identically. + part = part.replace("||", "|") + result.append(part) + if i < len(literals): + result.append(literals[i]) + # DEV-1492: the `=` → `==` rewrite runs on the rejoined string with a + # call-paren-aware scanner that leaves keyword-argument `=` alone + # inside non-scalar calls (transforms / parametric aggregations). Run + # last so the scanner sees lowercased keywords and the post-`<>` / + # post-`||` text — only its own pass can touch literal-spanning + # paren context correctly. + return _rewrite_comparison_equals("".join(result)) + + +def _classify_paren( + hist: List[Tuple[str, str]], +) -> Tuple[bool, Optional[str]]: + """Classify an open ``(`` as a CALL or GROUPING paren. + + A ``(`` is a call paren when the previous significant token is a + bare identifier (callable name) or a callable-suffix token (``)`` + / ``]``). Lowercase keywords (``and`` / ``or`` / ``not`` / ``in`` + / ``is``) do NOT make the next ``(`` a call paren — that's why + ``not(...)`` and ``x in (...)`` carry grouping parens. + + DEV-1492 iteration 3: a colon-aggregation context + (``revenue:first(...)``) makes the call a parametric aggregation + regardless of the callee name. ``first`` and ``last`` sit in both + :data:`ALL_TRANSFORMS` and the built-in aggregation set + (``_AMBIGUOUS_AGG_TRANSFORMS`` in ``slayer/core/formula.py``); + after a ``:`` they are always aggregations, never transforms. + Drop the callee to ``None`` so :func:`_is_kwarg_equals` takes the + aggregation/unknown branch (kwargs preserved after ``(`` or + ``,``). + """ + prev_kind = hist[-1][0] if hist else None + prev_text = hist[-1][1] if hist else "" + is_call = prev_kind == "NAME" or prev_text in (")", "]") + callee = hist[-1][1] if (is_call and prev_kind == "NAME") else None + if callee is not None and len(hist) >= 2 and hist[-2] == ("OTHER", ":"): + callee = None + return is_call, callee + + +def _is_kwarg_equals( + stack: List[Tuple[bool, Optional[str]]], + hist: List[Tuple[str, str]], +) -> bool: + """Whether a lone ``=`` is a Python keyword-argument separator. + + Three callee classes get different treatment (DEV-1492 iteration 2): + + * **Scalar** (``callee in SCALAR_FUNCTIONS``) — never a kwarg; + scalars reject keyword args by design. + * **Transform** (``callee in ALL_TRANSFORMS``) — the first + positional is always the value to transform, so a kwarg can + only appear AFTER a ``,``. This preserves the documented + predicate-input form (``consecutive_periods(status = 'paid')`` + where the SQL ``=`` is part of the predicate, not a kwarg). + * **Aggregation or unknown** — kwarg can be the first arg + (``weighted_avg(weight=qty)``, ``percentile(p=0.5)``), so the + ``=`` may follow either ``(`` or ``,``. + """ + top = stack[-1] if stack else None + if top is None or not top[0]: + return False + callee = top[1] + # Case-insensitive, matching the scalar-call parse branch below. + if callee is not None and callee.lower() in SCALAR_FUNCTIONS: + return False + prev_kind = hist[-1][0] if hist else None + if prev_kind != "NAME": + return False + prev_prev_kind = hist[-2][0] if len(hist) >= 2 else None + if callee is not None and callee in ALL_TRANSFORMS: + return prev_prev_kind == "COMMA" + return prev_prev_kind in ("LPAREN", "COMMA") + + +def _push_hist(hist: List[Tuple[str, str]], kind: str, text: str) -> None: + """Append ``(kind, text)`` and trim ``hist`` to the last 2 entries.""" + hist.append((kind, text)) + if len(hist) > 2: + del hist[0] + + +def _handle_pass_through( + m: "re.Match[str]", + out: List[str], + stack: List[Tuple[bool, Optional[str]]], + hist: List[Tuple[str, str]], +) -> None: + """Whitespace and string literals: emit verbatim, don't touch hist.""" + out.append(m.group(0)) + + +def _handle_ident( + m: "re.Match[str]", + out: List[str], + stack: List[Tuple[bool, Optional[str]]], + hist: List[Tuple[str, str]], +) -> None: + ident = m.group(0) + out.append(ident) + _push_hist(hist, "KW" if ident in _FILTER_KEYWORDS else "NAME", ident) + + +def _handle_other( + m: "re.Match[str]", + out: List[str], + stack: List[Tuple[bool, Optional[str]]], + hist: List[Tuple[str, str]], +) -> None: + """Numbers, compound ops (``==``/``<=``/``>=``/``!=``), brackets, and + any single char not otherwise classified (``:``, ``.``, ``+``, ``-``, + ``*``, ``/``, ``%``, ``<``, ``>``, ``!``, ``|``, ``{``, ``}``, ...).""" + text = m.group(0) + out.append(text) + _push_hist(hist, "OTHER", text) + + +def _handle_comma( + m: "re.Match[str]", + out: List[str], + stack: List[Tuple[bool, Optional[str]]], + hist: List[Tuple[str, str]], +) -> None: + out.append(",") + _push_hist(hist, "COMMA", ",") + + +def _handle_lparen( + m: "re.Match[str]", + out: List[str], + stack: List[Tuple[bool, Optional[str]]], + hist: List[Tuple[str, str]], +) -> None: + stack.append(_classify_paren(hist)) + out.append("(") + _push_hist(hist, "LPAREN", "(") + + +def _handle_rparen( + m: "re.Match[str]", + out: List[str], + stack: List[Tuple[bool, Optional[str]]], + hist: List[Tuple[str, str]], +) -> None: + if stack: + stack.pop() + out.append(")") + _push_hist(hist, "OTHER", ")") + + +def _handle_op_eq( + m: "re.Match[str]", + out: List[str], + stack: List[Tuple[bool, Optional[str]]], + hist: List[Tuple[str, str]], +) -> None: + out.append("=" if _is_kwarg_equals(stack, hist) else "==") + _push_hist(hist, "OTHER", "=") + + +_HANDLERS: Dict[str, Any] = { + "ws": _handle_pass_through, + "string": _handle_pass_through, + "ident": _handle_ident, + "number": _handle_other, + "op_eq2": _handle_other, + "op_eq": _handle_op_eq, + "lparen": _handle_lparen, + "rparen": _handle_rparen, + "lbrack": _handle_other, + "rbrack": _handle_other, + "comma": _handle_comma, + "other": _handle_other, +} + + +def _rewrite_comparison_equals(text: str) -> str: + """Rewrite SQL-style ``=`` to Python ``==`` except where Python would + treat the ``=`` as a keyword argument inside a non-scalar call. + + Per the architecture (``docs/architecture/parsing.md`` — scalars + reject kwargs, only ``AggCall`` / ``TransformCall`` carry kwargs), + a lone ``=`` is preserved (kwarg) iff: + + 1. the innermost open paren is a CALL paren (see + :func:`_classify_paren`), + 2. the callee of that innermost call is NOT in + :data:`SCALAR_FUNCTIONS` — scalars never accept kwargs, so a + ``=`` inside them is the user's SQL comparison + (``coalesce(status = 'paid', False)``), + 3. the ``=`` is immediately preceded (skipping whitespace) by an + identifier preceded (skipping whitespace) by the call's ``(`` + or by a ``,`` at that paren depth — Python's keyword-argument + grammar (see :func:`_is_kwarg_equals`). + + Compound operators (``==``, ``<=``, ``>=``, ``!=``) are emitted + verbatim by the tokenizer so their ``=`` is never touched. String + literals are tokenized as a unit (Python single/double-quoted with + backslash escapes) and pass through without perturbing the paren + stack or the previous-significant-token history. + + Token-class dispatch goes through :data:`_HANDLERS` keyed by the + regex's ``lastgroup`` (each token-class group is named and the + arms are mutually exclusive, so ``lastgroup`` is exactly the + matched arm). + """ + out: List[str] = [] + # Each frame: (is_call, callee). ``callee`` is the identifier text + # preceding the call's ``(``, or ``None`` (e.g. a callable expression + # like ``f()(x=1)`` where the prior token is ``)``). + stack: List[Tuple[bool, Optional[str]]] = [] + # Trailing window of the last 2 significant tokens (oldest-first). + # Categories: NAME (bare identifier), KW (lowercase Python keyword), + # LPAREN, COMMA, OTHER (everything else; string literals don't enter + # the history). + hist: List[Tuple[str, str]] = [] + for m in _SCAN_TOKEN_RE.finditer(text): + _HANDLERS[m.lastgroup](m, out, stack, hist) + return "".join(out) + + +# --------------------------------------------------------------------------- +# Reference walk (best-effort textual extraction) +# --------------------------------------------------------------------------- + + +def walk_parsed_refs( + parsed: ParsedExpr, +) -> Iterator[Union[Ref, DottedRef, AggCall]]: + """Yield every reference-bearing leaf node in a ``ParsedExpr`` tree. + + Yields ``Ref`` (bare identifier), ``DottedRef`` (dotted join path), and + ``AggCall`` (colon-syntax aggregation) nodes — the leaves a formula + actually references. This is the scope-free counterpart to the binder's + ``walk_value_keys``: callers that only need the *names* a formula touches + (schema-drift cascade attribution, memory entity tagging) walk the parse + tree directly instead of binding it against a scope. + + Descent rules (chosen to match the legacy ``parse_formula`` / + ``FieldSpec`` walk exactly): + + * ``AggCall`` is yielded as a unit — the aggregation's source / args / + kwargs are NOT descended (``weighted_avg(weight=quantity)`` surfaces + ``price``, never ``quantity``). + * ``TransformCall`` descends ``input`` only; positional args, kwargs, and + ``partition_by`` columns are opaque. + * ``ScalarCall`` descends every positional arg (``coalesce`` / ``nullif`` + wrapping aggregated or bare refs). + * ``Arith`` / ``UnaryOp`` / ``Cmp`` / ``BoolOp`` descend their operands. + * ``Literal`` and ``StarSource`` yield nothing. + """ + if isinstance(parsed, (Ref, DottedRef, AggCall)): + yield parsed + return + if isinstance(parsed, TransformCall): + yield from walk_parsed_refs(parsed.input) + return + if isinstance(parsed, ScalarCall): + for a in parsed.args: + yield from walk_parsed_refs(a) + return + if isinstance(parsed, Arith): + yield from walk_parsed_refs(parsed.left) + yield from walk_parsed_refs(parsed.right) + return + if isinstance(parsed, Cmp): + yield from walk_parsed_refs(parsed.left) + yield from walk_parsed_refs(parsed.right) + return + if isinstance(parsed, UnaryOp): + yield from walk_parsed_refs(parsed.operand) + return + if isinstance(parsed, BoolOp): + for op in parsed.operands: + yield from walk_parsed_refs(op) + return + # Literal / StarSource / TupleLit → no references. + # ``TupleLit`` carries only ``Literal`` elements by construction + # (see the ``ast.Compare`` branch of ``_convert``) so the walk stops + # here without descending — same shape as ``Literal``. + + +# --------------------------------------------------------------------------- +# Internals +# --------------------------------------------------------------------------- + + +def _reject_dunder_in_ast(node: ast.AST, *, original: str) -> None: + """Walk the parsed AST and reject any user identifier containing ``__``. + + Robust to string literals (they're ``ast.Constant`` nodes, not + identifier nodes) and to placeholder names generated by the colon + preprocessor (filtered by ``_PLACEHOLDER_PREFIX``). Walks ``Name`` + (``foo__bar``), ``Attribute.attr`` (``customers.foo__bar``), and + ``keyword.arg`` (``f(weight__bad=…)``). + """ + def _check(token: str) -> None: + if "__" in token and not token.startswith(_PLACEHOLDER_PREFIX): + raise ValueError( + f"Mode-B expression {original!r} contains double-" + f"underscore in identifier {token!r}: `__` is reserved " + f"for internal join-path aliases on the SQL side. Use " + f"single-dot DSL paths (e.g. `customers.region`) in " + f"queries and ModelMeasure formulas." + ) + + for child in ast.walk(node): + if isinstance(child, ast.Name): + _check(child.id) + elif isinstance(child, ast.Attribute): + _check(child.attr) + elif isinstance(child, ast.keyword) and child.arg is not None: + _check(child.arg) + + +def _preprocess_colons( + text: str, +) -> Tuple[str, Dict[int, Tuple[Union[Ref, DottedRef, StarSource], str]]]: + """Replace ``:`` with placeholder identifiers. + + Captures source kind + agg name. Any trailing ``(args)`` is left in + place so Python's AST parses it naturally as a Call. String literal + spans are skipped — the literal text is user data, not DSL syntax. + """ + agg_map: Dict[int, Tuple[Union[Ref, DottedRef, StarSource], str]] = {} + counter = [0] + literal_spans = [ + # CR review: use the escape-aware matcher so backslash-escaped + # quotes don't leak ``:sum`` colon rewrites into the string body. + (m.start(), m.end()) for m in _PY_STRING_LITERAL_RE.finditer(text) + ] + + def _in_literal(pos: int) -> bool: + return any(s <= pos < e for s, e in literal_spans) + + def _replace(match: re.Match) -> str: + if _in_literal(match.start()): + return match.group(0) + source_str = match.group(1) + agg_name = match.group(2) + source: Union[Ref, DottedRef, StarSource] + if source_str == "*": + source = StarSource() + elif "." in source_str: + source = DottedRef(parts=tuple(source_str.split("."))) + else: + source = Ref(name=source_str) + idx = counter[0] + counter[0] += 1 + agg_map[idx] = (source, agg_name) + return f"{_PLACEHOLDER_PREFIX}{idx}__" + + return _COLON_AGG_RE.sub(_replace, text), agg_map + + +def _convert(node: ast.AST, *, agg_map: Dict, original: str) -> ParsedExpr: # NOSONAR(S3776) — one-pass dispatch over ast node kinds (Constant/Name/Compare/BinOp/UnaryOp/BoolOp/Call/Attribute…) producing typed ParsedExpr; the branches are flat and short, and splitting hides the exhaustive ast-kind coverage one read scans for. Surfaces ParsedExpr.kind contract directly. + if isinstance(node, ast.Constant): + return _convert_constant(node, original=original) + + if isinstance(node, ast.Name): + m = _PLACEHOLDER_RE.match(node.id) + if m: + idx = int(m.group(1)) + source, agg = agg_map[idx] + return AggCall(source=source, agg=agg) + return Ref(name=node.id) + + if isinstance(node, ast.Attribute): + parts = _flatten_attribute(node, original=original) + return DottedRef(parts=tuple(parts)) + + if isinstance(node, ast.Call): + return _convert_call(node, agg_map=agg_map, original=original) + + if isinstance(node, ast.BinOp): + op_type = type(node.op) + if op_type is ast.BitOr: + # SQL ``||`` concat operator (``parse_filter_expr`` normalizes + # ``||`` → ``|``). Desugar to the existing ``concat`` scalar + # call so binding + per-dialect SQL emission are fully reused. + return ScalarCall( + name="concat", + args=( + _convert(node.left, agg_map=agg_map, original=original), + _convert(node.right, agg_map=agg_map, original=original), + ), + ) + if op_type not in _BIN_OP_MAP: + raise ValueError( + f"Invalid Mode-B expression {original!r}: unsupported " + f"binary operator {op_type.__name__}." + ) + return Arith( + op=_BIN_OP_MAP[op_type], + left=_convert(node.left, agg_map=agg_map, original=original), + right=_convert(node.right, agg_map=agg_map, original=original), + ) + + if isinstance(node, ast.UnaryOp): + op_type = type(node.op) + if op_type is ast.USub: + return UnaryOp( + op="-", + operand=_convert(node.operand, agg_map=agg_map, original=original), + ) + if op_type is ast.UAdd: + # `+x` is a no-op; collapse to the operand directly. + return _convert(node.operand, agg_map=agg_map, original=original) + if op_type is ast.Not: + return UnaryOp( + op="not", + operand=_convert(node.operand, agg_map=agg_map, original=original), + ) + raise ValueError( + f"Invalid Mode-B expression {original!r}: unsupported unary " + f"operator {op_type.__name__}." + ) + + if isinstance(node, ast.Compare): + if len(node.ops) != 1 or len(node.comparators) != 1: + raise ValueError( + f"Invalid Mode-B expression {original!r}: chained " + f"comparisons are not supported. Each Cmp must be a " + f"single comparison; split (e.g.) `1 < x < 10` into " + f"`1 < x and x < 10`." + ) + op_type = type(node.ops[0]) + if op_type not in _CMP_OP_MAP: + raise ValueError( + f"Invalid Mode-B expression {original!r}: unsupported " + f"comparison operator {op_type.__name__}." + ) + # DEV-1475: ``IN`` / ``NOT IN`` carry a literal-only tuple RHS + # (``status in ('completed', 'pending')``). Reject scalar RHS, + # empty RHS, and any non-literal element so the binder can fold + # the predicate into a single ``InKey`` with confidence. Signed + # numerics (``amount in (-1, -2)``) are admitted by collapsing + # ``UnaryOp(USub/UAdd, Constant(int|float))`` to a signed + # ``Literal`` at validation time (Codex review). + if op_type in (ast.In, ast.NotIn): + rhs_node = node.comparators[0] + if not isinstance(rhs_node, (ast.Tuple, ast.List)): + raise ValueError( + f"Invalid Mode-B expression {original!r}: the right-" + f"hand side of ``in`` / ``not in`` must be a tuple/" + f"list literal (e.g. ``status in ('a', 'b')``); got " + f"{type(rhs_node).__name__}." + ) + if not rhs_node.elts: + raise ValueError( + f"Invalid Mode-B expression {original!r}: empty " + f"tuple is not allowed on the right-hand side of " + f"``in`` / ``not in`` (dialect-dependent SQL); use " + f"a non-empty literal tuple." + ) + elements: List[Literal] = [] + for elt in rhs_node.elts: + converted = _convert_in_rhs_element( + elt, agg_map=agg_map, original=original, + ) + elements.append(converted) + return Cmp( + op=_CMP_OP_MAP[op_type], + left=_convert(node.left, agg_map=agg_map, original=original), + right=TupleLit(elements=tuple(elements)), + ) + return Cmp( + op=_CMP_OP_MAP[op_type], + left=_convert(node.left, agg_map=agg_map, original=original), + right=_convert(node.comparators[0], agg_map=agg_map, original=original), + ) + + if isinstance(node, ast.BoolOp): + op_str = "and" if isinstance(node.op, ast.And) else "or" + operands = tuple( + _convert(v, agg_map=agg_map, original=original) for v in node.values + ) + return BoolOp(op=op_str, operands=operands) + + raise ValueError( + f"Invalid Mode-B expression {original!r}: unsupported AST node " + f"{type(node).__name__}." + ) + + +def _convert_in_rhs_element( + node: ast.AST, *, agg_map: Dict, original: str, +) -> Literal: + """Convert one element of an ``IN`` / ``NOT IN`` literal-tuple RHS. + + The Python parser emits a negative numeric literal as + ``UnaryOp(USub, Constant(int|float))`` rather than a bare + ``Constant`` with a negative value, so a strict ``isinstance(_, + Literal)`` check against ``_convert``'s output would reject + ``amount in (-1, -2)``. This helper collapses the sign onto the + inner numeric before the literal check (Codex review). Boolean and + string literals are unaffected. + """ + if isinstance(node, ast.UnaryOp) and isinstance(node.op, (ast.USub, ast.UAdd)): + inner = node.operand + if ( + isinstance(inner, ast.Constant) + and isinstance(inner.value, (int, float)) + and not isinstance(inner.value, bool) + ): + signed = -inner.value if isinstance(node.op, ast.USub) else inner.value + if isinstance(signed, int): + return Literal(value=Decimal(signed)) + return Literal(value=Decimal(str(signed))) + converted = _convert(node, agg_map=agg_map, original=original) + if not isinstance(converted, Literal): + raise ValueError( + f"Invalid Mode-B expression {original!r}: every element on " + f"the right-hand side of ``in`` / ``not in`` must be a " + f"literal (string, number, or boolean); got " + f"{type(converted).__name__}." + ) + return converted + + +def _convert_constant(node: ast.Constant, *, original: str) -> Literal: + val = node.value + if isinstance(val, bool): + return Literal(value=val) + if val is None: + return Literal(value=None) + if isinstance(val, int): + return Literal(value=Decimal(val)) + if isinstance(val, float): + return Literal(value=Decimal(str(val))) + if isinstance(val, str): + return Literal(value=val) + raise ValueError( + f"Invalid Mode-B expression {original!r}: unsupported literal " + f"type {type(val).__name__}." + ) + + +def _flatten_attribute( + node: ast.Attribute, *, original: str, +) -> List[str]: + parts: List[str] = [node.attr] + cur: ast.AST = node.value + while isinstance(cur, ast.Attribute): + parts.append(cur.attr) + cur = cur.value + if isinstance(cur, ast.Name): + parts.append(cur.id) + else: + raise ValueError( + f"Invalid Mode-B expression {original!r}: unsupported " + f"attribute base {type(cur).__name__}." + ) + return list(reversed(parts)) + + +def _convert_kwarg_value(node: ast.AST, *, agg_map: Dict, original: str): + """Convert a call keyword-argument value. + + List / tuple values (e.g. ``partition_by=[region, channel]`` for the + rank family) convert to a tuple of converted elements so the parser + accepts the documented multi-column transform-kwarg grammar instead of + raising on the bare ``ast.List`` node; scalar values convert normally. + """ + if isinstance(node, (ast.List, ast.Tuple)): + return tuple( + _convert(e, agg_map=agg_map, original=original) for e in node.elts + ) + return _convert(node, agg_map=agg_map, original=original) + + +def _convert_call( + node: ast.Call, *, agg_map: Dict, original: str, +) -> ParsedExpr: + if not isinstance(node.func, ast.Name): + raise ValueError( + f"Invalid Mode-B expression {original!r}: function calls " + f"with non-name callee are not supported." + ) + func_name = node.func.id + + args = tuple( + _convert(a, agg_map=agg_map, original=original) for a in node.args + ) + # Reject ``**kwargs`` dictionary unpacking (``kw.arg is None``) rather + # than silently dropping it (CR) — a dropped ``**`` would change call + # semantics without warning. + if any(kw.arg is None for kw in node.keywords): + raise ValueError( + f"Invalid Mode-B expression {original!r}: dictionary unpacking " + f"(**kwargs) is not supported in calls." + ) + kwargs = tuple( + (kw.arg, _convert_kwarg_value(kw.value, agg_map=agg_map, original=original)) + for kw in node.keywords + if kw.arg is not None # guarded above; narrows kw.arg to str + ) + + # Aggregation placeholder? + m = _PLACEHOLDER_RE.match(func_name) + if m: + idx = int(m.group(1)) + source, agg = agg_map[idx] + return AggCall(source=source, agg=agg, args=args, kwargs=kwargs) + + # Transform? + if func_name in ALL_TRANSFORMS: + if not args: + raise ValueError( + f"Invalid Mode-B expression {original!r}: transform " + f"{func_name!r} requires at least one positional argument " + f"(the value to transform)." + ) + return TransformCall( + op=func_name, + input=args[0], + args=args[1:], + kwargs=kwargs, + ) + + # Scalar function? Matched case-INSENSITIVELY: SQL function names are + # case-insensitive and users write ``COALESCE(x, 0)`` as readily as + # ``coalesce(x, 0)``. The legacy parser lowercased before the allowlist + # lookup; matching exactly here rejected every SQL-cased formula. The + # name is normalised to lower case on the way into ``ScalarCall`` so the + # two spellings intern to ONE key rather than two slots computing the + # same value. + if func_name.lower() in SCALAR_FUNCTIONS: + if kwargs: + raise ValueError( + f"Invalid Mode-B expression {original!r}: scalar function " + f"{func_name!r} does not accept keyword arguments. Pass " + f"values positionally." + ) + return ScalarCall(name=func_name.lower(), args=args) + + # Otherwise — unknown. + raise UnknownFunctionError( + name=func_name, + location=original, + suggestion=( + f"Mode-B accepts only the closed scalar allowlist " + f"({sorted(SCALAR_FUNCTIONS)}), transforms " + f"({sorted(ALL_TRANSFORMS)}), and colon-syntax aggregations " + f"(e.g. `revenue:sum`). Function-style aggregations like " + f"`sum(revenue)` are normalised by the slack layer; if you " + f"see this error for one, slack normalization was bypassed." + ), + ) diff --git a/slayer/engine/variables.py b/slayer/engine/variables.py new file mode 100644 index 00000000..c7a38fe6 --- /dev/null +++ b/slayer/engine/variables.py @@ -0,0 +1,107 @@ +"""Stage 7b.1 (DEV-1450) — variable substitution in the new pipeline. + +Moves the ``{var}`` placeholder substitution that previously lived inside +the legacy enrichment path (``slayer.engine.enrichment.py:1162``) into a +small, pipeline-friendly module. + +Public surface: + +- :func:`merge_query_variables` collapses the four configured variable + layers (model defaults < outer query < stage query < runtime kwarg) + into the effective dict that populates + ``ResolvedSourceBundle.query_variables``. Precedence: runtime > stage > + outer > model_defaults. +- :func:`apply_variables_to_query` returns a copy of the input + ``SlayerQuery`` with ``{var}`` substituted in its ``filters`` list. The + helper always returns a fresh ``SlayerQuery`` instance for predictable + pipeline semantics. ``dry_run_placeholders=True`` fills any unresolved + valid placeholder with the legacy ``"0"`` sentinel instead of raising + — used by save-time dry-run SQL generation. Invalid placeholder names + still raise regardless of ``dry_run_placeholders``. + +Scope deliberately matches the legacy enrichment scope — +``SlayerQuery.filters`` is the only field this helper substitutes into. +Formula text, ``Column.sql``, ``Column.filter``, and +``SlayerModel.filters`` are NOT variable-substituted today, and this +module preserves that contract. + +The dormant module is unwired from the engine in this commit; stage +7b.15 (engine cutover) makes it the substitution path used by +``engine.execute`` and ``engine.save_model``. +""" + +from __future__ import annotations + +from typing import Any, Dict, Optional + +from slayer.core.query import ( + SlayerQuery, + extract_placeholder_names, + substitute_variables, +) + +_PLACEHOLDER_FILL_VALUE = "0" + + +def merge_query_variables( + *, + runtime: Optional[Dict[str, Any]], + stage: Optional[Dict[str, Any]], + outer: Optional[Dict[str, Any]], + model_defaults: Optional[Dict[str, Any]], +) -> Dict[str, Any]: + """Collapse the four variable layers into the effective dict. + + Precedence (highest wins): runtime > stage > outer > model_defaults. + ``None`` and empty-dict layers are identities. + """ + return { + **(model_defaults or {}), + **(outer or {}), + **(stage or {}), + **(runtime or {}), + } + + +def apply_variables_to_query( + *, + query: SlayerQuery, + variables: Optional[Dict[str, Any]] = None, + dry_run_placeholders: bool = False, +) -> SlayerQuery: + """Return a copy of ``query`` with ``{var}`` substituted in ``filters``. + + The returned ``SlayerQuery`` is always a fresh instance, including in + the no-op cases (``query.filters`` is ``None`` / empty / contains no + placeholders). ``variables=None`` is normalized to an empty dict. + When ``dry_run_placeholders=True``, unresolved valid placeholders are + filled with ``"0"`` instead of raising — the legacy save-time + dry-run behaviour. Invalid placeholder names still raise + ``ValueError`` regardless of ``dry_run_placeholders``, because the + dry-run shortcut is for missing *values*, not for bypassing name + validation. + """ + if query.filters is None: + return query.model_copy() + + effective: Dict[str, Any] = dict(variables or {}) + if dry_run_placeholders: + for placeholder in extract_placeholder_names(query): + effective.setdefault(placeholder, _PLACEHOLDER_FILL_VALUE) + + # Mode-B (Python-AST) query filters take the ``python`` escaping regime — + # SQL quote-doubling would silently corrupt a value via adjacent-literal + # concatenation once the AST layer re-renders it (DEV-1625). + substituted = [ + substitute_variables(filter_str=f, variables=effective, escape="python") + for f in query.filters + ] + return query.model_copy(update={"filters": substituted}) + + +__all__ = [ + "apply_variables_to_query", + "extract_placeholder_names", + "merge_query_variables", + "substitute_variables", +] diff --git a/slayer/facade/catalog.py b/slayer/facade/catalog.py index 86563141..6b9bb3d4 100644 --- a/slayer/facade/catalog.py +++ b/slayer/facade/catalog.py @@ -608,7 +608,7 @@ def _agg_output_type(*, column: Column, agg: str) -> DataType | None: wire schema is always derived from the actual ``LIMIT 0`` execution (§5.3), so any inference here is informational. """ - if agg in {"count", "count_distinct"}: + if agg in {"count", "count_distinct", "count_distinct_approx"}: return DataType.INT if agg in {"sum"}: # SUM(INT) → INT for SQLite/Postgres; SUM(DOUBLE) → DOUBLE. diff --git a/slayer/mcp/server.py b/slayer/mcp/server.py index 1fa09ee6..d7e4ab37 100644 --- a/slayer/mcp/server.py +++ b/slayer/mcp/server.py @@ -1222,9 +1222,9 @@ async def edit_model( "engine-managed (auto-derived from the backing query)." ) # Strip cache fields before save so engine.save_model can repopulate - # them from a fresh _query_as_model pass. (These are present here - # only because they were on the existing stored model, not from - # this edit.) + # them from a fresh expansion of the backing query. (These are + # present here only because they were on the existing stored + # model, not from this edit.) validated = validated.model_copy(update={ "columns": [], "backing_query_sql": None, diff --git a/slayer/memories/resolver.py b/slayer/memories/resolver.py index ab22c211..324817ec 100644 --- a/slayer/memories/resolver.py +++ b/slayer/memories/resolver.py @@ -33,19 +33,24 @@ from pydantic import BaseModel -from slayer.core.enums import BUILTIN_AGGREGATIONS -from slayer.core.errors import AmbiguousModelError, EntityResolutionError -from slayer.core.formula import ( - AggregatedMeasureRef, - ArithmeticField, - FieldSpec, - MixedArithmeticField, - TransformField, - parse_formula, +from slayer.core.errors import ( + AmbiguousModelError, + EntityResolutionError, + UnknownFunctionError, ) from slayer.core.models import SlayerModel from slayer.core.query import ColumnRef, SlayerQuery, TimeDimension from slayer.core.refs import strip_agg_suffix as _strip_agg_suffix +from slayer.engine.agg_registry import collect_reachable_agg_names +from slayer.engine.normalization import func_style_agg_to_colon +from slayer.engine.syntax import ( + AggCall, + ParsedExpr, + Ref, + StarSource, + parse_expr, + walk_parsed_refs, +) from slayer.memories.models import ( MEMORY_CANONICAL_PREFIX as _MEMORY_PREFIX, _validate_memory_id_charset, @@ -423,17 +428,34 @@ async def resolve_entity( # NOSONAR(S3776) — single linear dispatch matching # --------------------------------------------------------------------------- -def _formula_aggregated_refs(field: FieldSpec) -> Iterable[AggregatedMeasureRef]: - """Yield every ``AggregatedMeasureRef`` reachable inside ``field``.""" - if isinstance(field, AggregatedMeasureRef): - yield field - elif isinstance(field, TransformField): - yield from _formula_aggregated_refs(field.inner) - elif isinstance(field, (ArithmeticField, MixedArithmeticField)): - yield from field.agg_refs.values() - if isinstance(field, MixedArithmeticField): - for _ph, sub in field.sub_transforms: - yield from _formula_aggregated_refs(sub) +def _formula_entity_tokens(parsed: ParsedExpr) -> Iterable[str]: + """Yield every entity *token* referenced by a parsed Mode-B formula. + + Colon-syntax aggregations surface as ``":"`` (``*:count`` + included verbatim), bare and dotted refs surface as their textual form. + Each token is fed one-by-one into ``resolve_entity`` downstream, which + canonicalises it (stripping the agg suffix, collapsing ``*:count`` to the + model, walking dotted join paths). No binding — the resolver does its own + resolution. + + Aggregation args / kwargs (e.g. ``weighted_avg(weight=quantity)``) and + transform partition columns are opaque (legacy parity) — only the + aggregated source / inner value surfaces. + """ + for node in walk_parsed_refs(parsed): + if isinstance(node, AggCall): + source = node.source + if isinstance(source, StarSource): + src_name = "*" + elif isinstance(source, Ref): + src_name = source.name + else: # DottedRef + src_name = ".".join(source.parts) + yield f"{src_name}:{node.agg}" + elif isinstance(node, Ref): + yield node.name + else: # DottedRef + yield ".".join(node.parts) _FILTER_AGG_SUFFIX_RE = re.compile(r":\w+(?:\([^)]*\))?") @@ -539,28 +561,49 @@ def _add(forms: Iterable[str]) -> None: _add(result.canonical_forms) warnings.extend(result.warnings) - # 4. measures — parse each formula and walk for AggregatedMeasureRef. - named_measures = { - m.name: m.formula - for m in source_model.measures - if m.name is not None - } - extra_agg_names = frozenset( - a.name for a in source_model.aggregations - ) | BUILTIN_AGGREGATIONS + # 4. measures — parse each formula (Mode-B DSL) and resolve each + # referenced entity token. Function-style aggregations are rewritten to + # colon syntax first (quiet FUNC_STYLE_AGG slack helper). DEV-1500 — the + # custom-agg name set includes aggregations defined on *joined* models + # too, so entity extraction over a formula like + # ``rolling_avg(customers.score)`` (where ``rolling_avg`` lives on + # ``customers``) still produces the ``customers.score`` token instead of + # falling through to the unknown-function branch. + async def _resolve_join_target_for_resolver( + target_model_name: str, named_queries, # noqa: ARG001 + ): + try: + target = await storage.get_model( + target_model_name, data_source=source_model.data_source, + ) + except Exception: # noqa: BLE001 — best-effort; resolver must not raise + return None + if target is None: + return None + return (None, target) + + try: + reachable_agg_names = await collect_reachable_agg_names( + source_model=source_model, + resolve_join_target=_resolve_join_target_for_resolver, + named_queries={}, + ) + except Exception: # noqa: BLE001 — never let the walk break extraction + reachable_agg_names = None + custom_agg_names = reachable_agg_names or frozenset() for m in query.measures or []: if m.formula is None: continue try: - parsed = parse_formula( - m.formula, - extra_agg_names=extra_agg_names, - named_measures=named_measures or None, + parsed = parse_expr( + func_style_agg_to_colon( + m.formula, custom_agg_names=custom_agg_names + ) ) - except ValueError: - # Formula didn't parse as colon syntax — fall back to - # treating the bare formula text as an entity reference - # (handles ``formula="aov"``-style refs to named measures). + # IllegalWindowInFilterError is-a ValueError, so ValueError covers it. + except (ValueError, UnknownFunctionError): + # Not parseable as Mode-B DSL — fall back to treating the whole + # formula text as a single entity reference. result = await resolve_entity( m.formula, storage=storage, @@ -569,14 +612,9 @@ def _add(forms: Iterable[str]) -> None: _add(result.canonical_forms) warnings.extend(result.warnings) continue - for ref in _formula_aggregated_refs(parsed): - agg_token = ( - f"{ref.measure_name}:{ref.aggregation_name}" - if ref.aggregation_name - else ref.measure_name - ) + for token in _formula_entity_tokens(parsed): result = await resolve_entity( - agg_token, + token, storage=storage, source_model=source_model, ) diff --git a/slayer/pg_facade/connection.py b/slayer/pg_facade/connection.py index d649a550..420af3a9 100644 --- a/slayer/pg_facade/connection.py +++ b/slayer/pg_facade/connection.py @@ -20,6 +20,7 @@ import struct import time from collections.abc import Awaitable, Callable, Iterable +from typing import Iterator, List, Optional, Tuple import sqlglot import sqlglot.errors @@ -72,6 +73,154 @@ _BACKEND_SECRET = 0 _PARAM_PLACEHOLDER = re.compile(r"\$(\d+)") + +def _skip_line_comment(sql: str, i: int) -> int: + """``-- … newline`` (or EOF).""" + nl = sql.find("\n", i + 2) + return len(sql) if nl < 0 else nl + 1 + + +def _skip_block_comment(sql: str, i: int) -> int: + """``/* … */``, Postgres-style nested.""" + n = len(sql) + depth = 1 + i += 2 + while i < n and depth > 0: + if sql[i] == "/" and i + 1 < n and sql[i + 1] == "*": + depth += 1 + i += 2 + elif sql[i] == "*" and i + 1 < n and sql[i + 1] == "/": + depth -= 1 + i += 2 + else: + i += 1 + return i + + +def _skip_unquoted_identifier(sql: str, i: int) -> int: + """Postgres: letter|_ then alnum|_|$. Treats ``metric$1`` as one token.""" + n = len(sql) + i += 1 + while i < n and (sql[i].isalnum() or sql[i] in ("_", "$")): + i += 1 + return i + + +def _skip_e_string(sql: str, i: int) -> int: + """``E'…'`` / ``e'…'`` — backslash escapes AND ``''`` escape.""" + n = len(sql) + i += 2 # skip prefix + opening quote + while i < n: + if sql[i] == "\\" and i + 1 < n: + i += 2 + continue + if sql[i] == "'": + if i + 1 < n and sql[i + 1] == "'": + i += 2 + continue + return i + 1 + i += 1 + return i + + +def _skip_single_quoted_string(sql: str, i: int) -> int: + """``'…'`` — only the ``''`` doubled-quote escape (no backslash).""" + n = len(sql) + i += 1 + while i < n: + if sql[i] == "'": + if i + 1 < n and sql[i + 1] == "'": + i += 2 + continue + return i + 1 + i += 1 + return i + + +def _skip_double_quoted_identifier(sql: str, i: int) -> int: + """``"…"`` — only the ``""`` escape.""" + n = len(sql) + i += 1 + while i < n: + if sql[i] == '"': + if i + 1 < n and sql[i + 1] == '"': + i += 2 + continue + return i + 1 + i += 1 + return i + + +def _try_skip_dollar_quoted(sql: str, i: int) -> Optional[int]: + """If sql[i:] opens a ``$tag$ … $tag$`` literal, return the index just + past the closing tag; otherwise return None (so the caller can fall + through to ``$N`` placeholder matching). Requires a word boundary + before the opening ``$`` — otherwise ``ident$1`` would be misread.""" + n = len(sql) + prev = sql[i - 1] if i > 0 else "" + if prev.isalnum() or prev == "_": + return None + j = i + 1 + while j < n and (sql[j].isalnum() or sql[j] == "_"): + j += 1 + if j >= n or sql[j] != "$": + return None + tag = sql[i : j + 1] + end = sql.find(tag, j + 1) + return n if end < 0 else end + len(tag) + + +def _handle_dollar( + sql: str, i: int, +) -> Tuple[int, Optional[Tuple[int, int, int]]]: + """Resolve a ``$`` at ``i`` to either a dollar-quoted-string skip or a + ``$N`` placeholder match (or a single-char advance if neither). Returns + ``(new_i, placeholder_or_None)`` so the main walker stays a flat + dispatch.""" + dq_end = _try_skip_dollar_quoted(sql, i) + if dq_end is not None: + return dq_end, None + m = _PARAM_PLACEHOLDER.match(sql, i) + if m: + return m.end(), (int(m.group(1)), m.start(), m.end()) + return i + 1, None + + +def _iter_param_placeholders(sql: str) -> Iterator[Tuple[int, int, int]]: + """Yield ``(param_index, start, end)`` for every ``$N`` placeholder in + ``sql`` that is **not** inside a string literal, quoted identifier, + dollar-quoted string, or line/block comment. + + Mirrors Postgres' tokenizer well enough for standard-shape SQL coming + from libpq / asyncpg / psql / JDBC clients. Closes the regex-only path + that rewrote ``$N`` tokens inside literals and comments (Codex review on + PR #153). Per-lexical-context skip helpers (``_skip_*`` / ``_handle_*``) + own the state-machine; the main loop is a flat ``c → helper`` dispatch. + """ + i = 0 + n = len(sql) + while i < n: + c = sql[i] + if c == "-" and i + 1 < n and sql[i + 1] == "-": + i = _skip_line_comment(sql, i) + elif c == "/" and i + 1 < n and sql[i + 1] == "*": + i = _skip_block_comment(sql, i) + elif c in ("E", "e") and i + 1 < n and sql[i + 1] == "'": + i = _skip_e_string(sql, i) + elif c.isalpha() or c == "_": + i = _skip_unquoted_identifier(sql, i) + elif c == "'": + i = _skip_single_quoted_string(sql, i) + elif c == '"': + i = _skip_double_quoted_identifier(sql, i) + elif c == "$": + i, ph = _handle_dollar(sql, i) + if ph is not None: + yield ph + else: + i += 1 + + # Strips characteristics off a statement-initial ``BEGIN`` / ``START # TRANSACTION`` (``READ ONLY``, ``ISOLATION LEVEL …``, ``DEFERRABLE`` …) so the # sqlglot-based simple-query splitter — which rejects those forms — can still @@ -682,13 +831,19 @@ def _substitute_params(self, stmt: _PreparedStatement, bind: proto.BindMessage) ) literals.append(literal_for_substitution(value)) - def repl(match: "re.Match[str]") -> str: - idx = int(match.group(1)) + # Walk placeholders in lexical order, skipping ones inside string + # literals / quoted identifiers / dollar-quoted strings / comments. + parts: List[str] = [] + last = 0 + for idx, start, end in _iter_param_placeholders(stmt.sql): + parts.append(stmt.sql[last:start]) if 1 <= idx <= len(literals): - return literals[idx - 1] - return match.group(0) - - return _PARAM_PLACEHOLDER.sub(repl, stmt.sql) + parts.append(literals[idx - 1]) + else: + parts.append(stmt.sql[start:end]) + last = end + parts.append(stmt.sql[last:]) + return "".join(parts) def _empty_string_null_params_for_bind( self, *, sql: str, raw_values, oids: list[int], @@ -1482,7 +1637,7 @@ def _resolve_param_oids(stmt: _PreparedStatement) -> list[int]: """ declared = stmt.parameter_oids max_idx = max( - (int(m.group(1)) for m in _PARAM_PLACEHOLDER.finditer(stmt.sql)), + (idx for idx, _, _ in _iter_param_placeholders(stmt.sql)), default=0, ) count = max(len(declared), max_idx) diff --git a/slayer/sql/dialects/_alias_mangle.py b/slayer/sql/dialects/_alias_mangle.py deleted file mode 100644 index 382e30db..00000000 --- a/slayer/sql/dialects/_alias_mangle.py +++ /dev/null @@ -1,58 +0,0 @@ -"""DEV-1571: shared dotted-alias encoder/decoder. - -Used by ``BigqueryDialect`` (backtick-anchored regex) and ``TsqlDialect`` -(bracket-anchored regex). The two dialects need IDENTICAL bijective -encode/decode logic: BigQuery rejects dotted output-column names; T-SQL's -``ORDER BY`` parser does not resolve bracketed dotted identifiers as SELECT -aliases. The fix is the same — mangle ``.`` to ``___`` on emit, decode on -result-row keys. - -The bijection's only domain constraint is that ``decode_alias`` is the -inverse of ``encode_alias`` ONLY on the latter's image. A key like -``my___metric`` (no dot in the original) is OUTSIDE the image — calling -``decode_alias`` on it would corrupt the value to ``my.metric``. This -constraint never bites in practice because SLayer's projection aliases -are always model-qualified (``.``), so they always contain -at least one dot and always pass through ``encode_alias``. -""" - -from __future__ import annotations - - -_ALIAS_SEP = "___" - - -def encode_alias(alias: str) -> str: - """Forward encode: escape any pre-existing ``___`` to ``______``, then - map ``.`` to ``___``. - - Inverse is the left-to-right walker in :func:`decode_alias` which - consumes the longer ``______`` token BEFORE the shorter ``___``. - """ - return alias.replace(_ALIAS_SEP, _ALIAS_SEP * 2).replace(".", _ALIAS_SEP) - - -def decode_alias(key: str) -> str: - """Reverse decode of :func:`encode_alias`. - - Walks ``key`` left-to-right, consuming the escape-doubled ``______`` - BEFORE the plain ``___`` so the two encodings stay unambiguous. - - Domain constraint: inverse of ``encode_alias`` only on its image. - See module docstring. - """ - out: list[str] = [] - i = 0 - n = len(key) - esc = _ALIAS_SEP * 2 - while i < n: - if key.startswith(esc, i): - out.append(_ALIAS_SEP) - i += len(esc) - elif key.startswith(_ALIAS_SEP, i): - out.append(".") - i += len(_ALIAS_SEP) - else: - out.append(key[i]) - i += 1 - return "".join(out) diff --git a/slayer/sql/dialects/_tier2.py b/slayer/sql/dialects/_tier2.py index c5014e69..c49b4cb3 100644 --- a/slayer/sql/dialects/_tier2.py +++ b/slayer/sql/dialects/_tier2.py @@ -35,6 +35,13 @@ class RedshiftDialect(SqlDialect): log10_native: bool = True log2_native: bool = False + def build_null_safe_eq( + self, left: exp.Expression, right: exp.Expression, + ) -> exp.Expression: + """DEV-1708: Redshift (Postgres 8.0.2 fork) has no ``IS NOT DISTINCT + FROM`` — emit the expanded ``a = b OR (a IS NULL AND b IS NULL)``.""" + return self._expanded_null_safe_eq(left, right) + def build_approx_count_distinct( self, col_sql: str, @@ -128,6 +135,13 @@ class OracleDialect(SqlDialect): log10_native: bool = False log2_native: bool = False + def build_null_safe_eq( + self, left: exp.Expression, right: exp.Expression, + ) -> exp.Expression: + """DEV-1708: Oracle has no ``IS NOT DISTINCT FROM`` — emit the expanded + ``a = b OR (a IS NULL AND b IS NULL)``.""" + return self._expanded_null_safe_eq(left, right) + def build_approx_count_distinct( self, col_sql: str, diff --git a/slayer/sql/dialects/base.py b/slayer/sql/dialects/base.py index 7af32f60..6bb8a8a2 100644 --- a/slayer/sql/dialects/base.py +++ b/slayer/sql/dialects/base.py @@ -199,6 +199,37 @@ def backslash_escapes_strings(self) -> bool: """ return _sqlglot_backslash_escapes(self.sqlglot_name) + # ------------------------------------------------------------------ + # Null-safe equality (DEV-1708 / Codex F2) + # ------------------------------------------------------------------ + + def build_null_safe_eq( + self, left: exp.Expression, right: exp.Expression, + ) -> exp.Expression: + """A null-safe equality (``left`` and ``right`` compare equal, and two + NULLs compare equal) for the cross-model grain join-back's ``ON`` clause. + + Base (Postgres-family) uses sqlglot's ``NullSafeEQ`` → ``IS NOT DISTINCT + FROM``, which sqlglot also transpiles correctly for DuckDB / Snowflake / + BigQuery / Trino / Databricks / ClickHouse. MySQL overrides to ``<=>``; + SQLite to bare ``IS``; dialects with no native form (T-SQL / Oracle / + Redshift) to the expanded ``a = b OR (a IS NULL AND b IS NULL)``. + """ + return exp.NullSafeEQ(this=left, expression=right) + + @staticmethod + def _expanded_null_safe_eq( + left: exp.Expression, right: exp.Expression, + ) -> exp.Expression: + """``left = right OR (left IS NULL AND right IS NULL)`` — the portable + expansion for dialects without a native null-safe equality operator.""" + eq = exp.EQ(this=left.copy(), expression=right.copy()) + both_null = exp.And( + this=exp.Is(this=left.copy(), expression=exp.Null()), + expression=exp.Is(this=right.copy(), expression=exp.Null()), + ) + return exp.paren(exp.Or(this=eq, expression=exp.paren(both_null))) + # ------------------------------------------------------------------ # Date-trunc / time arithmetic # ------------------------------------------------------------------ diff --git a/slayer/sql/dialects/bigquery.py b/slayer/sql/dialects/bigquery.py index 0eace4ac..558404cd 100644 --- a/slayer/sql/dialects/bigquery.py +++ b/slayer/sql/dialects/bigquery.py @@ -18,9 +18,8 @@ ``rewrite_emitted_sql`` / ``decode_result_keys`` hooks on the base class have identity defaults; only ``BigqueryDialect`` (and ``TsqlDialect``, DEV-1571) override them today. The shared encode/decode bijection lives -in :mod:`slayer.sql.dialects._alias_mangle` and is reused by both -dialects — only the regex anchor (backticks here, brackets in T-SQL) -differs. +in :mod:`slayer.sql.naming` (DEV-1713) and is reused by both dialects — +only the regex anchor (backticks here, brackets in T-SQL) differs. """ from __future__ import annotations @@ -34,7 +33,7 @@ from sqlglot import exp from slayer.core.enums import TimeGranularity -from slayer.sql.dialects._alias_mangle import decode_alias, encode_alias +from slayer.sql.naming import decode_alias, encode_alias from slayer.sql.dialects.base import SqlDialect if TYPE_CHECKING: diff --git a/slayer/sql/dialects/sqlite.py b/slayer/sql/dialects/sqlite.py index 13f2994d..ae4957cd 100644 --- a/slayer/sql/dialects/sqlite.py +++ b/slayer/sql/dialects/sqlite.py @@ -407,6 +407,14 @@ class SqliteDialect(SqlDialect): log10_native: bool = True log2_native: bool = True + def build_null_safe_eq( + self, left: exp.Expression, right: exp.Expression, + ) -> exp.Expression: + """DEV-1708: SQLite's ``IS`` is null-safe on every supported version; + ``IS NOT DISTINCT FROM`` (what sqlglot emits for ``NullSafeEQ``) needs + SQLite ≥ 3.39, so anchor on bare ``IS`` instead.""" + return exp.Is(this=left, expression=right) + def build_date_trunc( self, col_expr: exp.Expression, diff --git a/slayer/sql/dialects/tsql.py b/slayer/sql/dialects/tsql.py index 59193fcb..8cf1a854 100644 --- a/slayer/sql/dialects/tsql.py +++ b/slayer/sql/dialects/tsql.py @@ -34,7 +34,7 @@ from sqlglot import exp from slayer.core.enums import TimeGranularity -from slayer.sql.dialects._alias_mangle import decode_alias, encode_alias +from slayer.sql.naming import decode_alias, encode_alias from slayer.sql.dialects.base import SqlDialect, _build_covar_decomposition @@ -71,6 +71,13 @@ class TsqlDialect(SqlDialect): log10_native: bool = True log2_native: bool = False + def build_null_safe_eq( + self, left: exp.Expression, right: exp.Expression, + ) -> exp.Expression: + """DEV-1708: T-SQL has no ``IS NOT DISTINCT FROM`` / ``<=>`` — emit the + portable expanded ``a = b OR (a IS NULL AND b IS NULL)``.""" + return self._expanded_null_safe_eq(left, right) + def build_approx_count_distinct( self, col_sql: str, @@ -130,6 +137,10 @@ def build_time_offset_expr( unit_map = { "year": "YEAR", "month": "MONTH", "day": "DAY", "quarter": "MONTH", "week": "WEEK", + # DEV-1572: a one-period shift of a Sunday-week is one week — same + # normalization the base ``_granularity_to_unit`` applies (without + # it, ``DATEADD(WEEK_SUNDAY, ...)`` is invalid T-SQL). + "week_sunday": "WEEK", "hour": "HOUR", "minute": "MINUTE", "second": "SECOND", } unit = unit_map.get(granularity, granularity.upper()) @@ -346,8 +357,7 @@ def rewrite_emitted_sql(self, sql: str) -> str: dotted alias shape. Uses the same bijection as ``BigqueryDialect`` (shared encode in - ``slayer.sql.dialects._alias_mangle``); only the regex anchor - differs. + ``slayer.sql.naming``); only the regex anchor differs. """ return _TSQL_DOTTED_ALIAS_RE.sub( lambda m: f"[{encode_alias(m.group(1))}]", sql diff --git a/slayer/sql/generator.py b/slayer/sql/generator.py index 82dab6b4..f00ca7f5 100644 --- a/slayer/sql/generator.py +++ b/slayer/sql/generator.py @@ -1,34 +1,297 @@ -"""SQL generator — converts EnrichedQuery to SQL via sqlglot AST. +"""SQL generator — converts a ``PlannedQuery`` to SQL via sqlglot AST. -The generator works exclusively with EnrichedQuery objects (fully resolved -SQL expressions). It never looks up model definitions — that's done by the -query engine's _enrich() step. +The generator works exclusively with ``PlannedQuery`` objects (typed value +keys interned into slots, each carrying its resolved expression, join path +and phase). It never looks up model definitions — every referenced model is +already loaded on the ``ResolvedSourceBundle`` it is handed. + +Entry points: ``generate_from_planned`` (one stage) and +``generate_planned_stages`` (a multi-stage DAG rendered to one statement). """ -import copy import logging import re -from typing import Any, NamedTuple +from typing import AbstractSet, Any, Dict, List, Literal, Optional, Set, Tuple, Union import sqlglot from sqlglot import exp from slayer.core.enums import ( + BUILTIN_AGGREGATIONS, BUILTIN_AGGREGATION_FORMULAS, BUILTIN_AGGREGATION_REQUIRED_PARAMS, DataType, TimeGranularity, ) -from slayer.core.errors import UnresolvableOrderColumnError -from slayer.engine.enriched import EnrichedMeasure, EnrichedQuery, public_projection_aliases +from pydantic import BaseModel, ConfigDict, field_validator + +from slayer.core.errors import AggregationNotAllowedError, UnresolvableOrderColumnError +from slayer.core.keys import _FrozenKey, _reroot_path_ref, reroot_aggregate_key +from slayer.core.models import Aggregation +from slayer.core.refs import agg_kwarg_canonical_str +from slayer.core.time_bounds import strip_frame_bounds +from slayer.core.window_duration import parse_window_duration as _parse_window_duration +from slayer.engine.column_expansion import ( + _is_trivial_base, + _walk_path_to_target_sync, + collect_root_scope_joined_paths, + expand_derived_refs_sync, +) +from slayer.engine.source_bundle import ( + stage_bundle_with_siblings, + synthetic_model_from_stage_schema, +) from slayer.sql.dialects import SqlDialect, get_dialect -from slayer.sql.reserved_keywords import ( - SLAYER_RESERVED_KEYWORDS, - prequote_reserved_identifiers, +from slayer.sql.naming import ( + AliasAllocator, + dialect_folds_case, + flat_name, + maybe_quote_ident, + quote_mixed_case_identifiers, + result_key, + result_key_from_alias, ) +from slayer.sql.reserved_keywords import prequote_reserved_identifiers +from slayer.sql.scope import ScopeFrame +from slayer.sql.scope_check import maybe_validate_scopes +from slayer.sql.stage_wrapper import build_flat_rename_wrapper + + + + +class ResolvedAggKwarg(BaseModel): + """DEV-1706 — a resolved parametric-aggregation kwarg value (2-kind tag). + + * ``kind="expr"`` — a trusted, scope-resolved sqlglot expression for a + column-ref kwarg (``ColumnKey`` / ``ColumnSqlKey``). Embedded directly; + the crossed join registered at spec-build (the DEV-1527 fix). + * ``kind="str"`` — the legacy canonical-string form (scalars via + ``agg_kwarg_canonical_str``, existing strings), consumed exactly as + before: ``_SAFE_AGG_PARAM_RE`` guard + ``_resolve_sql`` (percentile / + stat) or formula substitution (custom aggregations). + """ + + model_config = ConfigDict(frozen=True, arbitrary_types_allowed=True) + + kind: Literal["expr", "str"] + value: Union[exp.Expression, str] + + +class AggRenderSpec(BaseModel): + """DEV-1452 — typed input record for the dialect-aware aggregation + helpers (``_build_agg``, ``_build_percentile``, ``_build_stat_agg``, + ``_build_formula_agg``, ``_resolve_value_sql``, ``_resolve_agg_param``, + ``_build_ranked_subquery_from_planned``). + + Decouples the helpers from ``EnrichedMeasure`` so the legacy enrichment + pipeline can be deleted without forking dialect SQL emission. Carries + exactly the 11 fields the helpers empirically read; ``EnrichedMeasure`` + fields outside this set (``agg_args``, ``source_measure_name``, + ``distinct``, ``window``, ``user_declared``, ``label``, + ``filter_columns``) are deliberately NOT carried — ``count_distinct`` + dispatches on the agg name, and the positional time arg for + ``first`` / ``last`` is pre-resolved into ``time_column`` at spec-build + time. + """ + + model_config = ConfigDict(frozen=True, arbitrary_types_allowed=True) + + sql: str | None + """Column SQL expression (``Column.sql`` or its bare name); ``None`` for + ``*:count`` (renders as ``COUNT(*)``). + + Typed as ``str | None`` (not ``Optional[str]``) deliberately — the + field is **required** at construction; the explicit nullable form + documents that and dodges Sonar's S8396 false-positive on the + Pydantic-v2 ``Optional[X]``-implies-default-None misconception.""" + + name: str + """Source column name — qualified under ``model_name`` when ``sql`` is + None or a bare identifier. Empty for star-source aggregates.""" + + model_name: str + """Qualifier for unqualified column refs in ``sql`` / ``filter_sql`` / + aggregation params — the source relation.""" + + aggregation: str + """Aggregation name (``sum`` / ``count`` / ``percentile`` / …). Empty + string for the non-aggregation bare-column branch.""" + + alias: str + """Result-column alias used by the filtered first/last ranked-subquery + bookkeeping (``filtered_rn_map``, ``filtered_match_map`` lookups).""" + + aggregation_def: Optional[Aggregation] = None + """Custom-aggregation definition (formula + params) for aggregations + outside the built-in set. ``None`` for built-ins.""" + + agg_kwargs: Dict[str, ResolvedAggKwarg] = {} + """Query-time aggregation parameter overrides as typed 2-kind values + (DEV-1706 D-I). Column-ref kwargs arrive as ``kind="expr"`` (scope-resolved + at spec-build); everything else as ``kind="str"``. A bare ``str`` value is + coerced to ``kind="str"`` by ``_coerce_agg_kwargs`` so direct-construction + call sites keep working (this also carried the retired ``EnrichedMeasure`` + shim before DEV-1485 deleted it).""" + + @field_validator("agg_kwargs", mode="before") + @classmethod + def _coerce_agg_kwargs(cls, v: Any) -> Any: + """Coerce bare ``str`` kwarg values to ``ResolvedAggKwarg(kind="str")``; + pass ``ResolvedAggKwarg`` through; leave anything else for Pydantic to + reject (``bool`` / ``None`` never reach here from spec-build — they raise + earlier in ``agg_kwarg_canonical_str``).""" + if not isinstance(v, dict): + return v + coerced: Dict[str, Any] = {} + for key, val in v.items(): + if isinstance(val, (ResolvedAggKwarg, dict)): + coerced[key] = val + elif isinstance(val, str): + coerced[key] = ResolvedAggKwarg(kind="str", value=val) + else: + coerced[key] = val # bool / None / other → Pydantic rejects + return coerced + + filter_sql: Optional[str] = None + """Column-filter predicate (``Column.filter``) wired in at aggregation + time; the helpers wrap the aggregate as ``SUM(CASE WHEN THEN + END)``.""" + + time_column: Optional[str] = None + """Explicit time column for first/last ranking (overrides the query's + default). Pre-resolved from ``AggregateKey.args`` for the planner path.""" + + type: Optional[DataType] = None + """Declared outer-result type — when set, callers wrap the final + aggregate expression in ``CAST AS `` via ``_wrap_cast_for_type``.""" + + column_type: Optional[DataType] = None + """Source column's declared type — wraps the inner (pre-aggregation) + expression in CAST when the column.sql is a non-bare expression (e.g. + ``json_extract(...)``). Distinct from ``type`` which wraps the outer + aggregate.""" + + +class FirstLastRenderState(BaseModel): + """DEV-1501 — bundle of maps produced by + ``_build_first_last_base_select`` (host base) or + ``_render_cross_model_cte`` (cross-model CTE) that the HAVING render + path needs to thread into ``_build_agg`` so a HAVING aggregate + references the same ``_first_rn`` / ``_last_rn{suffix}`` column the + SELECT projects (instead of bare ``_last_rn``, which collapses + distinct time-column specs). + """ + + model_config = ConfigDict(frozen=True, arbitrary_types_allowed=True) + + rn_suffix_map: Dict[str, str] = {} + """Effective-time-column → rn suffix (``""`` / ``"_2"`` / …). Empty + when no first/last aggregates are in scope.""" + + default_time_col_sql: Optional[str] = None + """Fallback time column when a spec has no explicit ``time_column``. + ``None`` when every first/last spec carries an explicit arg.""" + + filtered_rn_map: Dict[str, str] = {} + """Per-spec-alias → dedicated rn column for filtered first/last + aggregates (Column.filter wired in at aggregation time).""" + + filtered_match_map: Dict[str, str] = {} + """Per-spec-alias → match-flag column for filtered first/last.""" + + agg_synth_alias: Optional[str] = None + """DEV-1501 Group A.3 — only set for the cross-model CTE single-agg + case. The cross-model CTE projects exactly one aggregate; if HAVING + references the same key, ``_render_filter_value_key_in_target_scope`` + must rebuild the synth with THIS alias so the ``filtered_rn_map`` / + ``filtered_match_map`` lookups (keyed by the synth alias) hit. Host- + base callers leave this ``None`` and rely on ``aliases_by_slot_id`` + threaded through ``_build_where_having_from_planned`` instead.""" + + value_alias_by_sql: Dict[str, str] = {} + """DEV-1708 Law 2: RESOLVED value text → ``_val_`` materialisation alias + for an aggregate whose SOURCE crosses a join. Keyed by the resolved + (qualified + ``Column.type`` inner-CAST) value emission — DEV-1709 — so + same-sql-different-type aggregates map to distinct materialisations. The + projected aggregate materialises the crossing value inside the ranked + subquery; a HAVING referencing the same aggregate must bind to the SAME + alias (not re-emit the raw crossing ref, which is out of scope in the + outer SELECT). Empty when every source is local.""" + + +def _iter_first_last_leaves(key) -> "list": # NOSONAR(S3776) — sequential isinstance dispatch over the closed ValueKey union; each branch is the per-type recursion contract for surfacing first/last AggregateKey leaves. Extracting per-type helpers would scatter the contract. + """DEV-1501 (Codex round 3): walk a composite ValueKey for first / + last ``AggregateKey`` leaves. + + Composite aggregate slots (``ArithmeticKey`` / ``ScalarCallKey``) + aren't separately materialised — their operand AggregateKeys are + inlined at the composite render path. Without surfacing the leaves + here, the ranked-subquery builder wouldn't see their distinct time + columns and the composite render would resolve every operand to + bare ``_last_rn``. + + Returns local first/last leaves only (cross-model operands raise in + the composite render path; row / literal / scalar-call / + transform / between / in branches recurse into operands without + surfacing themselves). + """ + from slayer.core.keys import ( + AggregateKey, + ArithmeticKey, + BetweenKey, + InKey, + ScalarCallKey, + ) + + out: list = [] + + def _walk(k) -> None: + if isinstance(k, AggregateKey): + if k.agg in ("first", "last") and not getattr( + k.source, "path", (), + ): + out.append(k) + return + if isinstance(k, ArithmeticKey): + for o in k.operands: + _walk(o) + return + if isinstance(k, ScalarCallKey): + for a in k.args: + _walk(a) + return + if isinstance(k, BetweenKey): + _walk(k.column) + _walk(k.low) + _walk(k.high) + return + if isinstance(k, InKey): + _walk(k.column) + # LiteralKey / ColumnKey / TimeTruncKey / TransformKey / etc.: + # not a first/last operand carrier; stop recursing. + + _walk(key) + return out + + + + +def _render_scalar_literal(v: Any) -> exp.Expression: + """Render a Python scalar (None / bool / int / float / Decimal / str) + as a bare sqlglot literal node. Used by the POST-phase filter renderer + for ``LiteralKey.value`` AND any non-key arg inside ``ScalarCallKey``. + """ + from decimal import Decimal + if v is None: + return exp.Null() + if isinstance(v, bool): + return exp.true() if v else exp.false() + if isinstance(v, (int, float, Decimal)): + return exp.Literal.number(str(v)) + return exp.Literal.string(str(v)) -def _wrap_cast_for_type(expr: exp.Expression, dt: DataType | None) -> exp.Expression: +def _wrap_cast_for_type(expr: exp.Expression, dt: Optional[DataType]) -> exp.Expression: """DEV-1361: wrap ``expr`` in ``CAST(expr AS )`` so the declared SLayer ``DataType`` is enforced in emitted SQL. @@ -56,6 +319,24 @@ def _wrap_cast_for_type(expr: exp.Expression, dt: DataType | None) -> exp.Expres return expr return exp.Cast(this=expr, to=exp.DataType(this=target)) + +def _filter_cast_type(dt: Optional[DataType]) -> Optional[DataType]: + """The CAST target to use when rendering a derived column inside a + WHERE / HAVING predicate (DEV-1450 #4a). + + Temporal types (``DATE`` / ``TIMESTAMP``) are suppressed: in a filter + the derived expression is COMPARED, not type-enforced, and + ``CAST(text AS TIMESTAMP)`` on SQLite gives the expression NUMERIC + affinity — truncating a string timestamp to its leading year and + breaking ``BETWEEN`` / comparison. A base temporal column in the same + position is never cast (it renders as a bare ``exp.Column``), so this + keeps the derived form on par. Non-temporal types pass through so a + derived numeric / boolean column still gets its enforcing CAST. + """ + if dt in (DataType.DATE, DataType.TIMESTAMP): + return None + return dt + logger = logging.getLogger(__name__) # Maps aggregation name (string) → SQL function name. @@ -78,76 +359,69 @@ def _wrap_cast_for_type(expr: exp.Expression, dt: DataType | None) -> exp.Expres # DEV-1317: statistical aggregations routed through _build_stat_agg. # stddev_samp/_pop and var_samp/_pop are 1-arg; corr / covar_samp / # covar_pop are 2-arg via the `other=` kwarg. SQLite gets these through -# registered Python UDFs; Postgres/DuckDB/ClickHouse use the native -# function emitted via sqlglot transpilation. MySQL and T-SQL have no -# native CORR / COVAR_SAMP / COVAR_POP — these use the -# variance-decomposition formula in _build_covar_formula instead. +# registered Python UDFs; Postgres/DuckDB/MySQL/ClickHouse use the +# native function emitted via sqlglot transpilation. MySQL has no +# native CORR / COVAR_SAMP / COVAR_POP — _build_stat_agg raises +# NotImplementedError there, mirroring _build_median. _STAT_AGG_NAMES: frozenset[str] = frozenset({ "stddev_samp", "stddev_pop", "var_samp", "var_pop", "corr", "covar_samp", "covar_pop", }) # Subset of _STAT_AGG_NAMES that take two columns (LHS + `other=` kwarg). -# Dialect-specific stat-agg behaviour (T-SQL name overrides, MySQL/T-SQL -# variance-decomposition formula, log10/log2 native flags) moved to the -# per-dialect classes under ``slayer/sql/dialects/`` (DEV-1542). _TWO_ARG_STAT_AGGS: frozenset[str] = frozenset({"corr", "covar_samp", "covar_pop"}) +# DEV-1450 stage 7b.13: aggregations dispatched through the built-in +# path (``_build_agg`` -> ``_build_*`` family). A name in this set always +# resolves to a built-in renderer; a name NOT in the set MUST resolve to +# a model-level ``Aggregation`` definition (``SlayerModel.aggregations``) +# or it's a hard error. Model-level overrides for built-in names ARE +# permitted and get threaded into ``AggRenderSpec.aggregation_def`` so +# ``_resolve_agg_param`` honours their default params (CodeRabbit +# fold-in on DEV-1452 PR #144 — the prior "synth adapter doesn't +# propagate aggregation_def for built-ins" TODO is now done). +# +# Name kept as ``_LOCAL_SLICE`` for grep continuity with 7b.8-7b.12 +# call sites and tests; the set is no longer local-only. +# +# DEV-1717: bound to the canonical ``BUILTIN_AGGREGATIONS`` enum rather than a +# hand-maintained duplicate. The two allowlists must stay byte-identical — a +# new built-in aggregation added to the enum is dispatched here automatically, +# so they can never silently desync (a lockstep-edit hazard CodeRabbit flagged +# when ``count_distinct_approx`` had to be added to both). +_BUILTIN_BAREARG_AGGS_LOCAL_SLICE: frozenset[str] = BUILTIN_AGGREGATIONS + +# DEV-1337: dialects with native single-arg `log10(x)` / `log2(x)`. sqlglot +# normalises both into a generic ``Log(this=Literal(base), expression=arg)`` +# AST and re-emits as ``LOG(base, x)`` for almost every dialect, which +# diverges from the recipe formula text and (on dialects without 2-arg +# ``LOG``) can break a previously working call. We rewrite the AST back +# to ``Anonymous(this='log10'|'log2', ...)``; the per-dialect native-alias +# decision is delegated to ``SqlDialect.should_use_native_log`` (DEV-1716). + # Transforms that use self-join CTEs instead of window functions. # This gives correct results at result-set edges (no NULLs when the DB has the data) # and handles gaps in time series correctly. _SELF_JOIN_TRANSFORMS = {"time_shift"} -# DEV-1539: compound AST shapes that ALWAYS need an outer ``(...)`` -# wrap when substituted into the HAVING text. Checked **before** the -# atomic list below because in sqlglot 30.4.3 ``exp.And`` / ``exp.Or`` -# inherit from ``exp.Func``; without this priority, a custom-aggregation -# formula rooted at an ``And``/``Or`` would mis-classify as atomic and -# emit ambiguous SQL like ``SUM(x) > 0 AND SUM(y) > 0 IS NULL`` instead -# of ``(SUM(x) > 0 AND SUM(y) > 0) IS NULL``. Mirrors the WHERE-side -# ``_COMPOUND_FILTER_INLINE_TYPES`` in ``slayer/engine/enrichment.py``. -_HAVING_AGG_COMPOUND_TYPES: tuple = ( - exp.Binary, - exp.Connector, - exp.Unary, - exp.Predicate, -) - -# DEV-1539: AST shapes considered atomic for HAVING aggregate -# substitution. Used as the fallback after the compound-types check. -# Mirrors the WHERE-side ``_ATOMIC_FILTER_INLINE_TYPES`` in -# ``slayer/engine/enrichment.py``. -_HAVING_AGG_ATOMIC_TYPES: tuple = ( - exp.Column, - exp.Literal, - exp.Func, # covers function calls, CAST, CASE, Anonymous, … - exp.Paren, - exp.Boolean, - exp.Null, -) - # Separator used when joining pre-rendered SQL fragments into a conjunctive # WHERE/HAVING clause; extracted as a constant so Sonar S1192 doesn't flag it # at every join site. _SQL_AND_JOINER = " AND " - -class _OrderColRef(NamedTuple): - """DEV-1645: resolved ORDER BY key. ``is_alias`` distinguishes a projected - output alias (emit whole-quoted, e.g. ``"orders.revenue_sum"``) from a - table.column fallback (emit SPLIT, e.g. ``ranked."time_mark"``), so a sort - on an unprojected/renamed column references the underlying FROM-scope column - instead of a nonexistent composite identifier.""" - text: str # whole resolved string (used for base_cols membership) - is_alias: bool # True => projected output alias; False => table.column fallback - qualifier: str | None # fallback only: FROM-scope alias - column: str | None # fallback only: underlying column short name - # DEV-1444: separator used between pretty-printed SELECT projection columns # (",\n "). Extracted as a constant so Sonar S1192 doesn't flag every # join site that follows the same pattern. _SQL_COL_SEP = ",\n " +# Repeated SQL keyword fragments — extracted so the same literal isn't +# duplicated across CTE / window emission sites (Sonar S1192). +_SQL_WITH = "WITH " +_SQL_PARTITION_BY = "PARTITION BY " +# Two-space-indented ``SELECT`` head for hand-assembled CTE bodies (shifted / +# consecutive-periods pairs), extracted so the literal isn't duplicated (S1192). +_SQL_SELECT_HEAD = "SELECT\n " + # Matches safe aggregation parameter values: identifiers, qualified names, numeric literals. _SAFE_AGG_PARAM_RE = re.compile( r'^(?:' @@ -158,7 +432,7 @@ class _OrderColRef(NamedTuple): ) -def _wrap_filter(sql_str: str, filter_sql: str | None) -> str: +def _wrap_filter(sql_str: str, filter_sql: Optional[str]) -> str: """Wrap ``sql_str`` in ``CASE WHEN filter_sql THEN ... END`` if a row-level filter is set; otherwise pass through unchanged. Used by the dialect-aware aggregate builders (``_build_percentile``, ``_build_stat_agg``, @@ -169,7 +443,43 @@ def _wrap_filter(sql_str: str, filter_sql: str | None) -> str: return sql_str return f"(CASE WHEN {filter_sql} THEN {sql_str} END)" -_WINDOW_DURATION_RE = re.compile(r"(?P\d+)(?Pmin|[ymwdhs])") + +def _first_bare_column_name(key) -> Optional[str]: + """Return the leaf name of the first bare column reference inside a + ROW-phase composite key (DEV-1576 / DEV-1717 error messages). + + Walks ``ArithmeticKey`` operands / ``ScalarCallKey`` args / a + ``TransformKey`` input for a ``ColumnKey`` / ``ColumnSqlKey`` leaf so the + "Bare measure name ''" error names the offending column. Returns + ``None`` when no column ref is found (caller falls back to the alias). + """ + from slayer.core.keys import ( + ArithmeticKey, + ColumnKey, + ColumnSqlKey, + ScalarCallKey, + TransformKey, + ) + + if isinstance(key, ColumnKey): + return key.leaf + if isinstance(key, ColumnSqlKey): + return key.column_name + if isinstance(key, ArithmeticKey): + children = key.operands + elif isinstance(key, ScalarCallKey): + children = key.args + elif isinstance(key, TransformKey): + children = [key.input] + else: + return None + for child in children: + name = _first_bare_column_name(child) + if name is not None: + return name + return None + + _WINDOW_UNIT_SQL = { "y": "year", "m": "month", @@ -179,7 +489,15 @@ def _wrap_filter(sql_str: str, filter_sql: str | None) -> str: "min": "minute", "s": "second", } -# `_WINDOW_UNIT_SQLITE` moved to ``slayer/sql/dialects/sqlite.py`` (DEV-1542). +_WINDOW_UNIT_SQLITE = { + "y": "years", + "m": "months", + "w": "days", + "d": "days", + "h": "hours", + "min": "minutes", + "s": "seconds", +} def _validate_agg_param_value(value: str, param_name: str, agg_name: str) -> None: @@ -208,55 +526,8 @@ def _validate_agg_param_value(value: str, param_name: str, agg_name: str) -> Non } -def _has_cross_model_filter(m: EnrichedMeasure) -> bool: - """Check if a measure's filter references a cross-model dimension. - - Local columns are qualified as "model.column" by resolve_filter_columns. - Cross-model columns have a different prefix (e.g., "loss_payment.has_flag"). - We detect cross-model by checking if any dotted column's prefix differs - from the measure's own model_name. - """ - if not m.filter_columns: - return False - for col in m.filter_columns: - if "." not in col: - continue - prefix = col.rsplit(".", 1)[0] - # "__" in prefix means a multi-hop join path (always cross-model) - if "__" in prefix: - return True - # Single segment prefix: cross-model if it's not the measure's model - if prefix != m.model_name: - return True - return False - - -def _is_windowed_measure(m: EnrichedMeasure) -> bool: - return bool(m.window) -def _parse_window_duration(value: str) -> list[tuple[int, str]]: - """Parse compact durations like 1y2m3w5d6h7min8s.""" - if not value: - raise ValueError("Window duration cannot be empty") - pos = 0 - parts: list[tuple[int, str]] = [] - for match in _WINDOW_DURATION_RE.finditer(value): - if match.start() != pos: - raise ValueError( - f"Invalid window duration '{value}'. Use syntax like '1y2m3w5d6h7min8s'." - ) - amount = int(match.group("num")) - unit = match.group("unit") - if amount <= 0: - raise ValueError(f"Window duration parts must be positive in '{value}'") - parts.append((amount, unit)) - pos = match.end() - if pos != len(value) or not parts: - raise ValueError( - f"Invalid window duration '{value}'. Use syntax like '1y2m3w5d6h7min8s'." - ) - return parts def _cte_name_from_alias(prefix: str, alias: str) -> str: @@ -266,71 +537,47 @@ def _cte_name_from_alias(prefix: str, alias: str) -> str: with aliases that already contain underscores. E.g.: - ``orders.revenue_sum`` -> ``_fm_orders__revenue_sum`` - ``orders_v2.revenue_sum`` -> ``_fm_orders_v2__revenue_sum`` + + DEV-1713: the ``.`` -> ``__`` flatten delegates to + :func:`slayer.sql.naming.flat_name` (single owner); this adds only the + non-identifier-character sanitisation on top. """ - sanitized = alias.replace(".", "__") + sanitized = flat_name(alias) sanitized = re.sub(r"[^a-zA-Z0-9_]", "_", sanitized) return prefix + sanitized -def _alias_prefixes(model_name: str) -> list: - """'a__b__c' → ['a', 'a__b', 'a__b__c']""" - parts = model_name.split("__") - return ["__".join(parts[: i + 1]) for i in range(len(parts))] +def _effective_src_filters(*, planned_query, plan) -> list: + """``planned_query.filters_by_phase`` as the windowed ``_src`` scope sees it + (DEV-1732): frame-bound residuals substituted for the host's predicates. + Returned as ONE list that both ``_resolve_where_filter_joins_via_scope`` and + ``_build_where_having_from_planned`` consume, so join discovery and + rendering are structurally guaranteed to agree. Entries whose filter is + wholly a frame bound need no substitution here — the planner already left + their ids out of ``plan.where_filter_ids``, and the caller's + ``skip_filter_ids`` drops them. -def _filter_dotted_columns(filters) -> list[str]: - """Yield each "__"-joined path-alias prefix referenced by every non-post - filter's dotted column. - - A filter on `a.b.c` produces ['a', 'a__b'] — the path-alias forms that - correspond to the joins required to evaluate the filter. Used by window - CTE pruning to keep filter-driven joins. + Returns the original list unchanged when the plan carries no rewrites, so a + query without a split conjunction emits byte-identical SQL. """ - out: list[str] = [] - for f in filters: - if getattr(f, "is_post_filter", False): - continue - for col in f.columns: - if "." not in col: - continue - parts = col.split(".") - for i in range(1, len(parts)): - out.append("__".join(parts[:i])) - return out + rewrites = getattr(plan, "src_filter_rewrites", None) + if not rewrites: + return planned_query.filters_by_phase + by_id = {r.filter_id: r.expression for r in rewrites} + return [ + fp if fp.id not in by_id + else fp.model_copy(update={"expression": by_id[fp.id]}) + for fp in planned_query.filters_by_phase + ] + + + + + + -def _needed_join_aliases(enriched: EnrichedQuery, extra_columns: list = ()) -> set: - """Compute which resolved_join aliases are needed for dimensions + extra dotted columns.""" - aliases: set = set() - for dim in enriched.dimensions: - if dim.model_name != enriched.model_name: - aliases.update(_alias_prefixes(dim.model_name)) - for td in enriched.time_dimensions: - if td.model_name != enriched.model_name: - aliases.update(_alias_prefixes(td.model_name)) - for col in extra_columns: - if "." in col: - parts = col.split(".") - for i in range(1, len(parts)): - aliases.add("__".join(parts[:i])) - return aliases - - -def _filter_references_available(f, available_aliases: set) -> bool: - """Check if all table references in a filter's columns are within a CTE's join set. - - Non-dotted columns (local to the base model) are always available. - Dotted columns like "warehouse.status" produce alias "warehouse" which - must be in available_aliases. - """ - for col in f.columns: - if "." not in col: - continue - parts = col.split(".") - table_alias = "__".join(parts[:-1]) - if table_alias not in available_aliases: - return False - return True # DEV-1444: digit-suffix tail patterns for OFFSET / LIMIT, each bounded @@ -349,6 +596,13 @@ def _filter_references_available(f, available_aliases: set) -> bool: ) _TRAILING_LIMIT_RE = re.compile(r"(?is)\s*LIMIT\s+\d+\s*\Z") +# A ``Column.sql`` that is just an unqualified identifier — i.e. the column +# renames a physical column rather than computing an expression. Used to +# reserve star-exported physical names against ``_val_`` collisions +# (DEV-1728). Deliberately rejects dots: ``regions.population`` is a crossing +# reference, not a column of the star-projected relation. +_BARE_IDENT_RE = re.compile(r"[A-Za-z_]\w*") + def _strip_trailing_pagination(sql: str) -> str: """DEV-1444: remove trailing ORDER BY / LIMIT / OFFSET clauses that @@ -398,98 +652,96 @@ def _strip_trailing_pagination(sql: str) -> str: class SQLGenerator: """Generates SQL from an EnrichedQuery.""" - def __init__(self, dialect: str | SqlDialect = "postgres"): + def __init__(self, dialect: "str | SqlDialect" = "postgres"): if isinstance(dialect, SqlDialect): self._dialect: SqlDialect = dialect else: self._dialect = get_dialect(dialect) + # DEV-1708 (D-E): the generation-wide alias allocator, installed by + # ``generate_from_planned`` for the duration of one render so inline + # forward ``_cm_*`` CTEs and the host base share ``_val_`` naming. + # ``None`` outside a render; direct-call helpers fall back to a local + # allocator. + self._gen_allocator: Optional[AliasAllocator] = None @property def dialect(self) -> str: """The sqlglot dialect name. Read-only — derived from ``self._dialect.sqlglot_name``. Mutating it would desync the - strategy object from the string sqlglot consumes.""" + strategy object from the string sqlglot consumes (DEV-1716).""" return self._dialect.sqlglot_name + def _new_allocator(self) -> AliasAllocator: + """Build an ``AliasAllocator`` carrying this generator's dialect + case-folding policy (DEV-1726): on case-folding dialects the + ``_taken`` comparison folds, so minted CTE / materialisation names can + never collide after the backend folds them. The ONLY construction + site in this module — pinned by test_dev1726_cte_case_folding — so a + new allocation path cannot silently lose dialect awareness.""" + return AliasAllocator(folds_case=dialect_folds_case(self.dialect)) + @staticmethod - def _maybe_quote_ident(ident: exp.Expression | None) -> None: - """Set ``quoted=True`` in place on ``ident`` when it is an unquoted - ``Identifier`` containing an uppercase letter (DEV-1645). No-op - otherwise (None, already-quoted, all-lowercase, non-Identifier).""" - if ( - isinstance(ident, exp.Identifier) - and not ident.quoted - and any(c.isupper() for c in ident.this) - ): - ident.set("quoted", True) + def _reserve_model_column_names(allocator: AliasAllocator, model) -> None: + """Reserve every name a ``.*`` projection of ``model`` can + export, so a minted ``_val_`` (Law-2 materialisation) never shadows a + real column (DEV-1728 / Codex F6). + + Both the SEMANTIC name and — when ``Column.sql`` is a bare identifier — + the PHYSICAL column name are reserved: a star-projection exports the + physical names, and the two differ whenever a column renames its source + (``Column(name="value", sql="_val_0")``). A non-bare ``Column.sql`` is an + expression, not a star-exported column, so it contributes nothing. + + Columns that exist in the database but not on the model are outside what + SLayer can see without reflection; a physical column literally named + ``_val_`` is the only way to hit that residual, which the underscore + prefix makes vanishingly unlikely. + """ + names: List[str] = [] + for c in model.columns: + names.append(c.name) + sql = getattr(c, "sql", None) + if sql and _BARE_IDENT_RE.fullmatch(sql.strip()): + names.append(sql.strip()) + allocator.reserve(*names) + + @staticmethod + def _maybe_quote_ident(ident: Optional[exp.Expression]) -> None: + """Thin delegator to :func:`slayer.sql.naming.maybe_quote_ident` + (DEV-1713 D-b: the mixed-case quoting policy is owned by the naming + module). Kept as a method so existing ``gen._maybe_quote_ident`` call + sites / tests are unchanged.""" + maybe_quote_ident(ident) @staticmethod def _quote_mixed_case_identifiers(node: exp.Expression) -> exp.Expression: - """DEV-1645: quote mixed-case DB identifiers so case-folding dialects - (Postgres/Redshift fold to lower; Snowflake/Oracle fold to upper) reach - the right physical object instead of silently folding to a non-existent - name. - - Context-aware: quotes only the **column-name leaf** of a ``Column`` - (``Column.this``) and the **physical-table name parts** of a ``Table`` - (``this``/``db``/``catalog``). It deliberately does NOT quote table - aliases or the qualifier side of a column reference — those are - SLayer-internal aliases that fold consistently within a query and never - need quoting; quoting them would only churn output and (for uppercase - model names) diverge from string-built references. Function names parse - to ``Anonymous``/``Func`` and string literals to ``Literal``, so neither - is touched. Idempotent. Applied at every parse / AST-construction site - (``_parse``, ``_parse_predicate``, ``_to_ident``, ``_to_table``).""" - if isinstance(node, exp.Column): - SQLGenerator._maybe_quote_ident(node.this) - elif isinstance(node, exp.Table): - SQLGenerator._maybe_quote_ident(node.this) - SQLGenerator._maybe_quote_ident(node.args.get("db")) - SQLGenerator._maybe_quote_ident(node.args.get("catalog")) - return node + """Thin delegator to + :func:`slayer.sql.naming.quote_mixed_case_identifiers` (DEV-1713 D-b). + Kept as a method so ``tree.transform(gen._quote_mixed_case_identifiers)`` + call sites / tests are unchanged. See the naming module for the policy + (DEV-1645 mixed-case quoting; DEV-1686 reserved-word dependency).""" + return quote_mixed_case_identifiers(node) def _to_ident(self, name: str) -> exp.Identifier: """Build a column/table-name identifier, quoting it when mixed-case (DEV-1645). Use for real DB column/table names — NOT for aliases or - qualifiers (those stay unquoted via plain ``exp.to_identifier``).""" + qualifiers (those stay unquoted via plain ``exp.to_identifier``, and + reserved-word aliases quote at emit via ``RESERVED_KEYWORDS``).""" ident = exp.to_identifier(name) self._maybe_quote_ident(ident) return ident - def _to_table(self, name: str, alias: str | None = None) -> exp.Expression: + def _to_table(self, name: str, alias: Optional[str] = None) -> exp.Expression: """Build a (possibly schema-qualified) table reference with mixed-case physical-name parts quoted (DEV-1645). The ``alias`` is SLayer-internal - and stays unquoted.""" + and stays unquoted (a reserved-word alias still quotes at emit through + ``RESERVED_KEYWORDS`` — DEV-1686).""" table = exp.to_table(name).transform(self._quote_mixed_case_identifiers) if alias is not None: table.set("alias", exp.TableAlias(this=exp.to_identifier(alias))) return table - def _maybe_quote_qualifier(self, name: str) -> str: - """DEV-1686: quote a SLayer-internal qualifier/alias iff it is a reserved - word. Non-reserved names (including mixed-case) stay bare, preserving the - DEV-1645 fold-consistent behaviour. Used for the string-built ``AS - `` sites that are NOT dot-adjacent (so the parse-time - ``prequote_reserved_identifiers`` cannot reach them).""" - return self._q(name) if name.lower() in SLAYER_RESERVED_KEYWORDS else name - - def _q(self, name: str) -> str: - """Dialect-aware identifier quoting (DEV-1571 Bug 3 generalised). - - Returns ``name`` wrapped in the dialect's natural identifier - quotes — backticks on MySQL/BigQuery, brackets on T-SQL, ANSI - double quotes on Postgres/SQLite/DuckDB/ClickHouse/Snowflake. - - Used across every site in this generator that previously - hardcoded ``f'"{name}"'``. The hardcoded form parses as a string - literal on MySQL and bypasses T-SQL's bracket-anchored alias - mangling (Bug 2), so the inner CTE-assembly code paths that fed - the failing time-shift / cross-model integration tests must - emit through this helper. - """ - return exp.Identifier(this=name, quoted=True).sql(dialect=self.dialect) - - def _parse(self, sql: str, *, dialect: str | None = None) -> exp.Expression: + def _parse(self, sql: str, *, dialect: Optional[str] = None) -> exp.Expression: """Parse ``sql`` via sqlglot, applying SLayer-specific AST rewrites. On SQLite, rewrites ``exp.JSONExtract`` to the function-call form so @@ -509,27 +761,44 @@ def _parse(self, sql: str, *, dialect: str | None = None) -> exp.Expression: """ d = dialect or self.dialect active = self._dialect if d == self.dialect else get_dialect(d) - # DEV-1686: quote bare reserved-word qualifiers/leaves so a generated - # string embedding e.g. ``grant.col`` parses (bare reserved words fail - # at parse time, which the emit-time RESERVED_KEYWORDS fix can't reach). - sql = prequote_reserved_identifiers(sql=sql, dialect=d) + # DEV-1686: quote any reserved-word qualifier/leaf (``grant.id`` → + # ``"grant".id``) before re-parsing a SLayer-built string, so a bare + # reserved word does not fail at parse time. No-op for ordinary SQL + # (only dot-adjacent reserved words are touched) and idempotent on + # already-quoted identifiers. + sql = prequote_reserved_identifiers(sql, dialect=d) tree = sqlglot.parse_one(sql, dialect=d) + # DEV-1716: PARSE-dialect keyed AST rewrite (SQLite rewrites + # JSONExtract to the function-call form — DEV-1331). Default identity. tree = active.rewrite_parsed_ast(tree) # Log-alias rewrite is multi-dialect; the per-base allowlist check # lives inside ``_rewrite_log_aliases`` so unsupported dialects # (oracle; tsql for log2) keep the canonical 2-arg LOG form. tree = tree.transform(self._rewrite_log_aliases) - # DEV-1645: quote mixed-case identifiers so case-folding dialects reach - # the right physical column/table. + # DEV-1645: quote mixed-case column/table identifiers so case-folding + # dialects reach the right physical object (see the method docstring + # for the DEV-1706 pull-forward rationale). tree = tree.transform(self._quote_mixed_case_identifiers) - # DEV-1576: target-keyed AST rewrite. Keyed to the generator's TARGET - # dialect (``self._dialect``), NOT the parse dialect — expressions are - # canonically parsed as Postgres regardless of target, so this is the - # only place a Postgres-only ROUND cast can fire without corrupting - # SQLite / DuckDB output. + # DEV-1716: TARGET-dialect keyed AST rewrite (Postgres wraps the first + # arg of a 2-arg ROUND in a numeric CAST — DEV-1576). Keyed to the + # generator's target dialect, not the parse dialect. return self._dialect.rewrite_target_ast(tree) - def _parse_predicate(self, sql: str, *, dialect: str | None = None) -> exp.Expression: + def _finalize_scalar_call(self, expr: exp.Expression) -> exp.Expression: + """Apply the target-dialect AST rewrite to a scalar-call expression + (DEV-1576 / DEV-1717). + + Scalar calls (``round``/``abs``/``coalesce``/…) in formulas are + assembled directly as ``exp.func(...)`` AST, never string-parsed, so + the ``rewrite_target_ast`` applied inside ``_parse`` never sees them. + Routing them through the same dialect hook here keeps the 2-arg + Postgres ``ROUND`` numeric-cast (and any future target rewrite) + consistent between parsed and AST-built expressions. Identity for + dialects whose ``rewrite_target_ast`` is a no-op. + """ + return self._dialect.rewrite_target_ast(expr) + + def _parse_predicate(self, sql: str, *, dialect: Optional[str] = None) -> exp.Expression: """Parse a bare WHERE/HAVING predicate expression (DEV-1378). ``sqlglot.parse_one(sql, dialect=...)`` falls back to a ``Command`` @@ -548,10 +817,9 @@ def _parse_predicate(self, sql: str, *, dialect: str | None = None) -> exp.Expre """ d = dialect or self.dialect active = self._dialect if d == self.dialect else get_dialect(d) - # DEV-1686: quote bare reserved-word qualifiers/leaves (e.g. a WHERE - # filter qualified to ``grant.amount`` or a joined ``grant.status``) - # before wrapping/parsing the predicate. - sql = prequote_reserved_identifiers(sql=sql, dialect=d) + # DEV-1686: quote reserved qualifiers/leaves before the re-parse (see + # ``_parse``). No-op for ordinary predicates; idempotent when quoted. + sql = prequote_reserved_identifiers(sql, dialect=d) wrapped = sqlglot.parse_one(f"SELECT 1 WHERE {sql}", dialect=d) where = wrapped.args.get("where") if where is None or where.this is None: # pragma: no cover — defensive @@ -560,160 +828,30 @@ def _parse_predicate(self, sql: str, *, dialect: str | None = None) -> exp.Expre ) tree = active.rewrite_parsed_ast(where.this) tree = tree.transform(self._rewrite_log_aliases) - # DEV-1645: quote mixed-case identifiers (same policy as ``_parse``) — - # this is the separate WHERE/HAVING/``filter_sql`` parser, so it needs - # the transform too or predicate-side mixed-case idents stay unquoted. + # DEV-1645: mixed-case identifier quoting (see ``_parse``). tree = tree.transform(self._quote_mixed_case_identifiers) - # DEV-1576: same target-keyed rewrite as ``_parse`` — a 2-arg ROUND - # over a DOUBLE in a Mode-A SQL filter needs the Postgres numeric cast. return self._dialect.rewrite_target_ast(tree) - def generate( - self, - enriched: EnrichedQuery, - *, - render_mode: str = "outer", - ) -> str: - """Generate SQL from a fully resolved EnrichedQuery. - - Architecture: - 1. Base CTE: simple (non-isolated) measures + dimensions - 2. Per-measure CTEs: cross-model measures + cross-model-filtered measures - 3. Combined: LEFT JOIN base + measure CTEs on shared dimensions - 4. Expressions/transforms stacked on top of combined - - Args: - enriched: Fully resolved query. - render_mode: ``"outer"`` (default) — SQL will be executed and - shown to the user; the outermost SELECT is trimmed to - ``public_projection_aliases(enriched)`` (DEV-1444). - ``"wrapped"`` — SQL is embedded into a larger structure - (``_query_as_model`` inner_sql, inner stages of - ``source_queries``); the outer SELECT keeps every alias - downstream references can reach. - """ - if render_mode not in ("outer", "wrapped"): - raise ValueError( - f"render_mode must be 'outer' or 'wrapped', got {render_mode!r}" - ) - has_isolated = any(_has_cross_model_filter(m) for m in enriched.measures) - has_windowed = any(_is_windowed_measure(m) for m in enriched.measures) - has_cross_model = bool(enriched.cross_model_measures) - has_measure_ctes = has_isolated or has_cross_model or has_windowed - has_computed = bool(enriched.expressions or enriched.transforms) - # DEV-1336: a post-filter on a windowed `Column.sql` (or any other - # post-classified filter) requires the outer `_filtered` wrap from - # `_generate_with_computed`, even when there are no expressions or - # transforms to layer. - has_post_filters = any(getattr(f, "is_post_filter", False) for f in enriched.filters) - - base_sql = self._generate_base(enriched=enriched, skip_isolated=has_measure_ctes) - - if not has_measure_ctes and not has_computed and not has_post_filters: - sql = base_sql - elif has_measure_ctes: - # Get structured CTE definitions (no WITH wrapper) - measure_ctes = self._build_combined(enriched=enriched, base_sql=base_sql) - if has_computed or has_post_filters: - # Pass CTE list to computed layer — it merges into a flat WITH - sql = self._generate_with_computed(enriched=enriched, prefix_ctes=measure_ctes) - else: - # No expressions: assemble CTEs + outer SELECT + pagination - sql = self._assemble_combined_sql(enriched=enriched, measure_ctes=measure_ctes) - else: - # No measure CTEs, just computed columns or post-filters - sql = self._generate_with_computed(enriched=enriched, base_sql=base_sql) - - if render_mode == "outer": - sql = self._apply_outer_projection_trim(sql=sql, enriched=enriched) - # Dialect-driven post-pass: BigQuery mangles dotted aliases here. - # Default hook is identity for every other dialect (Postgres-shaped - # SqlDialect base). Fires for BOTH render modes — inner CTE column - # names are subject to the same dialect alias rules as the outer - # projection. - sql = self._dialect.rewrite_emitted_sql(sql) - return sql - def _apply_outer_projection_trim( - self, *, sql: str, enriched: EnrichedQuery, - ) -> str: - """DEV-1444: wrap ``sql`` so its outermost SELECT projects exactly - the user-declared ``public_projection_aliases`` of ``enriched``, - in declared order. - - When the inner SELECT already projects exactly the public list, - the trim is a no-op (``sql`` returned unchanged). Otherwise an - outer wrapper is emitted:: - - SELECT - FROM () AS _outer - ORDER BY ... LIMIT N OFFSET M - - Moving ORDER BY / LIMIT / OFFSET to the outer wrapper preserves the - rendered SQL's row-ordering contract while keeping every hoisted - intermediate accessible inside the subquery scope (so the ORDER BY - can still reference a hidden alias like ``"orders.revenue_sum"`` - when no matching declared measure exists). - """ - public = public_projection_aliases(enriched) - if not public: - return sql - parsed = self._safe_parse_outer(sql) - if parsed is None: - return sql - # Fast path: when the inner SELECT already projects exactly the - # public alias list (in order), no wrapper is needed. - inner_aliases = [n.alias_or_name for n in parsed.expressions] - if inner_aliases == public: - return sql - # Detach ORDER BY / LIMIT / OFFSET from the inner so the outer - # wrapper can own them; the FROM-subquery scope exposes every - # alias their references may need. - order = parsed.args.pop("order", None) - limit = parsed.args.pop("limit", None) - offset_arg = parsed.args.pop("offset", None) - return self._build_outer_wrap( - inner_sql=sql, - public=public, - order=order, - limit=limit, - offset_arg=offset_arg, - ) - def _safe_parse_outer(self, sql: str): - """Parse ``sql`` via the generator's ``_parse`` (so AST rewrites - like LOG10/LOG2 alias preservation survive a round-trip). - Returns the ``exp.Select`` root or ``None`` when parsing fails - or the root isn't a Select — both signals tell the trim caller - to leave ``sql`` untouched. - """ - try: - parsed = self._parse(sql) - except Exception: - return None - if not isinstance(parsed, exp.Select): - return None - return parsed def _build_outer_wrap( self, *, inner_sql: str, - public: list[str], + public: List[str], order, limit, offset_arg, ) -> str: - """Thin delegate to ``self._dialect.emit_outer_wrap``. + """Thin delegate to ``self._dialect.emit_outer_wrap`` (DEV-1716). Strips trailing ORDER BY / LIMIT / OFFSET from ``inner_sql`` (text-level) before handing off to the dialect hook, then passes - the detached AST nodes for re-emission on the outer statement. - - The dialect hook owns the wrap-shape choice — base impl emits - today's derived-table form; ``TsqlDialect`` overrides to hoist - inner top-level CTEs to satisfy T-SQL's "WITH only as statement - prefix" rule (DEV-1571 Bug 1). + the detached AST nodes for re-emission on the outer statement. The + hook owns the wrap shape (base derived-table form; ``TsqlDialect`` + hoists inner CTEs) AND the dialect-correct identifier quoting of the + public-alias list (backticks / brackets / ANSI double quotes). """ if order is None and limit is None and offset_arg is None: stripped = inner_sql @@ -728,396 +866,50 @@ def _build_outer_wrap( parse=self._parse, ) - def _build_combined(self, enriched: EnrichedQuery, - base_sql: str) -> list[tuple[str, str]]: - """Build CTE definitions for per-measure isolation. - - Returns a list of (name, sql) tuples. The last entry is ("_combined", select) - which joins _base with all measure CTEs on shared dimensions. The caller - decides how to assemble these — either as a standalone WITH query or as - prefix CTEs for _generate_with_computed(). + def _quote_ident(self, name: str) -> str: + """Render ``name`` as ONE dialect-quoted identifier string (DEV-1716). + + Backticks on MySQL/BigQuery, brackets on T-SQL, ANSI double quotes on + Postgres/SQLite/DuckDB. Replaces raw ``f'"{name}"'`` sites in the + string-assembled CTE/projection paths so non-ANSI dialects get correct + quoting in the first place (a terminal string-rewrite can't fix ANSI + quotes — MySQL re-parses them as string literals). The BigQuery / T-SQL + alias-mangling ``rewrite_emitted_sql`` post-pass then fires on the + dotted quoted identifier. Identity round-trip on Postgres/SQLite (still + ``"name"``), so those emissions are unchanged. """ - ctes = [("_base", base_sql)] - - # Collect dimension aliases for JOIN conditions - dim_aliases = [d.alias for d in enriched.dimensions] - td_aliases = [td.alias for td in enriched.time_dimensions] - join_aliases = dim_aliases + td_aliases - - # Track all CTEs and their measure aliases - # Each entry: (cte_name, measure_alias, cte_join_aliases) - # cte_join_aliases is None to use the default join_aliases, or a list - # of surviving aliases when the CTE has fewer dimensions. - measure_cte_refs = [] - - # --- Cross-model measure CTEs --- - seen_cm_ctes: set = set() - for cm in enriched.cross_model_measures: - cte_name = _cte_name_from_alias("_cm_", cm.alias) - if cte_name in seen_cm_ctes: - measure_cte_refs.append((cte_name, cm.alias, None)) - continue - seen_cm_ctes.add(cte_name) - - if cm.rerooted_enriched is not None: - # Re-rooted subquery: full query with target model as source, - # all joins/filters resolved from the target's join graph. - cte_sql = self._generate_base(enriched=cm.rerooted_enriched) - ctes.append((cte_name, cte_sql)) - # Surviving dims may be fewer than shared dims (unreachable dropped) - surviving = ( - [d.alias for d in cm.rerooted_enriched.dimensions] - + [td.alias for td in cm.rerooted_enriched.time_dimensions] - ) - measure_cte_refs.append((cte_name, cm.alias, surviving)) - continue - else: - # Fallback: minimal source→target CTE (legacy path) - select = exp.Select() - group_exprs = [] - - for dim in cm.shared_dimensions: - col_expr = self._resolve_sql(sql=dim.sql, name=dim.name, model_name=cm.source_model_name, type=dim.type) - select = select.select(col_expr.as_(dim.alias)) - group_exprs.append(col_expr) - for td in cm.shared_time_dimensions: - col_expr = self._resolve_sql(sql=td.sql, name=td.name, model_name=cm.source_model_name) - td_expr = self._build_date_trunc(col_expr=col_expr, granularity=td.granularity) - select = select.select(td_expr.as_(td.alias)) - group_exprs.append(td_expr) - - agg_expr, _ = self._build_agg(measure=cm.measure) - # DEV-1361: cast the cross-model agg result if a result type - # was declared on the source ModelMeasure. - agg_expr = _wrap_cast_for_type(agg_expr, cm.measure.type) - select = select.select(agg_expr.as_(cm.alias)) - - # FROM source model - if cm.source_sql: - source_from = exp.Subquery( - this=self._parse(cm.source_sql), - alias=exp.to_identifier(cm.source_model_name), - ) - else: - source_from = self._to_table(cm.source_sql_table, alias=cm.source_model_name) - select = select.from_(source_from) - - # JOIN target model - if cm.target_model_sql: - target_join = exp.Subquery( - this=self._parse(cm.target_model_sql), - alias=exp.to_identifier(cm.target_model_name), - ) - else: - target_join = self._to_table(cm.target_model_sql_table, alias=cm.target_model_name) - join_on = exp.and_(*( - exp.EQ( - this=exp.Column(this=self._to_ident(src), table=exp.to_identifier(cm.source_model_name)), - expression=exp.Column(this=self._to_ident(tgt), table=exp.to_identifier(cm.target_model_name)), - ) - for src, tgt in cm.join_pairs - )) - select = select.join(target_join, on=join_on, join_type=cm.join_type.upper()) - - # Only include WHERE conditions whose tables are in this CTE - cm_available = {cm.source_model_name, cm.target_model_name} - original_filters = enriched.filters - enriched.filters = [f for f in original_filters - if _filter_references_available(f, cm_available)] - where_clause, _ = self._build_where_and_having(enriched=enriched) - enriched.filters = original_filters - if where_clause is not None: - select = select.where(where_clause) - for gb in group_exprs: - select = select.group_by(gb) - - ctes.append((cte_name, select.sql(dialect=self.dialect))) - measure_cte_refs.append((cte_name, cm.alias, None)) - - # --- Windowed aggregation CTEs --- - for measure in enriched.measures: - if not _is_windowed_measure(measure): - continue - cte_name = _cte_name_from_alias("_wm_", measure.alias) - ctes.append((cte_name, self._generate_window_measure_cte(enriched=enriched, measure=measure))) - measure_cte_refs.append((cte_name, measure.alias, None)) - - # --- Isolated filtered-measure CTEs --- - for measure in enriched.measures: - if not _has_cross_model_filter(measure): - continue - cte_name = _cte_name_from_alias("_fm_", measure.alias) - - # Measure aggregation without CASE WHEN (the join IS the filter) - unfiltered = copy.copy(measure) - unfiltered.filter_sql = None - unfiltered.filter_columns = [] - - # Only include dimension joins + this measure's filter joins - needed = _needed_join_aliases(enriched, extra_columns=measure.filter_columns) - - is_first_or_last = measure.aggregation in ("first", "last") - - if is_first_or_last and enriched.last_agg_time_column: - # Build a ranked subquery within this CTE so _last_rn/_first_rn - # columns exist for the MAX(CASE WHEN _rn = 1 ...) aggregate. - scoped = copy.copy(enriched) - scoped.measures = [unfiltered] - scoped.resolved_joins = [ - (t, a, c, j) for t, a, c, j in enriched.resolved_joins - if a in needed - ] - fm_available = needed | {enriched.model_name} - scoped.filters = [ - f for f in enriched.filters - if not f.is_post_filter and _filter_references_available(f, fm_available) - ] - - from_clause = self._build_from_clause(enriched=enriched) - ( - ranked_from, - rn_suffix_map, - _filtered_rn_map, - _filtered_match_map, - ) = self._build_last_ranked_from( - enriched=scoped, base_from=from_clause, - ) - - select = exp.Select() - group_exprs: list[exp.Expression] = [] - # Dimensions are already resolved inside the ranked subquery - for dim in enriched.dimensions: - col_expr = exp.Column(this=self._to_ident(dim.name)) - select = select.select(col_expr.as_(dim.alias)) - group_exprs.append(col_expr) - for td in enriched.time_dimensions: - col_expr = exp.Column(this=exp.to_identifier(f"_td_{td.name}")) - select = select.select(col_expr.as_(td.alias)) - group_exprs.append(col_expr) - - agg_expr, _ = self._build_agg( - measure=unfiltered, - rn_suffix_map=rn_suffix_map, - default_time_col=enriched.last_agg_time_column, - ) - agg_expr = _wrap_cast_for_type(agg_expr, measure.type) - select = select.select(agg_expr.as_(measure.alias)) - select = select.from_(ranked_from) - # WHERE already inside ranked subquery - else: - # Standard aggregation (sum, avg, etc.) - select = exp.Select() - group_exprs = [] - for dim in enriched.dimensions: - col_expr = self._resolve_sql(sql=dim.sql, name=dim.name, model_name=dim.model_name, type=dim.type) - select = select.select(col_expr.as_(dim.alias)) - group_exprs.append(col_expr) - for td in enriched.time_dimensions: - col_expr = self._resolve_sql(sql=td.sql, name=td.name, model_name=td.model_name) - td_expr = self._build_date_trunc(col_expr=col_expr, granularity=td.granularity) - select = select.select(td_expr.as_(td.alias)) - group_exprs.append(td_expr) - - agg_expr, _ = self._build_agg(measure=unfiltered) - agg_expr = _wrap_cast_for_type(agg_expr, measure.type) - select = select.select(agg_expr.as_(measure.alias)) - - from_clause = self._build_from_clause(enriched=enriched) - select = select.from_(from_clause) - - for target_table, target_alias, join_cond, jtype in enriched.resolved_joins: - if target_alias in needed: - if target_table.startswith("("): - join_target = exp.Subquery( - this=self._parse(target_table), - alias=exp.to_identifier(target_alias), - ) - else: - join_target = self._to_table(target_table, alias=target_alias) - join_on = self._parse(join_cond) - select = select.join(join_target, on=join_on, join_type=jtype.upper()) - - # Only include WHERE conditions whose tables are in this CTE - fm_available = needed | {enriched.model_name} - original_filters = enriched.filters - enriched.filters = [f for f in original_filters - if _filter_references_available(f, fm_available)] - where_clause, _ = self._build_where_and_having(enriched=enriched) - enriched.filters = original_filters - if where_clause is not None: - select = select.where(where_clause) - - for gb in group_exprs: - select = select.group_by(gb) - - ctes.append((cte_name, select.sql(dialect=self.dialect))) - measure_cte_refs.append((cte_name, measure.alias, None)) - - # --- Build combined SELECT: _base LEFT JOIN measure CTEs --- - base_cols = list(dim_aliases) + list(td_aliases) - for m in enriched.measures: - if not _has_cross_model_filter(m) and not _is_windowed_measure(m): - base_cols.append(m.alias) - final_parts = [f'_base.{self._q(a)}' for a in base_cols] - for cte_name, alias, _ in measure_cte_refs: - final_parts.append(f'{cte_name}.{self._q(alias)}') - - from_clause_str = "FROM _base" - joined_ctes: set = set() - for cte_name, _, cte_join_aliases in measure_cte_refs: - if cte_name in joined_ctes: - continue - joined_ctes.add(cte_name) - - # Use per-CTE join aliases when available (re-rooted CTEs may - # have fewer dims than the main query if some were unreachable). - effective_aliases = cte_join_aliases if cte_join_aliases is not None else join_aliases - join_on_parts = [] - for a in effective_aliases: - join_on_parts.append(f'_base.{self._q(a)} = {cte_name}.{self._q(a)}') - if join_on_parts: - from_clause_str += f"\nLEFT JOIN {cte_name} ON {' AND '.join(join_on_parts)}" - else: - from_clause_str += f"\nCROSS JOIN {cte_name}" - - combined_select = ( - f"SELECT {', '.join(final_parts)}\n" - f"{from_clause_str}" - ) - ctes.append(("_combined", combined_select)) - return ctes - - def _assemble_combined_sql(self, enriched: EnrichedQuery, - measure_ctes: list[tuple[str, str]]) -> str: - """Assemble measure CTEs into final SQL with pagination. - - The last entry in measure_ctes is the combined SELECT that joins _base - with measure CTEs. Earlier entries become WITH clauses. + return exp.to_identifier(name, quoted=True).sql(dialect=self.dialect) + + def _null_safe_join_pair_sql(self, *, left_sql: str, right_sql: str) -> str: + """Render one dialect-aware null-safe equality (DEV-1708 / Codex F2) for + a grain join-back ``ON`` clause. ``left_sql`` / ``right_sql`` are the + already-quoted qualified column strings (``_base."x"`` / ``_cm."x"``); + they are parsed back to AST so the dialect strategy's + ``build_null_safe_eq`` can wrap them (native ``IS NOT DISTINCT FROM`` / + ``<=>`` / ``IS``, or the expanded ``= … OR (… IS NULL AND … IS NULL)``).""" + left = self._parse(left_sql) + right = self._parse(right_sql) + return self._dialect.build_null_safe_eq(left, right).sql(dialect=self.dialect) + + def _ordered(self, order_col: exp.Expression, *, ascending: bool) -> exp.Ordered: + """Build an ``exp.Ordered`` node, suppressing sqlglot's NULLS-emulation + ``CASE WHEN`` on T-SQL (DEV-1571 Bug 2 / DEV-1716). + + On T-SQL, sqlglot emits ``CASE WHEN IS NULL THEN 1 ELSE 0 END, + `` to emulate NULLS ordering whenever ``nulls_first`` is unset; + the bracketed alias INSIDE the CASE WHEN mis-resolves against the FROM + scope (``Invalid column name``). Pinning ``nulls_first`` to T-SQL's + native default for the direction (FIRST on ASC, LAST on DESC) + suppresses the wrapper. No-op on every other dialect. """ - inner_ctes = measure_ctes[:-1] - combined_select = measure_ctes[-1][1] - - cte_strs = [f"{name} AS (\n{sql}\n)" for name, sql in inner_ctes] - sql = f"WITH {', '.join(cte_strs)}\n{combined_select}" + kwargs: dict = {"this": order_col, "desc": not ascending} + if self.dialect == "tsql": + kwargs["nulls_first"] = ascending + return exp.Ordered(**kwargs) - # ORDER BY: use _base. for dimensions (ambiguous across CTEs), - # bare alias for measure CTE columns (not in _base) - if enriched.order: - order_parts = [] - base_cols = set(d.alias for d in enriched.dimensions) | set(td.alias for td in enriched.time_dimensions) - base_cols |= { - m.alias for m in enriched.measures - if not _has_cross_model_filter(m) and not _is_windowed_measure(m) - } - for order_item in enriched.order: - col = order_item.column - ref = self._resolve_order_column(col=col, enriched=enriched) - direction = "ASC" if order_item.direction == "asc" else "DESC" - if ref.is_alias and ref.text in base_cols: - order_parts.append(f'_base.{self._q(ref.text)} {direction}') - elif ref.is_alias: - order_parts.append(f'{self._q(ref.text)} {direction}') - else: - order_parts.append(f'{self._order_split_sql(ref)} {direction}') - sql += "\nORDER BY " + ", ".join(order_parts) - if enriched.limit is not None: - sql += f"\nLIMIT {enriched.limit}" - if enriched.offset is not None: - sql += f"\nOFFSET {enriched.offset}" - - return sql - def _apply_pagination_to_sql(self, enriched: EnrichedQuery, sql: str) -> str: - """Apply ORDER BY, LIMIT, OFFSET to a raw SQL string.""" - if enriched.order: - order_parts = [] - for order_item in enriched.order: - col = order_item.column - ref = SQLGenerator._resolve_order_column(col=col, enriched=enriched) - direction = "ASC" if order_item.direction == "asc" else "DESC" - if ref.is_alias: - order_parts.append(f'{self._q(ref.text)} {direction}') - else: - order_parts.append(f'{self._order_split_sql(ref)} {direction}') - sql += "\nORDER BY " + ", ".join(order_parts) - if enriched.limit is not None: - sql += f"\nLIMIT {enriched.limit}" - if enriched.offset is not None: - sql += f"\nOFFSET {enriched.offset}" - return sql - def _generate_shifted_base(self, enriched: EnrichedQuery, transform) -> str: - """Generate a shifted sub-query for a time_shift transform. - - Shifts the time dimension column expression by -offset so that the - WHERE, SELECT, and GROUP BY all reference shifted time. Only includes - the target measure (not all measures). - - For example, time_shift(revenue:sum, -1, 'month') with date_range - [2024-03-01, 2024-03-31] produces a sub-query where the time column - is (created_at + INTERVAL '1' MONTH). This makes the WHERE fetch - February data and the GROUP BY bucket it into March, aligning with - the base query for a simple equality join. - """ - # Determine granularity: explicit or from time dim - gran = transform.granularity - if not gran: - for td in enriched.time_dimensions: - if td.alias == transform.time_alias: - gran = td.granularity.value - break - if not gran: - gran = "month" - # Find target measure - target_measure = next( - (m for m in enriched.measures if m.alias == transform.measure_alias), - None, - ) - if target_measure is None: - raise ValueError( - f"time_shift target measure '{transform.measure_alias}' not found " - f"in enriched query measures" - ) - - # Create shifted time dimensions with offset baked into td.sql - shifted_tds = [] - time_col_map: dict[str, str] = {} # original_qualified → shifted_sql - for td in enriched.time_dimensions: - shifted_td = copy.copy(td) - raw_sql = td.sql or td.name - raw_expr = self._resolve_sql(sql=raw_sql, name=td.name, model_name=td.model_name) - shifted_expr = self._build_time_offset_expr( - col_expr=raw_expr, offset=-transform.offset, granularity=gran, - ) - shifted_td.sql = shifted_expr.sql(dialect=self.dialect) - shifted_tds.append(shifted_td) - # Track for filter substitution - original_qualified = f"{enriched.model_name}.{td.name}" - time_col_map[original_qualified] = shifted_td.sql - - # Substitute time column references in filter SQL strings - shifted_filters = [] - for f in enriched.filters: - if f.is_post_filter: - continue - sf = copy.copy(f) - for orig, shifted_sql in time_col_map.items(): - sf.sql = sf.sql.replace(orig, f"({shifted_sql})") - shifted_filters.append(sf) - - # Build minimal enriched query with only the target measure - shifted_enriched = EnrichedQuery( - model_name=enriched.model_name, - sql_table=enriched.sql_table, - sql=enriched.sql, - resolved_joins=enriched.resolved_joins, - dimensions=list(enriched.dimensions), - measures=[target_measure], - time_dimensions=shifted_tds, - filters=shifted_filters, - ) - return self._generate_base(enriched=shifted_enriched) def _build_time_offset_expr(self, col_expr: exp.Expression, offset: int, granularity: str) -> exp.Expression: @@ -1133,7 +925,7 @@ def _build_time_offset_expr(self, col_expr: exp.Expression, offset: int, def _duration_interval_exprs(self, duration: str, sign: int = 1) -> list[exp.Expression]: """Return per-unit AST nodes that `_add_intervals_expr` will chain. - Delegates to the dialect strategy — Postgres-shape returns + Delegates to the dialect strategy (DEV-1716) — Postgres-shape returns ``exp.Interval`` nodes; SQLite returns DATETIME-modifier string literals with sign baked in. """ @@ -1144,7 +936,8 @@ def _granularity_interval_expr(self, granularity: TimeGranularity, sign: int = 1 if granularity == TimeGranularity.QUARTER: duration = "3m" elif granularity in (TimeGranularity.WEEK, TimeGranularity.WEEK_SUNDAY): - # DEV-1572: a one-period shift of a Sunday-week is one week. + # DEV-1572: a WEEK_SUNDAY shift spans one calendar week, same as WEEK + # (only the bucket anchor differs — Sunday vs Monday). duration = "1w" else: unit_to_duration = { @@ -1162,727 +955,49 @@ def _add_intervals_expr(self, expr: exp.Expression, intervals: list[exp.Expressi sign: int = 1) -> exp.Expression: """Compose `expr ± interval [± interval ...]` as AST. - Delegates to the dialect strategy — defaults to chained Add/Sub - with ``exp.Interval`` nodes; SQLite wraps as ``DATETIME(...)``; + Delegates to the dialect strategy (DEV-1716) — defaults to chained + Add/Sub with ``exp.Interval`` nodes; SQLite wraps as ``DATETIME(...)``; T-SQL chains ``DATEADD(...)`` calls. """ return self._dialect.add_intervals_expr( expr=expr, intervals=intervals, sign=sign, ) - def _build_window_source_cols( - self, - *, - enriched: EnrichedQuery, - td, - measure: EnrichedMeasure, - ) -> tuple[list[exp.Alias], list[exp.Condition]]: - """Build the SELECT columns and base equality predicates for the _src subquery. - - The trailing-window range predicate (`_src._w_time >= ...`) is added later - by the caller; only the equality joins on dims and other time dims are - produced here. - - Returns (source_cols, join_eqs) where source_cols are alias-wrapped - expressions ready to feed `exp.Select.select(...)` and join_eqs are - `exp.EQ` predicates ready to combine with `exp.and_`. - """ - source_cols: list[exp.Alias] = [] - join_eqs: list[exp.Condition] = [] - - def _src_col(name: str) -> exp.Column: - return exp.Column(this=exp.to_identifier(name), table=exp.to_identifier("_src")) - - def _base_col(alias: str) -> exp.Column: - return exp.Column(this=exp.to_identifier(alias), table=exp.to_identifier("_base")) - - for idx, dim in enumerate(enriched.dimensions): - col_expr = self._resolve_sql(sql=dim.sql, name=dim.name, model_name=dim.model_name, type=dim.type) - src_alias = f"_w_dim_{idx}" - source_cols.append(col_expr.as_(src_alias)) - join_eqs.append(exp.EQ(this=_src_col(src_alias), expression=_base_col(dim.alias))) - - # Equality-join on every other time dim so the trailing window does not - # fan out across their values when the query has 2+ time dimensions. - for idx, other_td in enumerate(enriched.time_dimensions): - if other_td.alias == td.alias: - continue - other_expr = self._resolve_sql( - sql=other_td.sql or other_td.name, - name=other_td.name, - model_name=other_td.model_name, - ) - other_bucket = self._build_date_trunc( - col_expr=other_expr, - granularity=other_td.granularity, - ) - other_alias = f"_w_td_{idx}" - source_cols.append(other_bucket.as_(other_alias)) - join_eqs.append(exp.EQ(this=_src_col(other_alias), expression=_base_col(other_td.alias))) - - raw_time_expr = self._resolve_sql(sql=td.sql or td.name, name=td.name, model_name=td.model_name) - source_cols.append(raw_time_expr.as_("_w_time")) - - value_expr = self._resolve_sql(sql=measure.sql or measure.name, name=measure.name, model_name=measure.model_name) - if measure.filter_sql: - # measure.filter_sql is a user-supplied predicate (originates from - # ``Column.filter`` / ``SlayerQuery.filters``); parse it via - # ``_parse_predicate`` so dialects whose statement keywords - # shadow function calls at expression start (SQLite / MySQL - # ``REPLACE``) don't fall back to a Command parse — DEV-1378. - filter_ast = self._parse_predicate(measure.filter_sql) - value_expr = exp.Case(ifs=[exp.If(this=filter_ast, true=value_expr)]) - source_cols.append(value_expr.as_("_w_value")) - - return source_cols, join_eqs - - def _window_referenced_aliases( - self, - *, - source_cols: list[exp.Alias], - measure: EnrichedMeasure, - filters, - ) -> set[str]: - """Aliases the windowed-CTE actually references; drives join pruning. - - Scans rendered `source_cols` SQL, the measure's filter_sql, and column - paths of every non-post query filter (so a WHERE on customers.x keeps - the customers join even if no other thing references it). Path aliases - use "__" so each is one identifier token; for multi-hop aliases like - "customers__regions" we also include every "__"-split prefix - ("customers") via `_alias_prefixes` so the transitive joins those - reference are kept too. - """ - rendered_cols = " ".join(c.sql(dialect=self.dialect) for c in source_cols) - referenced_text = rendered_cols - if measure.filter_sql: - referenced_text += " " + measure.filter_sql - referenced: set[str] = set() - for tok in re.findall(r'(?:^|[^\w."\'])([A-Za-z_]\w*)\.', referenced_text): - referenced.update(_alias_prefixes(tok)) - for col in _filter_dotted_columns(filters): - referenced.update(_alias_prefixes(col)) - return referenced - - def _build_window_source_select( - self, - *, - enriched: EnrichedQuery, - source_cols: list[exp.Alias], - measure: EnrichedMeasure, - ) -> exp.Select: - """Build the _src subquery: SELECT ... FROM ... [filtered JOINs] [WHERE ...] as AST. - - Only joins whose target_alias is referenced by source_cols (or by the - measure's filter SQL) are included — pulling in unrelated joins can - change row multiplicity for the windowed aggregation, breaking the - "adding a measure must not affect cardinality" core principle. - """ - select = exp.Select().select(*source_cols).from_(self._build_from_clause(enriched=enriched)) - referenced = self._window_referenced_aliases( - source_cols=source_cols, measure=measure, filters=enriched.filters, - ) - - for target_table, target_alias, join_cond, jtype in enriched.resolved_joins: - if target_alias not in referenced: - continue - if target_table.startswith("("): - join_target = exp.Subquery( - this=self._parse(target_table), - alias=exp.to_identifier(target_alias), - ) - else: - join_target = self._to_table(target_table, alias=target_alias) - join_on = self._parse(join_cond) - select = select.join(join_target, on=join_on, join_type=jtype.upper()) - - scoped = copy.copy(enriched) - scoped.time_dimensions = [ - t.model_copy(update={"date_range": None}) for t in enriched.time_dimensions - ] - where_clause, _ = self._build_where_and_having(enriched=scoped) - if where_clause is not None: - select = select.where(where_clause) - - return select - - def _generate_window_measure_cte(self, enriched: EnrichedQuery, measure: EnrichedMeasure) -> str: - if measure.aggregation not in ("sum", "avg"): - raise ValueError("Windowed aggregations are only supported for sum and avg") - if not measure.window or not measure.window_time_alias: - raise ValueError(f"Windowed measure '{measure.alias}' is missing window metadata") - - td = next((t for t in enriched.time_dimensions if t.alias == measure.window_time_alias), None) - if td is None: - raise ValueError(f"Windowed measure '{measure.alias}' could not resolve its time dimension") - - group_aliases = [d.alias for d in enriched.dimensions] + [t.alias for t in enriched.time_dimensions] - source_cols, join_eqs = self._build_window_source_cols( - enriched=enriched, td=td, measure=measure, - ) - src_select = self._build_window_source_select( - enriched=enriched, source_cols=source_cols, measure=measure, - ) - src_subq = exp.Subquery(this=src_select, alias=exp.TableAlias(this=exp.to_identifier("_src"))) - - frame_time = exp.Column(this=exp.to_identifier(td.alias), table=exp.to_identifier("_base")) - bucket_end = self._add_intervals_expr( - frame_time, - self._granularity_interval_expr(td.granularity, sign=1), - sign=1, - ) - lower_bound = self._add_intervals_expr( - bucket_end, - self._duration_interval_exprs(measure.window, sign=-1), - sign=-1, - ) - src_w_time = exp.Column(this=exp.to_identifier("_w_time"), table=exp.to_identifier("_src")) - # bucket_end may be referenced both as upper bound and as base for the - # lower bound — clone so the AST has independent subtrees. - on_expr = exp.and_( - *join_eqs, - exp.GTE(this=src_w_time, expression=lower_bound), - exp.LT(this=src_w_time.copy(), expression=bucket_end.copy()), - ) - - agg_cls = exp.Sum if measure.aggregation == "sum" else exp.Avg - agg_input = exp.Column(this=exp.to_identifier("_w_value"), table=exp.to_identifier("_src")) - - outer = exp.Select() - for a in group_aliases: - outer = outer.select(exp.Column(this=exp.to_identifier(a), table=exp.to_identifier("_base"))) - agg_expr = _wrap_cast_for_type(agg_cls(this=agg_input), measure.type) - outer = outer.select(agg_expr.as_(measure.alias)) - outer = outer.from_(exp.Table(this=exp.to_identifier("_base"))) - outer = outer.join(src_subq, on=on_expr, join_type="LEFT") - for a in group_aliases: - outer = outer.group_by(exp.Column(this=exp.to_identifier(a), table=exp.to_identifier("_base"))) - - return outer.sql(dialect=self.dialect, pretty=True) - - def _generate_base(self, enriched: EnrichedQuery, - skip_isolated: bool = False) -> str: - """Generate the base SELECT (measures, dimensions, filters).""" - from_clause = self._build_from_clause(enriched=enriched) - - # If any measure has first/last aggregation, prepend a ROW_NUMBER CTE - # to mark the latest (or earliest) row per group. - # When skip_isolated is set, only consider non-isolated measures — isolated - # first/last measures get their own ranked subquery in their CTE. - if skip_isolated: - has_first_or_last = any( - m.aggregation in ("first", "last") and not _has_cross_model_filter(m) - for m in enriched.measures - ) - else: - has_first_or_last = any(m.aggregation in ("first", "last") for m in enriched.measures) - rn_suffix_map: dict[str, str] = {} - filtered_rn_map: dict[str, str] = {} - filtered_match_map: dict[str, str] = {} - if has_first_or_last and enriched.last_agg_time_column: - ( - from_clause, - rn_suffix_map, - filtered_rn_map, - filtered_match_map, - ) = self._build_last_ranked_from( - enriched=enriched, base_from=from_clause, - ) - - select_columns = [] - group_by_columns = [] - - for dim in enriched.dimensions: - col_expr = self._resolve_sql(sql=dim.sql, name=dim.name, model_name=dim.model_name, type=dim.type) - if has_first_or_last: - # In ranked subquery, dimensions are already columns — reference - # directly (DEV-1645: quote mixed-case names so they match the - # ranked subquery's model.* output column on case-folding dialects) - col_expr = exp.Column(this=self._to_ident(dim.name)) - select_columns.append(col_expr.as_(dim.alias)) - group_by_columns.append(col_expr) - - for td in enriched.time_dimensions: - col_expr = self._resolve_sql(sql=td.sql, name=td.name, model_name=td.model_name) - if has_first_or_last: - # Time dimension is already truncated in the ranked subquery - col_expr = exp.Column(this=exp.to_identifier(f"_td_{td.name}")) - else: - col_expr = self._build_date_trunc(col_expr=col_expr, granularity=td.granularity) - select_columns.append(col_expr.as_(td.alias)) - group_by_columns.append(col_expr) - - has_aggregation = False - for measure in enriched.measures: - if skip_isolated and (_has_cross_model_filter(measure) or _is_windowed_measure(measure)): - continue # Will be handled in its own CTE - agg_expr, is_agg = self._build_agg( - measure=measure, - rn_suffix_map=rn_suffix_map, - default_time_col=enriched.last_agg_time_column, - filtered_rn_map=filtered_rn_map, - filtered_match_map=filtered_match_map, - ) - # DEV-1361: wrap the aggregation result in CAST when the measure - # has a declared result type. - if is_agg: - agg_expr = _wrap_cast_for_type(agg_expr, measure.type) - select_columns.append(agg_expr.as_(measure.alias)) - if is_agg: - has_aggregation = True - - # When all measures are isolated/cross-model and there are no dimensions, - # the base SELECT would be empty. Add a placeholder to produce valid SQL. - if not select_columns and skip_isolated: - select_columns.append(exp.Literal.number(1).as_("_placeholder")) - - where_clause, having_clause = self._build_where_and_having( - enriched=enriched, - rn_suffix_map=rn_suffix_map, - filtered_rn_map=filtered_rn_map, - ) - - select = exp.Select() - for col in select_columns: - select = select.select(col) - - select = select.from_(from_clause) - - # When using ranked subquery for type=last, WHERE is already inside the subquery - if where_clause is not None and not has_first_or_last: - select = select.where(where_clause) - - # Group by when there are aggregations, cross-model measures exist, - # isolated measures were skipped (to deduplicate the dimension spine), - # or the query is dim-only (auto-dedup distinct dim/time-dim tuples - # — applied before LIMIT so a row cap can't drop unique tuples). - # DEV-1543: the dim-only auto-dedup is gated on the query's - # ``distinct_dimension_values`` flag. Default True preserves the - # Cube.js-style dedup; setting False emits raw rows. - dim_only_dedup = ( - enriched.distinct_dimension_values - and bool(group_by_columns) - and not enriched.measures - ) - needs_group_by = ( - has_aggregation - or bool(enriched.cross_model_measures) - or skip_isolated - or dim_only_dedup - ) - if needs_group_by and group_by_columns: - for gb in group_by_columns: - select = select.group_by(gb) - - if having_clause is not None: - select = select.having(having_clause) - - # When no computed columns and no measure CTEs, apply order/limit/offset - # to the base query. Otherwise, they'll be applied to the outer query. - # DEV-1336: a post-filter requires the outer `_filtered` wrap from - # `_generate_with_computed`; pagination must apply to the filtered - # result, not to the unfiltered base. - has_post_filters = any(getattr(f, "is_post_filter", False) for f in enriched.filters) - if ( - not enriched.expressions - and not enriched.transforms - and not skip_isolated - and not has_post_filters - ): - select = self._apply_order_limit(select=select, enriched=enriched) - - # Append LEFT JOINs from resolved joins via sqlglot AST (works for both - # sql_table and inline-SQL models). - # When has_first_or_last is true, the joins were already injected inside the - # ranked subquery by _build_last_ranked_from — skip here to avoid duplicating. - # When skip_isolated, only include joins needed for dimensions (not filter-target - # joins of isolated measures, which would cause conflicting INNER JOIN intersections). - dim_only_aliases = _needed_join_aliases(enriched) if skip_isolated else None - if dim_only_aliases is not None: - # Also include aliases needed by WHERE-clause filters - for f in enriched.filters: - if not f.is_post_filter: - for col in f.columns: - if "." in col: - parts = col.split(".") - for i in range(1, len(parts)): - dim_only_aliases.add("__".join(parts[:i])) - resolved_joins = enriched.resolved_joins - if dim_only_aliases is not None: - resolved_joins = [(t, a, c, j) for t, a, c, j in resolved_joins if a in dim_only_aliases] - if resolved_joins and not has_first_or_last: - for target_table, target_alias, join_cond, jtype in resolved_joins: - if target_table.startswith("("): - # Inline-SQL target: parse as subquery - parsed_target = self._parse(target_table) - join_target = exp.Subquery( - this=parsed_target, alias=exp.to_identifier(target_alias), - ) - else: - join_target = self._to_table(target_table, alias=target_alias) - join_on = self._parse(join_cond) - select = select.join(join_target, on=join_on, join_type=jtype.upper()) - - sql = select.sql(dialect=self.dialect, pretty=True) - - return sql - - def _generate_with_computed(self, enriched: EnrichedQuery, - base_sql: str | None = None, - prefix_ctes: list[tuple[str, str]] | None = None) -> str: - """Wrap the base query as a CTE and add expressions/transforms as stacked CTE layers. - - Transforms that reference other transforms' outputs get their own CTE layer. - This handles arbitrary nesting like change(cumsum(revenue)). - - Args: - base_sql: Base SQL to wrap as "base" CTE (simple case, no measure CTEs). - prefix_ctes: Pre-built CTE list from _build_combined(). When provided, - these are used as the initial CTE stack instead of wrapping base_sql. - The last entry is the "combined" CTE with all measure values available. - """ - # Collect base aliases (includes all measures — combined SQL has them all) - base_aliases = [] - for dim in enriched.dimensions: - base_aliases.append(dim.alias) - for td in enriched.time_dimensions: - base_aliases.append(td.alias) - for m in enriched.measures: - base_aliases.append(m.alias) - for cm in enriched.cross_model_measures: - base_aliases.append(cm.alias) - # Build stacked CTEs. Each layer can reference aliases from previous layers. - if prefix_ctes is not None: - ctes = list(prefix_ctes) - else: - ctes = [("base", base_sql)] - available_aliases = set(base_aliases) # Aliases available in the current layer - - # All transforms go into a unified layering loop. Each iteration tries - # to resolve transforms whose inputs are available. Self-join transforms - # (time_shift, change, change_pct) get their own CTE with a LEFT JOIN. - # Window transforms (cumsum, lag, lead, rank, last) are batched into a - # single CTE layer with OVER() expressions. - # All measure aliases are available in base_sql (combined CTE includes - # cross-model and isolated filtered measures via LEFT JOIN). - pending_expressions = list(enriched.expressions) - pending_transforms = list(enriched.transforms) - layer_num = 0 - while pending_expressions or pending_transforms: - layer_num += 1 - prev_cte = ctes[-1][0] - added_this_layer = [] - remaining_expressions = [] - remaining_transforms = [] - - # Collect window transforms and expressions that can go in one layer - layer_parts = [self._q(a) for a in sorted(available_aliases)] - - for expr in pending_expressions: - if self._deps_available(expr.sql, available_aliases): - # DEV-1361: when the source ModelMeasure declared a - # result type, wrap the expression in CAST so the outer - # SELECT yields the typed value. - # DEV-1571 Bug 3 follow-up: ``expr.sql`` comes from - # enrichment (``_resolve_sql``) with hardcoded ANSI - # double-quoted aliases. Parse in postgres dialect - # (where ``"x"`` is an identifier) and re-emit in the - # active dialect so MySQL backticks / T-SQL brackets - # land throughout the expression. - parsed = self._parse(expr.sql, dialect="postgres") - if expr.type is not None: - parsed = _wrap_cast_for_type(parsed, expr.type) - expr_sql = parsed.sql(dialect=self.dialect) - layer_parts.append(f'{expr_sql} AS {self._q(expr.alias)}') - added_this_layer.append(expr.alias) - else: - remaining_expressions.append(expr) - - # Batch window-function transforms into this layer - deferred_self_joins = [] - deferred_consecutive_periods = [] - for t in pending_transforms: - if t.measure_alias not in available_aliases: - remaining_transforms.append(t) - elif t.transform in _SELF_JOIN_TRANSFORMS: - deferred_self_joins.append(t) # Handle after window layer - elif t.transform == "consecutive_periods": - deferred_consecutive_periods.append(t) - else: - window_sql = self._build_transform_sql(t) - # DEV-1361: wrap in CAST when the source ModelMeasure - # declared a result type (propagated to t.type at - # enrichment time). - if t.type is not None: - wrapped = _wrap_cast_for_type(self._parse(window_sql), t.type) - window_sql = wrapped.sql(dialect=self.dialect) - layer_parts.append(f'{window_sql} AS {self._q(t.alias)}') - added_this_layer.append(t.alias) - - # Emit window layer CTE if anything was added - if added_this_layer: - layer_name = f"step{layer_num}" - layer_select = "SELECT\n " + _SQL_COL_SEP.join(layer_parts) - ctes.append((layer_name, f"{layer_select}\nFROM {prev_cte}")) - available_aliases.update(added_this_layer) - - # Now emit each self-join transform as its own CTE layer. - # The shifted sub-query has the time offset baked into td.sql, - # so we always join on time column equality (calendar-based). - for t in deferred_self_joins: - src_cte = ctes[-1][0] - - shift_name = f"shifted_{t.name}" - shifted_sql = self._generate_shifted_base( - enriched=enriched, transform=t, - ) - ctes.append((shift_name, shifted_sql)) - - # Build the self-join CTE: src LEFT JOIN shifted ON time equality - time_col = self._q(t.time_alias) - join_cond = f'{src_cte}.{time_col} = {shift_name}.{time_col}' - # Also join on all dimension columns for correct matching - for dim in enriched.dimensions: - join_cond += ( - f' AND {src_cte}.{self._q(dim.alias)} ' - f'= {shift_name}.{self._q(dim.alias)}' - ) - col_sql = self._build_self_join_column( - transform=t.transform, right_table=shift_name, - measure_alias=t.measure_alias, - ) - join_cols = ", ".join( - f'{src_cte}.{self._q(a)}' for a in sorted(available_aliases) - ) - join_layer = f"sjoin_{t.name}" - join_sql = ( - f"SELECT {join_cols}, {col_sql} AS {self._q(t.alias)}\n" - f"FROM {src_cte}\n" - f"LEFT JOIN {shift_name}\n" - f" ON {join_cond}" - ) - ctes.append((join_layer, join_sql)) - available_aliases.add(t.alias) - added_this_layer.append(t.alias) - - # consecutive_periods needs two window layers: one to compute the - # reset group, then one to count within that group. Most SQL - # engines reject nested window functions in a single SELECT. - for t in deferred_consecutive_periods: - reset_layer, value_layer = self._build_consecutive_periods_ctes( - transform=t, - source_cte=ctes[-1][0], - available_aliases=available_aliases, - layer_num=layer_num, - ) - ctes.extend(reset_layer) - ctes.extend(value_layer) - available_aliases.add(t.alias) - added_this_layer.append(t.alias) - - if not added_this_layer: - remaining_transforms.extend(deferred_self_joins) - remaining_transforms.extend(deferred_consecutive_periods) - break # Nothing could be added — remaining items have unresolved deps - - pending_expressions = remaining_expressions - pending_transforms = remaining_transforms - - # Build final CTE clause - cte_strs = [f"{name} AS (\n{sql}\n)" for name, sql in ctes] - cte_clause = "WITH " + ",\n".join(cte_strs) - final_cte = ctes[-1][0] - # Build final SELECT - final_parts = [self._q(a) for a in sorted(available_aliases)] - - # Add any remaining expressions/transforms that couldn't be layered. - # DEV-1571 Bug 3 follow-up: re-emit each expression through the - # active dialect so ANSI-quoted aliases from enrichment become - # MySQL backticks / T-SQL brackets. - for expr in pending_expressions: - expr_sql = self._parse(expr.sql, dialect="postgres").sql(dialect=self.dialect) - final_parts.append(f'{expr_sql} AS {self._q(expr.alias)}') - for t in pending_transforms: - if t.transform in _SELF_JOIN_TRANSFORMS: - continue # Should not happen — self-joins are always materialized - if t.transform == "consecutive_periods": - raise ValueError("consecutive_periods could not be materialized") - window_sql = self._build_transform_sql(t) - if t.type is not None: - wrapped = _wrap_cast_for_type(self._parse(window_sql), t.type) - window_sql = wrapped.sql(dialect=self.dialect) - final_parts.append(f'{window_sql} AS {self._q(t.alias)}') - - outer_select = "SELECT\n " + _SQL_COL_SEP.join(final_parts) - - sql = f"{cte_clause}\n{outer_select}\nFROM {final_cte}" - - # Apply post-filters (filters referencing computed columns) BEFORE - # pagination, so LIMIT/OFFSET operate on the filtered result. - post_filters = [f for f in enriched.filters if f.is_post_filter] - if post_filters: - import re - model = enriched.model_name - conditions = [] - for f in post_filters: - qualified_sql = f.sql - for col_name in dict.fromkeys(f.columns): - qualified_sql = re.sub( - rf'(? bool: - """Check if all quoted aliases referenced in SQL are in the available set.""" - import re - refs = re.findall(r'"([^"]+)"', sql) - return all(ref in available for ref in refs) - def _build_consecutive_periods_ctes( - self, - transform, - source_cte: str, - available_aliases: set[str], - layer_num: int, - ) -> tuple[list[tuple[str, str]], list[tuple[str, str]]]: - partition_aliases = getattr(transform, "partition_aliases", []) or [] - reset_alias = _cte_name_from_alias("_cp_reset_", transform.alias) - reset_cte = _cte_name_from_alias(f"cp_reset_{layer_num}_", transform.alias) - value_cte = _cte_name_from_alias(f"cp_value_{layer_num}_", transform.alias) - - def _quoted_col(name: str) -> exp.Column: - return exp.Column(this=exp.to_identifier(name, quoted=True)) - - measure_col = _quoted_col(transform.measure_alias) - time_col = _quoted_col(transform.time_alias) - # Bare column inside exp.Order, NOT wrapped in exp.Ordered — sqlglot - # otherwise injects `NULLS LAST` on SQLite (and Spark/Databricks), - # changing streak/reset semantics for any NULL time values vs the - # pre-AST string-built `ORDER BY ` output. - order = exp.Order(expressions=[time_col]) - spec = exp.WindowSpec( - kind="ROWS", - start="UNBOUNDED", - start_side="PRECEDING", - end="CURRENT ROW", - ) - - # Wrap measure in an explicit boolean predicate so non-boolean argument - # expressions don't rely on dialect-specific truthiness coercion in - # CASE WHEN. Postgres rejects non-boolean WHEN outright; SQLite/MySQL - # coerce non-zero to true; ClickHouse has its own rules. - # When the inner expression is already boolean (e.g. - # `consecutive_periods(revenue:sum > 0)`), the numeric `<> 0` form - # is itself rejected by Postgres ("operator does not exist: - # boolean <> integer"), so we use the column directly inside CASE WHEN. - def _predicate() -> exp.Expression: - if getattr(transform, "predicate_is_boolean", False): - return exp.func("COALESCE", measure_col.copy(), exp.false()) - return exp.and_( - exp.Is(this=measure_col.copy(), expression=exp.Not(this=exp.Null())), - exp.NEQ(this=measure_col.copy(), expression=exp.Literal.number(0)), - ) - - source_col_exprs = [_quoted_col(a) for a in sorted(available_aliases)] - - # reset CTE: SELECT , SUM(CASE WHEN pred THEN 0 ELSE 1 END) - # OVER (PARTITION BY ... ORDER BY t ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) - # AS "" FROM source_cte - reset_case = exp.Case( - ifs=[exp.If(this=_predicate(), true=exp.Literal.number(0))], - default=exp.Literal.number(1), - ) - reset_window = exp.Window( - this=exp.Sum(this=reset_case), - partition_by=[_quoted_col(a) for a in partition_aliases] or None, - order=order, - spec=spec, - ) - reset_select = ( - exp.Select() - .select(*[c.copy() for c in source_col_exprs]) - .select(reset_window.as_(reset_alias, quoted=True)) - .from_(exp.Table(this=exp.to_identifier(source_cte))) - ) - # value CTE: SELECT , - # CASE WHEN pred THEN SUM(CASE WHEN pred THEN 1 ELSE 0 END) - # OVER (PARTITION BY ..., "" ORDER BY t ROWS ...) ELSE 0 END - # AS "" FROM reset_cte - value_inner_case = exp.Case( - ifs=[exp.If(this=_predicate(), true=exp.Literal.number(1))], - default=exp.Literal.number(0), - ) - value_partition = ( - [_quoted_col(a) for a in partition_aliases] + [_quoted_col(reset_alias)] - ) - value_window = exp.Window( - this=exp.Sum(this=value_inner_case), - partition_by=value_partition, - order=order.copy(), - spec=spec.copy(), - ) - value_outer_case = exp.Case( - ifs=[exp.If(this=_predicate(), true=value_window)], - default=exp.Literal.number(0), - ) - value_select = ( - exp.Select() - .select(*[c.copy() for c in source_col_exprs]) - .select(value_outer_case.as_(transform.alias, quoted=True)) - .from_(exp.Table(this=exp.to_identifier(reset_cte))) - ) - reset_sql = reset_select.sql(dialect=self.dialect, pretty=True) - value_sql = value_select.sql(dialect=self.dialect, pretty=True) - return [(reset_cte, reset_sql)], [(value_cte, value_sql)] def _build_date_trunc(self, col_expr: exp.Expression, granularity: TimeGranularity) -> exp.Expression: - """Build a DATE_TRUNC expression. Dispatches to the dialect strategy. + """Build a DATE_TRUNC expression. Dispatches to the dialect strategy + (DEV-1716). The dialect determines the wire form — DATE_TRUNC for - Postgres/DuckDB/ClickHouse, STRFTIME for SQLite (with CASE WHEN - for quarter and weekday-modifier for week), DATETRUNC for T-SQL. - Cast-wrapping of non-column operands is handled inside each - dialect's override. + Postgres/DuckDB/ClickHouse, STRFTIME for SQLite (with CASE WHEN for + quarter and weekday-modifier for week), DATETRUNC for T-SQL, native + Sunday-week for BigQuery. Cast-wrapping of non-column operands and the + WEEK_SUNDAY day-shift are handled inside the dialect (base) impl. """ return self._dialect.build_date_trunc( col_expr=col_expr, granularity=granularity, parse=self._parse, ) def _build_transform_sql(self, t) -> str: # NOSONAR S3776 — flat dispatch over transform names; per-transform SQL forms read better as one if/elif tree than as named helpers - """Build a window function SQL expression for a transform.""" - measure = self._q(t.measure_alias) - time_col = self._q(t.time_alias) if t.time_alias else None + """Build a window function SQL expression for a transform. + + DEV-1716: identifier refs are dialect-quoted (``_quote_ident``) so + MySQL/T-SQL/BigQuery get correct quotes; the subsequent + ``self._parse(window_sql)`` reads them back as identifiers (backticks + on MySQL, brackets on T-SQL) rather than string literals. + """ + measure = self._quote_ident(t.measure_alias) + time_col = self._quote_ident(t.time_alias) if t.time_alias else None partition_cols = getattr(t, "partition_aliases", []) or [] partition_clause = ( - "PARTITION BY " + ", ".join(self._q(a) for a in partition_cols) + _SQL_PARTITION_BY + ", ".join(self._quote_ident(a) for a in partition_cols) if partition_cols else "" ) @@ -1928,298 +1043,15 @@ def _build_transform_sql(self, t) -> str: # NOSONAR S3776 — flat dispatch ove else: raise ValueError(f"Unsupported transform: {t.transform}") - def _build_self_join_column(self, transform: str, right_table: str, - measure_alias: str) -> str: - """Build the SELECT expression for a self-join transform.""" - prev = f'{right_table}.{self._q(measure_alias)}' - if transform == "time_shift": - return prev - raise ValueError(f"Unknown self-join transform: {transform}") - - def _apply_order_limit(self, select: exp.Select, enriched: EnrichedQuery) -> exp.Select: - """Apply ORDER BY, LIMIT, OFFSET to a select expression. - - DEV-1571 Bug 2 follow-up: on MySQL / T-SQL, sqlglot emits a - ``CASE WHEN IS NULL THEN 1 ELSE 0 END, `` emulation - of NULLS LAST whenever ``nulls_first`` is unset. On T-SQL the - bracketed alias INSIDE the CASE WHEN is treated as a column-name - lookup against the FROM scope (NOT as a SELECT alias), so the - query fails with ``Invalid column name``. Pin ``nulls_first`` to - the dialect's native default for the requested direction - (NULLS FIRST on ASC, NULLS LAST on DESC) so sqlglot suppresses - the emulation. Behaviourally this means T-SQL inherits T-SQL's - native NULL ordering instead of the Postgres-shaped default — - users who need NULLS LAST on ASC on T-SQL must order on a - non-nullable column or use a coalesce expression. - """ - suppress_nulls_emulation = self.dialect == "tsql" - if enriched.order: - for order_item in enriched.order: - col = order_item.column - ref = self._resolve_order_column(col=col, enriched=enriched) - if ref.is_alias: - order_col = exp.Column(this=exp.to_identifier(ref.text, quoted=True)) - else: - order_col = exp.Column( - this=self._to_ident(ref.column), table=exp.to_identifier(ref.qualifier) - ) - ascending = order_item.direction == "asc" - ordered_kwargs: dict[str, Any] = {"this": order_col, "desc": not ascending} - if suppress_nulls_emulation: - # T-SQL native: NULLS FIRST on ASC, NULLS LAST on DESC. - ordered_kwargs["nulls_first"] = ascending - select = select.order_by(exp.Ordered(**ordered_kwargs)) - - if enriched.limit is not None: - select = select.limit(enriched.limit) - - if enriched.offset is not None: - select = select.offset(enriched.offset) - return select - def _order_split_sql(self, ref: _OrderColRef) -> str: - """DEV-1645: emit a non-projected ORDER BY key as a SPLIT - ``qualifier.column`` reference (mixed-case-quoted), not one - composite-quoted token.""" - col = exp.Column(this=self._to_ident(ref.column), table=exp.to_identifier(ref.qualifier)) - return col.sql(dialect=self.dialect) - @staticmethod - def _resolve_order_column(col, enriched: EnrichedQuery) -> _OrderColRef: - """Resolve an order column reference to a discriminated result. - - Users refer to columns by their short name (e.g., ``count``, - ``revenue_sum``). The enriched query stores fully qualified aliases - (e.g., ``orders._count``, ``orders.revenue_sum``). This method - matches the user-provided name against all enriched columns. - - When it matches a projected alias, the result carries ``is_alias=True`` - and the caller emits it whole-quoted (``"orders.revenue_sum"`` — that IS - the real output column name via ``AS "orders.revenue_sum"``). - - When no projected alias matches (DEV-1645: renamed via ``columns:``, or - an inner-stage dim the outer stage dropped), the result carries - ``is_alias=False`` with ``qualifier``/``column`` set, and the caller - emits a SPLIT ``qualifier.column`` reference (two identifiers) that - resolves against the FROM-scope table — instead of the old composite - ``"."`` token that Postgres rejects as UndefinedColumn. - - For ``*:count`` results, the internal name is ``_count`` but users - refer to it as ``count``. A fallback check for ``_name`` handles - this case. - """ - user_name = col.name - model_prefix = col.model or enriched.model_name - - # Build a lookup: short name → alias for all enriched columns - alias_lookup: dict[str, str] = {} - for d in enriched.dimensions: - alias_lookup[d.name] = d.alias - for td in enriched.time_dimensions: - alias_lookup[td.name] = td.alias - for m in enriched.measures: - alias_lookup[m.name] = m.alias - for e in enriched.expressions: - alias_lookup[e.name] = e.alias - for t in enriched.transforms: - alias_lookup[t.name] = t.alias - for cm in enriched.cross_model_measures: - alias_lookup[cm.name] = cm.alias - # Custom field names (e.g., {"formula": "x:count_distinct", "name": "my_name"}) - alias_lookup.update(enriched.field_name_aliases) - - # Direct match on the user-provided name - if user_name in alias_lookup: - return _OrderColRef(alias_lookup[user_name], True, None, None) - - # Qualified match for cross-model measures: - # col.model="customers", col.name="revenue_sum" → "customers.revenue_sum" - if col.model: - qualified = f"{col.model}.{col.name}" - if qualified in alias_lookup: - return _OrderColRef(alias_lookup[qualified], True, None, None) - - # Fallback for *:count → _count: user says "count", internal is "_count" - prefixed = f"_{user_name}" - if prefixed in alias_lookup: - return _OrderColRef(alias_lookup[prefixed], True, None, None) - - # Fallback: a non-projected order key is only safe to emit as a split - # reference against the BASE-model alias. DEV-1645: a joined qualifier - # (anything other than the base model) is rejected. Even when a filter - # pulls the join into the base FROM, the compiler's outer-wrapping - # layers (measure CTEs, pagination, the first/last ranked subquery, and - # projection trimming) relocate the ORDER BY into a scope where the - # joined table is unbound — emitting the reference there would produce - # invalid SQL. Ordering by a joined column therefore requires projecting - # it (add it to dimensions) or ordering by a projected field. The - # base-model qualifier is still emitted as a split reference (the - # documented measure-CTE ``_base`` base-column case is intentionally not - # rejected). - if model_prefix != enriched.model_name: - raise UnresolvableOrderColumnError(column=user_name, qualifier=model_prefix) - return _OrderColRef(f"{model_prefix}.{user_name}", False, model_prefix, user_name) # ------------------------------------------------------------------ # FROM / JOIN building # ------------------------------------------------------------------ - def _build_from_clause(self, enriched: EnrichedQuery) -> exp.Expression: - if enriched.sql_table: - return self._to_table(enriched.sql_table, alias=enriched.model_name) - elif enriched.sql: - parsed = self._parse(enriched.sql) - return exp.Subquery(this=parsed, alias=exp.to_identifier(enriched.model_name)) - else: - raise ValueError(f"Model '{enriched.model_name}' has neither sql_table nor sql defined") - - def _build_last_ranked_from( - self, - enriched: EnrichedQuery, - base_from: exp.Expression, - ) -> tuple[exp.Expression, dict[str, str], dict[str, str], dict[str, str]]: - """Build a ranked subquery for first/last aggregation. - - Wraps the source table in a subquery that adds ROW_NUMBER columns - for each distinct time column used by first/last measures. - Returns (subquery, rn_suffix_map, filtered_rn_map, filtered_match_map): - rn_suffix_map maps each effective time column to its ROW_NUMBER alias - suffix; filtered_rn_map and filtered_match_map both key by - EnrichedMeasure.alias and map to the dedicated ROW_NUMBER column and - boolean match-flag column for filtered first/last measures. The match - flag is needed by the outer aggregate so it doesn't have to re-emit - measure.filter_sql (which can reference joined-table columns that - aren't in scope outside this subquery). - """ - model = enriched.model_name - default_time_col = enriched.last_agg_time_column - - # Build SELECT * plus ROW_NUMBER - parts = [f"{model}.*"] - - # Add pre-computed time dimension expressions (DATE_TRUNC) - for td in enriched.time_dimensions: - col_expr = self._resolve_sql(sql=td.sql, name=td.name, model_name=td.model_name) - td_expr = self._build_date_trunc(col_expr=col_expr, granularity=td.granularity) - parts.append(f"{td_expr.sql(dialect=self.dialect)} AS _td_{td.name}") - - # Build PARTITION BY from query dimensions + time dimensions - # Must use full expressions (not aliases) since aliases aren't visible in OVER() - partition_parts = [] - for dim in enriched.dimensions: - col_expr = self._resolve_sql(sql=dim.sql, name=dim.name, model_name=dim.model_name, type=dim.type) - partition_parts.append(col_expr.sql(dialect=self.dialect)) - for td in enriched.time_dimensions: - col_expr = self._resolve_sql(sql=td.sql, name=td.name, model_name=td.model_name) - td_expr = self._build_date_trunc(col_expr=col_expr, granularity=td.granularity) - partition_parts.append(td_expr.sql(dialect=self.dialect)) - - partition_clause = f"PARTITION BY {', '.join(partition_parts)}" if partition_parts else "" - - # Collect distinct effective time columns from UNFILTERED first/last - # measures only — filtered ones get their own dedicated ROW_NUMBER - # columns later (so we'd otherwise emit a redundant _last_rn that - # nothing references). - # default_time_col is guaranteed non-None here (checked at call site) - assert default_time_col is not None - time_col_agg_types: dict[str, set[str]] = {} - for m in enriched.measures: - if m.aggregation in ("first", "last") and not m.filter_sql: - effective = m.time_column or default_time_col - if effective not in time_col_agg_types: - time_col_agg_types[effective] = set() - time_col_agg_types[effective].add(m.aggregation) - - # Assign stable suffixes: first sorted gets "", second gets "_2", etc. - sorted_time_cols = sorted(time_col_agg_types.keys()) - rn_suffix_map: dict[str, str] = {} - for i, tc in enumerate(sorted_time_cols): - rn_suffix_map[tc] = "" if i == 0 else f"_{i + 1}" - - # Generate ROW_NUMBER columns per distinct time column - for tc in sorted_time_cols: - tc_expr = self._resolve_sql(sql=tc, name=tc, model_name=model) - order_sql = tc_expr.sql(dialect=self.dialect) - suffix = rn_suffix_map[tc] - agg_types = time_col_agg_types[tc] - if "last" in agg_types: - parts.append(f"ROW_NUMBER() OVER ({partition_clause} ORDER BY {order_sql} DESC) AS _last_rn{suffix}") - if "first" in agg_types: - parts.append(f"ROW_NUMBER() OVER ({partition_clause} ORDER BY {order_sql} ASC) AS _first_rn{suffix}") - - # Generate dedicated ROW_NUMBER columns for filtered first/last measures. - # These push non-matching rows to the bottom of the ranking so that - # rn=1 picks the first matching row, not the globally first row. - # Also project a per-filter boolean *match flag* so the outer aggregate - # doesn't have to re-emit `measure.filter_sql` (which can reference - # joined-table columns that aren't visible outside the ranked subquery). - filtered_rn_map: dict[str, str] = {} - filtered_match_map: dict[str, str] = {} - filter_idx = 0 - # cache_key -> (rn_alias, match_alias) - seen_filters: dict[tuple[str, str, str], tuple[str, str]] = {} - for m in enriched.measures: - if m.aggregation in ("first", "last") and m.filter_sql: - effective_tc = m.time_column or default_time_col - tc_expr = self._resolve_sql(sql=effective_tc, name=effective_tc, model_name=model) - order_sql = tc_expr.sql(dialect=self.dialect) - cache_key = (m.filter_sql, effective_tc, m.aggregation) - if cache_key in seen_filters: - # Reuse existing columns for identical filter+time_col+agg - rn_alias, match_alias = seen_filters[cache_key] - else: - rn_alias = f"_{'first' if m.aggregation == 'first' else 'last'}_rn_f{filter_idx}" - match_alias = f"_match_f{filter_idx}" - order_dir = "ASC" if m.aggregation == "first" else "DESC" - parts.append( - f"ROW_NUMBER() OVER ({partition_clause} ORDER BY " - f"CASE WHEN {m.filter_sql} THEN 0 ELSE 1 END, " - f"{order_sql} {order_dir}) AS {rn_alias}" - ) - parts.append( - f"CASE WHEN {m.filter_sql} THEN 1 ELSE 0 END AS {match_alias}" - ) - seen_filters[cache_key] = (rn_alias, match_alias) - filter_idx += 1 - # Key by alias (unique per enriched measure) so two filtered - # measures that share source/agg but differ in filter or time - # column don't clobber each other. - filtered_rn_map[m.alias] = rn_alias - filtered_match_map[m.alias] = match_alias - - select_sql = ", ".join(parts) - from_sql = base_from.sql(dialect=self.dialect) - ranked_sql = f"SELECT {select_sql} FROM {from_sql}" - - # Apply LEFT JOINs from resolved_joins INSIDE the subquery so that - # filter expressions (and ORDER BY columns) referencing joined - # tables resolve. The outer query's join injection only matches - # `FROM
AS ` and would miss this subquery wrapper. - if enriched.resolved_joins: - # DEV-1686: the ``AS `` is not dot-adjacent, so the - # ``_parse`` prequote (which fixes ``join_cond`` qualifiers and the - # ``{model}.*`` projection when this string is re-parsed below) - # cannot reach it — quote a reserved alias explicitly. - join_sql_parts = [ - f"{jtype.upper()} JOIN {target_table} " - f"AS {self._maybe_quote_qualifier(target_alias)} ON {join_cond}" - for target_table, target_alias, join_cond, jtype in enriched.resolved_joins - ] - ranked_sql += " " + " ".join(join_sql_parts) - # Apply WHERE filters to the subquery (they filter raw data before ranking) - where_clause, _ = self._build_where_and_having(enriched=enriched) - if where_clause is not None: - ranked_sql += f" WHERE {where_clause.sql(dialect=self.dialect)}" - - parsed = self._parse(ranked_sql) - return ( - exp.Subquery(this=parsed, alias=exp.to_identifier(model)), - rn_suffix_map, - filtered_rn_map, - filtered_match_map, - ) # ------------------------------------------------------------------ # Column / measure resolution (from enriched SQL expressions) @@ -2252,10 +1084,10 @@ def _rewrite_log_aliases(self, node: exp.Expression) -> exp.Expression: def _resolve_sql( self, - sql: str | None, + sql: Optional[str], name: str, model_name: str, - type: DataType | None = None, + type: Optional[DataType] = None, ) -> exp.Expression: """Resolve an enriched SQL expression to a sqlglot AST node. @@ -2267,6 +1099,9 @@ def _resolve_sql( ``type``. """ if sql is None: + # DEV-1645: quote the mixed-case column leaf; the model qualifier is + # a SLayer-internal alias and stays unquoted (reserved names quote + # at emit). return exp.Column(this=self._to_ident(name), table=exp.to_identifier(model_name)) # Bare column name → qualify with model name # Use isidentifier() to distinguish column names from literals (e.g. "1") @@ -2274,111 +1109,156 @@ def _resolve_sql( return exp.Column(this=self._to_ident(sql), table=exp.to_identifier(model_name)) return _wrap_cast_for_type(self._parse(sql), type) - def _resolve_value_sql(self, measure: "EnrichedMeasure") -> str: - """Resolve ``measure.sql`` (or ``measure.name``) into a fully-qualified + def _resolve_value_sql(self, spec: AggRenderSpec) -> str: + """Resolve ``spec.sql`` (or ``spec.name``) into a fully-qualified SQL string for the value column. Mirrors what ``_build_agg`` does for the standard sum/avg/min/max path so the dialect-aware builders (median/percentile/stat-aggs/formula) emit the same qualified identifiers. """ return self._resolve_sql( - sql=measure.sql, - name=measure.name, - model_name=measure.model_name, - type=measure.column_type, + sql=spec.sql, + name=spec.name, + model_name=spec.model_name, + type=spec.column_type, ).sql(dialect=self.dialect) + def _agg_param_ast( + self, value: "ResolvedAggKwarg | str", *, model_name: str, + ) -> exp.Expression: + """Resolve a parametric-agg param value to a sqlglot AST. + + DEV-1706 (D-I): a ``ResolvedAggKwarg`` with ``kind="expr"`` is a trusted, + scope-resolved expression embedded directly; ``kind="str"`` (and a plain + model-level default ``str``) resolve through ``_resolve_sql`` so bare + identifiers qualify under ``model_name`` — the pre-DEV-1706 behaviour. + ``_SAFE_AGG_PARAM_RE`` guarding of ``kind="str"`` query values is applied + by the callers before this point. + """ + if isinstance(value, ResolvedAggKwarg): + if value.kind == "expr": + # Return a COPY: the same ResolvedAggKwarg (keyed by AggregateKey) + # is embedded into more than one AST when a C13 slot with two + # declared aliases visits the same key twice in base_render_order. + # sqlglot re-parents a node on attach, so sharing the node would + # corrupt the first tree — mirror ScopeFrame.resolve's .copy() + # discipline (slayer/sql/scope.py). + return value.value.copy() if isinstance(value.value, exp.Expression) \ + else self._parse(value.value) + raw = value.value + else: + raw = value + return self._resolve_sql(sql=raw, name=raw, model_name=model_name) + def _resolve_agg_param( self, - measure: "EnrichedMeasure", + spec: AggRenderSpec, *, name: str, agg_name: str, ) -> str: """Pull a named aggregation parameter, with query-time SQL-injection validation and model-level-default fallback. Returns the SQL string - with bare identifiers qualified under ``measure.model_name`` (via + with bare identifiers qualified under ``spec.model_name`` (via ``_resolve_sql``); qualified names and numeric literals pass through unchanged. Raises ``ValueError`` if neither source supplies the parameter — reused by ``_build_percentile`` (``p=``) and ``_build_stat_agg`` (``other=``); mirrors ``weighted_avg``'s ``weight=`` flow. """ - raw: str | None = None - if name in measure.agg_kwargs: - raw = measure.agg_kwargs[name] - _validate_agg_param_value(raw, name, agg_name) - elif measure.aggregation_def: - for param in measure.aggregation_def.params: + value: "ResolvedAggKwarg | str | None" = None + if name in spec.agg_kwargs: + value = spec.agg_kwargs[name] + # Guard the untrusted string forms: a ``kind="str"`` wrapper OR a + # bare ``str`` (the legacy ``EnrichedMeasure`` adapter and model-level + # defaults reach here unwrapped). ``kind="expr"`` is a trusted, + # bind-time-resolved expression and is embedded verbatim. + if isinstance(value, ResolvedAggKwarg): + if value.kind == "str": + _validate_agg_param_value(value.value, name, agg_name) + elif isinstance(value, str): + _validate_agg_param_value(value, name, agg_name) + elif spec.aggregation_def: + for param in spec.aggregation_def.params: if param.name == name: - raw = param.sql + value = param.sql break - if raw is None: + if value is None: raise ValueError( f"Aggregation '{agg_name}' requires parameter '{name}'. " f"Set it in the model's aggregation definition or at query time " f"(e.g., 'measure:{agg_name}({name}=column)')." ) - return self._resolve_sql( - sql=raw, name=raw, model_name=measure.model_name, + return self._agg_param_ast( + value, model_name=spec.model_name, ).sql(dialect=self.dialect) def _build_agg( self, - measure: EnrichedMeasure, - rn_suffix_map: dict[str, str] | None = None, - default_time_col: str | None = None, - filtered_rn_map: dict[str, str] | None = None, - filtered_match_map: dict[str, str] | None = None, + spec: "AggRenderSpec | None" = None, + rn_suffix_map: Optional[dict[str, str]] = None, + default_time_col: Optional[str] = None, + filtered_rn_map: Optional[dict[str, str]] = None, + filtered_match_map: Optional[dict[str, str]] = None, ) -> tuple[exp.Expression, bool]: - """Build an aggregation expression from an enriched measure.""" - agg_name = measure.aggregation + """Build an aggregation expression from an ``AggRenderSpec``.""" + if spec is None: # pragma: no cover — defensive + raise ValueError("_build_agg requires a 'spec'.") + agg_name = spec.aggregation if not agg_name: # Not an aggregation — raw expression - if measure.sql: + if spec.sql: return self._resolve_sql( - sql=measure.sql, - name=measure.name, - model_name=measure.model_name, - type=measure.column_type, + sql=spec.sql, + name=spec.name, + model_name=spec.model_name, + type=spec.column_type, ), False return exp.Column( - this=self._to_ident(measure.name), - table=exp.to_identifier(measure.model_name), + this=exp.to_identifier(spec.name), + table=exp.to_identifier(spec.model_name), ), False # --- first/last: MAX(CASE WHEN _rn = 1 THEN col END) --- if agg_name in ("first", "last"): col_expr = self._resolve_sql( - sql=measure.sql, - name=measure.name, - model_name=measure.model_name, - type=measure.column_type, + sql=spec.sql, + name=spec.name, + model_name=spec.model_name, + type=spec.column_type, ) col = col_expr.sql(dialect=self.dialect) suffix = "" - if rn_suffix_map and default_time_col: - effective_tc = measure.time_column or default_time_col - suffix = rn_suffix_map.get(effective_tc, "") + if rn_suffix_map is not None: + # DEV-1501: when no default ranking time column is in scope, + # every first/last spec is guaranteed to carry an explicit + # ``time_column`` (validated in + # ``_build_first_last_base_select``); so the suffix lookup + # must not gate on ``default_time_col`` being truthy, else + # distinct-time-column specs all collapse to ``_last_rn``. + effective_tc = spec.time_column or default_time_col + if effective_tc is not None: + suffix = rn_suffix_map.get(effective_tc, "") rn_col = f"_first_rn{suffix}" if agg_name == "first" else f"_last_rn{suffix}" # For filtered first/last, use the dedicated ROW_NUMBER column # that pushes non-matching rows to the bottom of the ranking. - # Look up by alias (unique per enriched measure) so two filtered - # measures sharing source/agg but with different filters map to - # their own respective rank columns. Use the per-measure match - # flag (also projected by the ranked subquery) instead of - # re-emitting measure.filter_sql here — the filter can reference - # joined-table columns that are not in scope outside the subquery. - if measure.filter_sql and filtered_rn_map: - filtered_rn = filtered_rn_map.get(measure.alias, rn_col) + # Look up by alias (unique per spec) so two filtered specs + # sharing source/agg but with different filters map to their + # own respective rank columns. Use the per-spec match flag + # (also projected by the ranked subquery) instead of + # re-emitting spec.filter_sql here — the filter can reference + # joined-table columns that are not in scope outside the + # subquery. + if spec.filter_sql and filtered_rn_map: + filtered_rn = filtered_rn_map.get(spec.alias, rn_col) match_col = ( - filtered_match_map.get(measure.alias) + filtered_match_map.get(spec.alias) if filtered_match_map else None ) # Fall back to the raw filter expression only if no match flag # was projected (legacy callers); accepts the leak risk. - filter_clause = f"{match_col} = 1" if match_col else measure.filter_sql + filter_clause = f"{match_col} = 1" if match_col else spec.filter_sql case_sql = ( f"MAX(CASE WHEN {filtered_rn} = 1 AND {filter_clause} " f"THEN {col} END)" @@ -2386,7 +1266,7 @@ def _build_agg( else: # ``col`` is already a fully-qualified SQL expression resolved # via ``_resolve_sql`` earlier in this branch, so we don't need - # to re-prefix ``measure.model_name``. (DEV-1333.) + # to re-prefix ``spec.model_name``. (DEV-1333.) case_sql = f"MAX(CASE WHEN {rn_col} = 1 THEN {col} END)" return self._parse(case_sql), True @@ -2396,44 +1276,51 @@ def _build_agg( # SQLite/ClickHouse/MySQL) so it gets its own builder rather than # going through the BUILTIN_AGGREGATION_FORMULAS path. if agg_name == "percentile": - return self._build_percentile(measure), True - # DEV-1595: approximate-distinct is dialect-aware (native function - # where the backend has one, exact COUNT(DISTINCT) fallback where - # it does not), so it routes to its own dialect-dispatching builder. - if agg_name == "count_distinct_approx": - return self._build_approx_count_distinct(measure), True + return self._build_percentile(spec), True # Statistical aggregates also dispatch to a dedicated builder so # the SQLite-UDF / native-function / NotImplementedError split # mirrors _build_median. if agg_name in _STAT_AGG_NAMES: - return self._build_stat_agg(measure), True - return self._build_formula_agg(measure, agg_name), True + return self._build_stat_agg(spec), True + # count_distinct_approx (DEV-1595): dialect-aware approximate- + # distinct — native function (DuckDB/ClickHouse/BigQuery/…) or the + # exact COUNT(DISTINCT) fallback (Postgres/SQLite/MySQL). Built like + # percentile/stat-agg (via _wrap_filter + _resolve_value_sql) so a + # row-level filter wraps as COUNT(DISTINCT (CASE WHEN ... END)). + if agg_name == "count_distinct_approx": + col_expr = _wrap_filter( + self._resolve_value_sql(spec), spec.filter_sql + ) + return self._dialect.build_approx_count_distinct( + col_sql=col_expr, parse=self._parse + ), True + return self._build_formula_agg(spec, agg_name), True # --- Resolve inner expression --- - if agg_name == "count" and measure.sql is None: + if agg_name == "count" and spec.sql is None: # COUNT(*) — if filtered, use COUNT(CASE WHEN filter THEN 1 END) - if measure.filter_sql: - case_sql = f"CASE WHEN {measure.filter_sql} THEN 1 END" + if spec.filter_sql: + case_sql = f"CASE WHEN {spec.filter_sql} THEN 1 END" inner = self._parse(case_sql) else: inner = exp.Star() - elif measure.sql: + elif spec.sql: inner = self._resolve_sql( - sql=measure.sql, - name=measure.name, - model_name=measure.model_name, - type=measure.column_type, + sql=spec.sql, + name=spec.name, + model_name=spec.model_name, + type=spec.column_type, ) else: inner = exp.Column( - this=self._to_ident(measure.name), - table=exp.to_identifier(measure.model_name), + this=exp.to_identifier(spec.name), + table=exp.to_identifier(spec.model_name), ) - # --- Apply measure-level filter as CASE WHEN wrapper --- - if measure.filter_sql and not (agg_name == "count" and measure.sql is None): + # --- Apply spec-level filter as CASE WHEN wrapper --- + if spec.filter_sql and not (agg_name == "count" and spec.sql is None): inner_sql = inner.sql(dialect=self.dialect) - case_sql = f"CASE WHEN {measure.filter_sql} THEN {inner_sql} END" + case_sql = f"CASE WHEN {spec.filter_sql} THEN {inner_sql} END" inner = self._parse(case_sql) # --- count_distinct --- @@ -2456,12 +1343,12 @@ def _build_agg( agg_class = agg_class_map[agg_func] return agg_class(this=inner), True - def _build_formula_agg(self, measure: EnrichedMeasure, agg_name: str) -> exp.Expression: + def _build_formula_agg(self, spec: AggRenderSpec, agg_name: str) -> exp.Expression: # NOSONAR(S3776) — sequential dispatch over formula source (aggregation_def vs built-in) and per-kind ResolvedAggKwarg substitution (DEV-1527); one cohesive template-substitution contract. """Build SQL for formula-based aggregations (weighted_avg, custom).""" # Get formula: from aggregation_def or built-in formula = None - if measure.aggregation_def and measure.aggregation_def.formula: - formula = measure.aggregation_def.formula + if spec.aggregation_def and spec.aggregation_def.formula: + formula = spec.aggregation_def.formula elif agg_name in BUILTIN_AGGREGATION_FORMULAS: formula = BUILTIN_AGGREGATION_FORMULAS[agg_name] @@ -2473,13 +1360,16 @@ def _build_formula_agg(self, measure: EnrichedMeasure, agg_name: str) -> exp.Exp # Collect param values: query-time overrides > aggregation_def defaults param_defaults = {} - if measure.aggregation_def: - param_defaults = {p.name: p.sql for p in measure.aggregation_def.params} - params = {**param_defaults, **measure.agg_kwargs} + if spec.aggregation_def: + param_defaults = {p.name: p.sql for p in spec.aggregation_def.params} + params = {**param_defaults, **spec.agg_kwargs} - # Validate query-time parameter values to prevent SQL injection - for pname, pval in measure.agg_kwargs.items(): - _validate_agg_param_value(pval, pname, agg_name) + # Validate query-time parameter values to prevent SQL injection. Only the + # untrusted ``kind="str"`` form is guarded; ``kind="expr"`` is a trusted, + # bind-time-resolved expression (DEV-1706 D-I). + for pname, pval in spec.agg_kwargs.items(): + if isinstance(pval, ResolvedAggKwarg) and pval.kind == "str": + _validate_agg_param_value(pval.value, pname, agg_name) # Validate required params required = BUILTIN_AGGREGATION_REQUIRED_PARAMS.get(agg_name, []) @@ -2492,58 +1382,46 @@ def _build_formula_agg(self, measure: EnrichedMeasure, agg_name: str) -> exp.Exp ) # Resolve {value} and {param_name} via _resolve_sql so bare identifiers - # are qualified under measure.model_name (matching the standard - # sum/avg/min/max path). When the measure carries a row-level filter, + # are qualified under spec.model_name (matching the standard + # sum/avg/min/max path). When the spec carries a row-level filter, # wrap row-level references (the value AND any column-ref params) in # CASE WHEN so non-matching rows contribute NULL to all terms — but # leave literal-default params unwrapped, since `(CASE WHEN ... THEN # 100 END)` for a constant `scale=100` would turn it into a row # expression and break grouped SQL semantics. - col_expr = _wrap_filter(self._resolve_value_sql(measure), measure.filter_sql) + col_expr = _wrap_filter(self._resolve_value_sql(spec), spec.filter_sql) substituted = formula.replace("{value}", col_expr) for param_name, param_val in params.items(): - param_ast = self._resolve_sql( - sql=param_val, name=param_val, model_name=measure.model_name, + param_ast = self._agg_param_ast( + param_val, model_name=spec.model_name, ) param_expr = param_ast.sql(dialect=self.dialect) - if measure.filter_sql and not isinstance(param_ast, exp.Literal): - param_expr = _wrap_filter(param_expr, measure.filter_sql) + if spec.filter_sql and not isinstance(param_ast, exp.Literal): + param_expr = _wrap_filter(param_expr, spec.filter_sql) substituted = substituted.replace(f"{{{param_name}}}", param_expr) return self._parse(substituted) def _build_median(self, inner: exp.Expression) -> exp.Expression: - """Build a median aggregation expression. Dispatches to the dialect.""" + """Build a median aggregation expression. Dispatches to the dialect + (DEV-1716) — MySQL/T-SQL raise NotImplementedError, SQLite/ClickHouse + emit ``median()``, others ``PERCENTILE_CONT(0.5)``.""" return self._dialect.build_median(inner=inner, parse=self._parse) - def _build_approx_count_distinct(self, measure: "EnrichedMeasure") -> exp.Expression: - """Build a dialect-aware approximate-distinct aggregation (DEV-1595). - - Resolves the value column (qualified under ``measure.model_name``) and, - when the measure carries a row-level filter, wraps it in - ``CASE WHEN filter THEN col END`` — composing with the metric-filter - push-down (Part 3.4) exactly as ``count_distinct`` does. Dispatches to - the dialect's ``build_approx_count_distinct``: the native function - (DuckDB / ClickHouse / BigQuery / …) or the exact ``COUNT(DISTINCT)`` - fallback (Postgres / SQLite / MySQL). - """ - col_expr = _wrap_filter(self._resolve_value_sql(measure), measure.filter_sql) - return self._dialect.build_approx_count_distinct(col_expr, parse=self._parse) - - def _build_percentile(self, measure: "EnrichedMeasure") -> exp.Expression: + def _build_percentile(self, spec: AggRenderSpec) -> exp.Expression: """Build a PERCENTILE_CONT(p) aggregation expression (dialect-dependent). - ``p`` comes from ``measure.agg_kwargs['p']`` (validated against + ``p`` comes from ``spec.agg_kwargs['p']`` (validated against SQL injection) or from a model-level ``Aggregation`` default. - Filter handling mirrors ``_build_formula_agg``: when the measure + Filter handling mirrors ``_build_formula_agg``: when the spec carries a row-level filter, the value column is wrapped in ``CASE WHEN ... END`` so non-matching rows contribute NULL and are ignored by the aggregate. Both the value column and ``p`` flow through ``_resolve_sql`` so bare identifiers are qualified - under ``measure.model_name`` and numeric literals pass through + under ``spec.model_name`` and numeric literals pass through unchanged. """ - p = self._resolve_agg_param(measure, name="p", agg_name="percentile") + p = self._resolve_agg_param(spec, name="p", agg_name="percentile") # `p` must be a numeric literal in [0, 1]. Without this guard a # caller could pass `measure:percentile(p=quantity)` (or a model- # level default like `p=pg_sleep(10)` that bypasses @@ -2564,20 +1442,16 @@ def _build_percentile(self, measure: "EnrichedMeasure") -> exp.Expression: f"Aggregation 'percentile' parameter 'p' must be in [0, 1]; got {p_float}." ) - # Pass the **original string** ``p`` (not ``p_float``) to the dialect - # so user literals like ``0.50`` / ``1`` / ``5e-2`` survive verbatim - # in the emitted SQL. Range validation above keeps the safety guard. - col_expr = _wrap_filter(self._resolve_value_sql(measure), measure.filter_sql) + # Pass the **original string** ``p`` (not ``p_float``) to the dialect so + # user literals like ``0.50`` / ``1`` / ``5e-2`` survive verbatim. + # DEV-1716: dialect owns the wire form (MySQL/T-SQL raise, SQLite UDF, + # ClickHouse parametric ``quantile(p)(x)``, others ``PERCENTILE_CONT``). + col_expr = _wrap_filter(self._resolve_value_sql(spec), spec.filter_sql) return self._dialect.build_percentile( p_str=p, col_sql=col_expr, parse=self._parse, ) - # ``_build_covar_formula`` lived here in DEV-1317 — moved to - # ``slayer/sql/dialects/base._build_covar_decomposition`` in DEV-1542 and - # is now called by MysqlDialect / TsqlDialect overrides of - # ``build_covar_2arg``. - - def _build_stat_agg(self, measure: "EnrichedMeasure") -> exp.Expression: + def _build_stat_agg(self, spec: AggRenderSpec) -> exp.Expression: """Build SQL for the statistical aggregations added in DEV-1317. Handles ``stddev_samp``, ``stddev_pop``, ``var_samp``, ``var_pop`` @@ -2587,33 +1461,39 @@ def _build_stat_agg(self, measure: "EnrichedMeasure") -> exp.Expression: ``corr`` / ``covar_*`` are not. SQLite gets them via Python UDFs registered in ``slayer.sql.sqlite_udfs`` — the UDFs alias sqlglot's transpiled names (e.g. ``var_samp`` → ``VARIANCE`` on - SQLite) so generator output resolves at runtime. MySQL and T-SQL - implement ``corr`` / ``covar_*`` via the variance-decomposition - formula in ``_build_covar_formula``. + SQLite) so generator output resolves at runtime. Both legs flow through ``_resolve_sql`` so bare identifiers are - qualified under ``measure.model_name`` (matches the standard + qualified under ``spec.model_name`` (matches the standard sum/avg/min/max path). Filter handling mirrors ``_build_percentile`` / ``_build_formula_agg``: a row-level filter wraps the value AND the ``other`` column in ``CASE WHEN filter THEN col END`` so non-matching rows contribute NULL — which the aggregates skip. """ - agg_name = measure.aggregation - - # Resolve the `other=` kwarg before the dialect guard so that a - # missing-required-param error takes priority over any dialect-specific - # error when both conditions hold — the missing-param message points at - # the actual user mistake. Closes Codex #5 on PR #82. - other_expr: str | None = None + agg_name = spec.aggregation + + # Resolve the `other=` kwarg before the MySQL guard so that a + # missing-required-param error takes priority over the + # MySQL-not-supported error when both conditions hold — the + # missing-param message points at the actual user mistake. Closes + # Codex #5 on PR #82. + # Resolve the `other=` kwarg BEFORE any dialect guard so a + # missing-required-param error takes priority over a dialect-specific + # error (the missing-param message points at the actual user mistake). + other_expr: Optional[str] = None if agg_name in _TWO_ARG_STAT_AGGS: other_expr = _wrap_filter( - self._resolve_agg_param(measure, name="other", agg_name=agg_name), - measure.filter_sql, + self._resolve_agg_param(spec, name="other", agg_name=agg_name), + spec.filter_sql, ) - col_expr = _wrap_filter(self._resolve_value_sql(measure), measure.filter_sql) + col_expr = _wrap_filter(self._resolve_value_sql(spec), spec.filter_sql) + # DEV-1716: the dialect owns the wire form — native CORR/COVAR on + # Postgres/DuckDB/ClickHouse, variance-decomposition formula on + # MySQL/T-SQL; canonical stddev/var name (sqlglot-transpiled) with the + # MySQL ``exp.Anonymous`` var_samp/var_pop bypass in the dialect class. if agg_name in _TWO_ARG_STAT_AGGS: assert other_expr is not None # set above when two-arg return self._dialect.build_covar_2arg( @@ -2630,111 +1510,8773 @@ def _build_stat_agg(self, measure: "EnrichedMeasure") -> exp.Expression: # WHERE / HAVING (filters still use ColumnRef for member resolution) # ------------------------------------------------------------------ - def _build_where_and_having( + + # ====================================================================== + # DEV-1450 stage 7b.8 — PlannedQuery → SQL. + # + # The legacy generator (everything above) consumes EnrichedQuery. This + # new entry point consumes the typed PlannedQuery from + # slayer/engine/stage_planner.py. The two paths coexist until the + # engine cutover (stage 7b.15) flips the default path. + # + # 7b.8 scope: local-only single-model queries — row-phase dims, local + # aggregates, Mode-B row filters, ORDER BY / LIMIT / OFFSET, dim-only + # dedup. Cross-model, time dimensions, transforms, and aggregate + # filtering raise NotImplementedError with an explicit stage marker + # so silent parity drift is impossible. + # ====================================================================== + + def generate_from_planned(self, planned_query, *, bundle) -> str: + """Render a typed ``PlannedQuery`` to SQL (public entry). + + DEV-1708 (D-E): installs a fresh generation-wide ``AliasAllocator`` for + the duration of this call and restores the caller's on exit. Inline + forward ``_cm_*`` CTEs and the host base share this one allocator, so + their ``_val_`` materialisation names never collide; a recursive + rerooted sub-generation (``_render_rerooted_cross_model_cte`` → + ``generate_from_planned``) is a self-contained statement and gets its + own allocator, with the parent's restored afterwards. + """ + prev_allocator = getattr(self, "_gen_allocator", None) + self._gen_allocator = self._new_allocator() + try: + return self._generate_from_planned_impl( + planned_query, bundle=bundle, + ) + finally: + self._gen_allocator = prev_allocator + + def _generate_from_planned_impl( # NOSONAR(S3776) — top-level dispatch over cross-model / transform-chain / plain branches plus the conditional outer-trim wrap. Each branch is a coherent compilation strategy; extracting would scatter the shared planned_query / slots_by_id / aliases_by_slot_id state across helpers without simplifying anything. self, - enriched: EnrichedQuery, - rn_suffix_map: dict[str, str] | None = None, - filtered_rn_map: dict[str, str] | None = None, - ) -> tuple[exp.Expression | None, exp.Expression | None]: - """Build WHERE and HAVING clauses from parsed filters. + planned_query, + *, + bundle, + ) -> str: + """Render a typed ``PlannedQuery`` to SQL. + + NOTE (DEV-1716): this is a STAGE renderer — its output feeds + ``generate_planned_stages``' flat-column stage-schema wrapper, so the + dialect ``rewrite_emitted_sql`` alias-mangling post-pass is applied by + the DB-bound terminal (``generate_planned_stages``), NOT here. Mangling + a stage's column names would break the downstream flat-name binding. + + Mirrors the local-only branch of ``_generate_base`` but reads + from typed PlannedQuery fields (``row_slots`` / ``aggregate_slots`` + / ``filters_by_phase`` / ``order`` / ``transform_layers``) + instead of ``EnrichedQuery``. Reuses legacy dialect helpers + (``_resolve_sql`` / ``_build_agg`` / ``_wrap_cast_for_type`` / + ``_parse_predicate`` / ``_build_date_trunc``) so dialect-specific + behavior is rendered identically to the legacy ``generate()`` + path — the parity oracle in ``tests/parity_oracle.py`` pins + this contract. + + Stage 7b.10 adds window-transform rendering: when + ``planned_query.transform_layers`` is non-empty, the base SELECT + is emitted as ``WITH base AS (...)``, Kahn-batched step CTEs + carry the window functions, and an outer wrap projects in + user-spec order. POST-phase filters that reference transform + slots wrap as ``SELECT * FROM (...) AS _filtered WHERE ...``. + ``time_shift`` / ``consecutive_periods`` layers raise + ``NotImplementedError`` with a ``7b.11`` marker. + """ + + source_model = bundle.source_model + if source_model is None: + raise ValueError( + "generate_from_planned requires bundle.source_model to be set", + ) + source_relation = planned_query.source_relation + + if ( + planned_query.cross_model_aggregate_plans + or planned_query.windowed_aggregate_plans + ): + return self._render_with_cross_model_plans( + planned_query=planned_query, bundle=bundle, + ) + + # 7b.10 — fail fast on transform ops this slice does not render + # (time_shift / consecutive_periods belong to 7b.11). Walks + # ``transform_layers`` for an explicit op match AND walks every + # ``TransformKey.input`` reachable from public slots so a + # ``change`` desugared into ``time_shift`` raises with the same + # marker. + self._validate_window_transform_ops_for_7b10( + planned_query=planned_query, + ) + + slots_by_id = { + s.id: s + for s in ( + list(planned_query.row_slots) + + list(planned_query.aggregate_slots) + + list(planned_query.combined_expression_slots) + ) + } + + # 7b.10 — slot key -> id lookup. ``PlannedQuery`` does not carry + # the ``ValueRegistry``, so the generator builds its own map. + # Used for resolving ``TransformKey.input`` / ``partition_keys`` / + # ``time_key`` references to step-CTE aliases. + slot_id_by_key: Dict[Any, str] = { + s.key: s.id for s in slots_by_id.values() + } + + public_proj_set: Set[str] = set(planned_query.projection) + # 7b.10 / DEV-1501 — base CTE projects hidden slots referenced as + # transform inputs / partition_keys / time_key / filter operands + # (AGGREGATE + POST phase) / order targets so step CTEs, HAVING, + # and the outer ORDER BY can name them. In the NO-transform path + # we additionally pass ``aggregates_only=True`` so only + # AggregateKey leaves get pulled in from order/filter walks — a + # hidden ROW order target (e.g. ``ORDER BY customer_id`` with + # ``customer_id`` not projected) would otherwise materialise into + # GROUP BY and silently change query grain. Hidden ROW order + # targets in the no-transform path keep raising NotImplementedError + # at the inline ORDER BY render path. + no_transform = not bool(planned_query.transform_layers) + extra_materialize_ids = self._collect_base_aux_slot_ids( + planned_query=planned_query, + slot_id_by_key=slot_id_by_key, + slots_by_id=slots_by_id, + include_order=True, + aggregates_only=no_transform, + ) + base_render_order = list(planned_query.projection) + [ + sid for sid in extra_materialize_ids if sid not in public_proj_set + ] + + # Build the base SELECT body. ``aliases_by_slot_id`` is a list + # of full aliases per slot, in projection visit order — needed + # so duplicate public_aliases on a single interned slot (DEV-1450 + # C13: two declared measures with the same key + different names) + # survive the CTE chain. ``available_alias_by_slot_id`` is the + # canonical "pick one" map used by transform-input / time-key / + # partition-key / order-entry lookups (any alias of the slot + # refers to the same column value, so any will do). + ( + base_select, + aliases_by_slot_id, + has_aggregation, + group_by_keys, + where_consumed, + first_last_state, + ) = self._build_base_select_for_planned( + planned_query=planned_query, + bundle=bundle, + source_model=source_model, + source_relation=source_relation, + base_render_order=base_render_order, + slots_by_id=slots_by_id, + ) + + where_clause, having_clause = self._build_where_having_from_planned( + planned_query=planned_query, + source_relation=source_relation, + source_model=source_model, + bundle=bundle, + first_last_state=first_last_state, + aliases_by_slot_id=aliases_by_slot_id, + ) + + # ``where_consumed`` is True for the first/last ranked-subquery path: + # the WHERE is applied INSIDE the ranked subquery (it must filter raw + # rows before ranking), so re-applying it on the outer SELECT would be + # both redundant and — for filters that should narrow the ranked set — + # semantically wrong. + if where_clause is not None and not where_consumed: + base_select = base_select.where(where_clause) + + # Match legacy _generate_base:1375 — dim-only-dedup OR + # has_aggregation triggers GROUP BY (dim-only emits GROUP BY + # before LIMIT so unique dim tuples can't silently drop past + # row N). + # DEV-1543: distinct_dimension_values=False opts out of the dim-only + # dedup GROUP BY, emitting raw rows instead of distinct tuples. + dim_only_dedup = ( + planned_query.distinct_dimension_values + and bool(group_by_keys) + and not has_aggregation + ) + needs_group_by = has_aggregation or dim_only_dedup + if needs_group_by and group_by_keys: + for gb in group_by_keys.values(): + base_select = base_select.group_by(gb) - ParsedFilter objects have pre-built SQL strings. Column names are - qualified with the model name for the WHERE clause. + if having_clause is not None: + base_select = base_select.having(having_clause) + + # No transforms → existing pre-7b.10 path: apply ORDER/LIMIT + # directly on the base select. DEV-1501: when the base + # materialised hidden order/filter aggregate slots (slot ids in + # ``base_render_order`` not in ``planned_query.projection``), + # wrap the base in an outer SELECT that trims to the public + # projection and moves ORDER BY / LIMIT / OFFSET to the outer + # level — mirrors the transform path's outer wrap shape, minus + # the step CTE chain. + if not planned_query.transform_layers: + public_slot_ids = set(planned_query.projection) + has_hidden_materialised = any( + sid not in public_slot_ids for sid in base_render_order + ) + if has_hidden_materialised: + return self._build_outer_trim_wrap_sql( + base_select=base_select, + planned_query=planned_query, + source_relation=source_relation, + aliases_by_slot_id=aliases_by_slot_id, + slots_by_id=slots_by_id, + bundle=bundle, + ) + base_select = self._apply_order_limit_from_planned( + select=base_select, + planned_query=planned_query, + source_relation=source_relation, + slots_by_id=slots_by_id, + source_model=source_model, + bundle=bundle, + aliases_by_slot_id=aliases_by_slot_id, + ) + return base_select.sql(dialect=self.dialect, pretty=True) + + # 7b.10 — transform layers present. Build the CTE chain. + base_cte_sql = base_select.sql(dialect=self.dialect, pretty=True) + ctes: list[tuple[str, str]] = [("base", base_cte_sql)] + # DEV-1692: collision-safe CTE-name allocator for the whole transform + # chain. The hoisted time_shift slot alias (``_time_shift_inner``) + # repeats across arithmetic-wrapped shifts, so two ``shifted_`` / + # ``sjoin_`` pairs would otherwise share a name (duplicate WITH). Every + # CTE name is reserved/allocated through this one allocator so the + # ``step`` / ``shifted_`` / ``sjoin_`` / ``cp_`` families never collide. + cte_allocator = self._new_allocator() + cte_allocator.reserve(*(name for name, _ in ctes)) + # Codex (PR #269): also reserve every already-projected column alias's + # BARE form so a hidden transform alias minted below + # (``_time_shift_inner`` / ``_consecutive_periods_inner``) can never + # shadow a real user column of that name — mirrors the legacy path + # seeding ``base_aliases`` into its allocator. + _alias_prefix = f"{source_relation}." + cte_allocator.reserve(*( + a[len(_alias_prefix):] if a.startswith(_alias_prefix) else a + for aliases in aliases_by_slot_id.values() + for a in aliases + )) + # "Pick one" map for transform-input / time-key / partition-key / + # order-entry / POST-filter lookups. Initialised from the first + # alias of every materialised slot. + available_alias_by_slot_id: Dict[str, str] = { + sid: aliases[0] + for sid, aliases in aliases_by_slot_id.items() + if aliases + } + + pending_layers = list(planned_query.transform_layers) + step_num = 0 + # 7b.11 — gather a global view of WHERE-able row-phase filters + # for the shifted CTE (which re-aggregates the source and needs + # the same WHERE minus BetweenKey date_range filters). Built + # once outside the loop since the source filters don't change + # across layers. + shifted_where_parts, shifted_where_join_paths = ( + self._build_shifted_cte_where_parts( + planned_query=planned_query, + source_relation=source_relation, + source_model=source_model, + bundle=bundle, + ) + ) + while pending_layers: + ready_window: list = [] + ready_time_shift: list = [] + ready_cp: list = [] + not_ready: list = [] + for layer in pending_layers: + if not self._transform_layer_deps_ready( + layer=layer, + slots_by_id=slots_by_id, + slot_id_by_key=slot_id_by_key, + available_alias_by_slot_id=available_alias_by_slot_id, + ): + not_ready.append(layer) + elif layer.op == "time_shift": + ready_time_shift.append(layer) + elif layer.op == "consecutive_periods": + ready_cp.append(layer) + else: + ready_window.append(layer) + if not (ready_window or ready_time_shift or ready_cp): + pending_ops = [layer.op for layer in pending_layers] + raise RuntimeError( + f"DEV-1450 stage 7b.11: transform layer dependencies " + f"could not be resolved; pending ops: {pending_ops!r}.", + ) + # --- Window batch (one step CTE per Kahn batch) ---------- + if ready_window: + step_num += 1 + step_name = cte_allocator.allocate_cte(f"step{step_num}") + prev_cte = ctes[-1][0] + carry_aliases_sorted = sorted( + a for aliases in aliases_by_slot_id.values() for a in aliases + ) + step_parts = [self._quote_ident(a) for a in carry_aliases_sorted] + for layer in ready_window: + for slot_id in layer.slot_ids: + slot = slots_by_id[slot_id] + alias = ( + slot.public_aliases[0] + if slot.public_aliases + else slot.declared_name + ) + full_alias = f"{source_relation}.{alias}" + window_sql = self._render_window_transform_sql( + slot=slot, + slots_by_id=slots_by_id, + slot_id_by_key=slot_id_by_key, + available_alias_by_slot_id=available_alias_by_slot_id, + planned_query=planned_query, + ) + if slot.type is not None: + wrapped = _wrap_cast_for_type( + self._parse(window_sql), slot.type, + ) + window_sql = wrapped.sql(dialect=self.dialect) + step_parts.append(f'{window_sql} AS {self._quote_ident(full_alias)}') + aliases_by_slot_id.setdefault(slot_id, []).append( + full_alias, + ) + available_alias_by_slot_id.setdefault( + slot_id, full_alias, + ) + step_sql = ( + "SELECT\n " + + _SQL_COL_SEP.join(step_parts) + + f"\nFROM {prev_cte}" + ) + ctes.append((step_name, step_sql)) + # --- time_shift layers (each gets shifted_ + sjoin_ pair) - + for layer in ready_time_shift: + for slot_id in layer.slot_ids: + slot = slots_by_id[slot_id] + self._emit_time_shift_ctes_for_planned( + slot=slot, + ctes=ctes, + cte_allocator=cte_allocator, + slots_by_id=slots_by_id, + slot_id_by_key=slot_id_by_key, + available_alias_by_slot_id=available_alias_by_slot_id, + aliases_by_slot_id=aliases_by_slot_id, + source_model=source_model, + source_relation=source_relation, + shifted_where_parts=shifted_where_parts, + shifted_where_join_paths=shifted_where_join_paths, + planned_query=planned_query, + bundle=bundle, + ) + # --- consecutive_periods layers (cp_reset_ + cp_value_ pair) + for layer in ready_cp: + for slot_id in layer.slot_ids: + slot = slots_by_id[slot_id] + self._emit_consecutive_periods_ctes_for_planned( + slot=slot, + ctes=ctes, + cte_allocator=cte_allocator, + slots_by_id=slots_by_id, + slot_id_by_key=slot_id_by_key, + available_alias_by_slot_id=available_alias_by_slot_id, + aliases_by_slot_id=aliases_by_slot_id, + planned_query=planned_query, + source_relation=source_relation, + ) + pending_layers = not_ready + + # 7b.11 — materialise POST-phase ArithmeticKey / ScalarCallKey + # slots that the user projected but no transform layer rendered. + # ``change(amount:sum)`` lowers to ``amount:sum - time_shift(...)``; + # the time_shift slot is rendered as a self-join CTE pair, but + # the outer ArithmeticKey slot that subtracts them needs its + # own step CTE. Same shape covers ``change_pct`` (division of + # arithmetic operands) and any future POST-phase non-transform + # slot the planner emits. + from slayer.core.keys import ( + ArithmeticKey as _ArithKey, + ScalarCallKey as _ScalarKey, + TransformKey as _TKey, + ) + unmaterialised: list = [] + for cslot in planned_query.combined_expression_slots: + if isinstance(cslot.key, _TKey): + # Transform-key slots are materialised by transform_layers. + continue + if cslot.id in aliases_by_slot_id: + continue + if isinstance(cslot.key, (_ArithKey, _ScalarKey)): + unmaterialised.append(cslot) + if unmaterialised: + step_num += 1 + step_name = f"step{step_num}" + prev_cte = ctes[-1][0] + carry_aliases_sorted = sorted( + a for aliases in aliases_by_slot_id.values() for a in aliases + ) + step_parts = [self._quote_ident(a) for a in carry_aliases_sorted] + for cslot in unmaterialised: + alias = ( + cslot.public_aliases[0] + if cslot.public_aliases + else cslot.declared_name + ) + full_alias = f"{source_relation}.{alias}" + rendered = self._render_value_key_against_aliases( + key=cslot.key, + slot_id_by_key=slot_id_by_key, + available_alias_by_slot_id=available_alias_by_slot_id, + ) + expr_sql = rendered.sql(dialect=self.dialect) + if cslot.type is not None: + wrapped = _wrap_cast_for_type( + self._parse(expr_sql), cslot.type, + ) + expr_sql = wrapped.sql(dialect=self.dialect) + step_parts.append(f'{expr_sql} AS {self._quote_ident(full_alias)}') + aliases_by_slot_id.setdefault(cslot.id, []).append( + full_alias, + ) + available_alias_by_slot_id.setdefault( + cslot.id, full_alias, + ) + step_sql = ( + "SELECT\n " + + _SQL_COL_SEP.join(step_parts) + + f"\nFROM {prev_cte}" + ) + ctes.append((step_name, step_sql)) + + # Inner SELECT inside _outer wrap: ALL carried aliases sorted + # (matches legacy _generate_with_computed:1607). + final_cte = ctes[-1][0] + inner_sorted = sorted( + a for aliases in aliases_by_slot_id.values() for a in aliases + ) + inner_sql = ( + "SELECT\n " + + _SQL_COL_SEP.join(self._quote_ident(a) for a in inner_sorted) + + f"\nFROM {final_cte}" + ) + + cte_clause = ( + _SQL_WITH + + ",\n".join(f"{name} AS (\n{sql}\n)" for name, sql in ctes) + ) + chain_sql = f"{cte_clause}\n{inner_sql}" + + # POST-phase filter wrap (filters referencing transform / arith + # slots). Mirrors legacy _generate_with_computed:1627-1648 — + # ``SELECT * FROM () AS _filtered WHERE ``. + post_filter_conditions = self._render_post_phase_filter_conditions( + planned_query=planned_query, + slot_id_by_key=slot_id_by_key, + available_alias_by_slot_id=available_alias_by_slot_id, + ) + if post_filter_conditions: + chain_sql = ( + f"SELECT *\nFROM (\n{chain_sql}\n) AS _filtered" + f"\nWHERE {_SQL_AND_JOINER.join(post_filter_conditions)}" + ) + + # Outer SELECT in user-projection order (public slots only). + # Per-slot index walks each slot's public_aliases so duplicate + # interned names (DEV-1450 C13) both surface in the result. + public_aliases_user_order: list[str] = [] + outer_alias_index: Dict[str, int] = {} + for sid in planned_query.projection: + slot = slots_by_id[sid] + if slot.hidden: + continue + all_aliases = aliases_by_slot_id.get(sid, []) + if not all_aliases: + continue + idx = outer_alias_index.setdefault(sid, 0) + alias = ( + all_aliases[idx] if idx < len(all_aliases) else all_aliases[-1] + ) + outer_alias_index[sid] = idx + 1 + public_aliases_user_order.append(alias) + return self._emit_planned_outer_wrap( + chain_sql=chain_sql, + public_aliases=public_aliases_user_order, + planned_query=planned_query, + slots_by_id=slots_by_id, + available_alias_by_slot_id=available_alias_by_slot_id, + ) + + # ----------------------------------------------------------------- + # Stage 7b.10 helpers + # ----------------------------------------------------------------- + + @staticmethod + def _validate_window_transform_ops_for_7b10(*, planned_query) -> None: + """Validate transform-layer op scope. + + 7b.11 lifted ``time_shift`` and ``consecutive_periods`` from + the deferred set — both render through dedicated self-join / + staged-window CTE pairs. The deferred set is now empty; the + function stays in place as a safety net for follow-up ops + added by later slices. + + It also enforces the **composite-input** rule that survives + from 7b.10: + + * ``time_shift`` requires a slottable leaf input (the legacy + self-join CTE re-aggregates the source — composite expressions + would need an inner expression layer). + * ``consecutive_periods`` accepts a slottable leaf OR a top-level + comparison ``ArithmeticKey`` (the boolean predicate shape + ``amount:sum > 0`` is the canonical user form). Other + composite shapes (numeric subtraction, scalar calls) are + rejected with a ``composite-input transforms`` marker so the + test suite's per-op composite assertions pin a unified message. """ - where_parts: list[str] = [] - having_parts: list[str] = [] + from slayer.core.keys import ( + AggregateKey, + ArithmeticKey, + BetweenKey, + ColumnKey, + ColumnSqlKey, + InKey, + ScalarCallKey, + TimeTruncKey, + TransformKey, + ) + + # 7b.11 lifted these — placeholder set for future slices. + deferred: set = set() + + leaf_kinds = (ColumnKey, ColumnSqlKey, AggregateKey, TimeTruncKey) + # Keep aligned with _emit_consecutive_periods_ctes_for_planned — + # the renderer dispatches arithmetic ops via _compose_arithmetic_op + # which supports these binary comparisons only. + _COMPARISON_OPS = {"==", "!=", "<", "<=", ">", ">="} + + def _walk(key) -> Optional[str]: + if isinstance(key, TransformKey): + if key.op in deferred: + return key.op + return _walk(key.input) + if isinstance(key, ArithmeticKey): + for o in key.operands: + found = _walk(o) + if found: + return found + return None + if isinstance(key, ScalarCallKey): + for a in key.args: + if isinstance( + a, + (TransformKey, ArithmeticKey, ScalarCallKey, BetweenKey, InKey), + ): + found = _walk(a) + if found: + return found + return None + if isinstance(key, BetweenKey): + for k in (key.column, key.low, key.high): + found = _walk(k) + if found: + return found + return None + if isinstance(key, InKey): + # DEV-1475: only LHS column can host a deferred transform. + return _walk(key.column) + return None - # Time dimension date ranges — use the resolved SQL expression - # (which may include a time offset for shifted sub-queries) - for td in enriched.time_dimensions: - if td.date_range and len(td.date_range) == 2: - col_expr = self._resolve_sql( - sql=td.sql or td.name, name=td.name, model_name=td.model_name, + # Explicit layer ops + composite-input enforcement. + for layer in planned_query.transform_layers: + if layer.op in deferred: + raise NotImplementedError( + f"DEV-1450 stage 7b.11: transform op {layer.op!r} " + f"(self-join CTE) deferred to a follow-up slice.", ) - col = col_expr.sql(dialect=self.dialect) - where_parts.append( - f"{col} BETWEEN '{td.date_range[0]}' AND '{td.date_range[1]}'" + if layer.op in ("time_shift", "consecutive_periods"): + # Walk the layer's slot ids and assert their TransformKey + # inputs satisfy the per-op composite-input rule. + slots_map = { + s.id: s + for s in ( + list(planned_query.row_slots) + + list(planned_query.aggregate_slots) + + list(planned_query.combined_expression_slots) + ) + } + for sid in layer.slot_ids: + slot = slots_map.get(sid) + if slot is None or not isinstance(slot.key, TransformKey): + continue + inner = slot.key.input + if isinstance(inner, leaf_kinds): + continue + if ( + layer.op == "consecutive_periods" + and isinstance(inner, ArithmeticKey) + and inner.op in _COMPARISON_OPS + ): + # Boolean predicate shape — accepted. + continue + raise ValueError( + f"Nesting a transform inside {layer.op!r} " + f"(input={type(inner).__name__}) is not supported. " + f"Compute the inner transform in an earlier stage of " + f"a multi-stage `source_queries` model and reference " + f"its output in this stage." + ) + + # Reachable trees of every slot we'll need to render. + slots = ( + list(planned_query.row_slots) + + list(planned_query.aggregate_slots) + + list(planned_query.combined_expression_slots) + ) + for slot in slots: + found_op = _walk(slot.key) + if found_op is not None: + raise NotImplementedError( + f"DEV-1450 stage 7b.11: transform op {found_op!r} " + f"(reached via slot id={slot.id!r}, key=" + f"{type(slot.key).__name__}) deferred to a follow-up " + f"slice.", ) - # Parsed filters - import re - model = enriched.model_name - for f in enriched.filters: - # Post-filters are applied later, on the outer wrapper - if f.is_post_filter: + @staticmethod + def _composite_has_remote_operand( + *, + key, + slots_by_id: Dict[str, Any], + slot_id_by_key: Dict[Any, str], + planned_query, + ) -> bool: + """Whether any operand of ``key`` is materialised OUTSIDE the base CTE. + + DEV-1733: a composite whose operands include a CROSS-MODEL aggregate + (``_cm_`` CTE) or a WINDOWED aggregate (``_wm_`` CTE) cannot render in + ``_base`` — the operand column is not in that scope. Such composites + are owned by the combined SELECT instead, which resolves each operand + to its CTE-qualified column. + """ + from slayer.core.keys import AggregateKey as _AggKey + from slayer.engine.binding import walk_value_keys + + remote_slot_ids = { + p.aggregate_slot_id + for p in planned_query.cross_model_aggregate_plans + } | { + p.aggregate_slot_id + for p in planned_query.windowed_aggregate_plans + } + for node in walk_value_keys(key): + if not isinstance(node, _AggKey): continue - if f.is_having: - # HAVING: reference the aggregate by looking up the measure's - # aggregation expression from the enriched query - having_sql = f.sql - for col_name in dict.fromkeys(f.columns): - # Find the measure and build its aggregate expression - for m in enriched.measures: - if m.name == col_name: - agg_expr, _ = self._build_agg( - measure=m, - rn_suffix_map=rn_suffix_map, - default_time_col=enriched.last_agg_time_column, - filtered_rn_map=filtered_rn_map, - ) - agg_sql = agg_expr.sql(dialect=self.dialect) - # DEV-1539: wrap the substituted aggregate - # in outer parens when it's a compound - # shape (arithmetic, AND/OR connector, - # NOT, BETWEEN, IN, LIKE, IS, …) so any - # surrounding comparator's precedence is - # explicit. The compound check fires - # first because in sqlglot 30.4.3 - # ``exp.And`` / ``exp.Or`` inherit from - # ``exp.Func``; a pure inverse-atomic check - # would mis-classify a connector-rooted - # aggregate as atomic. Single function-call - # aggregates (SUM/COUNT/AVG/…) match - # ``exp.Func`` and stay unwrapped. - if ( - isinstance(agg_expr, _HAVING_AGG_COMPOUND_TYPES) - or not isinstance(agg_expr, _HAVING_AGG_ATOMIC_TYPES) - ): - agg_sql = f"({agg_sql})" - # DEV-1539: lambda replacement (backslash - # safety) + trailing ``(?!\.)`` guard so a - # measure name doesn't mis-substitute inside - # a dotted continuation in ``having_sql``. - having_sql = re.sub( - rf'(? Set[str]: + """Return slot ids the base CTE must project beyond the public + projection. + + Walks every ``TransformKey`` in ``transform_layers`` for its + ``input`` / ``partition_keys`` / ``time_key`` deps; walks every + AGGREGATE- and POST-phase ``FilterPhase.expression`` for + slot-worthy deps; walks ``OrderEntry.slot_id`` keys when + ``include_order`` is True. Only ``ColumnKey`` / ``ColumnSqlKey`` + / ``TimeTruncKey`` / ``AggregateKey`` slot ids are returned + (those that the base CTE renders); transform slot ids are + excluded since they're materialised in step CTEs. + + DEV-1501: ``aggregates_only=True`` narrows leaf collection to + ``AggregateKey`` slots ONLY (row leaves on order/filter paths are + skipped). Used by the no-transform path so that materialising a + hidden order/filter aggregate does NOT accidentally pull a hidden + ROW dep into ``base_render_order`` (which would add it to GROUP + BY and silently change query grain). Composites still recurse so + their AggregateKey operands surface. + """ + from slayer.core.keys import ( + AggregateKey, + ArithmeticKey, + BetweenKey, + ColumnKey, + ColumnSqlKey, + InKey, + Phase, + ScalarCallKey, + TimeTruncKey, + TransformKey, + ) + + if aggregates_only: + base_kinds: Tuple[type, ...] = (AggregateKey,) + else: + base_kinds = (ColumnKey, ColumnSqlKey, TimeTruncKey, AggregateKey) + out: Set[str] = set() + + def _collect_from(key) -> None: + if isinstance(key, base_kinds): + sid = slot_id_by_key.get(key) + if sid is not None: + out.add(sid) + return + # ``aggregates_only`` mode: still SKIP non-aggregate row leaves + # at the leaf level — but the composite/walker branches below + # continue to recurse so their nested AggregateKey operands + # surface. + if aggregates_only and isinstance( + key, (ColumnKey, ColumnSqlKey, TimeTruncKey), + ): + return + if isinstance(key, TransformKey): + _collect_from(key.input) + for p in key.partition_keys: + _collect_from(p) + if key.time_key is not None: + _collect_from(key.time_key) + return + if isinstance(key, ArithmeticKey): + for o in key.operands: + _collect_from(o) + return + if isinstance(key, ScalarCallKey): + for a in key.args: + if isinstance( + a, + ( + TransformKey, ArithmeticKey, ScalarCallKey, + BetweenKey, InKey, ColumnKey, ColumnSqlKey, + TimeTruncKey, AggregateKey, + ), + ): + _collect_from(a) + return + if isinstance(key, BetweenKey): + _collect_from(key.column) + _collect_from(key.low) + _collect_from(key.high) + return + if isinstance(key, InKey): + # DEV-1475: only the LHS column references a slot; the + # RHS values are bare literals with no slot identity. + _collect_from(key.column) + return + # LiteralKey / StarKey / unknown: nothing to materialise. + + # Transform layer deps. + for layer in planned_query.transform_layers: + for slot_id in layer.slot_ids: + slot = slots_by_id.get(slot_id) + if slot is None: + continue + key = slot.key + if isinstance(key, TransformKey): + _collect_from(key.input) + for p in key.partition_keys: + _collect_from(p) + if key.time_key is not None: + _collect_from(key.time_key) + + # Filter deps for AGGREGATE-phase (HAVING) and POST-phase filters + # (the latter only in the transform path, where + # ``_render_post_phase_filter_conditions`` actually applies them). + # A hidden ``revenue:last(...) > 100`` HAVING aggregate needs the + # same ranked-subquery materialisation as the ORDER BY path, so + # its AggregateKey must reach ``base_render_order`` alongside the + # projected and order-only ones (DEV-1501). POST-phase walk is + # gated on the presence of transforms — POST filters reference + # ``TransformKey`` and so are planner-unreachable in no-transform + # queries; walking them anyway would silently materialise their + # operands without applying the filter (CodeRabbit DEV-1501 PR + # #159 Group B). + has_transforms = bool(planned_query.transform_layers) + for fp in planned_query.filters_by_phase: + if fp.phase == Phase.AGGREGATE: + pass # walk + elif fp.phase == Phase.POST and has_transforms: + pass # walk else: - # WHERE: qualify column names with model name - # Dotted names (joined columns) are already table-qualified - qualified_sql = f.sql - for col_name in dict.fromkeys(f.columns): - if "." in col_name: - # Already qualified (e.g., "customers.name") — keep as-is - pass - elif col_name.isidentifier(): - qualified_sql = re.sub( - rf'(? bool: + """A layer is ready when every slot-worthy dep its TransformKeys + reference (``input`` + ``partition_keys`` + ``time_key``) has + an alias materialised in a prior CTE. + """ + from slayer.core.keys import ( + AggregateKey, + ArithmeticKey, + BetweenKey, + ColumnKey, + ColumnSqlKey, + InKey, + ScalarCallKey, + TimeTruncKey, + TransformKey, + ) + + slotted_kinds = ( + ColumnKey, ColumnSqlKey, TimeTruncKey, AggregateKey, TransformKey, + ) + + def _ready(key) -> bool: + if isinstance(key, slotted_kinds): + sid = slot_id_by_key.get(key) + if sid is None: + # Not interned as a slot — can be inlined. + return True + return sid in available_alias_by_slot_id + if isinstance(key, ArithmeticKey): + return all(_ready(o) for o in key.operands) + if isinstance(key, ScalarCallKey): + for a in key.args: + if isinstance( + a, + ( + TransformKey, ArithmeticKey, ScalarCallKey, + BetweenKey, InKey, ColumnKey, ColumnSqlKey, + TimeTruncKey, AggregateKey, + ), + ) and not _ready(a): + return False + return True + if isinstance(key, BetweenKey): + return all( + _ready(k) for k in (key.column, key.low, key.high) + ) + if isinstance(key, InKey): + # DEV-1475: only LHS column needs slot readiness; RHS + # values are literals (always ready). + return _ready(key.column) + return True + + for slot_id in layer.slot_ids: + slot = slots_by_id.get(slot_id) + if slot is None or not isinstance(slot.key, TransformKey): + continue + tk = slot.key + if not _ready(tk.input): + return False + for p in tk.partition_keys: + if not _ready(p): + return False + if tk.time_key is not None and not _ready(tk.time_key): + return False + return True + + def _resolve_agg_inputs_via_scope( # NOSONAR(S3776) — one cohesive Law-1 discovery pass: three ordered sub-passes (Column.filter → source → kwargs) over the local aggregates via small closures sharing scope/resolved. Extracting them would scatter the ordered-registration contract that keeps the base FROM byte-identical. + self, *, base_render_order, slots_by_id, scope: ScopeFrame, + ) -> "Dict[Any, Dict[str, ResolvedAggKwarg]]": + """Resolve every LOCAL aggregate's join-crossing inputs through the host + ``scope`` (Law 1) — ``scope.resolve`` anchors each ref and registers the + joins it crosses into ``scope.join_paths``, the side effect that + base-pulls the crossed LEFT JOIN. + + Three ordered sub-passes over ``base_render_order`` preserve the pre- + resolver join-registration order (Column.filter → source → kwargs): + + 1. **``Column.filter`` predicates** (DEV-1494; replaces + ``_collect_column_filter_join_paths``). The Mode-A predicate is + dual-scanned via ``_filter_join_paths`` (raw + inline-expanded, so a + placeholder dotted ref that inlines to a constant still pulls its + join) and the paths registered into the scope. + 2. **derived aggregate SOURCES** (``ColumnSqlKey`` whose ``Column.sql`` + crosses a join — DEV-1502; replaces ``_collect_aggregate_source_ + join_paths``). Discovery only; the render spec re-expands the source. + 3. **column-ref KWARGS** (``weight=`` / ``other=`` — DEV-1527). + The resolved expression is returned, keyed by ``AggregateKey`` + (frozen/hashable) → ``{kwarg_name: ResolvedAggKwarg(kind="expr")}``, + for ``_build_agg_render_spec_from_planned`` to embed. Scalar + kwargs are left out (the spec builder canonical-stringifies them). + 3b. **template-fragment KWARGS** (DEV-1709): user-supplied string + kwargs and non-overridden model-default ``AggregationParam.sql`` + fragments are scanned for crossed paths (register-only) — the + fragment text substitutes verbatim into the aggregation template, + so its joins must be in the FROM. + 4. **first/last explicit TIME ARGS** (``amount:last(customers.signup_at)`` + — DEV-1710). Discovery only; the ranked subquery's ORDER BY re-renders + the arg via ``_resolve_explicit_time_col``. Replaces the legacy + ``_collect_joined_paths_for_base`` AGGREGATE arm. A path-bearing + derived (``ColumnSqlKey``) arg — the DEV-1526 residual — is skipped. + + Cross-model aggregates (non-empty ``source.path``) are skipped in every + sub-pass: their inputs are owned by the per-plan ``_cm_*`` CTE + (Stage 4 / DEV-1708). Recurses into composite AGGREGATE keys. + """ + from slayer.core.keys import ( + AggregateKey, + ArithmeticKey, + ColumnKey, + ColumnSqlKey, + Phase, + ScalarCallKey, + ) + + resolved: "Dict[Any, Dict[str, ResolvedAggKwarg]]" = {} + + def _walk(key, fn) -> None: + if isinstance(key, AggregateKey): + if not getattr(key.source, "path", ()): + fn(key) + elif isinstance(key, ArithmeticKey): + for o in key.operands: + _walk(o, fn) + elif isinstance(key, ScalarCallKey): + for a in key.args: + _walk(a, fn) + + def _for_each_local_agg(fn) -> None: + for sid in base_render_order: + slot = slots_by_id.get(sid) + if slot is not None and slot.phase == Phase.AGGREGATE: + _walk(slot.key, fn) + + def _resolve_column_filter(key) -> None: + cfk = key.column_filter_key + if cfk is None or not cfk.canonical_sql: + return + for p in self._filter_join_paths( + sql=cfk.canonical_sql, source_relation=scope.root_relation, + source_model=scope.root_model, bundle=scope.bundle, + ): + scope.join_paths.add(p) + + def _resolve_source(key) -> None: + if isinstance(key.source, ColumnSqlKey): + scope.resolve(key.source) # register-only; render re-expands + + def _resolve_kwargs(key) -> None: + kw: Dict[str, ResolvedAggKwarg] = {} + for kname, kval in key.kwargs: + if isinstance(kval, (ColumnKey, ColumnSqlKey)): + kw[kname] = ResolvedAggKwarg(kind="expr", value=scope.resolve(kval)) + if kw: + resolved[key] = kw + + def _resolve_fragment_kwargs(key) -> None: + # DEV-1709 (PR #271 Codex review): template-fragment kwargs — + # user-supplied str values and non-overridden model-default + # ``AggregationParam.sql`` fragments — are substituted into the + # aggregation template as qualified SQL text, so their crossed + # joins must register exactly like ``Column.filter`` predicates + # do (the widened Law-3 trigger isolates on them, and the CTE + # sub-render lands here). Scanned with the same + # ``_filter_join_paths`` pipeline; unparseable fragments + # contribute nothing. + fragments = [v for _, v in key.kwargs if isinstance(v, str)] + agg_def = next( + (a for a in (scope.root_model.aggregations or []) + if a.name == key.agg), + None, + ) + if agg_def is not None: + overridden = {name for name, _ in key.kwargs} + fragments.extend( + p.sql for p in (agg_def.params or []) + if p.name not in overridden and p.sql + ) + for frag in fragments: + for p in self._filter_join_paths( + sql=frag, source_relation=scope.root_relation, + source_model=scope.root_model, bundle=scope.bundle, + ): + scope.join_paths.add(p) + + def _resolve_first_last_time_arg(key) -> None: + # DEV-1710 Stage 6 — a first/last explicit ranking-time arg + # (``amount:last(customers.signup_at)``) crosses a join exactly like + # a source / kwarg does; resolving it through the scope registers + # that join (Law 1), so the ranked subquery's ORDER BY ref is in the + # base FROM. Replaces the legacy ``_collect_joined_paths_for_base`` + # AGGREGATE arm. Register-only: the render spec re-resolves via + # ``_resolve_explicit_time_col``. + arg = self._explicit_time_arg_of(key) + if arg is None: + return + # A path-bearing derived (ColumnSqlKey) arg is a hop PAST the target + # (the DEV-1526 residual the render seam raises on) — skip it here; + # anchoring against ``source_relation`` would register a bogus join. + if isinstance(arg, ColumnSqlKey) and arg.path: + return + scope.resolve(arg) + + _for_each_local_agg(_resolve_column_filter) + _for_each_local_agg(_resolve_source) + _for_each_local_agg(_resolve_kwargs) + _for_each_local_agg(_resolve_fragment_kwargs) + _for_each_local_agg(_resolve_first_last_time_arg) + return resolved + + def _resolve_agg_kwargs_for_key( + self, *, key, source_model, source_relation: str, bundle, + ) -> "Optional[Dict[str, ResolvedAggKwarg]]": + """Resolve a single LOCAL aggregate's column-ref kwargs + (``weighted_avg(weight=)`` / ``corr(other=)``) through a fresh + host ``ScopeFrame`` → ``{name: ResolvedAggKwarg(kind="expr")}`` or ``None``. + + The base SELECT uses the batch ``_resolve_agg_inputs_via_scope`` pass over + a shared host scope (which also registers the crossed joins). The HAVING + render path (``_render_value_key_for_filter``) has no such scope, so it + builds a throwaway one here purely to reproduce the SAME anchored kwarg + expression the SELECT emits — the crossed join is already base-pulled + (the HAVING aggregate is also a ``base_render_order`` slot), so the + throwaway scope's own ``join_paths`` are intentionally discarded. + """ + from slayer.core.keys import ColumnKey, ColumnSqlKey + + kwargs = getattr(key, "kwargs", None) + if bundle is None or not kwargs: + return None + allocator = self._new_allocator() + scope = ScopeFrame( + scope_id=allocator.next_scope_id(source_relation), + root_model=source_model, + root_relation=source_relation, + bundle=bundle, + dialect=self._dialect, + allocator=allocator, + ) + resolved = { + kname: ResolvedAggKwarg(kind="expr", value=scope.resolve(kval)) + for kname, kval in kwargs + if isinstance(kval, (ColumnKey, ColumnSqlKey)) + } + return resolved or None + + def _build_base_select_for_planned( # NOSONAR(S3776) — join-path collection and derived-dim expansion are extracted to helpers; the residual is the one cohesive per-slot ROW/AGGREGATE projection + GROUP-BY assembly pass. + self, + *, + planned_query, + bundle, + source_model, + source_relation: str, + base_render_order: List[str], + slots_by_id: Dict[str, Any], + skip_cross_model_aggs: bool = False, + skip_filter_ids: Optional[Set[str]] = None, + ): + """Build the base SELECT (sqlglot ``Select``) for ``generate_from_planned``. + + Iterates ``base_render_order`` (public projection followed by + aux materialisation slot ids), rendering each ROW / AGGREGATE + slot. POST-phase slots are skipped — step CTEs render them. + + Returns ``(base_select, aliases_by_slot_id, has_aggregation, + group_by_keys)``. ``aliases_by_slot_id`` is a list per slot to + preserve duplicate public aliases (DEV-1450 C13). + + DEV-1450 stage 7b.12: joined ROW slots (ColumnKey.path != () + and TimeTruncKey.column.path != ()) are rendered by walking + the bundle's join graph and emitting ``LEFT JOIN`` clauses in + the FROM. ``skip_cross_model_aggs=True`` is passed by the + cross-model orchestrator so the ``_base`` CTE omits AGGREGATE + slots that live in a per-plan ``_cm_*`` CTE. + """ + from slayer.core.enums import TimeGranularity + from slayer.core.keys import ( + AggregateKey, + ArithmeticKey, + ColumnKey, + ColumnSqlKey, + Phase, + ScalarCallKey, + TimeTruncKey, + ) + + # DEV-1706 Stage 2: the host base is a single scope; every join-crossing + # ref registers its path into ``host_scope.join_paths`` as a side effect + # of being resolved through the scope (Law 1 — discovery can never be + # forgotten). The legacy join collectors are gone: their work is now the + # scope passes below. The scope's ordered ``join_paths`` reproduce the + # collectors' first-seen registration order — derived dims → WHERE filters + # → Column.filter → source → kwargs → first/last time args — so the base + # FROM is byte-identical. + # + # Stage 2's host base has no projection boundary, so the allocator mints + # no ``_val_`` names here and a local instance suffices; the generation- + # wide allocator (D-E) arrives with the CTE scopes in Stage 4. + # + # Walk row slots for every joined DIMENSION path first (join-order + # position 1); the scope's paths (derived dims, filters, aggregate + # inputs, and — DEV-1710 Stage 6 — first/last time args) append after it. + needed_join_paths = self._collect_joined_paths_for_base( + base_render_order=base_render_order, + slots_by_id=slots_by_id, + order_slot_ids=[e.slot_id for e in planned_query.order], + ) + # DEV-1708 (D-E): share the generation-wide allocator so host-base and + # per-plan ``_cm_*`` CTE ``_val_`` names are globally unique. + host_allocator = self._gen_allocator or self._new_allocator() + host_scope = ScopeFrame( + scope_id=host_allocator.next_scope_id(source_relation), + root_model=source_model, + root_relation=source_relation, + bundle=bundle, + dialect=self._dialect, + allocator=host_allocator, + ) + # Pre-expand derived (ColumnSqlKey) ROW + TIME dimensions: inline + # sibling/joined derived refs (DEV-1333 / DEV-1410) and register any + # joins their SQL crosses into the scope (position 2). Returns the + # expanded-expr-by-slot-id map the render branch reads. + derived_expr_by_sid = self._expand_derived_row_dims( + base_render_order=base_render_order, slots_by_id=slots_by_id, + source_relation=source_relation, source_model=source_model, + bundle=bundle, scope=host_scope, + ) + # WHERE-phase filters referencing joined columns (direct, derived, or + # Mode-A ``__`` paths) register their joins into the scope too (position + # 3). Filters routed to a cross-model ``_cm_*`` CTE (``skip_filter_ids``) + # are applied there, not on ``_base`` — registering their join here would + # add an unused (and, for one-to-many joins, cardinality-changing) LEFT + # JOIN. + self._resolve_where_filter_joins_via_scope( + planned_query=planned_query, scope=host_scope, + skip_filter_ids=skip_filter_ids, + ) + # Every LOCAL aggregate's join-crossing inputs resolve through the scope + # next: ``Column.filter`` (position 4; DEV-1494), derived aggregate SOURCE + # (position 5; DEV-1502), column-ref KWARGS (position 6; DEV-1527 — + # ``weighted_avg(weight=)`` / ``corr(other=)``, whose resolved + # expression is embedded verbatim (``kind="expr"``) into the render spec, + # replacing the ``agg_kwarg_canonical_str`` round-trip that collapsed a + # derived column to a bare, non-existent name), and first/last explicit + # TIME ARGS (position 7; DEV-1710 — ``amount:last(customers.signup_at)``). + resolved_agg_kwargs = self._resolve_agg_inputs_via_scope( + base_render_order=base_render_order, + slots_by_id=slots_by_id, + scope=host_scope, + ) + # Merge the scope's registered paths (positions 2-7, in first-seen order) + # after the dimension paths (position 1) → byte-identical FROM. + for p in host_scope.join_paths.as_list(): + if p not in needed_join_paths: + needed_join_paths.append(p) + from_clause, base_joins = self._build_from_and_joins( + source_model=source_model, + source_relation=source_relation, + joined_paths=needed_join_paths, + bundle=bundle, + ) + + # DEV-1450: first/last AGGREGATIONS rank rows via a ROW_NUMBER + # subquery (mirrors legacy ``_generate_base`` + ``_build_last_ + # ranked_from``). + if self._has_first_last_aggregate( + base_render_order=base_render_order, slots_by_id=slots_by_id, + ): + # DEV-1503 — local first/last in ``_base`` alongside cross-model + # CTEs is supported: the ranked subquery wraps ``_base`` for the + # local measures; cross-model / filtered-local aggregates are + # deferred to their per-plan ``_cm_*`` CTEs (their slot ids are + # excluded from ``base_render_order`` by the caller, so + # ``_build_first_last_base_select`` never sees them and emits no + # dangling references). + return self._build_first_last_base_select( + planned_query=planned_query, + bundle=bundle, + source_model=source_model, + source_relation=source_relation, + base_render_order=base_render_order, + slots_by_id=slots_by_id, + from_clause=from_clause, + base_joins=base_joins, + skip_filter_ids=skip_filter_ids, + ) + + select_columns: list[exp.Expression] = [] + group_by_keys: Dict[str, exp.Expression] = {} + has_aggregation = False + alias_index: Dict[str, int] = {} + aliases_by_slot_id: Dict[str, List[str]] = {} + + def _record_alias(sid: str, full_alias: str) -> None: + aliases_by_slot_id.setdefault(sid, []).append(full_alias) + + for sid in base_render_order: + slot = slots_by_id[sid] + # DEV-1450 stage 7b.12: joined ROW slots emit the FULL + # dotted result-key form (``orders.customers.region_id``). + # The planner emits a flat ``customers__region_id`` + # declared_name for downstream stage binding (DEV-1449 / C4 + # contract), but the public projection alias must preserve + # the dotted path for the result-key contract (P10). Local + # slots keep the existing ``.`` + # form. + full_alias = self._full_alias_for_slot( + slot=slot, + source_relation=source_relation, + alias_index=alias_index, + ) + + if slot.phase == Phase.ROW: + key = slot.key + if isinstance(key, ColumnKey): + col_expr = self._joined_or_local_dim_expr( + path=key.path, + leaf=key.leaf, + source_model=source_model, + source_relation=source_relation, + bundle=bundle, + ) + select_columns.append(col_expr.copy().as_(full_alias)) + group_by_keys.setdefault(sid, col_expr) + _record_alias(sid, full_alias) + elif isinstance(key, TimeTruncKey): + col_expr = self._raw_time_col_expr_for_planned( + time_column=key.column, + source_model=source_model, + source_relation=source_relation, + bundle=bundle, + ) + trunc_expr = self._build_date_trunc( + col_expr=col_expr, + granularity=TimeGranularity(key.granularity), + ) + select_columns.append(trunc_expr.copy().as_(full_alias)) + group_by_keys.setdefault(sid, trunc_expr) + _record_alias(sid, full_alias) + elif isinstance(key, ColumnSqlKey): + # A derived column (``Column.sql`` set) used as a dimension, + # e.g. ``ratio = A.bar / B.foo_normalized`` (cross-table) or + # ``c2 = c1 * 2`` (sibling-derived chain). Local + # (``path == ()``) derived columns are pre-expanded above + # (sibling/joined refs inlined, joins pulled in); fall back + # to the non-expanded resolution for any other shape. + col_expr = derived_expr_by_sid.get(sid) + if col_expr is None: + col_expr = self._dim_column_expr_from_planned( + source_model=source_model, + source_relation=source_relation, + leaf=key.column_name, ) - where_parts.append(qualified_sql) + select_columns.append(col_expr.copy().as_(full_alias)) + group_by_keys.setdefault(sid, col_expr) + _record_alias(sid, full_alias) + elif isinstance(key, (ScalarCallKey, ArithmeticKey)): + # DEV-1576 / DEV-1717: a ROW-phase composite here is a + # non-aggregating measure expression (a bare column, or + # arithmetic / scalar-call over bare columns such as + # ``round(amount, 2)`` / ``abs(amount)`` / ``amount + 1``). + # Dimensions are ColumnKey / TimeTruncKey / ColumnSqlKey, + # already handled above; the only way to reach here with a + # composite key is a measure that never aggregates. Raise + # the same actionable "Bare measure name" error the + # enrich_query path raises rather than leaking an internal + # NotImplementedError. + bare = _first_bare_column_name(key) or full_alias + raise ValueError( + f"Bare measure name '{bare}' is not valid. " + f"Use colon syntax (e.g., '{bare}:sum', '{bare}:avg'). " + f"For COUNT(*), use '*:count'." + ) + else: + raise NotImplementedError( + f"DEV-1450 stage 7b.10+: row-phase key type " + f"{type(key).__name__} not supported in the " + f"local-only / time-dim slice." + ) - where_clause = None - if where_parts: - where_sql = _SQL_AND_JOINER.join(where_parts) - # DEV-1378: ``_parse_predicate`` wraps in SELECT context so a - # filter starting with ``replace(...)`` (a SQLite/MySQL - # statement keyword) is parsed as a function call rather - # than the REPLACE INTO statement form. - where_clause = self._parse_predicate(where_sql) + elif slot.phase == Phase.AGGREGATE: + key = slot.key + if not isinstance(key, AggregateKey): + # AGGREGATE-phase composite (arithmetic / scalar-call of + # aggregates, e.g. ``expensenet:avg + benchmarkexp:avg``). + # Render inline; cast the whole composite once. DEV-1527: + # thread the host scope's resolved column-ref kwargs so a + # crossing derived kwarg inside a composite operand + # (``amount:weighted_avg(weight=) + quantity:sum``) + # embeds its expanded join-anchored expression instead of a + # bare, non-existent name. + composite, any_agg = self._render_aggregate_composite_expr( + key=key, + slot=slot, + source_model=source_model, + source_relation=source_relation, + bundle=bundle, + resolved_agg_kwargs=resolved_agg_kwargs, + ) + if any_agg: + composite = _wrap_cast_for_type(composite, slot.type) + has_aggregation = True + select_columns.append(composite.copy().as_(full_alias)) + _record_alias(sid, full_alias) + continue + agg_path = getattr(key.source, "path", ()) + if agg_path: + if skip_cross_model_aggs: + # Cross-model aggregate; rendered by the per-plan + # ``_cm_*`` CTE. Skip in the host base. + continue + raise NotImplementedError( + f"DEV-1450 stage 7b.12: cross-model aggregate " + f"(source.path={agg_path!r}) reached the local " + f"base SELECT path. The cross-model orchestrator " + f"should have routed this through `_render_with_" + f"cross_model_plans`." + ) + # DEV-1450 stage 7b.12: ``column_filter_key`` is now + # propagated into the synthetic EnrichedMeasure's + # ``filter_sql`` field so ``_build_agg`` wraps the + # aggregate as ``SUM(CASE WHEN THEN col END)``. + synth = self._build_agg_render_spec_from_planned( + slot=slot, + key=key, + source_model=source_model, + source_relation=source_relation, + full_alias=full_alias, + bundle=bundle, + resolved_agg_kwargs=resolved_agg_kwargs.get(key), + ) + agg_expr, is_agg = self._build_agg(synth) + if is_agg: + agg_expr = _wrap_cast_for_type(agg_expr, slot.type) + has_aggregation = True + select_columns.append(agg_expr.copy().as_(full_alias)) + _record_alias(sid, full_alias) + else: + # POST-phase slot in projection — handled by step CTEs. + # Don't add to base select; step CTE will materialise. + continue - having_clause = None - if having_parts: - having_sql = _SQL_AND_JOINER.join(having_parts) - having_clause = self._parse_predicate(having_sql) + base_select = exp.Select() + for col in select_columns: + base_select = base_select.select(col) + base_select = base_select.from_(from_clause) + for join_expr, on_expr, join_type in base_joins: + base_select = base_select.join( + join_expr, on=on_expr, join_type=join_type, + ) + return ( + base_select, aliases_by_slot_id, has_aggregation, group_by_keys, + False, None, + ) - return where_clause, having_clause + def _has_first_last_aggregate( + self, *, base_render_order: List[str], slots_by_id: Dict[str, Any], + ) -> bool: + """True if any LOCAL ``first`` / ``last`` AGGREGATE slot appears in + the base render order — directly as an ``AggregateKey`` slot OR + as an operand inside a composite (``ArithmeticKey`` / + ``ScalarCallKey``) aggregate slot. + + Cross-model first/last (non-empty ``source.path``) is excluded — + it is not rendered by the ranked-subquery path (each cross-model + aggregate has its own CTE). DEV-1501 (Codex round 4): composite- + only first/last (e.g. ``last(created_at) + last(updated_at)`` + with no direct sibling) must still trigger the ranked-subquery + path; without composite-aware detection the composite render + would emit ``MAX(CASE WHEN _last_rn = 1 …)`` referencing a + column the bare-FROM never projects. + """ + from slayer.core.keys import AggregateKey, Phase + + for sid in base_render_order: + slot = slots_by_id.get(sid) + if slot is None or slot.phase != Phase.AGGREGATE: + continue + key = slot.key + if ( + isinstance(key, AggregateKey) + and key.agg in ("first", "last") + and not getattr(key.source, "path", ()) + ): + return True + # Composite slot (no direct AggregateKey): walk for first/ + # last AggregateKey leaves. The composite render needs the + # ranked subquery so each operand's ``_first_rn`` / + # ``_last_rn{suffix}`` column exists. + if not isinstance(key, AggregateKey) and _iter_first_last_leaves(key): + return True + return False + + def _resolve_ranking_time_column_from_planned( + self, + *, + base_render_order: List[str], + slots_by_id: Dict[str, Any], + source_model, + source_relation: str, + bundle, + ) -> Optional[str]: + """Resolve the default ORDER-BY time column for first/last + ROW_NUMBER ranking (mirrors legacy ``_resolve_last_agg_time``). + + Precedence (matching legacy): the first ``DATE`` / ``TIMESTAMP`` + regular dimension, then the first time-dimension slot's raw column, + then the model's ``default_time_dimension``. Returns the qualified + SQL string (e.g. ``"orders.created_at"`` / ``"stores.opened_at"``), + or ``None`` when nothing temporal is in scope. + + (The legacy ``main_time_dimension`` short-circuit and the + filter-referenced-date fallback are corner cases the spec permits + diverging on; they are not reproduced here.) + """ + from slayer.core.keys import ColumnKey, Phase, TimeTruncKey + + for sid in base_render_order: + slot = slots_by_id[sid] + if slot.phase == Phase.ROW and isinstance(slot.key, ColumnKey): + model = source_model + for hop in slot.key.path: + nxt = bundle.get_referenced_model(hop) + if nxt is None: + model = None + break + model = nxt + if model is None: + continue + col_def = next( + (c for c in model.columns if c.name == slot.key.leaf), None, + ) + if col_def is not None and col_def.type in ( + DataType.DATE, DataType.TIMESTAMP, + ): + return self._joined_or_local_dim_expr( + path=slot.key.path, leaf=slot.key.leaf, + source_model=source_model, + source_relation=source_relation, bundle=bundle, + ).sql(dialect=self.dialect) + for sid in base_render_order: + slot = slots_by_id[sid] + if slot.phase == Phase.ROW and isinstance(slot.key, TimeTruncKey): + return self._raw_time_col_expr_for_planned( + time_column=slot.key.column, source_model=source_model, + source_relation=source_relation, bundle=bundle, + ).sql(dialect=self.dialect) + if source_model.default_time_dimension: + return f"{source_relation}.{source_model.default_time_dimension}" + return None + + @staticmethod + def _explicit_time_arg_of(key): + """The explicit positional ranking-time arg of a ``first`` / ``last`` + aggregate, or ``None``. + + The SINGLE arg-selection contract shared by the three sites that must + never disagree on WHICH positional arg is the time column (DEV-1710 / + Codex F1): the raise-gate in ``_build_first_last_base_select``, the + join-discovery pass in ``_resolve_agg_inputs_via_scope``, and the render + seam ``_resolve_explicit_time_col``. Returns the FIRST positional arg + iff it is a ``ColumnKey`` / ``ColumnSqlKey``; ``None`` for a + non-first/last agg, empty args, or a first positional arg of any other + type (first/last never takes a leading non-column positional). + """ + from slayer.core.keys import ColumnKey, ColumnSqlKey + + if key.agg not in ("first", "last"): + return None + for a in key.args: + return a if isinstance(a, (ColumnKey, ColumnSqlKey)) else None + return None + + def _resolve_explicit_time_col( + self, + *, + key, + source_model, + source_relation: str, + bundle=None, + ) -> Optional[str]: + """Resolve the explicit positional time arg on a ``first`` / ``last`` + aggregate into a SQL string suitable for ``ORDER BY`` inside the + ranked subquery. + + Handles both bare-column refs (``ColumnKey`` — + ``amount:last(created_at)``) and derived-column refs (``ColumnSqlKey`` + — ``amount:last(net_amount_date)`` where ``net_amount_date`` has a + non-trivial ``Column.sql``). DEV-1710 Stage 6: when a ``bundle`` is + available the arg is anchored through a ``ScopeFrame`` (Law 1) — the + same resolver the host base / kwargs passes use — so a bare joined ref + qualifies to its ``__``-path alias, a derived expression's inner bare + refs qualify to ``source_relation`` (never ambiguous against a + same-named joined column), and reserved-word relations are quoted + (DEV-1686). Without a ``bundle`` (the render-spec unit path) it falls + back to bare-ident qualification / verbatim emit. + + Returns ``None`` for non-first/last aggs and when ``key.args`` is empty + or its first element is neither a ``ColumnKey`` nor a ``ColumnSqlKey`` + (see ``_explicit_time_arg_of``). A derived time arg (``ColumnSqlKey``) + whose ``path`` is non-empty AFTER the DEV-1707 cross-model reroot — a + column a hop PAST the target — raises ``NotImplementedError`` rather + than silently emitting against a relation the isolated CTE does not + join; that residual-hop case is tracked as DEV-1526 (Stage 4). The + analogous residual ``ColumnKey`` arg is caught loudly by the + scope-closure validator (``SLAYER_VALIDATE_SCOPES``) instead. + """ + from slayer.core.keys import ColumnKey, ColumnSqlKey + + arg = self._explicit_time_arg_of(key) + if arg is None: + return None + if isinstance(arg, ColumnSqlKey) and arg.path: + raise NotImplementedError( + f"Derived time column with a residual join path " + f"(path={arg.path!r}, column={arg.column_name!r}) on a " + f"first/last positional arg is not yet supported by " + f"the ranked-subquery builder: the isolated CTE does " + f"not pull the residual join. Post-DEV-1707 the " + f"cross-model reroot strips the target prefix, so this " + f"fires only for a time arg a hop PAST the target; " + f"tracked as DEV-1526 (Stage 4)." + ) + # Validate a derived arg's existence up front so the not-found case is a + # clear error rather than the resolver silently anchoring the bare name. + col = None + if isinstance(arg, ColumnSqlKey): + col = next( + (c for c in source_model.columns if c.name == arg.column_name), + None, + ) + if col is None: + raise ValueError( + f"Derived time column {arg.column_name!r} (positional " + f"arg of {key.agg!r}) not found on model " + f"{source_model.name!r}." + ) + if bundle is not None: + # Law 1 — anchor the arg through a throwaway host-rooted scope. Its + # ``join_paths`` are discarded (discovery is owned by the base + # aggregate-input pass, which registers the same join); this call is + # purely to reproduce the SAME anchored SQL the ORDER BY needs. Same + # throwaway-frame pattern as ``_resolve_agg_kwargs_for_key``. + allocator = self._new_allocator() + scope = ScopeFrame( + scope_id=allocator.next_scope_id(source_relation), + root_model=source_model, + root_relation=source_relation, + bundle=bundle, + dialect=self._dialect, + allocator=allocator, + ) + return scope.resolve(arg).sql(dialect=self.dialect) + # No bundle (defensive; the render-spec unit path): bare ColumnKey + # qualifies to its ``__``-path alias / source relation, a derived + # bare-ident qualifies to the source relation, else emit verbatim. + if isinstance(arg, ColumnKey): + relation = "__".join(arg.path) if arg.path else source_relation + return f"{relation}.{arg.leaf}" + col_sql = col.sql if col.sql else col.name + if col_sql.isidentifier(): + return f"{source_relation}.{col_sql}" + return self._parse(col_sql).sql(dialect=self.dialect) + + def _build_ranked_subquery_from_planned( # NOSONAR(S3776) — Group 2 already factored the per-spec ROW_NUMBER passes into _build_unfiltered_rn_columns / _build_filtered_rn_columns; what's left is exp.Select / from / joins / where assembly that has to live in one place. + self, + *, + source_relation: str, + default_time_col_sql: str, + partition_exprs: List[exp.Expression], + extra_projections: List[Tuple[str, exp.Expression]], + synth_specs: List[AggRenderSpec], + from_clause: exp.Expression, + base_joins: List, + where_clause: Optional[exp.Expression], + ) -> Tuple[exp.Expression, dict, dict, dict]: + """Build the ROW_NUMBER-ranked subquery that wraps the source for + first/last aggregation (planned-native port of + ``_build_last_ranked_from``). + + Projects ``source_relation.*`` plus the supplied ``extra_projections`` + (truncated time dimensions / joined dimensions referenced by the + outer SELECT) plus one ``ROW_NUMBER`` column per distinct + (effective-time-column, agg) pair. Filtered first/last measures get + a dedicated ranking column (non-matching rows pushed to the bottom) + and a boolean match flag. WHERE is applied INSIDE so it filters raw + rows before ranking. Returns ``(subquery, rn_suffix_map, + filtered_rn_map, filtered_match_map)``. + """ + partition_clause = "" + if partition_exprs: + partition_clause = _SQL_PARTITION_BY + ", ".join( + p.sql(dialect=self.dialect) for p in partition_exprs + ) + + select_exprs: List[exp.Expression] = [ + exp.Column(this=exp.Star(), table=exp.to_identifier(source_relation)), + ] + for alias, e in extra_projections: + select_exprs.append(e.copy().as_(alias)) + + unfiltered_exprs, rn_suffix_map = self._build_unfiltered_rn_columns( + synth_specs=synth_specs, + default_time_col_sql=default_time_col_sql, + partition_clause=partition_clause, + ) + select_exprs.extend(unfiltered_exprs) + + filtered_exprs, filtered_rn_map, filtered_match_map = ( + self._build_filtered_rn_columns( + synth_specs=synth_specs, + default_time_col_sql=default_time_col_sql, + partition_clause=partition_clause, + ) + ) + select_exprs.extend(filtered_exprs) + + inner = exp.Select() + for e in select_exprs: + inner = inner.select(e) + inner = inner.from_(from_clause) + for join_expr, on_expr, join_type in base_joins: + inner = inner.join(join_expr, on=on_expr, join_type=join_type) + if where_clause is not None: + inner = inner.where(where_clause) + subquery = exp.Subquery( + this=inner, alias=exp.to_identifier(source_relation), + ) + return subquery, rn_suffix_map, filtered_rn_map, filtered_match_map + + def _build_unfiltered_rn_columns( + self, + *, + synth_specs: List[AggRenderSpec], + default_time_col_sql: str, + partition_clause: str, + ) -> Tuple[List[exp.Expression], Dict[str, str]]: + """One ``ROW_NUMBER`` projection per distinct effective time column + for the unfiltered ``first`` / ``last`` specs. + + Each unique effective time column gets a stable suffix in render + order (first sorted gets ``""``, then ``"_2"``, ...); the same + time column shared by both ``first`` and ``last`` produces two + projections (`_first_rn{suffix}` ASC, `_last_rn{suffix}` DESC). + Returns ``(rn_select_exprs, rn_suffix_map)``. + """ + time_col_agg_types: Dict[str, set] = {} + for m in synth_specs: + if m.aggregation in ("first", "last") and not m.filter_sql: + eff = m.time_column or default_time_col_sql + time_col_agg_types.setdefault(eff, set()).add(m.aggregation) + sorted_tcs = sorted(time_col_agg_types) + rn_suffix_map: Dict[str, str] = { + tc: ("" if i == 0 else f"_{i + 1}") + for i, tc in enumerate(sorted_tcs) + } + rn_exprs: List[exp.Expression] = [] + for tc in sorted_tcs: + suffix = rn_suffix_map[tc] + if "last" in time_col_agg_types[tc]: + rn_exprs.append( + self._parse( + f"ROW_NUMBER() OVER ({partition_clause} " + f"ORDER BY {tc} DESC)" + ).as_(f"_last_rn{suffix}") + ) + if "first" in time_col_agg_types[tc]: + rn_exprs.append( + self._parse( + f"ROW_NUMBER() OVER ({partition_clause} " + f"ORDER BY {tc} ASC)" + ).as_(f"_first_rn{suffix}") + ) + return rn_exprs, rn_suffix_map + + def _build_filtered_rn_columns( + self, + *, + synth_specs: List[AggRenderSpec], + default_time_col_sql: str, + partition_clause: str, + ) -> Tuple[List[exp.Expression], Dict[str, str], Dict[str, str]]: + """One dedicated ``ROW_NUMBER`` + match-flag projection per distinct + ``(filter, time, agg)`` triple for the filtered ``first`` / ``last`` + specs. + + Filtered first/last needs to push non-matching rows past the + winners; emits ``ROW_NUMBER() OVER (... ORDER BY CASE WHEN + THEN 0 ELSE 1 END, "`` for any slot id in this map, and a + # ``_wm_`` CTE is joined into the combined FROM exactly as a + # ``_cm_`` one is. Without these entries the operand would fall + # through to the ``_base.`` fallback and read a plain + # aggregate (or dangle). + for plan in planned_query.windowed_aggregate_plans: + outer_composite_cm_map[plan.aggregate_slot_id] = ( + wm_cte_name_for_plan[plan.aggregate_slot_id], + wm_agg_col_for_plan[plan.aggregate_slot_id], + ) + + def _render_outer_composite(cslot) -> str: + rendered = self._render_filter_for_outer_wrapper( + key=cslot.key, + slot_by_key=slot_by_key, + cross_model_agg_slot_to_cm=outer_composite_cm_map, + aliases_by_slot_id=aliases_by_slot_id, + ) + if cslot.type is not None: + rendered = _wrap_cast_for_type(rendered, cslot.type) + return rendered.sql(dialect=self.dialect) + + # Projected outer composites: cycle through ``public_aliases`` + # for each occurrence in ``planned_query.projection``. C13 lets + # the same composite slot project under multiple user-declared + # names; emitting ``public_aliases[0]`` twice (and overwriting + # ``combined_aliases_by_slot_id[sid]``) would drop the second + # alias (CodeRabbit thread 2). + outer_emission_count: Dict[str, int] = {} + for sid in planned_query.projection: + if sid not in outer_composite_slot_ids: + continue + cslot = slots_by_id.get(sid) + if cslot is None: + continue + aliases_for_slot = list(cslot.public_aliases) or [ + cslot.declared_name, + ] + idx = outer_emission_count.get(sid, 0) + public_alias = ( + aliases_for_slot[idx] + if idx < len(aliases_for_slot) + else aliases_for_slot[-1] + ) + outer_emission_count[sid] = idx + 1 + full_alias = f"{source_relation}.{public_alias}" + combined_parts.append( + f'{_render_outer_composite(cslot)} AS {self._quote_ident(full_alias)}', + ) + combined_aliases_by_slot_id.setdefault(sid, []).append( + full_alias, + ) + # The first emitted alias is the canonical handle the + # combined-level ORDER BY references. + outer_composite_order_alias_by_sid.setdefault(sid, full_alias) + + # Order-only outer composites (Codex round 3 #1 / round 8): + # not in public projection but referenced by + # ``planned_query.order``. Rather than materialise as a + # hidden combined-SELECT column (which would leak as an + # extra public-result column on the no-transform path AND + # disappear from the cross-model transform chain's + # carry-forward dict), render the expression INLINE in the + # combined ORDER BY. ``outer_composite_order_expressions`` + # carries the rendered SQL for each order-only slot; the + # order-by builder emits `` {direction}`` bare. + projection_set_for_outer = set(planned_query.projection) + for entry in planned_query.order: + sid = entry.slot_id + if sid not in outer_composite_slot_ids: + continue + if sid in projection_set_for_outer: + continue + cslot = slots_by_id.get(sid) + if cslot is None: + continue + outer_composite_order_expressions[sid] = ( + _render_outer_composite(cslot) + ) + # Cross-model side: one entry per declared user alias, all + # referencing the CTE's aggregate column (canonical for the forward + # path; the sub-plan alias for the re-rooted path). When the public + # alias matches the CTE column name, no ``AS`` remap fires. + for plan in planned_query.cross_model_aggregate_plans: + agg_slot = slots_by_id[plan.aggregate_slot_id] + canonical_alias = canonical_alias_for_plan[plan.aggregate_slot_id] + agg_col_alias = agg_col_alias_for_plan[plan.aggregate_slot_id] + cte_name = _cte_name_from_alias("_cm_", canonical_alias) + # DEV-1495 bug 2 / DEV-1712: an order-by-only (hidden) cross-model + # aggregate never surfaces in the combined projection — its CTE is + # still joined below, and the ORDER BY references it CTE-qualified + # (``hidden_cte_order_refs``). Trimming it keeps the outer SELECT to + # the user-declared columns (Law 2 projection boundary). Only when + # there is NO transform chain: a hidden CMA feeding a transform + # layer (``cumsum(customers.revenue:sum)``) must stay projected so + # the step CTE can consume it — the transform outer wrap does the + # public-vs-hidden trim in that path. + trim_hidden = plan.hidden and not planned_query.transform_layers + public_aliases = ( + [] + if trim_hidden + else self._public_aliases_for_cross_model_agg( + slot=agg_slot, + source_relation=source_relation, + canonical_alias=canonical_alias, + ) + ) + for pub in public_aliases: + if pub == agg_col_alias: + combined_parts.append(f'{cte_name}.{self._quote_ident(agg_col_alias)}') + else: + combined_parts.append( + f'{cte_name}.{self._quote_ident(agg_col_alias)} AS {self._quote_ident(pub)}', + ) + combined_aliases_by_slot_id[plan.aggregate_slot_id] = list( + public_aliases, + ) + + # DEV-1714 Stage 10 — windowed side: project each ``_wm_`` CTE's + # aggregate column. Codex#2: one occurrence per declared user alias (C13 + # lets the same windowed key be selected under multiple names — the CTE + # holds one aggregate column, remapped ``AS`` each public alias). The + # column name already IS the primary dotted result key, so that occurrence + # needs no remap. Like the cross-model ``_cm_`` columns above, windowed + # columns are grouped after the ``_base`` projection rather than woven + # into ``planned_query.projection`` order — deterministic (measure + # declaration order) and harmless because results are keyed by name, not + # position. + for plan in planned_query.windowed_aggregate_plans: + agg_slot = slots_by_id[plan.aggregate_slot_id] + cte_name = wm_cte_name_for_plan[plan.aggregate_slot_id] + agg_col = wm_agg_col_for_plan[plan.aggregate_slot_id] + # DEV-1733: an order-only (hidden) windowed aggregate never surfaces + # in the combined projection — its ``_wm_`` CTE is still joined + # below and the ORDER BY references it CTE-qualified + # (``hidden_wm_order_ref``). Same trim predicate the hidden + # cross-model aggregate uses: with a transform chain on top the + # column must stay projected so the step CTE can consume it, and + # the transform outer wrap does the public-vs-hidden trim there. + if plan.hidden and not planned_query.transform_layers: + combined_aliases_by_slot_id[plan.aggregate_slot_id] = [] + continue + public_names = list(agg_slot.public_aliases) or ( + [agg_slot.public_name] if agg_slot.public_name else [] + ) + full_aliases = [f"{source_relation}.{p}" for p in public_names] or [agg_col] + for full in full_aliases: + if full == agg_col: + combined_parts.append(f'{cte_name}.{self._quote_ident(agg_col)}') + else: + combined_parts.append( + f'{cte_name}.{self._quote_ident(agg_col)} AS {self._quote_ident(full)}', + ) + combined_aliases_by_slot_id[plan.aggregate_slot_id] = list(full_aliases) + + from_clause_str = "FROM _base" + joined_cte_names: set = set() + for plan in planned_query.cross_model_aggregate_plans: + canonical_alias = canonical_alias_for_plan[plan.aggregate_slot_id] + cte_name = _cte_name_from_alias("_cm_", canonical_alias) + if cte_name in joined_cte_names: + continue + joined_cte_names.add(cte_name) + joinback_pairs = joinback_pairs_for_plan.get( + plan.aggregate_slot_id, [], + ) + if joinback_pairs: + # DEV-1708 / Codex F2: the grain join-back uses a dialect-aware + # NULL-SAFE equality so NULL dimension values and nullable + # truncated time grains join back instead of dropping their + # aggregate (a plain ``=`` yields NULL for NULL = NULL). + join_parts = [ + self._null_safe_join_pair_sql( + left_sql=f'_base.{self._quote_ident(host)}', + right_sql=f'{cte_name}.{self._quote_ident(cte_col)}', + ) + for host, cte_col in joinback_pairs + ] + from_clause_str += ( + f"\nLEFT JOIN {cte_name} ON " + _SQL_AND_JOINER.join(join_parts) + ) + else: + from_clause_str += f"\nCROSS JOIN {cte_name}" + + # DEV-1714 Stage 10 — LEFT JOIN each ``_wm_`` CTE back to ``_base`` on + # the shared grain (null-safe, so NULL-dim / nullable-grain groups keep + # a row; the windowed value for a NULL-dim group is NULL — the plain + # ``=`` inside the CTE never matches NULL, a documented consequence). + for plan in planned_query.windowed_aggregate_plans: + cte_name = wm_cte_name_for_plan[plan.aggregate_slot_id] + joinback_pairs = wm_joinback_pairs_for_plan.get( + plan.aggregate_slot_id, [], + ) + if joinback_pairs: + join_parts = [ + self._null_safe_join_pair_sql( + left_sql=f'_base.{self._quote_ident(host)}', + right_sql=f'{cte_name}.{self._quote_ident(cte_col)}', + ) + for host, cte_col in joinback_pairs + ] + from_clause_str += ( + f"\nLEFT JOIN {cte_name} ON " + _SQL_AND_JOINER.join(join_parts) + ) + else: + from_clause_str += f"\nCROSS JOIN {cte_name}" + + combined_select_sql = ( + f"SELECT {', '.join(combined_parts)}\n{from_clause_str}" + ) + + # DEV-1503 — outer combined-SELECT WHERE wrapper. AGGREGATE-phase + # host filters routed here in the classification pass above + # (``outer_where_filters``) render now against the joined-back + # ``_cm_*`` column for isolated-aggregate refs and ``_base.`` + # for any local operand (which the ``_add_local_aux_slots`` pass + # has materialised in ``_base``). + if outer_where_filters: + # Map EVERY cross-model aggregate slot (filtered-local AND + # forward / re-rooted) to its ``_cm_*`` CTE column — a mixed + # AGGREGATE filter like ``loss_payment_amt:sum > + # customers.revenue:sum`` triggers the outer wrapper through + # the filtered-local operand but ALSO has to resolve the + # forward cross-model operand on the same outer scope. If + # only filtered-local plans were mapped, the forward operand + # would fall through to the ``_base`` fallback and the + # renderer would raise (CodeRabbit thread 2). + cross_model_agg_slot_to_cm: Dict[str, Tuple[str, str]] = {} + for plan in planned_query.cross_model_aggregate_plans: + canonical_alias = canonical_alias_for_plan[plan.aggregate_slot_id] + cte_name = _cte_name_from_alias("_cm_", canonical_alias) + agg_col_alias = agg_col_alias_for_plan[plan.aggregate_slot_id] + cross_model_agg_slot_to_cm[plan.aggregate_slot_id] = ( + cte_name, agg_col_alias, + ) + outer_where_parts: List[str] = [] + for fp in outer_where_filters: + rendered = self._render_filter_for_outer_wrapper( + key=fp.expression.value_key, + slot_by_key=slot_by_key, + cross_model_agg_slot_to_cm=cross_model_agg_slot_to_cm, + aliases_by_slot_id=aliases_by_slot_id, + ) + if isinstance(rendered, (exp.And, exp.Or)): + rendered = exp.Paren(this=rendered) + outer_where_parts.append(rendered.sql(dialect=self.dialect)) + combined_select_sql += ( + "\nWHERE " + _SQL_AND_JOINER.join(outer_where_parts) + ) + + # DEV-1714 Stage 10 — POST-phase filters referencing a windowed measure + # render as an outer WHERE on the combined SELECT (never HAVING on the + # plain base aggregate), resolving each windowed slot to its ``_wm_`` + # CTE's joined-back aggregate column. + if planned_query.windowed_aggregate_plans: + wm_slot_to_cte: Dict[str, Tuple[str, str]] = { + p.aggregate_slot_id: ( + wm_cte_name_for_plan[p.aggregate_slot_id], + wm_agg_col_for_plan[p.aggregate_slot_id], + ) + for p in planned_query.windowed_aggregate_plans + } + wm_post_parts: List[str] = [] + for fp in planned_query.filters_by_phase: + if fp.phase != Phase.POST or fp.expression is None: + continue + rendered = self._render_filter_for_outer_wrapper( + key=fp.expression.value_key, + slot_by_key=slot_by_key, + cross_model_agg_slot_to_cm=wm_slot_to_cte, + aliases_by_slot_id=aliases_by_slot_id, + ) + if isinstance(rendered, (exp.And, exp.Or)): + rendered = exp.Paren(this=rendered) + wm_post_parts.append(rendered.sql(dialect=self.dialect)) + if wm_post_parts: + connector = "\nAND " if outer_where_filters else "\nWHERE " + combined_select_sql += connector + _SQL_AND_JOINER.join(wm_post_parts) + + # DEV-1450 stage 7b.15e (C2): a transform layer over a cross-model + # aggregate (``cumsum(customers.avg_score:avg)``) runs on TOP of the + # combined cross-model result — the combined SELECT becomes the base + # CTE and the window step CTEs / outer wrap are layered above it. + if planned_query.transform_layers: + if wm_ctes: + # Unreachable today — guard G4 rejects a windowed measure that + # coexists with a transform — but the combined SELECT already + # projects/joins the _wm_ CTEs, which this prelude omits. Fail + # loudly so lifting G4 (DEV-1504) can't silently emit a statement + # referencing undefined _wm_ CTEs. + raise NotImplementedError( + "DEV-1714 Stage 10: a windowed measure combined with a " + "transform layer is not supported (guarded at plan time by " + "G4); the cross-model transform chain does not carry `_wm_` " + "CTEs.", + ) + return self._render_cross_model_transform_chain( + prelude_ctes=[("_base", base_cte_sql)] + cm_ctes, + combined_select_sql=combined_select_sql, + planned_query=planned_query, + slots_by_id=slots_by_id, + combined_aliases_by_slot_id=combined_aliases_by_slot_id, + source_relation=source_relation, + ) + + all_ctes = [("_base", base_cte_sql)] + cm_ctes + wm_ctes + [("_combined", combined_select_sql)] + + # Stitch the WITH chain together. Inner CTEs first; the final + # ``_combined`` is the outermost FROM target. + cte_strs = [f"{name} AS (\n{sql}\n)" for name, sql in all_ctes[:-1]] + sql = f"WITH {', '.join(cte_strs)}\n{combined_select_sql}" + + # ORDER BY / LIMIT / OFFSET: emitted at the combined SELECT + # level. ORDER BY columns must be qualified — ``_base`` columns + # use ``_base."..."``, cross-model columns use the bare alias + # (only present on one side). + # DEV-1712 / DEV-1495 bug 2: hidden (order-only) cross-model aggregates + # are trimmed from the projection above, so their ORDER BY term must be + # CTE-qualified (``_cm_*.""``) rather than the bare + # combined-SELECT alias. + hidden_cte_order_refs: Dict[str, str] = {} + for plan in planned_query.cross_model_aggregate_plans: + # Only CMAs actually trimmed from the projection (hidden + no + # transform chain) need the CTE-qualified ORDER BY reference. + if not (plan.hidden and not planned_query.transform_layers): + continue + _canon = canonical_alias_for_plan[plan.aggregate_slot_id] + _agg_col = agg_col_alias_for_plan[plan.aggregate_slot_id] + _cte = _cte_name_from_alias("_cm_", _canon) + hidden_cte_order_refs[plan.aggregate_slot_id] = ( + f'{_cte}.{self._quote_ident(_agg_col)}' + ) + # DEV-1733: same treatment for a hidden (order-only) WINDOWED aggregate + # trimmed from the combined projection above — reference its ``_wm_`` + # CTE column rather than a bare alias the SELECT no longer emits. + for plan in planned_query.windowed_aggregate_plans: + if not (plan.hidden and not planned_query.transform_layers): + continue + hidden_cte_order_refs[plan.aggregate_slot_id] = ( + f'{wm_cte_name_for_plan[plan.aggregate_slot_id]}.' + f'{self._quote_ident(wm_agg_col_for_plan[plan.aggregate_slot_id])}' + ) + order_sql = self._build_combined_order_by_sql( + planned_query=planned_query, + slots_by_id=slots_by_id, + cma_slot_ids=cma_slot_ids, + cm_alias_for_plan=canonical_alias_for_plan, + # DEV-1714: windowed slots are referenced bare in the combined ORDER + # BY — they surface as a projected combined-SELECT column (from their + # ``_wm_`` CTE), so a ``_base.`` qualifier would dangle. + bare_order_slot_ids=set(order_only_local_ids) | windowed_slot_ids, + outer_composite_aliases=outer_composite_order_alias_by_sid, + outer_composite_expressions=outer_composite_order_expressions, + hidden_cte_order_refs=hidden_cte_order_refs, + ) + if order_sql: + sql += "\n" + order_sql + if planned_query.limit is not None: + sql += f"\nLIMIT {planned_query.limit}" + if planned_query.offset is not None: + sql += f"\nOFFSET {planned_query.offset}" + + # Outer projection trim — the inner already projects the public + # list in declared order, so the trim is normally a no-op. Skip + # the trim machinery here because the legacy path goes through + # an EnrichedQuery-driven ``_apply_outer_projection_trim`` that + # we don't have on the new side. Future slices may re-enable. + return sql + + def _render_cross_model_transform_chain( + self, + *, + prelude_ctes: List[Tuple[str, str]], + combined_select_sql: str, + planned_query, + slots_by_id: Dict[str, Any], + combined_aliases_by_slot_id: Dict[str, List[str]], + source_relation: str, + ) -> str: + """Render window-transform layers over a cross-model combined result. + + DEV-1450 stage 7b.15e (C2). The combined cross-model SELECT becomes the + ``base`` CTE; window step CTEs (``cumsum`` / ``lag`` / ``lead`` / + ``rank`` …) are layered above it exactly like the local transform path + in ``generate_from_planned``, then an outer wrap projects the public + slots in user order and applies ORDER BY / LIMIT / OFFSET. + + ``time_shift`` / ``consecutive_periods`` over a cross-model aggregate + re-aggregate the *source* and are out of slice scope — they raise. + """ + for layer in planned_query.transform_layers: + if layer.op in ("time_shift", "consecutive_periods"): + raise NotImplementedError( + f"DEV-1450 stage 7b.15e: self-join transform op " + f"{layer.op!r} is not yet rendered in a query that also has " + f"a cross-model aggregate (window transforms such as cumsum " + f"/ lag / lead / rank are). Factor the temporal transform " + f"(or change / change_pct, which desugar to time_shift) " + f"into an earlier stage.", + ) + + ctes: List[Tuple[str, str]] = list(prelude_ctes) + [ + ("base", combined_select_sql), + ] + aliases_by_slot_id: Dict[str, List[str]] = { + sid: list(a) for sid, a in combined_aliases_by_slot_id.items() + } + slot_id_by_key: Dict[Any, str] = { + s.key: s.id for s in slots_by_id.values() + } + available_alias_by_slot_id: Dict[str, str] = { + sid: a[0] for sid, a in aliases_by_slot_id.items() if a + } + + # Window-transform Kahn batches (one step CTE per ready batch). + pending_layers = list(planned_query.transform_layers) + step_num = 0 + while pending_layers: + ready: list = [] + not_ready: list = [] + for layer in pending_layers: + if self._transform_layer_deps_ready( + layer=layer, + slots_by_id=slots_by_id, + slot_id_by_key=slot_id_by_key, + available_alias_by_slot_id=available_alias_by_slot_id, + ): + ready.append(layer) + else: + not_ready.append(layer) + if not ready: + pending_ops = [layer.op for layer in pending_layers] + raise RuntimeError( + f"DEV-1450 stage 7b.15e: cross-model transform layer " + f"dependencies could not be resolved; pending ops: " + f"{pending_ops!r}.", + ) + step_num += 1 + step_name = f"step{step_num}" + prev_cte = ctes[-1][0] + carry_aliases_sorted = sorted( + a for aliases in aliases_by_slot_id.values() for a in aliases + ) + step_parts = [self._quote_ident(a) for a in carry_aliases_sorted] + for layer in ready: + for slot_id in layer.slot_ids: + slot = slots_by_id[slot_id] + alias = ( + slot.public_aliases[0] + if slot.public_aliases + else slot.declared_name + ) + full_alias = f"{source_relation}.{alias}" + window_sql = self._render_window_transform_sql( + slot=slot, + slots_by_id=slots_by_id, + slot_id_by_key=slot_id_by_key, + available_alias_by_slot_id=available_alias_by_slot_id, + planned_query=planned_query, + ) + if slot.type is not None: + window_sql = _wrap_cast_for_type( + self._parse(window_sql), slot.type, + ).sql(dialect=self.dialect) + step_parts.append(f'{window_sql} AS {self._quote_ident(full_alias)}') + aliases_by_slot_id.setdefault(slot_id, []).append(full_alias) + available_alias_by_slot_id.setdefault(slot_id, full_alias) + step_sql = ( + "SELECT\n " + + _SQL_COL_SEP.join(step_parts) + + f"\nFROM {prev_cte}" + ) + ctes.append((step_name, step_sql)) + pending_layers = not_ready + + # Materialise any projected POST-phase ArithmeticKey / ScalarCallKey + # slot a window layer didn't render (``cumsum(x) + 1``-style combos). + from slayer.core.keys import ( + ArithmeticKey as _ArithKey, + ScalarCallKey as _ScalarKey, + TransformKey as _TKey, + ) + unmaterialised: list = [] + for cslot in planned_query.combined_expression_slots: + if isinstance(cslot.key, _TKey): + continue + if cslot.id in aliases_by_slot_id: + continue + if isinstance(cslot.key, (_ArithKey, _ScalarKey)): + unmaterialised.append(cslot) + if unmaterialised: + step_num += 1 + step_name = f"step{step_num}" + prev_cte = ctes[-1][0] + carry_aliases_sorted = sorted( + a for aliases in aliases_by_slot_id.values() for a in aliases + ) + step_parts = [self._quote_ident(a) for a in carry_aliases_sorted] + for cslot in unmaterialised: + alias = ( + cslot.public_aliases[0] + if cslot.public_aliases + else cslot.declared_name + ) + full_alias = f"{source_relation}.{alias}" + rendered = self._render_value_key_against_aliases( + key=cslot.key, + slot_id_by_key=slot_id_by_key, + available_alias_by_slot_id=available_alias_by_slot_id, + ) + expr_sql = rendered.sql(dialect=self.dialect) + if cslot.type is not None: + expr_sql = _wrap_cast_for_type( + self._parse(expr_sql), cslot.type, + ).sql(dialect=self.dialect) + step_parts.append(f'{expr_sql} AS {self._quote_ident(full_alias)}') + aliases_by_slot_id.setdefault(cslot.id, []).append(full_alias) + available_alias_by_slot_id.setdefault(cslot.id, full_alias) + step_sql = ( + "SELECT\n " + + _SQL_COL_SEP.join(step_parts) + + f"\nFROM {prev_cte}" + ) + ctes.append((step_name, step_sql)) + + final_cte = ctes[-1][0] + inner_sorted = sorted( + a for aliases in aliases_by_slot_id.values() for a in aliases + ) + inner_sql = ( + "SELECT\n " + + _SQL_COL_SEP.join(self._quote_ident(a) for a in inner_sorted) + + f"\nFROM {final_cte}" + ) + cte_clause = ( + _SQL_WITH + + ",\n".join(f"{name} AS (\n{sql}\n)" for name, sql in ctes) + ) + chain_sql = f"{cte_clause}\n{inner_sql}" + + post_filter_conditions = self._render_post_phase_filter_conditions( + planned_query=planned_query, + slot_id_by_key=slot_id_by_key, + available_alias_by_slot_id=available_alias_by_slot_id, + ) + if post_filter_conditions: + chain_sql = ( + f"SELECT *\nFROM (\n{chain_sql}\n) AS _filtered" + f"\nWHERE {_SQL_AND_JOINER.join(post_filter_conditions)}" + ) + + public_aliases_user_order: list[str] = [] + outer_alias_index: Dict[str, int] = {} + for sid in planned_query.projection: + slot = slots_by_id[sid] + if slot.hidden: + continue + all_aliases = aliases_by_slot_id.get(sid, []) + if not all_aliases: + continue + idx = outer_alias_index.setdefault(sid, 0) + alias = ( + all_aliases[idx] if idx < len(all_aliases) else all_aliases[-1] + ) + outer_alias_index[sid] = idx + 1 + public_aliases_user_order.append(alias) + return self._emit_planned_outer_wrap( + chain_sql=chain_sql, + public_aliases=public_aliases_user_order, + planned_query=planned_query, + slots_by_id=slots_by_id, + available_alias_by_slot_id=available_alias_by_slot_id, + ) + + def _canonical_cross_model_alias( + self, + *, + source_relation: str, + key, + ) -> str: + """Build the canonical result-key alias for a cross-model + aggregate, IGNORING any user-declared ``name``. + + Used for CTE name + CTE projection alias so per-plan CTEs are + stable under renames and so multi-alias same-key slots (C13) + produce ONE shared CTE. The user-facing alias remapping + happens at the combined SELECT level via ``... AS + ""``. + + Format: ``..``. + ``canonical_agg_name`` collapses ``*`` to a leading ``_`` + (``*:count`` → ``_count``) per the result-key contract. + """ + from slayer.core.refs import canonical_agg_name + + path = getattr(key.source, "path", ()) + # Handle ColumnKey (``leaf``), ColumnSqlKey (``column_name`` — derived + # column source, almost universal for filtered-local measures whose + # ``Column.sql`` differs from ``Column.name``), and StarKey (no + # ``leaf`` / ``column_name`` → collapse to ``*``). Mirrors + # ``_aggregate_alias`` in ``cross_model_planner.py``. + measure_name = ( + getattr(key.source, "leaf", None) + or getattr(key.source, "column_name", None) + or "*" + ) + # DEV-1450 stage 7b.13: include kwarg suffix in cross-model + # alias so two distinct parametric aggs (``percentile(p=0.5)`` + # vs ``p=0.95``) produce distinct CTE names and column aliases. + # Legacy enrichment at ``query_engine.py:2160`` drops the + # signature suffix entirely -- a known legacy bug that + # produces ALIAS COLLISION when the same query has multiple + # parametric aggs against the same target.column. The new + # pipeline preserves slot identity here for correctness; + # parity tests for parametric cross-model aggs assert + # structural shape rather than bit-identical SQL. + canonical = canonical_agg_name( + measure_name=measure_name, + aggregation_name=key.agg, + agg_args=[agg_kwarg_canonical_str(a) for a in key.args] or None, + agg_kwargs={ + k: agg_kwarg_canonical_str(v) for k, v in key.kwargs + } or None, + ) + if path: + return f"{source_relation}." + ".".join(path) + f".{canonical}" + return f"{source_relation}.{canonical}" + + def _public_aliases_for_cross_model_agg( + self, + *, + slot, + source_relation: str, + canonical_alias: str, + ) -> List[str]: + """User-facing combined-SELECT aliases for this cross-model slot. + + Each declared ``name`` on the slot (P4 / C13) surfaces as one + entry. When no user names are declared we return a single + entry equal to ``canonical_alias`` so the combined SELECT + projects exactly once. The result is always ``. + ``. + """ + if not slot.public_aliases: + return [canonical_alias] + return [f"{source_relation}.{a}" for a in slot.public_aliases] + + def _render_rerooted_cross_model_cte( + self, + *, + plan, + bundle, + host_slots_by_id: Dict[str, Any], + host_source_relation: str, + ) -> Tuple[str, List[Tuple[str, str]], str]: + """Render a cross-model CTE from a nested re-rooted ``PlannedQuery``. + + DEV-1450 stage 7b.15e (C1). The sub-plan is rooted at the TARGET + model (``FROM target + joins``) so it preserves the host dimension + grain — the legacy ``_build_rerooted_enriched`` shape, now driven by + the typed pipeline. Reuses ``generate_from_planned`` to render the + sub-plan exactly like any base query. + + Returns ``(cte_sql, joinback_pairs, agg_col_alias)``: + * ``joinback_pairs`` — ``(host_base_alias, cte_column_alias)`` for the + combined ``LEFT JOIN ON`` (the two sides differ — the host aliases + dims under its own relation; the CTE under the target relation), + * ``agg_col_alias`` — the sub-plan's emitted alias for the aggregate. + """ + sub_plan = plan.rerooted_plan + # DEV-1503 — filtered-local (host-rooted) plans don't change the + # source_model: the sub-plan is rooted at the SAME host the outer + # plan binds against, and ``bundle.source_model`` already IS the + # host. The existing cross-model re-rooted path swaps ``source_model`` + # to the join target (which lives in ``referenced_models``); a + # filtered-local host name won't resolve there, so guard on + # ``cte_root_model`` first. + if plan.cte_root_model is not None: + host_model = bundle.source_model + if host_model is None or host_model.name != plan.cte_root_model: + raise ValueError( + f"Filtered-local CrossModelAggregatePlan " + f"cte_root_model={plan.cte_root_model!r} does not match " + f"the bundle's source model — planner/renderer drift.", + ) + rerooted_bundle = bundle + else: + target_model = bundle.get_referenced_model(plan.target_model) + if target_model is None: + raise ValueError( + f"Re-rooted CrossModelAggregatePlan target " + f"{plan.target_model!r} not in resolved source bundle.", + ) + rerooted_bundle = bundle.model_copy( + update={"source_model": target_model}, + ) + cte_sql = self.generate_from_planned(sub_plan, bundle=rerooted_bundle) + + sub_slots_by_id = { + s.id: s + for s in ( + list(sub_plan.row_slots) + + list(sub_plan.aggregate_slots) + + list(sub_plan.combined_expression_slots) + ) + } + target_relation = sub_plan.source_relation + + joinback_pairs: List[Tuple[str, str]] = [] + for host_sid, sub_sid in plan.rerooted_grain_pairs: + host_slot = host_slots_by_id.get(host_sid) + sub_slot = sub_slots_by_id.get(sub_sid) + if host_slot is None or sub_slot is None: + continue + host_alias = self._full_alias_for_slot( + slot=host_slot, + source_relation=host_source_relation, + alias_index={}, + ) + cte_alias = self._full_alias_for_slot( + slot=sub_slot, + source_relation=target_relation, + alias_index={}, + ) + joinback_pairs.append((host_alias, cte_alias)) + + agg_slot = sub_slots_by_id.get(plan.rerooted_agg_slot_id) + if agg_slot is None: + raise RuntimeError( + f"Re-rooted plan aggregate slot " + f"{plan.rerooted_agg_slot_id!r} not found in sub-plan.", + ) + agg_col_alias = self._full_alias_for_slot( + slot=agg_slot, + source_relation=target_relation, + alias_index={}, + ) + return cte_sql, joinback_pairs, agg_col_alias + + def _render_cross_model_cte( # NOSONAR(S3776) — single conceptual unit: shared-grain projection + GROUP BY classification + aggregate reroot (source / args / kwargs) + first/last ranked-subquery wrap + target-model-filter qualification + WHERE/HAVING routing. Each block is interdependent state for the same CTE; splitting forces the same cross-cutting state through helpers without simplifying anything. + self, + *, + plan, + agg_slot, + full_agg_alias: str, + bundle, + planned_query, + slots_by_id: Dict[str, Any], + base_projection_ids: Set[str], + ) -> Tuple[str, List[str]]: + """Render one ``_cm_<...>`` CTE body and return its SQL + + shared-grain alias list (for the outer ``LEFT JOIN ON`` clause). + + The CTE is rooted at the terminal target model (legacy + rerooted shape). Shared-grain slots whose key path is a prefix + of the target_path participate as both projection and GROUP BY + keys; slots with empty path (host-local dims) are excluded + since the legacy CROSS JOINs in that case. + + Filter routing reads ``plan.where_filter_ids`` / + ``plan.having_filter_ids`` / ``plan.target_model_filters`` so + the CTE renders each route without re-classifying. + """ + from slayer.core.enums import TimeGranularity + from slayer.core.keys import ( + ColumnKey, + ColumnSqlKey, + Phase, + TimeTruncKey, + ) + + target_model_name = plan.target_model + target_model = bundle.get_referenced_model(target_model_name) + if target_model is None: + raise ValueError( + f"CrossModelAggregatePlan target {target_model_name!r} " + f"not in resolved source bundle.", + ) + target_relation = target_model_name + + target_path = tuple(getattr(agg_slot.key.source, "path", ())) + + # Shared grain: project + GROUP BY any host slot whose key path + # matches a prefix of target_path. Local-only slots (path=()) + # don't participate at the CTE level; the legacy CROSS JOINs in + # that case so the host's GROUP BY broadcasts the global agg. + # + # Codex HIGH fold-in: the planner's ``shared_grain_slots`` + # currently includes ANY host ROW slot on the target path, + # including FILTER-ONLY slots that exist in the registry but + # are not in the host's public projection. A filter-only slot + # would over-GROUP the CTE and produce a join-back key that + # ``_base`` never projects (so the outer ``LEFT JOIN _cm_* ON + # _base."" = _cm_*.""`` references a missing + # column on the left side). Intersect with the host's actual + # projection ids so only projected slots flow into the CTE. + cte_select_columns: List[exp.Expression] = [] + # DEV-1728: two GROUP-BY lists — ``cte_group_by`` is the OUTER GROUP BY + # (an alias ref ``_val_`` for a first/last-materialised crossing + # grain, the raw expression otherwise); ``cte_partition_exprs`` is the + # ranked-subquery PARTITION BY, always the RAW expression (valid inside + # the subquery where the crossed join is bound). They are identical for + # every non-first/last query and every non-crossing grain. + cte_group_by: List[exp.Expression] = [] + cte_partition_exprs: List[exp.Expression] = [] + shared_grain_aliases: List[str] = [] + # DEV-1701: join paths crossed by a shared-grain derived TIME dimension's + # expanded ``Column.sql``. Collected during the loop (which runs before + # the CTE scope's join set is assembled) and merged into it below. + shared_grain_join_paths: List[Tuple[str, ...]] = [] + # DEV-1728: first/last grain materialisations (``_val_`` projections + # to inject INTO the ranked subquery for crossing derived grains). The + # generation-wide allocator is hoisted here so grain + source ``_val``s + # share one monotonic sequence. Reserve the target's physical column + # names (Codex F6): the ranked subquery re-exports ``target.*``, so a + # minted ``_val_`` must never collide with a real target column of + # that name — mirrors the host-path reservation in + # ``_build_first_last_base_select``. + cte_allocator = self._gen_allocator or self._new_allocator() + self._reserve_model_column_names(cte_allocator, target_model) + is_first_or_last = agg_slot.key.agg in ("first", "last") + grain_extra_projections: List[Tuple[str, exp.Expression]] = [] + for sid in plan.shared_grain_slots: + if sid not in base_projection_ids: + continue + slot = slots_by_id.get(sid) + if slot is None or slot.phase != Phase.ROW: + continue + key = slot.key + path: Tuple[str, ...] = () + if isinstance(key, ColumnKey): + path = key.path + elif isinstance(key, TimeTruncKey): + path = key.column.path + elif isinstance(key, ColumnSqlKey): + # DEV-1708 / DEV-1728: a plain derived (non-time) dim carries its + # own path; a path-bearing one renders here like any grain, a + # host-local ``path == ()`` one falls through to the CROSS-JOIN + # broadcast below (unchanged). + path = key.path + if not path: + # Local-only host dim — broadcast via CROSS JOIN. + continue + if path != target_path[: len(path)]: + # Off the join path; cross-branch dim doesn't share grain. + continue + # Build the column expression rooted at the target model. + # Single-hop case (path == target_path): bare leaf on target. + # Multi-hop intermediate case (path < target_path): would + # need an inner JOIN on the CTE's body. For 7b.12 we accept + # the single-hop common case and leave intermediate-hop + # shared grain as a follow-up. + if path != target_path: + raise NotImplementedError( + f"DEV-1450 stage 7b.12: shared-grain dimension on an " + f"intermediate hop ({path!r}) of cross-model agg " + f"target_path={target_path!r} not yet rendered in " + f"the typed pipeline. Use the terminal-target path " + f"or pull the dimension to the host base.", + ) + # Build the (untruncated) shared-grain column expression rooted at + # the target relation. A derived (ColumnSqlKey) column — base dim + # or time dimension — expands its Column.sql rooted at the target; + # a base column emits the bare ``target.leaf``. + from slayer.core.keys import ColumnSqlKey as _ColumnSqlKey + + grain_column = key.column if isinstance(key, TimeTruncKey) else key + if isinstance(grain_column, _ColumnSqlKey): + # DEV-1728: a derived (ColumnSqlKey) grain — plain dimension OR + # time dimension — expands its Column.sql rooted at the target and + # renders here. (The DEV-1708 raise for a PLAIN derived grain is + # gone: DEV-1713 fixed the naming half, so the host's dotted alias + # and the CTE join-back now agree.) + expanded_grain_sql = self._expand_derived_column_sql( + source_model=target_model, + source_relation=target_relation, + column_name=grain_column.column_name, + bundle=bundle, + ) + col_expr = self._parse(expanded_grain_sql) + leaf = grain_column.column_name + if not isinstance(key, TimeTruncKey): + # DEV-1728: a PLAIN derived grain is CAST to its declared type + # to match the host base's ``_wrap_cast_for_type`` (a bare + # column ref / TEXT is skipped there and here identically), so + # the join-back compares identically-typed values. A + # TimeTrunc-wrapped grain keeps ``_build_date_trunc``'s own + # temporal shape (no extra cast — parity with the host base). + grain_col = next( + (c for c in target_model.columns + if c.name == grain_column.column_name), + None, + ) + col_expr = _wrap_cast_for_type( + col_expr, grain_col.type if grain_col else None, + ) + # DEV-1701: register every further join the derived grain's + # expanded sql crosses (rooted at the target relation), so the + # CTE's FROM pulls it. Merged into the CTE join set below. + for _p in self._joined_paths_in_sql( + sql_expr=col_expr, source_relation=target_relation, + source_model=target_model, bundle=bundle, + ): + if _p not in shared_grain_join_paths: + shared_grain_join_paths.append(_p) + else: + leaf = grain_column.leaf + col_expr = exp.Column( + this=exp.to_identifier(leaf), + table=exp.to_identifier(target_relation), + ) + if isinstance(key, TimeTruncKey): + col_expr = self._build_date_trunc( + col_expr=col_expr, + granularity=TimeGranularity(key.granularity), + ) + # Host-side join-back uses the SAME alias as the host's + # base projection. For path-bearing slots that's the dotted + # form (e.g. ``orders.customers.created_at``); the host's + # ``_build_base_select_for_planned`` already aliases that + # way for joined ROW slots. + host_alias = planned_query.source_relation + "." + ".".join(path) + f".{leaf}" + # DEV-1728 Law 2: for a first/last aggregate the CTE's FROM is a + # ROW_NUMBER-ranked subquery that re-exports only ``target.*`` + rank + # columns. A grain whose expression CROSSES a join references a table + # bound ONLY inside that subquery, so the outer SELECT / GROUP BY + # cannot name it — materialise it as a ``_val_`` projection inside + # the subquery, group the outer SELECT on the alias, and keep the RAW + # expression for PARTITION BY (evaluated where the join is bound). A + # target-local grain (no crossing) needs no materialisation — it is + # re-exported by ``target.*``. + grain_crosses = is_first_or_last and bool( + self._joined_paths_in_sql( + sql_expr=col_expr, source_relation=target_relation, + source_model=target_model, bundle=bundle, + ) + ) + if grain_crosses: + val_alias = cte_allocator.allocate_val() + grain_extra_projections.append((val_alias, col_expr.copy())) + alias_ref = exp.column(val_alias) + cte_select_columns.append(alias_ref.copy().as_(host_alias)) + cte_group_by.append(alias_ref.copy()) + cte_partition_exprs.append(col_expr.copy()) + else: + cte_select_columns.append(col_expr.copy().as_(host_alias)) + cte_group_by.append(col_expr.copy()) + cte_partition_exprs.append(col_expr.copy()) + shared_grain_aliases.append(host_alias) + + # Aggregate column: synthesise an EnrichedMeasure ROOTED at the + # target so ``_build_agg`` resolves the source column on the + # right model (including ``column_filter_key`` CASE-WHEN). + # Mutate a copy of the key with ``source.path=()`` so the + # synthesise helper's local branch fires without re-checking + # path-based deferrals. DEV-1450 stage 7b.13: also reroot + # ``ColumnKey`` kwargs whose path matches the source's join path + # -- a user-qualified kwarg like + # ``customers.revenue:corr(other=customers.region_id)`` arrives + # here with both source and ``other`` rooted at ``("customers",)``. + # Stripping the prefix in lockstep means the synth helper's + # path-validation invariant (``kwarg.path == source.path``) holds. + # Re-root the aggregate SOURCE and ALL embedded refs (positional args + # AND column-valued kwargs) from the host's coordinate system into the + # target's local scope in one symmetric pass (DEV-1707). Covers a + # derived (ColumnSqlKey) source like ``customers.net:sum`` — otherwise + # the host-rooted derived key renders against the wrong alias inside + # the CTE — and the DEV-1476(c) explicit time arg + # ``customers.amount:last(customers.signup_at)``, whose positional arg + # must strip the host prefix in lockstep with the source so + # ``_resolve_explicit_time_col`` qualifies the time column under the + # target relation. ``column_filter_key`` rides through unchanged + # (owner-anchored, invariant under reroot). + cross_model_path = getattr(agg_slot.key.source, "path", ()) + local_agg_key = reroot_aggregate_key( + agg_slot.key, target_path=cross_model_path, + ) + # The local_agg_key was built from the target's own column. + # column_filter_key (if set) carries the canonical filter SQL + # from the target's Column.filter — the synth helper qualifies + # bare refs against target_model. + local_slot = agg_slot.model_copy(update={"key": local_agg_key}) + + # DEV-1708 Law 1: every expression rendered into this CTE enters through + # a single ScopeFrame rooted at the target relation. ``resolve`` anchors + # each ref and REGISTERS the joins it crosses into ``cte_scope.join_paths`` + # as a side effect — the CTE's FROM is built from that set below, so a + # crossed join can never be forgotten (replaces the ad-hoc + # ``_add_cte_join_paths`` closure + per-carrier collectors). The scope + # shares the generation-wide allocator (``cte_allocator``, hoisted above + # the grain loop) so ``_val_`` materialisation names (Law 2) are + # unique across the host base, the grain projections, and every CTE. + cte_scope = ScopeFrame( + scope_id=cte_allocator.next_scope_id(target_relation), + root_model=target_model, + root_relation=target_relation, + bundle=bundle, + dialect=self._dialect, + allocator=cte_allocator, + ) + # DEV-1701: merge the shared-grain derived-TIME-dim crossed joins + # collected in the loop above. + for _p in shared_grain_join_paths: + cte_scope.join_paths.add(_p) + # DEV-1526: register the rerooted aggregate SOURCE's crossed joins (a + # derived ``ColumnSqlKey`` source like ``customers_v2.deep_pop:sum`` whose + # ``Column.sql`` = ``regions.population`` must pull the customers_v2 → + # regions join into the CTE). Registration only — the render spec + # re-expands the source itself. + if isinstance(local_agg_key.source, ColumnSqlKey): + cte_scope.resolve(local_agg_key.source) + # DEV-1476(c) / Codex F1: register every positional ARG's crossed joins + # (the explicit first/last time arg may itself be a derived column whose + # sql crosses a further join — its ranking ORDER BY needs that join). + for _arg in local_agg_key.args: + if isinstance(_arg, (ColumnKey, ColumnSqlKey)): + cte_scope.resolve(_arg) + # DEV-1527 (cross-model remainder): resolve each column-ref KWARG through + # the scope — anchors the expanded expression AND registers its join — + # then embed it as a trusted ``kind="expr"`` into the render spec so a + # derived kwarg emits its expanded sql (``regions.weight``) instead of a + # bare, non-existent ``customers_v2.deep_weight``. + cte_resolved_kwargs: "Dict[str, ResolvedAggKwarg]" = {} + for _kname, _kval in local_agg_key.kwargs: + if isinstance(_kval, (ColumnKey, ColumnSqlKey)): + cte_resolved_kwargs[_kname] = ResolvedAggKwarg( + kind="expr", value=cte_scope.resolve(_kval), + ) + + synth = self._build_agg_render_spec_from_planned( + slot=local_slot, + key=local_agg_key, + source_model=target_model, + source_relation=target_relation, + full_alias=full_agg_alias, + bundle=bundle, + resolved_agg_kwargs=cte_resolved_kwargs or None, + ) + + # DEV-1476 bug (c): for first/last aggregates the FROM must be a + # ROW_NUMBER-ranked subquery so the ``MAX(CASE WHEN _last_rn = 1 + # THEN col END)`` expression has a ranking column. The local + # first/last path (``_build_first_last_base_select``) wraps via + # ``_build_ranked_subquery_from_planned``; mirror that here for + # the cross-model CTE. + # + # Codex round 2: when no explicit positional time arg was + # supplied, fall back to the target model's + # ``default_time_dimension`` (qualified under the target + # relation). If even that is unset, raise the standard + # "first/last requires a ranking time column" error rather than + # silently emitting an agg_expr that references a non-existent + # ``_first_rn`` / ``_last_rn`` column. (``is_first_or_last`` is computed + # once above the grain loop from ``agg_slot.key.agg`` — reroot preserves + # the aggregation name — so the grain materialisation and this branch + # agree.) + time_col_sql: Optional[str] = synth.time_column + if is_first_or_last and time_col_sql is None: + if target_model.default_time_dimension: + time_col_sql = ( + f"{target_relation}.{target_model.default_time_dimension}" + ) + else: + raise ValueError( + f"first/last aggregation requires a ranking time column " + f"(an explicit positional time arg, or the target " + f"model's default_time_dimension); none is resolvable " + f"for cross-model aggregate on target " + f"{target_model_name!r}." + ) + # WHERE: target-model-filters (qualified bare-identifier refs + # so ``deleted_at IS NULL`` becomes ``customers.deleted_at IS + # NULL`` to match the legacy enrichment's filter-column + # resolution) + host filters routed to WHERE. Computed up-front + # so the first/last branch can push them INSIDE the ranked + # subquery — otherwise rows excluded by a filter could still + # win ``_last_rn = 1`` and yield NULL aggregates. + # DEV-1494: join paths the CTE's own filters cross — the target measure's + # ``Column.filter`` and the target-model filters — registered into the + # CTE scope (Law 1). Each ``_cm_*`` CTE is an isolated per-(target, grain) + # computation, so adding these joins to ITS FROM affects only this + # measure (not siblings) — it resolves the filter's refs without the + # cross-measure cardinality concern DEV-1503 owns. Free-SQL predicates + # keep the quote-tolerant dual-scan of ``_filter_join_paths`` (raw + + # inline-expanded — the DEV-1494/dedup contract) while writing into the + # single ``cte_scope.join_paths`` set. + def _register_filter_join_paths(sql_text: Optional[str]) -> None: + if not sql_text: + return + for p in self._filter_join_paths( + sql=sql_text, source_relation=target_relation, + source_model=target_model, bundle=bundle, + ): + cte_scope.join_paths.add(p) + + if local_agg_key.column_filter_key is not None: + _register_filter_join_paths(local_agg_key.column_filter_key.canonical_sql) + + where_parts: List[exp.Expression] = [] + for filter_text in plan.target_model_filters: + # DEV-1450 #4b / DEV-1494: a target model filter referencing a + # non-trivial derived column on the target (bare OR a dotted ref to a + # derived column on a joined model) is inline-expanded; base-only + # filters keep the AST bare-ref qualification. The crossed join is + # pulled into this CTE's FROM via ``cte_scope.join_paths``. + _register_filter_join_paths(filter_text) + qualified = self._render_mode_a_predicate( + sql=filter_text, + source_model=target_model, + source_relation=target_relation, + bundle=bundle, + qualify_fallback=lambda s: self._qualify_column_filter_sql( + canonical_sql=s, + source_relation=target_relation, + source_model=target_model, + ), + ) + if not qualified: + continue + try: + where_parts.append(self._parse_predicate(qualified)) + except Exception: + raise ValueError( + f"Target model filter on {target_model_name!r} could " + f"not be parsed: {filter_text!r}", + ) + # DEV-1708 / Codex F4: pre-pass — walk the FULL ValueKey tree of every + # routed WHERE and HAVING filter (nested arithmetic / boolean / IN + # operands, aggregate leaves' source + args + kwargs + column_filter, + # derived ColumnSqlKey refs) and register the joins they cross into the + # CTE scope BEFORE the FROM is built. HAVING is rendered later (it needs + # the ranked-subquery rn maps), so its joins would otherwise register + # too late to reach the FROM. + self._register_routed_filter_joins( + planned_query=planned_query, + filter_ids=list(plan.where_filter_ids) + list(plan.having_filter_ids), + target_relation=target_relation, + target_model=target_model, + bundle=bundle, + scope=cte_scope, + target_path=target_path, + ) + cte_where = self._collect_routed_filters( + planned_query=planned_query, + filter_ids=plan.where_filter_ids, + target_relation=target_relation, + target_model=target_model, + bundle=bundle, + ) + if cte_where is not None: + where_parts.append(cte_where) + combined_where: Optional[exp.Expression] = None + if where_parts: + combined_where = ( + exp.and_(*where_parts) if len(where_parts) > 1 else where_parts[0] + ) + + # FROM: target table directly, OR a ROW_NUMBER-ranked subquery for + # first/last. Build the ranked subquery FIRST so its rank-column + # maps — including the filtered ``_last_rn_fN`` / ``_match_fN`` + # columns emitted when the measure's source column carries a + # ``Column.filter`` — can be threaded into ``_build_agg``. Without + # the filtered maps the agg references a bare ``_last_rn`` the + # subquery never projects. WHERE is pushed INSIDE so RN is computed + # over the filtered row set; otherwise a filtered-out row could win + # ``_last_rn = 1`` and the ``MAX(CASE WHEN _last_rn = 1 ...)`` + # aggregate would return NULL. + cte_join_paths = cte_scope.join_paths.as_list() + if cte_join_paths: + target_from, cte_base_joins = self._build_from_and_joins( + source_model=target_model, source_relation=target_relation, + joined_paths=cte_join_paths, bundle=bundle, + ) + else: + target_from = self._build_from_clause_from_planned( + source_model=target_model, source_relation=target_relation, + ) + cte_base_joins = [] + ranked_from: Optional[exp.Expression] = None + cte_value_alias_by_sql: Dict[str, str] = {} + if is_first_or_last: + assert time_col_sql is not None # narrowed by the guard above + # DEV-1708 Law 2 (DEV-1702 B2, forward variant): the ranked subquery + # re-exports only ``target.*`` + rank columns. If the first/last + # SOURCE value crosses a join, the crossing ref must be materialised + # as a ``_val_`` projection INSIDE the subquery and the outer + # aggregate rewritten to reference the alias — otherwise the outer + # ``MAX(CASE WHEN _last_rn = 1 THEN END)`` references a + # table bound only inside the subquery (out of scope). The FILTER + # refs are consumed inside the subquery (the ``_last_rn_fN`` / + # ``_match_fN`` rank columns) and need no outer alias. A LOCAL source + # value is already covered by ``target.*`` — no materialisation. + outer_synth = synth + # DEV-1728: seed with the crossing-grain ``_val_`` projections + # collected in the grain loop, so a first/last aggregate grouped by a + # crossing derived grain materialises that grain INSIDE the subquery + # too (the outer SELECT / GROUP BY reference the alias). + extra_projections: List[Tuple[str, exp.Expression]] = list( + grain_extra_projections, + ) + if synth.sql: + # DEV-1709: materialise the RESOLVED value (qualified + + # ``Column.type`` inner CAST for non-bare expressions) and + # key the alias map by that resolved text — mirrors the + # host-path materialisation in + # ``_build_first_last_base_select`` so typed non-bare + # sources keep ``MAX(CASE ... THEN CAST(x AS t) END)`` + # semantics inside the CTE too. + value_expr = self._resolve_sql( + sql=synth.sql, name=synth.name, + model_name=synth.model_name, type=synth.column_type, + ) + if self._joined_paths_in_sql( + sql_expr=value_expr, source_relation=target_relation, + source_model=target_model, bundle=bundle, + ): + val_alias = cte_allocator.allocate_val() + extra_projections.append((val_alias, value_expr)) + outer_synth = synth.model_copy(update={"sql": val_alias}) + # A HAVING on this same aggregate must reference the alias, + # not the raw crossing ref (out of scope in the outer SELECT). + cte_value_alias_by_sql[ + value_expr.sql(dialect=self.dialect) + ] = val_alias + ranked_from, rn_suffix_map, filtered_rn_map, filtered_match_map = ( + self._build_ranked_subquery_from_planned( + source_relation=target_relation, + default_time_col_sql=time_col_sql, + partition_exprs=list(cte_partition_exprs), + extra_projections=extra_projections, + synth_specs=[synth], + from_clause=target_from, + base_joins=cte_base_joins, + where_clause=combined_where, + ) + ) + agg_expr, is_agg = self._build_agg( + outer_synth, + rn_suffix_map=rn_suffix_map, + default_time_col=time_col_sql, + filtered_rn_map=filtered_rn_map, + filtered_match_map=filtered_match_map, + ) + else: + agg_expr, is_agg = self._build_agg(synth) + if is_agg: + agg_expr = _wrap_cast_for_type(agg_expr, agg_slot.type) + cte_select_columns.append(agg_expr.copy().as_(full_agg_alias)) + + # Assemble the CTE Select now that every projected column (shared + # grain + aggregate) is in ``cte_select_columns``. + cte_select = exp.Select() + for col in cte_select_columns: + cte_select = cte_select.select(col) + if is_first_or_last: + assert ranked_from is not None + cte_select = cte_select.from_(ranked_from) + else: + cte_select = cte_select.from_(target_from) + for join_expr, on_expr, join_type in cte_base_joins: + cte_select = cte_select.join( + join_expr, on=on_expr, join_type=join_type, + ) + if combined_where is not None: + cte_select = cte_select.where(combined_where) + + if cte_group_by: + for gb in cte_group_by: + cte_select = cte_select.group_by(gb) + + # DEV-1501 Group A.3: routed HAVING for cross-model first/last + # must use the SAME rn-based aggregate the CTE projects. Build a + # ``FirstLastRenderState`` carrying the rn maps + the single + # projected aggregate's full alias so HAVING's synth rebuild + # binds to the right ``_first_rn`` / ``_last_rn{suffix}`` / + # ``_last_rn_fN`` column (instead of a placeholder alias whose + # ``filtered_rn_map`` lookup misses and silently degrades to + # bare ``_last_rn`` + raw ``filter_sql``). + cm_first_last_state: Optional[FirstLastRenderState] = None + if is_first_or_last: + cm_first_last_state = FirstLastRenderState( + rn_suffix_map=dict(rn_suffix_map), + default_time_col_sql=time_col_sql, + filtered_rn_map=dict(filtered_rn_map), + filtered_match_map=dict(filtered_match_map), + agg_synth_alias=full_agg_alias, + value_alias_by_sql=dict(cte_value_alias_by_sql), + ) + cte_having = self._collect_routed_filters( + planned_query=planned_query, + filter_ids=plan.having_filter_ids, + target_relation=target_relation, + target_model=target_model, + bundle=bundle, + first_last_state=cm_first_last_state, + ) + if cte_having is not None: + cte_select = cte_select.having(cte_having) + + cte_sql = cte_select.sql(dialect=self.dialect, pretty=True) + return cte_sql, shared_grain_aliases + + def _register_routed_filter_joins( # NOSONAR(S3776) — a cohesive recursive ValueKey tree-walk dispatcher (the heavy AggregateKey arm is already extracted to _register_agg_key_joins); the remaining branches are the closed-union dispatch contract, mirroring the sibling walkers _value_key_join_paths / _collect_base_aux_slot_ids in this file. + self, + *, + planned_query, + filter_ids: List[str], + target_relation: str, + target_model, + bundle, + scope: ScopeFrame, + target_path: Tuple[str, ...], + ) -> None: + """DEV-1708 / Codex F4 — register the joins crossed by every routed + WHERE/HAVING filter into ``scope.join_paths``, walking the FULL + ``ValueKey`` tree so nested arithmetic/boolean/IN operands and aggregate + leaves (source + positional args + column-ref kwargs + ``column_filter``) + all contribute BEFORE the CTE FROM is assembled. + + Registration only — the render passes (``_collect_routed_filters`` for + WHERE, and the HAVING render below) emit the SQL themselves. The scope's + ``resolve`` anchors each typed leaf at the target relation and records + the path it crosses; free-SQL ``column_filter`` predicates keep the + quote-tolerant dual-scan of ``_filter_join_paths``. + """ + from slayer.core.keys import ( + AggregateKey, + ArithmeticKey, + BetweenKey, + ColumnKey, + ColumnSqlKey, + InKey, + ScalarCallKey, + ) + + if not filter_ids: + return + wanted = set(filter_ids) + + def _walk(vk) -> None: + if isinstance(vk, (ColumnKey, ColumnSqlKey)): + # Reroot a path-qualified leaf into the target's local scope by + # stripping the CTE's ``target_path`` prefix (NOT the ref's own + # path — a ref one hop past the target keeps its residual so the + # deeper join still registers), then resolve. + local = ( + _reroot_path_ref(vk, target_path=target_path) + if vk.path else vk + ) + scope.resolve(local) + elif isinstance(vk, AggregateKey): + self._register_agg_key_joins( + agg_key=vk, scope=scope, target_relation=target_relation, + target_model=target_model, bundle=bundle, + ) + elif isinstance(vk, ArithmeticKey): + for op in vk.operands: + _walk(op) + elif isinstance(vk, ScalarCallKey): + for a in vk.args: + _walk(a) + elif isinstance(vk, BetweenKey): + _walk(vk.column) + _walk(vk.low) + _walk(vk.high) + elif isinstance(vk, InKey): + _walk(vk.column) + + for fp in planned_query.filters_by_phase: + if fp.id in wanted and fp.expression is not None: + _walk(fp.expression.value_key) + + def _register_agg_key_joins( + self, *, agg_key, scope: ScopeFrame, target_relation: str, + target_model, bundle, + ) -> None: + """Register the joins an aggregate leaf crosses (source + positional + args + column-ref kwargs + ``column_filter``) into ``scope.join_paths`` + — the ``AggregateKey`` arm of ``_register_routed_filter_joins``'s tree + walk, extracted so the walker stays a thin dispatcher (DEV-1708).""" + from slayer.core.keys import ColumnKey, ColumnSqlKey + + cross_model_path = getattr(agg_key.source, "path", ()) + local_agg = reroot_aggregate_key(agg_key, target_path=cross_model_path) + if isinstance(local_agg.source, ColumnSqlKey): + scope.resolve(local_agg.source) + for a in local_agg.args: + if isinstance(a, (ColumnKey, ColumnSqlKey)): + scope.resolve(a) + for _k, v in local_agg.kwargs: + if isinstance(v, (ColumnKey, ColumnSqlKey)): + scope.resolve(v) + cfk = local_agg.column_filter_key + if cfk is not None and cfk.canonical_sql: + for p in self._filter_join_paths( + sql=cfk.canonical_sql, source_relation=target_relation, + source_model=target_model, bundle=bundle, + ): + scope.join_paths.add(p) + + def _collect_routed_filters( + self, + *, + planned_query, + filter_ids: List[str], + target_relation: str, + target_model, + bundle, + first_last_state: Optional[FirstLastRenderState] = None, + ) -> Optional[exp.Expression]: + """Build a conjunction of bound filter predicates by ID. + + Filters routed into a cross-model CTE bind in the CTE's local + scope (``customers.status`` resolves to the target's table). + For row-phase filters whose typed ``value_key`` already encodes + the join-target columns, ``_render_filter_value_key`` resolves + each leaf against the target model. + + Returns ``None`` when the requested filter set is empty so the + caller can skip emitting WHERE / HAVING. + """ + if not filter_ids: + return None + wanted = set(filter_ids) + parts: List[exp.Expression] = [] + for fp in planned_query.filters_by_phase: + if fp.id not in wanted: + continue + if fp.expression is None: + continue + ast = self._render_filter_value_key_in_target_scope( + value_key=fp.expression.value_key, + target_relation=target_relation, + target_model=target_model, + planned_query=planned_query, + bundle=bundle, + first_last_state=first_last_state, + ) + if ast is not None: + parts.append(ast) + if not parts: + return None + return exp.and_(*parts) if len(parts) > 1 else parts[0] + + def _render_filter_value_key_in_target_scope( # NOSONAR(S3776) — sequential isinstance dispatch over the closed ValueKey union with per-type cross-model target-scope rules (joined-column qualification, derived-column expansion, rn-state aware aggregate synth). Each branch carries the per-type cross-model render contract; extracting helpers would scatter the contract. + self, + *, + value_key, + target_relation: str, + target_model, + planned_query, + bundle, + first_last_state: Optional[FirstLastRenderState] = None, + ) -> Optional[exp.Expression]: + """Render a bound filter's value key as SQL with bare column + refs qualified against the cross-model CTE's local scope. + + The typed pipeline carries filter ASTs as ``ValueKey``-rooted + trees (``ArithmeticKey`` / ``AggregateKey`` / ``ColumnKey`` / + ``ColumnSqlKey`` / scalars). The CTE renderer reuses the legacy + ``_build_agg`` / column-resolution helpers via a small local + recursion that binds each leaf to the target model's relation alias. + """ + from slayer.core.keys import ( + AggregateKey, + ArithmeticKey, + BetweenKey, + ColumnKey, + ColumnSqlKey, + InKey, + LiteralKey, + ScalarCallKey, + ) + + if isinstance(value_key, ColumnSqlKey): + # DEV-1450 #4b: a routed filter on a DERIVED column owned by the + # CTE target — expand its Column.sql rooted at the target so it + # emits real SQL instead of falling through to a bogus literal. + if value_key.model != target_model.name: + raise NotImplementedError( + f"DEV-1450: cross-model filter on derived column " + f"{value_key.column_name!r} owned by {value_key.model!r} " + f"(not the CTE target {target_model.name!r}) is not yet " + f"rendered in the typed pipeline.", + ) + expanded = self._expand_derived_column_sql( + source_model=target_model, + source_relation=target_relation, + column_name=value_key.column_name, + bundle=bundle, + ) + return self._parse(expanded) + + if isinstance(value_key, ColumnKey): + # Cross-model filter on the joined-target path: the column + # lives on the target (single-hop) or on an intermediate + # hop. For 7b.12 we expect target-rooted refs only. + path = value_key.path + # ``value_key.path`` is a tuple of hop names ending at the + # target. The cross-model planner routes filters to the + # CTE only when the path == target_path (single-hop) or is + # a prefix (multi-hop). Both forms render against the + # target's local relation alias by leaf name. + if path and path[-1] != target_relation: + # Intermediate hop ref — not yet rendered. + raise NotImplementedError( + f"DEV-1450 stage 7b.12: cross-model filter on an " + f"intermediate hop ({path!r}) not yet rendered in " + f"the typed pipeline.", + ) + return exp.Column( + this=exp.to_identifier(value_key.leaf), + table=exp.to_identifier(target_relation), + ) + if isinstance(value_key, LiteralKey): + return self._literal_key_to_exp(value_key) + if isinstance(value_key, AggregateKey): + # HAVING-route: render the aggregate against the target. + # Reuse the synthesise helper with target_model as scope. + # DEV-1501 (Codex round 9): the routed AggregateKey carries + # source / args / kwargs still rooted at the cross-model path + # (``customers.regions.amount:last(customers.regions.opened_at)`` + # arrives with ``args=(ColumnKey(path=("customers","regions"), + # leaf="opened_at"),)``). Inside the target CTE scope every ref + # must qualify under the local relation, not the host-rooted + # ``__``-path alias — the SAME symmetric reroot the projection + # path applies (DEV-1707). Without it, the ranked subquery's + # ``ORDER BY`` qualifies the time column under a non-existent + # alias inside the CTE. + cross_model_path = getattr(value_key.source, "path", ()) + local_agg = reroot_aggregate_key( + value_key, target_path=cross_model_path, + ) + from slayer.engine.planned import ValueSlot as _Slot + tmp_slot = _Slot( + id="_cte_having_tmp", + key=local_agg, + declared_name="_having_agg", + phase=value_key.phase, + type=None, + ) + # DEV-1501 Group A.3: when the CTE projects a first/last + # aggregate, the projected spec's alias is the key for + # ``filtered_rn_map`` / ``filtered_match_map``. Reusing the + # SAME alias here lets ``_build_agg``'s lookup hit, binding + # the HAVING aggregate to the dedicated ``_last_rn_fN`` (and + # match-flag) instead of bare ``_last_rn`` + raw filter_sql. + having_full_alias = ( + first_last_state.agg_synth_alias + if first_last_state is not None and first_last_state.agg_synth_alias + else f"{target_relation}._having_agg" + ) + synth = self._build_agg_render_spec_from_planned( + slot=tmp_slot, + key=local_agg, + source_model=target_model, + source_relation=target_relation, + full_alias=having_full_alias, + bundle=bundle, + ) + # DEV-1708 Law 2: if the projected first/last materialised its + # crossing SOURCE value as a ``_val_`` column inside the ranked + # subquery, the HAVING aggregate must bind to that SAME alias — the + # outer ``MAX(CASE WHEN _last_rn = 1 THEN END)`` + # would otherwise reference a table bound only inside the subquery. + # DEV-1709: the alias map is keyed by the RESOLVED value text + # (qualified + typed inner CAST) so same-sql-different-type + # aggregates bind to their own materialisations. + if first_last_state is not None and synth.sql: + resolved_key = self._resolve_value_sql(synth) + if resolved_key in first_last_state.value_alias_by_sql: + synth = synth.model_copy(update={ + "sql": first_last_state.value_alias_by_sql[resolved_key], + }) + # Thread the cross-model CTE's rn maps so the HAVING + # aggregate uses the same ``_first_rn`` / ``_last_rn{suffix}`` + # / ``_last_rn_fN`` column the CTE SELECT projects. + rn_suffix_map = ( + first_last_state.rn_suffix_map if first_last_state else None + ) + default_time_col = ( + first_last_state.default_time_col_sql + if first_last_state else None + ) + filtered_rn_map = ( + first_last_state.filtered_rn_map if first_last_state else None + ) + filtered_match_map = ( + first_last_state.filtered_match_map if first_last_state else None + ) + expr, _ = self._build_agg( + synth, + rn_suffix_map=rn_suffix_map, + default_time_col=default_time_col, + filtered_rn_map=filtered_rn_map, + filtered_match_map=filtered_match_map, + ) + return expr + if isinstance(value_key, ArithmeticKey): + op = value_key.op + rendered_operands = [ + self._render_filter_value_key_in_target_scope( + value_key=op_key, + target_relation=target_relation, + target_model=target_model, + planned_query=planned_query, + bundle=bundle, + first_last_state=first_last_state, + ) + for op_key in value_key.operands + ] + return self._build_arith_or_cmp_ast(op=op, operands=rendered_operands) + if isinstance(value_key, ScalarCallKey): + # DEV-1708 (Codex): a routed filter wrapping a target ref in a scalar + # call (``abs(customers.deep_pop) > 5``) — render each arg in the + # target scope (a derived arg expands + pulls its join) and rebuild + # the call, mirroring the local filter-render path's ScalarCallKey + # branch (``like`` → ``exp.Like``; else ``func(NAME, *args)`` through + # the dialect rewrite). Without this the key falls through to the + # scalar fallback and emits its repr as a bogus string literal. + rendered_args = [ + self._render_filter_value_key_in_target_scope( + value_key=a, + target_relation=target_relation, + target_model=target_model, + planned_query=planned_query, + bundle=bundle, + first_last_state=first_last_state, + ) + for a in value_key.args + ] + if value_key.name == "like": + return exp.Like(this=rendered_args[0], expression=rendered_args[1]) + return self._finalize_scalar_call( + exp.func(value_key.name.upper(), *rendered_args), + ) + if isinstance(value_key, BetweenKey): + # DEV-1708: a routed ``date_range``-derived BETWEEN over a target + # (possibly crossing) column — render each operand in the target + # scope, mirroring the local filter path's ``exp.Between``. + def _render(k): + return self._render_filter_value_key_in_target_scope( + value_key=k, + target_relation=target_relation, + target_model=target_model, + planned_query=planned_query, + bundle=bundle, + first_last_state=first_last_state, + ) + return exp.Between( + this=_render(value_key.column), + low=_render(value_key.low), + high=_render(value_key.high), + ) + if isinstance(value_key, InKey): + # DEV-1475: cross-model IN filter — render the LHS column + # rooted at the CTE's target relation (so a bare ``name`` on + # ``stores`` becomes ``stores.name``), and the RHS literals + # inline. The cross-model routing path lands here only when + # the InKey's LHS column lives on the CTE target. + col_expr = self._render_filter_value_key_in_target_scope( + value_key=value_key.column, + target_relation=target_relation, + target_model=target_model, + planned_query=planned_query, + bundle=bundle, + first_last_state=first_last_state, + ) + value_exprs = [ + self._literal_key_to_exp(lit) for lit in value_key.values + ] + in_expr = exp.In(this=col_expr, expressions=value_exprs) + return exp.Not(this=in_expr) if value_key.negated else in_expr + # Scalars stored inline (Decimal / str / bool / None). + return self._literal_key_to_exp(value_key) + + def _literal_key_to_exp(self, value) -> exp.Expression: + """Convert a scalar / LiteralKey value to a sqlglot literal.""" + from slayer.core.keys import LiteralKey + from decimal import Decimal + + if isinstance(value, LiteralKey): + inner = value.value + else: + inner = value + if isinstance(inner, bool): + return exp.Boolean(this=inner) + if isinstance(inner, (int, float, Decimal)): + return exp.Literal.number(str(inner)) + if inner is None: + return exp.Null() + return exp.Literal.string(str(inner)) + + def _build_arith_or_cmp_ast( + self, + *, + op: str, + operands: List[exp.Expression], + ) -> exp.Expression: + """Build a sqlglot expression for a binary or unary op. + + Mirrors the small subset of operators the bound-filter renderer + emits: comparisons (``==``, ``!=``, ``<``, ``<=``, ``>``, + ``>=``, ``is``, ``is not``), boolean (``and``, ``or``, ``not``), + arithmetic (``+``, ``-``, ``*``, ``/``). + """ + if op == "not": + return exp.Not(this=operands[0]) + # ``and`` / ``or`` (Codex round 2): the binder produces n-ary + # boolean ``ArithmeticKey`` for ``a AND b AND c`` (three operands); + # the prior implementation took only ``operands[0]`` / ``[1]`` and + # silently dropped the third predicate from cross-model HAVING/ + # WHERE, broadening results. Fold over every operand the same + # way ``_compose_arithmetic_op`` and ``_build_arithmetic_for_filter`` + # already do. + if op in ("and", "or"): + node_cls = exp.And if op == "and" else exp.Or + acc = operands[0] + for o in operands[1:]: + acc = node_cls(this=acc, expression=o) + return acc + left, right = operands[0], operands[1] + # ``IS`` / ``IS NOT`` (Codex review): the typed pipeline's filter + # normalizer lowers SQL ``IS NULL`` / ``IS NOT NULL`` to Python + # ``is None`` / ``is not None``. Render against a ``Null`` literal + # as the standard SQL forms. + if op == "is": + return exp.Is(this=left, expression=right) + if op == "is not": + return exp.Not(this=exp.Is(this=left, expression=right)) + op_map = { + "==": exp.EQ, + "!=": exp.NEQ, + "<": exp.LT, + "<=": exp.LTE, + ">": exp.GT, + ">=": exp.GTE, + "+": exp.Add, + "-": exp.Sub, + "*": exp.Mul, + "/": exp.Div, + } + cls = op_map.get(op) + if cls is None: + raise NotImplementedError( + f"DEV-1450 stage 7b.12: arithmetic operator {op!r} not " + f"supported in cross-model filter rendering.", + ) + return cls(this=left, expression=right) + + def _build_combined_order_by_sql( + self, + *, + planned_query, + slots_by_id: Dict[str, Any], + cma_slot_ids: Set[str], + cm_alias_for_plan: Dict[str, str], + bare_order_slot_ids: Optional[Set[str]] = None, + outer_composite_aliases: Optional[Dict[str, str]] = None, + outer_composite_expressions: Optional[Dict[str, str]] = None, + hidden_cte_order_refs: Optional[Dict[str, str]] = None, + ) -> Optional[str]: + """Build the ORDER BY clause for the combined SELECT. + + PROJECTED local slots are referenced as ``_base.""`` + (legacy parity); cross-model slots are referenced as bare + ``""`` (they live in a single column projected from + the cross-model CTE). HIDDEN order-only local slots + (``bare_order_slot_ids``) are also referenced bare: they are + materialised in ``_base`` but TRIMMED from the combined public + projection, so the outermost ORDER BY must use the unqualified + alias — the ``_base.`` qualifier would dangle if an outer + projection-trim wrapper (which exposes only the bare public + aliases) is ever layered on top. The bare alias still resolves + unambiguously against ``_base`` in the combined FROM. + + DEV-1503 (Codex round 3 #2): outer-routed composite slots + (``outer_composite_aliases``) live ONLY in the combined SELECT's + projection — they are not materialised in ``_base``. Reference + them as bare aliases so the ORDER BY resolves against the outer + SELECT's own column list rather than ``_base.`` (which + would dangle). + """ + if not planned_query.order: + return None + bare_ids = bare_order_slot_ids or set() + outer_aliases = outer_composite_aliases or {} + outer_expressions = outer_composite_expressions or {} + hidden_cte_refs = hidden_cte_order_refs or {} + parts: List[str] = [] + for entry in planned_query.order: + slot = slots_by_id.get(entry.slot_id) + if slot is None: + continue + term = self._resolve_combined_order_term( + entry=entry, + slot=slot, + source_relation=planned_query.source_relation, + cma_slot_ids=cma_slot_ids, + cm_alias_for_plan=cm_alias_for_plan, + bare_ids=bare_ids, + outer_aliases=outer_aliases, + outer_expressions=outer_expressions, + hidden_cte_refs=hidden_cte_refs, + ) + if term is not None: + parts.append(term) + if not parts: + return None + return "ORDER BY " + ", ".join(parts) + + def _resolve_combined_order_term( + self, + *, + entry, + slot, + source_relation: str, + cma_slot_ids: Set[str], + cm_alias_for_plan: Dict[str, str], + bare_ids: Set[str], + outer_aliases: Dict[str, str], + outer_expressions: Optional[Dict[str, str]] = None, + hidden_cte_refs: Optional[Dict[str, str]] = None, + ) -> Optional[str]: + """Resolve one ``OrderEntry`` to its ``"alias" `` term. + + Cross-model agg slot → bare CTE alias; projected outer-composite + slot → bare combined-SELECT alias; order-only outer composite + (Codex round 8 / CodeRabbit) → inline `` `` so + no synthetic alias leaks into the combined projection and the + transform-chain carry-forward doesn't lose track; hidden + order-only local → bare ``_base`` alias; everything else → + qualified ``_base.""``. Returns ``None`` when the + cross-model alias map has no entry (the order slot can't be + rendered). + """ + direction = "ASC" if entry.direction == "asc" else "DESC" + # DEV-1712 / DEV-1733: a HIDDEN (order-only) aggregate that lives in its + # own CTE — cross-model (``_cm_``) or windowed (``_wm_``) — is trimmed + # from the combined projection, so the bare alias no longer names a + # projected column. Reference the CTE-qualified column instead. Checked + # BEFORE the ``cma_slot_ids`` gate because a windowed slot is not a + # cross-model slot and would otherwise fall through to the bare-alias + # branch below and dangle. + hidden_ref = (hidden_cte_refs or {}).get(entry.slot_id) + if hidden_ref is not None: + return f'{hidden_ref} {direction}' + if entry.slot_id in cma_slot_ids: + alias = cm_alias_for_plan.get(entry.slot_id) + if alias is None: + return None + return f'{self._quote_ident(alias)} {direction}' + if entry.slot_id in outer_aliases: + return f'{self._quote_ident(outer_aliases[entry.slot_id])} {direction}' + if outer_expressions and entry.slot_id in outer_expressions: + return f'{outer_expressions[entry.slot_id]} {direction}' + full_alias = self._full_alias_for_slot( + slot=slot, + source_relation=source_relation, + alias_index={}, + ) + if entry.slot_id in bare_ids: + return f'{self._quote_ident(full_alias)} {direction}' + return f'_base.{self._quote_ident(full_alias)} {direction}' + + def _full_alias_for_slot( + self, + *, + slot, + source_relation: str, + alias_index: Dict[str, int], + ) -> str: + """Build the SQL public alias for one ``ValueSlot``. + + Local slots use the legacy ``.`` form + where ``alias`` is the user-declared name (cycled via + ``_pick_alias_for_planned_slot`` for C13 multi-alias slots) or + the planner's canonical ``declared_name``. + + DEV-1450 stage 7b.12: joined ROW slots emit the FULL dotted + result-key form (``orders.customers.region_id``), preserving + the result-key contract (P10). The planner's flat + ``declared_name`` is the DEV-1449 / C4 downstream-stage binding + name and remains untouched on the slot for stage-2 references; + only the public SQL alias differs. + + DEV-1713 (D3 / DEV-1495 bug 1): the ROW branch covers all three + row key shapes — ``ColumnKey``, ``ColumnSqlKey`` (a joined DERIVED + column, which previously fell through to the flat ``declared_name`` + and surfaced as ``orders.customers__revenue``), and ``TimeTruncKey`` + over either. All route through :func:`slayer.sql.naming.result_key`, + the single owner of the dotted form; response_meta mirrors this + via the same builder so the two producers cannot drift. + """ + from slayer.core.keys import ( + ColumnKey, + ColumnSqlKey, + Phase, + TimeTruncKey, + column_leaf, + column_path, + ) + + if slot.phase == Phase.ROW: + key = slot.key + path: Tuple[str, ...] = () + leaf: Optional[str] = None + if isinstance(key, ColumnKey): + path, leaf = key.path, key.leaf + elif isinstance(key, ColumnSqlKey): + # DEV-1713: a joined derived column's leaf is its column_name. + path, leaf = key.path, key.column_name + elif isinstance(key, TimeTruncKey): + # DEV-1450 #4a: a derived TD's leaf is its column_name, so the + # public result-key shape matches the base-column TD. + path, leaf = column_path(key.column), column_leaf(key.column) + if path and leaf is not None: + return result_key( + source_relation=source_relation, path=path, leaf=leaf, + ) + # Local + AGGREGATE / POST slots: existing alias selection. The alias + # may embed hop dots (a cross-model measure alias such as + # ``customers.revenue_sum``), so use the canonical-alias builder. + if slot.public_aliases: + alias = self._pick_alias_for_planned_slot( + slot=slot, alias_index=alias_index, + ) + else: + alias = slot.declared_name + return result_key_from_alias(source_relation=source_relation, alias=alias) + + def _collect_joined_paths_for_base( + self, + *, + base_render_order: List[str], + slots_by_id: Dict[str, Any], + order_slot_ids: Optional[List[str]] = None, + ) -> List[Tuple[str, ...]]: + """Walk ROW slots in render order to collect unique joined DIMENSION + paths needed for projection / GROUP BY. + + Cross-model aggregate slots are NEVER walked — their joins live in + ``CrossModelAggregatePlan.join_chain`` and render inside the per-plan + ``_cm_*`` CTE. Local ``first`` / ``last`` explicit-time-arg joins are no + longer collected here either: DEV-1710 Stage 6 moved that discovery into + ``_resolve_agg_inputs_via_scope`` (sub-pass 4), where anchoring the arg + through the host ``ScopeFrame`` registers its crossed join as a Law-1 + side effect (bare, derived, and multi-hop args alike). + + ``order_slot_ids`` (DEV-1703 Phase 1) walks ORDER BY targets for the + same paths. An order-only joined row column is deliberately NOT in + ``base_render_order`` (materialising it there would project it and add + it to GROUP BY, changing the result grain), but the split reference the + ORDER BY emits still needs its join bound in the base FROM — Law 1 + applies to a sort key exactly as it does to a filter ref. + """ + from slayer.core.keys import ColumnKey, Phase, TimeTruncKey + + seen: set = set() + ordered: List[Tuple[str, ...]] = [] + + def _add(path: Tuple[str, ...]) -> None: + if not path or path in seen: + return + seen.add(path) + ordered.append(path) + + def _add_row_slot(sid: str) -> None: + slot = slots_by_id.get(sid) + if slot is None or slot.phase != Phase.ROW: + return + key = slot.key + if isinstance(key, ColumnKey): + _add(key.path) + elif isinstance(key, TimeTruncKey): + _add(key.column.path) + + for sid in base_render_order: + _add_row_slot(sid) + for sid in order_slot_ids or (): + _add_row_slot(sid) + return ordered + + def _build_from_and_joins( + self, + *, + source_model, + source_relation: str, + joined_paths: List[Tuple[str, ...]], + bundle, + ): + """Build ``(from_expr, joins)`` for a base SELECT. + + ``from_expr`` is the single-source Table/Subquery (same shape + ``_build_from_clause_from_planned`` would return). ``joins`` is + a list of ``(join_expr, on_expr, join_type)`` tuples the caller + attaches via ``Select.join`` after constructing the SELECT. + + Single-hop paths use the target's bare name as the table alias + (matching legacy: ``LEFT JOIN customers AS customers ON ...``); + multi-hop paths use the ``__``-delimited path alias for non- + leading hops (``LEFT JOIN regions AS customers__regions ON + ...``). The cross-model rerooted CTE re-uses this helper rooted + at the terminal target model with an empty join list, so the + same FROM shape applies. + """ + base_from = self._build_from_clause_from_planned( + source_model=source_model, source_relation=source_relation, + ) + joins: List = [] + if not joined_paths: + return base_from, joins + emitted_aliases: set = {source_relation} + for path in joined_paths: + current_model = source_model + current_alias = source_relation + for hop_idx, hop in enumerate(path): + join_def = next( + (j for j in current_model.joins if j.target_model == hop), + None, + ) + if join_def is None: + raise ValueError( + f"Model {current_model.name!r} has no join to " + f"{hop!r}; needed for joined path {path!r}.", + ) + next_model = bundle.get_referenced_model(hop) + if next_model is None: + raise ValueError( + f"Join target {hop!r} not in resolved source bundle.", + ) + next_alias = ( + hop if hop_idx == 0 + else f"{current_alias}__{hop}" + ) + if next_alias not in emitted_aliases: + join_on_parts = [] + for src_col, tgt_col in join_def.join_pairs: + # DEV-1645: the join keys are physical DB columns — + # quote them when mixed-case (``merchantId``) via + # ``_to_ident`` so a case-folding backend resolves them; + # the table qualifiers are SLayer-internal aliases + # (reserved names quote at emit via RESERVED_KEYWORDS). + join_on_parts.append(exp.EQ( + this=exp.Column( + this=self._to_ident(src_col), + table=exp.to_identifier(current_alias), + ), + expression=exp.Column( + this=self._to_ident(tgt_col), + table=exp.to_identifier(next_alias), + ), + )) + target_table = ( + next_model.sql_table or next_model.name + ) + if next_model.sql and not next_model.sql_table: + join_expr = exp.Subquery( + this=self._parse(next_model.sql), + alias=exp.to_identifier(next_alias), + ) + else: + # DEV-1686 reserved-word alias + DEV-1645 mixed-case + # physical-name quoting: ``FROM "Order" AS "order"``. + join_expr = self._to_table(target_table, alias=next_alias) + on_expr = ( + exp.and_(*join_on_parts) + if len(join_on_parts) > 1 + else join_on_parts[0] + ) + # Honor the model's declared join_type (default LEFT so a + # measure never changes cardinality; explicit INNER when the + # user declared it — e.g. existence-filter joins). Legacy + # rendered ``jtype.upper()`` here (generator.py:835/1242). + joins.append(( + join_expr, on_expr, join_def.join_type.value.upper(), + )) + emitted_aliases.add(next_alias) + current_model = next_model + current_alias = next_alias + return base_from, joins + + def _joined_or_local_dim_expr( + self, + *, + path: Tuple[str, ...], + leaf: str, + source_model, + source_relation: str, + bundle, + ) -> exp.Expression: + """Resolve a dimension column expression on either the host + model (empty path) or a joined target (non-empty path). + + For empty paths this delegates to ``_dim_column_expr_from_planned`` + which respects ``Column.sql`` for derived columns. For joined + paths the legacy emits a bare ``.`` column + ref — matching that shape so parity comparisons hold. + """ + if not path: + return self._dim_column_expr_from_planned( + source_model=source_model, + source_relation=source_relation, + leaf=leaf, + ) + current_alias = source_relation + current_model = source_model + for hop_idx, hop in enumerate(path): + target_alias = ( + hop if hop_idx == 0 + else f"{current_alias}__{hop}" + ) + current_alias = target_alias + target_model = bundle.get_referenced_model(hop) + if target_model is None: + raise ValueError( + f"Joined dim path {path!r}: target {hop!r} missing " + f"from the resolved source bundle.", + ) + current_model = target_model + col_def = next( + (c for c in current_model.columns if c.name == leaf), None, + ) + if col_def is None: + raise ValueError( + f"Column {leaf!r} not found on joined model " + f"{current_model.name!r}.", + ) + # Legacy emits the bare-table.column form for joined dims even + # when the column has a ``Column.sql`` override on the target; + # mirror that for parity. + return exp.Column( + this=exp.to_identifier(leaf), + table=exp.to_identifier(current_alias), + ) + + def _render_window_transform_sql( + self, + *, + slot, + slots_by_id: Dict[str, Any], + slot_id_by_key: Dict[Any, str], + available_alias_by_slot_id: Dict[str, str], + planned_query, + ) -> str: + """Render one window-transform slot as an OVER() expression. + + Direct port of ``_build_transform_sql:1794`` but reads from the + typed ``TransformKey`` instead of legacy ``EnrichedTransform``. + Auto-partition matches legacy: ``partition_aliases = query + dimensions only`` (NOT time dimensions) for non-rank ops; + rank-family defaults to no PARTITION BY. + """ + from slayer.core.keys import ( + ColumnKey, + Phase, + TransformKey, + ) + + key = slot.key + if not isinstance(key, TransformKey): + raise ValueError( + f"_render_window_transform_sql expected TransformKey, " + f"got {type(key).__name__}", + ) + + # Composite transform inputs — a transform whose ``input`` is an + # arithmetic / scalar-call expression rather than a slotted leaf + # (``cumsum(amount:sum / qty:sum)``; ``cumsum(change(x))`` which + # lowers to ``cumsum(x - time_shift(x))``). Render the input + # expression INLINE against the operands' already-materialised + # aliases — the Kahn readiness check (``_transform_layer_deps_ready`` + # → ``_ready(tk.input)``) guarantees every operand slot is in a + # prior CTE before this layer runs, so no extra inner CTE is needed. + from slayer.core.keys import ( + ArithmeticKey as _ArithKey, + ScalarCallKey as _ScalarKey, + ) + + if isinstance(key.input, (_ArithKey, _ScalarKey)): + measure = self._render_value_key_against_aliases( + key=key.input, + slot_id_by_key=slot_id_by_key, + available_alias_by_slot_id=available_alias_by_slot_id, + ).sql(dialect=self.dialect) + else: + # Resolve input alias (slotted leaf). + input_sid = slot_id_by_key.get(key.input) + if input_sid is None or input_sid not in available_alias_by_slot_id: + raise RuntimeError( + f"transform input not materialised: slot id={slot.id!r}, " + f"op={key.op!r}, input_key={key.input!r}.", + ) + input_alias = available_alias_by_slot_id[input_sid] + measure = self._quote_ident(input_alias) + + # Resolve time-key alias (None for rank-family without time). + time_alias: Optional[str] = None + if key.time_key is not None: + tk_sid = slot_id_by_key.get(key.time_key) + if tk_sid is None or tk_sid not in available_alias_by_slot_id: + raise RuntimeError( + f"transform time_key not materialised: " + f"slot id={slot.id!r}, op={key.op!r}, " + f"time_key={key.time_key!r}.", + ) + time_alias = self._quote_ident(available_alias_by_slot_id[tk_sid]) + + # Resolve partition aliases. Explicit partition_keys take + # precedence; otherwise auto-partition by query dimension slots + # (ColumnKey row-phase, hidden==False) — NOT TimeTruncKey slots + # (matches legacy enrichment.py:584 ``[d.alias for d in + # dimensions]``). + rank_family = {"rank", "percent_rank", "dense_rank", "ntile"} + if key.partition_keys: + partition_aliases: list[str] = [] + for pk in sorted( + key.partition_keys, key=lambda k: repr(k), + ): + pk_sid = slot_id_by_key.get(pk) + if pk_sid is None or pk_sid not in available_alias_by_slot_id: + raise RuntimeError( + f"transform partition_key not materialised: " + f"slot id={slot.id!r}, op={key.op!r}, " + f"partition_key={pk!r}.", + ) + partition_aliases.append( + available_alias_by_slot_id[pk_sid], + ) + elif key.op in rank_family: + partition_aliases = [] + else: + partition_aliases = [] + for sid in planned_query.projection: + row_slot = slots_by_id.get(sid) + if row_slot is None or row_slot.phase != Phase.ROW: + continue + if not isinstance(row_slot.key, ColumnKey): + # Skip TimeTruncKey row slots — matches legacy + # ``[d.alias for d in dimensions]``. + continue + alias = available_alias_by_slot_id.get(sid) + if alias is not None: + partition_aliases.append(alias) + + partition_clause = ( + _SQL_PARTITION_BY + ", ".join(self._quote_ident(a) for a in partition_aliases) + if partition_aliases + else "" + ) + order_clause = ( + f"ORDER BY {time_alias}" if time_alias else "" + ) + over_parts = " ".join(p for p in (partition_clause, order_clause) if p) + rank_order = f"ORDER BY {measure} DESC" + rank_over = " ".join(p for p in (partition_clause, rank_order) if p) + + kwarg_map = dict(key.kwargs) + op = key.op + + def _normalise_periods(raw: Any, *, kw: str = "periods") -> int: + """Reject bool / non-integral periods; accept int / integral + Decimal. Mirrors the strict validation the binder applies to + ``ntile.n`` and ``time_shift.periods``.""" + from decimal import Decimal + if isinstance(raw, bool): + raise ValueError( + f"transform {op!r} kwarg {kw!r} must be an integer; " + f"got bool {raw!r}.", + ) + if isinstance(raw, int): + return int(raw) + if isinstance(raw, Decimal): + if raw != raw.to_integral_value(): + raise ValueError( + f"transform {op!r} kwarg {kw!r} must be an " + f"integer; got {raw!r}.", + ) + return int(raw) + raise ValueError( + f"transform {op!r} kwarg {kw!r} must be an integer; " + f"got {type(raw).__name__} {raw!r}.", + ) + + if op == "cumsum": + return f"SUM({measure}) OVER ({over_parts})" + if op == "lag": + n = abs(_normalise_periods(kwarg_map.get("periods", 1))) + return f"LAG({measure}, {n}) OVER ({over_parts})" + if op == "lead": + n = abs(_normalise_periods(kwarg_map.get("periods", 1))) + return f"LEAD({measure}, {n}) OVER ({over_parts})" + if op == "rank": + return f"RANK() OVER ({rank_over})" + if op == "percent_rank": + return f"PERCENT_RANK() OVER ({rank_over})" + if op == "dense_rank": + return f"DENSE_RANK() OVER ({rank_over})" + if op == "ntile": + n = kwarg_map.get("n") + if not isinstance(n, int): + # Decimal-normalised int. + try: + n_int = int(n) + except (TypeError, ValueError): + raise ValueError( + f"ntile requires a positive integer n, got {n!r}", + ) + n = n_int + if n <= 0: + raise ValueError( + f"ntile requires a positive integer n, got {n!r}", + ) + return f"NTILE({n}) OVER ({rank_over})" + if op == "first": + return ( + f"FIRST_VALUE({measure}) OVER ({over_parts} " + f"ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING)" + ) + if op == "last": + if time_alias is None: + raise ValueError( + f"Transform 'last' requires an unambiguous time " + f"dimension (binder/planner gap; slot id={slot.id!r}).", + ) + return ( + f"FIRST_VALUE({measure}) OVER " + f"({partition_clause} ORDER BY {time_alias} DESC " + f"ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING)" + ) + raise NotImplementedError( + f"DEV-1450 stage 7b.10: transform op {op!r} not in the " + f"window-transform slice scope.", + ) + + def _render_post_phase_filter_conditions( # NOSONAR(S3776) — one cohesive walk of every POST-phase filter producing the outer-WHERE conditions: per-filter slot-id lookup, expr rebuild (Compare / BoolOp / UnaryOp / scalar wraps), alias resolution. Splitting hides the shared registry / alias-map state both wrap-CTE and outer-WHERE emission depend on. + self, + *, + planned_query, + slot_id_by_key: Dict[Any, str], + available_alias_by_slot_id: Dict[str, str], + ) -> List[str]: + """Render each POST-phase ``FilterPhase.expression`` to a SQL + string suitable for the outer ``WHERE`` after the CTE chain. + + Walks the typed value-key tree. Slot-worthy keys + (``AggregateKey`` / ``TransformKey`` / row-phase columns) are + replaced with quoted alias refs (``"orders.cumsum_amount_sum"``) + looked up through ``slot_id_by_key`` / + ``available_alias_by_slot_id``. Arithmetic / scalar-call + composition uses the same operator dispatch as the WHERE + renderer in ``_render_value_key_for_filter``. + """ + from slayer.core.keys import Phase + + out: List[str] = [] + for fp in planned_query.filters_by_phase: + if fp.phase != Phase.POST: + continue + if fp.expression is None: + raise ValueError( + f"POST-phase FilterPhase id={fp.id!r} has no typed " + f"expression; text-only POST filters are not supported.", + ) + rendered = self._render_value_key_against_aliases( + key=fp.expression.value_key, + slot_id_by_key=slot_id_by_key, + available_alias_by_slot_id=available_alias_by_slot_id, + ) + out.append(rendered.sql(dialect=self.dialect)) + return out + + def _render_value_key_against_aliases( + self, + *, + key, + slot_id_by_key: Dict[Any, str], + available_alias_by_slot_id: Dict[str, str], + ) -> exp.Expression: + """Render a typed ValueKey tree against already-materialised + aliases (used inside the ``_filtered`` wrapper). + + Slot-worthy keys → quoted ``exp.Column`` refs to their aliases. + ``ArithmeticKey`` / ``ScalarCallKey`` / ``BetweenKey`` / + ``InKey`` / ``LiteralKey`` compose recursively. + """ + from slayer.core.keys import ( + AggregateKey, + ArithmeticKey, + BetweenKey, + ColumnKey, + ColumnSqlKey, + InKey, + LiteralKey, + ScalarCallKey, + TimeTruncKey, + TransformKey, + ) + + slotted_kinds = ( + ColumnKey, ColumnSqlKey, TimeTruncKey, AggregateKey, TransformKey, + ) + all_key_kinds = ( + TransformKey, ArithmeticKey, ScalarCallKey, BetweenKey, InKey, + ColumnKey, ColumnSqlKey, TimeTruncKey, AggregateKey, LiteralKey, + ) + + def recurse(k) -> exp.Expression: + return self._render_value_key_against_aliases( + key=k, + slot_id_by_key=slot_id_by_key, + available_alias_by_slot_id=available_alias_by_slot_id, + ) + + if isinstance(key, slotted_kinds): + sid = slot_id_by_key.get(key) + if sid is None or sid not in available_alias_by_slot_id: + raise RuntimeError( + f"POST-phase filter references a key not materialised " + f"as a slot: {type(key).__name__} -> {key!r}.", + ) + alias = available_alias_by_slot_id[sid] + return exp.Column(this=exp.to_identifier(alias, quoted=True)) + + if isinstance(key, LiteralKey): + return _render_scalar_literal(key.value) + + if isinstance(key, ArithmeticKey): + return self._compose_arithmetic_op( + op=key.op, operands=[recurse(o) for o in key.operands], + ) + + if isinstance(key, ScalarCallKey): + args = [ + recurse(a) if isinstance(a, all_key_kinds) + else _render_scalar_literal(a) + for a in key.args + ] + if key.name == "like": + return exp.Like(this=args[0], expression=args[1]) + return self._finalize_scalar_call(exp.func(key.name.upper(), *args)) + + if isinstance(key, BetweenKey): + return exp.Between( + this=recurse(key.column), + low=recurse(key.low), + high=recurse(key.high), + ) + + if isinstance(key, InKey): + # DEV-1475: POST-phase IN filter — LHS column resolves to a + # quoted alias materialised in the ``_filtered`` wrapper; RHS + # literals are inlined as bare sqlglot scalars. + in_expr = exp.In( + this=recurse(key.column), + expressions=[recurse(lit) for lit in key.values], + ) + return exp.Not(this=in_expr) if key.negated else in_expr + + raise NotImplementedError( + f"DEV-1450 stage 7b.10: POST-phase filter key type " + f"{type(key).__name__} not yet supported.", + ) + + @staticmethod + def _paren_if_lower_prec( + child: exp.Expression, *, parent_prec: int, is_right: bool, op: str, + ) -> exp.Expression: + """Wrap ``child`` in parens when its arithmetic precedence is lower + than the parent op's (or equal, for the RIGHT operand of the + non-associative ``-`` / ``/``). Leaves / functions / casts / already- + parenthesised nodes are returned untouched. + """ + child_prec = { + exp.Add: 1, exp.Sub: 1, exp.Mul: 2, exp.Div: 2, + }.get(type(child)) + if child_prec is None: + return child + if child_prec < parent_prec: + return exp.Paren(this=child) + if child_prec == parent_prec and is_right and op in ("-", "/"): + return exp.Paren(this=child) + return child + + @staticmethod + def _compose_arithmetic_op( + *, op: str, operands: List[exp.Expression], + ) -> exp.Expression: + """Compose an arithmetic / comparison / boolean operator over + already-rendered operands. + + Accepts the operator aliases ``=``/``==``, ``<>``/``!=`` so the + rendered SQL surfaces the canonical SQL spellings for POST + filters. Unary ``-`` and N-ary ``and``/``or`` left-fold to the + sqlglot binary nodes. + """ + if len(operands) == 1: + if op == "not": + return exp.Not(this=operands[0]) + if op == "-": + return exp.Neg(this=operands[0]) + if len(operands) == 2: + lhs, rhs = operands + # ``IS`` / ``IS NOT`` (Codex review): see ``_build_arith_or_cmp_ast``. + if op == "is": + return exp.Is(this=lhs, expression=rhs) + if op == "is not": + return exp.Not(this=exp.Is(this=lhs, expression=rhs)) + binary = { + "+": exp.Add, "-": exp.Sub, "*": exp.Mul, "/": exp.Div, + "<": exp.LT, "<=": exp.LTE, ">": exp.GT, ">=": exp.GTE, + "==": exp.EQ, "=": exp.EQ, + "!=": exp.NEQ, "<>": exp.NEQ, + } + if op in binary: + # sqlglot does NOT add precedence parens for a nested AST, so + # ``Div(Sub(a, b), c)`` would render as ``a - b / c`` (wrong: + # ``b / c`` binds first). Parenthesise a lower-precedence + # operand — and an equal-precedence RIGHT operand under the + # non-associative ``-`` / ``/`` — so ``change_pct`` and friends + # emit ``(a - b) / c``. + arith_prec = {"+": 1, "-": 1, "*": 2, "/": 2} + parent_prec = arith_prec.get(op) + if parent_prec is not None: + lhs = SQLGenerator._paren_if_lower_prec( + lhs, parent_prec=parent_prec, is_right=False, op=op, + ) + rhs = SQLGenerator._paren_if_lower_prec( + rhs, parent_prec=parent_prec, is_right=True, op=op, + ) + return binary[op](this=lhs, expression=rhs) + if op == "and": + return exp.And(this=lhs, expression=rhs) + if op == "or": + return exp.Or(this=lhs, expression=rhs) + if len(operands) >= 2 and op in ("and", "or"): + node_cls = exp.And if op == "and" else exp.Or + acc = operands[0] + for rhs in operands[1:]: + acc = node_cls(this=acc, expression=rhs) + return acc + raise NotImplementedError( + f"DEV-1450 stage 7b.10: arithmetic op {op!r} arity " + f"{len(operands)} not supported in POST-filter rendering.", + ) + + def _emit_planned_outer_wrap( + self, + *, + chain_sql: str, + public_aliases: List[str], + planned_query, + slots_by_id: Dict[str, Any], + available_alias_by_slot_id: Dict[str, str], + ) -> str: + """Wrap ``chain_sql`` in the public-projection outer SELECT, through + the dialect strategy (DEV-1716). + + Delegates to ``SqlDialect.emit_outer_wrap`` rather than string-building + ``FROM () AS _outer`` inline. T-SQL overrides that hook to hoist + the inner top-level CTEs onto the outer statement, because SQL Server + accepts ``WITH`` only as a statement prefix and rejects + ``FROM (WITH ... SELECT ...) AS _outer`` with "Incorrect syntax near + the keyword 'WITH'" (DEV-1571 Bug 1). Every other dialect gets the + base impl, whose output is byte-identical to the previous inline + string. + + ORDER BY / LIMIT / OFFSET are resolved here from the typed plan (slot + id -> materialised alias) and handed to the hook as AST, since the + T-SQL override also transposes pagination to ``TOP`` / + ``FETCH NEXT n ROWS ONLY``. + """ + order_sql = self._planned_order_by_sql( + planned_query=planned_query, + slots_by_id=slots_by_id, + available_alias_by_slot_id=available_alias_by_slot_id, + ) + order_expr = ( + self._parse(f"SELECT 1 ORDER BY {order_sql}").args.get("order") + if order_sql + else None + ) + limit_expr = ( + exp.Limit(expression=exp.Literal.number(planned_query.limit)) + if planned_query.limit is not None + else None + ) + offset_expr = ( + exp.Offset(expression=exp.Literal.number(planned_query.offset)) + if planned_query.offset is not None + else None + ) + return self._dialect.emit_outer_wrap( + inner_sql=chain_sql, + public=public_aliases, + order=order_expr, + limit=limit_expr, + offset_arg=offset_expr, + parse=self._parse, + ) + + def _planned_order_by_sql( + self, + *, + planned_query, + slots_by_id: Dict[str, Any], + available_alias_by_slot_id: Dict[str, str], + ) -> str: + """Render the ORDER BY term list (without the keyword) for a planned + query whose sort keys resolve to CTE-chain aliases.""" + order_parts: list[str] = [] + for order_entry in planned_query.order: + slot = slots_by_id.get(order_entry.slot_id) + alias = available_alias_by_slot_id.get(order_entry.slot_id) + if slot is None or alias is None: + raise RuntimeError( + f"ORDER BY references slot id={order_entry.slot_id!r} " + f"not materialised in the CTE chain.", + ) + direction = ( + "ASC" if order_entry.direction == "asc" else "DESC" + ) + order_parts.append(f'{self._quote_ident(alias)} {direction}') + return ", ".join(order_parts) + + + # ----------------------------------------------------------------- + # Stage 7b.11 helpers — self-join CTE transforms (time_shift, + # consecutive_periods). change / change_pct desugar at plan time to + # time_shift + arithmetic, so the renderer only needs the two + # primitive shapes below. + # ----------------------------------------------------------------- + + def _build_shifted_cte_where_parts( + self, + *, + planned_query, + source_relation: str, + source_model, + bundle, + ) -> Tuple[List[str], List[Tuple[str, ...]]]: + """Build the WHERE clauses for the shifted CTE that re-aggregates + the source relation, plus the join paths those clauses cross. + + 7b.3c invariant, generalised by DEV-1732: a FRAME BOUND must be omitted + from the shifted inner CTE so the earliest visible bucket can still + carry a non-null shifted value. That covers the ``BetweenKey`` a + ``date_range`` produces AND the explicit relational spelling of the same + intent (``created_at >= '2024-01-01'``), which used to be propagated — + so the two spellings gave different numbers. A filter that is only + PARTLY a frame bound propagates as its residual population predicate. + + Other ROW-phase filters (e.g. ``status = 'active'``) are propagated + unchanged so the shifted aggregation runs over the same row population. + AGGREGATE / POST phase filters never apply to the shifted CTE + (they're outer-projection concerns). + + DEV-1711: a ROW filter referencing a JOINED column + (``stores.name = 'North'``) is now supported — the shifted CTE is a + real ``ScopeFrame`` whose FROM pulls the join, so the guard that used + to raise on joined refs is gone. The returned ``crossed_paths`` list is + registered into the caller's shifted scope so the LEFT JOIN the filter + needs is emitted. Filters over the same join set the base already + applies keep population parity between ``_base`` and the shifted CTE. + """ + from slayer.core.keys import Phase + + out: List[str] = [] + crossed_paths: List[Tuple[str, ...]] = [] + # DEV-1732: the frame-bound column set is computed once by the planner + # and carried on the plan, so this path and the windowed ``_src`` path + # cannot drift apart. + time_cols = frozenset(planned_query.frame_bound_columns) + for fp in planned_query.filters_by_phase: + if fp.phase != Phase.ROW: + continue + rendered = self._shifted_where_part( + fp=fp, source_relation=source_relation, + source_model=source_model, bundle=bundle, + time_columns=time_cols, + ) + if rendered is None: + continue + part, paths = rendered + out.append(part) + for p in paths: + if p not in crossed_paths: + crossed_paths.append(p) + return out, crossed_paths + + def _shifted_where_part( + self, *, fp, source_relation: str, source_model, bundle, + time_columns: "AbstractSet[Any]", + ) -> "Optional[Tuple[str, List[Tuple[str, ...]]]]": + """Render one ROW-phase filter for the shifted CTE, returning its SQL + plus the join paths it crosses — or ``None`` to omit it entirely. + + A filter that is wholly a FRAME BOUND on one of ``time_columns`` is + omitted; one that is partly a frame bound renders as its residual + population predicate (DEV-1732). This subsumes the old + ``isinstance(..., BetweenKey)`` special case: a ``date_range``'s + ``BetweenKey`` column is always a query time dimension's raw column, so + ``strip_frame_bounds`` returns ``None`` for it — same behaviour, one + rule. + + Mode-A ``text`` filters are exempt from the analysis and always + propagate (a model filter defines which rows EXIST, not the frame). + + ``time_columns`` is REQUIRED, deliberately (Codex): ``strip_frame_bounds`` + returns its input unchanged for an empty set, so a default would let a + future caller silently start rendering every ``date_range`` into the + shifted CTE — the exact 7b.3c regression this method exists to prevent. + + The join paths are collected per carrier kind (CodeRabbit): a TYPED + filter is scanned STRUCTURALLY on its already-rendered AST via + ``_joined_paths_in_sql`` — the expression is fully qualified/expanded, + so its crossed joins are visible directly and there is no text + round-trip that could silently swallow a parse failure. A Mode-A + ``text`` filter has only its string form, so it keeps the + ``_filter_join_paths`` dual raw + inline-expanded scan (the DEV-1494 + contract that surfaces a derived ref's expansion joins). + + Note the scan runs on the RESIDUAL, so the shifted CTE's join set + follows what it actually renders. + """ + if fp.expression is not None: + residual = strip_frame_bounds( + key=fp.expression.value_key, time_columns=time_columns, + ) + if residual is None: + return None # wholly a frame bound — omit from the shifted CTE. + rendered = self._render_value_key_for_filter( + key=residual, + source_relation=source_relation, + source_model=source_model, + bundle=bundle, + ) + if isinstance(rendered, (exp.And, exp.Or)): + rendered = exp.Paren(this=rendered) + paths = self._joined_paths_in_sql( + sql_expr=rendered, source_relation=source_relation, + source_model=source_model, bundle=bundle, + ) + return rendered.sql(dialect=self.dialect), paths + if fp.text is not None: + qualified = self._render_model_filter_sql( + sql=fp.text, + columns=fp.text_columns, + source_model=source_model, + source_relation=source_relation, + bundle=bundle, + ) + paths = self._filter_join_paths( + sql=qualified, source_relation=source_relation, + source_model=source_model, bundle=bundle, + ) + return qualified, paths + return None + + def _emit_time_shift_ctes_for_planned( # NOSONAR(S3776) — single conceptual unit for one time_shift slot: partition/time resolution through the shifted ScopeFrame + shifted-CTE body assembly + collision-safe CTE naming (cte_allocator) + sjoin grain join-back, all sharing tightly-coupled per-slot state (time_alias / input_alias / partition_specs / shifted_cte_name / carry aliases). Splitting forces that cross-cutting state through many-argument helpers without simplifying anything — same shape as the sibling _render_cross_model_cte's suppression. + self, + *, + slot, + ctes: list, + cte_allocator: AliasAllocator, + slots_by_id: Dict[str, Any], + slot_id_by_key: Dict[Any, str], + available_alias_by_slot_id: Dict[str, str], + aliases_by_slot_id: Dict[str, List[str]], + source_model, + source_relation: str, + shifted_where_parts: List[str], + shifted_where_join_paths: List[Tuple[str, ...]], + planned_query, + bundle, + ) -> None: + """Emit a ``shifted_`` + ``sjoin_`` CTE pair for + one time_shift transform slot. + + Legacy reference: ``slayer/sql/generator.py::_generate_shifted_base`` + and the sjoin assembly inside ``_generate_with_computed:1546``. + The typed implementation differs from legacy in two principled + ways: + + * **Inner reads raw data**: ``BetweenKey`` filters from + ``TimeDimension.date_range`` are omitted from the shifted CTE + (the 7b.3c invariant). Legacy instead substituted the time + column inside WHERE filters with a shifted expression to read + adjacent periods; the typed pipeline reads raw and lets the + outer projection re-apply the BETWEEN. + * **partition_keys**: DEV-1450 C6 — explicit ``partition_by`` on + ``change`` / ``time_shift`` threads through as additional + equality keys in the LEFT JOIN (not just query dimensions). + + DEV-1711 (Stage 7): the shifted CTE is a ``ScopeFrame`` (Laws 1 & 2). + Every partition key and the shift-axis time expression enters through + ``scope.resolve`` — anchoring the ref AND registering the join it + crosses in one call — so the shifted CTE's FROM (built from + ``scope.join_paths``) pulls exactly the LEFT JOINs the shifted + projection references. This makes CROSS-MODEL partitions (``stores. + name``), DERIVED dim partitions (local ``upper(status)`` or joined + ``stores.tier``), SECONDARY time-dimension partitions, and joined-column + ROW filters all work, and removes the joinless-CTE guards. The sjoin + grain join-back (time axis + every partition) is dialect-aware + null-safe (Codex F2) so NULL dim / NULL time-bucket groups keep their + shifted value instead of silently dropping. + """ + from slayer.core.enums import TimeGranularity + from slayer.core.keys import ( + AggregateKey, + ColumnKey, + ColumnSqlKey, + TimeTruncKey, + TransformKey, + ) + + key = slot.key + if not isinstance(key, TransformKey) or key.op != "time_shift": + raise ValueError( + f"expected time_shift TransformKey, got " + f"{type(key).__name__} (op={getattr(key, 'op', None)!r})", + ) + inner_key = key.input + time_key = key.time_key + if not isinstance(inner_key, (AggregateKey, ColumnKey, ColumnSqlKey)): + raise NotImplementedError( + f"DEV-1450 stage 7b.11: composite-input transforms " + f"(layer op='time_shift' input={type(inner_key).__name__}) " + f"are deferred to a follow-up slice. slot id={slot.id!r}." + ) + if not isinstance(time_key, TimeTruncKey): + raise ValueError( + f"time_shift requires a TimeTruncKey time_key; got " + f"{type(time_key).__name__} (slot id={slot.id!r}).", + ) + + # Resolve periods kwarg (binder defaulted to None if missing — + # validation raised already in that case). + periods_raw = next( + (v for k, v in key.kwargs if k == "periods"), None, + ) + if periods_raw is None: + raise ValueError( + f"time_shift requires 'periods' kwarg; planner gap " + f"(slot id={slot.id!r}).", + ) + from decimal import Decimal + if isinstance(periods_raw, bool): + raise ValueError( + f"time_shift periods must be an integer; got bool {periods_raw!r}", + ) + if isinstance(periods_raw, Decimal): + if periods_raw != periods_raw.to_integral_value(): + raise ValueError( + f"time_shift periods must be an integer; got {periods_raw!r}", + ) + periods = int(periods_raw) + elif isinstance(periods_raw, int): + periods = int(periods_raw) + else: + raise ValueError( + f"time_shift periods must be an integer; got " + f"{type(periods_raw).__name__} {periods_raw!r}", + ) + + # The aliases the shifted CTE needs to project. + # 1. The time-trunc column (shifted, then DATE_TRUNC'd) AS its + # own alias matching the base CTE. + time_sid = slot_id_by_key.get(time_key) + if time_sid is None or time_sid not in available_alias_by_slot_id: + raise RuntimeError( + f"time_shift time_key not materialised in base CTE: " + f"slot id={slot.id!r}, time_key={time_key!r}.", + ) + time_alias = available_alias_by_slot_id[time_sid] + + # 2. The aggregate / column input under its base alias. + input_sid = slot_id_by_key.get(inner_key) + if input_sid is None or input_sid not in available_alias_by_slot_id: + raise RuntimeError( + f"time_shift input not materialised in base CTE: " + f"slot id={slot.id!r}, input={inner_key!r}.", + ) + input_alias = available_alias_by_slot_id[input_sid] + + # DEV-1711 (Law 1): the shifted CTE is a ScopeFrame. Every partition + # key and the shift-axis time expression enters through ``resolve``, + # which anchors the ref AND registers the join it crosses. The FROM + # (built below from ``shifted_scope.join_paths``) then pulls exactly + # those LEFT JOINs — a cross-model / derived / secondary-time partition + # can never reference an unjoined table. The scope shares the + # generation-wide allocator so any ``_val_`` names stay unique + # across the base and every CTE. + shifted_allocator = self._gen_allocator or self._new_allocator() + shifted_scope = ScopeFrame( + scope_id=shifted_allocator.next_scope_id(source_relation), + root_model=source_model, + root_relation=source_relation, + bundle=bundle, + dialect=self._dialect, + allocator=shifted_allocator, + ) + + # 3. partition_keys (DEV-1450 C6) + auto-include query dimensions. + # + # Legacy auto-joins on EVERY query dimension regardless of + # partition_by (``_generate_with_computed:1559``). Without this, + # ``time_shift(amount:sum, periods=-1)`` with ``status`` in + # ``dimensions`` would broadcast the prior-period total across + # every status value. The typed pipeline mirrors this AND extends it + # (DEV-1711): the sjoin grain is EVERY projected dimension — joined + # ``ColumnKey``, derived ``ColumnSqlKey``, and SECONDARY ``TimeTruncKey`` + # (a second time dim, distinct from the shift axis) — plus any explicit + # ``partition_keys`` (C6). The shift axis itself is the time-join + # column, excluded by slot id. + from slayer.core.keys import Phase as _Phase + partition_specs: list[tuple[str, str, exp.Expression]] = [] + # entries: (slot_id, base_alias, resolved_expr_for_select_and_group_by) + seen_partition_sids: set = set() + + def _resolve_partition_expr(pk_obj) -> exp.Expression: + # A SECONDARY time dimension renders as DATE_TRUNC over its resolved + # (possibly joined / derived) raw column; a plain / derived column + # renders as its resolved expression. ``resolve`` registers the + # crossed join in both cases (Law 1). + if isinstance(pk_obj, TimeTruncKey): + raw = shifted_scope.resolve(pk_obj.column) + return self._build_date_trunc( + col_expr=raw, + granularity=TimeGranularity(pk_obj.granularity), + ) + if isinstance(pk_obj, (ColumnKey, ColumnSqlKey)): + return shifted_scope.resolve(pk_obj) + raise NotImplementedError( + f"time_shift partition on {type(pk_obj).__name__} is not " + f"supported (only column / derived-column / time-dimension " + f"partitions render in the shifted CTE). slot id={slot.id!r}.", + ) + + def _add_partition(pk_obj, *, where: str) -> None: + pk_sid = slot_id_by_key.get(pk_obj) + if pk_sid is None or pk_sid not in available_alias_by_slot_id: + raise RuntimeError( + f"time_shift {where} not materialised: " + f"slot id={slot.id!r}, key={pk_obj!r}.", + ) + # The shift axis is the time-join column, never a partition pair. + if pk_sid == time_sid or pk_sid in seen_partition_sids: + return + pk_alias = available_alias_by_slot_id[pk_sid] + partition_specs.append((pk_sid, pk_alias, _resolve_partition_expr(pk_obj))) + seen_partition_sids.add(pk_sid) + + # Auto-include EVERY projected row dimension (column, derived column, or + # secondary time dimension); the shift axis is skipped by slot id above. + for sid in planned_query.projection: + dim_slot = slots_by_id.get(sid) + if dim_slot is None or dim_slot.phase != _Phase.ROW: + continue + if not isinstance(dim_slot.key, (ColumnKey, ColumnSqlKey, TimeTruncKey)): + continue + _add_partition(dim_slot.key, where="query dimension") + + # Explicit partition_keys (DEV-1450 C6) may add more (deduped by slot id + # against the auto-included dims — see the DEV-1711 dedup test). + for pk in sorted(key.partition_keys, key=lambda k: repr(k)): + _add_partition(pk, where="partition_key") + + # DEV-1711 defensive completeness: a LOCAL aggregate whose source / + # column-filter / kwargs cross a join is isolated upstream (Stage 5) and + # would have raised 7b.15e before reaching a time_shift CTE, so these + # registrations are provably no-ops today — but routing them through the + # scope keeps Law 1 total (no render path skips join discovery). + if isinstance(inner_key, AggregateKey): + if isinstance(inner_key.source, ColumnSqlKey): + shifted_scope.resolve(inner_key.source) + for _kname, _kval in inner_key.kwargs: + if isinstance(_kval, (ColumnKey, ColumnSqlKey)): + shifted_scope.resolve(_kval) + if inner_key.column_filter_key is not None: + for _p in self._filter_join_paths( + sql=inner_key.column_filter_key.canonical_sql, + source_relation=source_relation, + source_model=source_model, bundle=bundle, + ): + shifted_scope.join_paths.add(_p) + + # Build the shifted time-column expression. Calendar offset is + # ``-periods`` units in the SHIFT granularity (periods=-1 -> +1 unit). + # The shift granularity is the explicit 3rd arg + # (``time_shift(x, -1, 'year')``) when given, else the query time + # dimension's granularity — so a year-shift over a month bucket + # yields "same month, previous year" (YoY). The DATE_TRUNC below + # always uses the TD granularity (the join/bucket axis). + shift_gran_raw = next( + (v for k, v in key.kwargs if k == "granularity"), None, + ) + shift_granularity = ( + str(shift_gran_raw) if shift_gran_raw is not None + else time_key.granularity + ) + # DEV-1450 #4a / DEV-1711: the shift-axis raw time expression resolves + # through the SAME scope (Law 1) — a derived (ColumnSqlKey) time column + # yields its EXPANDED expression, and a JOINED time axis (``stores. + # opened_at``) registers its join so the shifted FROM binds it. The + # calendar offset and DATE_TRUNC then apply over that expression. + raw_time_col_expr = shifted_scope.resolve(time_key.column) + shifted_raw_expr = self._build_time_offset_expr( + col_expr=raw_time_col_expr, + offset=-periods, + granularity=shift_granularity, + ) + shifted_trunc_expr = self._build_date_trunc( + col_expr=shifted_raw_expr, + granularity=TimeGranularity(time_key.granularity), + ) + + # Build the shifted CTE. + shifted_select_parts: list[str] = [] + shifted_group_by: list[str] = [] + + # Projected: time-trunc shifted under the base time alias. + shifted_trunc_sql = shifted_trunc_expr.sql(dialect=self.dialect) + shifted_select_parts.append( + f'{shifted_trunc_sql} AS {self._quote_ident(time_alias)}', + ) + shifted_group_by.append(shifted_trunc_sql) + + # partition_keys: SELECT + GROUP BY under their base aliases. + for _, pk_alias, pk_expr in partition_specs: + pk_sql = pk_expr.sql(dialect=self.dialect) + shifted_select_parts.append(f'{pk_sql} AS {self._quote_ident(pk_alias)}') + shifted_group_by.append(pk_sql) + + # Aggregate: re-emit the AggregateKey using the same synth / + # _build_agg dance the base CTE uses. + if isinstance(inner_key, AggregateKey): + # Build a synth EnrichedMeasure for _build_agg. + # + # The renderer needs a slot-like input with declared_name + + # type. Pull from the inner aggregate's slot to keep typed + # CAST behavior aligned with the base. + inner_slot = slots_by_id.get(input_sid) + if inner_slot is None: + raise RuntimeError( + f"inner aggregate slot {input_sid!r} not found", + ) + synth = self._build_agg_render_spec_from_planned( + slot=inner_slot, + key=inner_key, + source_model=source_model, + source_relation=source_relation, + full_alias=input_alias, + bundle=bundle, + ) + agg_expr, _ = self._build_agg(synth) + agg_expr = _wrap_cast_for_type(agg_expr, inner_slot.type) + shifted_select_parts.append( + f'{agg_expr.sql(dialect=self.dialect)} AS {self._quote_ident(input_alias)}', + ) + else: + # Row-level column input (not aggregated). Resolve through the scope + # so a joined / derived input registers its join and anchors + # correctly (Law 1), same as every other ref in this CTE. + col_expr = shifted_scope.resolve(inner_key) + shifted_select_parts.append( + f'{col_expr.sql(dialect=self.dialect)} AS {self._quote_ident(input_alias)}', + ) + shifted_group_by.append(col_expr.sql(dialect=self.dialect)) + + # DEV-1711: register the join paths the shifted WHERE filters cross + # (computed once by ``_build_shifted_cte_where_parts``) so a joined-column + # ROW filter (``stores.name = 'North'``) pulls its LEFT JOIN into this + # CTE. Then build the FROM from the scope's full registered set — a + # crossed join can never be forgotten because discovery is a side effect + # of resolving each ref above. + for _p in shifted_where_join_paths: + shifted_scope.join_paths.add(_p) + shifted_join_paths = shifted_scope.join_paths.as_list() + if shifted_join_paths: + from_clause, shifted_joins = self._build_from_and_joins( + source_model=source_model, + source_relation=source_relation, + joined_paths=shifted_join_paths, + bundle=bundle, + ) + else: + from_clause = self._build_from_clause_from_planned( + source_model=source_model, source_relation=source_relation, + ) + shifted_joins = [] + + from_parts = [f"FROM {from_clause.sql(dialect=self.dialect)}"] + for join_expr, on_expr, join_type in shifted_joins: + from_parts.append( + f"{join_type} JOIN {join_expr.sql(dialect=self.dialect)} " + f"ON {on_expr.sql(dialect=self.dialect)}" + ) + + shifted_sql_parts = [_SQL_SELECT_HEAD + ",\n ".join(shifted_select_parts)] + shifted_sql_parts.extend(from_parts) + if shifted_where_parts: + shifted_sql_parts.append( + "WHERE " + _SQL_AND_JOINER.join(shifted_where_parts), + ) + if shifted_group_by: + shifted_sql_parts.append( + "GROUP BY\n " + ",\n ".join(shifted_group_by), + ) + shifted_sql = "\n".join(shifted_sql_parts) + + # Pick the slot's user-facing alias(es). DEV-1450 C13: two + # declared measures sharing a structural key intern to ONE + # slot with multiple ``public_aliases``; the sjoin CTE projects + # the shifted measure under EACH alias so the outer SELECT + # carries both. + # DEV-1692: a HIDDEN inner time_shift slot's declared_name + # (``_time_shift_inner``) is NOT unique across sibling shifts with + # different offsets — two would project + resolve downstream under the + # same column, silently collapsing ``growth_2m`` onto ``growth_1m``'s + # shift. Allocate a unique internal alias for the hidden case; USER + # aliases (public_aliases, already unique) are left untouched. + if slot.public_aliases: + slot_aliases: List[str] = list(slot.public_aliases) + else: + slot_aliases = [cte_allocator.allocate_cte(slot.declared_name)] + cte_name_alias = slot_aliases[0] + # DEV-1692: allocate collision-free CTE names too. + shifted_cte_name = cte_allocator.allocate_cte(f"shifted_{cte_name_alias}") + sjoin_cte_name = cte_allocator.allocate_cte(f"sjoin_{cte_name_alias}") + + ctes.append((shifted_cte_name, shifted_sql)) + + # Build the sjoin CTE: LEFT JOIN prev_cte + shifted on time + + # partition equalities. Carry every prev_cte alias forward, + # then add the shifted measure under EACH of the slot's public + # aliases (DEV-1450 C13). + prev_cte = ctes[-2][0] # the CTE just before the shifted CTE + carry_aliases_sorted = sorted( + a for aliases in aliases_by_slot_id.values() for a in aliases + ) + sjoin_select_parts = [ + f'{prev_cte}.{self._quote_ident(a)}' for a in carry_aliases_sorted + ] + slot_full_aliases: List[str] = [] + for slot_alias in slot_aliases: + full_slot_alias = f"{source_relation}.{slot_alias}" + slot_full_aliases.append(full_slot_alias) + sjoin_select_parts.append( + f'{shifted_cte_name}.{self._quote_ident(input_alias)} AS {self._quote_ident(full_slot_alias)}', + ) + + # JOIN conditions: time equality + every partition equality, all + # dialect-aware NULL-SAFE (DEV-1711 / Codex F2). The sjoin is a grain + # join-back — a NULL dimension value (e.g. a LEFT-joined ``stores.name`` + # with no matching store) or a NULL time bucket must match its own group + # instead of silently dropping to a NULL shifted value under plain ``=``. + # + # The predicate is built from AST nodes DIRECTLY — not via + # ``_null_safe_join_pair_sql``'s string round-trip — because a dotted + # public alias (``orders.created_at``) re-parses on BigQuery/T-SQL as a + # multi-part reference and the DEV-1713 alias mangling then corrupts it + # (``base.`orders.created_at``` → ``base___orders`.`created_at```). The + # alias as a single ``quoted=True`` identifier matches the SELECT parts' + # ``_quote_ident`` output byte-for-byte on every dialect and survives the + # post-generation mangling intact. + def _grain_eq(a: str) -> str: + left = exp.Column( + this=exp.to_identifier(a, quoted=True), + table=exp.to_identifier(prev_cte), + ) + right = exp.Column( + this=exp.to_identifier(a, quoted=True), + table=exp.to_identifier(shifted_cte_name), + ) + return self._dialect.build_null_safe_eq(left, right).sql(dialect=self.dialect) + + join_conds = [_grain_eq(time_alias)] + for _, pk_alias, _ in partition_specs: + join_conds.append(_grain_eq(pk_alias)) + + sjoin_sql = ( + "SELECT " + ", ".join(sjoin_select_parts) + + f"\nFROM {prev_cte}" + + f"\nLEFT JOIN {shifted_cte_name}" + + "\n ON " + _SQL_AND_JOINER.join(join_conds) + ) + ctes.append((sjoin_cte_name, sjoin_sql)) + + # Record EACH alias in both the per-slot list (C13 carry-forward + # in the outer SELECT) and the "pick one" map (transform input / + # filter / order lookups by downstream layers). + for full_slot_alias in slot_full_aliases: + aliases_by_slot_id.setdefault(slot.id, []).append(full_slot_alias) + # ``available_alias_by_slot_id`` is "pick one" — first alias wins. + available_alias_by_slot_id.setdefault(slot.id, slot_full_aliases[0]) + + def _emit_consecutive_periods_ctes_for_planned( # NOSONAR(S3776) — one cohesive per-slot consecutive_periods emission: predicate-shape decision, unique hidden alias plus collision-safe reset and value CTE names, the reset-group window layer, then the count-within-group window layer. Each block shares the slot registry and alias maps and cte_allocator; extracting helpers would scatter that contract without simplifying it. + self, + *, + slot, + ctes: list, + cte_allocator: AliasAllocator, + slots_by_id: Dict[str, Any], + slot_id_by_key: Dict[Any, str], + available_alias_by_slot_id: Dict[str, str], + aliases_by_slot_id: Dict[str, List[str]], + planned_query, + source_relation: str, + ) -> None: + """Emit ``cp_reset_`` + ``cp_value_`` CTEs for one + consecutive_periods transform slot. + + Supersedes the legacy ``_build_consecutive_periods_ctes`` (deleted + with the enrichment stack in DEV-1485). The typed implementation + differs from it in two principled ways: + + * The predicate-shape decision (boolean vs numeric) is read + from the TransformKey input shape (validated by + ``_validate_window_transform_ops_for_7b10``) rather than the + legacy ``predicate_is_boolean`` field. + * The inner aggregate is materialised in the base CTE as a + hidden slot (via the planner's ``_iter_slot_deps`` walk), so + the predicate text references that base alias directly — no + legacy ``_inner_`` step CTE needed. + """ + from slayer.core.keys import ( + AggregateKey, + ArithmeticKey, + ColumnKey, + ColumnSqlKey, + Phase, + TimeTruncKey, + TransformKey, + ) + + key = slot.key + if not isinstance(key, TransformKey) or key.op != "consecutive_periods": + raise ValueError( + f"expected consecutive_periods TransformKey, got " + f"{type(key).__name__} (op={getattr(key, 'op', None)!r})", + ) + inner_key = key.input + time_key = key.time_key + if not isinstance(time_key, TimeTruncKey): + raise ValueError( + f"consecutive_periods requires a TimeTruncKey time_key; " + f"got {type(time_key).__name__} (slot id={slot.id!r}).", + ) + + # Resolve the time-key alias. + time_sid = slot_id_by_key.get(time_key) + if time_sid is None or time_sid not in available_alias_by_slot_id: + raise RuntimeError( + f"consecutive_periods time_key not materialised: " + f"slot id={slot.id!r}.", + ) + time_alias = available_alias_by_slot_id[time_sid] + + # Build the predicate SQL referencing already-materialised base + # CTE aliases. Two shapes accepted by the validator: + # * Slottable leaf: numeric truthiness via IS NOT NULL AND <> 0. + # * Comparison ArithmeticKey: rendered + wrapped in COALESCE(, FALSE). + leaf_kinds = (ColumnKey, ColumnSqlKey, AggregateKey, TimeTruncKey) + if isinstance(inner_key, leaf_kinds): + input_sid = slot_id_by_key.get(inner_key) + if input_sid is None or input_sid not in available_alias_by_slot_id: + raise RuntimeError( + f"consecutive_periods input not materialised: " + f"slot id={slot.id!r}, input={inner_key!r}.", + ) + input_alias = available_alias_by_slot_id[input_sid] + predicate_sql = ( + f'{self._quote_ident(input_alias)} IS NOT NULL AND {self._quote_ident(input_alias)} <> 0' + ) + predicate_is_boolean = False + elif isinstance(inner_key, ArithmeticKey): + comparison_ops = {"==", "!=", "<", "<=", ">", ">=", "=", "<>"} + if inner_key.op not in comparison_ops: + raise NotImplementedError( + f"DEV-1450 stage 7b.11: composite-input transforms " + f"(layer op='consecutive_periods' input=" + f"ArithmeticKey op={inner_key.op!r}) are deferred to " + f"a follow-up slice (slot id={slot.id!r}).", + ) + rendered = self._render_value_key_against_aliases( + key=inner_key, + slot_id_by_key=slot_id_by_key, + available_alias_by_slot_id=available_alias_by_slot_id, + ) + predicate_sql = rendered.sql(dialect=self.dialect) + predicate_is_boolean = True + else: + raise NotImplementedError( + f"DEV-1450 stage 7b.11: consecutive_periods input " + f"{type(inner_key).__name__} not supported.", + ) + + # COALESCE / numeric wrap. + if predicate_is_boolean: + pred_in_case = f"COALESCE({predicate_sql}, FALSE)" + else: + pred_in_case = predicate_sql + + # Auto-partition by query dimensions (ColumnKey row-phase slots + # only — NOT TimeTruncKey, matching legacy). + partition_aliases: list[str] = [] + for sid in planned_query.projection: + row_slot = slots_by_id.get(sid) + if row_slot is None or row_slot.phase != Phase.ROW: + continue + if not isinstance(row_slot.key, ColumnKey): + continue + alias = available_alias_by_slot_id.get(sid) + if alias is not None: + partition_aliases.append(alias) + + # DEV-1692: a HIDDEN inner consecutive_periods slot's declared_name + # (``_consecutive_periods_inner``) is NOT unique across sibling slots — + # two would collide on ``full_slot_alias`` / ``cp_reset_alias`` and + # collapse downstream, the same failure mode fixed for time_shift. + # Allocate a unique internal alias for the hidden case; USER aliases + # (already unique) are left untouched. + if slot.public_aliases: + slot_alias = slot.public_aliases[0] + else: + slot_alias = cte_allocator.allocate_cte(slot.declared_name) + full_slot_alias = f"{source_relation}.{slot_alias}" + cp_reset_alias = f"_cp_reset_{full_slot_alias}" + + # Build the reset CTE. + prev_cte = ctes[-1][0] + carry_aliases_sorted = sorted( + a for aliases in aliases_by_slot_id.values() for a in aliases + ) + carry_select = ",\n ".join(self._quote_ident(a) for a in carry_aliases_sorted) + partition_clause = ( + _SQL_PARTITION_BY + ", ".join(self._quote_ident(a) for a in partition_aliases) + if partition_aliases + else "" + ) + over_reset = " ".join(p for p in ( + partition_clause, + f'ORDER BY {self._quote_ident(time_alias)}', + "ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW", + ) if p) + reset_window_sql = ( + f'SUM(CASE WHEN {pred_in_case} THEN 0 ELSE 1 END) ' + f'OVER ({over_reset}) AS {self._quote_ident(cp_reset_alias)}' + ) + cp_reset_cte_name = cte_allocator.allocate_cte(f"cp_reset_{slot_alias}") + cp_reset_sql = ( + _SQL_SELECT_HEAD + carry_select + + ",\n " + reset_window_sql + + f"\nFROM {prev_cte}" + ) + ctes.append((cp_reset_cte_name, cp_reset_sql)) + + # Build the value CTE — references the cp_reset CTE's added + # column in PARTITION BY so each run of true predicate is + # counted within its own reset group. + value_partition_aliases = partition_aliases + [cp_reset_alias] + value_partition_clause = _SQL_PARTITION_BY + ", ".join( + self._quote_ident(a) for a in value_partition_aliases + ) + over_value = " ".join(( + value_partition_clause, + f'ORDER BY {self._quote_ident(time_alias)}', + "ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW", + )) + # Outer CASE WHEN guarantees rows where the predicate is false + # surface as 0 (legacy parity). + value_inner_window_sql = ( + f'SUM(CASE WHEN {pred_in_case} THEN 1 ELSE 0 END) ' + f'OVER ({over_value})' + ) + value_outer_case = ( + f'CASE WHEN {pred_in_case} ' + f'THEN {value_inner_window_sql} ELSE 0 END ' + f'AS {self._quote_ident(full_slot_alias)}' + ) + cp_value_cte_name = cte_allocator.allocate_cte(f"cp_value_{slot_alias}") + cp_value_sql = ( + _SQL_SELECT_HEAD + carry_select + + ",\n " + value_outer_case + + f"\nFROM {cp_reset_cte_name}" + ) + ctes.append((cp_value_cte_name, cp_value_sql)) + + # Record the slot's alias for downstream lookups. + aliases_by_slot_id.setdefault(slot.id, []).append(full_slot_alias) + available_alias_by_slot_id.setdefault(slot.id, full_slot_alias) + + @staticmethod + def _pick_alias_for_planned_slot(*, slot, alias_index: dict) -> str: + """Pick the next alias for a slot in projection order. + + Mirrors ``stage_planner._emit_stage_schema``: per-slot index + picks the next ``public_aliases`` entry; falls back to + ``declared_name`` when the alias list is exhausted (kept + symmetric with the planner; unreachable for properly-interned + slots but defensive). + """ + idx = alias_index.setdefault(slot.id, 0) + if idx < len(slot.public_aliases): + alias = slot.public_aliases[idx] + else: + alias = slot.declared_name + alias_index[slot.id] = idx + 1 + return alias + + def _qualify_column_filter_sql( + self, + *, + canonical_sql: Optional[str], + source_relation: str, + source_model, + ) -> Optional[str]: + """Qualify bare-identifier column refs in a Mode-A filter fragment. + + ``Column.filter`` is Mode-A SQL like ``"status = 'paid'"``; + ``_build_agg`` wraps the aggregate argument as ``SUM(CASE WHEN + THEN col END)`` and inserts the filter text verbatim. + Without qualification, ``status`` resolves against the implicit + outermost scope at the agg-rendering site, which differs between + the host base CTE and a re-rooted cross-model CTE. Legacy + ``resolve_filter_columns`` qualifies bare refs to + ``.``; mirror that on the parsed AST so a + rerooted CTE renders the same ``customers.status = 'active'`` + the host base would render. + + Only bare ``exp.Column`` nodes (no table qualifier) whose name + matches a column on ``source_model`` get qualified. Already- + qualified refs (``other.col``) and function-call AST nodes pass + through unchanged. + """ + if not canonical_sql: + return None + try: + ast = self._parse_predicate(canonical_sql) + except Exception: + # Unparseable filter SQL — fall back to the raw text. The + # legacy path bubbled up the same shape (the enrichment + # parse failure surfaces at query time). + return canonical_sql + known_names = {c.name for c in source_model.columns} + for col in ast.find_all(exp.Column): + if col.args.get("table") is not None: + continue + ident = col.this + if not isinstance(ident, exp.Identifier): + continue + if ident.name in known_names: + col.set("table", exp.to_identifier(source_relation)) + return ast.sql(dialect=self.dialect) + + def _column_ref_is_derived( + self, *, col: exp.Column, source_model, source_relation: str, bundle, + include_dotted: bool, + ) -> bool: + """True iff a single ``exp.Column`` ref resolves to a non-trivial DERIVED + column — bare on ``source_model``, or (when ``include_dotted``) a dotted + ``__``-path alias resolving through ``bundle`` to a derived column on a + joined model. Dotted resolution starts from ``source_relation`` (the + alias the rest of the Mode-A path uses), not ``source_model.name``. + """ + if col.args.get("db") or col.args.get("catalog"): + return False + ident = col.this + if not isinstance(ident, exp.Identifier): + return False + tbl = col.args.get("table") + if tbl is None: + return self._is_nontrivial_derived(source_model, ident.name) + if not include_dotted: + return False + target, _ = _walk_path_to_target_sync( + source_model=source_model, + source_alias=source_relation, + table_alias=tbl.name, + resolve_model=bundle.get_referenced_model, + is_root=True, + ) + return target is not None and self._is_nontrivial_derived(target, ident.name) + + def _predicate_references_derived( + self, *, parsed: exp.Expression, source_model, source_relation: str, + bundle, include_dotted: bool, + ) -> bool: + """True iff the parsed Mode-A predicate references a non-trivial DERIVED + column (see :meth:`_column_ref_is_derived`). Drives whether the predicate + must be inline-expanded (vs. cheaply qualified). DEV-1494. + """ + return any( + self._column_ref_is_derived( + col=col, source_model=source_model, + source_relation=source_relation, bundle=bundle, + include_dotted=include_dotted, + ) + for col in parsed.find_all(exp.Column) + ) + + def _render_mode_a_predicate( + self, + *, + sql: Optional[str], + source_model, + source_relation: str, + bundle, + qualify_fallback, + include_dotted_derived: bool = True, + ) -> Optional[str]: + """Render a Mode-A predicate (``Column.filter`` / ``SlayerModel.filters``) + with DERIVED refs inline-expanded and base refs qualified — the shared + core for the column-filter and model-filter render paths (DEV-1494). + + If the predicate references a non-trivial derived column (bare, or — when + ``include_dotted_derived`` — a dotted ref to a derived column on a joined + model), it is inline-expanded via ``expand_derived_refs_sync`` so the + crossed joins resolve and no dangling ``.`` (a + non-physical column) survives — mirroring the query-level filter path. + Otherwise ``qualify_fallback(sql)`` does the cheap bare-ref qualification, + preserving each caller's exact non-derived output (regex for model + filters, AST for column filters). On sqlglot parse failure the predicate + falls through to ``qualify_fallback`` unchanged, so a dialect-specific + fragment never raises earlier than today. ``include_dotted_derived`` is + ``False`` for the cross-model ``_cm_*`` CTE target-filter path, which has + no mechanism to add a deeper join an expansion would cross (DEV-1503). + """ + if not sql: + return None + if bundle is not None: + try: + parsed = self._parse_predicate(sql) + except Exception: + parsed = None + if parsed is not None and self._predicate_references_derived( + parsed=parsed, source_model=source_model, + source_relation=source_relation, bundle=bundle, + include_dotted=include_dotted_derived, + ): + root = self._expand_degenerate_derived_root( + parsed=parsed, source_model=source_model, + source_relation=source_relation, bundle=bundle, + include_dotted_derived=include_dotted_derived, + ) + if root is not None: + return root + expanded = expand_derived_refs_sync( + sql=sql, + model=source_model, + alias_path=source_relation, + resolve_model=bundle.get_referenced_model, + dialect=self.dialect, + ) + if expanded is not None: + return expanded + return qualify_fallback(sql) + + def _expand_degenerate_derived_root( + self, *, parsed: exp.Expression, source_model, source_relation: str, + bundle, include_dotted_derived: bool, + ) -> Optional[str]: + """When the whole predicate IS a single derived-column ref + (``filter="is_eu"`` / ``filter="loss_payment.has_flag"``), expand it + directly — ``expand_derived_refs_sync`` rewrites refs via in-place + ``col.replace``, a no-op on the AST root. A dotted root is walked to its + target model + canonical ``__`` alias and expanded with ``is_root=False`` + so further-joined refs prefix correctly. Returns the expanded SQL, or + ``None`` when ``parsed`` is not such a derived single-column root. + """ + if not isinstance(parsed, exp.Column) or ( + parsed.args.get("db") or parsed.args.get("catalog") + ): + return None + tbl = parsed.args.get("table") + if tbl is None: + if self._is_nontrivial_derived(source_model, parsed.name): + return self._expand_derived_column_sql( + source_model=source_model, + source_relation=source_relation, + column_name=parsed.name, + bundle=bundle, + ) + return None + if not include_dotted_derived: + return None + target, canonical = _walk_path_to_target_sync( + source_model=source_model, + source_alias=source_relation, + table_alias=tbl.name, + resolve_model=bundle.get_referenced_model, + is_root=True, + ) + if ( + target is not None + and canonical is not None + and self._is_nontrivial_derived(target, parsed.name) + ): + return self._expand_derived_column_sql( + source_model=target, + source_relation=canonical, + column_name=parsed.name, + bundle=bundle, + is_root=False, + ) + return None + + def _filter_join_paths( + self, *, sql: Optional[str], source_relation: str, source_model, bundle, + ) -> List[Tuple[str, ...]]: + """Join paths a Mode-A filter (``Column.filter`` / ``SlayerModel.filters``) + needs (DEV-1494). + + Scans BOTH the un-inlined predicate — so a placeholder dotted ref + (``loss_payment.has_flag``, the dbt join-trigger idiom) keeps its alias + even when it inlines to a constant — AND the inline-expanded predicate — + so a bare/dotted DERIVED ref surfaces the joins its expansion crosses + (``is_eu`` → ``customers``; ``loss_payment.deep_flag`` → + ``loss_payment__claim``). The union is required because inlining drops the + placeholder alias while the raw form can't see a derived expansion's + crossed joins. Each parse is tolerant — an unparseable side yields no + paths rather than raising earlier than today. + """ + if not sql: + return [] + seen: set = set() + ordered: List[Tuple[str, ...]] = [] + + def _scan(text: Optional[str]) -> None: + if not text: + return + try: + parsed = self._parse_predicate(text) + except Exception: + return + for p in self._joined_paths_in_sql( + sql_expr=parsed, source_relation=source_relation, + source_model=source_model, bundle=bundle, + ): + if p not in seen: + seen.add(p) + ordered.append(p) + + _scan(sql) + rendered = self._render_mode_a_predicate( + sql=sql, source_model=source_model, source_relation=source_relation, + bundle=bundle, qualify_fallback=lambda s: s, + ) + if rendered is not None and rendered != sql: + _scan(rendered) + return ordered + + def _expand_derived_row_dims( # NOSONAR(S3776) — one cohesive per-slot pass expanding derived ROW/TIME dimensions and registering the joins they cross. + self, *, base_render_order, slots_by_id, source_relation: str, + source_model, bundle, scope: ScopeFrame, + ) -> Dict[str, exp.Expression]: + """Pre-expand derived (``ColumnSqlKey``) ROW dimensions and derived TIME + dimensions for the base SELECT: inline sibling/joined derived refs + (DEV-1333 / DEV-1410), register any joins their SQL crosses into + ``scope.join_paths`` (Law 1 — the join-discovery side effect), and return + the expanded-expr-by-slot-id map the render branch reads from. Extracted + from ``_build_base_select_for_planned``. + """ + from slayer.core.keys import ColumnSqlKey, Phase, TimeTruncKey + + def _add(path: Tuple[str, ...]) -> None: + if path: + scope.join_paths.add(path) + + derived_expr_by_sid: Dict[str, exp.Expression] = {} + for sid in base_render_order: + slot = slots_by_id.get(sid) + if slot is None or slot.phase != Phase.ROW: + continue + key = slot.key + # DEV-1450 #4a: a derived (ColumnSqlKey) TIME dimension expands the + # same way; pull any joins its SQL crosses into the FROM so the + # DATE_TRUNC over the expanded expression resolves. + if isinstance(key, TimeTruncKey) and isinstance(key.column, ColumnSqlKey): + raw = self._raw_time_col_expr_for_planned( + time_column=key.column, source_model=source_model, + source_relation=source_relation, bundle=bundle, + ) + # DEV-1701: register the join to the derived TD's OWNING model + # (``key.column.path``) plus every further join its expanded sql + # crosses — parity with the plain joined-derived-dimension branch + # below. ``is_root=False`` (in ``_raw_time_col_expr_for_planned``) + # already anchored the inner refs at the host-path alias, so the + # scan and the render agree. + _add(key.column.path) + for p in self._joined_paths_in_sql( + sql_expr=raw, source_relation=source_relation, + source_model=source_model, bundle=bundle, + ): + _add(p) + continue + if not isinstance(key, ColumnSqlKey): + continue + # Local refs (``path == ()``) expand rooted at the source relation; a + # CROSS-MODEL derived dim (``B.foo``, ``path == ("B",)``) expands + # rooted at the ``__``-path alias of the owning joined model, with + # ``is_root=False`` so a further-joined ref carries the full prefix + # (``B`` reaching ``C`` → ``B__C``). + if key.path: + owner_model = bundle.get_referenced_model(key.path[-1]) + if owner_model is None: + continue + owner_relation = "__".join(key.path) + else: + owner_model = source_model + owner_relation = source_relation + expanded_sql = self._expand_derived_column_sql( + source_model=owner_model, source_relation=owner_relation, + column_name=key.column_name, bundle=bundle, is_root=not key.path, + ) + col = next( + (c for c in owner_model.columns if c.name == key.column_name), None, + ) + expr = _wrap_cast_for_type( + self._parse(expanded_sql), col.type if col is not None else None, + ) + derived_expr_by_sid[sid] = expr + _add(key.path) # the join to the owning model itself (cross-model) + for p in self._joined_paths_in_sql( + sql_expr=expr, source_relation=source_relation, + source_model=source_model, bundle=bundle, + ): + _add(p) + return derived_expr_by_sid + + def _expand_column_filter_sql( + self, + *, + canonical_sql: Optional[str], + source_relation: str, + source_model, + bundle=None, + ) -> Optional[str]: + """Render a ``Column.filter`` Mode-A predicate for the aggregation-time + CASE-WHEN wrapper (``SUM(CASE WHEN THEN col END)``). Inlines + derived refs (bare or dotted-to-joined-derived) so the crossed joins + resolve; otherwise qualifies bare refs. DEV-1494; see + ``_render_mode_a_predicate``. + """ + if bundle is None: + return self._qualify_column_filter_sql( + canonical_sql=canonical_sql, + source_relation=source_relation, + source_model=source_model, + ) + return self._render_mode_a_predicate( + sql=canonical_sql, + source_model=source_model, + source_relation=source_relation, + bundle=bundle, + qualify_fallback=lambda s: self._qualify_column_filter_sql( + canonical_sql=s, + source_relation=source_relation, + source_model=source_model, + ), + ) + + def _build_from_clause_from_planned( + self, + *, + source_model, + source_relation: str, + ) -> exp.Expression: + if source_model.sql_table: + # DEV-1686 reserved-word alias + DEV-1645 mixed-case physical-name + # quoting via ``_to_table``. + return self._to_table(source_model.sql_table, alias=source_relation) + if source_model.sql: + return exp.Subquery( + this=self._parse(source_model.sql), + alias=exp.to_identifier(source_relation), + ) + raise NotImplementedError( + f"DEV-1450 stage 7b.12+: query-backed models (source_queries) " + f"deferred to multi-stage slices. Model " + f"{source_model.name!r} has neither sql_table nor sql set." + ) + + def _dim_column_expr_from_planned( + self, *, source_model, source_relation: str, leaf: str, + ) -> exp.Expression: + col = next( + (c for c in source_model.columns if c.name == leaf), None, + ) + if col is None: + raise ValueError( + f"Column {leaf!r} not found on model " + f"{source_model.name!r}", + ) + return self._resolve_sql( + sql=col.sql, name=col.name, model_name=source_relation, + type=col.type, + ) + + def _raw_time_col_expr_for_planned( + self, *, time_column, source_model, source_relation: str, bundle, + ) -> exp.Expression: + """Untruncated time expression for a ``TimeTruncKey.column`` + (DEV-1450 #4a), agnostic to base vs derived. + + * ``ColumnKey`` → the (possibly joined) bare column expression. + * ``ColumnSqlKey`` → the EXPANDED ``Column.sql``, rooted at the host + relation for a local derived column, or at the ``__``-path alias + for a joined one. The DATE_TRUNC is applied by the caller. + """ + from slayer.core.keys import ColumnKey, ColumnSqlKey + + if isinstance(time_column, ColumnKey): + return self._joined_or_local_dim_expr( + path=time_column.path, + leaf=time_column.leaf, + source_model=source_model, + source_relation=source_relation, + bundle=bundle, + ) + if isinstance(time_column, ColumnSqlKey): + if time_column.path: + joined_model = bundle.get_referenced_model(time_column.path[-1]) + if joined_model is None: + raise ValueError( + f"Time dimension references derived column " + f"{time_column.column_name!r} on joined model " + f"{time_column.path[-1]!r} which is not in the resolved " + f"source bundle.", + ) + # DEV-1701: a JOINED derived TIME dimension whose ``Column.sql`` + # crosses a FURTHER join must anchor its inner refs at the + # host-path alias (``customers_v2__regions``), not the bare + # direct-join alias (``regions``) — otherwise the host base + # SELECT references a table its FROM never joins. ``is_root= + # False`` carries the full ``__`` prefix, exactly as the plain + # joined-derived-dimension branch in ``_expand_derived_row_dims`` + # does. The ``continue``-less callers (base render, ranked + # subquery, default-time-col) all render in the host frame. + expanded_sql = self._expand_derived_column_sql( + source_model=joined_model, + source_relation="__".join(time_column.path), + column_name=time_column.column_name, + bundle=bundle, + is_root=False, + ) + else: + expanded_sql = self._expand_derived_column_sql( + source_model=source_model, + source_relation=source_relation, + column_name=time_column.column_name, + bundle=bundle, + ) + return self._parse(expanded_sql) + raise NotImplementedError( + f"Unsupported TimeTruncKey column type: {type(time_column).__name__}", + ) + + def _expand_derived_column_sql( + self, *, source_model, source_relation: str, column_name: str, bundle, + is_root: bool = True, + ) -> str: + """Expand a derived ``Column.sql`` (a ``ColumnSqlKey`` target) into a + fully-qualified SQL string, recursively inlining references to other + derived columns on the same model or on joined models (DEV-1333 / + DEV-1410). Bare identifiers qualify to ``source_relation``; joined + refs qualify to their ``__``-canonical path alias. + + ``is_root`` is ``False`` when the derived column lives on a JOINED + model (a cross-model derived dimension, ``source_relation`` being the + ``__``-path alias). A further-joined reference inside that column's + sql then resolves to the full path (``B`` reaching ``C`` → + ``B__C``), not the bare child alias. + + Synchronous: resolves join targets through ``bundle.get_referenced_ + model`` (every model is already loaded — P11). Returns the column's + own ``name`` when ``sql`` is unset (bare base column). + """ + col = next( + (c for c in source_model.columns if c.name == column_name), None, + ) + if col is None: + raise ValueError( + f"Derived column {column_name!r} not found on model " + f"{source_model.name!r}", + ) + if col.sql is None: + return col.name + expanded = expand_derived_refs_sync( + sql=col.sql, + model=source_model, + alias_path=source_relation, + resolve_model=bundle.get_referenced_model, + dialect=self.dialect, + is_root=is_root, + ) + return expanded if expanded is not None else col.sql + + def _joined_paths_in_sql( + self, *, sql_expr: exp.Expression, source_relation: str, source_model, + bundle, + ) -> List[Tuple[str, ...]]: + """Collect the join paths referenced by table qualifiers inside an + (already-expanded) SQL expression. + + Each ROOT-scope ``.`` whose ``alias`` is not the source + relation and fully resolves as a join walk on ``source_model`` + contributes its path prefixes (``a__b`` → ``("a",)`` and + ``("a", "b")``) so ``_build_from_and_joins`` pulls the LEFT JOINs into + the FROM. Aliases that don't resolve as a join path (CTE / subquery + aliases) are skipped, as are refs inside a nested scope (subquery / + set-op branch) — those belong to the inner rowset, not the outer FROM. + Prefixes are only emitted once the FULL alias path resolves, so a + partially-matching alias never injects a spurious outer join. + + Thin shim over the shared ``collect_root_scope_joined_paths`` helper + — see ``column_filter_paths._walk_root_scope_paths`` for the planner + side using the same primitive. + """ + return collect_root_scope_joined_paths( + parsed=sql_expr, + source_model=source_model, + source_relation=source_relation, + bundle=bundle, + ) + + def _resolve_where_filter_joins_via_scope( + self, *, planned_query, scope: ScopeFrame, + skip_filter_ids: Optional[Set[str]] = None, + filters_override: "Optional[List[Any]]" = None, + ) -> None: + """Register into ``scope.join_paths`` the joins every WHERE-phase filter + references (Law 1 — discovery is a side effect of resolving the filter + through the scope). A 1:1 replacement for the former + ``_collect_filter_join_paths`` (wrap-and-reuse, D-G): it delegates to the + same ``_value_key_join_paths`` / ``_filter_join_paths`` sub-scanners in + the same ``filters_by_phase`` order, so the base FROM stays byte-identical. + + Covers three shapes: + * typed joined column ref (``customers.regions.name == 'US'``) — + ``ColumnKey.path``; + * typed derived column whose ``Column.sql`` crosses a join + (``is_eu = 1`` where ``is_eu`` references ``customers.region``) — + ``ColumnSqlKey``, expanded then scanned; + * Mode-A ``SlayerModel.filters`` text with a ``__`` join path + (``customers__regions.name = 'EU'``) — parsed and scanned. + + Filters routed to a per-plan ``_cm_*`` CTE (``skip_filter_ids``) are + applied there, not on the host base, so their joins are not registered + here. + + ``filters_override`` (DEV-1732) replaces the filter list being scanned — + the windowed ``_src`` scope passes the SAME rewritten list it renders, so + discovery and rendering can never disagree about what the CTE contains. + """ + from slayer.core.keys import Phase + + skip = skip_filter_ids or set() + filters = ( + planned_query.filters_by_phase + if filters_override is None else filters_override + ) + for fp in filters: + if fp.phase != Phase.ROW or fp.id in skip: + continue + if fp.expression is not None: + for p in self._value_key_join_paths( + key=fp.expression.value_key, source_model=scope.root_model, + source_relation=scope.root_relation, bundle=scope.bundle, + ): + scope.join_paths.add(p) + elif fp.text is not None: + # DEV-1450 #4b / DEV-1494: discover joins from BOTH the + # un-inlined text (a placeholder dotted ref like + # ``loss_payment.has_flag`` keeps its alias even when it inlines + # to a constant) AND the inline-expanded text (a bare/dotted + # DERIVED ref like ``is_eu`` surfaces the join its expansion + # crosses). See ``_filter_join_paths``. + for p in self._filter_join_paths( + sql=fp.text, source_relation=scope.root_relation, + source_model=scope.root_model, bundle=scope.bundle, + ): + scope.join_paths.add(p) + + def _value_key_join_paths( # NOSONAR(S3776) — one cohesive recursive ValueKey-tree walk; complexity is the per-key-type dispatch. + self, *, key, source_model, source_relation: str, bundle, + ) -> List[Tuple[str, ...]]: + """Join paths a typed filter ``ValueKey`` tree references (DEV-1450 / + DEV-1475): a direct ``ColumnKey.path``; a derived ``ColumnSqlKey`` + (local or joined — expanded then scanned for the joins its ``sql`` + crosses); and recursively through ``ArithmeticKey`` / ``ScalarCallKey`` / + ``BetweenKey`` / ``InKey`` operands. Sub-scanner shared by + ``_resolve_where_filter_joins_via_scope``; ``_joined_paths_in_sql`` + already emits path prefixes, and ``ColumnKey.path`` prefixes are + expanded here. + """ + from slayer.core.keys import ( + ArithmeticKey, + BetweenKey, + ColumnKey, + ColumnSqlKey, + InKey, + ScalarCallKey, + ) + + out: List[Tuple[str, ...]] = [] + + def _add(path: Tuple[str, ...]) -> None: + for i in range(1, len(path) + 1): + prefix = tuple(path[:i]) + if prefix and prefix not in out: + out.append(prefix) + + def _scan(parsed: exp.Expression) -> None: + for p in self._joined_paths_in_sql( + sql_expr=parsed, source_relation=source_relation, + source_model=source_model, bundle=bundle, + ): + if p not in out: + out.append(p) + + def _derived_paths(*, model, relation, column_name) -> None: + _scan(self._parse(self._expand_derived_column_sql( + source_model=model, source_relation=relation, + column_name=column_name, bundle=bundle, + ))) + + def _walk(k) -> None: + if isinstance(k, ColumnKey): + _add(k.path) + elif isinstance(k, ColumnSqlKey): + # Joined derived ref also pulls the walk to its owning model. + _add(k.path) + model = ( + bundle.get_referenced_model(k.path[-1]) if k.path + else source_model + ) + if model is not None: + _derived_paths( + model=model, + relation="__".join(k.path) if k.path else source_relation, + column_name=k.column_name, + ) + elif isinstance(k, ArithmeticKey): + for o in k.operands: + _walk(o) + elif isinstance(k, ScalarCallKey): + for a in k.args: + _walk(a) + elif isinstance(k, BetweenKey): + _walk(k.column) + _walk(k.low) + _walk(k.high) + elif isinstance(k, InKey): + # DEV-1475: only the LHS column of an IN can carry a join path. + _walk(k.column) + + _walk(key) + return out + + def _resolve_aggregation_def( + self, + *, + key, + source_model, + src_leaf: str, + ): + """Look up the model-level ``Aggregation`` definition for ``key.agg``, + if any. Returns the matched ``Aggregation`` or ``None``. + + The lookup runs for built-ins too (a user model is allowed to + override default params for a built-in, e.g. supply a default + ``weight=`` for ``weighted_avg``), and ``_resolve_agg_param`` + relies on that override surfacing in + ``AggRenderSpec.aggregation_def``. Only when the name is NOT a + built-in does a lookup miss raise — an unknown non-built-in is a + hard error. + """ + agg_def = next( + (a for a in (source_model.aggregations or []) if a.name == key.agg), + None, + ) + if agg_def is None and key.agg not in _BUILTIN_BAREARG_AGGS_LOCAL_SLICE: + raise AggregationNotAllowedError( + column=src_leaf, + agg=key.agg, + reason=( + f"unknown aggregation {key.agg!r} — not a built-in " + f"and not defined in {source_model.name!r}." + f"aggregations." + ), + ) + return agg_def + + def _validate_aggregate_kwarg_paths( + self, + *, + key, + source, + src_leaf: str, + ) -> None: + """Reject CROSS-MODEL aggregates' kwarg column refs whose join path + disagrees with the aggregate source path. + + For a target-rooted aggregate, a kwarg path that doesn't match the + source path after reroot prefix-stripping would silently bind the + kwarg to a different model than the aggregate value column — + meaningless SQL semantically; any residual mismatch surfaces here. + + DEV-1709: LOCAL aggregates (``source.path == ()``) are exempt — a + structurally-crossing kwarg (``weighted_avg(weight=customers.w)``) + is now a supported crossing INPUT: the widened Law-3 trigger + isolates the aggregate host-rooted, and inside that CTE's + sub-render the kwarg resolves through the host scope (join + registration + path-aliased emission). Both bare-column + (``ColumnKey``) and derived-column (``ColumnSqlKey``) kwarg + refs go through this gate (CodeRabbit fold-in on PR #144). + """ + from slayer.core.keys import ColumnKey, ColumnSqlKey + + if not source.path: + return + for kname, kval in key.kwargs: + if isinstance(kval, (ColumnKey, ColumnSqlKey)) and kval.path != source.path: + raise AggregationNotAllowedError( + column=src_leaf, + agg=key.agg, + reason=( + f"kwarg {kname!r} references " + f"{type(kval).__name__} with path {kval.path!r}; " + f"aggregate source path is {source.path!r}. " + f"Cross-model kwargs must share the source's " + f"join path." + ), + ) + + def _build_agg_render_spec_from_planned( # NOSONAR(S3776) — sequential isinstance dispatch over StarKey / ColumnKey / ColumnSqlKey with helper extractions for aggregation-def lookup, kwarg path validation, and explicit-time-arg resolution. Further splitting would scatter the per-source-kind contract. + self, + *, + slot, + key, + source_model, + source_relation: str, + full_alias: str, + bundle=None, + resolved_agg_kwargs: "Optional[Dict[str, ResolvedAggKwarg]]" = None, + ) -> AggRenderSpec: + """Build an ``AggRenderSpec`` from a planned aggregate slot so + ``_build_agg`` / ``_resolve_sql`` / ``_wrap_cast_for_type`` emit + dialect-correct SQL without forking the agg-emission codebase. + + Replaces the legacy ``_build_agg_render_spec_from_planned`` + adapter (DEV-1452 Stage A). Mirrors ``enrichment.py:431`` + ``sql = column.sql or column.name`` so ``COUNT(*)`` (StarKey source) + and ``COUNT(col)`` (ColumnKey source with sql=None on a bare column) + take their distinct branches inside ``_build_agg``. + """ + from slayer.core.keys import ColumnKey, ColumnSqlKey, StarKey + + # ``slot`` may be ``None`` when this spec is built for a HAVING term + # whose aggregate isn't a declared projection slot; the result type is + # then unknown (no outer CAST needed for a comparison operand). + slot_type = slot.type if slot is not None else None + source = key.source + if isinstance(source, StarKey): + # Legacy enrichment (enrichment.py:~388) rejects any + # non-count aggregation on ``*`` — e.g. ``*:sum`` or + # ``*:median`` would otherwise plan and render as + # ``SUM(*)`` / ``MEDIAN(*)``, which is meaningless. + # Mirror that rejection here so the typed pipeline can't + # silently emit invalid SQL (Codex MEDIUM fold-in). + if key.agg != "count": + raise ValueError( + f"Aggregation {key.agg!r} not allowed with measure " + f"'*' — use '*:count' for COUNT(*)." + ) + if key.args or key.kwargs: + raise ValueError( + f"'*:count' takes no args or kwargs; got " + f"args={key.args!r}, kwargs={key.kwargs!r}." + ) + return AggRenderSpec( + name="", + sql=None, + aggregation=key.agg, + alias=full_alias, + model_name=source_relation, + type=slot_type, + ) + if isinstance(source, (ColumnKey, ColumnSqlKey)): + # ColumnKey is a bare / trivial column (``sql`` None or a bare + # identifier remap); ColumnSqlKey is a derived column (``Column.sql`` + # set to a non-trivial expression — ``amount * 2``). Both resolve + # the same way: look up the column on the model and aggregate + # ``col.sql`` (the derived expression) or ``col.name`` (bare). + src_leaf = ( + source.leaf + if isinstance(source, ColumnKey) + else source.column_name + ) + # ``first`` / ``last`` aggregations rank rows via a ROW_NUMBER + # subquery (built in ``_build_ranked_subquery_from_planned``) and + # pick ``rn = 1`` through ``MAX(CASE WHEN _rn = 1 THEN col END)``. + # An explicit positional arg (``latest_amount:last(created_at)`` + # or ``…:last(derived_time_col)``) overrides the query's default + # ranking time column; the helper handles both bare-column + # (``ColumnKey``) and derived-column (``ColumnSqlKey``) args. + explicit_time_col = self._resolve_explicit_time_col( + key=key, + source_model=source_model, + source_relation=source_relation, + bundle=bundle, + ) + agg_def = self._resolve_aggregation_def( + key=key, source_model=source_model, src_leaf=src_leaf, + ) + self._validate_aggregate_kwarg_paths( + key=key, source=source, src_leaf=src_leaf, + ) + col = next( + (c for c in source_model.columns if c.name == src_leaf), + None, + ) + if col is None: + raise ValueError( + f"Aggregate source column {src_leaf!r} not found " + f"on model {source_model.name!r}", + ) + # DEV-1452 Stage B — for derived (``ColumnSqlKey``) aggregate + # sources, the inner bare refs in ``Column.sql`` must qualify + # to ``source_relation`` (legacy enrichment did this pre-CAST + # via ``_enrich``'s derived-ref expansion; the typed pipeline + # never invoked the expander on aggregate sources, so the + # rendered SQL kept bare ``amount`` where it should be + # ``orders.amount``). + if ( + isinstance(source, ColumnSqlKey) + and col.sql is not None + and bundle is not None + ): + sql_text = self._expand_derived_column_sql( + source_model=source_model, + source_relation=source_relation, + column_name=col.name, + bundle=bundle, + ) + else: + sql_text = col.sql if col.sql else col.name + # DEV-1527: a column-ref kwarg (``weight=`` / ``other=``) + # that the pre-FROM scope pass resolved is embedded as a trusted + # ``kind="expr"`` expression (its join already base-pulled); every + # other kwarg (scalar / string / a column-ref on a path this call + # has no scope for — e.g. the cross-model CTE build) canonical- + # stringifies as before and coerces to ``kind="str"`` via the + # ``AggRenderSpec`` before-validator (guarded downstream by + # ``_validate_agg_param_value`` / ``_SAFE_AGG_PARAM_RE``). + resolved_kw = resolved_agg_kwargs or {} + agg_kwargs_str = { + k: (resolved_kw[k] if k in resolved_kw else agg_kwarg_canonical_str(v)) + for k, v in key.kwargs + } + # DEV-1450 stage 7b.12: propagate ``AggregateKey.column_filter_key`` + # into ``AggRenderSpec.filter_sql`` so ``_build_agg`` wraps the + # aggregate argument as ``SUM(CASE WHEN THEN col END)``. + # Legacy ``resolve_filter_columns`` qualifies bare-identifier refs + # in the filter with the host model name (so ``status = 'paid'`` + # becomes ``orders.status = 'paid'``); mirror that here on the + # parsed AST so dialect-independent wiring works in the new + # pipeline. + filter_sql = self._expand_column_filter_sql( + canonical_sql=( + key.column_filter_key.canonical_sql + if key.column_filter_key is not None + else None + ), + source_relation=source_relation, + source_model=source_model, + bundle=bundle, + ) + return AggRenderSpec( + name=col.name, + sql=sql_text, + aggregation=key.agg, + alias=full_alias, + model_name=source_relation, + type=slot_type, + column_type=col.type, + filter_sql=filter_sql, + agg_kwargs=agg_kwargs_str, + aggregation_def=agg_def, + time_column=explicit_time_col, + ) + raise NotImplementedError( + f"AggregateKey source {type(source).__name__} not supported.", + ) + + def _build_where_having_from_planned( # NOSONAR(S3776) — one cohesive pass over filters_by_phase routing each entry to WHERE / HAVING / POST by phase, with the per-carrier (typed vs Mode-A text) rendering and the HAVING grouped-column guard inline. The complexity is pre-existing; DEV-1732 added only the `filters_override` list selection. Splitting the phase routing from the rendering would thread slot_by_key / first_last_state / where_parts / having_parts through helpers without simplifying anything. + self, + *, + planned_query, + source_relation: str, + source_model, + bundle, + skip_filter_ids: Optional[Set[str]] = None, + first_last_state: Optional[FirstLastRenderState] = None, + aliases_by_slot_id: Optional[Dict[str, List[str]]] = None, + filters_override: "Optional[List[Any]]" = None, + ): + """``filters_override`` (DEV-1732) replaces ``filters_by_phase`` as the + list being rendered — see ``_effective_src_filters``.""" + from slayer.core.keys import Phase + + skip = skip_filter_ids or set() + # key -> slot map so a HAVING term's local AggregateKey renders as the + # same aggregate expression the base SELECT emits. + slot_by_key: Dict[Any, Any] = { + s.key: s + for s in ( + list(planned_query.row_slots) + + list(planned_query.aggregate_slots) + + list(planned_query.combined_expression_slots) + ) + } + where_parts: list[str] = [] + having_parts: list[str] = [] + filters = ( + planned_query.filters_by_phase + if filters_override is None else filters_override + ) + for fp in filters: + if fp.id in skip: + # DEV-1450 stage 7b.12: filters routed into a per-plan + # cross-model CTE (where_filter_ids / having_filter_ids) + # are rendered there; the host base must not double- + # apply them. + continue + if fp.phase == Phase.POST: + # 7b.10: POST-phase filters are handled in the outer + # wrapper by ``_render_post_phase_filter_conditions`` + # (after the CTE chain, before pagination). Skip them + # here so the base WHERE doesn't try to render them. + continue + if fp.phase not in (Phase.ROW, Phase.AGGREGATE): + raise NotImplementedError( + f"DEV-1450 stage 7b.10+: unsupported filter phase " + f"{fp.phase!r}. filter id={fp.id!r}." + ) + # AGGREGATE-phase filters referencing a LOCAL aggregate render as a + # HAVING clause; a cross-model aggregate ref raises inside the + # value-key walker (it routes via the per-plan CTE instead). + target_parts = ( + having_parts if fp.phase == Phase.AGGREGATE else where_parts + ) + if fp.phase == Phase.AGGREGATE and fp.expression is not None: + # A HAVING that references a bare (non-aggregated) row column + # which is NOT in the query's GROUP BY would emit invalid SQL + # (``HAVING orders.status = 'x'`` with status ungrouped). Reject + # early with the legacy phrasing. + grouped = { + s.key + for s in planned_query.row_slots + if s.id in set(planned_query.projection) + } + for ck in self._direct_local_column_keys(fp.expression.value_key): + if ck not in grouped: + raise ValueError( + f"Filter references column {ck.leaf!r} in a HAVING " + f"(aggregate) predicate, but it is not in the " + f"query's dimensions / GROUP BY." + ) + if fp.expression is not None: + # Typed predicate (Mode-B DSL or planner-emitted + # BetweenKey) — render through the value-key walker. + # DEV-1501: thread ``first_last_state`` so HAVING + # aggregates reference the same ``_first_rn`` / + # ``_last_rn{suffix}`` columns the base SELECT projects, + # AND thread ``aliases_by_slot_id`` so the synth's + # ``full_alias`` matches the materialised spec's alias — + # required for ``filtered_rn_map`` / ``filtered_match_map`` + # lookups (which are keyed by the full alias the + # ranked-subquery builder used). + rendered = self._render_value_key_for_filter( + key=fp.expression.value_key, + source_relation=source_relation, + source_model=source_model, + bundle=bundle, + slot_by_key=slot_by_key, + first_last_state=first_last_state, + aliases_by_slot_id=aliases_by_slot_id, + ) + # Match the legacy DSL parser, which wraps top-level + # boolean expressions in parens — legacy WHERE for a + # compound filter emits ``WHERE (a AND b)`` rather than + # ``WHERE a AND b``. Wrapping at the top level only (not + # recursively) reproduces legacy output without affecting + # single-comparison or single-BETWEEN filters. + if isinstance(rendered, (exp.And, exp.Or)): + rendered = exp.Paren(this=rendered) + target_parts.append(rendered.sql(dialect=self.dialect)) + elif fp.text is not None: + # Mode-A SQL filter (SlayerModel.filters) — qualify bare + # column refs with the source relation, mirroring + # legacy `_build_where_and_having` at generator.py:2566. + # DEV-1450 #4b: a reference to a non-trivial derived column + # is inline-expanded (and pulls its crossed joins into the + # FROM via _resolve_where_filter_joins_via_scope). + target_parts.append(self._render_model_filter_sql( + sql=fp.text, + columns=fp.text_columns, + source_model=source_model, + source_relation=source_relation, + bundle=bundle, + )) + else: + raise ValueError( + f"FilterPhase id={fp.id!r} has neither expression " + f"nor text (planner gap).", + ) + + where_clause = None + if where_parts: + where_clause = self._parse_predicate(_SQL_AND_JOINER.join(where_parts)) + having_clause = None + if having_parts: + having_clause = self._parse_predicate(_SQL_AND_JOINER.join(having_parts)) + return where_clause, having_clause + + @staticmethod + def _qualify_mode_a_sql_filter( + *, + sql: str, + columns, + source_model, + source_relation: str, + ) -> str: + """Qualify bare-identifier column references in a Mode-A SQL + filter — mirrors legacy ``_build_where_and_having`` at + ``slayer/sql/generator.py:2566-2580``. + + For each name in ``columns``: + * Already-dotted refs are left alone (``orders.id`` stays). + * Non-identifier tokens (SQL keywords picked up by the regex + extractor) are left alone. + * Bare identifiers matching a model column name are rewritten + to ``.``. The negative lookbehind + ``(? bool: + """True iff ``name`` is a column on ``model`` whose ``Column.sql`` is a + non-trivial expression (set, and not just a bare-identifier remap).""" + col = next((c for c in model.columns if c.name == name), None) + return col is not None and col.sql is not None and not _is_trivial_base( + column=col, + ) + + def _render_model_filter_sql( + self, + *, + sql: str, + columns, + source_model, + source_relation: str, + bundle, + ) -> str: + """Render a ``SlayerModel.filters`` Mode-A SQL predicate (DEV-1450 #4b / + DEV-1494). + + Inlines references to non-trivial derived columns — bare on + ``source_model`` or dotted-to-a-derived-column-on-a-joined-model — so the + crossed joins resolve; otherwise qualifies bare base refs via the regex + path (``_qualify_mode_a_sql_filter``), byte-identical to legacy. Thin + wrapper over ``_render_mode_a_predicate``. + """ + rendered = self._render_mode_a_predicate( + sql=sql, + source_model=source_model, + source_relation=source_relation, + bundle=bundle, + qualify_fallback=lambda s: self._qualify_mode_a_sql_filter( + sql=s, + columns=columns, + source_model=source_model, + source_relation=source_relation, + ), + ) + return rendered if rendered is not None else sql + + def _render_value_key_for_filter( # NOSONAR(S3776) — sequential isinstance dispatch over the closed filter-ValueKey union. Each branch carries the per-type filter-render contract (local vs joined column qualification, derived-column expansion, aggregate-with-rn-state synth, etc.); extracting per-branch helpers would scatter the contract. + self, + *, + key, + source_relation: str, + source_model, + bundle, + slot_by_key: Optional[Dict[Any, Any]] = None, + first_last_state: Optional[FirstLastRenderState] = None, + aliases_by_slot_id: Optional[Dict[str, List[str]]] = None, + ) -> exp.Expression: + """Render a ValueKey tree to sqlglot for WHERE / HAVING rendering. + + Supports ``ColumnKey`` (local AND joined ``path != ()`` — emitted as + ``<__path_alias>.``; the join is pulled into the FROM by + ``_resolve_where_filter_joins_via_scope``), ``ColumnSqlKey`` (derived column — + expanded inline, sibling/joined refs resolved), ``LiteralKey``, + ``ArithmeticKey``, ``ScalarCallKey``, ``BetweenKey``, and a LOCAL + ``AggregateKey`` (for HAVING — rendered as the bare aggregate + expression so it works on dialects that reject SELECT aliases in + HAVING). Cross-model aggregate refs (``path != ()``) and + ``TransformKey`` / ``TimeTruncKey`` are deferred to later slices. + """ + from decimal import Decimal + + from slayer.core.keys import ( + AggregateKey, + ArithmeticKey, + BetweenKey, + ColumnKey, + ColumnSqlKey, + InKey, + LiteralKey, + ScalarCallKey, + StarKey, + TimeTruncKey, + TransformKey, + ) + + if isinstance(key, AggregateKey): + # HAVING term: render the aggregate as its expression (``COUNT(*)``, + # ``SUM(amount)``), not the SELECT alias — Postgres rejects output + # aliases in HAVING. Cross-model aggregates (non-empty source path) + # are routed into a per-plan CTE instead (handled by the caller). + if getattr(key.source, "path", ()): + raise NotImplementedError( + f"DEV-1450 stage 7b.12: cross-model aggregate ref in " + f"filter (path={key.source.path!r}) routes via the " + f"per-plan CTE, not inline HAVING." + ) + slot = (slot_by_key or {}).get(key) + # DEV-1501 Group A.2: when the slot was materialised in the + # base SELECT, ``_build_filtered_rn_columns`` keyed its + # ``filtered_rn_map`` / ``filtered_match_map`` by the FULL + # ALIAS the materialised spec used. The HAVING synth must + # reuse the same alias — bare placeholder ``__having_ref__`` + # would miss the lookup and fall back to the unfiltered + # ``_last_rn`` + raw ``filter_sql``. + having_full_alias = "__having_ref__" + if ( + aliases_by_slot_id is not None + and slot is not None + and aliases_by_slot_id.get(slot.id) + ): + having_full_alias = aliases_by_slot_id[slot.id][0] + # DEV-1527: resolve this local aggregate's column-ref kwargs + # (``weighted_avg(weight=)`` / ``corr(other=)``) through a + # host scope so a derived/crossing kwarg renders its expanded, join- + # anchored expression HERE too — matching the base SELECT — instead of + # collapsing to a bare, non-existent name. The crossed join is already + # base-pulled by ``_resolve_agg_inputs_via_scope`` (this HAVING + # aggregate is also a ``base_render_order`` slot), so the throwaway + # scope is used only to reproduce the same anchored expression. + having_kwargs = self._resolve_agg_kwargs_for_key( + key=key, source_model=source_model, + source_relation=source_relation, bundle=bundle, + ) + synth = self._build_agg_render_spec_from_planned( + slot=slot, + key=key, + source_model=source_model, + source_relation=source_relation, + full_alias=having_full_alias, + bundle=bundle, + resolved_agg_kwargs=having_kwargs, + ) + # DEV-1501: thread the rn suffix maps from the base SELECT + # so a HAVING reference to a hidden first/last aggregate + # binds to the same ``_first_rn`` / ``_last_rn{suffix}`` + # column the base projects (instead of bare ``_last_rn``, + # which collapses distinct time-column specs). + rn_suffix_map = ( + first_last_state.rn_suffix_map if first_last_state else None + ) + default_time_col = ( + first_last_state.default_time_col_sql + if first_last_state + else None + ) + filtered_rn_map = ( + first_last_state.filtered_rn_map if first_last_state else None + ) + filtered_match_map = ( + first_last_state.filtered_match_map if first_last_state else None + ) + agg_expr, _is_agg = self._build_agg( + synth, + rn_suffix_map=rn_suffix_map, + default_time_col=default_time_col, + filtered_rn_map=filtered_rn_map, + filtered_match_map=filtered_match_map, + ) + return agg_expr + + if isinstance(key, ColumnKey): + if key.path != (): + # Joined column ref (``customers.regions.name``) — emit the + # ``__``-canonical path alias (``customers__regions.name``). + # The join is pulled into the FROM by + # ``_resolve_where_filter_joins_via_scope``. + return exp.Column( + this=exp.to_identifier(key.leaf), + table=exp.to_identifier("__".join(key.path)), + ) + col = next( + (c for c in source_model.columns if c.name == key.leaf), + None, + ) + if col is None: + raise ValueError( + f"Filter references column {key.leaf!r} which is " + f"not found on model {source_model.name!r}", + ) + return self._resolve_sql( + sql=col.sql, + name=col.name, + model_name=source_relation, + type=col.type, + ) + if isinstance(key, ColumnSqlKey): + if key.path != (): + # Joined derived-column ref (``policy_amount.premium.has_premium``). + # Expand the column's ``sql`` rooted at the JOINED model, + # qualifying bare refs to the ``__``-canonical path alias; the + # join itself is pulled into the FROM by + # ``_resolve_where_filter_joins_via_scope`` (which adds ``key.path``). + joined_model = bundle.get_referenced_model(key.path[-1]) + if joined_model is None: + raise ValueError( + f"Filter references derived column {key.column_name!r} " + f"on joined model {key.path[-1]!r} which is not in the " + f"resolved source bundle.", + ) + path_alias = "__".join(key.path) + expanded_sql = self._expand_derived_column_sql( + source_model=joined_model, + source_relation=path_alias, + column_name=key.column_name, + bundle=bundle, + ) + col = next( + (c for c in joined_model.columns if c.name == key.column_name), + None, + ) + return _wrap_cast_for_type( + self._parse(expanded_sql), + _filter_cast_type(col.type if col is not None else None), + ) + # Derived column (``Column.sql`` set) — expand inline, resolving + # sibling / joined derived refs and pulling crossed joins into the + # FROM (via ``_resolve_where_filter_joins_via_scope``). + expanded_sql = self._expand_derived_column_sql( + source_model=source_model, + source_relation=source_relation, + column_name=key.column_name, + bundle=bundle, + ) + col = next( + (c for c in source_model.columns if c.name == key.column_name), + None, + ) + return _wrap_cast_for_type( + self._parse(expanded_sql), + _filter_cast_type(col.type if col is not None else None), + ) + if isinstance(key, LiteralKey): + return self._scalar_to_sqlglot(key.value) + if isinstance(key, ArithmeticKey): + operands = [ + self._render_value_key_for_filter( + key=o, + source_relation=source_relation, + source_model=source_model, + bundle=bundle, + slot_by_key=slot_by_key, + first_last_state=first_last_state, + aliases_by_slot_id=aliases_by_slot_id, + ) + for o in key.operands + ] + return self._build_arithmetic_for_filter( + op=key.op, operands=operands, + ) + if isinstance(key, ScalarCallKey): + args = [] + for a in key.args: + if isinstance(a, (Decimal, str, bool)) or a is None: + args.append(self._scalar_to_sqlglot(a)) + else: + args.append(self._render_value_key_for_filter( + key=a, + source_relation=source_relation, + source_model=source_model, + bundle=bundle, + slot_by_key=slot_by_key, + first_last_state=first_last_state, + aliases_by_slot_id=aliases_by_slot_id, + )) + if key.name == "like": + return exp.Like(this=args[0], expression=args[1]) + # DEV-1576: a 2-arg ROUND needs the Postgres numeric cast, so it + # must be a TYPED node (exp.Round) routed through the target-dialect + # rewrite. Only ROUND is retyped: the string-hygiene functions + # (substr / concat / lower / ...) must emit literally as written + # (DEV-1484), which exp.func would break by transpiling them per + # dialect — so they stay as Anonymous passthrough. + typed = exp.func(key.name.upper(), *args) + if isinstance(typed, exp.Round): + return self._finalize_scalar_call(typed) + return exp.Anonymous(this=key.name.upper(), expressions=args) + if isinstance(key, BetweenKey): + col_expr = self._render_value_key_for_filter( + key=key.column, + source_relation=source_relation, + source_model=source_model, + bundle=bundle, + slot_by_key=slot_by_key, + first_last_state=first_last_state, + aliases_by_slot_id=aliases_by_slot_id, + ) + low_expr = self._render_value_key_for_filter( + key=key.low, + source_relation=source_relation, + source_model=source_model, + bundle=bundle, + slot_by_key=slot_by_key, + first_last_state=first_last_state, + aliases_by_slot_id=aliases_by_slot_id, + ) + high_expr = self._render_value_key_for_filter( + key=key.high, + source_relation=source_relation, + source_model=source_model, + bundle=bundle, + slot_by_key=slot_by_key, + first_last_state=first_last_state, + aliases_by_slot_id=aliases_by_slot_id, + ) + return exp.Between(this=col_expr, low=low_expr, high=high_expr) + if isinstance(key, InKey): + # DEV-1475: render the LHS column through the normal filter + # path (local + joined paths both supported via ColumnKey / + # ColumnSqlKey), and the RHS as a sequence of scalar + # literals. Wrap in ``exp.Not`` for ``not in``. + col_expr = self._render_value_key_for_filter( + key=key.column, + source_relation=source_relation, + source_model=source_model, + bundle=bundle, + slot_by_key=slot_by_key, + first_last_state=first_last_state, + aliases_by_slot_id=aliases_by_slot_id, + ) + value_exprs = [ + self._scalar_to_sqlglot(lit.value) for lit in key.values + ] + in_expr = exp.In(this=col_expr, expressions=value_exprs) + return exp.Not(this=in_expr) if key.negated else in_expr + if isinstance(key, ( + AggregateKey, TransformKey, TimeTruncKey, StarKey, + )): + raise NotImplementedError( + f"DEV-1450 stage 7b.10+: filter rendering for " + f"{type(key).__name__} deferred to later slice." + ) + raise NotImplementedError( + f"Unsupported ValueKey type in filter: {type(key).__name__}", + ) + + def _render_filter_for_outer_wrapper( # NOSONAR(S3776) — sequential isinstance dispatch over the closed filter-ValueKey union for the DEV-1503 outer-WHERE wrapper. Mirrors ``_render_value_key_for_filter`` shape but substitutes slot refs with the combined-SELECT's table-qualified columns (``_cm_*`` / ``_base``); per-branch helpers would scatter the substitution contract. + self, + *, + key, + slot_by_key: Dict[Any, Any], + cross_model_agg_slot_to_cm: Dict[str, Tuple[str, str]], + aliases_by_slot_id: Dict[str, List[str]], + ) -> exp.Expression: + """Render a ValueKey tree for the DEV-1503 outer combined-SELECT WHERE. + + Used when an AGGREGATE-phase host filter references a filtered-local + isolated aggregate (``loss_payment_amt:sum > 1000`` where + ``loss_payment_amt`` has a join-crossing ``Column.filter``). The + filtered aggregate lives in a ``_cm_*`` CTE that LEFT JOINs back to + ``_base``; the outer combined SELECT is non-aggregating, so the + comparison renders as plain WHERE on the joined-back column rather + than HAVING-into-the-CTE (which would surface host rows as NULL + instead of dropping them). + + Slot-bearing leaves resolve via: + + * Isolated aggregate slot → ``.""`` from + ``cross_model_agg_slot_to_cm`` (the CTE's emitted aggregate column). + * Any other slot (row column, joined dim, local aggregate operand) + → ``_base.""`` from ``aliases_by_slot_id``. The + generator's aux-slot pass (``_add_local_aux_slots(aggregates_only= + True)``) promotes non-isolated aggregate operands into + ``base_render_order`` so this lookup always succeeds. + + Cross-model aggregates with ``path != ()`` (forward-path) are not + expected here — the planner routes those via plan-level + ``where_filter_ids`` / ``having_filter_ids`` instead. + """ + from decimal import Decimal + + from slayer.core.keys import ( + AggregateKey, + ArithmeticKey, + BetweenKey, + ColumnKey, + ColumnSqlKey, + InKey, + LiteralKey, + ScalarCallKey, + StarKey, + TimeTruncKey, + TransformKey, + ) + + def _slot_alias_column(slot) -> Optional[exp.Expression]: + sid = slot.id + cm_entry = cross_model_agg_slot_to_cm.get(sid) + if cm_entry is not None: + cte_name, agg_col_alias = cm_entry + return exp.Column( + this=exp.to_identifier(agg_col_alias, quoted=True), + table=exp.to_identifier(cte_name), + ) + aliases = aliases_by_slot_id.get(sid) or [] + if not aliases: + return None + return exp.Column( + this=exp.to_identifier(aliases[0], quoted=True), + table=exp.to_identifier("_base"), + ) + + if isinstance(key, AggregateKey): + slot = slot_by_key.get(key) + if slot is not None: + resolved = _slot_alias_column(slot) + if resolved is not None: + return resolved + raise NotImplementedError( + f"DEV-1503 outer-WHERE wrapper: AggregateKey " + f"{key!r} has no slot/alias resolution. " + f"Forward-path cross-model aggregates route via plan " + f"where/having ids instead.", + ) + if isinstance(key, (ColumnKey, ColumnSqlKey, TimeTruncKey)): + slot = slot_by_key.get(key) + if slot is not None: + resolved = _slot_alias_column(slot) + if resolved is not None: + return resolved + raise NotImplementedError( + f"DEV-1503 outer-WHERE wrapper: {type(key).__name__} " + f"{key!r} has no base alias — operand promotion did not " + f"materialise it in ``_base``.", + ) + if isinstance(key, LiteralKey): + return self._scalar_to_sqlglot(key.value) + if isinstance(key, ArithmeticKey): + operands = [ + self._render_filter_for_outer_wrapper( + key=o, + slot_by_key=slot_by_key, + cross_model_agg_slot_to_cm=cross_model_agg_slot_to_cm, + aliases_by_slot_id=aliases_by_slot_id, + ) + for o in key.operands + ] + return self._build_arithmetic_for_filter( + op=key.op, operands=operands, + ) + if isinstance(key, ScalarCallKey): + args = [] + for a in key.args: + if isinstance(a, (Decimal, str, bool)) or a is None: + args.append(self._scalar_to_sqlglot(a)) + else: + args.append(self._render_filter_for_outer_wrapper( + key=a, + slot_by_key=slot_by_key, + cross_model_agg_slot_to_cm=cross_model_agg_slot_to_cm, + aliases_by_slot_id=aliases_by_slot_id, + )) + if key.name == "like": + return exp.Like(this=args[0], expression=args[1]) + # DEV-1576: a 2-arg ROUND needs the Postgres numeric cast, so it + # must be a TYPED node (exp.Round) routed through the target-dialect + # rewrite. Only ROUND is retyped: the string-hygiene functions + # (substr / concat / lower / ...) must emit literally as written + # (DEV-1484), which exp.func would break by transpiling them per + # dialect — so they stay as Anonymous passthrough. + typed = exp.func(key.name.upper(), *args) + if isinstance(typed, exp.Round): + return self._finalize_scalar_call(typed) + return exp.Anonymous(this=key.name.upper(), expressions=args) + if isinstance(key, BetweenKey): + col_expr = self._render_filter_for_outer_wrapper( + key=key.column, + slot_by_key=slot_by_key, + cross_model_agg_slot_to_cm=cross_model_agg_slot_to_cm, + aliases_by_slot_id=aliases_by_slot_id, + ) + low_expr = self._render_filter_for_outer_wrapper( + key=key.low, + slot_by_key=slot_by_key, + cross_model_agg_slot_to_cm=cross_model_agg_slot_to_cm, + aliases_by_slot_id=aliases_by_slot_id, + ) + high_expr = self._render_filter_for_outer_wrapper( + key=key.high, + slot_by_key=slot_by_key, + cross_model_agg_slot_to_cm=cross_model_agg_slot_to_cm, + aliases_by_slot_id=aliases_by_slot_id, + ) + return exp.Between(this=col_expr, low=low_expr, high=high_expr) + if isinstance(key, InKey): + col_expr = self._render_filter_for_outer_wrapper( + key=key.column, + slot_by_key=slot_by_key, + cross_model_agg_slot_to_cm=cross_model_agg_slot_to_cm, + aliases_by_slot_id=aliases_by_slot_id, + ) + value_exprs = [ + self._scalar_to_sqlglot(lit.value) for lit in key.values + ] + in_expr = exp.In(this=col_expr, expressions=value_exprs) + return exp.Not(this=in_expr) if key.negated else in_expr + if isinstance(key, (TransformKey, StarKey)): + raise NotImplementedError( + f"DEV-1503 outer-WHERE wrapper: filter rendering for " + f"{type(key).__name__} not supported on outer wrapper.", + ) + raise NotImplementedError( + f"DEV-1503 outer-WHERE wrapper: unsupported ValueKey type " + f"{type(key).__name__}", + ) + + @staticmethod + def _direct_local_column_keys(key) -> "List[Any]": + """Local ``ColumnKey``s that appear as DIRECT (non-aggregated) operands + of a predicate tree — used to reject a HAVING that compares an + ungrouped row column. The walk stops at ``AggregateKey`` / + ``TransformKey`` (their inner columns are aggregated, not grouped). + """ + from slayer.core.keys import ( + AggregateKey, + ArithmeticKey, + BetweenKey, + ColumnKey, + InKey, + ScalarCallKey, + TransformKey, + ) + + out: List[Any] = [] + + def _walk(k) -> None: + if isinstance(k, ColumnKey): + if k.path == (): + out.append(k) + return + if isinstance(k, (AggregateKey, TransformKey)): + return # inner refs are aggregated / windowed, not grouped + if isinstance(k, ArithmeticKey): + for o in k.operands: + _walk(o) + elif isinstance(k, ScalarCallKey): + for a in k.args: + _walk(a) + elif isinstance(k, BetweenKey): + _walk(k.column) + _walk(k.low) + _walk(k.high) + elif isinstance(k, InKey): + # DEV-1475: only the LHS column can be a direct local + # row-column; literal RHS values aren't grouped against. + _walk(k.column) + + _walk(key) + return out + + @staticmethod + def _scalar_to_sqlglot(v) -> exp.Expression: + from decimal import Decimal + + if v is None: + return exp.Null() + if isinstance(v, bool): + return exp.Boolean(this=v) + if isinstance(v, Decimal): + return exp.Literal.number(str(v)) + if isinstance(v, str): + return exp.Literal.string(v) + raise NotImplementedError( + f"Unsupported scalar in filter: type={type(v).__name__} " + f"value={v!r}", + ) + + @staticmethod + def _paren_if_binary(node: exp.Expression) -> exp.Expression: + """DEV-1539: wrap a multi-term operand in ``(...)`` when it is a + ``Binary`` (arithmetic ``a + b``, or an ``AND``/``OR`` connector) so a + surrounding comparator's precedence is explicit by inspection, not only + by SQL operator-precedence rules — ``(a + b) > 7``, not ``a + b > 7``. + Bare columns, literals, function calls, and already-enclosed forms + (``CAST(...)`` / ``Paren``) are not ``Binary`` and pass through.""" + return exp.Paren(this=node) if isinstance(node, exp.Binary) else node + + @staticmethod + def _build_arithmetic_for_filter( # NOSONAR(S3776) — sequential per-operator dispatch (==/!= → EQ/NEQ, comparison, arithmetic) with DEV-1539 precedence paren-wrapping; each branch is the per-op emission contract. + *, op: str, operands: list, + ) -> exp.Expression: + # DSL ``==``/``!=`` map to sqlglot EQ/NEQ; sqlglot then emits the + # dialect-correct SQL operator (postgres ``=``/``!=``). DEV-1539: a + # multi-term comparison operand is parenthesised so its precedence is + # explicit (``(a + b) > 7`` / ``x = (a OR b)``). + _cmp = { + "==": exp.EQ, "=": exp.EQ, "!=": exp.NEQ, "<>": exp.NEQ, + "<": exp.LT, "<=": exp.LTE, ">": exp.GT, ">=": exp.GTE, + } + cmp_cls = _cmp.get(op) + if cmp_cls is not None: + return cmp_cls( + this=SQLGenerator._paren_if_binary(operands[0]), + expression=SQLGenerator._paren_if_binary(operands[1]), + ) + if op == "+": + # Unary plus is a no-op; legacy never emits it explicitly. + if len(operands) == 1: + return operands[0] + return exp.Add(this=operands[0], expression=operands[1]) + if op == "-": + # Unary minus: the binder represents ``-x`` / ``-10`` as + # ``ArithmeticKey(op="-", operands=(x,))`` — handle the + # single-operand form so a filter like ``amount > -10`` + # doesn't crash with IndexError. + if len(operands) == 1: + return exp.Neg(this=operands[0]) + return exp.Sub( + this=SQLGenerator._paren_if_lower_prec( + operands[0], parent_prec=1, is_right=False, op="-", + ), + expression=SQLGenerator._paren_if_lower_prec( + operands[1], parent_prec=1, is_right=True, op="-", + ), + ) + if op == "*": + return exp.Mul( + this=SQLGenerator._paren_if_lower_prec( + operands[0], parent_prec=2, is_right=False, op="*", + ), + expression=SQLGenerator._paren_if_lower_prec( + operands[1], parent_prec=2, is_right=True, op="*", + ), + ) + if op == "/": + return exp.Div( + this=SQLGenerator._paren_if_lower_prec( + operands[0], parent_prec=2, is_right=False, op="/", + ), + expression=SQLGenerator._paren_if_lower_prec( + operands[1], parent_prec=2, is_right=True, op="/", + ), + ) + if op == "and": + result = operands[0] + for o in operands[1:]: + result = exp.And(this=result, expression=o) + return result + if op == "or": + result = operands[0] + for o in operands[1:]: + result = exp.Or(this=result, expression=o) + return result + if op == "not": + return exp.Not(this=operands[0]) + # ``IS`` / ``IS NOT`` (Codex round 2): the filter normalizer lowers + # SQL ``IS NULL`` / ``IS NOT NULL`` to Python ``is None`` / ``is + # not None``. Render against the rhs (a ``Null`` literal) as the + # standard SQL forms. Without these branches a local-stage filter + # ``deleted_at IS NULL`` parses and binds but raises here at SQL + # generation. Mirrors the patches in ``_build_arith_or_cmp_ast`` + # and ``_compose_arithmetic_op``. + if op == "is": + return exp.Is(this=operands[0], expression=operands[1]) + if op == "is not": + return exp.Not(this=exp.Is(this=operands[0], expression=operands[1])) + raise NotImplementedError( + f"DEV-1450 stage 7b.8: ArithmeticKey op {op!r} not " + f"supported in filter rendering." + ) + + def _build_outer_trim_wrap_sql( + self, + *, + base_select: exp.Select, + planned_query, + source_relation: str, + aliases_by_slot_id: Dict[str, List[str]], + slots_by_id: Dict[str, Any], + bundle, + ) -> str: + """DEV-1501 — wrap a no-transform base SELECT in an outer SELECT + that projects ONLY the public projection slots (trimming hidden + materialised aggregates from the result), then moves ORDER BY / + LIMIT / OFFSET to the outer level so they reference the full + materialised aliases. + + Same shape as the transform path's outer wrap minus the step CTE + chain. Preserves C13 duplicate-public-alias semantics by walking + ``planned_query.projection`` slot-by-slot and cycling aliases per + slot (mirroring the transform path's ``outer_alias_index``). + + Built via sqlglot AST + ``.sql(dialect=…)`` so identifier quoting + is dialect-correct (Postgres / SQLite / DuckDB / ClickHouse use + ``"…"``; MySQL uses backticks). String-built quoted identifiers + would silently degrade to string literals on MySQL. + """ + public_aliases: list[str] = [] + outer_alias_index: Dict[str, int] = {} + for sid in planned_query.projection: + slot = slots_by_id[sid] + if slot.hidden: + continue + all_aliases = aliases_by_slot_id.get(sid, []) + if not all_aliases: + continue + idx = outer_alias_index.setdefault(sid, 0) + alias = ( + all_aliases[idx] if idx < len(all_aliases) else all_aliases[-1] + ) + outer_alias_index[sid] = idx + 1 + public_aliases.append(alias) + + outer_select = exp.Select() + for alias in public_aliases: + outer_select = outer_select.select( + exp.Column(this=exp.to_identifier(alias, quoted=True)), + ) + outer_select = outer_select.from_( + exp.Subquery(this=base_select, alias=exp.to_identifier("_outer")), + ) + + # Outer ORDER BY references each order entry's materialised alias + # — the first alias per slot is canonical (C13-duplicate aliases + # of a single slot share the same column value). Reuse + # ``_apply_order_limit_from_planned`` to apply ORDER BY / LIMIT / + # OFFSET so the dialect-aware sqlglot emission path is shared. + return self._apply_order_limit_from_planned( + select=outer_select, + planned_query=planned_query, + source_relation=source_relation, + slots_by_id=slots_by_id, + source_model=None, + bundle=bundle, + aliases_by_slot_id=aliases_by_slot_id, + ).sql(dialect=self.dialect, pretty=True) + + def _apply_order_limit_from_planned( # NOSONAR(S3776) — per-order-entry slot-kind dispatch (hidden materialised aggregate vs hidden NYI vs declared public alias) plus LIMIT/OFFSET tail. Each branch is the per-kind resolution contract; extracting helpers would scatter the alias-lookup chain. + self, + *, + select: exp.Select, + planned_query, + source_relation: str, + slots_by_id: dict, + source_model=None, + bundle=None, + aliases_by_slot_id: Optional[Dict[str, List[str]]] = None, + ) -> exp.Select: + """ORDER BY entries reference slot ids — resolve to the slot's + public or materialised alias and emit ``ORDER BY + "source_relation.alias" ASC|DESC`` (quoted-identifier form). + + DEV-1501: hidden AggregateKey slots that have been MATERIALISED + in the base SELECT (via Change 2's aggregate-only walk over + order/filter deps) resolve to their materialised full alias from + ``aliases_by_slot_id``. This is called either on the inner base + SELECT (when no outer wrap is needed) or on the outer wrap (when + hidden materialised columns are trimmed). In the outer-wrap + path, the inner subquery exposes the materialised alias as a + column the outer SELECT can reference by quoted identifier. + + Hidden ROW / TransformKey / cross-model targets remain + unsupported (``Change 2``'s ``aggregates_only=True`` keeps row + targets out of ``base_render_order``, preserving today's + ``NotImplementedError``). + """ + from slayer.core.keys import ( + AggregateKey, + ArithmeticKey, + ColumnKey, + ColumnSqlKey, + ScalarCallKey, + TimeTruncKey, + TransformKey, + ) + + # DEV-1733: the EXACT set of hidden key kinds that resolve to a + # materialised alias. Deliberately enumerated rather than "any hidden + # slot that happens to carry an alias" — a hidden ROW slot with an + # alias must still hit the split-emission / invariant branches below, + # never be ordered on as a bare column that is not in the GROUP BY. + _MATERIALISED_ORDER_KINDS = ( + AggregateKey, ArithmeticKey, ScalarCallKey, TransformKey, + ) + + for order_entry in planned_query.order: + slot = slots_by_id.get(order_entry.slot_id) + if slot is None: + continue + if slot.hidden: + # DEV-1501: hidden AGGREGATE slots are now materialised + # in the base SELECT (Change 2). Resolve to the + # materialised full alias from ``aliases_by_slot_id`` and + # reference it by quoted identifier — identical shape to + # the non-hidden public-alias branch below. + aliases = ( + aliases_by_slot_id.get(slot.id, []) + if aliases_by_slot_id is not None + else [] + ) + if aliases and isinstance(slot.key, _MATERIALISED_ORDER_KINDS): + full_alias = aliases[0] + order_col = exp.Column( + this=exp.to_identifier(full_alias, quoted=True), + ) + ascending = order_entry.direction == "asc" + select = select.order_by( + self._ordered(order_col, ascending=ascending), + ) + continue + # DEV-1712 (Law 2, split emission): a hidden ROW column ordered + # in an UNGROUPED query. The plan-time order validation + # (``plan_query``) guarantees the only hidden ROW slot that + # reaches here is a bare column in a query with no GROUP BY — + # grouped row columns are rejected or MAX-wrapped up front, and + # aggregates take the branch above. Emit a SPLIT + # ``.`` reference (mixed-case-aware) against + # the base FROM scope, identical to how the column would render + # if it were a projected dimension. + # + # DEV-1703 Phase 1: a JOINED column is emitted the same way, + # under its ``__`` path alias (``customers__regions.name``). The + # row IS the grain in an ungrouped query, so the bare reference + # is legal; Law 1 pulls the crossed join into the base FROM (see + # ``_collect_joined_paths_for_base``, which walks order targets). + key = slot.key + row_key = key.column if isinstance(key, TimeTruncKey) else key + if ( + source_model is not None + and isinstance(row_key, ColumnKey) + ): + order_col = self._joined_or_local_dim_expr( + path=row_key.path, leaf=row_key.leaf, + source_model=source_model, + source_relation=source_relation, bundle=bundle, + ) + ascending = order_entry.direction == "asc" + select = select.order_by( + self._ordered(order_col, ascending=ascending), + ) + continue + # A LOCAL DERIVED column (``ColumnSqlKey``, path empty): resolve + # its ``Column.sql`` through a throwaway host scope. That both + # anchors the expansion AND surfaces whether the SQL crosses a + # join. A hidden order-only derived column is NOT projected, so + # its join was never pulled into the base FROM — ordering on it + # would reference an unbound table. Reject that (project it), + # rather than emit invalid SQL; a non-crossing derived column + # (e.g. a bare mixed-case identifier) orders on its expression. + if ( + source_model is not None + and bundle is not None + and isinstance(row_key, ColumnSqlKey) + and not row_key.path + ): + # Detect join crossing via a throwaway scope (register-only); + # the resolved expr is discarded — its expansion lacks the + # DEV-1645 mixed-case quoting the planned-dim helper applies. + allocator = self._new_allocator() + scope = ScopeFrame( + scope_id=allocator.next_scope_id(source_relation), + root_model=source_model, + root_relation=source_relation, + bundle=bundle, + dialect=self._dialect, + allocator=allocator, + ) + scope.resolve(row_key) + if scope.join_paths: + # The derived column IS local (``orders.cust_region``); + # it merely depends on an unpulled join. Report its own + # qualified name, not a fabricated ``customers.cust_region``. + raise UnresolvableOrderColumnError( + column=row_key.column_name, qualifier=source_relation, + ) + # Non-crossing local derived column — emit through the + # planned-dim helper so the expansion is quoted identically + # to a projected dimension (mixed-case-safe). + order_col = self._joined_or_local_dim_expr( + path=(), leaf=row_key.column_name, + source_model=source_model, + source_relation=source_relation, bundle=bundle, + ) + ascending = order_entry.direction == "asc" + select = select.order_by( + self._ordered(order_col, ascending=ascending), + ) + continue + # Defensive: any other hidden shape should have been rejected at + # plan time (transform / composite / joined / grouped-row). + raise NotImplementedError( + f"ORDER BY references a hidden slot (id={slot.id!r}, key=" + f"{type(slot.key).__name__}) that was not resolved at plan " + f"time — this is an internal invariant violation." + ) + # DEV-1713: resolve to the SAME full alias the projection emits — + # a joined ROW dimension projects under the DOTTED result key + # (``orders.customers.regions.name``), so the ORDER BY must match + # it, not the flat ``declared_name`` (``customers__regions__name``), + # which would name a column the SELECT never projects. + full_alias = self._full_alias_for_slot( + slot=slot, source_relation=source_relation, alias_index={}, + ) + order_col = exp.Column( + this=exp.to_identifier(full_alias, quoted=True), + ) + ascending = order_entry.direction == "asc" + select = select.order_by( + self._ordered(order_col, ascending=ascending), + ) + + if planned_query.limit is not None: + select = select.limit(planned_query.limit) + if planned_query.offset is not None: + select = select.offset(planned_query.offset) + + return select + + +# =========================================================================== +# DEV-1450 stage 7b.8 — module-level shim entry point. +# =========================================================================== + + +def generate_from_planned( + planned_query, + *, + bundle, + dialect: str = "postgres", +) -> str: + """Render a ``PlannedQuery`` to SQL. + + Module-level entry point: constructs an ``SQLGenerator`` for the + requested dialect and delegates to the instance method, which + reuses the legacy dialect helpers (``_resolve_sql`` / + ``_build_agg`` / ``_wrap_cast_for_type`` / ``_parse_predicate``) + so dialect-specific behavior is rendered identically to the + legacy ``SQLGenerator.generate()`` path. + + Stage 7b.8 scope: single-model queries with dimensions, local + aggregates, Mode-B row filters, ORDER BY, LIMIT/OFFSET, and dim- + only deduplication. Cross-model aggregates, time dimensions, + window transforms, self-join CTE transforms, and HAVING-phase + filters raise ``NotImplementedError`` with a stage marker so + silent parity drift is impossible (slices 7b.9–7b.13 land each + behavior in turn). + """ + return SQLGenerator(dialect=dialect).generate_from_planned( + planned_query, bundle=bundle, + ) + + +def _bundle_for_stage(planned_query, bundle, schema_by_name): + """Pick the per-stage bundle a single DAG stage renders against. + + The stage's host model comes from the planner (``render_source_model`` — + the stage's OWN source / overlay / synthetic-over-sibling) so the + generator's FROM / joins bind against exactly what the binder used. A + StageSchema chain stage carries no ``render_source_model``; the generator + builds a synthetic model over the upstream CTE. Either way, synthetic + models for the OTHER sibling stages are threaded into ``referenced_models`` + so a join / cross-model ref that targets a sibling resolves to its CTE. + + A plain single-model query (no upstream schema, no render model) renders + against the original bundle unchanged. + """ + ds = (bundle.source_model.data_source if bundle.source_model else "") or "_stage" + relation = planned_query.source_relation + if planned_query.render_source_model is not None: + source = planned_query.render_source_model + elif relation in schema_by_name: + source = synthetic_model_from_stage_schema( + name=relation, schema=schema_by_name[relation], data_source=ds, + ) + else: + return bundle + sibling_schemas = {n: s for n, s in schema_by_name.items() if n != relation} + return stage_bundle_with_siblings( + bundle=bundle, source_model=source, + sibling_schemas=sibling_schemas, data_source=ds, + ) + + +def generate_planned_stages( + planned_queries, + *, + bundle, + dialect: str = "postgres", +) -> str: + """Render a multi-stage DAG (``plan_stages`` output) to one SQL string. + + Each non-root stage becomes a CTE ``() AS ()``; + the column-alias list flattens the stage's result-key projection + (``orders.amount_sum``) to the flat names downstream stages bound against + (``amount_sum``), so no per-stage rename wrapper is needed. The root + stage is the outer SELECT and carries the public result keys. Stage CTEs + are prepended to any CTEs the root already emits (cross-model / transform + stages), since the root reads ``FROM ``. + + ``planned_queries`` is the topo-ordered list from ``plan_stages`` (root + last). A single-stage list delegates straight to ``generate_from_planned``. + """ + if not planned_queries: + raise ValueError("generate_planned_stages requires at least one stage") + if len(planned_queries) == 1: + # DEV-1716: single-stage DB-bound terminal — apply the dialect alias + # mangling post-pass (BigQuery / T-SQL; identity otherwise). + sql = generate_from_planned( + planned_queries[0], bundle=bundle, dialect=dialect, + ) + sql = get_dialect(dialect).rewrite_emitted_sql(sql) + # DEV-1705: validate the final POST-mangle, pre-RLS statement (env-gated). + maybe_validate_scopes(sql, dialect=dialect) + return sql + + schema_by_name = { + p.stage_schema.relation_name: p.stage_schema + for p in planned_queries + if p.stage_schema is not None + } + + # (cte_name, rename-wrapped stage AST) in dependency order. + stage_ctes: List[Tuple[str, exp.Expression]] = [] + root_sql: Optional[str] = None + for planned in planned_queries: + stage_bundle = _bundle_for_stage(planned, bundle, schema_by_name) + stage_sql = generate_from_planned( + planned, bundle=stage_bundle, dialect=dialect, + ) + if planned is planned_queries[-1]: + root_sql = stage_sql + continue + if planned.stage_schema is None: + raise ValueError( + "non-root stage must carry a stage_schema for CTE chaining; " + f"source_relation={planned.source_relation!r}", + ) + stage_ctes.append(( + planned.stage_schema.relation_name, + _stage_rename_wrapper( + planned=planned, stage_sql=stage_sql, dialect=dialect, + ), + )) + + assert root_sql is not None + root_ast = sqlglot.parse_one(root_sql, dialect=dialect) + + # The root may already carry CTEs (cross-model / transform stages emit + # ``WITH base AS ...``). Those read FROM the stage relations, so the + # stage CTEs must come FIRST. ``Select.with_`` appends; build the order + # explicitly: clear the root's own CTEs, add the stage CTEs (dependency + # order), then re-append the root's original CTEs. + existing_with = root_ast.args.get("with_") + existing_ctes = ( + list(existing_with.expressions) if existing_with is not None else [] + ) + if existing_with is not None: + root_ast.set("with_", None) + + for name, wrapped in stage_ctes: + root_ast = root_ast.with_(name, as_=wrapped, dialect=dialect) + for cte in existing_ctes: + root_ast = root_ast.with_(cte.args["alias"], as_=cte.this, dialect=dialect) + + # DEV-1716: terminal emit of the multi-stage root — apply the dialect + # rewrite_emitted_sql post-pass (BigQuery / T-SQL alias mangling; identity + # otherwise). The re-parse/with_ grafting above can surface dotted aliases + # the per-stage emits already mangled, so mangle once more here + # (idempotent) to catch the root's own projection. + sql = root_ast.sql(dialect=dialect, pretty=True) + sql = get_dialect(dialect).rewrite_emitted_sql(sql) + # DEV-1705: validate the final POST-mangle, pre-RLS multi-stage root + # (env-gated). One validation per final terminal (single- vs multi-stage). + maybe_validate_scopes(sql, dialect=dialect) + return sql + + +def _stage_rename_wrapper(*, planned, stage_sql, dialect): + """Wrap a rendered intermediate-stage SQL so its output columns are the + flat names downstream stages bound against. + + Thin adapter around :func:`slayer.sql.stage_wrapper.build_flat_rename_wrapper` + (DEV-1452 Stage B decision B) — pulls ``source_relation`` and the + expected StageSchema column names off the ``PlannedQuery`` and forwards + to the shared helper. The migrated ``_expand_query_backed_model`` path + calls the helper directly with names derived from the typed plan. + """ + return build_flat_rename_wrapper( + source_relation=planned.source_relation, + stage_sql=stage_sql, + expected_columns=[c.name for c in planned.stage_schema.columns], + dialect=dialect, + ) diff --git a/slayer/sql/naming.py b/slayer/sql/naming.py new file mode 100644 index 00000000..6b579790 --- /dev/null +++ b/slayer/sql/naming.py @@ -0,0 +1,347 @@ +"""DEV-1706 Stage 2 — minimal, collision-safe alias allocator. + +A single ``AliasAllocator`` is created per top-level ``generate_from_planned`` +call and threaded to every ``ScopeFrame`` built during that call. It mints: + +* ``_val_`` materialisation aliases (Law 2 — projection-boundary columns), +* CTE names, + +seeded from every name already in scope (bundle relations, ``__``-path join +aliases, public projection aliases, model names) so a minted name can never +collide with a user column, a path alias, or a reserved public alias. It also +hands out generation-local ``ScopeFrame`` ids. + +The allocator is the *minimal* collision primitive (subsumes DEV-1692's +collision check). DEV-1713 Stage 9 grew this module into the single owner of +every alias / result-key decision: + +* :func:`result_key` / :func:`result_key_from_alias` — the DOTTED user-facing + FINAL-stage keys (``orders.customers.regions.name``); +* :func:`flat_name` — the ``__``-joined INNER-stage downstream schema names + (``customers__regions__name``, the StageSchema bind contract); +* :func:`encode_alias` / :func:`decode_alias` — the BigQuery / T-SQL dotted + alias mangling bijection (DEV-1571), relocated here from the dialect package; +* :func:`quote_mixed_case_identifiers` / :func:`maybe_quote_ident` — the + DEV-1645 mixed-case identifier-quoting policy, relocated here from the + generator; +* :func:`assert_unique_cte_names` — the DEV-1692 per-``WITH``-scope CTE + name-collision belt. +""" + +from __future__ import annotations + +from typing import Optional, Tuple + +import sqlglot +from pydantic import BaseModel, ConfigDict, PrivateAttr +from sqlglot import exp + +# --------------------------------------------------------------------------- +# Dialect case-folding policy (DEV-1726). +# +# SLayer-minted names (CTE families, ``_val_`` materialisation aliases) are +# emitted unquoted, so on case-folding backends two names differing only in +# case fold to the same identifier — two user measure aliases ``Foo``/``foo`` +# both driving time_shift CTEs would produce a duplicate ``WITH`` name. The +# policy of WHICH sqlglot dialects fold lives HERE (naming policy, per the +# Stage-9 ownership decision) because this module must stay an import leaf — +# dialect modules import from it. +# +# Membership notes (confirmed against vendor docs, sqlglot's +# NORMALIZATION_STRATEGY, and — for SQLite/DuckDB — empirically): +# * BigQuery FOLDS: GoogleSQL's case-sensitivity table marks "aliases within +# a query" (which CTE names are) case-insensitive; only real table/dataset +# names are case-sensitive. This corrects the DEV-1726 issue text. +# * SQLite and DuckDB reject case-differing CTE names even when QUOTED. +# * MySQL / T-SQL fold DELIBERATELY despite platform/collation dependence: +# folding is rename-only-safe (every reference uses the allocated name), +# while not folding leaves the collision live on the majority configs +# (Windows/macOS MySQL, default-collation SQL Server). +# * ClickHouse identifiers are case-sensitive — exact comparison. +# * Unknown dialect strings compare exact (previous behavior, fail-safe). +# +# The fold KEY is ``str.lower()`` — parity with sqlglot's +# ``normalize_identifier``; ``str.casefold()`` would over-equate (``ß``→``ss``). +# --------------------------------------------------------------------------- + +CASE_FOLDING_SQLGLOT_DIALECTS: frozenset[str] = frozenset({ + "postgres", "redshift", "snowflake", "oracle", "mysql", "tsql", + "sqlite", "duckdb", "trino", "presto", "databricks", "spark", "bigquery", +}) + +# Explicit, so "does not fold" is a decision, not an omission: every registry +# dialect must appear in exactly one of the two sets (pinned by +# tests/test_dev1726_cte_case_folding.py against the dialect registry). +KNOWN_CASE_SENSITIVE_SQLGLOT_DIALECTS: frozenset[str] = frozenset({"clickhouse"}) + + +def dialect_folds_case(dialect: str) -> bool: + """True iff ``dialect`` case-folds unquoted identifiers (CTE names in + particular). Input is normalized via ``strip().lower()``; an unknown + dialect string returns False (exact comparison — fail-safe).""" + return dialect.strip().lower() in CASE_FOLDING_SQLGLOT_DIALECTS + + +class AliasAllocator(BaseModel): + """Per-generation collision-safe name allocator (mutable). + + With ``folds_case=True`` (case-folding dialects — DEV-1726, set via + :func:`dialect_folds_case` by ``SQLGenerator._new_allocator``), every + ``_taken`` comparison folds with ``str.lower()`` while names are still + returned in the caller's original case — so ``shifted_Foo`` blocks + ``shifted_foo`` and the second mint walks to ``shifted_foo_2``. + """ + + model_config = ConfigDict(arbitrary_types_allowed=True) + + # Fold every _taken comparison with str.lower() (DEV-1726). Comparison + # only: allocated names keep the caller's original case. + folds_case: bool = False + + # External names the allocator must avoid (user columns, join aliases, + # public projection aliases, model names). Stored FOLDED when folds_case. + # NOSONAR lines below: Pydantic v2 PrivateAttr idiom — the annotation is the + # attribute's runtime type after model init; the ``PrivateAttr(...)`` sentinel + # is replaced by Pydantic. S5890 can't model this and is a false positive. + _reserved: set[str] = PrivateAttr(default_factory=set) # NOSONAR(S5890) + # Names already handed out by this allocator. + _used: set[str] = PrivateAttr(default_factory=set) # NOSONAR(S5890) + # Monotonic ``_val_`` cursor (never reset per scope, so sibling scopes + # in one generation cannot mint the same ``_val_0``). + _val_seq: int = PrivateAttr(default=0) # NOSONAR(S5890) + # Monotonic scope-id cursor. + _scope_seq: int = PrivateAttr(default=0) # NOSONAR(S5890) + + def _fold(self, name: str) -> str: + """The comparison key: ``str.lower()`` when folding, else identity.""" + return name.lower() if self.folds_case else name + + def reserve(self, *names: str) -> None: + """Mark ``names`` as taken so they are never allocated.""" + self._reserved.update(self._fold(n) for n in names) + + def _taken(self, name: str) -> bool: + key = self._fold(name) + return key in self._reserved or key in self._used + + def allocate(self, preferred: str) -> str: + """Return ``preferred`` if free, else ``preferred_2``, ``preferred_3``, …""" + candidate = preferred + suffix = 2 + while self._taken(candidate): + candidate = f"{preferred}_{suffix}" + suffix += 1 + self._used.add(self._fold(candidate)) + return candidate + + def allocate_val(self) -> str: + """Return the next free ``_val_`` materialisation alias.""" + while True: + candidate = f"_val_{self._val_seq}" + self._val_seq += 1 + if not self._taken(candidate): + self._used.add(self._fold(candidate)) + return candidate + + def allocate_cte(self, preferred: str) -> str: + """Return a collision-safe CTE name (same walk as :meth:`allocate`).""" + return self.allocate(preferred) + + def next_scope_id(self, root_relation: str) -> str: + """Return a generation-local ``ScopeFrame`` id, ``#``. + + Ephemeral — used only for in-generation materialisation dedup; it is + never emitted into SQL, result keys, or persisted state (D-F / Codex L1). + """ + scope_id = f"{root_relation}#{self._scope_seq}" + self._scope_seq += 1 + return scope_id + + +# --------------------------------------------------------------------------- +# Result-key / flat-name builders (DEV-1713 Stage 9). +# +# A query renders as either a FINAL stage (its columns are the user-facing +# result keys — DOTTED, ``orders.customers.regions.name``) or an INNER stage +# of a multi-stage DAG (its columns are downstream bind names — ``__``-joined, +# ``customers__regions__name``). These two builders are the single owners of +# those two forms; the planner's is-final flag picks between them so the two +# can never mix (D3 / DEV-1495 bug 1). +# --------------------------------------------------------------------------- + + +def result_key(*, source_relation: str, path: Tuple[str, ...] = (), leaf: str) -> str: + """Build the DOTTED final-stage result key from STRUCTURED parts. + + ``source_relation`` then each ``path`` hop then ``leaf``, dot-joined: + ``result_key(source_relation="orders", path=("customers",), leaf="revenue")`` + → ``"orders.customers.revenue"``. + + ``leaf`` must not contain a dot — hop information belongs in ``path`` so + ownership is unambiguous. For an already-canonical relative alias that + legitimately embeds hop dots (a cross-model measure alias such as + ``customers.revenue_sum``), use :func:`result_key_from_alias` instead. + """ + if "." in leaf: + raise ValueError( + f"result_key leaf must not contain '.': {leaf!r}. Pass hops via " + f"`path`, or use result_key_from_alias for a canonical dotted alias." + ) + return ".".join((source_relation, *path, leaf)) + + +def result_key_from_alias(*, source_relation: str, alias: str) -> str: + """Build a final-stage result key from an already-canonical relative + ``alias`` that may embed hop dots (e.g. a cross-model measure alias + ``customers.revenue_sum`` → ``orders.customers.revenue_sum``).""" + return f"{source_relation}.{alias}" + + +def flat_name(dotted: str, *, strip_relation: Optional[str] = None) -> str: + """Flatten a dotted name to its ``__``-joined INNER-stage bind name. + + When ``strip_relation`` is given, the exact ``f"{strip_relation}."`` + prefix is removed first (a dot-boundary match, so ``strip_relation='orders'`` + strips ``orders.`` but never the char prefix of a sibling ``orders_archive``). + Remaining dots become ``__``: + ``flat_name("orders.customers.revenue", strip_relation="orders")`` → + ``"customers__revenue"``. + """ + remainder = dotted + if strip_relation is not None: + prefix = f"{strip_relation}." + if remainder.startswith(prefix): + remainder = remainder[len(prefix):] + return remainder.replace(".", "__") + + +# --------------------------------------------------------------------------- +# BigQuery / T-SQL dotted-alias mangling bijection (DEV-1571). +# +# Relocated from ``slayer/sql/dialects/_alias_mangle.py`` (DEV-1713 D-a) so the +# naming module owns the result-key <-> wire-identifier bijection. Used by +# ``BigqueryDialect`` (backtick-anchored regex) and ``TsqlDialect`` (bracket- +# anchored regex): both need IDENTICAL encode/decode logic — BigQuery rejects +# dotted output-column names; T-SQL's ORDER BY parser does not resolve bracketed +# dotted identifiers as SELECT aliases. The fix is the same: mangle ``.`` to +# ``___`` on emit, decode on result-row keys. +# +# The bijection's only domain constraint is that ``decode_alias`` inverts +# ``encode_alias`` ONLY on the latter's image. A key like ``my___metric`` (no +# dot in the original) is OUTSIDE the image — decoding it would corrupt the +# value to ``my.metric``. This never bites because SLayer projection aliases are +# always model-qualified (``.``), so they always contain a dot +# and always pass through ``encode_alias``. +# --------------------------------------------------------------------------- + +_ALIAS_SEP = "___" + + +def encode_alias(alias: str) -> str: + """Forward encode: escape any pre-existing ``___`` to ``______``, then + map ``.`` to ``___``. Inverse is :func:`decode_alias`.""" + return alias.replace(_ALIAS_SEP, _ALIAS_SEP * 2).replace(".", _ALIAS_SEP) + + +def decode_alias(key: str) -> str: + """Reverse of :func:`encode_alias`. Walks ``key`` left-to-right, consuming + the escape-doubled ``______`` BEFORE the plain ``___`` so the two encodings + stay unambiguous. Inverse of ``encode_alias`` only on its image (see the + module-level bijection note).""" + out: list[str] = [] + i = 0 + n = len(key) + esc = _ALIAS_SEP * 2 + while i < n: + if key.startswith(esc, i): + out.append(_ALIAS_SEP) + i += len(esc) + elif key.startswith(_ALIAS_SEP, i): + out.append(".") + i += len(_ALIAS_SEP) + else: + out.append(key[i]) + i += 1 + return "".join(out) + + +# --------------------------------------------------------------------------- +# Mixed-case identifier quoting (DEV-1645). +# +# Relocated from ``SQLGenerator`` (DEV-1713 D-b) so the naming module owns the +# identifier-quoting policy. Case-folding dialects (Postgres/Redshift fold to +# lower; Snowflake/Oracle to upper) reach the wrong physical object unless a +# mixed-case identifier is quoted. The generator keeps thin delegators. +# --------------------------------------------------------------------------- + + +def maybe_quote_ident(ident: Optional[exp.Expression]) -> None: + """Set ``quoted=True`` in place on ``ident`` when it is an unquoted + ``Identifier`` containing an uppercase letter. No-op otherwise (None, + already-quoted, all-lowercase, non-Identifier).""" + if ( + isinstance(ident, exp.Identifier) + and not ident.quoted + and any(c.isupper() for c in ident.this) + ): + ident.set("quoted", True) + + +def quote_mixed_case_identifiers(node: exp.Expression) -> exp.Expression: + """Quote mixed-case DB identifiers so case-folding dialects reach the right + physical object. Context-aware: quotes only the column-name leaf of a + ``Column`` and the physical-table name parts of a ``Table`` — never table + aliases or the qualifier side of a column reference (SLayer-internal + aliases that fold consistently within a query). Idempotent; intended as a + ``.transform(...)`` callback.""" + if isinstance(node, exp.Column): + maybe_quote_ident(node.this) + elif isinstance(node, exp.Table): + maybe_quote_ident(node.this) + maybe_quote_ident(node.args.get("db")) + maybe_quote_ident(node.args.get("catalog")) + return node + + +# --------------------------------------------------------------------------- +# CTE name-collision belt (DEV-1692). +# --------------------------------------------------------------------------- + + +def assert_unique_cte_names(sql: str, *, dialect: str = "postgres") -> None: + """Assert every CTE name is unique WITHIN each ``WITH`` scope. + + CTE names must be unique inside a single ``WITH`` clause, but the same name + may legally recur in a separate nested ``WITH`` scope (an inner subquery); + each ``exp.With`` is validated independently. Raises ``ValueError`` on a + same-scope duplicate — the loud failure the DEV-1692 de-collision guards + against (a duplicate ``shifted_*`` CTE otherwise silently shadows). + + On case-folding dialects (:func:`dialect_folds_case`) names are compared + case-folded, REGARDLESS of identifier quoting (DEV-1726). That is + deliberately over-strict for quoted names on Postgres/Snowflake/Oracle: + this belt validates SLayer's own allocator-sanitized output — which never + quotes CTE names — so a fold-collision here always signals an + allocator-bypass bug, never a legitimately-distinct quoted pair. It is + not a general-purpose validator of arbitrary SQL. + """ + fold = dialect_folds_case(dialect) + parsed = sqlglot.parse_one(sql, dialect=dialect) + for with_node in parsed.find_all(exp.With): + names = [cte.alias_or_name for cte in with_node.expressions] + seen: dict[str, str] = {} + for name in names: + key = name.lower() if fold else name + if key in seen: + first = seen[key] + fold_note = ( + f" ({first!r} and {name!r} case-fold to {key!r} on " + f"{dialect})" + if first != name + else "" + ) + raise ValueError( + f"Duplicate CTE name {name!r} within one WITH scope" + f"{fold_note}: {names}" + ) + seen[key] = name diff --git a/slayer/sql/scope.py b/slayer/sql/scope.py new file mode 100644 index 00000000..c9e93db8 --- /dev/null +++ b/slayer/sql/scope.py @@ -0,0 +1,223 @@ +"""DEV-1706 Stage 2 — ``ScopeFrame`` + the single resolver (Laws 1 & 2). + +A query renders as a tree of SELECT scopes, each rooted at one relation. Every +expression enters a scope through :meth:`ScopeFrame.resolve`, which: + +* **Law 1 (anchored rendering):** expands derived refs (reserved-word + identifiers prequoted — DEV-1686; multi-term derived expansions parenthesised + — DEV-1539), anchors every reference at the scope root or a ``__``-path join + alias, and REGISTERS each crossed join path into ``join_paths`` in the same + call. Discovery is a side effect of rendering — it can never be forgotten. +* **Law 2 (projection boundaries):** when a ``consumer`` scope is named, the + value is materialised as a ``_val_`` projection in THIS (producing) scope + and a bare alias is returned for the consumer. Materialisations dedup by a + scope-safe key (producing-scope id + anchored AST + dialect — Codex F6). + +Stage 2 migrates the host base SELECT, which is a single scope with no +projection boundary, so the materialise branch is exercised only by direct unit +tests here; Stage 4 is its first generated-SQL consumer. The resolver reuses the +existing engine-layer expansion/scan helpers (D-G wrap-and-reuse). +""" + +from __future__ import annotations + +from typing import List, Optional, Tuple, Union + +import sqlglot +from pydantic import BaseModel, ConfigDict, Field +from sqlglot import exp + +from slayer.core.keys import ColumnKey, ColumnSqlKey +from slayer.core.models import SlayerModel +from slayer.engine.column_expansion import ( + collect_root_scope_joined_paths, + expand_derived_refs_sync, +) +from slayer.engine.source_bundle import ResolvedSourceBundle +from slayer.sql.dialects.base import SqlDialect +from slayer.sql.naming import AliasAllocator +from slayer.sql.reserved_keywords import ( + install_reserved_keywords, + prequote_reserved_identifiers, +) + +# The resolver relies on sqlglot's reserved-word quoting on emit (DEV-1686). +install_reserved_keywords() + +# A ref that can enter a scope. Stage 2 exercises structural column refs, derived +# columns, and free Mode-A / predicate text; later stages widen this union. +Ref = Union[ColumnKey, ColumnSqlKey, str] + + +class _OrderedPathSet: + """Insertion-ordered, de-duplicated set of ``__``-join-path tuples. + + Backed by a dict so membership is O(1) and iteration/`as_list` preserve + first-seen order — the join emission order ``_build_from_and_joins`` reads. + """ + + def __init__(self) -> None: + self._d: "dict[Tuple[str, ...], None]" = {} + + def add(self, path: Tuple[str, ...]) -> None: + self._d.setdefault(path, None) + + def __contains__(self, path: object) -> bool: + return path in self._d + + def __iter__(self): + return iter(self._d) + + def __len__(self) -> int: + return len(self._d) + + def as_list(self) -> List[Tuple[str, ...]]: + return list(self._d) + + +class Materialization(BaseModel): + """A Law-2 ``_val_`` projection produced in a scope for a consumer.""" + + model_config = ConfigDict(frozen=True, arbitrary_types_allowed=True) + + alias: str + expr: exp.Expression # anchored template, projected in the producing scope + # (producing_scope_id, ast.sql(dialect=sqlglot_name), sqlglot_name) — Codex F6/M3. + dedup_key: Tuple[str, str, str] + + +class ScopeFrame(BaseModel): + """One SELECT scope rooted at ``root_relation`` (Laws 1 & 2).""" + + model_config = ConfigDict(arbitrary_types_allowed=True) + + scope_id: str # generation-local, ephemeral, never emitted (D-F / Codex L1) + root_model: SlayerModel + root_relation: str + bundle: ResolvedSourceBundle + dialect: SqlDialect + allocator: AliasAllocator + join_paths: _OrderedPathSet = Field(default_factory=_OrderedPathSet) + materializations: List[Materialization] = Field(default_factory=list) + + # ---- Law 1 ------------------------------------------------------------- + def resolve(self, ref: Ref, *, consumer: "ScopeFrame | None" = None) -> exp.Expression: + """Anchor ``ref`` in this scope, register the joins it crosses, and — + when a ``consumer`` scope is named — materialise it and return the bare + alias for the consumer. + """ + template = self._anchor(ref) + for path in collect_root_scope_joined_paths( + parsed=template, + source_model=self.root_model, + source_relation=self.root_relation, + bundle=self.bundle, + ): + self.join_paths.add(path) + + if consumer is not None and not self.may_inline(self.join_paths.as_list()): + alias = self._materialize(template) + return exp.column(alias) + # Return a copy so a caller attaching this into its tree can never + # corrupt a value the scope (or another caller) also holds (D-L / M1). + return template.copy() + + def resolve_predicate_sql(self, ref: Ref) -> Optional[str]: + """Resolve a predicate ref to a SQL string for WHERE/HAVING builders.""" + expr = self.resolve(ref) + return None if expr is None else expr.sql(dialect=self.dialect.sqlglot_name) + + def _anchor(self, ref: Ref) -> exp.Expression: + if isinstance(ref, ColumnKey): + alias = self.root_relation if not ref.path else "__".join(ref.path) + return exp.Column( + this=exp.to_identifier(ref.leaf), + table=exp.to_identifier(alias), + ) + if isinstance(ref, ColumnSqlKey): + model = self._model_for(ref.model) + col = next( + (c for c in model.columns if c.name == ref.column_name), None, + ) + if col is None: + raw_sql = ref.column_name + elif col.sql: + raw_sql = col.sql + else: + raw_sql = col.name + # DEV-1711: a derived column ON a JOINED model (``path`` non-empty, + # e.g. ``stores.tier`` where ``tier`` lives on the joined ``stores``) + # must anchor at the ``__``-path alias with ``is_root=False`` so a + # bare inner ref (``name``) qualifies to ``stores.name`` — and a + # further-joined inner ref (``regions.population``) to the full + # ``stores__regions`` path (the DEV-1701 shape). A local derived + # column (empty path) keeps anchoring at the scope root. + if ref.path: + alias_path = "__".join(ref.path) + is_root = False + else: + alias_path = self.root_relation + is_root = True + expanded = expand_derived_refs_sync( + sql=raw_sql, + model=model, + alias_path=alias_path, + resolve_model=self.bundle.get_referenced_model, + dialect=self.dialect.sqlglot_name, + is_root=is_root, + ) + return self._parse(expanded or raw_sql) + if isinstance(ref, str): + prequoted = prequote_reserved_identifiers( + ref, dialect=self.dialect.sqlglot_name, + ) + expanded = expand_derived_refs_sync( + sql=prequoted, + model=self.root_model, + alias_path=self.root_relation, + resolve_model=self.bundle.get_referenced_model, + dialect=self.dialect.sqlglot_name, + is_root=True, + ) + return self._parse(expanded or prequoted) + raise NotImplementedError( + f"ScopeFrame.resolve does not yet handle ref type {type(ref).__name__}", + ) + + def _model_for(self, name: str) -> SlayerModel: + if name == self.root_model.name: + return self.root_model + return self.bundle.get_referenced_model(name) or self.root_model + + def _parse(self, sql: str) -> exp.Expression: + return sqlglot.parse_one(sql, dialect=self.dialect.sqlglot_name) + + # ---- Law 2 ------------------------------------------------------------- + def may_inline(self, crossed_paths: List[Tuple[str, ...]]) -> bool: # NOSONAR(S1172) — crossed_paths is the documented v1 API seam; the Stage-N inlining optimisation reads it, hardcoded False until then. + """Whether a crossing value may be inlined back into the consumer scope + instead of materialised. Hardcoded ``False`` in v1 (the seam Stage-N+ + optimisation grows into).""" + return False + + def _materialize(self, template: exp.Expression) -> str: + key = ( + self.scope_id, + template.sql(dialect=self.dialect.sqlglot_name), + self.dialect.sqlglot_name, + ) + for m in self.materializations: + if m.dedup_key == key: + return m.alias + alias = self.allocator.allocate_val() + self.materializations.append( + Materialization(alias=alias, expr=template, dedup_key=key), + ) + return alias + + def apply_materializations(self, select: exp.Select) -> exp.Select: + """Project each materialisation as ``