diff --git a/.claude/skills/slayer-models.md b/.claude/skills/slayer-models.md index 83e857bb..62bf05b4 100644 --- a/.claude/skills/slayer-models.md +++ b/.claude/skills/slayer-models.md @@ -88,7 +88,9 @@ Saved query-backed models support two access patterns: Variable precedence (highest first): runtime kwarg > stage `.variables` > outer query `.variables` > `model.query_variables`. -**Variables in model SQL (DEV-1625)**: the same `{var}` mechanism also substitutes into a model's **raw-SQL (Mode A) surfaces** — `SlayerModel.sql`, `SlayerModel.filters`, `Column.sql`, `Column.filter` — for a query's **direct source model** (the primitive for parameterizing hand-written SQL, e.g. Cube `FILTER_PARAMS`). Same precedence and `{{`/`}}` escaping. Contract: **raise-on-missing once any variable is in play** (a `query_variables` default or a caller value); a **fully variable-free execution leaves braces as literals** so raw brace literals like `'{1,2,3}'` survive untouched. String values are Mode-A-escaped (write the quotes yourself: `WHERE region = '{region}'`; trusted input — not dialect-aware, so avoid untrusted values on backslash-escaping backends like MySQL); `inspect_model` shows the literal `{var}` template. Nested `source_queries` stages, query-backed direct sources, join targets, and cross-model targets are deferred (DEV-1678) — a `{var}` there stays literal and errors on the stray placeholder. +**Variables in model SQL (DEV-1625)**: the same `{var}` mechanism also substitutes into a model's **raw-SQL (Mode A) surfaces** — `SlayerModel.sql`, `SlayerModel.filters`, `Column.sql`, `Column.filter` — for a query's **direct source model** (the primitive for parameterizing hand-written SQL, e.g. Cube `FILTER_PARAMS`). Same precedence and `{{`/`}}` escaping. Contract: **raise-on-missing once any variable is in play** (a `query_variables` default or a caller value); a **fully variable-free execution leaves braces as literals** so raw brace literals like `'{1,2,3}'` survive untouched. String values are Mode-A- and dialect-aware-escaped (write the quotes yourself: `WHERE region = '{region}'`; the quote/backslash escaping follows the datasource dialect so values round-trip on standard AND backslash-escaping backends like MySQL/ClickHouse — DEV-1727; still trusted input, and only *quoted* literals are escaped); a **list** value renders an injection-safe `IN`-list body (`region IN ({regions})` with `{"regions": ["US","CA"]}` → `region IN ('US', 'CA')`; write the parens, elements auto-quoted; empty list raises); `inspect_model` shows the literal `{var}` template. Nested `source_queries` stages, query-backed direct sources, join targets, and cross-model targets are deferred (DEV-1678) — a `{var}` there stays literal and errors on the stray placeholder. + +**Optional blocks (DEV-1730)**: Mode-A surfaces also support `{? pred ?}` — the predicate renders (parenthesised) when every inner `{var}` is supplied, else the whole block collapses to `(1=1)`. This is the SLayer form of a Cube `FILTER_PARAMS` optional pushdown (`{? region IN ({regions}) ?}` → `(region IN ('US','CA'))` or `(1=1)`). Put `AND` outside the block and open `WHERE` with `1=1`; a block needs ≥1 var, doesn't nest, collapses even on a zero-variable call, and is Mode-A-only. `inspect` lists a model's placeholders as **required** (bare, no default) vs **optional** (in-block or defaulted), derived from the SQL (`extract_model_variables`). `slayer import-cube` reads Cube `.js` as well as YAML and maps `FILTER_PARAMS` to these forms — requiredness from member `meta.required` (`--ignore-required-meta` forces optional). You **cannot** supply `columns` or `backing_query_sql` when saving a query-backed model — they're engine-managed cache; the save path rejects them. Caches refresh **only on save paths**: `engine.save_model()` and `create_model_from_query(save=True)`. `engine.execute()` never writes to storage — even on stale or empty caches. diff --git a/.claude/skills/slayer-query.md b/.claude/skills/slayer-query.md index 45931dbb..b6950970 100644 --- a/.claude/skills/slayer-query.md +++ b/.claude/skills/slayer-query.md @@ -76,7 +76,7 @@ Result column naming: `revenue:sum` → `orders.revenue_sum` (colon becomes unde **Top-N filtering**: use `"rank() <= N"` (e.g. `"rank(revenue:sum) <= 10"`) — dialect-portable and auto-promoted to a post-filter on the outer query. Raw `OVER (...)` SQL inside a filter or `ModelMeasure.formula` is rejected with an actionable error. Filtering on a `Column` whose `sql` contains a window function is also rejected (DEV-1369): use `rank()` / `dense_rank()` / `percent_rank()` / `ntile(n=)` for top-N, or factor the windowed expression into an earlier stage of a multi-stage `source_queries` model. -**Variable substitution**: `{var}` placeholders in filter strings are substituted from the query's `variables` dict (or per-model defaults). Use `{{`/`}}` for literal braces. Write the surrounding quotes yourself (`status = '{status}'`); string values are auto-escaped so an embedded quote stays inside the literal. Numbers (incl. bool) insert verbatim; non-finite floats are rejected; undefined vars raise. The same `{var}` mechanism also fills the raw-SQL (Mode A) surfaces of the query's direct source model — `SlayerModel.sql`, `SlayerModel.filters`, `Column.sql`, `Column.filter` (DEV-1625). See slayer-models skill for details. +**Variable substitution**: `{var}` placeholders in filter strings are substituted from the query's `variables` dict (or per-model defaults). Use `{{`/`}}` for literal braces. Write the surrounding quotes yourself (`status = '{status}'`); string values are auto-escaped so an embedded quote, backslash, or control char (newline/tab) stays inside the literal and parses cleanly (DEV-1727). Numbers (incl. bool) insert verbatim; non-finite floats are rejected; undefined vars raise. A **list** value renders an injection-safe `IN`-list for an `in`/`not in` filter (`region in ({regions})` with `{"regions": ["US","CA"]}` → `region IN ('US', 'CA')`) — write the parens, omit per-element quotes (auto-quoted); empty list raises. The same `{var}` mechanism also fills the raw-SQL (Mode A) surfaces of the query's direct source model — `SlayerModel.sql`, `SlayerModel.filters`, `Column.sql`, `Column.filter` (DEV-1625) — which additionally support optional blocks `{? pred ?}` that collapse to `(1=1)` when their vars are absent (Cube `FILTER_PARAMS` form, DEV-1730). See slayer-models skill for details. ## Executing diff --git a/CLAUDE.md b/CLAUDE.md index c9a59e03..5c4c6a72 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -53,7 +53,10 @@ data dir, override with `$SLAYER_STORAGE`. - Dots denote join paths in queries (`customers.regions.name`); `__` denotes path aliases in model SQL (`customers__regions.name`) - Models are keyed by `(data_source, name)`; joins resolve within the parent model's datasource - Models/queries/datasource configs carry a `version` field; storage migrations run automatically on load (`slayer/storage/migrations.py`) -- Filters support `{variable}` placeholders from `query.variables`; datasource configs support `${ENV_VAR}` +- Filters support `{variable}` placeholders from `query.variables` (scalars, plus lists → auto-quoted `IN`-list body: `region IN ({regions})`). Values are trusted input; string escaping IS dialect-aware (DEV-1727) but only applies to *quoted* literals. Datasource configs support `${ENV_VAR}` +- Mode-A raw-SQL surfaces also support optional blocks `{? pred ?}` — render parenthesised when every inner `{var}` is supplied, else collapse to `(1=1)` (Cube `FILTER_PARAMS` optional-pushdown form, DEV-1730). `extract_model_variables(model)` classifies placeholders required vs optional; surfaced in the inspect skeleton +- `slayer import-cube` reads Cube `.yml`/`.yaml` **and `.js`** (`cube()`/`view()` via esprima); FILTER_PARAMS → `{var}`/`{? ?}`, requiredness from member `meta.required` (`--ignore-required-meta` to force optional). See `docs/cube/cube_import.md` +- A model may declare a variable `list_valued` in `meta.cube_variables` (importers do this for generated `col IN ({var})` templates); the engine then wraps a bare scalar into a one-element list. Hand-written model SQL keeps the author-writes-the-quotes convention — a scalar string substitutes unquoted ## Database Support diff --git a/CUBE_IMPORT_SPEC.md b/CUBE_IMPORT_SPEC.md new file mode 100644 index 00000000..0e1b7c3f --- /dev/null +++ b/CUBE_IMPORT_SPEC.md @@ -0,0 +1,539 @@ +# Spec: Cube → SLayer ingestion (DEV-1608) + +Convert Cube (Cube.js / Cube.dev) data-model YAML into persisted `SlayerModel`s, +mirroring the existing `import-dbt` importer (`slayer/dbt/`). Two stages: + +- **Stage 1 (this PR)** — everything that maps cleanly without the Tesseract + engine: cubes, measures, dimensions, joins, segments, `extends`, and **views** + (via facade models). Plus any Tesseract feature that turns out to be an easy + win (assessment below: effectively none — see §9). +- **Stage 2 (follow-up issue/PR)** — the Tesseract-only feature set and the + hard edges Stage 1 routes to the report. Fully designed in §9, not built here. + +Conversion is **fully offline**: types come from Cube's declared dimension / +measure types; no database connection is required or used. Everything that does +not convert is captured in a **structured report** (Pydantic object + JSON file). + +--- + +## 1. Package layout (mirrors `slayer/dbt/`) + +```text +slayer/cube/ + __init__.py + models.py # Pydantic shapes for parsed Cube YAML + parser.py # walk dir, collect .yml/.yaml, Jinja-detect, parse cubes:/views: + refs.py # {CUBE}/{member}/{cube.member} → SLayer SQL / DSL translator + extends.py # extends-graph resolution + member flattening + converter.py # CubeToSlayerConverter → CubeConversionResult + report.py # CubeConversionReport / CubeConversionIssue / categories +``` + +CLI: `import-cube` subcommand in `slayer/cli.py` + `_run_import_cube`. + +The converter never touches the DB and never needs an engine. Persistence is +`run_sync(storage.save_model(model))`, exactly as `_run_import_dbt` does for +table-backed models. + +--- + +## 2. Parser (`parser.py`) + +- Recursively collect `*.yml` / `*.yaml` (skip hidden dirs, `target`, etc.) — + reuse the dbt parser's `_collect_yaml_paths` shape. +- **Jinja detection (Codex #6 — order resolved):** Cube's own SQL-ref syntax + uses **single** braces (`{CUBE}`, `{cube.member}`) so it never collides with + Jinja's double-brace / brace-percent. **Try YAML-parsing first.** + - If the file parses as YAML: scan only the **templatable fields** of each + member (`sql`, `filter`, `case` predicates, join `sql`) for `{{`/`{%`. A + member with a Jinja marker is skipped + reported (`requires_templating`), + keeping the rest of the cube. (Plain YAML containing one templated SQL member + is thus mostly imported — the earlier "scan raw text, skip whole file" rule + is replaced by this.) + - File-level skip applies **only** when the file fails to parse as YAML + *because of* templating directives (e.g. `{% for %}` generating list items) — + then skip the whole file + `requires_templating` report. +- Parse the two top-level keys: `cubes:` (list) and `views:` (list). Tolerate + single-object forms (wrap in a list), matching the dbt parser's leniency. +- Per-item `model_validate` into the `models.py` shapes; on `ValidationError`, + log + emit a `parse_error` report issue and continue (never abort the run for + one bad file). + +### Parsed Cube shapes (`models.py`, Pydantic v2, no dataclasses) + +```text +CubeMeasure: name, type, sql?, title?, description?, public?=True, meta?, + format?, filters?: list[{sql}], drill_members?, rolling_window?, + multi_stage?=False, time_shift?, grain?, filter? (Tesseract), + case? (Tesseract) +CubeDimension: name, sql?, type='string', title?, description?, public?=True, + meta?, format?, primary_key?=False, sub_query?=False, case?, + granularities?, latitude?, longitude? (geo) +CubeJoin: name (target cube), relationship, sql (ON clause) +CubeSegment: name, sql, title?, description?, public?=True, meta? +CubeCube: name, sql_table?, sql?, sql_alias?, extends?, data_source?, + title?, description?, public?=True, meta?, refresh_key?, + calendar?, measures?, dimensions?, joins?, segments?, + hierarchies?, pre_aggregations?, access_policy? +CubeViewCubeRef: join_path, includes?(list|'*'), excludes?, prefix?=False, + alias?, per-member overrides (alias/title/description/format/meta) +CubeView: name, cubes: list[CubeViewCubeRef], extends?, title?, description?, + public?=True, meta?, folders?, default_filters?, access_policy? +CubeProject: cubes: list[CubeCube], views: list[CubeView] +``` + +Unknown keys are tolerated (Cube evolves); they are ignored unless listed in the +"unmapped infra" set (§7), which is stashed in `meta`. + +--- + +## 3. Reference translator (`refs.py`) — the core correctness surface + +Cube `sql`/`filter` strings use Cube's curly reference syntax. Translate to +SLayer SQL (Mode A, for `Column.sql` / `Column.filter` / model `filters`) or +SLayer DSL (Mode B, for calculated-measure `ModelMeasure.formula`). These are +**not Jinja** and are translated, not skipped. + +| Cube ref | Meaning | SLayer rewrite | +|---|---|---| +| `{CUBE}.col` | own table column | bare `col` (SLayer auto-qualifies) | +| `{CUBE}` (bare) | own table alias | drop / context-specific (see joins) | +| `{member}` (same cube) | sibling member on this cube | bare `member` — SLayer inlines sibling derived columns recursively | +| `{other_cube.member}` | member on a joined cube | Mode A: `other_cube.member` (single hop) or `a__b.member` (multi-hop); SLayer inlines derived joined columns | +| `{measure}` inside a calc measure | another measure (post-agg) | Mode B: bare `measure` (resolves as `ModelMeasure` name) | + +Mechanics: +- A single-pass regex over the string, **skipping SQL string literals** + (reuse the `_STRING_LITERAL_RE` idea from `core/models.py`). +- Distinguish `{X}` (one dotless token → same-cube member or `CUBE`), + `{X.Y}` (cube.member), and `{CUBE}` / `{CUBE.col}` specially. +- Multi-hop `{a.b.c}` → SLayer multi-dot `a.b.c`, which `Column.sql`'s own + `_fix_multidot_sql` validator then converts to `a__b.c` (we lean on the + existing model-side normalization rather than duplicating it). +- **Boundary / report rule (user-established):** after translation, if the + result is not valid for its target mode, do **not** emit a broken member — + route it to the report: + - `Column.sql` / `Column.filter` / model `filters` are validated by Pydantic + construction (Mode A `parse_sql_predicate`). A `ValidationError` → catch → + `complex_sql` report issue, member dropped. + - Calculated-measure `ModelMeasure.formula` is Mode B. If the translated + formula is clean DSL (arithmetic / `||` / the `SCALAR_PASSTHROUGH` scalar + set / measure refs) it converts; otherwise (`CASE WHEN`, raw SQL funcs Mode B + rejects, unresolved refs) → `complex_measure` report issue, measure dropped. + +--- + +## 4. Cube → model conversion (`converter.py`) + +Top loop: **for each cube → one table-owning model; for each view → one facade +model** (§6). Both emit ordinary `SlayerModel`s. Order: resolve `extends` first +(§5), then convert cubes, then views (views need cube models to exist for +member/measure lookup and fan-out detection). + +### 4.1 Cube → SlayerModel + +| Cube | SLayer | Notes | +|---|---|---| +| `name` | `name` | | +| `sql_table` | `sql_table` | verbatim (Cube uses `schema.table`) | +| `sql` | `sql` | `{CUBE}`/`{member}` translated; if it references measures or is otherwise un-rewritable → `complex_sql` report and the cube is dropped (`no_source`), since a cube with no usable source can't be emitted (see the source rule below) | +| `data_source` (per-cube) | — | converter scopes ALL models under the single `--datasource`; per-cube `data_source` → `unmapped_infra` report + stashed in `meta.cube_unmapped.data_source` | +| `title` | — | no model title field; → `meta.cube_title` (info-level), not dropped silently | +| `description` | `description` | | +| `public: false` | `hidden: true` | | +| `meta` (incl `ai_context`) | `meta` | merged under the model's `meta` (preserved verbatim) | +| `sql_alias`, `refresh_key`, `calendar`, `hierarchies`, `pre_aggregations`, `access_policy` | — | §7 report + `meta.cube_unmapped.*` | +| `measures` | `columns` + `measures` | §4.2 | +| `dimensions` | `columns` | §4.3 | +| `joins` | `joins` | §4.4 | +| `segments` | `columns` (boolean) | §4.5 | + +A cube must end with exactly one source. Normal case: `sql_table`. If only `sql` +is given and it translates cleanly → `sql`. If neither is usable, emit nothing +for that cube and report `no_source` (error severity). + +### 4.2 Measures — Column + ModelMeasure split (same idiom as dbt) + +Aggregating measures (`type` ∈ `count`, `count_distinct`, `count_distinct_approx`, +`sum`, `avg`, `min`, `max`): + +- The `sql` expression → a `Column` (`DataType.DOUBLE`, `NumberFormat(FLOAT)` + default unless `format` maps — §8). Bare-identifier `sql` → Column named after + the column; non-trivial expression → Column named `_col`. Collisions + resolved by `_col` suffix (reuse dbt's logic). +- **Column dedup key must include the filter + window state** (Codex #4). The + dbt converter groups measures by `sql` expression alone (safe because dbt + measures carry no per-measure column state). Cube measures put `filters` on the + emitted `Column.filter`, so two measures sharing one `sql` but differing in + `filters` (or `rolling_window`) **must** become distinct columns — the dedup + key is `(translated_sql, translated_filter, window_spec)`, not `sql` alone. + Otherwise a filter would bleed across measures. +- The measure → a `ModelMeasure` whose formula is `:`. +- `type: count` with no `sql` → `ModelMeasure(formula="*:count")` (COUNT(*)), + no Column needed. `count` **with** `sql` → `:count`. +- Aggregation name map: `count_distinct`→`count_distinct`, + `count_distinct_approx`→`count_distinct` (+ `lossy_mapping` info report: SLayer + has no approximate distinct), others 1:1. + +Calculated measures (`type` ∈ `number`, `string`, `time`, `boolean`) — these are +post-aggregation expressions referencing other measures (e.g. +`sql: "{revenue} / {count}"`): + +- → a `ModelMeasure` whose formula is the Mode-B-translated expression, + `type` set to the mapped `DataType`. Convert if clean DSL; else `complex_measure` + report (§3 boundary). + +Per-measure extras: +- `filters: [{sql}, …]` (conditional aggregation) → the Column carries a `filter` + (the AND of the translated predicates). Same Column+ModelMeasure split; the + filter lives on the Column (dbt simple-filtered-metric idiom). +- `format` → §8. +- `title`→`ModelMeasure.label`, `description`→`ModelMeasure.description`, + `meta`→`ModelMeasure.meta`. +- `rolling_window` with a **finite `trailing`** and no `leading`/`offset` → + windowed aggregation `:(window='')` (Cube `1 month`→`1m`, + `7 day`→`7d`, etc.). `unbounded` / `leading` / `offset` / `rolling_window` on a + non-sum/avg agg → `unsupported_rolling_window` report, measure falls back to the + plain aggregation (still emitted) with a warning. +- `drill_members` → §7 report + `meta.cube_unmapped` on the ModelMeasure. +- `multi_stage`, `time_shift`, `grain`, `filter` (Tesseract), `case` (Tesseract) + → §9 (Stage 2). Measure is emitted as its plain aggregation if one exists, + else routed to report as `deferred_stage2`. + +### 4.3 Dimensions → Column + +| Cube dim | SLayer Column | Notes | +|---|---|---| +| `type: string` | `type=TEXT` | | +| `type: number` | `type=DOUBLE` | Cube doesn't distinguish int/float; refine later via `slayer ingest` | +| `type: boolean` | `type=BOOLEAN` | | +| `type: time` | `type=TIMESTAMP` | | +| `sql` | `Column.sql` (translated); omitted when it's just `{CUBE}.` | | +| `primary_key: true` | `Column.primary_key=True` | PK columns auto-restricted to count/count_distinct | +| `case:` (CASE-WHEN dim) | `Column.sql` built as `CASE WHEN … THEN … ELSE … END` from `when[].{sql,label}` + `else` | **Stage 1** — `case` *dimension* is not Tesseract | +| `title`→`label`, `description`, `meta`, `format` | direct | | +| `type: geo` (+ `latitude`/`longitude`) | — | §7 report + `meta.cube_unmapped.geo`; not split into lat/long columns in Stage 1 | +| `sub_query: true` | — | §7 report + `meta.cube_unmapped`; correlated per-row measure has no SLayer equivalent | +| `granularities:` (custom) | — | base time Column still emitted; custom grains → §7 report (SLayer granularity is query-time enum) | +| `type: switch` | — | §9 (Tesseract) | +| `links:`, `order:` | — | presentation; §7 report | + +### 4.4 Joins + +`CubeJoin.sql` is an ON clause like `{CUBE}.customer_id = {customers.id}`. + +- Parse the ON into equality column pairs. Conjunctions (`A=B AND C=D`) → + multiple `join_pairs` (SLayer supports composite keys). The qualifier matching + `{CUBE}` is the source column; the qualifier matching `{.…}` is the + target column. +- **Resolve each ON side to a physical column name** (Codex #2 — verified: the + SQL generator emits raw `alias.src = alias.tgt` from `join_pairs`; it does NOT + expand `Column.sql`). `{customers.id}` means the `id` *member*, which may have + `sql: "{CUBE}.customer_id"`. So follow each ON ref to its member's `sql`: if it + is a bare physical identifier, use it; if the member's `sql` is a non-trivial + expression (function, arithmetic, CASE), the column pair can't be expressed in + `join_pairs` → `unsupported_join` report, join dropped. Same for a `{CUBE}.col` + side that resolves to a derived dimension. +- `target_model` = the join `name` (the target cube). +- `relationship` (`many_to_one` / `one_to_many` / `one_to_one`, plus legacy + `belongs_to`/`has_many`/`has_one`) is **not stored on `ModelJoin`** (SLayer + joins are all LEFT). It IS recorded internally and used for view fan-out + detection (§6). Emitted `ModelJoin.join_type = LEFT`. +- Non-equi ON (ranges, function calls, inequalities), or an ON that doesn't + resolve to clean column pairs → `unsupported_join` report; the join is dropped + (both cubes still exist as models, just not auto-joined). + +### 4.5 Segments → boolean derived Column + +Each segment `{name, sql}` → `Column(name=, sql=, +type=BOOLEAN)`. Filterable (`name = true`) and group-able. `title`/`description`/ +`meta` carried onto the Column. Also recorded in the report (`segment_as_column`, +info). Name-collision with an existing column/measure → `_seg` suffix + warning. + +--- + +### 4.6 Namespace allocation & emit-time safety (Codex #5, #7) + +The core `SlayerModel` validators reject duplicate column names, duplicate +measure names, **any column↔measure overlap**, and `ModelMeasure` names that +shadow a built-in transform (`cumsum`, `rank`, `change`, `lag`, …). A naive +converter that lets these collide makes **whole-model construction throw**, +losing the entire model. So: + +- **Preflight namespace allocator** per emitted model/view: allocate column and + measure names against a shared seen-set (columns + measures share one + namespace). Dimension/segment/measure/entity name clashes are resolved by a + deterministic suffix (`_col` / `_seg`) **before** constructing the model. +- A Cube measure/metric name that shadows a SLayer transform, or a collision + that can't be safely renamed, is routed to the report (don't emit it) rather + than thrown. +- **Offline validation helper** (Codex #7 — verified: `Column.sql` is *not* + SQL-validated at construction; only `Column.filter` / `SlayerModel.filters` + parse a predicate, and `ModelMeasure.formula` only rejects raw `OVER`). After + building each model, run an explicit offline pass: sqlglot-parse every + translated `Column.sql`, `parse_sql_predicate` every filter, and formula-parse + every `ModelMeasure.formula`. A parse failure routes that member to the report + (`complex_sql` / `complex_measure`) and drops it — so a structurally-broken + member never persists to fail later at enrichment. + +## 5. `extends` (flatten) — `extends.py` + +> **Approach for this PR: flatten.** Native persisted model inheritance +> (making `ModelExtension` a saveable source mode) is tracked separately in +> **DEV-1610** and is deliberately *not* a dependency of DEV-1608. When DEV-1610 +> lands, `extends.py` swaps from flattening to emitting +> `SlayerModel(extends=ModelExtension(...))` — a ~10-line change. Flattening is +> faithful to Cube's own compile-time materialization of extended cubes, so it +> is not a stopgap-quality result. + +- Build the extends graph across all cubes. Resolve transitively (multi-level), + detect cycles → `extends_cycle` report (error) and skip the cycle members. +- Flatten: a child inherits the parent's measures / dimensions / joins / + segments; **child members win** on name conflict. Child's own `sql_table`/`sql` + override the parent's source. +- Every cube is still emitted as its own model (hidden iff `public: false`), so + an abstract base (`public: false`, only extended) becomes a **hidden** model + AND its members are flattened into children. Lossless, mirrors Cube's + `public: false`, matches SLayer's hidden-model convention. +- Views can `extends` other views — same flattening over view member lists. + +--- + +## 6. Views → facade models — the key structural mapping + +A Cube view owns no table; it re-exports members from cubes along a `join_path`. +SLayer has no view type, so a view becomes a **thin regular `SlayerModel` +anchored on the join_path root cube's table**: + +- **Source mode mirrors the root cube's emitted model** (Codex #3): copy + `sql_table` *or* `sql` — whichever the root cube produced. If the root cube was + not emitted (e.g. it failed conversion), drop + report the view + (`ambiguous_view_root` / `disconnected_view`). Never hard-code `sql_table`. +- `joins` = the joins implied by the view's `join_path`s (reuse the root cube's + existing join definitions by walking the path; each hop must correspond to a + declared cube join). +- Each included **dimension** → a derived `Column` whose `sql` references the + joined cube's column: `Column(name=, sql="customers.name")` + (single-dot Mode A joined ref; multi-hop uses `a__b.col`). Root-cube dimensions + reference their own column. +- Each included **measure** → a `ModelMeasure`, but it **must reference the + underlying `Column`, not the Cube measure name** (Codex #1 — verified: + `query_engine.py:2345` resolves cross-model aggs via + `target_model.get_column(name)`, never `get_measure`). So for a Cube measure + `revenue {type: sum, sql: amount}` whose emitted underlying column is `amount` + (or `_col` for an expression): + - **root-cube** measure → the facade also **carries the underlying `Column`** + (its physical/derived expression), and the `ModelMeasure` is a local + `amount:sum` (or `:`). A root-cube measure can't be re-exported by + bare name because the facade is a *separate* model that doesn't own the root + cube's measures. + - **joined-cube** measure → cross-model `customers.amount:sum` + (`joinpath.:`); the underlying column already lives on + the joined cube's emitted model, so no copy is needed. + - **filtered** Cube measure → the underlying column carries the `filter` + (root: copied onto the facade; joined: already on the joined model), so + `customers.:` still applies the CASE-WHEN correctly. + - **`count`** → `joinpath.*:count` (or local `*:count`); `customers.*:count` + resolves via the `measure_name == "*"` path. + - **calculated** (`type: number/…`, references multiple measures) re-export → + inline the cross-model **column** refs if the result is clean Mode B + (`customers.amount:sum / customers.*:count`); otherwise → `complex_measure` + report, that measure dropped from the view. +- The converter therefore threads, from cube-measure conversion, the + `(underlying_column_name, aggregation, filter?)` triple for every measure so + the facade builder can synthesize the correct `.:` formula. +- `prefix: true` → exported names are `_`. Per-member `alias` → + the exported name. `title`/`description`/`format`/`meta` overrides → the + Column/ModelMeasure fields. +- `default_filters` (`{member, operator, values, unless?}`) → model `filters` + (Mode A SQL predicates built from operator+values; `member` resolved to its + SQL column / joined ref). Operators that don't translate to plain SQL → + `unsupported_default_filter` report, that filter dropped. +- `excludes` / `includes: '*'` honored when selecting members. +- `meta: {cube_kind: "view"}` stamped so the report and future round-trips can + identify facade models. + +**Common case built in Stage 1.** Routed to the report (not guessed) when: +- the view's members span cubes **not on one connected join tree** rooted at the + join_path root (`disconnected_view`), +- the join_path root is **ambiguous** / not derivable (`ambiguous_view_root`), +- a hop in the path is `one_to_many` / `many_to_one`-reversed such that the + facade would **fan out** the root and double-count a root measure + (`view_fanout_risk` — detected via the recorded `relationship`), +- `folders` present → parked in `meta.cube_unmapped.folders` + `folders_unmapped` + report (no SLayer hierarchy concept), +- any per-member override or operator that doesn't map → reported, that member / + filter skipped, rest of the view still emitted. + +--- + +## 7. "No SLayer home" features — report + stash in meta + +For every Cube feature with no SLayer equivalent, emit a structured report issue +AND preserve the raw Cube fragment under a namespaced key on the owning entity's +`meta`: `meta.cube_unmapped.`. Set covers (per §4/§6): + +`pre_aggregations`, `refresh_key`, `calendar`, `hierarchies`, `drill_members`, +`access_policy`, `sql_alias`, per-cube `data_source`, `geo` dims (+lat/long), +`sub_query` dims, custom `granularities`, dimension `links`/`order`, view +`folders`, `multi_stage`/`time_shift`/`grain`/Tesseract bits (§9). + +Genuinely-semantic metadata is **not** in this bucket — `title`→`label`, +`description`, `meta`/`ai_context`, `format` always carry over directly to the +proper SLayer field. + +--- + +## 8. Format mapping (`format` → `NumberFormat`) + +`NumberFormat` has `type ∈ {percent, currency, integer, float}`, `precision`, +`symbol` (currency only). Map Cube formats: + +- `percent` → `PERCENT` +- `currency` (+ `currency_symbol` if present) → `CURRENCY` (symbol) +- `number` / numeric d3-ish → `FLOAT` (with `precision` if a `_N` suffix is + parseable) +- `accounting`, `abbr`, arbitrary d3-format strings, `imageUrl`/`link`/`id` → + `unsupported_format` report; format dropped (the field/measure still emitted). +- **`NumberFormat.symbol` guard** (Codex #8 — verified `format.py:40`: `symbol` + is rejected unless `type == CURRENCY`, and is auto-defaulted to `$` for + currency). Never pass `symbol` for non-currency mappings. A Cube format payload + carrying a symbol-like field on a non-currency type, or an otherwise invalid + format, is reported + dropped rather than allowed to raise at model + construction. + +--- + +## 9. Stage 2 (Tesseract) — designed here, built in the follow-up issue + +`CUBEJS_TESSERACT_SQL_PLANNER`-only features. **Assessment: none are easy wins** +— each lacks a clean SLayer mapping, so all are deferred. Stage 1 routes any cube +using them to the report as `deferred_stage2` (the cube's non-Tesseract members +still convert). + +| Tesseract feature | Why no clean SLayer map | Proposed Stage-2 approach | +|---|---|---| +| `switch` dimension | Query-time selectable dimension; SLayer dimensions are static | No direct map. Possibly enumerate switch cases into N separate columns + a report note; needs design. | +| `number_agg` measure | Aggregates an expression that itself contains aggregations (multi-stage) | Multi-stage `source_queries` model: inner stage materializes the inner aggregation, outer stage re-aggregates. Needs the query-backed-model builder. | +| `case` measure | Conditional measure keyed on a `switch` dimension | Depends on `switch`; without it, a pure `CASE WHEN` over a condition → a filtered Column (the one borderline "maybe easy" — evaluate during impl, default to defer). | +| measure `filter` (`exclude`/`keep_only`/`mode`) | Grain manipulation at aggregation time | No SLayer grain-override; candidate for a multi-stage rewrite. | +| `multi_stage` + `time_shift` + `grain` (non-Tesseract but same family) | Cube's measure-level time-shift/grain grammar ≠ SLayer's query-time `time_shift`/transform model | Map finite `rolling_window` trailing in Stage 1 (§4.2); defer `time_shift`/`grain` to the multi-stage rewrite. | + +Plus the Stage-1 "hard edges" routed to the report: disconnected/ambiguous/ +fan-out-risk views, non-equi joins, complex SQL/measures, custom granularities, +geo split, sub_query dims. The follow-up issue tackles these alongside Tesseract. + +--- + +## 10. Structured report (`report.py`) + +```python +class CubeIssueCategory(str, Enum): # requires_templating, parse_error, + complex_sql, complex_measure, lossy_mapping, unsupported_join, + unsupported_rolling_window, unsupported_format, unsupported_default_filter, + segment_as_column, unmapped_infra, geo_unmapped, subquery_unmapped, + granularity_unmapped, disconnected_view, ambiguous_view_root, view_fanout_risk, + folders_unmapped, extends_cycle, no_source, deferred_stage2 + +class CubeConversionIssue(BaseModel): + category: CubeIssueCategory + severity: Literal["info","warning","error"] + cube: str | None; view: str | None; member: str | None + message: str + raw: str | None # raw Cube fragment when useful + +class CubeConversionReport(BaseModel): + issues: list[CubeConversionIssue] + model_count: int; hidden_count: int; view_count: int + # counts derived; helpers to filter by category/severity + +class CubeConversionResult(BaseModel): # converter return + models: list[SlayerModel] + report: CubeConversionReport +``` + +(No `Dict`-typed LLM-output fields; this is internal, so plain fields are fine. +No dataclasses.) + +--- + +## 11. CLI + +```text +slayer import-cube --datasource NAME [--storage PATH] + [--report PATH] [--include-hidden] +``` + +- Recursively parse, convert, `storage.save_model` each model, print a console + summary (imported models with column/measure counts + `[hidden]`, then issues + grouped by severity), and **always write the JSON report** to + `/cube_import_report.json` (override with `--report PATH`). +- `_run_import_cube` mirrors `_run_import_dbt` structurally. +- `--datasource` is just the SLayer datasource name to file models under; it need + not exist or be reachable (offline). (`--include-hidden` reserved for parity; + cubes are already emitted hidden when `public: false`, so it mainly governs + whether hidden models print — keep minimal.) +- The `slayer/cube/` converter API stays importable for programmatic use. + +--- + +## 12. Tests (TDD — full suite first, per `feedback_tdd_style.md`) + +Mirror `tests/test_dbt_*`. New files: + +- `tests/test_cube_parser.py` — YAML collection; Jinja file-skip + member-skip + + report; single-object tolerance; malformed-file `parse_error` continue. +- `tests/test_cube_refs.py` — `{CUBE}.col`→`col`; `{member}`→bare; `{cube.member}` + → joined ref; multi-hop; string-literal skipping; calc-measure Mode-B + translation; un-rewritable → report boundary. +- `tests/test_cube_converter.py` — cube→model 1:1; measure Column+ModelMeasure + split (count/count_distinct/approx/sum/avg/min/max); `*:count`; calc measures; + measure `filters`→Column.filter; finite rolling_window→`window=`; + dimensions (string/number/bool/time, `case` dim, primary_key); joins (single + + composite + non-equi→report); segments→boolean column; format mapping; + unmapped infra→report+meta stash. +- `tests/test_cube_extends.py` — single + multi-level flatten; child-wins; abstract + base emitted hidden; cycle→report. +- `tests/test_cube_views.py` — facade model: dims→derived columns, measures→ + local/cross-model ModelMeasures, prefix/alias/overrides, default_filters→model + filters, excludes/`*`; disconnected/ambiguous/fanout/folders→report. +- `tests/test_cube_report.py` — categories, severities, counts, JSON round-trip. +- `tests/test_cube_cli.py` (or fold into converter) — `import-cube` writes models + + JSON report; offline (no datasource needed); console summary. +- `tests/fixtures/cube_project/` — hand-written sample `.yml` (cubes, a view, + extends, segments, a Jinja file, a Tesseract cube) covering the above. +- **`tests/test_cube_smoke.py` — enrich/execute converted models, not just + assert converter output** (Codex test-gap). Build a tiny SQLite datasource + + converted models and actually run queries: (a) a view cross-model measure that + references the underlying joined column (`customers.amount:sum`), (b) a view + rooted on a `sql`-backed cube, (c) a multi-hop facade dimension, (d) two + filtered same-`sql` measures returning different values, (e) `show_sql` on a + facade measure. These catch the §6/§4.4 mapping breaks at the SQL layer. +- **Negative validator-boundary construction tests** (Codex test-gap): a converter + input that would yield column↔measure namespace overlap, a measure named after + a transform (`cumsum`), a Cube name containing `.`/`:`, an ON that yields empty + `join_pairs`, a facade whose root cube wasn't emitted, and a non-currency + format carrying a symbol — assert each is routed to the **report** (model still + emitted where possible), NOT raised as an unhandled `ValidationError`. + +Run the full non-integration suite after implementation; fix all failures. + +--- + +## 13. Docs (update on user-facing change) + +- New `docs/cube/cube_import.md` (mirror `docs/dbt/dbt_import.md`): mapping + tables, the non-mapping catalog, the report, CLI usage. Link from `mkdocs.yml`. +- `CLAUDE.md` — short note under an importer/CLI section. +- `.claude/skills/` — mention `import-cube` where `import-dbt` is referenced. + +--- + +## 14. Explicit non-goals (Stage 1) + +- No live DB connection / type refinement / sample profiling (run `slayer ingest` + afterward). +- No Jinja/Python template rendering. +- No Tesseract features built (designed in §9). +- No MCP/REST surface (CLI + importable API only), matching `import-dbt`. +- No round-trip SLayer→Cube export. diff --git a/DECISIONS.md b/DECISIONS.md index 02bb7695..496c8b6f 100644 --- a/DECISIONS.md +++ b/DECISIONS.md @@ -65,3 +65,7 @@ implementation detail. Include issue refs when known. - 2026-07-22 — Case-colliding ids rejected in the YAML backend only (#249): ids are filenames there and case variants alias on macOS/Windows; SQLite keeps case variants as distinct rows (so such stores can't export to YAML without renames); backends opt in via the `_ids_collide_as_filenames` flag + shared collision helpers; case-variant reads/deletes are exact-directory-entry checked (not-found / no-op, never the wrong file). - 2026-07-31 — RLS policy restructured around one required `ruleset` (DEV-1718 / #260), superseding the `data_filters` rule list of DEV-1578/DEV-1627: a `SessionPolicy` carries exactly one `ColumnFilterRuleset` or `JoinFilterRuleset`, discriminated on an explicit `kind` (no inference — a kind-less dict fails to discriminate), and no-filtering is `policy=None` rather than an empty policy. The join ruleset hoists the tenant anchor (`table`/`column`/`value`) up from the rules, so nested rules carry only `target_table` + `join_path` and classification becomes fully structural and DB-free (`has_column` is never probed); an explicit `whitelist` replaces the mandatory blocking-column backstop, so a table that is neither the anchor, a join target, nor whitelisted fails closed. Join paths may be written from either endpoint and are normalized target-first, so the correlated EXISTS always lands its terminal predicate on the anchor. - 2026-08-03 — Mode-A `{variable}` substitution (DEV-1625 / #270): the `{var}` mechanism now fills a query's DIRECT source model's four raw-SQL surfaces — `SlayerModel.sql`/`.filters`, `Column.sql`/`.filter` — not just query-level filters; the primitive for parameterizing hand-written model SQL (e.g. Cube `FILTER_PARAMS`). `substitute_variables(..., escape="sql"|"python")` picks the escaping regime by layer: SQL quote-doubling for sqlglot-parsed Mode-A, Python backslash-escaping for the Python-AST Mode-B filters (SQL doubling would silently corrupt a Mode-B value via adjacent-literal concatenation). Contract: raise-on-missing once any variable is in play; a fully variable-free execution leaves braces as literals so raw brace literals (`'{1,2,3}'`) survive. Values are trusted input; the escaping is not dialect-aware (MySQL backslash caveat). Nested source_queries stages / join-target / cross-model-target lineages are deferred (DEV-1678); dialect-aware + control-char escaping deferred (DEV-1727). +- 2026-08-03 — List-valued `{variable}` substitution (DEV-1730 / #270): a `list`/`tuple` variable renders an injection-safe `IN`-list body through the same `_render_variable_value` choke point, so the one branch covers every consumer (Mode-A engine pass, Mode-B enrichment, `get_column_types` defaults probe) with no schema change. Quoting is deliberately **asymmetric** vs scalars: a scalar string's quotes are author-written (`status = '{v}'`), but list elements are **auto-quoted** at render time — a single `{var}` placeholder can't carry per-element quotes, so `col IN ({regions})` renders `IN ('US', 'CA')`. Mode-B (`escape="python"`) appends a **trailing comma** (`('US',)`) so the Python-AST parser always reads a tuple, never `str` containment for a 1-element list. An **empty list raises** rather than emitting `IN ()` (invalid SQL): "no filter" semantics belong to a sentinel default, not an empty list (DEV-1730 acceptance). Per-element escaping reuses `_escape_string_value`, so DEV-1727's dialect-aware escaping composes automatically. +- 2026-08-03 — Optional blocks + Cube JS/FILTER_PARAMS import (DEV-1730 / #270): a Mode-A-only `{? ... ?}` block renders its content parenthesised when every inner `{var}` is supplied, else collapses to the neutral `(1=1)` — the SLayer form of a Cube `FILTER_PARAMS` optional pushdown. Blocks live in the same `substitute_variables` (escape="sql") scanner as `{var}`/`{{`/`}}`, must contain ≥1 var, do not nest, and are rejected in Mode-B. A block-bearing model runs substitution even on a zero-variable call so its blocks collapse (the `_substitute_model_sql_surfaces` fast-path now checks for `{?` too); a block-free, required-only model with zero variables is still left untouched (the documented DEV-1625 raw-brace-literal boundary). `extract_model_variables(model)` derives required (bare, no default) vs optional (in-block or defaulted) from the four Mode-A surfaces — structural, nothing persisted, surfaced additively in the inspect skeleton `Variables:` line. The Cube importer gains a **JavaScript front-end** (esprima ESTree parser, a new core dep) that parses `cube()`/`view()` into the same `CubeCube`/`CubeView` shapes as YAML (dynamic values → report + skip member). FILTER_PARAMS refs are carried JS→converter as structured `CubeFilterParamRef` on the transient `CubeCube` (sentinels in the surface text; no arrow-body re-parse, sidestepping the `{var}`-vs-`{FILTER_PARAMS…}` brace clash); the converter resolves sentinels AFTER `translate_cube_refs` so the introduced `{var}` are never eaten. Requiredness (bare vs block) is decided in the converter alone via `honor_required_meta` (default on; CLI `--ignore-required-meta`) AND the member's `meta.required`; with the flag off a scalar-position arrow collapses to Cube's own `(1=1)::TIMESTAMP` booby-trap, faithfully. Cross-cube refs, unknown members, and generated-name collisions (`d`→`d_from` clashing member `d_from`) drop the cube (`filter_params_unsupported`); each logical variable is reported once (`filter_params_variable`) and stashed in `meta.cube_variables`. `render_probe_text` (blocks→`(1=1)`, bare vars→`0`) is the single import-time validation renderer, matching runtime collapse. +- 2026-08-04 — Dialect-aware / complete escaping for Mode-A `{variable}` substitution (DEV-1727), hardening DEV-1625. `substitute_variables(..., escape="sql")` is now **dialect-aware** and **fail-closed**: it gained a required keyword-only `backslash_escapes` signal (`bool | None`, raises if `None` in sql mode) so a caller rendering raw SQL can never silently under-escape. On backslash-escaping dialects (MySQL/ClickHouse/Snowflake/Redshift/BigQuery/Databricks/Spark) it doubles the backslash before escaping the single quote; on standard dialects it keeps the `''` quote-doubling. The double quote is deliberately left untouched — inside a single-quoted literal `\"` is NOT a recognised escape on 6 of the 7 backslash dialects (only MySQL), so escaping it would corrupt the value. The regime is DERIVED from sqlglot's own tokenizer via `SqlDialect.backslash_escapes_strings` (= `"\\" in tokenizer.STRING_ESCAPES`, guarded + 14-dialect pinned) so our escaping can never drift from the parser that reads the substituted SQL. `escape="python"` (Mode-B) additionally encodes the full C0 control range (`\t`/`\n`/`\r` named, rest `\xNN`) so raw newlines/NUL no longer break `ast.parse`. Engine fail-closed: `_substitute_model_sql_surfaces` / `_render_probe_model` require a `dialect`, threaded from the resolved datasource — no bare bool to forget. Assumes MySQL's default `sql_mode` (backslash escapes on); `NO_BACKSLASH_ESCAPES` servers are a sqlglot-layer-wide limitation, documented not fixed. The SQLite backslash end-to-end gap stays a pinned strict-xfail (pre-existing, out of scope). Bound parameters rejected (don't fit substitute-into-raw-SQL). Nested/join/cross-model lineages remain DEV-1678. +- 2026-08-04 — Declared list-valued `{variable}` coercion (DEV-1730 follow-up): a scalar supplied for a variable the model declares `list_valued` is wrapped into a one-element list before Mode-A substitution, so an importer-generated `col IN ({var})` renders `IN ('US')` rather than the unquoted `IN (US)`. The generic scalar rule (author writes the quotes, so `{var}` also works in numeric/fragment positions like `amount >= {floor}` and `{d}::TIMESTAMP`) is CORRECT and unchanged — it just presumes an author who can see the SQL position, which a machine-generated fixed template does not have; the caller cannot supply per-element quotes through parentheses the importer wrote. Silent-wrong-answer risk drove the fix over a raise: `region IN (US)` parses as a column reference, so it fails at the database with a confusing message, or resolves against a real column and returns wrong rows. Opt-in is a front-end-NEUTRAL flag: the Cube converter writes `list_valued: ref.kind == "string"` into each `meta.cube_variables` entry (arrow forms splice pre-quoted scalars and stay `False`), and the engine reads only that flag — never Cube's `kind` taxonomy — so a future list-shaped front-end opts in the same way. Coercion lives at the single Mode-A choke point `_substitute_model_sql_surfaces` (execution and the `_render_probe_model` type-probe both route through it, so it cannot be bypassed) via `coerce_declared_list_variables` / `list_valued_variable_names` in `slayer/core/query.py`. Scope is deliberately narrow: only `str`/`int`/`float`/`bool` are wrapped; `list`/`tuple` pass through (the **empty list still raises** — "no filter" belongs to an optional block or a sentinel default); `None`/`dict` are left for `_render_variable_value` to reject with its own naming error; hand-written models declare nothing and are untouched. Follow-on from the same review: `declares_variables(model)` (any non-empty `meta.cube_variables`) now also defeats the DEV-1625 zero-variable fast path, via the shared `_model_needs_substitution_pass` predicate used by both `_substitute_model_sql_surfaces` and `_render_probe_model`. This closes the fast-path hole for a GENERATED model whose pushdowns are all required (no `{? ?}` block to force the pass): such a model used to emit a bare `{var}` into the SQL on a zero-variable call instead of raising the documented missing-variable error. The hole stays open — deliberately — for hand-written models, which declare nothing and keep the raw-brace-literal protection (`'{1,2,3}'`). The `list_valued` flag is matched with `is True`, not truthiness, since `meta` is user-extensible and a stray `1` or the string `"false"` must not switch substitution semantics. The bag is also SELF-IDENTIFYING — an entry counts only with a string `member` (the shape every importer writes) — so a hand-written `meta` that reuses the `cube_variables` key is not mistaken for generated SQL and silently stripped of its brace-literal protection. diff --git a/docs/concepts/models.md b/docs/concepts/models.md index 89fd4fc8..030c2c34 100644 --- a/docs/concepts/models.md +++ b/docs/concepts/models.md @@ -386,15 +386,29 @@ Rules: - **Same precedence** as everywhere else (runtime kwarg > stage > outer query > `model.query_variables`), and the same `{{`/`}}` literal-brace escaping. - **Raise on missing — once any variable is in play.** As soon as at least one variable is supplied (a `query_variables` default, or a caller/stage/runtime value), every `{var}` placeholder must resolve or execution raises `Undefined variable` — a parameterized model is meant to fail without its value, not silently render a neutral predicate. -- **Variable-free executions treat braces as literals.** When there is *no* variable in play at all (no `query_variables` and no caller variables), the four surfaces are left untouched: a raw brace literal (e.g. a Postgres array `'{1,2,3}'`) survives verbatim, and a placeholder-shaped token like `'{status}'` is emitted as-is rather than raising. This is the deliberate contract that lets brace-bearing SQL coexist with the feature. If a model *does* use variables, escape any literal braces as `{{`/`}}`. -- **String escaping is Mode-A-aware.** Write the surrounding quotes yourself (`WHERE region = '{region}'`); a string value's embedded single quotes are doubled so it stays inside that literal. Numbers (including booleans) insert verbatim; non-finite floats are rejected. -- **`inspect` / `inspect_model` show the literal template** (`{floor}`), not a rendered value. +- **Variable-free executions treat braces as literals.** When there is *no* variable in play at all (no `query_variables` and no caller variables), the four surfaces are left untouched: a raw brace literal (e.g. a Postgres array `'{1,2,3}'`) survives verbatim, and a placeholder-shaped token like `'{status}'` is emitted as-is rather than raising. This is the deliberate contract that lets brace-bearing SQL coexist with the feature. If a model *does* use variables, escape any literal braces as `{{`/`}}`. Two kinds of model are exempt and always run substitution — one carrying an optional `{? ... ?}` block (so the block collapses), and a **generated** model that declares its variables in `meta.cube_variables` (an importer wrote the template, so there is no brace-literal ambiguity to protect and a missing variable raises normally). +- **String escaping is Mode-A- and dialect-aware.** Write the surrounding quotes yourself (`WHERE region = '{region}'`); a string value's embedded single quotes are escaped so it stays inside that literal. The regime follows the datasource dialect: standard backends (SQLite, Postgres, DuckDB, …) double the quote (`''`); backslash-escaping backends (MySQL, ClickHouse, Snowflake, Redshift, BigQuery, Databricks, Spark) also escape backslashes, so a value like `a\'b` can't break out of the literal (DEV-1727). Numbers (including booleans) insert verbatim; non-finite floats are rejected. +- **List values render an `IN`-list.** A `list` (or `tuple`) variable renders a comma-separated, injection-safe `IN`-list body. Write the parentheses yourself and **omit** per-element quotes — each string element is auto-quoted for you (the opposite of the scalar-string rule above, because a single placeholder can't carry per-element quotes): + + ```json + { "filters": ["region IN ({regions})"] } + ``` + + with `variables={"regions": ["US", "CA"]}` renders `region IN ('US', 'CA')`. Elements must be strings or numbers; an **empty list raises** (`IN ()` is invalid SQL — for "no filter" semantics use a sentinel default rather than an empty list). Passing a bare scalar here (`{"regions": "US"}`) renders `IN (US)` — unquoted, per the scalar rule above — so wrap single values in a one-element list: `{"regions": ["US"]}`. (The exception is a variable a *generated* model declares list-valued: see [importing Cube](../cube/cube_import.md#scalars-are-accepted-for-the-string-form), where the template is machine-written and a scalar is normalized for you.) +- **Optional blocks `{? ... ?}` — a filter that disappears when its variable is absent.** Wrap a predicate in `{? ... ?}` (Mode-A surfaces only): when every `{var}` inside is supplied, the block renders parenthesised; when any is missing, the whole block collapses to the neutral `(1=1)`. This is the SLayer form of a Cube `FILTER_PARAMS` optional pushdown. Put the `AND` outside the block so the collapse leaves valid SQL (open your `WHERE` with `1=1`): + + ```json + { "sql": "SELECT * FROM orders WHERE 1=1 AND {? region IN ({regions}) ?}" } + ``` + + With `variables={"regions": ["US","CA"]}` this renders `... WHERE 1=1 AND (region IN ('US', 'CA'))`; with no `regions` supplied it renders `... WHERE 1=1 AND (1=1)`. A block must contain at least one `{var}`; blocks do not nest; a block collapses even on a **zero-variable** call (unlike bare placeholders, which are left literal when no variable is in play at all). Optional blocks are rejected in Mode-B query filters. +- **`inspect` / `inspect_model` show the literal template** (`{floor}`), not a rendered value. A `Variables:` line lists the model's placeholders classified **required** (bare, no default — omitting it raises) vs **optional** (inside a block, or carrying a `query_variables` default). The classification is derived from the SQL, not stored, so it can never drift from the template. **Scope (DEV-1625):** substitution currently applies to the **direct source model** of a query. Nested `source_queries` stages, query-backed direct sources, join-target models, and cross-model-target models are the deferred follow-up ([DEV-1678](https://linear.app/motley-ai/issue/DEV-1678)). A `{var}` in one of those lineages is left untouched (and surfaces as an error on the stray placeholder) until that lands. -**Trusted input.** Substituted values are treated as trusted, not attacker-controlled. The Mode-A escaping doubles single quotes so a value stays inside the quoted literal you wrote, but it is deliberately *not* dialect-aware: on a backend where a backslash escapes the following quote (e.g. **MySQL** with default settings), a value containing a backslash immediately before a quote could still break out of the literal. Do not feed untrusted end-user input through `variables`; fuller dialect-aware escaping / bound parameters is tracked as a follow-up. +**Trusted input.** Substituted values are still treated as trusted, not attacker-controlled — prefer not to feed untrusted end-user input through `variables`. The Mode-A escaping is now dialect-aware (DEV-1727): it keeps a string value inside the quoted literal you wrote on every supported dialect, including backslash-escaping backends like MySQL and ClickHouse. Two residual caveats: a `{var}` placed in an **unquoted** position is still raw substitution (only *quoted* string literals are escaped); and the backslash-dialect escaping assumes the server's **default** string mode — a MySQL server running with `sql_mode=NO_BACKSLASH_ESCAPES` treats backslash as an ordinary char, which the whole sqlglot dialect layer (not just this feature) assumes is off. -**Known limitation:** a string value containing a backslash also does not round-trip through every backend (notably SQLite, whose driver does not unescape backslashes) — a pre-existing dialect quirk, independent of variable substitution. +**Known limitation:** a string value containing a backslash does not round-trip through **SQLite** end-to-end (its driver does not unescape backslashes) — a pre-existing dialect quirk, independent of variable substitution and out of scope for DEV-1727. ### Variable precedence diff --git a/docs/concepts/queries.md b/docs/concepts/queries.md index 9dace5db..055ea85c 100644 --- a/docs/concepts/queries.md +++ b/docs/concepts/queries.md @@ -269,7 +269,19 @@ Filters support `{variable_name}` placeholders, substituted from the query's `va This produces the filter `status = 'completed' AND amount > 100`. - Variable names must be alphanumeric + underscore (`[a-zA-Z_][a-zA-Z0-9_]*`) -- Values must be strings or numbers. **You write the surrounding quotes** in the template (`status = '{status}'`); the string value is then automatically escaped so an embedded quote (e.g. `O'Brien`) can't break out of that literal. Numbers (including booleans) are inserted verbatim; non-finite floats (`nan`/`inf`) are rejected. +- Values must be strings, numbers, or **lists**. **You write the surrounding quotes** in the template (`status = '{status}'`); the string value is then automatically escaped so an embedded quote (e.g. `O'Brien`), backslash, or control character (newline, tab) can't break out of that literal or the filter parser (DEV-1727). Numbers (including booleans) are inserted verbatim; non-finite floats (`nan`/`inf`) are rejected. +- **List values render an `IN`-list.** A list variable powers an `in` / `not in` filter — write the parentheses and omit per-element quotes (each string element is auto-quoted): + + ```json + { + "source_model": "orders", + "measures": ["*:count"], + "filters": ["region in ({regions})"], + "variables": {"regions": ["US", "CA"]} + } + ``` + + An empty list raises (`IN ()` is invalid SQL). Under the hood a list renders as a Python tuple (with a trailing comma, e.g. `('US',)`) so that even a single-element list is treated as membership rather than a bare value — you may see that trailing comma in `show_sql` output; it is not a typo. - `{{` and `}}` produce literal `{` and `}` - Undefined variables raise an error - The same `{variable}` mechanism also works in the **raw-SQL (Mode A) surfaces** of a model — see [Variables in model SQL](models.md#variables-in-model-sql). diff --git a/docs/concepts/references.md b/docs/concepts/references.md index 10abbb12..9e3a292f 100644 --- a/docs/concepts/references.md +++ b/docs/concepts/references.md @@ -19,7 +19,7 @@ SLayer has two distinct expression layers and the rules for what each one accept * User-supplied multi-dot input (`a.b.c`) is auto-rewritten to `a__b.c` at validation time with a warning. * Other derived columns of the same model (or of a joined model via `__`) are recursively expanded so chains like `A.ratio = "A.bar / B.foo_normalized"` (where `B.foo_normalized` is itself derived) work. * `ModelMeasure` names are not visible from SQL mode — saved measures are DSL-only. -* `{variable}` placeholders are substituted into these Mode-A surfaces from the merged variable set (raise-on-missing once any variable is in play; a fully variable-free execution leaves braces as literals; Mode-A-aware string escaping). See [Variables in model SQL](models.md#variables-in-model-sql). +* `{variable}` placeholders are substituted into these Mode-A surfaces from the merged variable set (raise-on-missing once any variable is in play; a fully variable-free execution leaves braces as literals; Mode-A- and dialect-aware string escaping, so quoted values round-trip on backslash-escaping backends like MySQL/ClickHouse — DEV-1727). See [Variables in model SQL](models.md#variables-in-model-sql). ### DSL mode (queries + `ModelMeasure.formula`) diff --git a/docs/cube/cube_import.md b/docs/cube/cube_import.md new file mode 100644 index 00000000..f6ce4461 --- /dev/null +++ b/docs/cube/cube_import.md @@ -0,0 +1,215 @@ +# Importing Cube definitions + +SLayer can import [Cube](https://cube.dev) (Cube.js / Cube.dev) data models — +cubes and views, in **YAML or JavaScript** — and convert them to SLayer models. +The conversion is **fully offline**: data types come from Cube's declared +dimension / measure types, so no database connection is required. Everything that +can't be mapped cleanly is captured in a structured JSON report rather than +silently dropped. + +## Quick start + +```bash +slayer import-cube ./cube_project --datasource my_postgres --storage ./slayer_data +``` + +This recursively reads every `.yml`/`.yaml` **and `.js`** file under the path +(skipping `node_modules` / `target`), extracts `cubes:` / `views:` (YAML) and +`cube('Name', {...})` / `view(...)` calls (JavaScript), writes SLayer model files +to the storage directory, and writes `cube_import_report.json` next to it. + +`--datasource` is just the SLayer datasource name to file the models under — it +does not need to exist or be reachable. After importing, run `slayer ingest` +against a live connection to profile sample values and refine numeric types. + +## What gets converted + +### Cubes → models + +Each cube becomes one `SlayerModel` anchored on its `sql_table` (or `sql`). + +| Cube | SLayer | +|------|--------| +| `name` | `name` | +| `sql_table` / `sql` | `sql_table` / `sql` (with `{CUBE}`/`{member}` refs translated) | +| `description` | `description` | +| `public: false` | `hidden: true` | +| `meta` (incl. `ai_context`) | `meta` | +| `title` | `meta.cube_title` | + +### Measures → columns + measures + +Cube bakes the aggregation into each measure; SLayer separates the row-level +expression (a `Column`) from the named aggregation (a `ModelMeasure`). + +```yaml +# Cube +measures: + - { name: total_revenue, type: sum, sql: "{CUBE}.amount" } +# SLayer +columns: + - { name: amount, type: DOUBLE } +measures: + - { name: total_revenue, formula: "amount:sum" } +``` + +- `count` with no `sql` → `*:count`; `count_distinct_approx` → `count_distinct`. +- Conditional `filters:` become a `CASE WHEN` on the column's `filter`. Two + measures over the same expression but different filters get distinct columns. +- A finite trailing `rolling_window` becomes a windowed aggregation + (`amount:sum(window='30d')`). +- Calculated measures (`type: number/string/time/boolean`) referencing other + measures become a `ModelMeasure` formula (`{revenue} / {count}` → `revenue / count`). +- `format` maps to `NumberFormat` (`percent`, `currency`, `number`). + +### Dimensions → columns + +`string`→`TEXT`, `number`→`DOUBLE`, `boolean`→`BOOLEAN`, `time`→`TIMESTAMP`. +`primary_key: true` carries over. A `case:` dimension becomes a `CASE WHEN` +column. + +### Joins + +A join's ON clause (`{CUBE}.customer_id = {customers.id}`) becomes +`join_pairs`; member references resolve to their physical columns. Composite +(`AND`-joined) keys are supported. All joins emit as `LEFT`. + +### Segments → boolean columns + +Each segment becomes a boolean column carrying the predicate, so it stays +filterable (`completed = true`) and group-able. + +### Views → facade models + +A Cube view (which owns no table) becomes a thin model anchored on its +`join_path` root cube: included dimensions become derived columns +(`sql: "customers.name"`), and included measures become local or cross-model +`ModelMeasure`s that reference the measure's **underlying column** — a joined +measure `revenue` with `sql: {CUBE}.amount` becomes `customers.amount:sum`, not +`customers.revenue:sum`. `prefix: true` prepends the cube name, and +`default_filters` become model filters. + +### `extends` + +Cube inheritance is **flattened** at import time — a child inherits the parent's +members (child wins on conflicts), and abstract bases (`public: false`) are +emitted as hidden models. + +### JavaScript configs + +`cube('Name', { ... })` and `view('Name', { ... })` calls in `.js` files are +parsed (via a pure-Python ESTree parser) into the same shapes as the YAML path, +so every mapping above applies equally. Supported: template-literal strings +(`` `...` ``) including `${CUBE}` / `${member}` / `${a.b}` refs, object and array +literals, `//` and `/* */` comments, `module.exports =` / `export default` +wrappers, `import` / `export` (ES-module) configs, and multiple cubes per file. +Member keys are accepted in either `camelCase` (`primaryKey`) or `snake_case` +(`primary_key`); `meta` is preserved verbatim. + +Anything **dynamic** — a helper call (`sql: buildSql()`), a spread (`...base`), a +bare identifier reference (`sql: someConst`), or a computed key — can't be +resolved offline. The affected member is skipped with a report issue and the rest +of the cube still converts (one bad dimension does not sink the model). The parser +targets ES2017; a config using newer syntax is reported rather than crashing. One +Stage-1 gap: a JS **view** whose `join_path` is written as a bare cube identifier +(`join_path: Orders`) rather than a string is skipped — write it as a string +(`join_path: 'Orders'`) for now. + +### FILTER_PARAMS pushdowns + +Cube's `FILTER_PARAMS...filter(...)` renders a member's filter when +the caller supplies one, else the neutral `1 = 1`. SLayer represents this with +[`{variable}` substitution](../concepts/models.md#variables-in-model-sql): + +- **String form** `.filter('col')` → an optional block `{? col IN ({member}) ?}` + — the caller passes a list (`{"brand": ["Acme", "Zeta"]}`), a bare scalar + (`{"brand": "Acme"}`, normalized to a one-element list — see below), or omits + it and the block collapses to `(1=1)`. +- **Arrow form** `.filter((from, to) => ...)` → the body is emitted with `from` / + `to` spliced as the pre-quoted variables `{_from}` / `{_to}`. + A bare-param body used in a scalar position (`...filter((from,to)=>from)::TIMESTAMP`) + is handled too. + +#### Scalars are accepted for the string form + +The general [`{variable}` rule](../concepts/models.md#variables-in-model-sql) puts +quoting on whoever writes the template — a scalar string substitutes *unquoted*, +which is what makes `amount >= {floor}` and `'{d}'::TIMESTAMP` both expressible. +That rule assumes an author who can see the SQL position, and the importer's +`col IN ({member})` template has none: the parentheses are generated, so the +caller has nowhere to put the quotes and a bare `"Acme"` would otherwise render +`IN (Acme)` — a column reference that parses cleanly and then fails at the +database. + +So the importer marks each string-form variable `list_valued` in +`meta.cube_variables`, and the engine wraps a scalar into a one-element list +before substituting. An entry counts as a declaration only when it carries a +string `member` (the shape the importer always writes), so unrelated `meta` +that happens to reuse the `cube_variables` key is never mistaken for generated +SQL. In `IN (...)` position a scalar and a one-element list mean +the same thing, so `{"brand": "Acme"}` and `{"brand": ["Acme"]}` are equivalent. +This applies **only** to variables a model declares that way — hand-written +model SQL keeps the author-written-quotes convention unchanged. An **empty list** +still raises (`IN ()` is invalid SQL); to mean "no filter", omit the variable and +let the block collapse. + +Whether a pushdown is **required** (bare `{var}`, omitting it raises a clear error) +or **optional** (wrapped in a block) is decided by the referenced member's +`meta.required`: a truthy value ⇒ required. Pass `--ignore-required-meta` to emit +every pushdown as optional (literal Cube semantics) instead. Each emitted variable +is listed in the report (`filter_params_variable`), and the model stashes the +member/required/kind/description of each variable under `meta.cube_variables`. + +Two pushdown shapes are representable in Stage 1: the **string form** (set +membership — `IN` / equals) and the **arrow form** (a date range built from the +`from` / `to` bounds, as above). Cube's other per-operator filter helpers +(`contains`, `gt`, `startsWith`, …) are not represented as their own forms. A +cross-cube reference, an unknown member, a generated-name collision, or an +arrow form in a YAML (non-JS) config is reported as `filter_params_unsupported` +and drops the cube. + +## What does not map (reported) + +These are recorded in `cube_import_report.json` and, where useful, preserved +under `meta.cube_unmapped.`: + +- Caching / infra: `pre_aggregations`, `refresh_key`, `calendar`, `sql_alias`. +- Presentation: `hierarchies`, `drill_members`, folders, dimension `links`/`order`. +- Security: `access_policy`. +- No SLayer equivalent: `geo` dimensions, `sub_query` dimensions, custom + `granularities`, per-cube `data_source`. +- Non-equality / non-column join ON clauses (the join is dropped). +- Files or members using Jinja templating (`{{ }}` / `{% %}`) — skipped, since + conversion is offline and does not render templates. + +### Tesseract features (deferred) + +Features that require the Tesseract SQL planner — `switch` dimensions, +`number_agg` measures, `case` measures, and the measure `filter` grain control — +have no clean SLayer mapping yet and are reported as `deferred_stage2`. The +cube's other members still convert. + +## The report + +`CubeConversionResult` carries the emitted `models` and a `CubeConversionReport` +of categorized issues (each with a category, severity, the owning cube/view/member, +a message, and the raw Cube fragment when useful). The CLI always writes it to +`cube_import_report.json` (override with `--report PATH`) and prints a summary +grouped by severity. + +## CLI reference + +```text +slayer import-cube [options] + +Arguments: + cube_project_path Path to the Cube project (or its model directory) + +Options: + --datasource NAME SLayer datasource name for the imported models (required) + --storage PATH Storage directory / .db file (default: platform path) + --report PATH JSON report path (default: /cube_import_report.json) + --include-hidden Also print hidden (public: false) models in the summary + --ignore-required-meta Emit every FILTER_PARAMS pushdown as optional (ignore + member meta.required) +``` diff --git a/docs/database-support.md b/docs/database-support.md index e70256f6..fa652472 100644 --- a/docs/database-support.md +++ b/docs/database-support.md @@ -139,6 +139,13 @@ If you need percentiles on MySQL, the recommended options are: - Define a custom `Aggregation` on the model with whatever `GROUP_CONCAT`- based or windowed expression suits your data shape and group sizes. +**String-literal escaping** (`{variable}` substitution and generated SQL +generally) assumes MySQL's **default** `sql_mode` where a backslash escapes +the following character. A server running with `sql_mode=NO_BACKSLASH_ESCAPES` +treats backslash as an ordinary char; SLayer's entire sqlglot-based MySQL +emission — not just variable substitution — assumes that mode is off, so +backslash-bearing string values are not supported on such servers. + ### SQL Server (T-SQL) caveats T-SQL has `STDEV`/`STDEVP`/`VAR`/`VARP` (not `STDDEV_SAMP`/`STDDEV_POP`/ diff --git a/docs/examples/14_variable_substitution/variable_substitution.md b/docs/examples/14_variable_substitution/variable_substitution.md index 266505b3..49df210a 100644 --- a/docs/examples/14_variable_substitution/variable_substitution.md +++ b/docs/examples/14_variable_substitution/variable_substitution.md @@ -47,6 +47,38 @@ SQL — not as a filter on the result. The `{floor}` is substituted into the model's `WHERE` before the query runs. Write the surrounding quotes yourself for string values (`region = '{region}'`); SLayer escapes the value so an embedded quote can't break out of the literal. +The one place you *don't* write quotes is an `IN`-list — `region IN ({regions})` +takes a list and quotes each element for you (a bare string there would land +unquoted). Models generated by `import-cube` normalize a scalar to a +one-element list for those pushdowns. + +## Optional blocks — a filter that vanishes when its value is absent + +The surfaces above **require** their variables. But a Cube `FILTER_PARAMS` +pushdown is *optional*: it filters when the caller supplies a value and becomes a +no-op when they don't. SLayer expresses that with an **optional block** +`{? ... ?}` on a Mode-A surface — it renders (parenthesised) when every `{var}` +inside is supplied, and collapses to the neutral `(1=1)` otherwise. A **list** +value renders an injection-safe `IN`-list (write the parens; *string* elements +are auto-quoted, numbers stay unquoted). Open the `WHERE` with `1=1` so the +collapse leaves valid SQL. + +```json +{ + "source_model": { + "name": "orders_by_store", + "data_source": "jaffle_shop", + "sql": "SELECT o.id, s.name AS store_name FROM orders o LEFT JOIN stores s ON o.store_id = s.id WHERE 1=1 AND {? s.name IN ({stores}) ?}" + }, + "measures": ["*:count"], + "variables": {"stores": ["Brooklyn", "Philadelphia"]} +} +``` + +With `stores` supplied this renders `... AND (s.name IN ('Brooklyn', 'Philadelphia'))`; +omit `stores` and the whole block becomes `... AND (1=1)`, counting every order. +This is exactly how `slayer import-cube` represents an optional Cube +`FILTER_PARAMS` pushdown — see [Importing Cube definitions](../../cube/cube_import.md#filter_params-pushdowns). ## Precedence @@ -61,14 +93,19 @@ query or per call. ## Contract - **Raise on missing — once any variable is in play.** As soon as one variable is - supplied, every `{var}` must resolve or execution raises. A parameterized model - is meant to fail loudly without its value, not silently match nothing. + supplied, every *required* `{var}` must resolve or execution raises. A + parameterized model is meant to fail loudly without its value, not silently + match nothing. (The exception is a `{var}` **inside a `{? ... ?}` block** — + when it's absent the whole block collapses to `(1=1)` instead of raising.) - **Fully variable-free executions treat braces as literals**, so a raw brace literal such as a Postgres array `'{1,2,3}'` survives untouched. Use `{{` / `}}` for literal braces in a model that *does* use variables. -- **Trusted input.** Values are treated as trusted, not attacker-controlled. The - Mode-A escaping is not dialect-aware, so avoid untrusted values on - backslash-escaping backends like MySQL. +- **Trusted input.** Values are treated as trusted, not attacker-controlled — + prefer not to feed untrusted end-user input through `variables`. The Mode-A + escaping is dialect-aware (DEV-1727), so a quoted string value stays inside its + literal on every backend, including backslash-escaping ones like MySQL and + ClickHouse; only *quoted* literals are escaped, and a `{var}` in an unquoted + position is still raw substitution. Substitution currently applies to a query's **direct source model**; nested `source_queries` stages, join targets, and cross-model targets are a tracked diff --git a/docs/examples/14_variable_substitution/variable_substitution_nb.ipynb b/docs/examples/14_variable_substitution/variable_substitution_nb.ipynb index 67429bf8..9a766a4e 100644 --- a/docs/examples/14_variable_substitution/variable_substitution_nb.ipynb +++ b/docs/examples/14_variable_substitution/variable_substitution_nb.ipynb @@ -30,10 +30,10 @@ "id": "dff26f5e", "metadata": { "execution": { - "iopub.execute_input": "2026-08-03T11:04:31.921375Z", - "iopub.status.busy": "2026-08-03T11:04:31.920640Z", - "iopub.status.idle": "2026-08-03T11:04:32.212128Z", - "shell.execute_reply": "2026-08-03T11:04:32.211469Z" + "iopub.execute_input": "2026-08-04T10:22:58.180073Z", + "iopub.status.busy": "2026-08-04T10:22:58.179647Z", + "iopub.status.idle": "2026-08-04T10:22:58.509255Z", + "shell.execute_reply": "2026-08-04T10:22:58.507587Z" } }, "outputs": [], @@ -67,10 +67,10 @@ "id": "7ef41cd4", "metadata": { "execution": { - "iopub.execute_input": "2026-08-03T11:04:32.214011Z", - "iopub.status.busy": "2026-08-03T11:04:32.213851Z", - "iopub.status.idle": "2026-08-03T11:04:32.373902Z", - "shell.execute_reply": "2026-08-03T11:04:32.373324Z" + "iopub.execute_input": "2026-08-04T10:22:58.514845Z", + "iopub.status.busy": "2026-08-04T10:22:58.514398Z", + "iopub.status.idle": "2026-08-04T10:22:58.683309Z", + "shell.execute_reply": "2026-08-04T10:22:58.683020Z" } }, "outputs": [ @@ -78,8 +78,8 @@ "name": "stdout", "output_type": "stream", "text": [ - "Brooklyn 255,284 orders $ 2,828,722.39\n", - "Philadelphia 194,223 orders $ 2,140,972.43\n" + "Brooklyn 255,791 orders $ 2,785,451.75\n", + "Philadelphia 187,076 orders $ 2,164,465.40\n" ] } ], @@ -120,10 +120,10 @@ "id": "b6ff57ed", "metadata": { "execution": { - "iopub.execute_input": "2026-08-03T11:04:32.375561Z", - "iopub.status.busy": "2026-08-03T11:04:32.375432Z", - "iopub.status.idle": "2026-08-03T11:04:32.431157Z", - "shell.execute_reply": "2026-08-03T11:04:32.429271Z" + "iopub.execute_input": "2026-08-04T10:22:58.684412Z", + "iopub.status.busy": "2026-08-04T10:22:58.684332Z", + "iopub.status.idle": "2026-08-04T10:22:58.713555Z", + "shell.execute_reply": "2026-08-04T10:22:58.712974Z" } }, "outputs": [ @@ -131,8 +131,8 @@ "name": "stdout", "output_type": "stream", "text": [ - "order_total >= 0: 661,036 orders\n", - "order_total >= 50: 15,083 orders\n", + "order_total >= 0: 655,380 orders\n", + "order_total >= 50: 15,218 orders\n", "\n", "Generated WHERE: orders_over_floor.order_total >= 50\n" ] @@ -181,10 +181,10 @@ "id": "84aaa071", "metadata": { "execution": { - "iopub.execute_input": "2026-08-03T11:04:32.437208Z", - "iopub.status.busy": "2026-08-03T11:04:32.436529Z", - "iopub.status.idle": "2026-08-03T11:04:32.472382Z", - "shell.execute_reply": "2026-08-03T11:04:32.471057Z" + "iopub.execute_input": "2026-08-04T10:22:58.715290Z", + "iopub.status.busy": "2026-08-04T10:22:58.715127Z", + "iopub.status.idle": "2026-08-04T10:22:58.728822Z", + "shell.execute_reply": "2026-08-04T10:22:58.728508Z" } }, "outputs": [ @@ -192,7 +192,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "{'floored_orders.floor_used': 60, 'floored_orders._count': 8384}\n" + "{'floored_orders.floor_used': 60, 'floored_orders._count': 9947}\n" ] } ], @@ -237,10 +237,10 @@ "id": "c2ba901c", "metadata": { "execution": { - "iopub.execute_input": "2026-08-03T11:04:32.475010Z", - "iopub.status.busy": "2026-08-03T11:04:32.474685Z", - "iopub.status.idle": "2026-08-03T11:04:32.508696Z", - "shell.execute_reply": "2026-08-03T11:04:32.507998Z" + "iopub.execute_input": "2026-08-04T10:22:58.730482Z", + "iopub.status.busy": "2026-08-04T10:22:58.730376Z", + "iopub.status.idle": "2026-08-04T10:22:58.756381Z", + "shell.execute_reply": "2026-08-04T10:22:58.756152Z" } }, "outputs": [ @@ -248,8 +248,8 @@ "name": "stdout", "output_type": "stream", "text": [ - "raw sum = $7,339,099.99\n", - "scaled (x2) = $14,678,199.98\n" + "raw sum = $7,375,596.31\n", + "scaled (x2) = $14,751,192.62\n" ] } ], @@ -290,10 +290,10 @@ "id": "e050aa9f", "metadata": { "execution": { - "iopub.execute_input": "2026-08-03T11:04:32.510123Z", - "iopub.status.busy": "2026-08-03T11:04:32.509976Z", - "iopub.status.idle": "2026-08-03T11:04:32.542530Z", - "shell.execute_reply": "2026-08-03T11:04:32.541253Z" + "iopub.execute_input": "2026-08-04T10:22:58.757490Z", + "iopub.status.busy": "2026-08-04T10:22:58.757418Z", + "iopub.status.idle": "2026-08-04T10:22:58.813798Z", + "shell.execute_reply": "2026-08-04T10:22:58.813544Z" } }, "outputs": [ @@ -301,8 +301,8 @@ "name": "stdout", "output_type": "stream", "text": [ - "SUM over all orders = $7,339,099.99\n", - "SUM over orders >= $50 (CASE) = $1,062,350.27\n" + "SUM over all orders = $7,375,596.31\n", + "SUM over orders >= $50 (CASE) = $1,103,442.44\n" ] } ], @@ -346,10 +346,10 @@ "id": "09acf0d4", "metadata": { "execution": { - "iopub.execute_input": "2026-08-03T11:04:32.544276Z", - "iopub.status.busy": "2026-08-03T11:04:32.544064Z", - "iopub.status.idle": "2026-08-03T11:04:32.615142Z", - "shell.execute_reply": "2026-08-03T11:04:32.613080Z" + "iopub.execute_input": "2026-08-04T10:22:58.815214Z", + "iopub.status.busy": "2026-08-04T10:22:58.815134Z", + "iopub.status.idle": "2026-08-04T10:22:58.863983Z", + "shell.execute_reply": "2026-08-04T10:22:58.862249Z" } }, "outputs": [ @@ -357,8 +357,8 @@ "name": "stdout", "output_type": "stream", "text": [ - "model default (floor=100) : 96\n", - "query variable (floor=50) : 15,083\n" + "model default (floor=100) : 89\n", + "query variable (floor=50) : 15,218\n" ] }, { @@ -414,10 +414,10 @@ "id": "73da5a81", "metadata": { "execution": { - "iopub.execute_input": "2026-08-03T11:04:32.619323Z", - "iopub.status.busy": "2026-08-03T11:04:32.618943Z", - "iopub.status.idle": "2026-08-03T11:04:32.660767Z", - "shell.execute_reply": "2026-08-03T11:04:32.660308Z" + "iopub.execute_input": "2026-08-04T10:22:58.868294Z", + "iopub.status.busy": "2026-08-04T10:22:58.867935Z", + "iopub.status.idle": "2026-08-04T10:22:58.912283Z", + "shell.execute_reply": "2026-08-04T10:22:58.912028Z" } }, "outputs": [ @@ -426,7 +426,13 @@ "output_type": "stream", "text": [ "Missing variable raises:\n", - " Undefined variable 'floor' in filter: 'order_total >= {floor}'. Available variables: ['unrelated']\n", + " Undefined variable 'floor' in filter: 'order_total >= {floor}'. Available variables: ['unrelated']\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ "\n", "Value O'Brien is escaped in the generated SQL:\n", " stores.name = 'O''Brien'\n" @@ -460,30 +466,71 @@ }, { "cell_type": "markdown", - "id": "5ab05221", + "id": "d2f1a1d4", "metadata": {}, + "source": "## Optional blocks — a filter that vanishes when its value is absent\n\nEvery surface above **requires** its variables. A Cube `FILTER_PARAMS` pushdown,\nthough, is *optional*: it filters when the caller supplies a value and becomes a\nno-op when they don't. SLayer expresses that with an **optional block**\n`{? ... ?}` on a Mode-A surface — it renders (parenthesised) when every `{var}`\ninside is supplied, and collapses to the neutral `(1=1)` otherwise. A **list**\nvalue renders an injection-safe `IN`-list (write the parens; *string* elements\nare auto-quoted, numbers stay unquoted). Open the `WHERE` with `1=1` so the\ncollapse leaves valid SQL. This is exactly how `slayer import-cube` represents an\noptional Cube `FILTER_PARAMS` pushdown." + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "caae0817", + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-04T10:22:58.913226Z", + "iopub.status.busy": "2026-08-04T10:22:58.913150Z", + "iopub.status.idle": "2026-08-04T10:22:58.992546Z", + "shell.execute_reply": "2026-08-04T10:22:58.992129Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "no filter (block collapses to (1=1)) : 655,380\n", + "stores IN [Brooklyn, Philadelphia] : 442,867\n", + "\n", + "Generated (list -> IN-list): s.name IN ('Brooklyn', 'Philadelphia')\n" + ] + } + ], "source": [ - "## Scope & summary\n", + "orders_by_store = {\n", + " \"name\": \"orders_by_store\",\n", + " \"data_source\": \"jaffle_shop\",\n", + " \"sql\": (\n", + " \"SELECT o.id, o.order_total, s.name AS store_name \"\n", + " \"FROM orders o LEFT JOIN stores s ON o.store_id = s.id \"\n", + " \"WHERE 1=1 AND {? s.name IN ({stores}) ?}\"\n", + " ),\n", + " \"columns\": [\n", + " {\"name\": \"id\", \"sql\": \"id\", \"type\": \"TEXT\", \"primary_key\": True},\n", + " {\"name\": \"order_total\", \"sql\": \"order_total\", \"type\": \"DOUBLE\"},\n", + " {\"name\": \"store_name\", \"sql\": \"store_name\", \"type\": \"TEXT\"},\n", + " ],\n", + "}\n", "\n", - "| Surface | Layer | Example |\n", - "|---|---|---|\n", - "| query `filters` | Mode B (DSL) | `\"stores.name = '{region}'\"` |\n", - "| `SlayerModel.filters` | Mode A (raw SQL) | `\"order_total >= {floor}\"` |\n", - "| `SlayerModel.sql` | Mode A (raw SQL) | `\"... WHERE order_total >= {floor}\"` |\n", - "| `Column.sql` | Mode A (raw SQL) | `\"order_total * {mult}\"` |\n", - "| `Column.filter` | Mode A (raw SQL) | `\"order_total >= {floor}\"` |\n", + "# Omitted -> the block collapses to (1=1): every order counts.\n", + "allc = engine.execute_sync(query={\"source_model\": orders_by_store, \"measures\": [\"*:count\"]})\n", "\n", - "**Contract:** raise-on-missing once any variable is in play; a fully\n", - "variable-free execution leaves braces as literals (so raw brace literals like a\n", - "Postgres array `'{1,2,3}'` survive). Use `{{` / `}}` for literal braces in a\n", - "model that *does* use variables.\n", + "# Supplied (a list) -> renders `s.name IN ('Brooklyn', 'Philadelphia')`.\n", + "some = engine.execute_sync(query={\n", + " \"source_model\": orders_by_store,\n", + " \"measures\": [\"*:count\"],\n", + " \"variables\": {\"stores\": [\"Brooklyn\", \"Philadelphia\"]},\n", + "})\n", "\n", - "**Scope:** substitution applies to a query's **direct source model**. Nested\n", - "`source_queries` stages, join targets, and cross-model targets are a tracked\n", - "follow-up. String values are treated as trusted input (not attacker-controlled);\n", - "the escaping is not dialect-aware, so avoid untrusted values on backslash-escaping\n", - "backends like MySQL." + "print(f\"no filter (block collapses to (1=1)) : {allc.data[0]['orders_by_store._count']:>10,}\")\n", + "print(f\"stores IN [Brooklyn, Philadelphia] : {some.data[0]['orders_by_store._count']:>10,}\")\n", + "clause = [ln.strip() for ln in some.sql.splitlines() if \"IN (\" in ln][0]\n", + "print(\"\\nGenerated (list -> IN-list):\", clause)" ] + }, + { + "cell_type": "markdown", + "id": "5ab05221", + "metadata": {}, + "source": "## Scope & summary\n\n| Surface | Layer | Example |\n|---|---|---|\n| query `filters` | Mode B (DSL) | `\"stores.name = '{region}'\"` |\n| `SlayerModel.filters` | Mode A (raw SQL) | `\"order_total >= {floor}\"` |\n| `SlayerModel.sql` | Mode A (raw SQL) | `\"... WHERE order_total >= {floor}\"` |\n| `Column.sql` | Mode A (raw SQL) | `\"order_total * {mult}\"` |\n| `Column.filter` | Mode A (raw SQL) | `\"order_total >= {floor}\"` |\n| optional block | Mode A (raw SQL) | `\"... AND {? s.name IN ({stores}) ?}\"` |\n\n**Contract:** raise-on-missing once any variable is in play — for *required*\nplaceholders; a `{var}` **inside a `{? ... ?}` block** instead collapses the\nblock to `(1=1)` when absent. A fully variable-free execution leaves braces as\nliterals (so raw brace literals like a Postgres array `'{1,2,3}'` survive); use\n`{{` / `}}` for literal braces in a model that *does* use variables. A **list**\nvalue renders an injection-safe `IN`-list body (*string* elements auto-quoted,\nnumbers unquoted) — together the primitives for representing Cube `FILTER_PARAMS`\npushdowns.\n\n**Scope:** substitution applies to a query's **direct source model**. Nested\n`source_queries` stages, join targets, and cross-model targets are a tracked\nfollow-up. String values are treated as trusted input (not attacker-controlled);\nthe escaping is dialect-aware (DEV-1727), so a quoted value stays inside its\nliteral on backslash-escaping backends like MySQL and ClickHouse too." } ], "metadata": { @@ -507,4 +554,4 @@ }, "nbformat": 4, "nbformat_minor": 5 -} +} \ No newline at end of file diff --git a/poetry.lock b/poetry.lock index 238961f3..537300cc 100644 --- a/poetry.lock +++ b/poetry.lock @@ -1680,6 +1680,17 @@ duckdb = ">=0.5.0" packaging = ">=21" sqlalchemy = ">=1.3.22" +[[package]] +name = "esprima" +version = "4.0.1" +description = "ECMAScript parsing infrastructure for multipurpose analysis in Python" +optional = false +python-versions = "*" +groups = ["main"] +files = [ + {file = "esprima-4.0.1.tar.gz", hash = "sha256:08db1a876d3c2910db9cfaeb83108193af5411fc3a3a66ebefacd390d21323ee"}, +] + [[package]] name = "executing" version = "2.2.1" @@ -7960,4 +7971,4 @@ sqlserver = ["pyodbc"] [metadata] lock-version = "2.1" python-versions = "^3.11" -content-hash = "b4fe34cc49941759e67dfadecc27663a9f4467d0322b06a0a797f8ce5d1703cd" +content-hash = "aa61d7145c9569f852cf60d41cdc64a079de4a68ed23f50afbea30c488b96d17" diff --git a/pyproject.toml b/pyproject.toml index 844f2cf1..9f848e62 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -51,6 +51,13 @@ sqlalchemy-bigquery = {version = ">=1.11", optional = true, python = "<3.15"} duckdb = ">=0.9" duckdb-engine = ">=0.13" jafgen = "^0.4.14" +# esprima is a pure-Python ESTree parser (zero transitive deps) used by the +# Cube JavaScript-config importer (`slayer import-cube` on `.js` files, +# DEV-1730). Core dep so `import-cube` works after a single +# `pip install motley-slayer`. Frozen upstream at ES2017 — sufficient for the +# declarative cube()/view() config subset; newer syntax surfaces as a report +# issue rather than a crash. +esprima = ">=4.0" # rank-bm25 is unmaintained (last release 0.2.2); revisit if supply-chain # concerns arise — the actively-maintained alternative is `bm25s`. rank-bm25 = ">=0.2.2" diff --git a/slayer/cli.py b/slayer/cli.py index db116992..c752675d 100644 --- a/slayer/cli.py +++ b/slayer/cli.py @@ -378,6 +378,29 @@ def main(): # NOSONAR(S3776) — linear top-level CLI command dispatch (one eli ) _add_storage_arg(import_dbt_parser) + # ── import-cube ─────────────────────────────────────────────────── + import_cube_parser = subparsers.add_parser( + "import-cube", + help="Import Cube (Cube.js / Cube.dev) YAML and JavaScript data models into SLayer models", + epilog="""\ +examples: + slayer import-cube ./cube_project --datasource my_postgres + slayer import-cube ./cube_project/model --datasource my_postgres --report ./report.json +""", + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + import_cube_parser.add_argument("cube_project_path", help="Path to the Cube project (or its model directory)") + import_cube_parser.add_argument("--datasource", required=True, help="SLayer datasource name to file the imported models under") + import_cube_parser.add_argument("--report", default=None, help="Path for the JSON conversion report (default: /cube_import_report.json)") + import_cube_parser.add_argument("--include-hidden", action="store_true", help="Also print hidden (public: false) models in the summary") + import_cube_parser.add_argument( + "--ignore-required-meta", action="store_true", + help="Emit every FILTER_PARAMS pushdown as an optional filter (literal " + "Cube semantics), ignoring a member's meta.required marker. By " + "default a truthy meta.required makes that filter required " + "(raise-on-missing).") + _add_storage_arg(import_cube_parser) + # ── import-osi ──────────────────────────────────────────────────── import_osi_parser = subparsers.add_parser( "import-osi", @@ -819,6 +842,8 @@ def main(): # NOSONAR(S3776) — linear top-level CLI command dispatch (one eli _run_recommend_root_model(args) elif args.command == "import-dbt": _run_import_dbt(args) + elif args.command == "import-cube": + _run_import_cube(args) elif args.command == "import-osi": _run_import_osi(args) elif args.command == "models": @@ -1533,6 +1558,71 @@ def _run_import_dbt(args): ) +def _run_import_cube(args): + from slayer.cube.converter import CubeToSlayerConverter + from slayer.cube.parser import parse_cube_project + from slayer.cube.report import CubeConversionIssue, CubeIssueCategory + + storage = _resolve_storage(args) + project, parse_issues = parse_cube_project(args.cube_project_path) + + if not project.cubes and not project.views: + print(f"No cubes or views found in {args.cube_project_path}") + sys.exit(1) + + result = CubeToSlayerConverter( + project=project, data_source=args.datasource, parse_issues=parse_issues, + honor_required_meta=not args.ignore_required_meta, + ).convert() + saved = 0 + for model in result.models: + try: + run_sync(storage.save_model(model)) + saved += 1 + except Exception as exc: # noqa: BLE001 — one bad model shouldn't abort the import + result.report.add(CubeConversionIssue( + category=CubeIssueCategory.SAVE_FAILED, severity="error", + cube=model.name, + message=f"Failed to save model '{model.name}': {exc}")) + + _print_cube_import_summary(result, include_hidden=args.include_hidden) + report_path = _write_cube_report(result, args) + print( + f"\nDone: {saved} of {result.report.model_count} models saved " + f"({result.report.hidden_count} hidden, {result.report.view_count} views), " + f"{len(result.report.issues)} report issues. Report: {report_path}" + ) + + +def _print_cube_import_summary(result, *, include_hidden): + for model in result.models: + if model.hidden and not include_hidden: + continue + suffix = " [hidden]" if model.hidden else "" + kind = " (view)" if (model.meta or {}).get("cube_kind") == "view" else "" + print( + f"Imported model: {model.name}{kind}{suffix} " + f"({len(model.columns)} columns, {len(model.measures)} measures)" + ) + for severity in ("error", "warning", "info"): + for issue in result.report.by_severity(severity): + print(f" {severity.upper()} [{issue.category.value}/{issue.context}]: {issue.message}") + + +def _write_cube_report(result, args) -> str: + import os + + storage_base = args.storage or args.models_dir or _STORAGE_DEFAULT + if storage_base.endswith(".db"): + storage_base = os.path.dirname(storage_base) or "." + report_path = args.report or os.path.join(storage_base, "cube_import_report.json") + # Report path is an intended, user-specified CLI output location. + os.makedirs(os.path.dirname(report_path) or ".", exist_ok=True) # NOSONAR(S8707) + with open(report_path, "w", encoding="utf-8") as fh: # NOSONAR(S8707) + fh.write(result.report.model_dump_json(indent=2)) + return report_path + + def _run_import_osi(args): from slayer.osi.converter import OsiConversionError, OsiToSlayerConverter from slayer.osi.parser import parse_osi_path diff --git a/slayer/core/query.py b/slayer/core/query.py index f8b7a0da..2ddc4eba 100644 --- a/slayer/core/query.py +++ b/slayer/core/query.py @@ -12,10 +12,17 @@ import re from typing import Annotated, Any, Literal -from pydantic import BaseModel, BeforeValidator, ConfigDict, field_validator, model_validator +from pydantic import ( + BaseModel, + BeforeValidator, + ConfigDict, + Field, + field_validator, + model_validator, +) from slayer.core.enums import TimeGranularity -from slayer.core.models import ModelMeasure +from slayer.core.models import ModelMeasure, SlayerModel from slayer.sql.window_detect import WINDOW_IN_FILTER_ERROR, has_window_function from slayer.storage.migrations import migrate as _migrate_schema @@ -39,76 +46,263 @@ def _validate_query_filter_string(formula: str) -> None: raise ValueError(f"Filter '{formula}' {WINDOW_IN_FILTER_ERROR}") -def _escape_string_value(value: str, escape: Literal["sql", "python"]) -> str: +# C0 control characters (U+0000–U+001F) → Python string-literal escapes, used +# by the ``"python"`` regime (DEV-1727). A raw newline / carriage return / NUL +# inside a single-quoted literal makes ``ast.parse`` raise, so every C0 char is +# encoded: ``\t``/``\n``/``\r`` as their named escape, the rest as ``\xNN``. +# Encoding the whole C0 range (not just the three ast-breakers) keeps the +# substituted filter single-line and printable, at zero behavioural cost — +# ``ast.parse`` recovers the identical value either way. +_C0_NAMED_ESCAPES = {"\t": "\\t", "\n": "\\n", "\r": "\\r"} +_C0_ESCAPE_MAP = { + chr(codepoint): _C0_NAMED_ESCAPES.get(chr(codepoint), f"\\x{codepoint:02x}") + for codepoint in range(0x20) +} +_C0_RE = re.compile(r"[\x00-\x1f]") + + +def _escape_string_value( + value: str, escape: Literal["sql", "python"], *, backslash_escapes: bool +) -> str: """Escape a string variable value for the target expression layer. The value is inserted, unquoted, into a quoted literal the author already wrote (``status = '{v}'``); escaping keeps it from breaking out of that - literal. Two layers, two escaping regimes (DEV-1625): + literal. Two layers, two escaping regimes (DEV-1625, hardened in DEV-1727): + + - ``"sql"`` — Mode-A raw-SQL surfaces are parsed by sqlglot. The regime is + **dialect-aware** (``backslash_escapes``): + + * ``False`` (standard dialects — SQLite/Postgres/DuckDB/T-SQL/Trino/ + Presto/Oracle): a backslash is an ordinary literal char, so only the + single quote is doubled (``'`` → ``''``). + * ``True`` (backslash dialects — MySQL/ClickHouse/Snowflake/Redshift/ + BigQuery/Databricks/Spark): a backslash escapes the next char, so it is + doubled FIRST (``\\`` → ``\\\\``) and the single quote is + backslash-escaped (``'`` → ``\\'``). The double quote is left untouched + — inside a single-quoted literal it is an ordinary char on every + dialect, and ``\\"`` is not a recognised escape on 6 of the 7 backslash + dialects (only MySQL), so escaping it would corrupt the value. - - ``"sql"`` — Mode-A raw-SQL surfaces are parsed by sqlglot. A single quote - is doubled (``'`` → ``''``); backslash is an ordinary character and is - left untouched. - ``"python"`` — Mode-B query filters are parsed by SLayer's Python-AST formula parser, where SQL quote-doubling would be read as adjacent-literal concatenation (``'O''Brien'`` → ``'OBrien'``). So backslash is doubled - FIRST, then both quote styles are backslash-escaped, matching Python - string-literal rules so ``ast.parse`` recovers the original value. + FIRST, then both quote styles are backslash-escaped, then every C0 + control char (U+0000–U+001F) is encoded, matching Python string-literal + rules so ``ast.parse`` recovers the original value. This matters because a + raw newline/CR/NUL in a single-quoted Python literal is a ``SyntaxError`` + (or "null bytes" error), so leaving control chars unescaped would make + ``ast.parse`` reject an otherwise valid value. SQL literals permit raw + newlines, so the ``"sql"`` branch leaves them alone. + ``backslash_escapes`` is ignored here. """ if escape == "sql": + if backslash_escapes: + # Double the backslash before escaping the quote (order matters). + return value.replace("\\", "\\\\").replace("'", "\\'") return value.replace("'", "''") - # python: order matters — double the backslash before escaping quotes. - return value.replace("\\", "\\\\").replace("'", "\\'").replace('"', '\\"') + # python: order matters — double the backslash before escaping quotes, then + # encode C0 control chars so ast.parse recovers a single-line literal. + escaped = value.replace("\\", "\\\\").replace("'", "\\'").replace('"', '\\"') + return _C0_RE.sub(lambda m: _C0_ESCAPE_MAP[m.group(0)], escaped) + + +def _render_list_value( + name: str, + value: "list | tuple", + escape: Literal["sql", "python"], + *, + backslash_escapes: bool, +) -> str: + """Render a ``list``/``tuple`` variable value into an ``IN``-list body + (DEV-1730 multi-value pushdown). + + The intended template shape is ``col IN ({var})`` — the author writes the + parentheses; this renders the comma-separated body only. Unlike a scalar + string (where the author writes the surrounding quotes), each string element + is **auto-quoted** here: a single placeholder can't carry per-element quotes, + so quoting has to happen at render time. Elements are escaped per the target + layer via :func:`_escape_string_value`, so DEV-1727's escaping composes. + + - ``str`` element → auto-quoted + escaped (``O'Brien`` → ``'O''Brien'`` in + sql mode; ``'O\\'Brien'`` in python mode). + - ``int``/``float``/``bool`` element → bare via ``str()``; a non-finite float + element raises (it can never render a valid literal). + - ``escape="python"`` appends a **trailing comma** so the Mode-B Python-AST + parser always reads a tuple — ``x in ('A',)`` (1-tuple membership), never + ``x in ('A')`` (which parses as ``str`` containment). + - An **empty** list/tuple raises: ``IN ()`` is invalid SQL, and "no filter" + semantics belong to a sentinel default (see DEV-1730). + - ``None``, nested list/tuple, dict, or any other element type raises, + naming the variable. + """ + if len(value) == 0: + raise ValueError( + f"Variable '{name}' cannot be an empty list; 'IN ()' is invalid SQL. " + f"For 'no filter' semantics, use a sentinel default (see DEV-1730)." + ) + rendered: list[str] = [] + for element in value: + if isinstance(element, str): + rendered.append( + "'" + + _escape_string_value( + value=element, escape=escape, backslash_escapes=backslash_escapes + ) + + "'" + ) + # bool is an int subclass and is accepted (renders True/False). + elif isinstance(element, (int, float)): + if isinstance(element, float) and not math.isfinite(element): + raise ValueError( + f"Variable '{name}' list element must be finite, got {element!r}" + ) + rendered.append(str(element)) + else: + raise ValueError( + f"Variable '{name}' list element must be a string, number, or bool, " + f"got {type(element).__name__}" + ) + joined = ", ".join(rendered) + # Mode-B (python) needs the trailing comma to force tuple parsing; Mode-A + # (sql) must NOT have it (``IN (1, 2,)`` is a syntax error in most dialects). + return f"{joined}," if escape == "python" else joined def _render_variable_value( - name: str, value: Any, escape: Literal["sql", "python"] + name: str, + value: Any, + escape: Literal["sql", "python"], + *, + backslash_escapes: bool, ) -> str: """Render a single resolved variable value into substitution text. Strings are escaped for the target layer (see :func:`_escape_string_value`); numbers (including ``bool``) pass through via ``str()`` but non-finite floats - raise (they can never render a valid literal); anything else raises. + raise (they can never render a valid literal). A ``list``/``tuple`` renders + an ``IN``-list body (see :func:`_render_list_value`); anything else raises. """ + # list/tuple first: an IN-list body (DEV-1730). Checked before str so the + # scalar path only ever sees a single value. + if isinstance(value, (list, tuple)): + return _render_list_value( + name=name, value=value, escape=escape, backslash_escapes=backslash_escapes + ) if isinstance(value, str): - return _escape_string_value(value=value, escape=escape) + return _escape_string_value( + value=value, escape=escape, backslash_escapes=backslash_escapes + ) # bool is an int subclass and is accepted (renders True/False). if isinstance(value, (int, float)): if isinstance(value, float) and not math.isfinite(value): raise ValueError(f"Variable '{name}' must be finite, got {value!r}") return str(value) raise ValueError( - f"Variable '{name}' must be a string or number, got {type(value).__name__}" + f"Variable '{name}' must be a string, number, or list/tuple, " + f"got {type(value).__name__}" ) -def substitute_variables( - filter_str: str, variables: dict[str, Any], *, escape: Literal["sql", "python"] -) -> str: - """Substitute {variable} placeholders in a filter or raw-SQL string. +_BLOCK_OPEN = "{?" +_BLOCK_CLOSE = "?}" - - {var_name} is replaced with the variable's value (str or number). - - {{ and }} are escaped to literal { and }. - - Variable names must be alphanumeric + underscore. - - Raises ValueError for undefined variables or invalid variable names. - ``escape`` (required, keyword-only) selects the escaping regime for string - values by expression layer — ``"sql"`` for Mode-A raw-SQL surfaces, - ``"python"`` for Mode-B query filters. See :func:`_escape_string_value`. - Numbers (including ``bool``) pass through via ``str()``; non-finite floats - (``nan``/``inf``) raise, since they can never render a valid literal. +def _find_block_end(text: str, start: int) -> int: + """Scan from ``start`` (just past a ``{?``) to the matching ``?}``. - Example: - substitute_variables("status = '{status_val}'", {"status_val": "active"}, escape="sql") - → "status = 'active'" + Returns the index of the closing ``?}``. Raises on a nested ``{?`` (blocks + do not nest) or if no close is found before end-of-string. ``{{``/``}}`` + escapes are skipped so they can never masquerade as block delimiters. + """ + i, n = start, len(text) + while i < n: + two = text[i:i + 2] + if two in ("{{", "}}"): + i += 2 + continue + if two == _BLOCK_OPEN: + raise ValueError( + f"Nested optional block '{{?' is not allowed in: {text!r}" + ) + if two == _BLOCK_CLOSE: + return i + i += 1 + raise ValueError( + f"Unterminated optional block (missing '?}}') in: {text!r}" + ) - substitute_variables("amount > {min_amount}", {"min_amount": 100}, escape="sql") - → "amount > 100" + +def _split_blocks(text: str) -> list[tuple[str, str]]: + """Split ``text`` into ``("text", s)`` / ``("block", inner)`` parts. + + Top-level ``{? ... ?}`` spans become ``block`` parts (inner text only); + everything else is ``text``. ``{{``/``}}`` escapes are preserved verbatim in + ``text`` parts (the downstream var pass renders them). Raises on a stray + ``?}`` (a close with no open). Blocks never nest (enforced here). """ - if escape not in ("sql", "python"): + parts: list[tuple[str, str]] = [] + buf: list[str] = [] + i, n = 0, len(text) + while i < n: + two = text[i:i + 2] + if two in ("{{", "}}"): + buf.append(two) + i += 2 + continue + if two == _BLOCK_OPEN: + parts.append(("text", "".join(buf))) + buf = [] + end = _find_block_end(text, i + 2) + parts.append(("block", text[i + 2:end])) + i = end + 2 + continue + if two == _BLOCK_CLOSE: + raise ValueError( + f"Unexpected '?}}' (optional-block close without open) in: {text!r}" + ) + buf.append(text[i]) + i += 1 + parts.append(("text", "".join(buf))) + return parts + + +def _block_var_names(inner: str, whole: str) -> list[str]: + """Return the valid ``{var}`` names inside a block's ``inner`` text. + + Raises if the block carries an invalid ``{...}`` name, or if it contains no + ``{var}`` at all (an optional block with nothing to key on is a mistake — + it would render identically whether or not any variable is supplied). + """ + names: list[str] = [] + for match in _VAR_PATTERN.finditer(inner): + if match.group(0) in ("{{", "}}"): + continue + if match.group(1) is not None: + names.append(match.group(1)) + else: + raise ValueError( + f"Invalid variable name '{match.group(2)}' in optional block " + f"of: {whole!r}." + ) + if not names: raise ValueError( - f"Invalid escape mode {escape!r}; expected 'sql' or 'python'." + f"Optional block '{{? ... ?}}' must contain at least one " + f"{{variable}} in: {whole!r}." ) + return names + + +def _contains_block_delimiter(text: str) -> bool: + return _BLOCK_OPEN in text or _BLOCK_CLOSE in text + + +def _make_var_replacer( + filter_str: str, variables: dict, escape: str, backslash_escapes: bool +): + """Build the ``re.sub`` replacement callable for ``{var}`` / ``{{`` / ``}}`` + tokens, closed over the resolved escaping regime. Extracted from + :func:`substitute_variables` to keep its cognitive complexity in check.""" def _replace(match: re.Match) -> str: full = match.group(0) @@ -125,7 +319,10 @@ def _replace(match: re.Match) -> str: f"Available variables: {sorted(variables.keys())}" ) return _render_variable_value( - name=valid_name, value=variables[valid_name], escape=escape + name=valid_name, + value=variables[valid_name], + escape=escape, + backslash_escapes=backslash_escapes, ) # Group 2: invalid variable name (matched {something} but name was invalid) bad_name = match.group(2) @@ -134,7 +331,104 @@ def _replace(match: re.Match) -> str: f"Variable names must contain only letters, digits, and underscores." ) - return _VAR_PATTERN.sub(_replace, filter_str) + return _replace + + +def substitute_variables( + filter_str: str, + variables: dict[str, Any], + *, + escape: Literal["sql", "python"], + backslash_escapes: bool | None = None, +) -> str: + """Substitute {variable} placeholders in a filter or raw-SQL string. + + - {var_name} is replaced with the variable's value (str, number, or list). + - {{ and }} are escaped to literal { and }. + - Variable names must be alphanumeric + underscore. + - Raises ValueError for undefined variables or invalid variable names. + + ``escape`` (required, keyword-only) selects the escaping regime for string + values by expression layer — ``"sql"`` for Mode-A raw-SQL surfaces, + ``"python"`` for Mode-B query filters. See :func:`_escape_string_value`. + Numbers (including ``bool``) pass through via ``str()``; non-finite floats + (``nan``/``inf``) raise, since they can never render a valid literal. + + ``backslash_escapes`` (keyword-only) is the **dialect-aware** signal for the + ``"sql"`` regime (DEV-1727) and is **fail-closed**: with ``escape="sql"`` it + is REQUIRED (``None`` raises), so a caller that renders raw SQL can never + silently under-escape on a backslash dialect. Derive it from + ``SqlDialect.backslash_escapes_strings``. It is ignored for + ``escape="python"`` (Mode-B escaping is dialect-independent). + + A ``list``/``tuple`` value renders an ``IN``-list body (DEV-1730). The + template writes the parentheses (``col IN ({var})``) and each string element + is **auto-quoted** — the opposite of the scalar-string convention where the + author writes the quotes (``status = '{v}'``). See :func:`_render_list_value`. + + Example: + substitute_variables("status = '{status_val}'", {"status_val": "active"}, escape="sql", backslash_escapes=False) + → "status = 'active'" + + substitute_variables("amount > {min_amount}", {"min_amount": 100}, escape="sql", backslash_escapes=False) + → "amount > 100" + + substitute_variables("region IN ({regions})", {"regions": ["US", "CA"]}, escape="sql", backslash_escapes=False) + → "region IN ('US', 'CA')" + """ + if escape not in ("sql", "python"): + raise ValueError( + f"Invalid escape mode {escape!r}; expected 'sql' or 'python'." + ) + if escape == "sql" and backslash_escapes is None: + raise ValueError( + "escape='sql' requires backslash_escapes to be specified: True on " + "backslash-escaping dialects (MySQL, ClickHouse, Snowflake, " + "Redshift, BigQuery, Databricks, Spark), False on standard dialects " + "(SQLite, Postgres, DuckDB, ...). Derive it from " + "SqlDialect.backslash_escapes_strings." + ) + # Normalise to a concrete bool for the value renderers; python mode ignores + # the signal (Mode-B escaping is dialect-independent). + effective_backslash_escapes = bool(backslash_escapes) if escape == "sql" else False + _replace = _make_var_replacer( + filter_str, variables, escape, effective_backslash_escapes + ) + + # Optional blocks {? ... ?} are a Mode-A-only construct (DEV-1730). The + # Mode-B Python-AST filter layer rejects them outright. + if escape == "python": + if _contains_block_delimiter(filter_str): + raise ValueError( + f"Optional blocks '{{? ... ?}}' are not supported in Mode-B " + f"(python) filters: {filter_str!r}." + ) + return _VAR_PATTERN.sub(_replace, filter_str) + + # Fast path: no block delimiters -> the original single-pass regex sub. + if not _contains_block_delimiter(filter_str): + return _VAR_PATTERN.sub(_replace, filter_str) + + return _render_block_segments(filter_str, variables, _replace) + + +def _render_block_segments(filter_str: str, variables: dict, replace_fn) -> str: + """Render a Mode-A string that contains at least one ``{? ... ?}`` block. + + Plain text segments substitute normally; a block renders parenthesised when + every inner ``{var}`` is supplied, else collapses to the neutral ``(1=1)``. + """ + out: list[str] = [] + for kind, segment in _split_blocks(filter_str): + if kind == "text": + out.append(_VAR_PATTERN.sub(replace_fn, segment)) + continue + names = _block_var_names(segment, filter_str) + if all(name in variables for name in names): + out.append("(" + _VAR_PATTERN.sub(replace_fn, segment).strip() + ")") + else: + out.append("(1=1)") + return "".join(out) def extract_placeholder_names(query: "SlayerQuery") -> set: @@ -153,6 +447,217 @@ def extract_placeholder_names(query: "SlayerQuery") -> set: return found +def _probe_replace(match: re.Match) -> str: + full = match.group(0) + if full == "{{": + return "{" + if full == "}}": + return "}" + return "0" # any {var} (valid or not) -> a syntactically safe literal + + +def render_probe_text(text: str) -> str: + """Render a Mode-A surface for a syntax-only sqlglot parse (DEV-1730). + + Optional blocks collapse to ``(1=1)`` (their absent-value form) and every + remaining ``{var}`` becomes the literal ``0`` — enough for sqlglot to parse + structure without any caller variables. Shared by every converter validation + path so import-time validation matches execution-time rendering. + """ + out: list[str] = [] + for kind, segment in _split_blocks(text): + if kind == "block": + out.append("(1=1)") + else: + out.append(_VAR_PATTERN.sub(_probe_replace, segment)) + return "".join(out) + + +class ModelVariables(BaseModel): + """Structural classification of a model's Mode-A ``{var}`` placeholders. + + ``required`` vars have no default and are not inside an optional block, so a + query that omits them raises. ``optional`` vars either sit inside a ``{? ?}`` + block (collapse to ``(1=1)`` when absent) or carry a ``query_variables`` + default. Derived on demand from the four Mode-A surfaces — nothing is + persisted, so there is no schema-version impact (DEV-1730). + """ + + required: list[str] = Field(default_factory=list) + optional: list[str] = Field(default_factory=list) + + +def extract_variable_refs(text: str) -> tuple[set[str], set[str]]: + """Return ``(bare_names, blocked_names)`` referenced in a Mode-A ``text``. + + ``bare_names`` appear outside any ``{? ?}`` block; ``blocked_names`` appear + inside one. A name may land in both sets (used bare in one place and blocked + in another) — the caller resolves the precedence. + """ + bare: set[str] = set() + blocked: set[str] = set() + try: + parts = _split_blocks(text) + except ValueError: + # Malformed block delimiters (stray '?}' / unterminated '{?'): classify + # structurally as block-free rather than breaking read-only inspection + # (extract_model_variables runs unguarded from the inspect skeleton). + # Execution still raises through substitute_variables. + parts = [("text", text)] + for kind, segment in parts: + target = bare if kind == "text" else blocked + for match in _VAR_PATTERN.finditer(segment): + if match.group(0) in ("{{", "}}"): + continue + if match.group(1): + target.add(match.group(1)) + return bare, blocked + + +def extract_model_variables(model: SlayerModel) -> ModelVariables: + """Classify a model's Mode-A ``{var}`` placeholders as required / optional. + + Walks the four Mode-A surfaces — ``SlayerModel.sql``, ``SlayerModel.filters``, + ``Column.sql``, ``Column.filter`` — the same surfaces DEV-1625 substitutes. + A bare occurrence with no ``query_variables`` default is required; everything + else (inside a block, or defaulted) is optional. A bare-without-default + occurrence anywhere wins, so a var used both bare and blocked is required. + """ + surfaces: list[str] = [] + if model.sql: + surfaces.append(model.sql) + surfaces.extend(f for f in (model.filters or []) if f) + for col in model.columns: + if col.sql: + surfaces.append(col.sql) + if col.filter: + surfaces.append(col.filter) + + bare: set[str] = set() + blocked: set[str] = set() + for surface in surfaces: + s_bare, s_blocked = extract_variable_refs(surface) + bare |= s_bare + blocked |= s_blocked + + defaults = set(model.query_variables or {}) + required = {name for name in bare if name not in defaults} + optional = (bare | blocked) - required + return ModelVariables( + required=sorted(required), optional=sorted(optional) + ) + + +def declared_variable_specs(model: SlayerModel) -> dict[str, dict]: + """The model's DECLARED Mode-A variable bag (``name -> spec``), or ``{}``. + + An importer that generates parameterized model SQL records what it emitted + under ``meta.cube_variables``, so the engine can tell a generated template + apart from hand-written SQL that merely contains braces. ``meta`` is + user-extensible, so every layer is shape-checked and a malformed bag + degrades to "nothing declared" rather than raising during a query. + + An entry counts as a declaration only if it carries a NON-EMPTY string + ``member`` — the shape every importer writes (a member name is always a + parsed identifier). That makes the bag SELF-IDENTIFYING, so a hand-written + ``meta`` that happens to reuse the ``cube_variables`` key for something else + (``{"cube_variables": {"note": {}}}``) is not mistaken for generated SQL. + The distinction matters: :func:`declares_variables` disables the + zero-variable brace-literal fast path, so a false positive would make a + previously-working model with raw braces start raising. + """ + declared = (model.meta or {}).get("cube_variables") + if not isinstance(declared, dict): + return {} + return { + name: spec + for name, spec in declared.items() + if isinstance(name, str) and isinstance(spec, dict) and _is_member_name(spec) + } + + +def _is_member_name(spec: dict) -> bool: + """True if ``spec`` carries the non-empty string ``member`` that marks it as + a real importer-written variable declaration.""" + member = spec.get("member") + return isinstance(member, str) and bool(member) + + +def declares_variables(model: SlayerModel) -> bool: + """True if the model declares its Mode-A variables (importer-generated SQL). + + Such a model is unambiguously parameterized, so the DEV-1625 brace-literal + protection — which leaves surfaces untouched on a zero-variable call so raw + braces like a Postgres array ``'{1,2,3}'`` survive — must NOT apply: leaving + a declared ``{var}`` unrendered emits it into SQL instead of raising the + documented missing-variable error. Hand-written models declare nothing and + keep the protection. + """ + return bool(declared_variable_specs(model)) + + +def list_valued_variable_names(model: SlayerModel) -> set[str]: + """Names of the model's Mode-A variables DECLARED to fill an ``IN``-list. + + The generic ``{var}`` contract puts quoting on the template author — a + scalar string renders unquoted so ``{var}`` also works in numeric and + fragment positions (``order_total >= {floor}``, ``{d}::TIMESTAMP``). That + reasoning needs an author who can see the position, and it breaks down for + a MACHINE-generated surface: the Cube importer emits the fixed template + ``col IN ({var})``, so the caller can never write the quotes, and a scalar + string would render ``IN (US)`` — a column reference that sqlglot parses + happily and the database rejects (or, worse, silently resolves). + + An importer therefore declares such a variable with ``list_valued: true`` in + ``meta.cube_variables``; :func:`coerce_declared_list_variables` acts on it. + Only that neutral flag is read here — not Cube's ``kind`` taxonomy — so a + future front-end emitting a different list-shaped template opts in the same + way. Returns an empty set for a hand-written model (nothing declared), which + keeps the generic scalar convention untouched. + + The flag must be exactly ``True``: ``meta`` is user-extensible, and matching + on truthiness would let a stray ``1`` or the string ``"false"`` silently + switch a variable's substitution semantics. + """ + return { + name + for name, spec in declared_variable_specs(model).items() + if spec.get("list_valued") is True + } + + +def coerce_declared_list_variables( + variables: dict[str, Any], *, list_valued: set[str] +) -> dict[str, Any]: + """Wrap a scalar supplied for a declared list-valued variable in a + one-element list, so it renders ``IN ('US')`` rather than ``IN (US)``. + + In ``IN (...)`` position a scalar and a one-element list are semantically + identical, so this is a normalisation, not a guess — there is no competing + reading of ``{"regions": "US"}`` against ``region IN ({regions})``. + + Only ``str``/``int``/``float``/``bool`` are wrapped. A ``list``/``tuple`` + passes through unchanged (including the empty list, which keeps raising — + ``IN ()`` is invalid SQL and "no filter" belongs to an optional block or a + sentinel default). Any other type is left alone so + :func:`_render_variable_value` still raises its own naming error. Returns + the input dict unchanged when nothing needs wrapping; never mutates it. + """ + if not list_valued: + return variables + coerced: dict[str, Any] | None = None + for name in list_valued: + if name not in variables: + continue + value = variables[name] + # bool is an int subclass and is accepted, matching the list renderer. + if isinstance(value, (str, int, float)): + if coerced is None: + coerced = dict(variables) + coerced[name] = [value] + return variables if coerced is None else coerced + + class ColumnRef(BaseModel): """Reference to a dimension by name. diff --git a/slayer/cube/__init__.py b/slayer/cube/__init__.py new file mode 100644 index 00000000..f5adfeaa --- /dev/null +++ b/slayer/cube/__init__.py @@ -0,0 +1,3 @@ +"""Cube (Cube.js / Cube.dev) data-model ingestion — parse Cube YAML and +convert to SLayer models. Mirrors ``slayer/dbt/``. See DEV-1608. +""" diff --git a/slayer/cube/converter.py b/slayer/cube/converter.py new file mode 100644 index 00000000..bbea10f2 --- /dev/null +++ b/slayer/cube/converter.py @@ -0,0 +1,945 @@ +"""Convert a parsed CubeProject into SLayer models. + +Mirrors ``slayer/dbt/converter.py``. For each cube → one table-owning model; +for each view → one facade model. Everything that can't map cleanly is recorded +on the ``CubeConversionReport``. See DEV-1608. +""" + +import logging +import re + +import sqlglot +from pydantic import BaseModel + +from slayer.core.enums import DataType, JoinType +from slayer.core.format import NumberFormat, NumberFormatType +from slayer.core.formula import ALL_TRANSFORMS, parse_formula +from slayer.core.models import Column, ModelJoin, ModelMeasure, SlayerModel +from slayer.core.query import render_probe_text +from slayer.cube.extends import flatten_cube_extends, flatten_view_extends +from slayer.cube.filter_params import ( + apply_filter_params, + parse_string_filter_params, +) +from slayer.cube.models import ( + CubeCube, + CubeDimension, + CubeFilterParamRef, + CubeMeasure, + CubeView, +) +from slayer.cube.refs import is_bare_identifier, parse_join_on, translate_cube_refs +from slayer.cube.report import ( + CubeConversionIssue, + CubeConversionReport, + CubeConversionResult, + CubeIssueCategory, +) + +logger = logging.getLogger(__name__) + +_AGG_TYPES = {"sum", "avg", "min", "max", "count", "count_distinct", "count_distinct_approx"} +_CALC_TYPES = {"number", "string", "time", "boolean"} +_DEFERRED_MEASURE_TYPES = {"number_agg"} +_DIM_TYPE_MAP = { + "string": DataType.TEXT, "number": DataType.DOUBLE, + "boolean": DataType.BOOLEAN, "time": DataType.TIMESTAMP, +} +_FORMAT_MAP = { + "percent": NumberFormatType.PERCENT, "currency": NumberFormatType.CURRENCY, + "number": NumberFormatType.FLOAT, +} +_DURATION_UNITS = {"day": "d", "month": "m", "week": "w", "year": "y", + "hour": "h", "minute": "min", "second": "s"} +_STAR_COUNT = "*:count" +_CUBE_INFRA_FIELDS = ("refresh_key", "calendar", "hierarchies", "access_policy", "sql_alias") + + +def _sql_str_literal(value) -> str: + """Render a value as a single-quoted SQL string literal, escaping quotes.""" + return "'" + str(value).replace("'", "''") + "'" + + +def _case_when_predicates(dim) -> list[dict]: + """Return the ``when`` predicate dicts of a CASE-WHEN dimension (each carries + a ``sql``), or ``[]`` for a non-case dimension. These are Mode-A surfaces too, + so they can host FILTER_PARAMS (DEV-1730).""" + if not dim.case: + return [] + return [w for w in dim.case.get("when", []) if isinstance(w, dict)] + + +class _MeasureInfo(BaseModel): + """How a converted cube measure can be re-aggregated by a view facade.""" + kind: str # "agg" | "calc" | "star_count" + underlying_col: str | None = None + agg: str | None = None + emitted_name: str | None = None # the emitted ModelMeasure name (for pruning) + + +class _Names: + """Shared column+measure namespace allocator.""" + + def __init__(self) -> None: + self.used: set[str] = set() + + def take(self, base: str, suffix: str = "_col") -> str: + name = base + while name in self.used: + name = name + suffix + self.used.add(name) + return name + + def reserve(self, name: str) -> None: + self.used.add(name) + + +def _map_format(fmt, report: CubeConversionReport, *, cube: str, member: str) -> NumberFormat | None: + if fmt is None: + return None + ftype = fmt.get("type") if isinstance(fmt, dict) else fmt + nf_type = _FORMAT_MAP.get(ftype) if isinstance(ftype, str) else None + if nf_type is None: + report.add(CubeConversionIssue( + category=CubeIssueCategory.UNSUPPORTED_FORMAT, severity="info", + cube=cube, member=member, message=f"Unsupported format '{fmt}'; dropped.", + )) + return None + kwargs = {"type": nf_type} + if nf_type == NumberFormatType.CURRENCY and isinstance(fmt, dict) and fmt.get("currency_symbol"): + kwargs["symbol"] = fmt["currency_symbol"] # symbol ONLY for currency (Codex #8) + try: + return NumberFormat(**kwargs) + except Exception: # noqa: BLE001 + report.add(CubeConversionIssue( + category=CubeIssueCategory.UNSUPPORTED_FORMAT, severity="info", + cube=cube, member=member, message=f"Invalid format '{fmt}'; dropped.", + )) + return None + + +def _window_from_rolling(rolling, report, *, cube, member) -> str | None: + trailing = rolling.get("trailing") + if rolling.get("leading") or rolling.get("offset") or trailing in (None, "unbounded"): + report.add(CubeConversionIssue( + category=CubeIssueCategory.UNSUPPORTED_ROLLING_WINDOW, severity="warning", + cube=cube, member=member, + message="Only finite trailing rolling_window maps; fell back to plain aggregation.", + )) + return None + m = re.match(r"(\d+)\s+(\w+)", str(trailing)) + unit = _DURATION_UNITS.get(m.group(2).rstrip("s")) if m else None + if not m or unit is None: + report.add(CubeConversionIssue( + category=CubeIssueCategory.UNSUPPORTED_ROLLING_WINDOW, severity="warning", + cube=cube, member=member, message=f"Unparseable rolling_window '{trailing}'.", + )) + return None + return f"{m.group(1)}{unit}" + + +class CubeToSlayerConverter: + """Convert a CubeProject into SLayer models + a structured report.""" + + def __init__( + self, project, data_source: str, parse_issues=None, + *, honor_required_meta: bool = True, + ) -> None: + self.project = project + self.data_source = data_source + self.parse_issues = parse_issues or [] + # DEV-1730: honor a member's truthy ``meta.required`` by emitting its + # FILTER_PARAMS pushdown as a required (raise-on-missing) variable rather + # than an optional block. ``--ignore-required-meta`` flips this off. + self.honor_required_meta = honor_required_meta + self._cubes: dict[str, CubeCube] = {} + self._models: dict[str, SlayerModel] = {} + # model name → {measure name → _MeasureInfo} + self._measure_info: dict[str, dict[str, _MeasureInfo]] = {} + # Per-cube FILTER_PARAMS state (set in _convert_cube, consumed by the + # Mode-A translate helpers). + self._active_refs: list[CubeFilterParamRef] = [] + self._active_required: set[str] = set() + + # ── pipeline ─────────────────────────────────────────────────────────── + + def convert(self) -> CubeConversionResult: + report = CubeConversionReport(issues=list(self.parse_issues)) + + cubes, cube_issues = flatten_cube_extends(self.project.cubes) + report.issues.extend(cube_issues) + views, view_issues = flatten_view_extends(self.project.views) + report.issues.extend(view_issues) + + self._cubes = {c.name: c for c in cubes} + models: list[SlayerModel] = [] + for cube in cubes: + model = self._convert_cube(cube, report) + if model is not None: + models.append(model) + self._models[model.name] = model + + for view in views: + model = self._convert_view(view, report) + if model is not None: + models.append(model) + self._models[model.name] = model + + report.model_count = len(models) + report.hidden_count = sum(1 for m in models if m.hidden) + report.view_count = sum( + 1 for m in models if (m.meta or {}).get("cube_kind") == "view") + return CubeConversionResult(models=models, report=report) + + # ── cube → model ─────────────────────────────────────────────────────── + + def _convert_cube(self, cube: CubeCube, report: CubeConversionReport) -> SlayerModel | None: + # DEV-1730: gather + validate FILTER_PARAMS refs BEFORE any translation, + # so a bad ref drops the cube cleanly with no half-built variable entries. + setup = self._setup_filter_params(cube, report) + if setup is None: + return None + cube, refs = setup + + source = self._cube_source(cube, report) + if source is None: + return None + + meta, unmapped = self._build_meta_and_unmapped(cube, refs, report) + + names = _Names() + columns: list[Column] = [] + measures: list[ModelMeasure] = [] + info: dict[str, _MeasureInfo] = {} + + for dim in cube.dimensions: + self._convert_dimension(cube, dim, columns, names, unmapped, report) + dedup: dict[tuple, str] = {} + for meas in cube.measures: + self._convert_measure(cube, meas, columns, measures, names, dedup, info, report) + for seg in cube.segments: + self._convert_segment(cube, seg, columns, names, report) + + joins = self._convert_joins(cube, report) + columns, measures = self._validate_offline(cube.name, columns, measures, report) + self._dedisambiguate_namespace(columns, measures, report, cube=cube.name) + # Keep _measure_info in sync with what actually survived validation, so + # view facades never re-export a measure the model no longer has. + surviving = {m.name for m in measures} + info = {k: v for k, v in info.items() if v.emitted_name in surviving} + + if unmapped: + meta["cube_unmapped"] = unmapped + + try: + model = SlayerModel( + name=cube.name, data_source=self.data_source, + hidden=not cube.public, description=cube.description, + meta=meta or None, columns=columns, measures=measures, joins=joins, + **source, + ) + # Illegal model name etc. → report, don't crash the run. + except Exception as exc: # noqa: BLE001 + report.add(CubeConversionIssue( + category=CubeIssueCategory.PARSE_ERROR, severity="error", + cube=cube.name, message=f"Could not build model: {exc}", + )) + return None + self._measure_info[cube.name] = info + self._report_filter_param_variables(cube, refs, report) + return model + + def _build_meta_and_unmapped( + self, cube: CubeCube, refs, report: CubeConversionReport + ) -> tuple[dict, dict]: + """Assemble the model ``meta`` (cube meta + title + FILTER_PARAMS + variables) and the ``unmapped`` infra bag, reporting each stashed field.""" + meta: dict = dict(cube.meta or {}) + cube_vars = self._build_cube_variables(cube, refs) + if cube_vars: + meta["cube_variables"] = cube_vars + if cube.title: + meta["cube_title"] = cube.title + unmapped: dict = {} + if cube.data_source: + unmapped["data_source"] = cube.data_source + if cube.pre_aggregations: + unmapped["pre_aggregations"] = cube.pre_aggregations + for field in _CUBE_INFRA_FIELDS: + val = getattr(cube, field, None) + if val is not None: + unmapped[field] = val + for key in unmapped: + report.add(CubeConversionIssue( + category=CubeIssueCategory.UNMAPPED_INFRA, severity="warning", + cube=cube.name, message=f"'{key}' has no SLayer equivalent; stashed in meta.", + raw=str(unmapped[key]), + )) + return meta, unmapped + + # ── FILTER_PARAMS (DEV-1730) ──────────────────────────────────────────── + + def _setup_filter_params( + self, cube: CubeCube, report: CubeConversionReport + ) -> tuple[CubeCube, list[CubeFilterParamRef]] | None: + """Prepare + validate FILTER_PARAMS for a cube, priming the per-cube + translate state. Returns ``(working_cube, refs)`` or ``None`` (a + validation error was reported and the cube must be dropped).""" + cube, refs = self._prepare_filter_params(cube) + fp_error = self._validate_filter_params(cube, refs) + if fp_error is not None: + report.add(fp_error) + return None + self._active_refs = refs + self._active_required = self._required_members(cube, refs) + return cube, refs + + def _prepare_filter_params( + self, cube: CubeCube + ) -> tuple[CubeCube, list[CubeFilterParamRef]]: + """Return ``(working_cube, refs)``. JS cubes arrive with structured refs + + sentinelised surfaces; YAML cubes carry raw ``{FILTER_PARAMS...}`` text + which is scanned here (on a deep copy) into refs + sentinels.""" + if cube.filter_params: + return cube, list(cube.filter_params) + if not self._cube_mentions_filter_params(cube): + return cube, [] + cube = cube.model_copy(deep=True) + refs: list[CubeFilterParamRef] = [] + + def scan(text: str | None) -> str | None: + nonlocal refs + if not text or "{FILTER_PARAMS." not in text: + return text + ext = parse_string_filter_params( + text, host_cube=cube.name, start_index=len(refs)) + refs.extend(ext.refs) + # Unsupported occurrences are left verbatim; _validate_filter_params + # sees the still-present {FILTER_PARAMS...} text and drops the cube. + return ext.text + + cube.sql = scan(cube.sql) + for dim in cube.dimensions: + dim.sql = scan(dim.sql) + for when in _case_when_predicates(dim): + when["sql"] = scan(when.get("sql")) + for meas in cube.measures: + meas.sql = scan(meas.sql) + for f in meas.filters: + f.sql = scan(f.sql) + for seg in cube.segments: + seg.sql = scan(seg.sql) + return cube, refs + + def _cube_mentions_filter_params(self, cube: CubeCube) -> bool: + surfaces = [cube.sql] + surfaces += [d.sql for d in cube.dimensions] + surfaces += [w.get("sql") for d in cube.dimensions for w in _case_when_predicates(d)] + surfaces += [m.sql for m in cube.measures] + surfaces += [f.sql for m in cube.measures for f in m.filters] + surfaces += [s.sql for s in cube.segments] + return any(s and "{FILTER_PARAMS." in s for s in surfaces) + + def _validate_filter_params( + self, cube: CubeCube, refs: list[CubeFilterParamRef] + ) -> CubeConversionIssue | None: + """Validate FILTER_PARAMS refs; return a drop-the-cube issue or None.""" + if self._cube_mentions_filter_params(cube): + return CubeConversionIssue( + category=CubeIssueCategory.FILTER_PARAMS_UNSUPPORTED, severity="error", + cube=cube.name, + message="Unsupported FILTER_PARAMS form (cross-cube, arrow-in-YAML, " + "or unparseable); cube dropped.") + valid_members = {d.name for d in cube.dimensions} | {m.name for m in cube.measures} + var_owner: dict[str, str] = {} + for ref in refs: + if ref.cube != cube.name: + return CubeConversionIssue( + category=CubeIssueCategory.FILTER_PARAMS_UNSUPPORTED, severity="error", + cube=cube.name, member=ref.member, + message=f"Cross-cube FILTER_PARAMS reference " + f"'{ref.cube}.{ref.member}' (host cube is '{cube.name}'); " + f"cube dropped.") + if ref.member not in valid_members: + return CubeConversionIssue( + category=CubeIssueCategory.FILTER_PARAMS_UNSUPPORTED, severity="error", + cube=cube.name, member=ref.member, + message=f"FILTER_PARAMS references unknown member " + f"'{ref.member}'; cube dropped.") + for var in ref.var_names: + prior = var_owner.get(var) + if prior is not None and prior != ref.member: + return CubeConversionIssue( + category=CubeIssueCategory.FILTER_PARAMS_UNSUPPORTED, severity="error", + cube=cube.name, member=ref.member, + message=f"FILTER_PARAMS variable '{var}' is generated by both " + f"member '{prior}' and '{ref.member}'; cube dropped.") + var_owner[var] = ref.member + return None + + def _required_members(self, cube: CubeCube, refs) -> set[str]: + if not self.honor_required_meta: + return set() + return {r.member for r in refs if self._member_required(cube, r.member)} + + def _member_required(self, cube: CubeCube, member: str) -> bool: + entity = next((d for d in cube.dimensions if d.name == member), None) \ + or next((m for m in cube.measures if m.name == member), None) + return bool(entity and (entity.meta or {}).get("required")) + + def _member_description(self, cube: CubeCube, member: str) -> str | None: + entity = next((d for d in cube.dimensions if d.name == member), None) \ + or next((m for m in cube.measures if m.name == member), None) + return entity.description if entity is not None else None + + def _build_cube_variables(self, cube: CubeCube, refs) -> dict: + """Stash one entry per emitted variable under ``meta.cube_variables``. + + ``list_valued`` is the front-end-NEUTRAL half of the contract, and the + only field the engine reads: the string form emits the fixed template + ``col IN ({var})``, whose parentheses the importer — not the caller — + wrote, so the caller cannot supply the per-element quotes a scalar + placeholder normally expects. Flagging it lets the engine coerce a bare + scalar to a one-element list instead of rendering an unquoted identifier + (DEV-1730). ``kind`` stays Cube's own taxonomy, for the report/humans. + """ + required = self._required_members(cube, refs) + out: dict = {} + for ref in refs: + for var in ref.var_names: + out[var] = { + "member": ref.member, + "required": ref.member in required, + "kind": ref.kind, + "list_valued": ref.kind == "string", + "description": self._member_description(cube, ref.member), + } + return out + + def _report_filter_param_variables(self, cube, refs, report) -> None: + # Dedup by variable NAME, not by member: one member can generate several + # variables across refs (e.g. a scalar arrow yielding `_from` and a range + # arrow yielding `_from`+`_to`), and each logical variable must be + # reported exactly once so the report matches meta.cube_variables. + seen_vars: set[str] = set() + required = self._active_required + for ref in refs: + new_vars = [v for v in ref.var_names if v not in seen_vars] + if not new_vars: + continue + seen_vars.update(new_vars) + req = "required" if ref.member in required else "optional" + report.add(CubeConversionIssue( + category=CubeIssueCategory.FILTER_PARAMS_VARIABLE, severity="info", + cube=cube.name, member=ref.member, + message=f"FILTER_PARAMS member '{ref.member}' → {req} variable(s) " + f"{new_vars}.")) + + def _resolve_fp(self, translated: str) -> str: + """Replace FILTER_PARAMS sentinels in already-``translate_cube_refs``-ed + text with their SLayer Mode-A form (block / bare). A no-op when the text + carries no sentinels.""" + return apply_filter_params( + translated, self._active_refs, required_members=self._active_required) + + def _cube_source(self, cube: CubeCube, report) -> dict | None: + if cube.sql_table: + return {"sql_table": cube.sql_table} + if cube.sql: + translated = self._resolve_fp( + translate_cube_refs(cube.sql, mode="sql", cube=cube.name)) + try: + sqlglot.parse_one(render_probe_text(translated)) + except Exception: # noqa: BLE001 + report.add(CubeConversionIssue( + category=CubeIssueCategory.COMPLEX_SQL, severity="error", + cube=cube.name, message="Cube 'sql' could not be translated; cube dropped.", + )) + return None + return {"sql": translated} + report.add(CubeConversionIssue( + category=CubeIssueCategory.NO_SOURCE, severity="error", + cube=cube.name, message="Cube has no sql_table/sql source; dropped.", + )) + return None + + # ── dimensions ───────────────────────────────────────────────────────── + + def _convert_dimension(self, cube, dim: CubeDimension, columns, names, unmapped, report) -> None: + if dim.type == "switch": + report.add(CubeConversionIssue( + category=CubeIssueCategory.DEFERRED_STAGE2, severity="warning", + cube=cube.name, member=dim.name, + message="`switch` dimension is a Tesseract feature (Stage 2); skipped.", + )) + return + if dim.type == "geo": + unmapped.setdefault("geo", []).append( + {"name": dim.name, "latitude": dim.latitude, "longitude": dim.longitude}) + report.add(CubeConversionIssue( + category=CubeIssueCategory.GEO_UNMAPPED, severity="warning", + cube=cube.name, member=dim.name, message="geo dimension has no SLayer type; stashed.")) + return + if dim.sub_query: + unmapped.setdefault("sub_query", []).append(dim.name) + report.add(CubeConversionIssue( + category=CubeIssueCategory.SUBQUERY_UNMAPPED, severity="warning", + cube=cube.name, member=dim.name, message="sub_query dimension has no SLayer equivalent.")) + return + if dim.granularities: + report.add(CubeConversionIssue( + category=CubeIssueCategory.GRANULARITY_UNMAPPED, severity="info", + cube=cube.name, member=dim.name, + message="Custom granularities are query-time in SLayer; base column kept.")) + + if dim.case: + sql = self._build_case_sql(cube, dim.case) + elif dim.sql: + translated = self._resolve_fp( + translate_cube_refs(dim.sql, mode="sql", cube=cube.name)) + sql = None if translated == dim.name else translated + else: + sql = None + + name = names.take(dim.name) + columns.append(Column( + name=name, sql=sql, type=_DIM_TYPE_MAP.get(dim.type, DataType.TEXT), + primary_key=dim.primary_key, hidden=not dim.public, + label=dim.title, description=dim.description, meta=dim.meta, + format=_map_format(dim.format, report, cube=cube.name, member=dim.name), + )) + + def _build_case_sql(self, cube, case: dict) -> str: + parts = ["CASE"] + for when in case.get("when", []): + cond = self._resolve_fp( + translate_cube_refs(when.get("sql", ""), mode="sql", cube=cube.name)) + parts.append(f"WHEN {cond} THEN {_sql_str_literal(when.get('label', ''))}") + if case.get("else"): + parts.append(f"ELSE {_sql_str_literal(case['else'].get('label', ''))}") + parts.append("END") + return " ".join(parts) + + # ── measures ─────────────────────────────────────────────────────────── + + def _convert_measure(self, cube, meas: CubeMeasure, columns, measures, names, dedup, info, report) -> None: + if meas.type in _DEFERRED_MEASURE_TYPES or meas.case is not None: + report.add(CubeConversionIssue( + category=CubeIssueCategory.DEFERRED_STAGE2, severity="warning", + cube=cube.name, member=meas.name, + message=f"Measure type/shape '{meas.type}' is a Tesseract feature (Stage 2); skipped.")) + return + if meas.drill_members: + report.add(CubeConversionIssue( + category=CubeIssueCategory.UNMAPPED_INFRA, severity="info", + cube=cube.name, member=meas.name, message="drill_members has no SLayer equivalent.")) + if meas.multi_stage or meas.time_shift or meas.grain or meas.filter is not None: + report.add(CubeConversionIssue( + category=CubeIssueCategory.DEFERRED_STAGE2, severity="info", + cube=cube.name, member=meas.name, + message="multi_stage/time_shift/grain/filter on measure deferred to Stage 2; " + "emitted as plain aggregation if possible.")) + + if meas.name in ALL_TRANSFORMS: + report.add(CubeConversionIssue( + category=CubeIssueCategory.COMPLEX_MEASURE, severity="warning", + cube=cube.name, member=meas.name, + message=f"Measure name '{meas.name}' shadows a SLayer transform; skipped.")) + return + + # Reserve the measure's name BEFORE creating its underlying column, so a + # bare-identifier column yields (`rate` → `rate_col`) and the measure keeps + # the Cube name (dbt-importer idiom). Collision with a dimension column + # falls through to the `_measure` suffix. + final_name = names.take(meas.name, suffix="_measure") + if meas.type in _CALC_TYPES and meas.sql: + self._convert_calc_measure(cube, meas, measures, names, final_name, info, report) + return + self._convert_agg_measure(cube, meas, columns, measures, names, dedup, info, report, final_name) + + def _convert_agg_measure(self, cube, meas, columns, measures, names, dedup, info, report, final_name) -> None: + agg = "count_distinct" if meas.type == "count_distinct_approx" else meas.type + if meas.type == "count_distinct_approx": + report.add(CubeConversionIssue( + category=CubeIssueCategory.LOSSY_MAPPING, severity="info", + cube=cube.name, member=meas.name, + message="count_distinct_approx → exact count_distinct (SLayer has no approx).")) + + window = _window_from_rolling(meas.rolling_window, report, cube=cube.name, member=meas.name) \ + if meas.rolling_window else None + + if meas.type == "count" and not meas.sql: + formula = _STAR_COUNT + (f"(window='{window}')" if window else "") + if self._emit_measure(measures=measures, names=names, final_name=final_name, + formula=formula, meas=meas, report=report, cube_name=cube.name): + info[meas.name] = _MeasureInfo(kind="star_count", emitted_name=final_name) + return + + translated = self._resolve_fp( + translate_cube_refs(meas.sql, mode="sql", cube=cube.name)) + filter_pred = self._measure_filter(cube, meas) + col_name = self._get_or_create_column( + meas, translated, filter_pred, columns, names, dedup, report, cube) + formula = f"{col_name}:{agg}" + (f"(window='{window}')" if window else "") + if self._emit_measure(measures=measures, names=names, final_name=final_name, + formula=formula, meas=meas, report=report, cube_name=cube.name): + info[meas.name] = _MeasureInfo( + kind="agg", underlying_col=col_name, agg=agg, emitted_name=final_name) + + def _convert_calc_measure(self, cube, meas, measures, names, final_name, info, report) -> None: + formula = translate_cube_refs(meas.sql, mode="dsl", cube=cube.name) + # Pass `names` (not None) so a failed calc measure releases its reserved + # name, matching the aggregate path — otherwise a later same-named member + # gets an unnecessary suffix. + if self._emit_measure(measures=measures, names=names, final_name=final_name, + formula=formula, meas=meas, report=report, cube_name=cube.name, + result_type=_DIM_TYPE_MAP.get(meas.type)): + info[meas.name] = _MeasureInfo(kind="calc", emitted_name=final_name) + + def _measure_filter(self, cube, meas) -> str | None: + if not meas.filters: + return None + preds = [self._resolve_fp(translate_cube_refs(f.sql, mode="sql", cube=cube.name)) + for f in meas.filters] + return " AND ".join(preds) if preds else None + + def _get_or_create_column(self, meas, translated_sql, filter_pred, columns, names, dedup, report, cube) -> str: + key = (translated_sql, filter_pred) + if key in dedup: + return dedup[key] + base = translated_sql if is_bare_identifier(translated_sql) else f"{meas.name}_col" + col_name = names.take(base) + columns.append(Column( + name=col_name, + sql=None if col_name == translated_sql else translated_sql, + type=DataType.DOUBLE, filter=filter_pred, + format=_map_format(meas.format, report, cube=cube.name, member=meas.name), + )) + dedup[key] = col_name + return col_name + + def _emit_measure(self, measures, names, final_name, formula, meas, report, cube_name, *, result_type=None) -> bool: + try: + measures.append(ModelMeasure( + name=final_name, formula=formula, label=meas.title, + description=meas.description, type=result_type, meta=meas.meta)) + return True + except Exception as exc: # noqa: BLE001 + if names is not None: + names.used.discard(final_name) + report.add(CubeConversionIssue( + category=CubeIssueCategory.COMPLEX_MEASURE, severity="warning", + cube=cube_name, member=meas.name, + message=f"Measure '{meas.name}' could not be built: {exc}")) + return False + + # ── segments ─────────────────────────────────────────────────────────── + + def _convert_segment(self, cube, seg, columns, names, report) -> None: + name = names.take(seg.name, suffix="_seg") + columns.append(Column( + name=name, + sql=self._resolve_fp(translate_cube_refs(seg.sql, mode="sql", cube=cube.name)), + type=DataType.BOOLEAN, hidden=not seg.public, + label=seg.title, description=seg.description, meta=seg.meta)) + report.add(CubeConversionIssue( + category=CubeIssueCategory.SEGMENT_AS_COLUMN, severity="info", + cube=cube.name, member=seg.name, + message=f"Segment '{seg.name}' mapped to a boolean column.")) + + # ── joins ────────────────────────────────────────────────────────────── + + def _convert_joins(self, cube, report) -> list[ModelJoin]: + joins: list[ModelJoin] = [] + for cj in cube.joins: + if cj.name not in self._cubes: + report.add(CubeConversionIssue( + category=CubeIssueCategory.UNSUPPORTED_JOIN, severity="warning", + cube=cube.name, member=cj.name, + message=f"Join target cube '{cj.name}' is not available; dropped.")) + continue + pairs = parse_join_on(cj.sql, source_cube=cube.name, target_cube=cj.name) + resolved = self._resolve_join_pairs(cube, cj, pairs) if pairs else None + if not resolved: + report.add(CubeConversionIssue( + category=CubeIssueCategory.UNSUPPORTED_JOIN, severity="warning", + cube=cube.name, member=cj.name, + message=f"Join ON '{cj.sql}' is not an equality of physical columns; dropped.")) + continue + joins.append(ModelJoin(target_model=cj.name, join_pairs=resolved, join_type=JoinType.LEFT)) + return joins + + def _resolve_join_pairs(self, cube, cj, pairs) -> list[list[str]] | None: + target = self._cubes.get(cj.name) + out: list[list[str]] = [] + for src_member, tgt_member in pairs: + src = self._physical_col(cube, src_member) + tgt = self._physical_col(target, tgt_member) if target else tgt_member + if src is None or tgt is None: + return None + out.append([src, tgt]) + return out + + def _physical_col(self, cube, member: str) -> str | None: + if cube is None: + return member + dim = next((d for d in cube.dimensions if d.name == member), None) + if dim is None or dim.sql is None: + return member + translated = translate_cube_refs(dim.sql, mode="sql", cube=cube.name) + return translated.strip() if is_bare_identifier(translated) else None + + # ── offline validation + namespace safety ────────────────────────────── + + def _validate_offline(self, cube_name, columns, measures, report): + good_cols = [] + dropped: set[str] = set() + for col in columns: + if col.sql is None: + good_cols.append(col) + continue + try: + # A column sql may carry {? ?} blocks / {var} placeholders + # (DEV-1730); probe-render before the syntax-only parse. + sqlglot.parse_one(render_probe_text(col.sql)) + good_cols.append(col) + except Exception: # noqa: BLE001 + dropped.add(col.name) + report.add(CubeConversionIssue( + category=CubeIssueCategory.COMPLEX_SQL, severity="warning", + cube=cube_name, member=col.name, + message=f"Column sql does not parse; dropped: {col.sql!r}")) + col_names = {c.name for c in good_cols} + known = col_names | {m.name for m in measures if m.name} + good_measures = [] + for m in measures: + ref_col = m.formula.split(":")[0].strip() + if ref_col in dropped: + continue + if not self._formula_parses(m.formula, known): + report.add(CubeConversionIssue( + category=CubeIssueCategory.COMPLEX_MEASURE, severity="warning", + cube=cube_name, member=m.name, + message=f"Measure formula does not parse; dropped: {m.formula!r}")) + continue + good_measures.append(m) + return good_cols, good_measures + + def _formula_parses(self, formula: str, known_names: set[str]) -> bool: + nm = dict.fromkeys(known_names, _STAR_COUNT) + try: + parse_formula(formula, named_measures=nm or None) + return True + except Exception: # noqa: BLE001 + return False + + def _dedisambiguate_namespace(self, columns, measures, report, *, cube) -> None: + # The allocator already guarantees uniqueness; this is a defensive check. + col_names = {c.name for c in columns} + for m in measures: + if m.name in col_names: # pragma: no cover — allocator prevents this + report.add(CubeConversionIssue( + category=CubeIssueCategory.COMPLEX_MEASURE, severity="info", + cube=cube, member=m.name, message="measure/column name overlap auto-resolved.")) + + # ── views → facade models ────────────────────────────────────────────── + + def _convert_view(self, view: CubeView, report) -> SlayerModel | None: + if not view.cubes: + report.add(CubeConversionIssue( + category=CubeIssueCategory.AMBIGUOUS_VIEW_ROOT, severity="warning", + view=view.name, message="View has no cubes; skipped.")) + return None + root_cube_name = view.cubes[0].join_path.split(".")[0] + root_cube = self._cubes.get(root_cube_name) + root_model = self._models.get(root_cube_name) + if root_cube is None or root_model is None: + report.add(CubeConversionIssue( + category=CubeIssueCategory.AMBIGUOUS_VIEW_ROOT, severity="warning", + view=view.name, + message=f"View root cube '{root_cube_name}' was not emitted; view dropped.")) + return None + + source = {"sql_table": root_model.sql_table} if root_model.sql_table else {"sql": root_model.sql} + meta = {"cube_kind": "view"} + unmapped: dict = {} + if view.folders: + unmapped["folders"] = view.folders + report.add(CubeConversionIssue( + category=CubeIssueCategory.FOLDERS_UNMAPPED, severity="info", + view=view.name, message="Folders have no SLayer hierarchy; stashed in meta.")) + + names = _Names() + columns: list[Column] = [] + measures: list[ModelMeasure] = [] + join_targets: set[str] = set() + + for ref in view.cubes: + self._convert_view_ref(view, ref, root_cube_name, root_model, + columns, measures, names, join_targets, report) + + joins = [j for j in root_model.joins if j.target_model in join_targets] + filters = self._view_default_filters(view, root_cube_name, report) + if unmapped: + meta["cube_unmapped"] = unmapped + + try: + return SlayerModel( + name=view.name, data_source=self.data_source, + hidden=not view.public, description=view.description, meta=meta, + columns=columns, measures=measures, joins=joins, filters=filters, + **source, + ) + except Exception as exc: # noqa: BLE001 + report.add(CubeConversionIssue( + category=CubeIssueCategory.PARSE_ERROR, severity="error", + view=view.name, message=f"Could not build facade model: {exc}")) + return None + + def _convert_view_ref(self, view, ref, root_cube_name, root_model, + columns, measures, names, join_targets, report) -> None: + path = ref.join_path.split(".") + cube_name = path[-1] + cube = self._cubes.get(cube_name) + cube_model = self._models.get(cube_name) + is_root = (len(path) == 1 and cube_name == root_cube_name) + + if cube is None or cube_model is None: + report.add(CubeConversionIssue( + category=CubeIssueCategory.DISCONNECTED_VIEW, severity="warning", + view=view.name, member=ref.join_path, + message=f"View member cube '{cube_name}' not available; skipped.")) + return + + if not is_root: + join = next((j for j in root_cube.joins if j.name == cube_name), None) \ + if (root_cube := self._cubes.get(root_cube_name)) else None + if join is None or not any(j.target_model == cube_name for j in root_model.joins): + report.add(CubeConversionIssue( + category=CubeIssueCategory.DISCONNECTED_VIEW, severity="warning", + view=view.name, member=ref.join_path, + message=f"'{cube_name}' is not joined to root '{root_cube_name}'; skipped.")) + return + if join.relationship in ("one_to_many", "has_many"): + report.add(CubeConversionIssue( + category=CubeIssueCategory.VIEW_FANOUT_RISK, severity="warning", + view=view.name, member=ref.join_path, + message=f"Join to '{cube_name}' is {join.relationship}; root measures may fan out.")) + join_targets.add(cube_name) + + prefix = f"{ref.alias or cube_name}_" if ref.prefix else "" + dim_names, meas_names = self._selected_members(cube, ref, view, report) + + for dname in dim_names: + self._facade_dimension(cube, cube_model, dname, prefix, is_root, + columns, names) + for mname in meas_names: + self._facade_measure(view, cube_name, cube_model, mname, prefix, is_root, + columns, measures, names, report) + + def _selected_members(self, cube, ref, view, report) -> tuple[list[str], list[str]]: + dims = [d.name for d in cube.dimensions if d.type not in ("geo", "switch") and not d.sub_query] + meas = [m.name for m in cube.measures] + exclude = set(ref.excludes or []) + if ref.includes in (None, "*"): + # `includes: "*"` must not re-export members the cube marked private. + private = {d.name for d in cube.dimensions if not d.public} \ + | {m.name for m in cube.measures if not m.public} + chosen = [n for n in dims + meas if n not in exclude and n not in private] + else: + chosen = self._include_names(ref.includes, exclude, view, report) + chosen_set = set(chosen) + return ([d for d in dims if d in chosen_set], [m for m in meas if m in chosen_set]) + + def _include_names(self, includes, exclude, view, report) -> list[str]: + """Extract member names from an ``includes`` list. Cube's per-member + override object form (``{name, format, meta, …}``) is accepted so it + doesn't crash the view; the overrides are reported as unsupported. An + entry with no valid ``name`` (Cube requires one) — e.g. ``{}`` or + ``{"alias": ...}`` — is reported as a parse error rather than silently + dropped or reported with a confusing ``member=None``.""" + names: list[str] = [] + for entry in includes: + name = entry.get("name") if isinstance(entry, dict) else entry + if not isinstance(name, str) or not name: + report.add(CubeConversionIssue( + category=CubeIssueCategory.PARSE_ERROR, severity="warning", + view=view.name, + message=f"View include entry {entry!r} has no valid 'name'; skipped.")) + continue + if isinstance(entry, dict) and any(k != "name" for k in entry): + report.add(CubeConversionIssue( + category=CubeIssueCategory.UNMAPPED_INFRA, severity="info", + view=view.name, member=name, + message=f"Per-member override on '{name}' is not applied (Stage 1).")) + if name not in exclude: + names.append(name) + return names + + def _facade_dimension(self, cube, cube_model, dname, prefix, is_root, columns, names) -> None: + col = cube_model.get_column(dname) + if col is None: + return + if is_root: + sql = col.sql if col.sql else col.name + else: + sql = f"{cube.name}.{col.name}" + exported = names.take(f"{prefix}{dname}") + columns.append(Column( + name=exported, sql=sql, type=col.type, label=col.label, + description=col.description, format=col.format)) + + def _facade_measure(self, view, cube_name, cube_model, mname, prefix, is_root, + columns, measures, names, report) -> None: + info = self._measure_info.get(cube_name, {}).get(mname) + if info is None: + return + exported = names.take(f"{prefix}{mname}", suffix="_measure") + if info.kind == "star_count": + formula = _STAR_COUNT if is_root else f"{cube_name}.{_STAR_COUNT}" + elif info.kind == "agg": + base = f"{info.underlying_col}:{info.agg}" + if is_root: + src_col = cube_model.get_column(info.underlying_col) + if src_col is not None and not any(c.name == info.underlying_col for c in columns): + columns.append(src_col.model_copy()) # carry the underlying column onto the facade + formula = base + else: + formula = f"{cube_name}.{info.underlying_col}:{info.agg}" + else: + report.add(CubeConversionIssue( + category=CubeIssueCategory.COMPLEX_MEASURE, severity="info", + view=view.name, member=mname, + message=f"Calculated measure '{mname}' re-export not supported in Stage 1.")) + return + try: + measures.append(ModelMeasure(name=exported, formula=formula)) + except Exception as exc: # noqa: BLE001 + report.add(CubeConversionIssue( + category=CubeIssueCategory.COMPLEX_MEASURE, severity="warning", + view=view.name, member=mname, message=f"Facade measure failed: {exc}")) + + def _view_default_filters(self, view, root_cube_name, report) -> list[str]: + filters: list[str] = [] + for df in view.default_filters or []: + member = df.get("member", "") + op = df.get("operator") + values = df.get("values") or [] + col = self._resolve_view_member(member, root_cube_name) + if op == "equals" and len(values) == 1: + filters.append(f"{col} = {_sql_str_literal(values[0])}") + elif op in ("equals", "in") and values: + vlist = ", ".join(_sql_str_literal(v) for v in values) + filters.append(f"{col} IN ({vlist})") + else: + report.add(CubeConversionIssue( + category=CubeIssueCategory.UNSUPPORTED_DEFAULT_FILTER, severity="info", + view=view.name, member=member, + message=f"default_filter operator '{op}' not mapped; dropped.")) + return filters + + def _resolve_view_member(self, member: str, root_cube_name: str) -> str: + parts = member.split(".") + if parts and parts[0] == root_cube_name: + parts = parts[1:] + return ".".join(parts) if parts else member diff --git a/slayer/cube/extends.py b/slayer/cube/extends.py new file mode 100644 index 00000000..43e634db --- /dev/null +++ b/slayer/cube/extends.py @@ -0,0 +1,126 @@ +"""Resolve Cube `extends` by flattening base members into children. + +DEV-1608 §5. Native persisted inheritance is tracked separately in DEV-1610; +this module flattens (faithful to Cube's own compile-time materialization). +""" + +from slayer.cube.models import CubeCube, CubeView +from slayer.cube.report import CubeConversionIssue, CubeIssueCategory + + +def _merge_by_name(parent_items: list, child_items: list) -> list: + """Merge two member lists by ``.name``; child wins on conflict, parent order + preserved, child-new appended.""" + merged: dict[str, object] = {item.name: item for item in parent_items} + for item in child_items: + merged[item.name] = item + return list(merged.values()) + + +def _merge_cube(parent: CubeCube, child: CubeCube) -> CubeCube: + if child.sql_table or child.sql: + sql_table, sql = child.sql_table, child.sql + else: + sql_table, sql = parent.sql_table, parent.sql + return child.model_copy(update={ + "extends": None, + "sql_table": sql_table, + "sql": sql, + "dimensions": _merge_by_name(parent.dimensions, child.dimensions), + "measures": _merge_by_name(parent.measures, child.measures), + "joins": _merge_by_name(parent.joins, child.joins), + "segments": _merge_by_name(parent.segments, child.segments), + }) + + +def _find_cyclic(by_name: dict) -> set[str]: + """Return every node that lies on an ``extends`` cycle. + + Detecting the full cycle up front (rather than only the closing frame) means + *no* node on a cycle inherits — matching the "flattened without inheritance" + contract. Nodes that merely *extend into* a cycle are not cyclic; they + inherit the cyclic node's own (un-merged) members. + """ + cyclic: set[str] = set() + for start in by_name: + path: list[str] = [] + seen: set[str] = set() + cur = start + while cur in by_name: + nxt = by_name[cur].extends + if not nxt or nxt not in by_name: + break + if nxt in seen or nxt == cur: + if nxt in path: + cyclic.update(path[path.index(nxt):]) + cyclic.add(nxt) + cyclic.add(cur) + break + path.append(cur) + seen.add(cur) + cur = nxt + return cyclic + + +def flatten_cube_extends( + cubes: list[CubeCube], +) -> tuple[list[CubeCube], list[CubeConversionIssue]]: + """Flatten the cube `extends` graph (child wins; multi-level transitive; + cycles reported). Every cube is still returned (hidden iff ``public: false`` + is handled downstream by the converter).""" + by_name = {c.name: c for c in cubes} + cyclic = _find_cyclic(by_name) + issues = [CubeConversionIssue( + category=CubeIssueCategory.EXTENDS_CYCLE, severity="error", cube=name, + message=f"Cube '{name}' is in an extends cycle; flattened without inheritance.") + for name in sorted(cyclic)] + resolved: dict[str, CubeCube] = {} + + def resolve(name: str) -> CubeCube: + if name in resolved: + return resolved[name] + cube = by_name[name] + if name in cyclic or not cube.extends or cube.extends not in by_name: + resolved[name] = cube + return cube + merged = _merge_cube(resolve(cube.extends), cube) + resolved[name] = merged + return merged + + return [resolve(c.name) for c in cubes], issues + + +def _merge_view(parent: CubeView, child: CubeView) -> CubeView: + return child.model_copy(update={ + "extends": None, + "cubes": list(parent.cubes) + list(child.cubes), + "default_filters": ((parent.default_filters or []) + (child.default_filters or [])) or None, + "folders": ((parent.folders or []) + (child.folders or [])) or None, + }) + + +def flatten_view_extends( + views: list[CubeView], +) -> tuple[list[CubeView], list[CubeConversionIssue]]: + """Flatten the view `extends` graph by concatenating member-contributing + cube refs (the converter dedups members).""" + by_name = {v.name: v for v in views} + cyclic = _find_cyclic(by_name) + issues = [CubeConversionIssue( + category=CubeIssueCategory.EXTENDS_CYCLE, severity="error", view=name, + message=f"View '{name}' is in an extends cycle; flattened without inheritance.") + for name in sorted(cyclic)] + resolved: dict[str, CubeView] = {} + + def resolve(name: str) -> CubeView: + if name in resolved: + return resolved[name] + view = by_name[name] + if name in cyclic or not view.extends or view.extends not in by_name: + resolved[name] = view + return view + merged = _merge_view(resolve(view.extends), view) + resolved[name] = merged + return merged + + return [resolve(v.name) for v in views], issues diff --git a/slayer/cube/filter_params.py b/slayer/cube/filter_params.py new file mode 100644 index 00000000..ad9aeff9 --- /dev/null +++ b/slayer/cube/filter_params.py @@ -0,0 +1,273 @@ +"""Shared Cube ``FILTER_PARAMS`` translation (DEV-1730). + +Cube's ``FILTER_PARAMS...filter()`` renders one of two things +per query: the member's filter (``col IN (values)`` / a date-range predicate) +when the caller supplied a filter on that member, else the neutral ``1 = 1``. +SLayer represents that with a Mode-A optional block ``{? ... ?}`` (optional +member) or a bare ``{var}`` (a member the importer classified as required). + +This module is front-end-agnostic: the JS parser feeds arrow-body *segments* +extracted from the ESTree AST; the YAML path feeds a string col-expr via +:func:`parse_string_filter_params`. Ref construction and rendering live here so +both paths converge. Requiredness (block vs bare) is applied downstream by the +converter, which alone knows ``honor_required_meta`` + the member's ``meta``. +""" + +import re + +from pydantic import BaseModel, Field + +from slayer.cube.models import CubeFilterParamRef + +# A sentinel carries NUL bytes so it can never occur in real SQL and survives +# ``translate_cube_refs`` untouched (it has no braces). Resolved by the converter +# before any sqlglot parse. +_SENTINEL_FMT = "\x00SLAYER_FP_{i}\x00" + + +def filter_param_sentinel(index: int) -> str: + """Return the unique surface-text sentinel for FILTER_PARAMS ref ``index``.""" + return _SENTINEL_FMT.format(i=index) + + +def _member_var(member: str, param_suffix: str) -> str: + """Map an arrow param position (``"from"``/``"to"``) to a variable name.""" + return f"{member}_{param_suffix}" + + +def build_string_ref( + *, cube: str, member: str, col_expr: str, sentinel: str +) -> CubeFilterParamRef: + """Build a ref for the string-arg form ``.filter('col_expr')`` → the + membership body ``col_expr IN ({member})`` (the query supplies a list).""" + body = f"{col_expr} IN ({{{member}}})" + return CubeFilterParamRef( + cube=cube, member=member, kind="string", + body_template=body, var_names=[member], sentinel=sentinel, + ) + + +def build_arrow_ref( + *, cube: str, member: str, segments: list[tuple[str, str]], sentinel: str +) -> CubeFilterParamRef: + """Build a ref for the arrow form ``.filter((from, to) => )``. + + ``segments`` is the arrow body decomposed by the front-end into + ``("lit", text)`` (verbatim SQL) and ``("param", "from"|"to")`` (a date + bound). Params render **pre-quoted** (``'{member_from}'``) because Cube + splices quoted literals. A body that is a single bare param is ``arrow_value`` + (used in scalar SELECT position); anything else is ``arrow_range``. + """ + parts: list[str] = [] + var_names: list[str] = [] + for kind, value in segments: + if kind == "lit": + parts.append(value) + elif kind == "param": + var = _member_var(member, value) + parts.append(f"'{{{var}}}'") + if var not in var_names: + var_names.append(var) + else: # pragma: no cover — defensive + raise ValueError(f"Unknown arrow segment kind {kind!r}") + is_value = len(segments) == 1 and segments[0][0] == "param" + return CubeFilterParamRef( + cube=cube, member=member, + kind="arrow_value" if is_value else "arrow_range", + body_template="".join(parts), var_names=var_names, sentinel=sentinel, + ) + + +def render_filter_param(ref: CubeFilterParamRef, *, required: bool) -> str: + """Render a ref to SLayer Mode-A text: bare body if required (raise-on-missing), + else wrapped in an optional block (collapse to ``(1=1)`` when absent). + + A ref with **no** variables (a degenerate arrow whose body references neither + ``from`` nor ``to``) is always emitted bare — an optional block needs ≥1 + variable to key on, so a var-less block would pass import validation but raise + at runtime. Rendering it bare keeps the constant predicate and stays valid. + """ + if required or not ref.var_names: + return ref.body_template + return "{? " + ref.body_template + " ?}" + + +def apply_filter_params( + text: str, refs: list[CubeFilterParamRef], *, required_members: set[str] +) -> str: + """Replace each ref's sentinel in ``text`` with its rendered Mode-A form, + treating members in ``required_members`` as required (bare).""" + for ref in refs: + text = text.replace( + ref.sentinel, render_filter_param(ref, required=ref.member in required_members) + ) + return text + + +# ── YAML text path (string-arg only) ──────────────────────────────────────── + + +class FilterParamsUnsupported(BaseModel): + """A FILTER_PARAMS occurrence the front-end could not translate.""" + + member: str | None = None + raw: str + reason: str + + +class FilterParamsExtraction(BaseModel): + """Result of scanning a raw-SQL surface for FILTER_PARAMS occurrences.""" + + text: str + refs: list[CubeFilterParamRef] = Field(default_factory=list) + unsupported: list[FilterParamsUnsupported] = Field(default_factory=list) + + +_FP_START = "{FILTER_PARAMS." +_IDENT = re.compile(r"[A-Za-z_]\w*") + + +def _scan_filter_call(text: str, start: int) -> tuple[str, str, str, int] | None: + """Parse one ``{FILTER_PARAMS.cube.member.filter()}`` starting at + ``start`` (index of the opening ``{``). + + Returns ``(cube, member, arg, end)`` where ``end`` is just past the closing + ``}``; ``None`` if the text at ``start`` is not a well-formed occurrence. + Balances parentheses inside ``.filter(...)`` while respecting single-quoted + string literals (``''`` escaping), so arg expressions containing ``)`` / ``,`` + don't truncate the match. + """ + pos = start + len(_FP_START) + m = _IDENT.match(text, pos) + if not m: + return None + cube = m.group(0) + pos = m.end() + if not text.startswith(".", pos): + return None + pos += 1 + m = _IDENT.match(text, pos) + if not m: + return None + member = m.group(0) + pos = m.end() + if not text.startswith(".filter(", pos): + return None + pos += len(".filter(") + scanned = _scan_balanced_arg(text, pos) + if scanned is None: + return None + arg, close = scanned + if not text.startswith("}", close + 1): + return None + return cube, member, arg, close + 2 + + +def _scan_balanced_arg(text: str, pos: int) -> tuple[str, int] | None: + """From just inside ``.filter(`` at ``pos``, return ``(arg_text, close_index)`` + for the matching ``)``, or ``None`` if unbalanced. Parentheses are balanced + while respecting single-quoted string literals (``''`` escaping), so a ``)`` + or ``,`` inside a literal doesn't truncate the argument.""" + arg_start = pos + depth = 1 + n = len(text) + while pos < n: + c = text[pos] + if c == "'": + pos = _skip_sq_literal(text, pos) + continue + if c == "(": + depth += 1 + elif c == ")": + depth -= 1 + if depth == 0: + return text[arg_start:pos], pos + pos += 1 + return None + + +def _skip_sq_literal(text: str, pos: int) -> int: + """``pos`` is at a single-quote opening a literal; return the index just past + its closing quote (``''`` is an escaped quote, not a close).""" + n = len(text) + pos += 1 + while pos < n: + if text[pos] == "'": + if pos + 1 < n and text[pos + 1] == "'": + pos += 2 + continue + return pos + 1 + pos += 1 + return pos + + +def parse_string_filter_params( + text: str, *, host_cube: str, start_index: int = 0 +) -> FilterParamsExtraction: + """Scan a raw-SQL surface for FILTER_PARAMS occurrences (YAML text path). + + Only the string-arg form ``.filter('col_expr')`` is supported here (arrow + forms require the JS AST). Cross-cube refs (cube segment ≠ ``host_cube``) and + arrow forms are reported as unsupported. Each supported occurrence is replaced + with a sentinel and captured as a :class:`CubeFilterParamRef`. + + ``start_index`` offsets the sentinel numbering so multiple surfaces of one + cube can be scanned without sentinel collisions. + """ + out: list[str] = [] + refs: list[CubeFilterParamRef] = [] + unsupported: list[FilterParamsUnsupported] = [] + i = 0 + n = len(text) + while i < n: + if text.startswith(_FP_START, i): + parsed = _scan_filter_call(text, i) + if parsed is not None: + cube, member, arg, end = parsed + raw = text[i:end] + ref = _classify_string_arg( + cube=cube, member=member, arg=arg.strip(), raw=raw, + host_cube=host_cube, index=start_index + len(refs), + ) + if isinstance(ref, CubeFilterParamRef): + refs.append(ref) + out.append(ref.sentinel) + else: + unsupported.append(ref) + out.append(raw) # leave verbatim; the cube will be dropped + i = end + continue + out.append(text[i]) + i += 1 + return FilterParamsExtraction( + text="".join(out), refs=refs, unsupported=unsupported + ) + + +def _classify_string_arg( + *, cube: str, member: str, arg: str, raw: str, host_cube: str, index: int +): + """Return a CubeFilterParamRef for a supported string-arg occurrence, else a + FilterParamsUnsupported.""" + if cube != host_cube: + return FilterParamsUnsupported( + member=member, raw=raw, + reason=f"cross-cube FILTER_PARAMS reference '{cube}.{member}' " + f"(host cube is '{host_cube}') is not supported (Stage 1).", + ) + if "=>" in arg: + return FilterParamsUnsupported( + member=member, raw=raw, + reason="arrow-form FILTER_PARAMS is only supported via the JS " + "front-end, not in YAML text.", + ) + if len(arg) >= 2 and arg[0] in "'\"" and arg[-1] == arg[0]: + col_expr = arg[1:-1] + return build_string_ref( + cube=cube, member=member, col_expr=col_expr, + sentinel=filter_param_sentinel(index), + ) + return FilterParamsUnsupported( + member=member, raw=raw, + reason=f"unrecognised FILTER_PARAMS argument: {arg!r}.", + ) diff --git a/slayer/cube/js_parser.py b/slayer/cube/js_parser.py new file mode 100644 index 00000000..cd83af95 --- /dev/null +++ b/slayer/cube/js_parser.py @@ -0,0 +1,475 @@ +"""Parse JavaScript Cube configs into ``CubeCube`` / ``CubeView`` (DEV-1730). + +Handles the declarative ``cube('Name', {...})`` / ``view(...)`` subset via an +esprima ESTree AST, converting object/array/template literals into the same dict +shapes the YAML front-end feeds to ``CubeCube.model_validate`` — so the converter +stays front-end-agnostic. Anything dynamic (helper calls, spreads, identifier +refs, computed keys) is reported as an issue and the affected member/cube is +skipped ("report, don't crash", mirroring the YAML path). + +``${CUBE}`` / ``${member}`` / ``${a.b}`` interpolations become single-brace Cube +refs (``{CUBE}`` …); ``${FILTER_PARAMS...}`` interpolations become structured +``CubeFilterParamRef`` entries + sentinels in the surface text (see +``slayer.cube.filter_params``). +""" + +import logging +import re + +import esprima +from pydantic import BaseModel, Field + +from slayer.cube.filter_params import ( + build_arrow_ref, + build_string_ref, + filter_param_sentinel, +) +from slayer.cube.models import ( + CubeCube, + CubeDimension, + CubeFilterParamRef, + CubeJoin, + CubeMeasure, + CubeSegment, + CubeView, + CubeViewCubeRef, +) +from slayer.cube.report import CubeConversionIssue, CubeIssueCategory + +logger = logging.getLogger(__name__) + +_CUBE_FIELDS = set(CubeCube.model_fields) +_VIEW_FIELDS = set(CubeView.model_fields) +_DIM_FIELDS = set(CubeDimension.model_fields) +_MEAS_FIELDS = set(CubeMeasure.model_fields) +_SEG_FIELDS = set(CubeSegment.model_fields) +_JOIN_FIELDS = set(CubeJoin.model_fields) +_VIEW_CUBE_FIELDS = set(CubeViewCubeRef.model_fields) + +# member-map sections: key → (target field set) +_MEMBER_SECTIONS = { + "dimensions": _DIM_FIELDS, + "measures": _MEAS_FIELDS, + "segments": _SEG_FIELDS, +} +_PARAM_SUFFIX = {0: "from", 1: "to"} + + +class CubeJsParseResult(BaseModel): + cubes: list[CubeCube] = Field(default_factory=list) + views: list[CubeView] = Field(default_factory=list) + issues: list[CubeConversionIssue] = Field(default_factory=list) + + +class _DynamicValue(Exception): + """Raised when a value can't be statically converted (dynamic JS).""" + + def __init__(self, reason: str) -> None: + super().__init__(reason) + self.reason = reason + + +def _camel_to_snake(name: str) -> str: + return re.sub(r"(?") -> CubeJsParseResult: + """Parse a JavaScript Cube config source into cubes + views + issues.""" + issues: list[CubeConversionIssue] = [] + cubes: list[CubeCube] = [] + views: list[CubeView] = [] + tree = _parse_js_source(source, path, issues) + if tree is None: + return CubeJsParseResult(issues=issues) + + for call in _iter_cube_calls(tree): + _Walker(path=path, issues=issues).convert_call(call, cubes, views) + return CubeJsParseResult(cubes=cubes, views=views, issues=issues) + + +def _parse_js_source(source: str, path: str, issues: list[CubeConversionIssue]): + """Parse ``source`` to an ESTree, tolerating both plain script configs and + ES-module configs. ``parseScript`` is tried first (the classic global-``cube`` + style); on failure ``parseModule`` handles top-level ``import`` / ``export`` + (which ``parseScript`` rejects). Returns the tree, or ``None`` + a reported + issue when neither parses.""" + opts = {"range": True, "comment": True} + try: + return esprima.parseScript(source, opts) + except Exception: # noqa: BLE001 — retry as a module before giving up + pass + try: + return esprima.parseModule(source, opts) + except Exception as exc: # noqa: BLE001 — surface as a report issue + issues.append(CubeConversionIssue( + category=CubeIssueCategory.PARSE_ERROR, severity="warning", + message=f"File '{path}' could not be parsed as JavaScript: {exc}", + )) + return None + + +def _iter_cube_calls(tree): + """Yield top-level ``cube(...)`` / ``view(...)`` CallExpression nodes, + unwrapping ``module.exports = ``, ``export default``, and ``const x = `` + wrappers.""" + for stmt in tree.body: + for node in _candidate_expressions(stmt): + if _is_cube_or_view_call(node): + yield node + + +def _candidate_expressions(stmt): + t = stmt.type + if t == "ExpressionStatement": + expr = stmt.expression + if expr.type == "AssignmentExpression": + return [expr.right] + return [expr] + if t == "ExportDefaultDeclaration": + return [stmt.declaration] + if t == "VariableDeclaration": + return [d.init for d in stmt.declarations if d.init is not None] + if t == "ExportNamedDeclaration" and stmt.declaration is not None: + return _candidate_expressions(stmt.declaration) + return [] + + +def _is_cube_or_view_call(node) -> bool: + return ( + node is not None + and node.type == "CallExpression" + and node.callee.type == "Identifier" + and node.callee.name in ("cube", "view") + ) + + +class _Walker: + """Per-file walker; accumulates issues and per-cube FILTER_PARAMS refs.""" + + def __init__(self, *, path: str, issues: list[CubeConversionIssue]) -> None: + self.path = path + self.issues = issues + self._fp_refs: list[CubeFilterParamRef] = [] + self._fp_counter = 0 + + # ── entry ──────────────────────────────────────────────────────────── + + def convert_call(self, call, cubes: list, views: list) -> None: + kind = call.callee.name + name = self._name_arg(call.arguments[0]) if call.arguments else None + if name is None: + self.issues.append(CubeConversionIssue( + category=CubeIssueCategory.PARSE_ERROR, severity="warning", + message=f"{kind}() in '{self.path}' has no static string name; skipped.")) + return + if len(call.arguments) < 2 or call.arguments[1].type != "ObjectExpression": + self.issues.append(CubeConversionIssue( + category=CubeIssueCategory.PARSE_ERROR, severity="warning", + cube=name, message=f"{kind}('{name}') has no object definition; skipped.")) + return + + self._fp_refs = [] + try: + body = self._cube_object(call.arguments[1], cube=name) + except _DynamicValue as exc: + self.issues.append(CubeConversionIssue( + category=CubeIssueCategory.COMPLEX_SQL, severity="warning", + cube=name, message=f"{kind}('{name}') has a dynamic definition: {exc.reason}; skipped.")) + return + body["name"] = name + if kind == "cube": + self._build_cube(name, body, cubes) + else: + self._build_view(name, body, views) + + def _cube_object(self, node, *, cube: str) -> dict: + """Convert a cube()/view() top-level object. Member-map sections + (dimensions/measures/segments/joins) are converted with per-member error + isolation so one dynamic member does not sink the whole cube; a dynamic + NON-member top-level property is fatal to the cube (re-raised).""" + out: dict = {} + for prop in node.properties: + if prop.type != "Property": + raise _DynamicValue("spread / non-literal property at top level") + key = self._key_name(prop) + if key in _MEMBER_SECTIONS or key == "joins": + out[key] = self._member_map(prop.value, section=key, cube=cube) + elif key == "meta": + out[key] = self._raw_meta(prop.value) + else: + out[key] = self._value(prop.value) + return out + + def _member_map(self, node, *, section: str, cube: str) -> dict: + """Convert a ``{name: {...}}`` member map, isolating per-member failures.""" + if node.type != "ObjectExpression": + raise _DynamicValue(f"'{section}' is not an object literal") + out: dict = {} + for prop in node.properties: + if prop.type != "Property": + self.issues.append(CubeConversionIssue( + category=CubeIssueCategory.COMPLEX_SQL, severity="warning", + cube=cube, message=f"Spread/dynamic entry in '{section}'; skipped.")) + continue + member_name: str | None = None + # Snapshot the FILTER_PARAMS ref count so a member that fails partway + # (after a template literal already appended a ref + sentinel) rolls + # back cleanly — otherwise a skipped member could leave a dangling + # ref that drops the whole cube or emits a phantom variable. + fp_mark = len(self._fp_refs) + try: + # _key_name is inside the try so a computed key ([name]: {...}) + # skips just this member instead of sinking the cube. + member_name = self._key_name(prop) + out[member_name] = self._object(prop.value) + except _DynamicValue as exc: + del self._fp_refs[fp_mark:] + label = member_name or "" + self.issues.append(CubeConversionIssue( + category=CubeIssueCategory.COMPLEX_SQL, severity="warning", + cube=cube, member=member_name, + message=f"'{section}.{label}' is dynamic ({exc.reason}); skipped.")) + return out + + def _build_cube(self, name, body, cubes: list) -> None: + d = _normalize_keys(body, _CUBE_FIELDS) + _member_maps_to_lists(d) + d["filter_params"] = [r.model_dump() for r in self._fp_refs] + try: + cubes.append(CubeCube.model_validate(d)) + except Exception as exc: # noqa: BLE001 + self.issues.append(CubeConversionIssue( + category=CubeIssueCategory.PARSE_ERROR, severity="warning", + cube=name, message=f"Could not build cube '{name}': {exc}")) + + def _build_view(self, name, body, views: list) -> None: + d = _normalize_keys(body, _VIEW_FIELDS) + cubes = d.get("cubes") + if isinstance(cubes, list): + d["cubes"] = [_normalize_keys(c, _VIEW_CUBE_FIELDS) if isinstance(c, dict) else c + for c in cubes] + try: + views.append(CubeView.model_validate(d)) + except Exception as exc: # noqa: BLE001 + self.issues.append(CubeConversionIssue( + category=CubeIssueCategory.PARSE_ERROR, severity="warning", + view=name, message=f"Could not build view '{name}': {exc}")) + + # ── value conversion ───────────────────────────────────────────────── + + def _name_arg(self, node) -> str | None: + if node.type == "Literal" and isinstance(node.value, str): + return node.value + if node.type == "TemplateLiteral" and not node.expressions: + return "".join(q.value.cooked for q in node.quasis) + return None + + def _object(self, node, *, in_meta: bool = False) -> dict: + out: dict = {} + for prop in node.properties: + if prop.type != "Property": + raise _DynamicValue("spread / non-literal property") + key = self._key_name(prop) + if key == "meta" and not in_meta: + out[key] = self._raw_meta(prop.value) + else: + out[key] = self._value(prop.value, in_meta=in_meta) + return out + + def _key_name(self, prop) -> str: + if prop.computed: + raise _DynamicValue("computed property key") + k = prop.key + if k.type == "Identifier": + return k.name + if k.type == "Literal": + return str(k.value) + raise _DynamicValue("non-literal property key") + + def _value(self, node, *, in_meta: bool = False): + t = node.type + if t == "Literal": + return node.value + if t == "TemplateLiteral": + return self._template_literal(node) + if t == "ObjectExpression": + return self._object(node, in_meta=in_meta) + if t == "ArrayExpression": + return [self._value(el, in_meta=in_meta) for el in node.elements + if el is not None] + if t == "UnaryExpression" and node.operator in ("-", "+") \ + and node.argument.type == "Literal" \ + and isinstance(node.argument.value, (int, float)): + return -node.argument.value if node.operator == "-" else node.argument.value + raise _DynamicValue(f"unsupported {t}") + + def _raw_meta(self, node): + """Convert a ``meta`` subtree verbatim — no key normalisation, no ref + translation (meta is opaque user JSON).""" + t = node.type + if t == "Literal": + return node.value + if t == "ObjectExpression": + return {self._key_name(p): self._raw_meta(p.value) for p in node.properties + if p.type == "Property"} + if t == "ArrayExpression": + return [self._raw_meta(el) for el in node.elements if el is not None] + if t == "UnaryExpression" and node.operator in ("-", "+") \ + and node.argument.type == "Literal" \ + and isinstance(node.argument.value, (int, float)): + return -node.argument.value if node.operator == "-" else node.argument.value + raise _DynamicValue(f"unsupported meta value {t}") + + # ── template literals + FILTER_PARAMS ──────────────────────────────── + + def _template_literal(self, node) -> str: + parts: list[str] = [] + quasis = node.quasis + exprs = node.expressions + for i, quasi in enumerate(quasis): + parts.append(quasi.value.cooked) + if i < len(exprs): + parts.append(self._interpolation(exprs[i])) + return "".join(parts) + + def _interpolation(self, expr) -> str: + t = expr.type + if t == "Identifier": + return "{" + expr.name + "}" + if t == "MemberExpression": + return "{" + self._member_path(expr) + "}" + if t == "CallExpression": + return self._filter_params(expr) + raise _DynamicValue(f"unsupported interpolation {t}") + + def _member_path(self, expr) -> str: + parts: list[str] = [] + cur = expr + while cur.type == "MemberExpression": + if cur.computed or cur.property.type != "Identifier": + raise _DynamicValue("computed member access in interpolation") + parts.append(cur.property.name) + cur = cur.object + if cur.type != "Identifier": + raise _DynamicValue("non-identifier member root") + parts.append(cur.name) + return ".".join(reversed(parts)) + + def _filter_params(self, call) -> str: + chain = self._filter_call_chain(call) + if chain is None: + raise _DynamicValue("unsupported call in interpolation (not FILTER_PARAMS)") + cube, member = chain + sentinel = filter_param_sentinel(self._fp_counter) + self._fp_counter += 1 + arg = call.arguments[0] if call.arguments else None + ref = self._build_fp_ref(cube=cube, member=member, arg=arg, sentinel=sentinel) + self._fp_refs.append(ref) + return sentinel + + def _filter_call_chain(self, call) -> tuple[str, str] | None: + """If ``call`` is ``FILTER_PARAMS...filter(...)``, return + ``(cube, member)``; else ``None``.""" + callee = call.callee + if callee.type != "MemberExpression" or callee.computed: + return None + if callee.property.type != "Identifier" or callee.property.name != "filter": + return None + try: + path = self._member_path(callee.object).split(".") + except _DynamicValue: + return None + if len(path) != 3 or path[0] != "FILTER_PARAMS": + return None + return path[1], path[2] + + def _build_fp_ref(self, *, cube, member, arg, sentinel) -> CubeFilterParamRef: + if arg is None: + raise _DynamicValue("FILTER_PARAMS.filter() with no argument") + if arg.type == "Literal" and isinstance(arg.value, str): + return build_string_ref( + cube=cube, member=member, col_expr=arg.value, sentinel=sentinel) + if arg.type == "ArrowFunctionExpression": + segments = self._arrow_segments(arg) + return build_arrow_ref( + cube=cube, member=member, segments=segments, sentinel=sentinel) + raise _DynamicValue("unsupported FILTER_PARAMS argument") + + def _arrow_segments(self, arrow) -> list[tuple[str, str]]: + suffix_by_name: dict[str, str] = {} + for idx, param in enumerate(arrow.params): + if param.type != "Identifier" or idx not in _PARAM_SUFFIX: + raise _DynamicValue("unsupported FILTER_PARAMS arrow params") + suffix_by_name[param.name] = _PARAM_SUFFIX[idx] + segments: list[tuple[str, str]] = [] + self._collect_arrow_body(arrow.body, suffix_by_name, segments) + return segments + + def _collect_arrow_body(self, node, suffix_by_name, segments) -> None: + t = node.type + if t == "Literal" and isinstance(node.value, str): + segments.append(("lit", node.value)) + elif t == "Identifier": + segments.append(("param", self._param_suffix(node, suffix_by_name))) + elif t == "BinaryExpression" and node.operator == "+": + self._collect_arrow_body(node.left, suffix_by_name, segments) + self._collect_arrow_body(node.right, suffix_by_name, segments) + elif t == "TemplateLiteral": + self._collect_template_body(node, suffix_by_name, segments) + else: + raise _DynamicValue(f"unsupported FILTER_PARAMS arrow body ({t})") + + def _param_suffix(self, node, suffix_by_name) -> str: + if node.name not in suffix_by_name: + raise _DynamicValue(f"unknown arrow param '{node.name}'") + return suffix_by_name[node.name] + + def _collect_template_body(self, node, suffix_by_name, segments) -> None: + for i, quasi in enumerate(node.quasis): + if quasi.value.cooked: + segments.append(("lit", quasi.value.cooked)) + if i < len(node.expressions): + self._collect_arrow_body(node.expressions[i], suffix_by_name, segments) + + +# ── dict post-processing (shape normalisation) ────────────────────────────── + + +def _normalize_keys(d: dict, fields: set[str]) -> dict: + """camelCase → snake_case only when the snaked key is a known field for this + level; unknown keys pass through verbatim. ``meta`` is never touched.""" + out: dict = {} + for k, v in d.items(): + if k == "meta": + out[k] = v + continue + nk = _camel_to_snake(k) + out[nk if nk in fields else k] = v + return out + + +def _member_maps_to_lists(d: dict) -> None: + """Convert Cube's member maps (``{name: {...}}``) into the list-of-dicts + shape the Pydantic models expect, injecting ``name`` and normalising each + member's keys against its own field set.""" + for key, fields in _MEMBER_SECTIONS.items(): + m = d.get(key) + if isinstance(m, dict): + d[key] = _map_to_member_list(m, fields) + joins = d.get("joins") + if isinstance(joins, dict): + d["joins"] = _map_to_member_list(joins, _JOIN_FIELDS) + pre = d.get("pre_aggregations") + if isinstance(pre, dict): + d["pre_aggregations"] = [ + {"name": k, **(v if isinstance(v, dict) else {})} for k, v in pre.items() + ] + + +def _map_to_member_list(m: dict, fields: set[str]) -> list[dict]: + items: list[dict] = [] + for name, body in m.items(): + entry = _normalize_keys(body, fields) if isinstance(body, dict) else {} + entry["name"] = name + items.append(entry) + return items diff --git a/slayer/cube/models.py b/slayer/cube/models.py new file mode 100644 index 00000000..fa279042 --- /dev/null +++ b/slayer/cube/models.py @@ -0,0 +1,159 @@ +"""Pydantic v2 models for parsed Cube (Cube.js / Cube.dev) YAML objects. + +Lightweight representations of Cube's ``cubes:`` and ``views:`` YAML. We don't +depend on any Cube runtime — these shapes are populated directly from the YAML +by ``slayer.cube.parser`` and consumed by ``slayer.cube.converter``. + +Unknown keys are tolerated (Cube's schema evolves); Pydantic's default +``extra="ignore"`` drops them, except for the fields explicitly captured below +so the converter can either map them or stash the raw value under +``meta.cube_unmapped.`` (see the spec on DEV-1608, §7). +""" + +from typing import Any, Literal + +from pydantic import BaseModel, Field + + +class CubeFilterParamRef(BaseModel): + """A captured Cube ``FILTER_PARAMS...filter(...)`` occurrence + (DEV-1730). + + Front-ends (JS parser / YAML text scan) replace each occurrence with + ``sentinel`` in the surface text and record the structured ref here; the + converter resolves the sentinel to SLayer Mode-A text once requiredness is + known. ``body_template`` is the rendered inner SQL with ``{var}`` placeholders + (e.g. ``pr."BRAND" IN ({brand})`` or ``'{fulfillment_date_from}'``). + """ + + cube: str + member: str + kind: Literal["string", "arrow_value", "arrow_range"] + body_template: str + var_names: list[str] = Field(default_factory=list) + sentinel: str + + +class CubeMeasureFilter(BaseModel): + """One entry of a Cube measure ``filters:`` list — a conditional-aggregation + predicate (``{sql: "..."}``).""" + + sql: str + + +class CubeMeasure(BaseModel): + name: str + type: str # count, count_distinct, count_distinct_approx, sum, avg, min, max, + # number, string, time, boolean, number_agg (Tesseract) + sql: str | None = None + title: str | None = None + description: str | None = None + public: bool = True + meta: dict[str, Any] | None = None + format: Any | None = None # str ("percent"/"currency"/…) or object + filters: list[CubeMeasureFilter] = Field(default_factory=list) + drill_members: list[str] = Field(default_factory=list) + rolling_window: dict[str, Any] | None = None + # Multi-stage / Tesseract-leaning fields (mostly Stage-2 — see §9). + multi_stage: bool = False + time_shift: Any | None = None + grain: Any | None = None + filter: Any | None = None # Tesseract grain filter (exclude/keep_only/mode) + case: Any | None = None # Tesseract conditional measure (switch-keyed) + + +class CubeDimension(BaseModel): + name: str + sql: str | None = None + type: str = "string" # string, number, boolean, time, geo, switch (Tesseract) + title: str | None = None + description: str | None = None + public: bool = True + meta: dict[str, Any] | None = None + format: Any | None = None + primary_key: bool = False + sub_query: bool = False + case: dict[str, Any] | None = None # CASE-WHEN dimension (Stage 1) + granularities: list[dict[str, Any]] | None = None # custom time granularities + latitude: dict[str, Any] | None = None # geo + longitude: dict[str, Any] | None = None # geo + links: Any | None = None # presentation + order: str | None = None # presentation + + +class CubeJoin(BaseModel): + name: str # target cube name + relationship: str = "many_to_one" + sql: str # ON clause, e.g. "{CUBE}.customer_id = {customers.id}" + + +class CubeSegment(BaseModel): + name: str + sql: str + title: str | None = None + description: str | None = None + public: bool = True + meta: dict[str, Any] | None = None + + +class CubeCube(BaseModel): + name: str + sql_table: str | None = None + sql: str | None = None + sql_alias: str | None = None + extends: str | None = None + data_source: str | None = None + title: str | None = None + description: str | None = None + public: bool = True + meta: dict[str, Any] | None = None + refresh_key: dict[str, Any] | None = None + calendar: bool | None = None + measures: list[CubeMeasure] = Field(default_factory=list) + dimensions: list[CubeDimension] = Field(default_factory=list) + joins: list[CubeJoin] = Field(default_factory=list) + segments: list[CubeSegment] = Field(default_factory=list) + hierarchies: list[dict[str, Any]] | None = None + pre_aggregations: list[dict[str, Any]] | None = None + access_policy: list[dict[str, Any]] | None = None + # FILTER_PARAMS occurrences captured from this cube's raw-SQL surfaces + # (sentinels sit in the surface text). Populated by the front-end; resolved + # by the converter once requiredness is known (DEV-1730). + filter_params: list[CubeFilterParamRef] = Field(default_factory=list) + + +class CubeViewCubeRef(BaseModel): + """One entry of a view's ``cubes:`` list — a cube reached via ``join_path`` + contributing a set of members to the view. + + ``includes`` is a list of member names, the string ``"*"``, or Cube's + per-member override object form (``[{name, alias, title, format, meta}, …]``). + The converter extracts the member names and reports per-member overrides as + unsupported (Stage 1) rather than silently dropping them. + """ + + join_path: str + includes: list[str | dict[str, Any]] | str | None = None + excludes: list[str] = Field(default_factory=list) + prefix: bool = False + alias: str | None = None # renames the cube for member prefixing + + +class CubeView(BaseModel): + name: str + cubes: list[CubeViewCubeRef] = Field(default_factory=list) + extends: str | None = None + title: str | None = None + description: str | None = None + public: bool = True + meta: dict[str, Any] | None = None + folders: list[dict[str, Any]] | None = None + default_filters: list[dict[str, Any]] | None = None + access_policy: list[dict[str, Any]] | None = None + + +class CubeProject(BaseModel): + """Aggregated result of parsing all YAML files in a Cube project.""" + + cubes: list[CubeCube] = Field(default_factory=list) + views: list[CubeView] = Field(default_factory=list) diff --git a/slayer/cube/parser.py b/slayer/cube/parser.py new file mode 100644 index 00000000..455dccaa --- /dev/null +++ b/slayer/cube/parser.py @@ -0,0 +1,190 @@ +"""Parse a Cube project directory into a CubeProject. + +Walks the directory for ``*.yml`` / ``*.yaml``, extracts top-level ``cubes:`` +and ``views:``, and surfaces Jinja-templated files/members + malformed files as +report issues. See DEV-1608 §2. +""" + +import logging +import os + +import yaml + +from slayer.cube.models import CubeCube, CubeProject, CubeView +from slayer.cube.refs import contains_jinja +from slayer.cube.report import CubeConversionIssue, CubeIssueCategory + +logger = logging.getLogger(__name__) + + +def _collect_paths(directory: str, suffixes: tuple[str, ...]) -> list[str]: + paths: list[str] = [] + for root, dirs, files in os.walk(directory): + dirs[:] = [d for d in dirs + if not d.startswith(".") and d not in ("target", "node_modules")] + for filename in sorted(files): + if filename.startswith("."): + continue + if filename.endswith(suffixes): + paths.append(os.path.join(root, filename)) + return paths + + +def _collect_yaml_paths(directory: str) -> list[str]: + return _collect_paths(directory, (".yaml", ".yml")) + + +def _str_has_jinja(value) -> bool: + return isinstance(value, str) and contains_jinja(value) + + +def _member_has_jinja(item) -> bool: + """True if any templatable field of a member (``sql`` / ``filter`` / + measure ``filters`` / ``case`` predicates) contains Jinja.""" + if not isinstance(item, dict): + return False + if _str_has_jinja(item.get("sql")) or _str_has_jinja(item.get("filter")): + return True + if any(_str_has_jinja((f or {}).get("sql")) for f in item.get("filters") or []): + return True + case = item.get("case") + if isinstance(case, dict): + return any(_str_has_jinja((w or {}).get("sql")) for w in case.get("when") or []) + return False + + +def _filter_jinja(items: list, *, cube_name, path, issues, has_jinja) -> list: + kept = [] + for item in items: + if has_jinja(item): + issues.append(CubeConversionIssue( + category=CubeIssueCategory.REQUIRES_TEMPLATING, severity="warning", + cube=cube_name, + member=item.get("name") if isinstance(item, dict) else None, + message=f"Member in '{path}' uses Jinja templating; skipped.", + )) + else: + kept.append(item) + return kept + + +def _strip_jinja_members(raw_cube: dict, issues: list, path: str) -> None: + """Drop members whose templatable fields contain Jinja, reporting each.""" + cube_name = raw_cube.get("name") + for key in ("dimensions", "measures", "segments"): + items = raw_cube.get(key) + if isinstance(items, list): + raw_cube[key] = _filter_jinja( + items, cube_name=cube_name, path=path, issues=issues, + has_jinja=_member_has_jinja) + joins = raw_cube.get("joins") + if isinstance(joins, list): + raw_cube["joins"] = _filter_jinja( + joins, cube_name=cube_name, path=path, issues=issues, + has_jinja=lambda j: _str_has_jinja((j or {}).get("sql"))) + + +def _as_list(value) -> list: + if value is None: + return [] + return value if isinstance(value, list) else [value] + + +def _load_yaml(path: str, issues: list): + try: + with open(path, encoding="utf-8") as fh: + raw_text = fh.read() + except (OSError, UnicodeDecodeError) as exc: + # UnicodeDecodeError (invalid UTF-8) is a ValueError, not an OSError — + # catch it too so a single bad file becomes a PARSE_ERROR warning + # instead of aborting the whole import. + issues.append(CubeConversionIssue( + category=CubeIssueCategory.PARSE_ERROR, severity="warning", + message=f"File '{path}' could not be read: {exc}", + )) + return None + try: + return yaml.safe_load(raw_text) + except yaml.YAMLError: + templated = contains_jinja(raw_text) + category = (CubeIssueCategory.REQUIRES_TEMPLATING if templated + else CubeIssueCategory.PARSE_ERROR) + issues.append(CubeConversionIssue( + category=category, severity="warning", + message=f"File '{path}' could not be parsed as YAML" + + (" (Jinja templating)." if templated else "."), + )) + return None + + +def _parse_cubes(data: dict, path: str, cubes: list, issues: list) -> None: + for raw_cube in _as_list(data.get("cubes")): + if not isinstance(raw_cube, dict): + continue + _strip_jinja_members(raw_cube, issues, path) + try: + cubes.append(CubeCube.model_validate(raw_cube)) + except Exception as exc: # noqa: BLE001 — keep parsing the rest + issues.append(CubeConversionIssue( + category=CubeIssueCategory.PARSE_ERROR, severity="warning", + cube=raw_cube.get("name"), + message=f"Failed to parse cube in '{path}': {exc}", + )) + + +def _parse_views(data: dict, path: str, views: list, issues: list) -> None: + for raw_view in _as_list(data.get("views")): + if not isinstance(raw_view, dict): + continue + try: + views.append(CubeView.model_validate(raw_view)) + except Exception as exc: # noqa: BLE001 — keep parsing the rest + issues.append(CubeConversionIssue( + category=CubeIssueCategory.PARSE_ERROR, severity="warning", + view=raw_view.get("name"), + message=f"Failed to parse view in '{path}': {exc}", + )) + + +def parse_cube_project(project_path: str) -> tuple[CubeProject, list[CubeConversionIssue]]: + """Parse a Cube project directory. + + Returns the parsed ``CubeProject`` plus parse-time issues + (``requires_templating`` for Jinja, ``parse_error`` for malformed files). + """ + cubes: list[CubeCube] = [] + views: list[CubeView] = [] + issues: list[CubeConversionIssue] = [] + + for path in _collect_yaml_paths(project_path): + data = _load_yaml(path, issues) + if not isinstance(data, dict): + continue + _parse_cubes(data, path, cubes, issues) + _parse_views(data, path, views, issues) + + _parse_js_files(project_path, cubes, views, issues) + + return CubeProject(cubes=cubes, views=views), issues + + +def _parse_js_files(project_path: str, cubes: list, views: list, issues: list) -> None: + """Discover + parse ``.js`` Cube configs (DEV-1730), merging cubes / views / + issues into the same aggregation as the YAML front-end.""" + from slayer.cube.js_parser import parse_cube_js # local: keeps esprima lazy + + for path in _collect_paths(project_path, (".js",)): + try: + with open(path, encoding="utf-8") as fh: + source = fh.read() + except (OSError, UnicodeDecodeError) as exc: + # UnicodeDecodeError (invalid UTF-8) subclasses ValueError, not + # OSError — catch it too so one bad file doesn't abort the import. + issues.append(CubeConversionIssue( + category=CubeIssueCategory.PARSE_ERROR, severity="warning", + message=f"File '{path}' could not be read: {exc}")) + continue + result = parse_cube_js(source, path=path) + cubes.extend(result.cubes) + views.extend(result.views) + issues.extend(result.issues) diff --git a/slayer/cube/refs.py b/slayer/cube/refs.py new file mode 100644 index 00000000..a149380b --- /dev/null +++ b/slayer/cube/refs.py @@ -0,0 +1,129 @@ +"""Translate Cube curly references to SLayer SQL (Mode A) / DSL (Mode B). + +Cube `sql`/`filter` strings use `{CUBE}`, `{member}`, `{cube.member}` — single +braces, distinct from Jinja's `{{ }}` / `{% %}` (which are detected and skipped +upstream). See DEV-1608 §3. +""" + +import re + +_JINJA_RE = re.compile(r"\{\{|\{%|%\}|\}\}") +_LITERAL_RE = re.compile(r"'(?:''|[^'])*'") # SQL string literal, doubled-quote aware +_REF_RE = re.compile(r"\{([^{}]+)\}") + +# Operand forms accepted inside a join ON clause. +_OPERAND_BRACE_DOT = re.compile(r"^\{([A-Za-z_]\w*)\}\.(\w+)$") # {CUBE}.col +_OPERAND_BRACED = re.compile(r"^\{([^{}]+)\}$") # {cube.col} +# `\bAND\b` (no surrounding `\s+` quantifiers) avoids the polynomial-backtracking +# shape Sonar S5852 flags; operands are whitespace-stripped after the split. +_AND_SPLIT = re.compile(r"\bAND\b", re.IGNORECASE) +_IDENTIFIER_RE = re.compile(r"^[A-Za-z_]\w*$") + + +def contains_jinja(text: str) -> bool: + """True if ``text`` contains Jinja markers (`{{ }}` or `{% %}`). + + Cube's own `{CUBE}` / `{cube.member}` single-brace refs are NOT Jinja. + """ + return bool(_JINJA_RE.search(text)) + + +def translate_cube_refs(text: str, *, mode: str, cube: str | None = None) -> str: + """Translate Cube curly refs in ``text`` (string literals are left intact). + + - ``{CUBE}.col`` → ``col`` (SLayer auto-qualifies bare names) + - bare ``{CUBE}`` → the cube name (table reference) + - ``{member}`` → ``member`` (same-cube sibling) + - ``{cube.member}`` / ``{a.b.c}`` → ``cube.member`` / ``a.b.c`` (dotted) + + ``mode`` (``"sql"``/``"dsl"``) documents the intended target layer for the + caller; it is validated but does not change the syntactic rewrite. + """ + if mode not in ("sql", "dsl"): + raise ValueError(f"mode must be 'sql' or 'dsl', got {mode!r}") + literal_spans = [m.span() for m in _LITERAL_RE.finditer(text)] + + def _in_literal(pos: int) -> bool: + return any(s <= pos < e for s, e in literal_spans) + + out: list[str] = [] + last = 0 + for m in _REF_RE.finditer(text): + if m.start() < last or _in_literal(m.start()): + continue + out.append(text[last:m.start()]) + next_char = text[m.end()] if m.end() < len(text) else "" + replacement, extra = _resolve_ref(m.group(1).strip(), next_char, cube) + out.append(replacement) + last = m.end() + extra + out.append(text[last:]) + return "".join(out) + + +def _resolve_ref(inner: str, next_char: str, cube: str | None) -> tuple[str, int]: + """Resolve one ``{ref}`` to ``(replacement, extra_chars_consumed)``.""" + if inner == "CUBE": + if next_char == ".": + return "", 1 # `{CUBE}.col` → drop the `{CUBE}` and the following dot + return cube or "", 0 + return inner, 0 # `{member}` / `{a.b}` → inner verbatim + + +def _operand_ref(operand: str) -> tuple[str, str] | None: + """Parse one side of an equality into ``(qualifier, column)``. + + Returns ``None`` for anything that isn't a bare Cube column reference + (function calls, arithmetic, literals) — those can't go in ``join_pairs``. + """ + operand = operand.strip() + m = _OPERAND_BRACE_DOT.match(operand) + if m: + return m.group(1), m.group(2) + m = _OPERAND_BRACED.match(operand) + if m: + parts = m.group(1).strip().split(".") + if len(parts) >= 2: + return parts[0], ".".join(parts[1:]) + return None + + +def parse_join_on(on_sql: str, *, source_cube: str, target_cube: str) -> list[list[str]] | None: + """Parse a Cube join ON clause into SLayer ``join_pairs``. + + Returns ``[[src_col, tgt_col], ...]`` for an equality (or AND-conjunction of + equalities); ``None`` for any non-equality / non-column ON. The column names + are the *member* names as written — the converter resolves them to physical + columns (and drops the join if a member's sql is non-trivial). + """ + pairs: list[list[str]] = [] + for part in _AND_SPLIT.split(on_sql.strip()): + pair = _equality_pair(part, source_cube=source_cube, target_cube=target_cube) + if pair is None: + return None + pairs.append(pair) + return pairs or None + + +def _equality_pair(part: str, *, source_cube: str, target_cube: str) -> list[str] | None: + """Parse one ``A = B`` equality into ``[src_col, tgt_col]`` or ``None``.""" + if part.count("=") != 1 or any(op in part for op in ("<", ">", "!")): + return None + left, right = part.split("=") + lhs = _operand_ref(left) + rhs = _operand_ref(right) + if lhs is None or rhs is None: + return None + src_col = tgt_col = None + for qualifier, col in (lhs, rhs): + if qualifier in ("CUBE", source_cube): + src_col = col + elif qualifier == target_cube: + tgt_col = col + if src_col is None or tgt_col is None: + return None + return [src_col, tgt_col] + + +def is_bare_identifier(sql: str) -> bool: + """True if ``sql`` is a single bare column identifier (usable in join_pairs).""" + return bool(_IDENTIFIER_RE.match(sql.strip())) diff --git a/slayer/cube/report.py b/slayer/cube/report.py new file mode 100644 index 00000000..70189be8 --- /dev/null +++ b/slayer/cube/report.py @@ -0,0 +1,85 @@ +"""Structured report for Cube → SLayer conversion (DEV-1608, §10). + +Everything the converter cannot map cleanly is recorded as a +``CubeConversionIssue`` rather than silently dropped or raised. The full +``CubeConversionResult`` (models + report) is returned by the converter and the +report is also written to JSON by the CLI. +""" + +from enum import Enum +from typing import Literal + +from pydantic import BaseModel, Field + +from slayer.core.models import SlayerModel + + +class CubeIssueCategory(str, Enum): + REQUIRES_TEMPLATING = "requires_templating" + PARSE_ERROR = "parse_error" + COMPLEX_SQL = "complex_sql" + COMPLEX_MEASURE = "complex_measure" + LOSSY_MAPPING = "lossy_mapping" + UNSUPPORTED_JOIN = "unsupported_join" + UNSUPPORTED_ROLLING_WINDOW = "unsupported_rolling_window" + UNSUPPORTED_FORMAT = "unsupported_format" + UNSUPPORTED_DEFAULT_FILTER = "unsupported_default_filter" + SEGMENT_AS_COLUMN = "segment_as_column" + UNMAPPED_INFRA = "unmapped_infra" + GEO_UNMAPPED = "geo_unmapped" + SUBQUERY_UNMAPPED = "subquery_unmapped" + GRANULARITY_UNMAPPED = "granularity_unmapped" + DISCONNECTED_VIEW = "disconnected_view" + AMBIGUOUS_VIEW_ROOT = "ambiguous_view_root" + VIEW_FANOUT_RISK = "view_fanout_risk" + FOLDERS_UNMAPPED = "folders_unmapped" + EXTENDS_CYCLE = "extends_cycle" + NO_SOURCE = "no_source" + DEFERRED_STAGE2 = "deferred_stage2" + SAVE_FAILED = "save_failed" + FILTER_PARAMS_VARIABLE = "filter_params_variable" + FILTER_PARAMS_UNSUPPORTED = "filter_params_unsupported" + + +Severity = Literal["info", "warning", "error"] + + +class CubeConversionIssue(BaseModel): + category: CubeIssueCategory + severity: Severity = "warning" + cube: str | None = None + view: str | None = None + member: str | None = None + message: str + raw: str | None = None # raw Cube fragment when useful + + @property + def context(self) -> str: + return self.cube or self.view or self.member or "general" + + +class CubeConversionReport(BaseModel): + issues: list[CubeConversionIssue] = Field(default_factory=list) + model_count: int = 0 + hidden_count: int = 0 + view_count: int = 0 + + def add(self, issue: CubeConversionIssue) -> None: + self.issues.append(issue) + + def by_category(self, category: CubeIssueCategory) -> list[CubeConversionIssue]: + return [i for i in self.issues if i.category == category] + + def by_severity(self, severity: Severity) -> list[CubeConversionIssue]: + return [i for i in self.issues if i.severity == severity] + + @property + def has_errors(self) -> bool: + return any(i.severity == "error" for i in self.issues) + + +class CubeConversionResult(BaseModel): + """Return value of ``CubeToSlayerConverter.convert``.""" + + models: list[SlayerModel] = Field(default_factory=list) + report: CubeConversionReport = Field(default_factory=CubeConversionReport) diff --git a/slayer/engine/query_engine.py b/slayer/engine/query_engine.py index 679acdc4..8f2c0fc9 100644 --- a/slayer/engine/query_engine.py +++ b/slayer/engine/query_engine.py @@ -35,7 +35,11 @@ ColumnRef, SlayerQuery, TimeDimension, + _contains_block_delimiter, + coerce_declared_list_variables, + declares_variables, extract_placeholder_names, + list_valued_variable_names, substitute_variables, ) from slayer.core.recommend import ( @@ -67,7 +71,7 @@ resolve_entity, ) from slayer.sql.client import SlayerSQLClient -from slayer.sql.dialects import dialect_for_ds_type, get_dialect +from slayer.sql.dialects import SqlDialect, dialect_for_ds_type, get_dialect from slayer.sql import engine_factory from slayer.sql.engine_factory import _runtime_fingerprint from slayer.sql.generator import SQLGenerator @@ -260,26 +264,86 @@ def _merge_query_variables( return {**(outer or {}), **(stage or {}), **(runtime or {})} +def _model_has_optional_block(model: SlayerModel) -> bool: + """True if any Mode-A surface carries an optional ``{? ... ?}`` block. + + Such a model must run through substitution even with zero variables so its + blocks collapse to ``(1=1)`` (DEV-1730) instead of leaking the raw ``{?`` + delimiters into emitted SQL. + """ + surfaces = [model.sql, *(model.filters or [])] + for col in model.columns: + surfaces.append(col.sql) + surfaces.append(col.filter) + return any(s and _contains_block_delimiter(s) for s in surfaces) + + +def _model_needs_substitution_pass(model: SlayerModel) -> bool: + """True if substitution must run even when NO variables are supplied. + + Two independent reasons, both of which defeat the DEV-1625 zero-variable + fast path (which exists so raw brace literals like a Postgres array + ``'{1,2,3}'`` survive verbatim in models that don't use variables): + + - an optional ``{? ... ?}`` block must collapse to ``(1=1)`` rather than + leak its delimiters into the SQL (DEV-1730), and + - the model DECLARES its variables, so it is importer-generated and has no + brace-literal ambiguity to protect — skipping the pass would emit a bare + ``{var}`` into the SQL instead of raising the documented missing-variable + error. This closes the fast-path hole for a generated model whose + pushdowns are all required (no block to force the pass). + """ + return _model_has_optional_block(model) or declares_variables(model) + + def _substitute_model_sql_surfaces( - *, model: SlayerModel, variables: dict[str, Any] + *, model: SlayerModel, variables: dict[str, Any], dialect: SqlDialect ) -> SlayerModel: """Return a copy of ``model`` with ``{var}`` substituted into the four Mode-A (raw-SQL) surfaces: ``SlayerModel.sql``, ``SlayerModel.filters``, ``Column.sql``, and ``Column.filter`` (DEV-1625). - No-op copy when ``variables`` is empty — so a model that uses no variables - is never touched and raw brace literals (e.g. Postgres arrays ``'{1,2,3}'``) - survive verbatim. Uses the hardened :func:`substitute_variables` - (raise-on-missing, string single-quote escaping). Never mutates the input - model — it may be a shared cached object. Only these four surfaces change; - Mode-B surfaces (``ModelMeasure.formula`` etc.) are copied through as-is. + ``dialect`` is REQUIRED (DEV-1727 fail-closed): the Mode-A escaping regime + is dialect-aware, and deriving ``backslash_escapes`` from the resolved + dialect here — rather than accepting an optional bool — means no caller can + render raw SQL for a backslash dialect (MySQL/ClickHouse/...) while silently + under-escaping the value. + + No-op copy when ``variables`` is empty AND the model needs no pass of its + own (see :func:`_model_needs_substitution_pass`) — so a model that uses no + variables is never touched and raw brace literals (e.g. Postgres arrays + ``'{1,2,3}'``) survive verbatim. A block-bearing or variable-declaring + model, however, must still run — so its blocks collapse to ``(1=1)``, and a + declared-but-missing variable raises, even on a zero-variable call + (DEV-1730). Uses the hardened + :func:`substitute_variables` (raise-on-missing, dialect-aware escaping, block + collapse). Never mutates the input model — it may be a shared cached object. + Only these four surfaces change; Mode-B surfaces (``ModelMeasure.formula`` + etc.) are copied through as-is. + + A scalar passed for a variable the model DECLARES list-valued (an importer's + generated ``col IN ({var})`` template) is wrapped to a one-element list + first — see :func:`coerce_declared_list_variables`. This is the one + substitution choke point for Mode-A model surfaces (execution and the + type-probe both route through here), so the coercion cannot be bypassed. """ - if not variables: + if not variables and not _model_needs_substitution_pass(model): return model + variables = coerce_declared_list_variables( + variables, list_valued=list_valued_variable_names(model) + ) + backslash_escapes = dialect.backslash_escapes_strings + def _sub(text: str) -> str: - # Mode-A surfaces are parsed by sqlglot → escape string values SQL-style. - return substitute_variables(filter_str=text, variables=variables, escape="sql") + # Mode-A surfaces are parsed by sqlglot → escape string values SQL-style + # in the target dialect's regime. + return substitute_variables( + filter_str=text, + variables=variables, + escape="sql", + backslash_escapes=backslash_escapes, + ) new_columns = [] for col in model.columns: @@ -298,18 +362,22 @@ def _sub(text: str) -> str: return model.model_copy(update=model_updates) -def _render_probe_model(model: SlayerModel) -> SlayerModel: +def _render_probe_model(model: SlayerModel, *, dialect: SqlDialect) -> SlayerModel: """Substitute a template model's OWN ``query_variables`` defaults into its Mode-A surfaces for type-probing (DEV-1625). - Defaults only — the probe has no caller variables and must honour the - no-dummy-fill decision. A rendered virtual model (``source_model_origin`` - set) is returned unchanged. An undefaulted ``{var}`` raises here and is - caught by the probe's graceful-``{}`` handler in ``get_column_types``. + ``dialect`` is REQUIRED (DEV-1727 fail-closed) and threads into the + dialect-aware Mode-A escaping. Defaults only — the probe has no caller + variables and must honour the no-dummy-fill decision. A rendered virtual + model (``source_model_origin`` set) is returned unchanged. An undefaulted + ``{var}`` raises here and is caught by the probe's graceful-``{}`` handler + in ``get_column_types``. """ - if model.source_model_origin is None and model.query_variables: + if model.source_model_origin is None and ( + model.query_variables or _model_needs_substitution_pass(model) + ): return _substitute_model_sql_surfaces( - model=model, variables=model.query_variables + model=model, variables=model.query_variables, dialect=dialect ) return model @@ -1114,10 +1182,18 @@ async def _prepare_pipeline( # model's own defaults are the lowest layer. if model.source_model_origin is None: effective_vars = {**(model.query_variables or {}), **(query.variables or {})} - if effective_vars: - model = _substitute_model_sql_surfaces( - model=model, variables=effective_vars - ) + # Always call: the helper is a no-op for a variable-free, block-free + # model (preserving DEV-1625 raw-brace-literal protection), but a + # block-bearing model must still run so its {? ?} blocks collapse to + # (1=1) even on a zero-variable call (DEV-1730). DEV-1727: escaping is + # dialect-aware — pass the resolved datasource's dialect so backslash + # dialects (MySQL/ClickHouse/…) get the hardened regime, standard + # dialects the '' doubling. + model = _substitute_model_sql_surfaces( + model=model, + variables=effective_vars, + dialect=dialect_for_ds_type(datasource.type), + ) # Enrich: SlayerQuery + model → EnrichedQuery _t = timing.start() @@ -1930,8 +2006,13 @@ async def get_column_types( # rendered (from its own query_variables defaults) before probe SQL # is generated; an undefaulted {var} raises here and degrades to {} # via the surrounding except, so partially-defaulted models stay safe. + # DEV-1727: pass the resolved datasource's dialect so probe-SQL + # escaping matches the backend that parses it. enriched = await self._enrich( - query=probe_query, model=_render_probe_model(model) + query=probe_query, + model=_render_probe_model( + model, dialect=dialect_for_ds_type(datasource.type) + ), ) dialect = self._dialect_for_type(datasource.type) generator = SQLGenerator(dialect=dialect) diff --git a/slayer/inspect/model_render.py b/slayer/inspect/model_render.py index 5c48a89d..27f100f5 100644 --- a/slayer/inspect/model_render.py +++ b/slayer/inspect/model_render.py @@ -20,7 +20,7 @@ from slayer.core.enums import DataType from slayer.core.models import Column, SlayerModel -from slayer.core.query import SlayerQuery +from slayer.core.query import SlayerQuery, extract_model_variables from slayer.engine.ingestion import _friendly_db_error from slayer.engine.profiling import ( _is_sample_cached, @@ -598,7 +598,9 @@ def model_skeleton_fields( """Cheap, DB-free structured skeleton of a model. Shape: ``{name, canonical_id, description, column_names, measure_names, - aggregation_names, joins_to}``. Used by ``inspect(model, compact=True)`` + aggregation_names, joins_to, variables}`` — ``variables`` is + ``{required, optional}`` (DEV-1730), the Mode-A ``{var}`` / ``{? ?}`` + placeholders classified structurally. Used by ``inspect(model, compact=True)`` JSON and by each entry of ``inspect(datasource, compact=False)``'s ``models`` list (DEV-1588). ``description`` is truncated by ``max_chars``; ``canonical_id`` falls back to the bare name when ``data_source`` is unset @@ -607,6 +609,7 @@ def model_skeleton_fields( canonical_id = ( f"{model.data_source}.{model.name}" if model.data_source else model.name ) + mv = extract_model_variables(model) return { "name": model.name, "canonical_id": canonical_id, @@ -615,6 +618,7 @@ def model_skeleton_fields( "measure_names": [m.name for m in model.measures if m.name is not None], "aggregation_names": [a.name for a in model.aggregations], "joins_to": sorted({j.target_model for j in model.joins}), + "variables": {"required": mv.required, "optional": mv.optional}, } @@ -630,7 +634,9 @@ def render_model_skeleton( An optional truncated description line (only when set), then four lines — ``Columns`` / ``Measures`` / ``Aggregations`` / ``Joins to`` — always present, each empty value rendered ``_(none)_`` (aligned to - ``models_summary(compact)``). The caller prepends the ``#``/``##`` heading. + ``models_summary(compact)``), plus a fifth ``Variables`` line when the model + is parameterised with Mode-A ``{var}`` / ``{? ?}`` placeholders (DEV-1730). + The caller prepends the ``#``/``##`` heading. """ fields = model_skeleton_fields(model=model, max_chars=max_chars) lines: list[str] = [] @@ -640,9 +646,25 @@ def render_model_skeleton( lines.append(f"Measures: {_skeleton_csv(fields['measure_names'])}") lines.append(f"Aggregations: {_skeleton_csv(fields['aggregation_names'])}") lines.append(f"Joins to: {_skeleton_csv(fields['joins_to'])}") + var_line = _render_variables_line(fields["variables"]) + if var_line: + lines.append(var_line) return "\n".join(lines) +def _render_variables_line(variables: dict[str, list[str]]) -> str | None: + """Render the model-variable line for sql-mode models parameterised with + ``{var}`` / ``{? ?}`` (DEV-1730). Returns ``None`` when the model takes no + variables so a plain table-backed model's skeleton is unchanged. + """ + required = variables.get("required") or [] + optional = variables.get("optional") or [] + if not required and not optional: + return None + parts = [f"{name} (required)" for name in required] + list(optional) + return f"Variables: {', '.join(parts)}" + + async def render_model_inspection( # NOSONAR(S3776) — faithful extraction of the inspect_model tool body; the section-gating + cache-miss + dual markdown/json render is intentionally a single linear pass *, model: SlayerModel, diff --git a/slayer/sql/dialects/base.py b/slayer/sql/dialects/base.py index 00e1a57d..7af32f60 100644 --- a/slayer/sql/dialects/base.py +++ b/slayer/sql/dialects/base.py @@ -12,11 +12,13 @@ from __future__ import annotations +from functools import lru_cache from typing import TYPE_CHECKING, Any from collections.abc import Callable from pydantic import BaseModel, ConfigDict from sqlglot import exp +from sqlglot.dialects.dialect import Dialect as _SqlglotDialect from slayer.core.enums import TimeGranularity @@ -142,6 +144,29 @@ def _build_covar_decomposition( # --------------------------------------------------------------------------- +@lru_cache(maxsize=None) +def _sqlglot_backslash_escapes(sqlglot_name: str) -> bool: + """Whether ``sqlglot``'s tokenizer for ``sqlglot_name`` treats a backslash + as a string-literal escape character (DEV-1727). + + This is the single source of truth for the Mode-A ``{var}`` escaping regime: + deriving it from the same tokenizer that later PARSES the substituted SQL + means our escaping can never drift from the parser. ``STRING_ESCAPES`` is a + semi-internal sqlglot attribute; guard it so a future sqlglot change that + renames/reshapes it fails loudly here rather than silently mis-escaping. + """ + tokenizer = _SqlglotDialect.get_or_raise(sqlglot_name).tokenizer_class + escapes = getattr(tokenizer, "STRING_ESCAPES", None) + if not isinstance(escapes, (list, tuple, set, frozenset)): + raise RuntimeError( + f"Cannot derive the backslash-escaping regime for sqlglot dialect " + f"{sqlglot_name!r}: its tokenizer's STRING_ESCAPES is " + f"{type(escapes).__name__}, expected a collection of strings. A " + f"sqlglot upgrade may have changed this internal API (DEV-1727)." + ) + return "\\" in escapes + + class SqlDialect(BaseModel): """Strategy class encapsulating one database's SQL-generation quirks. @@ -159,6 +184,21 @@ class SqlDialect(BaseModel): log10_native: bool = True log2_native: bool = True + @property + def backslash_escapes_strings(self) -> bool: + """Whether this dialect's string literals treat a backslash as an escape + character (MySQL/ClickHouse/Snowflake/Redshift/BigQuery/Databricks/Spark) + rather than an ordinary char (SQLite/Postgres/DuckDB/T-SQL/Trino/Presto/ + Oracle). + + Drives DEV-1727 dialect-aware Mode-A ``{var}`` escaping: pass this to + ``substitute_variables(..., backslash_escapes=...)`` so a value like + ``a\\'b`` stays inside its quoted literal on every backend. Derived from + sqlglot's tokenizer (see :func:`_sqlglot_backslash_escapes`) so it can't + disagree with the parser; a pinning test freezes the expected value. + """ + return _sqlglot_backslash_escapes(self.sqlglot_name) + # ------------------------------------------------------------------ # Date-trunc / time arithmetic # ------------------------------------------------------------------ diff --git a/sonar-project.properties b/sonar-project.properties index 3335f6ce..e7c42511 100644 --- a/sonar-project.properties +++ b/sonar-project.properties @@ -16,3 +16,12 @@ sonar.organization=motleyai # shared bits into a helper would tangle dialect concerns without changing # the per-dialect specifics. Suppress at the file level. sonar.cpd.exclusions=tests/integration/test_integration_mysql.py,tests/integration/test_integration_clickhouse.py,tests/integration/test_integration_sqlserver.py + +# DEV-1730: the Cube-JS import fixtures under tests/fixtures/cube_js/ are Cube +# FILTER_PARAMS DSL, not executable JavaScript. SonarJS misreads the +# `${FILTER_PARAMS...filter((from, to) => ...)}` arrows inside SQL template +# literals (e.g. S7770), and a NOSONAR comment can't live inside the SQL string. +# Suppress JS issue analysis on that fixture tree. +sonar.issue.ignore.multicriteria=cubejs_fixtures +sonar.issue.ignore.multicriteria.cubejs_fixtures.ruleKey=javascript:* +sonar.issue.ignore.multicriteria.cubejs_fixtures.resourceKey=tests/fixtures/cube_js/** diff --git a/tests/dialects/test_base.py b/tests/dialects/test_base.py index e8035c61..36e58288 100644 --- a/tests/dialects/test_base.py +++ b/tests/dialects/test_base.py @@ -458,3 +458,100 @@ def test_duckdb_and_sqlite_rewrite_target_ast_leave_round_uncast() -> None: tree = sqlglot.parse_one("ROUND(x, 2)", dialect="postgres") out = d.rewrite_target_ast(tree).sql(dialect=d.sqlglot_name).upper() assert "CAST(" not in out + + +# --------------------------------------------------------------------------- +# backslash_escapes_strings — DEV-1727 dialect-aware Mode-A {var} escaping +# --------------------------------------------------------------------------- + +from slayer.sql.dialects import ( # noqa: E402 + ClickhouseDialect, + DuckdbDialect, + MysqlDialect, + PostgresDialect, + SnowflakeDialect, + SqliteDialect, + TsqlDialect, +) +from slayer.sql.dialects._tier2 import ( # noqa: E402 + DatabricksDialect, + PrestoDialect, + RedshiftDialect, + SparkDialect, + TrinoDialect, +) + +# Pin the expected backslash-escaping regime for every dialect class. This is +# the single source of truth the DEV-1727 escaping keys off, DERIVED from +# sqlglot's tokenizer (SqlDialect.backslash_escapes_strings). Freezing the +# expected value here means a sqlglot upgrade that shifts a dialect's string +# escaping fails THIS test loudly for review rather than silently changing how +# {var} values are escaped in generated SQL. +# +# Standard (backslash is an ordinary literal char → only '' doubling): +# sqlite, postgres, duckdb, tsql, trino, presto, oracle +# Backslash-escaping (a backslash escapes the next char in a string literal): +# mysql, clickhouse, snowflake, redshift, bigquery, databricks, spark +_BACKSLASH_ESCAPES_PINS = [ + (SqliteDialect(), False), + (PostgresDialect(), False), + (DuckdbDialect(), False), + (TsqlDialect(), False), + (TrinoDialect(), False), + (PrestoDialect(), False), + (OracleDialect(), False), + (MysqlDialect(), True), + (ClickhouseDialect(), True), + (SnowflakeDialect(), True), + (RedshiftDialect(), True), + (BigqueryDialect(), True), + (DatabricksDialect(), True), + (SparkDialect(), True), +] + + +@pytest.mark.parametrize( + "dialect,expected", + _BACKSLASH_ESCAPES_PINS, + ids=[d.sqlglot_name for d, _ in _BACKSLASH_ESCAPES_PINS], +) +def test_backslash_escapes_strings_pin(dialect: SqlDialect, expected: bool) -> None: + assert dialect.backslash_escapes_strings is expected + + +def test_backslash_escapes_strings_matches_sqlglot_for_all_dialects() -> None: + """The property must agree with sqlglot's own tokenizer table for every + registered dialect — the derivation and the parser can never disagree.""" + from sqlglot.dialects.dialect import Dialect + + from slayer.sql.dialects import _ALL_DIALECTS + + for d in _ALL_DIALECTS: + expected = "\\" in Dialect.get_or_raise(d.sqlglot_name).tokenizer_class.STRING_ESCAPES + assert d.backslash_escapes_strings is expected, d.sqlglot_name + + +def test_backslash_escapes_strings_is_bool() -> None: + # Contract: a plain bool (not a truthy set/other), so callers can pass it + # straight into substitute_variables(backslash_escapes=...). + assert isinstance(MysqlDialect().backslash_escapes_strings, bool) + assert isinstance(SqliteDialect().backslash_escapes_strings, bool) + + +def test_backslash_escapes_derivation_guards_missing_string_escapes(monkeypatch) -> None: + """If a future sqlglot reshapes/removes the tokenizer's STRING_ESCAPES + attribute, the derivation must raise a clear RuntimeError (not silently + mis-escape or blow up with an obscure error).""" + from sqlglot.dialects.dialect import Dialect + + from slayer.sql.dialects import base as base_mod + + base_mod._sqlglot_backslash_escapes.cache_clear() + tokenizer_cls = Dialect.get_or_raise("postgres").tokenizer_class + # Simulate the attribute changing shape (a str is not a collection). + monkeypatch.setattr(tokenizer_cls, "STRING_ESCAPES", "not-a-collection", raising=False) + try: + with pytest.raises(RuntimeError, match="STRING_ESCAPES"): + base_mod._sqlglot_backslash_escapes("postgres") + finally: + base_mod._sqlglot_backslash_escapes.cache_clear() diff --git a/tests/fixtures/cube_js/rr_drivers.js b/tests/fixtures/cube_js/rr_drivers.js new file mode 100644 index 00000000..73bfa9d8 --- /dev/null +++ b/tests/fixtures/cube_js/rr_drivers.js @@ -0,0 +1,193 @@ +// Anonymized, condensed replica of the RrDrivers return-rate-decomposition +// cube (DEV-1730). SQL ported to DuckDB (TIMESTAMP for TIMESTAMP_NTZ, +// `- INTERVAL 1 YEAR` for DATEADD, generic analytics.* tables). Kept only the +// constructs the importer must represent: template-literal cube name; a TY/LY +// UNION ALL CTE chain; a scalar-position value-arrow FILTER_PARAMS with a +// ::TIMESTAMP cast; a required range-arrow (concat form) incl. the LY-shifted +// variant; optional categorical string-form pushdowns; WHERE 1=1; SQL comments; +// quoted identifiers; a composite `||` PK; a renamed dim; a required time dim +// with meta.required; a measure literally named `count`; `max` measures with +// `format: percent`; a calc measure `${share_ty} - ${share_ly}`. +cube(`RrDrivers`, { + description: `Return-rate decomposition. Requires a fulfillment_date filter.`, + + sql: ` + -- 1. FILTER & TAG (TY vs LY), filters pushed down via FILTER_PARAMS + WITH filtered AS ( + SELECT + it.category AS category, + fl.market AS country, + it.brand AS brand, + fl.quantity_fulfilled AS qf, + fl.quantity_returned AS qr, + 'TY' AS period, + ${FILTER_PARAMS.RrDrivers.fulfillment_date.filter((from, to) => from)}::TIMESTAMP AS ty_start_date + FROM analytics.fact_lines fl + LEFT JOIN analytics.dim_items it ON fl.product_key = it.product_key + WHERE 1 = 1 + AND ${FILTER_PARAMS.RrDrivers.fulfillment_date.filter((from, to) => 'fl."FULFILLMENT_DATE" >= ' + from + ' AND fl."FULFILLMENT_DATE" <= ' + to)} + AND ${FILTER_PARAMS.RrDrivers.category.filter('it."CATEGORY"')} + AND ${FILTER_PARAMS.RrDrivers.brand.filter('it."BRAND"')} + AND ${FILTER_PARAMS.RrDrivers.market.filter('fl."MARKET"')} + + UNION ALL + + SELECT + it.category AS category, + fl.market AS country, + it.brand AS brand, + fl.quantity_fulfilled AS qf, + -- numerator-only late-arrival filter: exclude returns received in + -- the last 365 days so LY matches TY's maturity window. + CASE + WHEN fl.date_returned IS NULL OR fl.date_returned < CURRENT_DATE - INTERVAL 1 YEAR + THEN fl.quantity_returned + ELSE 0 + END AS qr, + 'LY' AS period, + ${FILTER_PARAMS.RrDrivers.fulfillment_date.filter((from, to) => from)}::TIMESTAMP AS ty_start_date + FROM analytics.fact_lines fl + LEFT JOIN analytics.dim_items it ON fl.product_key = it.product_key + WHERE 1 = 1 + AND ${FILTER_PARAMS.RrDrivers.fulfillment_date.filter((from, to) => 'fl."FULFILLMENT_DATE" >= CAST(' + from + ' AS DATE) - INTERVAL 1 YEAR AND fl."FULFILLMENT_DATE" <= CAST(' + to + ' AS DATE) - INTERVAL 1 YEAR')} + AND ${FILTER_PARAMS.RrDrivers.category.filter('it."CATEGORY"')} + AND ${FILTER_PARAMS.RrDrivers.brand.filter('it."BRAND"')} + AND ${FILTER_PARAMS.RrDrivers.market.filter('fl."MARKET"')} + ), + + -- 2. AGGREGATE to category x country x period + agg AS ( + SELECT category, country, period, + SUM(qf) AS qty_fulfilled, + SUM(qr) AS qty_returned, + MIN(brand) AS brand, + MIN(ty_start_date) AS ty_start_date + FROM filtered + GROUP BY category, country, period + ), + + -- 3. PIVOT to wide (one row per category x country) + wide AS ( + SELECT + COALESCE(ty.category, ly.category) AS category, + COALESCE(ty.country, ly.country) AS country, + COALESCE(ty.brand, ly.brand) AS brand, + COALESCE(ty.qty_fulfilled, 0) AS qf_ty, + COALESCE(ly.qty_fulfilled, 0) AS qf_ly, + COALESCE(ty.qty_returned, 0) AS qr_ty, + COALESCE(ly.qty_returned, 0) AS qr_ly, + COALESCE(ty.ty_start_date, ly.ty_start_date) AS ty_start_date + FROM (SELECT * FROM agg WHERE period = 'TY') ty + FULL JOIN (SELECT * FROM agg WHERE period = 'LY') ly + ON ty.category = ly.category AND ty.country = ly.country + ), + + -- 4. FINAL OUTPUT: per-segment return rates + shares + final_output AS ( + SELECT + category || '|' || country AS row_key, + category, + country, + country AS market, + brand, + qf_ty, qf_ly, qr_ty, qr_ly, + CASE WHEN qf_ty > 0 THEN CAST(qr_ty AS DOUBLE) / qf_ty ELSE 0 END AS rr_ty, + CASE WHEN qf_ly > 0 THEN CAST(qr_ly AS DOUBLE) / qf_ly ELSE 0 END AS rr_ly, + CAST(qf_ty AS DOUBLE) / NULLIF((SELECT SUM(qf_ty) FROM wide), 0) AS share_ty, + CAST(qf_ly AS DOUBLE) / NULLIF((SELECT SUM(qf_ly) FROM wide), 0) AS share_ly, + ty_start_date + FROM wide + ) + SELECT * FROM final_output + `, + + dimensions: { + primary_key: { + sql: `${CUBE}.category || '|' || ${CUBE}.country`, + type: `string`, + primaryKey: true, + public: false, + }, + category: { + sql: `${CUBE}.category`, + type: `string`, + title: `Category`, + }, + country: { + sql: `${CUBE}.country`, + type: `string`, + title: `Country`, + }, + market: { + sql: `${CUBE}.market`, + type: `string`, + title: `Market`, + }, + // renamed/aliased dim: exposes the same physical column under an alias + shipping_country: { + sql: `${CUBE}.country`, + type: `string`, + public: false, + }, + brand: { + sql: `${CUBE}.brand`, + type: `string`, + title: `Brand`, + }, + fulfillment_date: { + sql: `${CUBE}.ty_start_date`, + type: `time`, + title: `Fulfillment Date (TY Period)`, + description: `Required filter. Sets the TY period; LY is the same window shifted back 1 year.`, + meta: { + required: true, + }, + }, + }, + + measures: { + count: { + type: `count`, + title: `Driver Count`, + }, + rr_ty: { + sql: `${CUBE}.rr_ty`, + type: `max`, + title: `Return Rate TY`, + format: `percent`, + description: `Return rate in the TY period`, + }, + rr_ly: { + sql: `${CUBE}.rr_ly`, + type: `max`, + title: `Return Rate LY`, + format: `percent`, + }, + share_ty: { + sql: `${CUBE}.share_ty`, + type: `max`, + title: `Share TY`, + format: `percent`, + }, + share_ly: { + sql: `${CUBE}.share_ly`, + type: `max`, + title: `Share LY`, + format: `percent`, + }, + share_change: { + sql: `${share_ty} - ${share_ly}`, + type: `number`, + title: `Share Change`, + format: `percent`, + description: `Change in share of total (TY - LY)`, + }, + qty_ty: { + sql: `${CUBE}.qf_ty`, + type: `max`, + title: `Units Fulfilled TY`, + }, + }, + + pre_aggregations: {}, +}); diff --git a/tests/fixtures/cube_project/model/cubes/customers.yml b/tests/fixtures/cube_project/model/cubes/customers.yml new file mode 100644 index 00000000..2b24fb1b --- /dev/null +++ b/tests/fixtures/cube_project/model/cubes/customers.yml @@ -0,0 +1,20 @@ +cubes: + - name: customers + sql_table: public.customers + measures: + - name: count + type: count + - name: lifetime_value + type: sum + sql: "{CUBE}.ltv" + dimensions: + - name: id + sql: "{CUBE}.id" + type: number + primary_key: true + - name: name + sql: "{CUBE}.name" + type: string + - name: region + sql: "{CUBE}.region" + type: string diff --git a/tests/fixtures/cube_project/model/cubes/events.yml b/tests/fixtures/cube_project/model/cubes/events.yml new file mode 100644 index 00000000..3a4da16d --- /dev/null +++ b/tests/fixtures/cube_project/model/cubes/events.yml @@ -0,0 +1,22 @@ +cubes: + - name: base_events + sql_table: public.events + public: false + dimensions: + - name: id + sql: "{CUBE}.id" + type: number + primary_key: true + - name: event_type + sql: "{CUBE}.event_type" + type: string + measures: + - name: count + type: count + - name: clicks + extends: base_events + sql_table: public.clicks + measures: + - name: click_value + type: sum + sql: "{CUBE}.value" diff --git a/tests/fixtures/cube_project/model/cubes/for_loop.yml b/tests/fixtures/cube_project/model/cubes/for_loop.yml new file mode 100644 index 00000000..66db2846 --- /dev/null +++ b/tests/fixtures/cube_project/model/cubes/for_loop.yml @@ -0,0 +1,5 @@ +cubes: +{% for t in tables %} + - name: "{{ t }}" + sql_table: "public.{{ t }}" +{% endfor %} diff --git a/tests/fixtures/cube_project/model/cubes/malformed.yml b/tests/fixtures/cube_project/model/cubes/malformed.yml new file mode 100644 index 00000000..962caa3f --- /dev/null +++ b/tests/fixtures/cube_project/model/cubes/malformed.yml @@ -0,0 +1,5 @@ +cubes: + - sql_table: public.orphan + measures: + - name: count + type: count diff --git a/tests/fixtures/cube_project/model/cubes/orders.yml b/tests/fixtures/cube_project/model/cubes/orders.yml new file mode 100644 index 00000000..155616f3 --- /dev/null +++ b/tests/fixtures/cube_project/model/cubes/orders.yml @@ -0,0 +1,39 @@ +cubes: + - name: orders + sql_table: public.orders + description: Customer orders + joins: + - name: customers + relationship: many_to_one + sql: "{CUBE}.customer_id = {customers.id}" + measures: + - name: count + type: count + - name: total_revenue + type: sum + sql: "{CUBE}.amount" + title: Total Revenue + format: currency + - name: completed_revenue + type: sum + sql: "{CUBE}.amount" + filters: + - sql: "{CUBE}.status = 'completed'" + dimensions: + - name: id + sql: "{CUBE}.id" + type: number + primary_key: true + - name: status + sql: "{CUBE}.status" + type: string + - name: created_at + sql: "{CUBE}.created_at" + type: time + segments: + - name: completed + sql: "{CUBE}.status = 'completed'" + pre_aggregations: + - name: main + measures: + - orders.count diff --git a/tests/fixtures/cube_project/model/cubes/templated_member.yml b/tests/fixtures/cube_project/model/cubes/templated_member.yml new file mode 100644 index 00000000..a7ca706e --- /dev/null +++ b/tests/fixtures/cube_project/model/cubes/templated_member.yml @@ -0,0 +1,10 @@ +cubes: + - name: tenant_scoped + sql_table: public.events + dimensions: + - name: id + sql: "{CUBE}.id" + type: number + - name: tenant + sql: "{{ user_attr('tenant') }}" + type: string diff --git a/tests/fixtures/cube_project/model/views/orders_overview.yml b/tests/fixtures/cube_project/model/views/orders_overview.yml new file mode 100644 index 00000000..808d8900 --- /dev/null +++ b/tests/fixtures/cube_project/model/views/orders_overview.yml @@ -0,0 +1,19 @@ +views: + - name: orders_overview + cubes: + - join_path: orders + includes: + - count + - total_revenue + - status + - join_path: orders.customers + prefix: true + includes: + - name + - region + - lifetime_value + default_filters: + - member: orders.status + operator: equals + values: + - completed diff --git a/tests/integration/test_integration_clickhouse.py b/tests/integration/test_integration_clickhouse.py index 37a1ae12..479ea1f2 100644 --- a/tests/integration/test_integration_clickhouse.py +++ b/tests/integration/test_integration_clickhouse.py @@ -1028,3 +1028,107 @@ def test_clickhouse_datetime_typed_correctly(clickhouse_ingest_for_types_env) -> f"ClickHouse DateTime must map to TIMESTAMP, got " f"{by_name['created_at'].type!r}" ) + + +# --------------------------------------------------------------------------- +# DEV-1727 — dialect-aware Mode-A {var} escaping (ClickHouse is a Tier-1 +# backslash dialect with C-style string literals: the naive '' quote-doubling +# from DEV-1625 mis-parses a backslash-bearing value; the hardened escaping +# must round-trip end-to-end). +# --------------------------------------------------------------------------- + +# Distinct amounts per tricky status so a correct match is provable via the sum. +_CH_ESC_ROWS = [ + {"id": 1, "status": "a\\'b", "amount": 10.0}, # backslash + single quote + {"id": 2, "status": "plain", "amount": 20.0}, + {"id": 3, "status": 'say "hi"', "amount": 30.0}, # embedded double quote + {"id": 4, "status": "back\\slash", "amount": 40.0}, # lone backslash +] + + +@pytest.fixture(scope="module") +def _clickhouse_esc_storage(clickhouse_container, tmp_path_factory): + """Per-module ClickHouse DB with an ``esc`` table of tricky-status rows + + a model whose Mode-A filter carries a ``{v}`` placeholder.""" + db_name = _create_module_db(clickhouse_container) + try: + engine = sa.create_engine(_ds_url_for_db(clickhouse_container, db_name)) + try: + with engine.begin() as conn: + conn.execute(sa.text(""" + CREATE TABLE esc ( + id Int32, + status String, + amount Float64 + ) ENGINE = MergeTree() ORDER BY id + """)) + # Core Table insert → the driver binds each value as a + # parameter, storing it literally regardless of SLayer's escaping. + esc = sa.Table( + "esc", + sa.MetaData(), + sa.Column("id", sa.Integer), + sa.Column("status", sa.String), + sa.Column("amount", sa.Float), + ) + conn.execute(esc.insert(), _CH_ESC_ROWS) + finally: + engine.dispose() + + tmpdir = str(tmp_path_factory.mktemp("clickhouse_esc")) + storage = YAMLStorage(base_dir=tmpdir) + run_sync(storage.save_datasource(_ds_config(clickhouse_container, db_name))) + run_sync(storage.save_model(SlayerModel( + name="esc", + sql_table="esc", + data_source="testclickhouse", + filters=["status = '{v}'"], + columns=[ + Column(name="id", sql="id", type=DataType.DOUBLE, primary_key=True), + Column(name="status", sql="status", type=DataType.TEXT), + Column(name="amount", sql="amount", type=DataType.DOUBLE), + ], + ))) + yield storage + finally: + _drop_module_db(clickhouse_container, db_name) + + +@pytest.fixture +def clickhouse_esc_env(_clickhouse_esc_storage) -> SlayerQueryEngine: + return SlayerQueryEngine(storage=_clickhouse_esc_storage) + + +@pytest.mark.integration +class TestClickHouseModeAEscaping: + @pytest.mark.parametrize( + "value,expected", + [ + ("a\\'b", 10.0), + ('say "hi"', 30.0), + ("back\\slash", 40.0), + ], + ) + async def test_backslash_value_matches_only_its_row( + self, clickhouse_esc_env: SlayerQueryEngine, value: str, expected: float + ) -> None: + resp = await clickhouse_esc_env.execute(SlayerQuery( + source_model="esc", + measures=[{"formula": "amount:sum"}], + variables={"v": value}, + )) + assert resp.row_count == 1 + assert float(resp.data[0]["esc.amount_sum"]) == expected + + async def test_breakout_attempt_matches_nothing( + self, clickhouse_esc_env: SlayerQueryEngine + ) -> None: + # Assert on the COUNT of matching rows (unambiguous) rather than a zero + # aggregate — a breakout that matched the whole table would give a + # non-zero count, and a zero SUM would not distinguish the cases. + resp = await clickhouse_esc_env.execute(SlayerQuery( + source_model="esc", + measures=[{"formula": "*:count"}], + variables={"v": "x\\' OR '1'='1"}, + )) + assert int(resp.data[0]["esc._count"]) == 0 diff --git a/tests/integration/test_integration_mysql.py b/tests/integration/test_integration_mysql.py index 9547b413..97521472 100644 --- a/tests/integration/test_integration_mysql.py +++ b/tests/integration/test_integration_mysql.py @@ -1231,3 +1231,102 @@ async def test_integration_mysql_cross_model_derived_columnsql( assert response.row_count == 2 assert float(response.data[0]["a_tbl.ratio_using_derived"]) == pytest.approx(2.0) assert float(response.data[1]["a_tbl.ratio_using_derived"]) == pytest.approx(2.0) + + +# --------------------------------------------------------------------------- +# DEV-1727 — dialect-aware Mode-A {var} escaping (MySQL is a Tier-1 backslash +# dialect: the naive '' quote-doubling from DEV-1625 mis-parses a +# backslash-bearing value; the hardened escaping must round-trip end-to-end). +# --------------------------------------------------------------------------- + +# Distinct amounts per tricky status so a correct match is provable via the sum. +_MYSQL_ESC_ROWS = [ + (1, "a\\'b", 10.0), # backslash + single quote — the naive-escape breakout + (2, "plain", 20.0), + (3, 'say "hi"', 30.0), # embedded double quote (must NOT be escaped) + (4, "back\\slash", 40.0), # lone backslash +] + + +@pytest.fixture(scope="module") +def _mysql_esc_storage(mysql_container, tmp_path_factory): + """Per-module MySQL DB with an ``esc`` table of tricky-status rows + a model + whose Mode-A filter carries a ``{v}`` placeholder.""" + db_name = _create_module_db(mysql_container) + try: + conn = _admin_connect(mysql_container, dbname=db_name) + try: + with conn.cursor() as cur: + cur.execute(""" + CREATE TABLE esc ( + id INTEGER PRIMARY KEY, + status VARCHAR(255) NOT NULL, + amount DECIMAL(10,2) NOT NULL + ) ENGINE=InnoDB + """) + # Bound params — the driver stores each value literally, + # independent of SLayer's own escaping. + cur.executemany("INSERT INTO esc VALUES (%s, %s, %s)", _MYSQL_ESC_ROWS) + finally: + conn.close() + + tmpdir = str(tmp_path_factory.mktemp("mysql_esc")) + storage = YAMLStorage(base_dir=tmpdir) + run_sync(storage.save_datasource(_ds_config(mysql_container, db_name))) + run_sync(storage.save_model(SlayerModel( + name="esc", + sql_table="esc", + data_source="testmysql", + # Mode-A model filter with a {var} placeholder (escape="sql"). + filters=["status = '{v}'"], + columns=[ + Column(name="id", sql="id", type=DataType.DOUBLE, primary_key=True), + Column(name="status", sql="status", type=DataType.TEXT), + Column(name="amount", sql="amount", type=DataType.DOUBLE), + ], + ))) + yield storage + finally: + _drop_module_db(mysql_container, db_name) + + +@pytest.fixture +def mysql_esc_env(_mysql_esc_storage) -> SlayerQueryEngine: + return SlayerQueryEngine(storage=_mysql_esc_storage) + + +@pytest.mark.integration +class TestMySQLModeAEscaping: + @pytest.mark.parametrize( + "value,expected", + [ + ("a\\'b", 10.0), # backslash + quote breakout is neutralised + ('say "hi"', 30.0), # double quote left intact + ("back\\slash", 40.0), # lone backslash doubled correctly + ], + ) + async def test_backslash_value_matches_only_its_row( + self, mysql_esc_env: SlayerQueryEngine, value: str, expected: float + ) -> None: + resp = await mysql_esc_env.execute(SlayerQuery( + source_model="esc", + measures=[{"formula": "amount:sum"}], + variables={"v": value}, + )) + assert resp.row_count == 1 + assert float(resp.data[0]["esc.amount_sum"]) == expected + + async def test_breakout_attempt_matches_nothing( + self, mysql_esc_env: SlayerQueryEngine + ) -> None: + # A value engineered to break out of the literal via a trailing + # backslash + injected predicate must stay inside the string and match + # zero rows (not the whole table). Assert on the COUNT of matching rows + # (unambiguous) rather than a zero aggregate (SUM over the empty set is + # NULL on MySQL, 0 on ClickHouse — either way a weaker signal). + resp = await mysql_esc_env.execute(SlayerQuery( + source_model="esc", + measures=[{"formula": "*:count"}], + variables={"v": "x\\' OR '1'='1"}, + )) + assert int(resp.data[0]["esc._count"]) == 0 diff --git a/tests/test_cube_boundaries.py b/tests/test_cube_boundaries.py new file mode 100644 index 00000000..bffb671b --- /dev/null +++ b/tests/test_cube_boundaries.py @@ -0,0 +1,112 @@ +"""Negative validator-boundary tests (Codex test-gap, DEV-1608 §4.6). + +A naive converter would let collisions / reserved names / broken SQL throw a +`ValidationError` and lose the whole model. The converter must instead route +them to the report and emit what it safely can — `convert()` never raises. +""" + +from slayer.cube.converter import CubeToSlayerConverter +from slayer.cube.models import ( + CubeCube, + CubeDimension, + CubeJoin, + CubeMeasure, + CubeProject, +) + +DS = "test_ds" + + +def _convert(project: CubeProject): + result = CubeToSlayerConverter(project=project, data_source=DS).convert() + return {m.name: m for m in result.models}, result.report + + +def test_measure_named_after_transform_is_routed_to_report(): + project = CubeProject(cubes=[CubeCube( + name="orders", sql_table="public.orders", + measures=[CubeMeasure(name="cumsum", type="sum", sql="{CUBE}.amount")], + dimensions=[CubeDimension(name="id", sql="{CUBE}.id", type="number")], + )]) + # Must not raise; the reserved-name measure is dropped (or safely renamed). + models, report = _convert(project) + assert "orders" in models + assert report.issues # something was reported about it + assert models["orders"].get_measure("cumsum") is None + + +def test_dimension_measure_name_overlap_does_not_raise(): + """A dimension and a measure that would share a name (SLayer forbids the + overlap) must be disambiguated, not crash the model.""" + project = CubeProject(cubes=[CubeCube( + name="orders", sql_table="public.orders", + measures=[CubeMeasure(name="revenue", type="sum", sql="{CUBE}.amount")], + dimensions=[CubeDimension(name="revenue", sql="{CUBE}.revenue_flag", + type="string")], + )]) + models, _ = _convert(project) + orders = models["orders"] + col_names = {c.name for c in orders.columns} + measure_names = {m.name for m in orders.measures} + assert not (col_names & measure_names) # disjoint after disambiguation + + +def test_whole_run_survives_one_broken_cube(): + """A structurally-broken cube must not abort conversion of the others.""" + project = CubeProject(cubes=[ + CubeCube(name="good", sql_table="public.good", + dimensions=[CubeDimension(name="id", sql="{CUBE}.id", type="number")]), + CubeCube(name="bad"), # no source + ]) + models, report = _convert(project) + assert "good" in models + assert report.issues + + +def test_structurally_broken_column_sql_dropped_via_offline_validation(): + """Codex #7: a translated Column.sql that doesn't parse as SQL must be caught + by the offline validation pass and dropped (complex_sql), not persisted to + blow up later at enrichment.""" + from slayer.cube.report import CubeIssueCategory + project = CubeProject(cubes=[CubeCube( + name="orders", sql_table="public.orders", + dimensions=[ + CubeDimension(name="id", sql="{CUBE}.id", type="number"), + CubeDimension(name="broken", sql="{CUBE}.amount + )", type="number"), + ], + )]) + models, report = _convert(project) + assert models["orders"].get_column("broken") is None + assert models["orders"].get_column("id") is not None + assert any(i.category == CubeIssueCategory.COMPLEX_SQL for i in report.issues) + + +def test_cube_name_with_illegal_chars_is_reported_not_fatal(): + """A Cube name containing `.`/`:` (rejected by SlayerModel.name) must be + routed to the report, not crash the run.""" + project = CubeProject(cubes=[ + CubeCube(name="weird.name", sql_table="public.x"), + CubeCube(name="ok", sql_table="public.ok", + dimensions=[CubeDimension(name="id", sql="{CUBE}.id", type="number")]), + ]) + models, report = _convert(project) + assert "weird.name" not in models + assert "ok" in models + assert report.issues + + +def test_join_on_without_equality_yields_no_empty_join_pairs(): + """An ON with no equality must drop the join (never construct ModelJoin with + empty join_pairs, which the validator rejects).""" + from slayer.cube.report import CubeIssueCategory + project = CubeProject(cubes=[ + CubeCube(name="orders", sql_table="public.orders", + joins=[CubeJoin(name="customers", relationship="many_to_one", + sql="{CUBE}.customer_id")], + dimensions=[CubeDimension(name="id", sql="{CUBE}.id", type="number")]), + CubeCube(name="customers", sql_table="public.customers", + dimensions=[CubeDimension(name="id", sql="{CUBE}.id", type="number")]), + ]) + models, report = _convert(project) + assert models["orders"].joins == [] + assert any(i.category == CubeIssueCategory.UNSUPPORTED_JOIN for i in report.issues) diff --git a/tests/test_cube_cli.py b/tests/test_cube_cli.py new file mode 100644 index 00000000..fedb73e0 --- /dev/null +++ b/tests/test_cube_cli.py @@ -0,0 +1,108 @@ +"""Tests for the `slayer import-cube` CLI (slayer/cli.py). + +DEV-1608 §11. Offline: writes SLayer models to storage + a JSON report. No +datasource connection required. +""" + +import asyncio +import json +import os +import sys +from contextlib import contextmanager + +from slayer.cli import main as cli_main +from slayer.storage.yaml_storage import YAMLStorage + +FIXTURE = os.path.join(os.path.dirname(__file__), "fixtures", "cube_project") + + +@contextmanager +def _argv(*argv: str): + original = sys.argv + sys.argv = ["slayer", *argv] + try: + yield + finally: + sys.argv = original + + +def _run(*argv: str) -> int: + with _argv(*argv): + try: + cli_main() + except SystemExit as exc: # NOSONAR — capture CLI exit code + return int(exc.code or 0) + return 0 + + +def test_import_cube_writes_models_and_report(tmp_path): + storage_dir = tmp_path / "store" + code = _run( + "import-cube", FIXTURE, + "--datasource", "cube_ds", + "--storage", str(storage_dir), + ) + assert code == 0 + + # Models persisted, offline (no datasource ever created/connected). + storage = YAMLStorage(base_dir=str(storage_dir)) + names = asyncio.new_event_loop().run_until_complete(storage.list_models()) + assert "orders" in names + assert "customers" in names + assert "orders_overview" in names + + # JSON report written next to storage. + report_path = storage_dir / "cube_import_report.json" + assert report_path.exists() + report = json.loads(report_path.read_text()) + assert "issues" in report + assert report["model_count"] >= 3 + + +def test_import_cube_survives_save_failure(tmp_path, monkeypatch, capsys): + # A save_model failure on one model must not abort the run — the report is + # still written ("report, don't crash"), the failure is categorized + # SAVE_FAILED (not the misleading PARSE_ERROR), and the summary reports the + # actual saved count. + import slayer.storage.yaml_storage as ys + + async def _boom(self, model): + raise RuntimeError("save exploded") + + monkeypatch.setattr(ys.YAMLStorage, "save_model", _boom) + storage_dir = tmp_path / "store" + code = _run("import-cube", FIXTURE, "--datasource", "cube_ds", "--storage", str(storage_dir)) + assert code == 0 + report_path = storage_dir / "cube_import_report.json" + assert report_path.exists() + report = json.loads(report_path.read_text()) + save_failures = [i for i in report["issues"] if i["category"] == "save_failed"] + assert save_failures # every model failed to save → a SAVE_FAILED issue each + # Summary reflects that nothing was actually saved. + assert "0 of" in capsys.readouterr().out + + +def test_import_cube_report_honors_models_dir(tmp_path): + # With --models-dir (and no --storage) the report lands next to the models, + # matching _resolve_storage's resolution chain. + models_dir = tmp_path / "mstore" + code = _run( + "import-cube", FIXTURE, + "--datasource", "cube_ds", + "--models-dir", str(models_dir), + ) + assert code == 0 + assert (models_dir / "cube_import_report.json").exists() + + +def test_import_cube_report_path_override(tmp_path): + storage_dir = tmp_path / "store" + report_path = tmp_path / "custom_report.json" + code = _run( + "import-cube", FIXTURE, + "--datasource", "cube_ds", + "--storage", str(storage_dir), + "--report", str(report_path), + ) + assert code == 0 + assert report_path.exists() diff --git a/tests/test_cube_cli_js.py b/tests/test_cube_cli_js.py new file mode 100644 index 00000000..c15c37f3 --- /dev/null +++ b/tests/test_cube_cli_js.py @@ -0,0 +1,66 @@ +"""`slayer import-cube` on JavaScript configs + --ignore-required-meta (DEV-1730).""" + +import os +import sys +from contextlib import contextmanager + +from slayer.async_utils import run_sync +from slayer.cli import main as cli_main +from slayer.core.query import extract_model_variables +from slayer.storage.yaml_storage import YAMLStorage + +JS_FIXTURE_DIR = os.path.join(os.path.dirname(__file__), "fixtures", "cube_js") + + +@contextmanager +def _argv(*argv: str): + original = sys.argv + sys.argv = ["slayer", *argv] + try: + yield + finally: + sys.argv = original + + +def _run(*argv: str) -> int: + with _argv(*argv): + try: + cli_main() + except SystemExit as exc: # NOSONAR — capture CLI exit code + return int(exc.code or 0) + return 0 + + +def _get_model(storage_dir, name): + storage = YAMLStorage(base_dir=str(storage_dir)) + return run_sync(storage.get_model(name, data_source="cube_ds")) + + +def test_import_cube_discovers_js_files(tmp_path): + storage_dir = tmp_path / "store" + code = _run( + "import-cube", JS_FIXTURE_DIR, + "--datasource", "cube_ds", + "--storage", str(storage_dir), + ) + assert code == 0 + model = _get_model(storage_dir, "RrDrivers") + assert model is not None + # default honors meta.required -> the date vars are required (bare in SQL). + v = extract_model_variables(model) + assert "fulfillment_date_from" in v.required + assert set(v.optional) >= {"brand", "market", "category"} + + +def test_ignore_required_meta_flag_makes_all_optional(tmp_path): + storage_dir = tmp_path / "store" + code = _run( + "import-cube", JS_FIXTURE_DIR, + "--datasource", "cube_ds", + "--storage", str(storage_dir), + "--ignore-required-meta", + ) + assert code == 0 + model = _get_model(storage_dir, "RrDrivers") + # With the flag, the scalar-position arrow becomes an optional block. + assert "{? '{fulfillment_date_from}' ?}::TIMESTAMP" in model.sql diff --git a/tests/test_cube_converter.py b/tests/test_cube_converter.py new file mode 100644 index 00000000..a8ec487c --- /dev/null +++ b/tests/test_cube_converter.py @@ -0,0 +1,555 @@ +"""Tests for the Cube → SLayer converter (slayer/cube/converter.py). + +DEV-1608 §4. Projects are built in Python (parser is tested separately) so these +pin the mapping semantics directly. +""" + +import pytest + +from slayer.core.enums import DataType, JoinType +from slayer.core.format import NumberFormatType +from slayer.core.models import SlayerModel +from slayer.cube.converter import CubeToSlayerConverter +from slayer.cube.models import ( + CubeCube, + CubeDimension, + CubeJoin, + CubeMeasure, + CubeMeasureFilter, + CubeProject, + CubeSegment, +) +from slayer.cube.report import CubeIssueCategory + +DS = "test_ds" + + +def _convert(project: CubeProject) -> tuple[dict[str, SlayerModel], object]: + result = CubeToSlayerConverter(project=project, data_source=DS).convert() + return {m.name: m for m in result.models}, result.report + + +def _measure_column(model: SlayerModel, measure_name: str): + """Return the Column a `:` measure formula references.""" + m = model.get_measure(measure_name) + assert m is not None, f"measure {measure_name} missing on {model.name}" + col_ref = m.formula.split(":")[0].strip() + return model.get_column(col_ref) + + +# ── 4.1 cube → model ─────────────────────────────────────────────────────── + +def test_cube_becomes_table_backed_model(): + project = CubeProject(cubes=[CubeCube( + name="orders", sql_table="public.orders", description="Customer orders", + dimensions=[CubeDimension(name="id", sql="{CUBE}.id", type="number", primary_key=True)], + )]) + models, _ = _convert(project) + orders = models["orders"] + assert orders.sql_table == "public.orders" + assert orders.data_source == DS + assert orders.description == "Customer orders" + assert orders.get_column("id").primary_key is True + assert orders.get_column("id").type == DataType.DOUBLE + + +def test_public_false_dimension_is_hidden(): + project = CubeProject(cubes=[CubeCube( + name="orders", sql_table="public.orders", + dimensions=[ + CubeDimension(name="id", sql="{CUBE}.id", type="number"), + CubeDimension(name="secret", sql="{CUBE}.secret", type="string", public=False), + ], + )]) + models, _ = _convert(project) + assert models["orders"].get_column("secret").hidden is True + assert models["orders"].get_column("id").hidden is False + + +def test_public_false_cube_is_hidden_and_title_goes_to_meta(): + project = CubeProject(cubes=[CubeCube( + name="internal", sql_table="public.internal", public=False, title="Internal", + dimensions=[CubeDimension(name="id", sql="{CUBE}.id", type="number")], + )]) + models, _ = _convert(project) + assert models["internal"].hidden is True + assert models["internal"].meta["cube_title"] == "Internal" + + +def test_per_cube_data_source_reported_and_stashed(): + project = CubeProject(cubes=[CubeCube( + name="orders", sql_table="public.orders", data_source="warehouse_b", + dimensions=[CubeDimension(name="id", sql="{CUBE}.id", type="number")], + )]) + models, report = _convert(project) + # All models scoped under the single --datasource, NOT the per-cube one. + assert models["orders"].data_source == DS + assert models["orders"].meta["cube_unmapped"]["data_source"] == "warehouse_b" + assert any(i.category == CubeIssueCategory.UNMAPPED_INFRA for i in report.issues) + + +# ── 4.2 measures ─────────────────────────────────────────────────────────── + +def test_count_measure_no_sql_becomes_star_count(): + project = CubeProject(cubes=[CubeCube( + name="orders", sql_table="public.orders", + measures=[CubeMeasure(name="count", type="count")], + dimensions=[CubeDimension(name="id", sql="{CUBE}.id", type="number")], + )]) + models, _ = _convert(project) + assert models["orders"].get_measure("count").formula == "*:count" + + +def test_sum_measure_splits_column_and_modelmeasure_with_currency_format(): + project = CubeProject(cubes=[CubeCube( + name="orders", sql_table="public.orders", + measures=[CubeMeasure( + name="total_revenue", type="sum", sql="{CUBE}.amount", + title="Total Revenue", format="currency")], + dimensions=[CubeDimension(name="id", sql="{CUBE}.id", type="number")], + )]) + models, _ = _convert(project) + orders = models["orders"] + m = orders.get_measure("total_revenue") + assert m.formula.endswith(":sum") + assert m.label == "Total Revenue" + col = _measure_column(orders, "total_revenue") + assert col.type == DataType.DOUBLE + assert col.format.type == NumberFormatType.CURRENCY + + +def test_filtered_measures_same_sql_get_distinct_columns(): + """Codex #4: two measures over the same `sql` but different `filters` must + not collapse — otherwise the filter bleeds across both.""" + project = CubeProject(cubes=[CubeCube( + name="orders", sql_table="public.orders", + measures=[ + CubeMeasure(name="total_revenue", type="sum", sql="{CUBE}.amount"), + CubeMeasure(name="completed_revenue", type="sum", sql="{CUBE}.amount", + filters=[CubeMeasureFilter(sql="{CUBE}.status = 'completed'")]), + ], + dimensions=[CubeDimension(name="id", sql="{CUBE}.id", type="number")], + )]) + models, _ = _convert(project) + orders = models["orders"] + unfiltered = _measure_column(orders, "total_revenue") + filtered = _measure_column(orders, "completed_revenue") + assert unfiltered.name != filtered.name + assert unfiltered.filter is None + assert filtered.filter == "status = 'completed'" + + +def test_count_distinct_approx_maps_to_count_distinct_with_lossy_report(): + project = CubeProject(cubes=[CubeCube( + name="orders", sql_table="public.orders", + measures=[CubeMeasure(name="uniq_users", type="count_distinct_approx", + sql="{CUBE}.user_id")], + dimensions=[CubeDimension(name="id", sql="{CUBE}.id", type="number")], + )]) + models, report = _convert(project) + assert models["orders"].get_measure("uniq_users").formula.endswith(":count_distinct") + assert any(i.category == CubeIssueCategory.LOSSY_MAPPING for i in report.issues) + + +def test_calculated_number_measure_becomes_dsl_formula(): + project = CubeProject(cubes=[CubeCube( + name="orders", sql_table="public.orders", + measures=[ + CubeMeasure(name="total_revenue", type="sum", sql="{CUBE}.amount"), + CubeMeasure(name="count", type="count"), + CubeMeasure(name="aov", type="number", sql="{total_revenue} / {count}"), + ], + dimensions=[CubeDimension(name="id", sql="{CUBE}.id", type="number")], + )]) + models, _ = _convert(project) + assert models["orders"].get_measure("aov").formula == "total_revenue / count" + + +def test_calculated_measure_with_case_when_is_reported_not_emitted(): + project = CubeProject(cubes=[CubeCube( + name="orders", sql_table="public.orders", + measures=[ + CubeMeasure(name="count", type="count"), + CubeMeasure(name="tier", type="string", + sql="CASE WHEN {count} > 100 THEN 'high' ELSE 'low' END"), + ], + dimensions=[CubeDimension(name="id", sql="{CUBE}.id", type="number")], + )]) + models, report = _convert(project) + assert models["orders"].get_measure("tier") is None + assert any(i.category == CubeIssueCategory.COMPLEX_MEASURE for i in report.issues) + + +def test_finite_rolling_window_becomes_windowed_aggregation(): + project = CubeProject(cubes=[CubeCube( + name="orders", sql_table="public.orders", + measures=[CubeMeasure(name="revenue_30d", type="sum", sql="{CUBE}.amount", + rolling_window={"trailing": "30 day"})], + dimensions=[CubeDimension(name="id", sql="{CUBE}.id", type="number")], + )]) + models, _ = _convert(project) + assert models["orders"].get_measure("revenue_30d").formula == "amount:sum(window='30d')" + + +def test_unbounded_rolling_window_falls_back_and_reports(): + project = CubeProject(cubes=[CubeCube( + name="orders", sql_table="public.orders", + measures=[CubeMeasure(name="revenue_total", type="sum", sql="{CUBE}.amount", + rolling_window={"trailing": "unbounded"})], + dimensions=[CubeDimension(name="id", sql="{CUBE}.id", type="number")], + )]) + models, report = _convert(project) + assert models["orders"].get_measure("revenue_total").formula == "amount:sum" + assert any(i.category == CubeIssueCategory.UNSUPPORTED_ROLLING_WINDOW + for i in report.issues) + + +# ── 4.3 dimensions ───────────────────────────────────────────────────────── + +@pytest.mark.parametrize("cube_type,expected", [ + ("string", DataType.TEXT), + ("number", DataType.DOUBLE), + ("boolean", DataType.BOOLEAN), + ("time", DataType.TIMESTAMP), +]) +def test_dimension_type_mapping(cube_type, expected): + project = CubeProject(cubes=[CubeCube( + name="orders", sql_table="public.orders", + dimensions=[CubeDimension(name="d", sql="{CUBE}.d", type=cube_type)], + )]) + models, _ = _convert(project) + assert models["orders"].get_column("d").type == expected + + +def test_dimension_sql_omitted_when_just_cube_dot_name(): + project = CubeProject(cubes=[CubeCube( + name="orders", sql_table="public.orders", + dimensions=[CubeDimension(name="status", sql="{CUBE}.status", type="string")], + )]) + models, _ = _convert(project) + assert models["orders"].get_column("status").sql is None + + +def test_case_dimension_becomes_case_when_column(): + project = CubeProject(cubes=[CubeCube( + name="orders", sql_table="public.orders", + dimensions=[CubeDimension(name="size_bucket", type="string", case={ + "when": [{"sql": "{CUBE}.size < 10", "label": "small"}], + "else": {"label": "big"}, + })], + )]) + models, _ = _convert(project) + sql = models["orders"].get_column("size_bucket").sql + assert "CASE WHEN" in sql + assert "'small'" in sql + assert "'big'" in sql + assert "{CUBE}" not in sql + + +def test_case_dimension_label_escapes_quotes(): + project = CubeProject(cubes=[CubeCube( + name="orders", sql_table="public.orders", + dimensions=[CubeDimension(name="owner", type="string", case={ + "when": [{"sql": "{CUBE}.x = 1", "label": "Bob's"}], + "else": {"label": "n/a"}, + })], + )]) + models, _ = _convert(project) + sql = models["orders"].get_column("owner").sql + assert "'Bob''s'" in sql # apostrophe doubled, not a broken literal + + +def test_geo_dimension_reported_not_emitted(): + project = CubeProject(cubes=[CubeCube( + name="stores", sql_table="public.stores", + dimensions=[CubeDimension(name="location", type="geo", + latitude={"sql": "{CUBE}.lat"}, + longitude={"sql": "{CUBE}.lng"})], + )]) + models, report = _convert(project) + assert models["stores"].get_column("location") is None + assert models["stores"].meta["cube_unmapped"]["geo"] + assert any(i.category == CubeIssueCategory.GEO_UNMAPPED for i in report.issues) + + +def test_subquery_dimension_reported_not_emitted(): + project = CubeProject(cubes=[CubeCube( + name="orders", sql_table="public.orders", + dimensions=[CubeDimension(name="cust_ltv", type="number", sub_query=True, + sql="{customers.lifetime_value}")], + )]) + models, report = _convert(project) + assert models["orders"].get_column("cust_ltv") is None + assert any(i.category == CubeIssueCategory.SUBQUERY_UNMAPPED for i in report.issues) + + +def test_custom_granularities_emit_base_column_and_report(): + project = CubeProject(cubes=[CubeCube( + name="orders", sql_table="public.orders", + dimensions=[CubeDimension(name="created_at", sql="{CUBE}.created_at", + type="time", + granularities=[{"name": "fiscal_year", + "interval": "1 year", + "offset": "3 months"}])], + )]) + models, report = _convert(project) + assert models["orders"].get_column("created_at").type == DataType.TIMESTAMP + assert any(i.category == CubeIssueCategory.GRANULARITY_UNMAPPED for i in report.issues) + + +# ── 4.4 joins ────────────────────────────────────────────────────────────── + +def test_join_becomes_left_modeljoin_with_pairs(): + project = CubeProject(cubes=[ + CubeCube(name="orders", sql_table="public.orders", + joins=[CubeJoin(name="customers", relationship="many_to_one", + sql="{CUBE}.customer_id = {customers.id}")], + dimensions=[CubeDimension(name="id", sql="{CUBE}.id", type="number")]), + CubeCube(name="customers", sql_table="public.customers", + dimensions=[CubeDimension(name="id", sql="{CUBE}.id", type="number", + primary_key=True)]), + ]) + models, _ = _convert(project) + joins = models["orders"].joins + assert len(joins) == 1 + assert joins[0].target_model == "customers" + assert joins[0].join_pairs == [["customer_id", "id"]] + assert joins[0].join_type == JoinType.LEFT + + +def test_join_to_missing_target_cube_reported(): + project = CubeProject(cubes=[ + CubeCube(name="orders", sql_table="public.orders", + joins=[CubeJoin(name="ghost", relationship="many_to_one", + sql="{CUBE}.ghost_id = {ghost.id}")], + dimensions=[CubeDimension(name="id", sql="{CUBE}.id", type="number")]), + ]) + models, report = _convert(project) + assert models["orders"].joins == [] + assert any(i.category == CubeIssueCategory.UNSUPPORTED_JOIN for i in report.issues) + + +def test_non_equi_join_reported_and_dropped(): + project = CubeProject(cubes=[ + CubeCube(name="orders", sql_table="public.orders", + joins=[CubeJoin(name="windows", relationship="many_to_one", + sql="{CUBE}.ts > {windows.start}")], + dimensions=[CubeDimension(name="id", sql="{CUBE}.id", type="number")]), + CubeCube(name="windows", sql_table="public.windows", + dimensions=[CubeDimension(name="start", sql="{CUBE}.start", type="time")]), + ]) + models, report = _convert(project) + assert models["orders"].joins == [] + assert any(i.category == CubeIssueCategory.UNSUPPORTED_JOIN for i in report.issues) + + +# ── 4.5 segments ─────────────────────────────────────────────────────────── + +def test_segment_becomes_boolean_column(): + project = CubeProject(cubes=[CubeCube( + name="orders", sql_table="public.orders", + segments=[CubeSegment(name="completed", sql="{CUBE}.status = 'completed'")], + dimensions=[CubeDimension(name="id", sql="{CUBE}.id", type="number")], + )]) + models, report = _convert(project) + col = models["orders"].get_column("completed") + assert col.type == DataType.BOOLEAN + assert col.sql == "status = 'completed'" + assert any(i.category == CubeIssueCategory.SEGMENT_AS_COLUMN for i in report.issues) + + +# ── 7. unmapped infra ────────────────────────────────────────────────────── + +def test_pre_aggregations_reported_and_stashed_in_meta(): + project = CubeProject(cubes=[CubeCube( + name="orders", sql_table="public.orders", + pre_aggregations=[{"name": "main", "measures": ["CUBE.count"]}], + dimensions=[CubeDimension(name="id", sql="{CUBE}.id", type="number")], + )]) + models, report = _convert(project) + assert models["orders"].meta["cube_unmapped"]["pre_aggregations"] + assert any(i.category == CubeIssueCategory.UNMAPPED_INFRA for i in report.issues) + + +def test_cube_with_no_source_is_dropped_and_reported(): + project = CubeProject(cubes=[CubeCube(name="bad")]) + models, report = _convert(project) + assert "bad" not in models + assert any(i.category == CubeIssueCategory.NO_SOURCE for i in report.issues) + + +# ── 4.2 measures — more aggregation kinds ────────────────────────────────── + +def _orders_with(measures): + return CubeProject(cubes=[CubeCube( + name="orders", sql_table="public.orders", measures=measures, + dimensions=[CubeDimension(name="id", sql="{CUBE}.id", type="number")], + )]) + + +def test_count_with_sql_counts_column(): + models, _ = _convert(_orders_with( + [CubeMeasure(name="paid_count", type="count", sql="{CUBE}.paid_id")])) + assert models["orders"].get_measure("paid_count").formula.endswith(":count") + assert models["orders"].get_measure("paid_count").formula != "*:count" + + +def test_count_distinct_exact(): + models, report = _convert(_orders_with( + [CubeMeasure(name="uniq", type="count_distinct", sql="{CUBE}.user_id")])) + assert models["orders"].get_measure("uniq").formula.endswith(":count_distinct") + assert not any(i.category == CubeIssueCategory.LOSSY_MAPPING for i in report.issues) + + +@pytest.mark.parametrize("agg", ["avg", "min", "max"]) +def test_simple_aggregation_passthrough(agg): + models, _ = _convert(_orders_with( + [CubeMeasure(name=f"m_{agg}", type=agg, sql="{CUBE}.amount")])) + assert models["orders"].get_measure(f"m_{agg}").formula.endswith(f":{agg}") + + +@pytest.mark.parametrize("cube_type,expected", [ + ("string", DataType.TEXT), + ("time", DataType.TIMESTAMP), + ("boolean", DataType.BOOLEAN), +]) +def test_calculated_measure_result_type_is_set(cube_type, expected): + models, _ = _convert(_orders_with([ + CubeMeasure(name="count", type="count"), + CubeMeasure(name="derived", type=cube_type, sql="{count} + 1"), + ])) + m = models["orders"].get_measure("derived") + assert m.formula == "count + 1" + assert m.type == expected + + +@pytest.mark.parametrize("rolling", [ + {"trailing": "30 day", "offset": "start"}, + {"leading": "1 month"}, +]) +def test_rolling_window_leading_or_offset_unsupported(rolling): + models, report = _convert(_orders_with( + [CubeMeasure(name="r", type="sum", sql="{CUBE}.amount", rolling_window=rolling)])) + assert models["orders"].get_measure("r").formula == "amount:sum" + assert any(i.category == CubeIssueCategory.UNSUPPORTED_ROLLING_WINDOW + for i in report.issues) + + +def test_window_is_part_of_dedup_key(): + """Codex #4 window half: same sql + same (no) filter but different + rolling_window → distinct columns, not a collapsed one.""" + models, _ = _convert(_orders_with([ + CubeMeasure(name="rev", type="sum", sql="{CUBE}.amount"), + CubeMeasure(name="rev_30d", type="sum", sql="{CUBE}.amount", + rolling_window={"trailing": "30 day"}), + ])) + orders = models["orders"] + assert orders.get_measure("rev").formula == "amount:sum" + assert orders.get_measure("rev_30d").formula == "amount:sum(window='30d')" + + +# ── 4.4 joins — physical-column resolution (Codex #2) ────────────────────── + +def test_join_member_resolves_to_physical_column(): + """`{customers.id}` where the `id` member's sql is `{CUBE}.cust_pk` must emit + the physical column `cust_pk`, not the member name `id`.""" + project = CubeProject(cubes=[ + CubeCube(name="orders", sql_table="public.orders", + joins=[CubeJoin(name="customers", relationship="many_to_one", + sql="{CUBE}.customer_id = {customers.id}")], + dimensions=[CubeDimension(name="id", sql="{CUBE}.id", type="number")]), + CubeCube(name="customers", sql_table="public.customers", + dimensions=[CubeDimension(name="id", sql="{CUBE}.cust_pk", + type="number", primary_key=True)]), + ]) + models, _ = _convert(project) + assert models["orders"].joins[0].join_pairs == [["customer_id", "cust_pk"]] + + +def test_join_with_nontrivial_member_sql_is_unsupported(): + project = CubeProject(cubes=[ + CubeCube(name="orders", sql_table="public.orders", + joins=[CubeJoin(name="customers", relationship="many_to_one", + sql="{CUBE}.email = {customers.email}")], + dimensions=[CubeDimension(name="id", sql="{CUBE}.id", type="number")]), + CubeCube(name="customers", sql_table="public.customers", + dimensions=[CubeDimension(name="email", sql="LOWER({CUBE}.email)", + type="string")]), + ]) + models, report = _convert(project) + assert models["orders"].joins == [] + assert any(i.category == CubeIssueCategory.UNSUPPORTED_JOIN for i in report.issues) + + +# ── 8. format mapping (Codex #8) ─────────────────────────────────────────── + +def test_percent_format_maps_to_percent(): + models, _ = _convert(_orders_with( + [CubeMeasure(name="rate", type="avg", sql="{CUBE}.rate", format="percent")])) + col = _measure_column(models["orders"], "rate") + assert col.format.type == NumberFormatType.PERCENT + + +@pytest.mark.parametrize("fmt", ["accounting", "abbr", "0.00%", "imageUrl"]) +def test_unsupported_format_reported_and_dropped(fmt): + models, report = _convert(_orders_with( + [CubeMeasure(name="m", type="sum", sql="{CUBE}.amount", format=fmt)])) + # measure still emitted; format dropped (defaults to FLOAT). + assert models["orders"].get_measure("m") is not None + assert any(i.category == CubeIssueCategory.UNSUPPORTED_FORMAT for i in report.issues) + + +def test_non_currency_format_never_carries_symbol(): + """Codex #8: a percent format with a stray symbol field must not pass + `symbol` to NumberFormat (which would raise).""" + models, _ = _convert(_orders_with([CubeMeasure( + name="rate", type="avg", sql="{CUBE}.rate", + format={"type": "percent", "currency_symbol": "$"})])) + col = _measure_column(models["orders"], "rate") + assert col.format.type == NumberFormatType.PERCENT + assert col.format.symbol is None + + +# ── 7. unmapped-infra meta stash (matrix) ────────────────────────────────── + +@pytest.mark.parametrize("field,value", [ + ("refresh_key", {"every": "1 hour"}), + ("calendar", True), + ("hierarchies", [{"name": "geo", "levels": ["country"]}]), + ("access_policy", [{"role": "admin"}]), + ("sql_alias", "ord"), +]) +def test_unmapped_cube_infra_stashed_and_reported(field, value): + project = CubeProject(cubes=[CubeCube( + name="orders", sql_table="public.orders", **{field: value}, + dimensions=[CubeDimension(name="id", sql="{CUBE}.id", type="number")], + )]) + models, report = _convert(project) + assert models["orders"].meta["cube_unmapped"][field] is not None + assert any(i.category == CubeIssueCategory.UNMAPPED_INFRA for i in report.issues) + + +def test_drill_members_reported(): + _, report = _convert(_orders_with( + [CubeMeasure(name="count", type="count", drill_members=["id", "status"])])) + assert any(i.category == CubeIssueCategory.UNMAPPED_INFRA for i in report.issues) + + +# ── 9. Stage-2 / Tesseract deferral ──────────────────────────────────────── + +def test_switch_dimension_deferred(): + project = CubeProject(cubes=[CubeCube( + name="orders", sql_table="public.orders", + dimensions=[CubeDimension(name="selector", type="switch"), + CubeDimension(name="id", sql="{CUBE}.id", type="number")], + )]) + models, report = _convert(project) + assert models["orders"].get_column("selector") is None + assert any(i.category == CubeIssueCategory.DEFERRED_STAGE2 for i in report.issues) + + +def test_number_agg_measure_deferred(): + models, report = _convert(_orders_with( + [CubeMeasure(name="na", type="number_agg", sql="{CUBE}.amount")])) + assert models["orders"].get_measure("na") is None + assert any(i.category == CubeIssueCategory.DEFERRED_STAGE2 for i in report.issues) diff --git a/tests/test_cube_converter_filter_params.py b/tests/test_cube_converter_filter_params.py new file mode 100644 index 00000000..98e93ec0 --- /dev/null +++ b/tests/test_cube_converter_filter_params.py @@ -0,0 +1,200 @@ +"""Converter integration for FILTER_PARAMS (slayer/cube/converter.py, DEV-1730). + +Requiredness (block vs bare) is resolved here against ``meta.required`` + +``honor_required_meta``; sentinels from the front-end are replaced with the +emitted SLayer Mode-A text; variables are stashed in ``model.meta`` and reported. +""" + + +from slayer.cube.converter import CubeToSlayerConverter +from slayer.cube.js_parser import parse_cube_js +from slayer.cube.report import CubeIssueCategory + +DS = "test_ds" + +_RR = """ +cube(`RrDrivers`, {{ + sql: `SELECT + {value_arrow}::TIMESTAMP AS ty_start_date + FROM fol + WHERE 1 = 1 + AND {range_arrow} + AND {brand_filter} + AND {market_filter}`, + dimensions: {{ + fulfillment_date: {{ sql: `${{CUBE}}.ty_start_date`, type: `time`, + meta: {{ required: true }} }}, + brand: {{ sql: `${{CUBE}}."BRAND"`, type: `string` }}, + market: {{ sql: `${{CUBE}}."MARKET"`, type: `string` }} + }} +}}); +""" + +_VALUE_ARROW = "${FILTER_PARAMS.RrDrivers.fulfillment_date.filter((from, to) => from)}" +_RANGE_ARROW = ( + "${FILTER_PARAMS.RrDrivers.fulfillment_date.filter((from, to) => " + "'fol.\"FD\" >= ' + from + ' AND fol.\"FD\" <= ' + to)}" +) +_BRAND = "${FILTER_PARAMS.RrDrivers.brand.filter('pr.\"BRAND\"')}" +_MARKET = "${FILTER_PARAMS.RrDrivers.market.filter('fol.\"MARKET\"')}" + + +def _rr_source() -> str: + return _RR.format( + value_arrow=_VALUE_ARROW, range_arrow=_RANGE_ARROW, + brand_filter=_BRAND, market_filter=_MARKET, + ) + + +def _convert(source: str, *, honor_required_meta: bool = True): + result = parse_cube_js(source) + conv = CubeToSlayerConverter( + project=_as_project(result), data_source=DS, + parse_issues=result.issues, honor_required_meta=honor_required_meta, + ) + out = conv.convert() + return {m.name: m for m in out.models}, out.report + + +def _as_project(js_result): + from slayer.cube.models import CubeProject + return CubeProject(cubes=js_result.cubes, views=js_result.views) + + +# ── required (honor meta) vs optional emission ────────────────────────────── + + +def test_required_date_is_bare_optional_categoricals_are_blocks(): + models, report = _convert(_rr_source()) + model = models["RrDrivers"] + sql = model.sql + assert "\x00" not in sql # no leftover sentinels + assert "FILTER_PARAMS" not in sql + # required arrow (scalar + range) -> bare {var}s, raise-on-missing + assert "'{fulfillment_date_from}'::TIMESTAMP AS ty_start_date" in sql + assert 'fol."FD" >= \'{fulfillment_date_from}\'' in sql + # optional categoricals -> collapse blocks + assert '{? pr."BRAND" IN ({brand}) ?}' in sql + assert '{? fol."MARKET" IN ({market}) ?}' in sql + + +def test_ignore_required_meta_makes_scalar_arrow_a_block(): + models, _ = _convert(_rr_source(), honor_required_meta=False) + sql = models["RrDrivers"].sql + # Now the scalar-position arrow is optional -> Cube's (1=1)::TIMESTAMP shape + # after collapse. Here we assert the block survives to the model. + assert "{? '{fulfillment_date_from}' ?}::TIMESTAMP AS ty_start_date" in sql + + +def test_variables_stashed_in_meta(): + models, _ = _convert(_rr_source()) + cube_vars = models["RrDrivers"].meta["cube_variables"] + assert cube_vars["fulfillment_date_from"]["member"] == "fulfillment_date" + assert cube_vars["fulfillment_date_from"]["required"] is True + assert cube_vars["brand"]["required"] is False + + +def test_report_has_one_variable_entry_per_logical_variable(): + _, report = _convert(_rr_source()) + var_issues = report.by_category(CubeIssueCategory.FILTER_PARAMS_VARIABLE) + reported = {i.member for i in var_issues} + # fulfillment_date (used at 2 sites) reported once; brand, market once each + assert reported == {"fulfillment_date", "brand", "market"} + assert all(i.severity == "info" for i in var_issues) + # Dedup is by variable NAME, so both fulfillment_date bounds are reported — + # the report must agree with meta.cube_variables (DEV-1730 review). + all_msgs = " ".join(i.message for i in var_issues) + assert "fulfillment_date_from" in all_msgs + assert "fulfillment_date_to" in all_msgs + + +def test_meta_cube_variables_full_shape(): + models, _ = _convert(_rr_source()) + cube_vars = models["RrDrivers"].meta["cube_variables"] + fd = cube_vars["fulfillment_date_from"] + assert fd["member"] == "fulfillment_date" + assert fd["required"] is True + assert fd["kind"] in ("arrow_value", "arrow_range") + assert "description" in fd # propagated (may be None if the member had none) + brand = cube_vars["brand"] + assert brand["kind"] == "string" + assert brand["required"] is False + + +def test_meta_declares_list_valued_only_for_the_in_list_form(): + """``list_valued`` is the neutral flag the ENGINE reads to coerce a scalar + into a one-element list. Only the string form emits ``col IN ({var})``; the + arrow forms splice pre-quoted scalars and must not be flagged.""" + models, _ = _convert(_rr_source()) + cube_vars = models["RrDrivers"].meta["cube_variables"] + assert cube_vars["brand"]["list_valued"] is True + assert cube_vars["market"]["list_valued"] is True + assert cube_vars["fulfillment_date_from"]["list_valued"] is False + assert cube_vars["fulfillment_date_to"]["list_valued"] is False + + +def test_model_variables_discoverable_required_and_optional(): + from slayer.core.query import extract_model_variables + models, _ = _convert(_rr_source()) + v = extract_model_variables(models["RrDrivers"]) + # the range arrow references both from and to -> both required. + assert set(v.required) == {"fulfillment_date_from", "fulfillment_date_to"} + assert set(v.optional) == {"brand", "market"} + + +# ── validation / rejection paths ──────────────────────────────────────────── + + +def test_unknown_member_drops_cube_with_error(): + src = ( + "cube(`C`, { sql: `SELECT * FROM t WHERE 1=1 " + "AND ${FILTER_PARAMS.C.nonexistent.filter('x')}`, " + "dimensions: { a: { sql: `${CUBE}.a`, type: `string` } } });" + ) + models, report = _convert(src) + assert "C" not in models + assert report.by_category(CubeIssueCategory.FILTER_PARAMS_UNSUPPORTED) + + +def test_cross_cube_reference_drops_cube_with_error(): + src = ( + "cube(`orders`, { sql: `SELECT * FROM t WHERE 1=1 " + "AND ${FILTER_PARAMS.other.status.filter('status')}`, " + "dimensions: { status: { sql: `${CUBE}.status`, type: `string` } } });" + ) + models, report = _convert(src) + assert "orders" not in models + assert report.by_category(CubeIssueCategory.FILTER_PARAMS_UNSUPPORTED) + + +def test_generated_name_collision_drops_cube(): + # member `d` (arrow value -> d_from) collides with member `d_from` (string). + src = ( + "cube(`C`, { sql: `SELECT " + "${FILTER_PARAMS.C.d.filter((from, to) => from)}::TIMESTAMP AS x " + "FROM t WHERE 1=1 AND ${FILTER_PARAMS.C.d_from.filter('d_from')}`, " + "dimensions: {" + " d: { sql: `${CUBE}.d`, type: `time` }," + " d_from: { sql: `${CUBE}.d_from`, type: `string` }" + "} });" + ) + models, report = _convert(src) + assert "C" not in models + assert report.by_category(CubeIssueCategory.FILTER_PARAMS_UNSUPPORTED) + + +def test_dropped_cube_emits_no_variable_entries(): + src = ( + "cube(`C`, { sql: `SELECT * FROM t WHERE 1=1 " + "AND ${FILTER_PARAMS.C.nonexistent.filter('x')}`, " + "dimensions: { a: { sql: `${CUBE}.a`, type: `string` } } });" + ) + _, report = _convert(src) + assert not report.by_category(CubeIssueCategory.FILTER_PARAMS_VARIABLE) + + +def test_generated_model_sql_parses_after_probe_render(): + # The converter must validate the sentinel-resolved SQL; a model that + # survives conversion has parseable SQL once blocks collapse + vars fill. + models, _ = _convert(_rr_source()) + assert "RrDrivers" in models # survived offline validation diff --git a/tests/test_cube_extends.py b/tests/test_cube_extends.py new file mode 100644 index 00000000..3e6c219b --- /dev/null +++ b/tests/test_cube_extends.py @@ -0,0 +1,103 @@ +"""Tests for Cube `extends` flattening (slayer/cube/extends.py + converter). + +DEV-1608 §5. Flatten base members into children (child wins); abstract bases +emitted hidden; cycles reported. +""" + +from slayer.cube.converter import CubeToSlayerConverter +from slayer.cube.models import CubeCube, CubeDimension, CubeMeasure, CubeProject +from slayer.cube.report import CubeIssueCategory + +DS = "test_ds" + + +def _convert(project: CubeProject): + result = CubeToSlayerConverter(project=project, data_source=DS).convert() + return {m.name: m for m in result.models}, result.report + + +def test_child_inherits_base_members(): + project = CubeProject(cubes=[ + CubeCube(name="base_events", sql_table="public.events", public=False, + dimensions=[CubeDimension(name="id", sql="{CUBE}.id", type="number", + primary_key=True), + CubeDimension(name="event_type", sql="{CUBE}.event_type", + type="string")], + measures=[CubeMeasure(name="count", type="count")]), + CubeCube(name="clicks", extends="base_events", sql_table="public.clicks", + measures=[CubeMeasure(name="click_value", type="sum", + sql="{CUBE}.value")]), + ]) + models, _ = _convert(project) + clicks = models["clicks"] + # Inherited dimensions + measure, plus the child's own measure. + assert clicks.get_column("id") is not None + assert clicks.get_column("event_type") is not None + assert clicks.get_measure("count") is not None + assert clicks.get_measure("click_value") is not None + assert clicks.sql_table == "public.clicks" # child source wins + + +def test_abstract_base_emitted_hidden(): + project = CubeProject(cubes=[ + CubeCube(name="base_events", sql_table="public.events", public=False, + dimensions=[CubeDimension(name="id", sql="{CUBE}.id", type="number")]), + CubeCube(name="clicks", extends="base_events", sql_table="public.clicks"), + ]) + models, _ = _convert(project) + assert models["base_events"].hidden is True + assert models["clicks"].hidden is False + + +def test_child_overrides_inherited_member(): + project = CubeProject(cubes=[ + CubeCube(name="base", sql_table="public.base", public=False, + dimensions=[CubeDimension(name="label", sql="{CUBE}.old_label", + type="string")]), + CubeCube(name="child", extends="base", sql_table="public.child", + dimensions=[CubeDimension(name="label", sql="{CUBE}.new_label", + type="string")]), + ]) + models, _ = _convert(project) + # Child wins on name conflict. + assert models["child"].get_column("label").sql == "new_label" + + +def test_multi_level_extends_chain(): + project = CubeProject(cubes=[ + CubeCube(name="a", sql_table="public.a", public=False, + dimensions=[CubeDimension(name="x", sql="{CUBE}.x", type="number")]), + CubeCube(name="b", extends="a", sql_table="public.b", public=False, + dimensions=[CubeDimension(name="y", sql="{CUBE}.y", type="number")]), + CubeCube(name="c", extends="b", sql_table="public.c", + dimensions=[CubeDimension(name="z", sql="{CUBE}.z", type="number")]), + ]) + models, _ = _convert(project) + c = models["c"] + assert c.get_column("x") is not None # from a, transitively + assert c.get_column("y") is not None # from b + assert c.get_column("z") is not None # own + + +def test_extends_cycle_is_reported(): + project = CubeProject(cubes=[ + CubeCube(name="a", extends="b", sql_table="public.a"), + CubeCube(name="b", extends="a", sql_table="public.b"), + ]) + _, report = _convert(project) + assert any(i.category == CubeIssueCategory.EXTENDS_CYCLE for i in report.issues) + + +def test_extends_cycle_flattens_without_inheritance(): + """Every node on the cycle keeps only its own members — no ancestor merges + a partially-resolved cyclic node.""" + project = CubeProject(cubes=[ + CubeCube(name="a", extends="b", sql_table="public.a", + dimensions=[CubeDimension(name="a_col", sql="{CUBE}.a_col", type="number")]), + CubeCube(name="b", extends="a", sql_table="public.b", + dimensions=[CubeDimension(name="b_col", sql="{CUBE}.b_col", type="number")]), + ]) + models, _ = _convert(project) + assert models["a"].get_column("a_col") is not None + assert models["a"].get_column("b_col") is None # did NOT inherit across the cycle + assert models["b"].get_column("a_col") is None diff --git a/tests/test_cube_filter_params.py b/tests/test_cube_filter_params.py new file mode 100644 index 00000000..4f0c01fe --- /dev/null +++ b/tests/test_cube_filter_params.py @@ -0,0 +1,160 @@ +"""Shared FILTER_PARAMS translation (slayer/cube/filter_params.py, DEV-1730). + +The FILTER_PARAMS micro-grammar maps to SLayer Mode-A optional blocks / plain +vars. Ref construction is AST/text-agnostic (the JS parser feeds arrow segments, +the YAML path feeds a string col-expr); requiredness (block vs bare) is applied +downstream by the converter. These pin the emitted SQL text for each form. +""" + + +from slayer.cube.filter_params import ( + apply_filter_params, + build_arrow_ref, + build_string_ref, + filter_param_sentinel, + parse_string_filter_params, + render_filter_param, +) +from slayer.cube.models import CubeFilterParamRef + +# ── ref construction ──────────────────────────────────────────────────────── + + +def test_string_ref_builds_in_list_body(): + ref = build_string_ref( + cube="RrDrivers", member="brand", col_expr='pr."BRAND"', + sentinel=filter_param_sentinel(0), + ) + assert ref.kind == "string" + assert ref.member == "brand" + assert ref.body_template == 'pr."BRAND" IN ({brand})' + assert ref.var_names == ["brand"] + + +def test_arrow_value_ref_single_prequoted_param(): + ref = build_arrow_ref( + cube="RrDrivers", member="fulfillment_date", + segments=[("param", "from")], sentinel=filter_param_sentinel(0), + ) + assert ref.kind == "arrow_value" + assert ref.body_template == "'{fulfillment_date_from}'" + assert ref.var_names == ["fulfillment_date_from"] + + +def test_arrow_range_ref_concat_body(): + ref = build_arrow_ref( + cube="RrDrivers", member="fulfillment_date", + segments=[ + ("lit", 'fol."FD" >= '), + ("param", "from"), + ("lit", ' AND fol."FD" <= '), + ("param", "to"), + ], + sentinel=filter_param_sentinel(0), + ) + assert ref.kind == "arrow_range" + assert ref.body_template == ( + 'fol."FD" >= \'{fulfillment_date_from}\' ' + 'AND fol."FD" <= \'{fulfillment_date_to}\'' + ) + assert ref.var_names == ["fulfillment_date_from", "fulfillment_date_to"] + + +def test_arrow_ref_preserves_ly_shifted_wrapper(): + ref = build_arrow_ref( + cube="RrDrivers", member="fulfillment_date", + segments=[ + ("lit", 'fol."FD" >= DATEADD(YEAR, -1, '), + ("param", "from"), + ("lit", ") AND fol.\"FD\" <= DATEADD(YEAR, -1, "), + ("param", "to"), + ("lit", ")"), + ], + sentinel=filter_param_sentinel(0), + ) + assert "DATEADD(YEAR, -1, '{fulfillment_date_from}')" in ref.body_template + assert "DATEADD(YEAR, -1, '{fulfillment_date_to}')" in ref.body_template + + +# ── render: optional (block) vs required (bare) ───────────────────────────── + + +def test_render_optional_wraps_in_block(): + ref = build_string_ref( + cube="c", member="brand", col_expr="brand", sentinel=filter_param_sentinel(0) + ) + assert render_filter_param(ref, required=False) == "{? brand IN ({brand}) ?}" + + +def test_render_required_is_bare(): + ref = build_string_ref( + cube="c", member="brand", col_expr="brand", sentinel=filter_param_sentinel(0) + ) + assert render_filter_param(ref, required=True) == "brand IN ({brand})" + + +def test_apply_filter_params_replaces_sentinels_by_requiredness(): + s0, s1 = filter_param_sentinel(0), filter_param_sentinel(1) + r0 = build_arrow_ref( + cube="c", member="fulfillment_date", segments=[("param", "from")], sentinel=s0 + ) + r1 = build_string_ref(cube="c", member="brand", col_expr="brand", sentinel=s1) + text = f"SELECT {s0}::TIMESTAMP AS d FROM t WHERE 1=1 AND {s1}" + out = apply_filter_params( + text, [r0, r1], required_members={"fulfillment_date"} + ) + assert out == ( + "SELECT '{fulfillment_date_from}'::TIMESTAMP AS d " + "FROM t WHERE 1=1 AND {? brand IN ({brand}) ?}" + ) + + +# ── YAML text path (string-arg only) ──────────────────────────────────────── + + +def test_parse_string_filter_params_extracts_and_sentinels(): + text = "WHERE 1=1 AND {FILTER_PARAMS.orders.status.filter('o.status')}" + result = parse_string_filter_params(text, host_cube="orders") + assert len(result.refs) == 1 + ref = result.refs[0] + assert ref.member == "status" + assert ref.body_template == "o.status IN ({status})" + # the ref's sentinel replaced the FILTER_PARAMS ref in the returned text + assert "FILTER_PARAMS" not in result.text + assert ref.sentinel in result.text + assert not result.unsupported + + +def test_parse_string_filter_params_arrow_in_yaml_is_unsupported(): + text = "AND {FILTER_PARAMS.orders.d.filter((from, to) => from)}" + result = parse_string_filter_params(text, host_cube="orders") + assert result.refs == [] + assert len(result.unsupported) == 1 + assert "d" in result.unsupported[0].raw or "d" == result.unsupported[0].member + + +def test_parse_string_filter_params_cross_cube_is_unsupported(): + # Stage-1: the cube segment must equal the host cube. + text = "AND {FILTER_PARAMS.other.status.filter('status')}" + result = parse_string_filter_params(text, host_cube="orders") + assert result.refs == [] + assert len(result.unsupported) == 1 + + +def test_parse_string_filter_params_no_filter_params_is_noop(): + text = "WHERE deleted_at IS NULL" + result = parse_string_filter_params(text, host_cube="orders") + assert result.text == text + assert result.refs == [] + assert result.unsupported == [] + + +def test_filter_param_ref_is_cube_model_validatable(): + # CubeCube must be able to carry these (JS parser populates them). + d = { + "cube": "c", "member": "brand", "kind": "string", + "body_template": "brand IN ({brand})", "var_names": ["brand"], + "sentinel": filter_param_sentinel(0), + } + ref = CubeFilterParamRef.model_validate(d) + assert ref.member == "brand" diff --git a/tests/test_cube_js_e2e_duckdb.py b/tests/test_cube_js_e2e_duckdb.py new file mode 100644 index 00000000..3ea18a3a --- /dev/null +++ b/tests/test_cube_js_e2e_duckdb.py @@ -0,0 +1,156 @@ +"""End-to-end: import the anonymized RrDrivers JS fixture and execute it on +DuckDB (in-process, DEV-1730 acceptance bar). + +Proves the FILTER_PARAMS-bearing single-CTE-chain cube is not just representable +but *executable* as a SLayer sql-mode model: required date pushdown, optional +categorical IN pushdowns, and the required-omitted clean raise. +""" +import os + +import pytest + +duckdb = pytest.importorskip("duckdb") + +from slayer.core.models import DatasourceConfig +from slayer.core.query import SlayerQuery, extract_model_variables +from slayer.cube.converter import CubeToSlayerConverter +from slayer.cube.js_parser import parse_cube_js +from slayer.cube.models import CubeProject +from slayer.engine.query_engine import SlayerQueryEngine +from slayer.storage.yaml_storage import YAMLStorage + +_FIXTURE = os.path.join( + os.path.dirname(__file__), "fixtures", "cube_js", "rr_drivers.js" +) +_FULL_VARS = {"fulfillment_date_from": "2025-01-01", "fulfillment_date_to": "2025-12-31"} + + +def _seed(db_path: str) -> None: + con = duckdb.connect(db_path) + con.execute("CREATE SCHEMA analytics") + con.execute( + "CREATE TABLE analytics.fact_lines(fulfillment_date DATE, product_key INT, " + "market TEXT, quantity_fulfilled INT, quantity_returned INT, date_returned DATE)" + ) + con.execute("CREATE TABLE analytics.dim_items(product_key INT, category TEXT, brand TEXT)") + con.executemany( + "INSERT INTO analytics.dim_items VALUES (?,?,?)", + [(1, "Shoes", "Acme"), (2, "Hats", "Zeta")], + ) + con.executemany( + "INSERT INTO analytics.fact_lines VALUES (?,?,?,?,?,?)", + [ + ("2025-03-01", 1, "US", 10, 2, "2025-06-01"), + ("2025-04-01", 1, "EU", 5, 1, None), + ("2025-05-01", 2, "US", 8, 4, "2025-03-15"), + ("2024-03-01", 1, "US", 6, 3, "2024-06-01"), + ("2024-04-01", 2, "US", 4, 1, "2024-02-01"), + ], + ) + con.close() + + +async def _import_and_engine(tmp_path): + db_path = str(tmp_path / "rr.duckdb") + _seed(db_path) + storage = YAMLStorage(base_dir=str(tmp_path / "store")) + await storage.save_datasource( + DatasourceConfig(name="rr_ds", type="duckdb", database=db_path) + ) + with open(_FIXTURE, encoding="utf-8") as fh: + source = fh.read() + parsed = parse_cube_js(source) + assert not [i for i in parsed.issues if i.severity == "error"], \ + [i.message for i in parsed.issues] + result = CubeToSlayerConverter( + project=CubeProject(cubes=parsed.cubes, views=parsed.views), + data_source="rr_ds", parse_issues=parsed.issues, + ).convert() + assert not result.report.has_errors, [i.message for i in result.report.issues if i.severity == "error"] + model = next(m for m in result.models if m.name == "RrDrivers") + await storage.save_model(model) + return SlayerQueryEngine(storage=storage), model + + +def _sole_value(resp): + assert resp.row_count == 1, f"expected 1 row: {resp.data}" + return next(iter(resp.data[0].values())) + + +async def test_import_yields_expected_variable_contract(tmp_path): + _, model = await _import_and_engine(tmp_path) + v = extract_model_variables(model) + assert set(v.required) == {"fulfillment_date_from", "fulfillment_date_to"} + assert set(v.optional) == {"brand", "market", "category"} + + +async def test_count_with_optional_omitted_is_unfiltered(tmp_path): + engine, _ = await _import_and_engine(tmp_path) + q = SlayerQuery( + source_model="RrDrivers", measures=[{"formula": "count"}], variables=_FULL_VARS, + ) + resp = await engine.execute(q) + assert _sole_value(resp) == 3 + + +async def test_count_with_brand_pushdown(tmp_path): + engine, _ = await _import_and_engine(tmp_path) + q = SlayerQuery( + source_model="RrDrivers", measures=[{"formula": "count"}], + variables={**_FULL_VARS, "brand": ["Acme"]}, + ) + resp = await engine.execute(q) + assert _sole_value(resp) == 2 + + +async def test_count_with_scalar_brand_pushdown(tmp_path): + """A bare scalar for an importer-generated ``IN ({var})`` pushdown must + behave exactly like the one-element list — the importer wrote the + parentheses, so the caller has nowhere to put the quotes. Without the + declared-list coercion this renders ``IN (Acme)`` and DuckDB fails on the + unknown column reference.""" + engine, _ = await _import_and_engine(tmp_path) + q = SlayerQuery( + source_model="RrDrivers", measures=[{"formula": "count"}], + variables={**_FULL_VARS, "brand": "Acme"}, + ) + resp = await engine.execute(q) + assert _sole_value(resp) == 2 + + +async def test_scalar_and_list_pushdowns_agree(tmp_path): + engine, _ = await _import_and_engine(tmp_path) + + async def _count(brand): + q = SlayerQuery( + source_model="RrDrivers", measures=[{"formula": "count"}], + variables={**_FULL_VARS, "brand": brand}, + ) + return _sole_value(await engine.execute(q)) + + assert await _count("Zeta") == await _count(["Zeta"]) == 1 + + +async def test_grouped_max_measure(tmp_path): + engine, _ = await _import_and_engine(tmp_path) + q = SlayerQuery( + source_model="RrDrivers", dimensions=["category"], + measures=[{"formula": "qty_ty"}], variables=_FULL_VARS, + ) + resp = await engine.execute(q) + by_cat = {} + for row in resp.data: + cat = row["RrDrivers.category"] + qty = next(v for k, v in row.items() if "qty_ty" in k) + by_cat[cat] = qty + assert by_cat == {"Shoes": 10, "Hats": 8} + + +async def test_required_filter_omitted_raises_and_names_variable(tmp_path): + engine, _ = await _import_and_engine(tmp_path) + q = SlayerQuery( + source_model="RrDrivers", measures=[{"formula": "count"}], + variables={"brand": ["Acme"]}, # required date vars omitted + ) + with pytest.raises(Exception, match="fulfillment_date_from"): + await engine.execute(q) diff --git a/tests/test_cube_js_parser.py b/tests/test_cube_js_parser.py new file mode 100644 index 00000000..bd17bbad --- /dev/null +++ b/tests/test_cube_js_parser.py @@ -0,0 +1,319 @@ +"""JavaScript Cube-config front-end (slayer/cube/js_parser.py, DEV-1730). + +Parses the declarative ``cube('Name', {...})`` / ``view(...)`` subset via an +esprima ESTree AST into the same ``CubeCube`` / ``CubeView`` shapes the YAML +front-end produces, so the converter is front-end-agnostic. FILTER_PARAMS +interpolations are captured as structured refs (sentinels in the surface text). +""" + + +from slayer.cube.js_parser import parse_cube_js +from slayer.cube.report import CubeIssueCategory + + +def _one_cube(source: str): + result = parse_cube_js(source) + assert not result.issues, [i.message for i in result.issues] + assert len(result.cubes) == 1 + return result.cubes[0] + + +# ── basic shapes ──────────────────────────────────────────────────────────── + + +def test_template_literal_cube_name_and_sql_table(): + cube = _one_cube( + "cube(`Orders`, { sql_table: `public.orders`, " + "dimensions: { id: { sql: `${CUBE}.id`, type: `number`, primaryKey: true } } });" + ) + assert cube.name == "Orders" + assert cube.sql_table == "public.orders" + assert len(cube.dimensions) == 1 + assert cube.dimensions[0].name == "id" + assert cube.dimensions[0].primary_key is True # camelCase primaryKey normalised + + +def test_string_literal_cube_name(): + cube = _one_cube("cube('Orders', { sql_table: 'public.orders' });") + assert cube.name == "Orders" + + +def test_dimensions_object_becomes_ordered_list_with_names(): + cube = _one_cube( + "cube(`C`, { sql_table: `t`, dimensions: {" + " a: { sql: `${CUBE}.a`, type: `string` }," + " b: { sql: `${CUBE}.b`, type: `string` }," + " c: { sql: `${CUBE}.c`, type: `string` }" + "} });" + ) + assert [d.name for d in cube.dimensions] == ["a", "b", "c"] + + +def test_cube_ref_interpolation_becomes_single_brace(): + cube = _one_cube( + "cube(`C`, { sql_table: `t`, dimensions: {" + " pk: { sql: `${CUBE}.\"ID\" || '|' || ${CUBE}.effect`, type: `string`, primaryKey: true, public: false }" + "} });" + ) + dim = cube.dimensions[0] + assert dim.sql == '{CUBE}."ID" || \'|\' || {CUBE}.effect' + assert dim.primary_key is True + assert dim.public is False + + +def test_measure_named_count_and_max_with_format(): + cube = _one_cube( + "cube(`C`, { sql_table: `t`, measures: {" + " count: { type: `count`, title: `Driver Count` }," + " share_ty: { sql: `${CUBE}.share_ty`, type: `max`, format: `percent`, description: `Share TY` }" + "} });" + ) + by_name = {m.name: m for m in cube.measures} + assert by_name["count"].type == "count" + assert by_name["count"].title == "Driver Count" + assert by_name["share_ty"].type == "max" + assert by_name["share_ty"].format == "percent" + + +def test_calc_measure_interpolation_becomes_brace_refs(): + cube = _one_cube( + "cube(`C`, { sql_table: `t`, measures: {" + " share_change: { sql: `${share_ty} - ${share_ly}`, type: `number`, format: `percent` }" + "} });" + ) + m = cube.measures[0] + assert m.sql == "{share_ty} - {share_ly}" + assert m.type == "number" + + +def test_meta_object_preserved_verbatim_including_capitals(): + cube = _one_cube( + "cube(`C`, { sql_table: `t`, dimensions: {" + " d: { sql: `${CUBE}.d`, type: `time`, meta: { required: true, camelKey: 42 } }" + "} });" + ) + meta = cube.dimensions[0].meta + assert meta == {"required": True, "camelKey": 42} + + +def test_line_and_block_comments_tolerated(): + cube = _one_cube( + "cube(`C`, {\n" + " // a line comment\n" + " sql_table: `t`, /* block */ dimensions: {" + " a: { sql: `${CUBE}.a`, type: `string` } // trailing\n" + " }\n" + "});" + ) + assert cube.name == "C" + assert [d.name for d in cube.dimensions] == ["a"] + + +def test_empty_pre_aggregations_object_ok(): + cube = _one_cube("cube(`C`, { sql_table: `t`, pre_aggregations: {} });") + assert cube.name == "C" + + +def test_negative_number_literal(): + cube = _one_cube( + "cube(`C`, { sql_table: `t`, dimensions: {" + " a: { sql: `${CUBE}.a`, type: `number`, meta: { lo: -5 } }" + "} });" + ) + assert cube.dimensions[0].meta == {"lo": -5} + + +# ── multiple objects / views ──────────────────────────────────────────────── + + +def test_multiple_cubes_in_one_file(): + result = parse_cube_js( + "cube(`A`, { sql_table: `a` });\ncube(`B`, { sql_table: `b` });" + ) + assert {c.name for c in result.cubes} == {"A", "B"} + + +def test_view_is_parsed_with_parity(): + result = parse_cube_js( + "view(`MyView`, { cubes: [ { join_path: `orders`, includes: `*` } ] });" + ) + assert not result.issues, [i.message for i in result.issues] + assert len(result.views) == 1 + assert result.views[0].name == "MyView" + assert result.views[0].cubes[0].join_path == "orders" + + +def test_module_exports_wrapper_recognised(): + result = parse_cube_js("module.exports = cube(`C`, { sql_table: `t` });") + assert len(result.cubes) == 1 + assert result.cubes[0].name == "C" + + +def test_es_module_export_default_recognised(): + # `export default` requires module parsing — parseScript rejects it, so the + # parser must retry with parseModule (DEV-1730 review). + result = parse_cube_js("export default cube(`C`, { sql_table: `t` });") + assert not result.issues, [i.message for i in result.issues] + assert [c.name for c in result.cubes] == ["C"] + + +def test_es_module_with_import_recognised(): + result = parse_cube_js( + "import { foo } from 'helpers';\n" + "export default cube(`C`, { sql_table: `t` });" + ) + assert [c.name for c in result.cubes] == ["C"] + + +def test_computed_member_key_skips_only_that_member(): + # A computed key ([name]: {...}) is dynamic; per-member isolation must skip + # just that member, not the whole cube (DEV-1730 review). + result = parse_cube_js( + "cube(`C`, { sql_table: `t`, dimensions: {" + " ok: { sql: `${CUBE}.ok`, type: `string` }," + " [dynamicName]: { sql: `${CUBE}.x`, type: `string` }" + "} });" + ) + assert len(result.cubes) == 1 + assert [d.name for d in result.cubes[0].dimensions] == ["ok"] + assert result.issues + + +def test_skipped_dynamic_member_does_not_leak_filter_params_refs(): + # A member that captures a FILTER_PARAMS ref in one field, then fails on a + # later dynamic field, must roll back the ref (else it dangles on the cube). + result = parse_cube_js( + "cube(`C`, { sql: `SELECT 1`, dimensions: {" + " bad: { sql: `${FILTER_PARAMS.C.brand.filter('b')}`, type: helperCall() }" + "} });" + ) + assert len(result.cubes) == 1 + assert result.cubes[0].filter_params == [] # rolled back with the member + + +# ── FILTER_PARAMS capture ─────────────────────────────────────────────────── + + +def test_filter_params_string_form_captured_as_ref(): + cube = _one_cube( + "cube(`RrDrivers`, { sql: `SELECT * FROM t WHERE 1=1 " + "AND ${FILTER_PARAMS.RrDrivers.brand.filter('pr.\"BRAND\"')}`, " + "dimensions: { brand: { sql: `${CUBE}.\"BRAND\"`, type: `string` } } });" + ) + assert len(cube.filter_params) == 1 + ref = cube.filter_params[0] + assert ref.member == "brand" + assert ref.kind == "string" + assert ref.body_template == 'pr."BRAND" IN ({brand})' + # the sentinel replaced the FILTER_PARAMS interpolation in the sql text + assert "FILTER_PARAMS" not in cube.sql + assert ref.sentinel in cube.sql + + +def test_filter_params_arrow_value_and_range_captured(): + cube = _one_cube( + "cube(`RrDrivers`, { sql: `SELECT " + "${FILTER_PARAMS.RrDrivers.fulfillment_date.filter((from, to) => from)}::TIMESTAMP AS d " + "FROM t WHERE " + "${FILTER_PARAMS.RrDrivers.fulfillment_date.filter((from, to) => 'x >= ' + from + ' AND x <= ' + to)}`, " + "dimensions: { fulfillment_date: { sql: `${CUBE}.d`, type: `time`, meta: { required: true } } } });" + ) + kinds = sorted(r.kind for r in cube.filter_params) + assert kinds == ["arrow_range", "arrow_value"] + rng = next(r for r in cube.filter_params if r.kind == "arrow_range") + assert rng.var_names == ["fulfillment_date_from", "fulfillment_date_to"] + assert "x >= '{fulfillment_date_from}'" in rng.body_template + + +# ── error / report paths ──────────────────────────────────────────────────── + + +def test_syntax_error_reports_parse_error(): + result = parse_cube_js("cube(`C`, { sql_table: ") + assert result.cubes == [] + assert any(i.category == CubeIssueCategory.PARSE_ERROR for i in result.issues) + + +def test_dynamic_value_reports_and_skips_member_not_cube(): + # A dimension whose sql is a bare identifier reference (not a literal) + # is dynamic -> that dimension is skipped with an issue, cube still built. + result = parse_cube_js( + "cube(`C`, { sql_table: `t`, dimensions: {" + " ok: { sql: `${CUBE}.ok`, type: `string` }," + " bad: { sql: someHelper(), type: `string` }" + "} });" + ) + assert len(result.cubes) == 1 + cube = result.cubes[0] + assert [d.name for d in cube.dimensions] == ["ok"] + assert result.issues # a dynamic-construct issue was reported + + +def test_spread_in_object_reports_issue(): + result = parse_cube_js( + "cube(`C`, { ...base, sql_table: `t` });" + ) + assert result.issues + + +def test_bare_identifier_value_is_dynamic_and_skipped(): + result = parse_cube_js( + "cube(`C`, { sql_table: `t`, dimensions: {" + " ok: { sql: `${CUBE}.ok`, type: `string` }," + " bad: { sql: someConst, type: `string` }" + "} });" + ) + assert len(result.cubes) == 1 + assert [d.name for d in result.cubes[0].dimensions] == ["ok"] + assert result.issues + + +def test_unknown_capitalized_key_does_not_break_parse(): + # An unmodelled camelCase key must not be blanket-normalised into a known + # field; the cube still parses (extra keys are ignored by the model). + result = parse_cube_js( + "cube(`C`, { sql_table: `t`, myCustomThing: `whatever`," + " dimensions: { a: { sql: `${CUBE}.a`, type: `string` } } });" + ) + assert len(result.cubes) == 1 + assert result.cubes[0].name == "C" + assert [d.name for d in result.cubes[0].dimensions] == ["a"] + + +# ── interpolation forms ───────────────────────────────────────────────────── + + +def test_member_and_dotted_interpolation_forms(): + cube = _one_cube( + "cube(`C`, { sql_table: `t`, dimensions: {" + " a: { sql: `${member}`, type: `string` }," + " b: { sql: `${a.b}.x`, type: `string` }" + "} });" + ) + by_name = {d.name: d for d in cube.dimensions} + assert by_name["a"].sql == "{member}" + assert by_name["b"].sql == "{a.b}.x" + + +# ── template-literal escapes (cooked authoritative) ───────────────────────── + + +def test_newline_preserved_in_template(): + cube = _one_cube("cube(`C`, { sql: `line1\nline2` });") + assert cube.sql == "line1\nline2" + + +def test_escaped_backslash_becomes_single_backslash(): + cube = _one_cube(r"cube(`C`, { sql: `a \\ b` });") + assert cube.sql == r"a \ b" + + +def test_escaped_backtick_preserved(): + cube = _one_cube(r"cube(`C`, { sql: `a \` b` });") + assert cube.sql == "a ` b" + + +def test_escaped_interpolation_is_literal_dollar_brace(): + # \${x} is not an interpolation — cooked to a literal ${x}. + cube = _one_cube(r"cube(`C`, { sql: `SELECT \${x} FROM t` });") + assert cube.sql == "SELECT ${x} FROM t" diff --git a/tests/test_cube_parser.py b/tests/test_cube_parser.py new file mode 100644 index 00000000..50d8d62f --- /dev/null +++ b/tests/test_cube_parser.py @@ -0,0 +1,163 @@ +"""Tests for the Cube project parser (slayer/cube/parser.py). + +DEV-1608 §2. Walk a directory, parse cubes:/views:, skip + report Jinja +(file-level and member-level) and malformed files without aborting the run. +""" + +import os +import textwrap + +from slayer.cube.parser import parse_cube_project +from slayer.cube.report import CubeIssueCategory + +FIXTURE = os.path.join(os.path.dirname(__file__), "fixtures", "cube_project") + + +def test_parses_cubes_and_views(): + project, _issues = parse_cube_project(FIXTURE) + cube_names = {c.name for c in project.cubes} + view_names = {v.name for v in project.views} + assert {"orders", "customers", "base_events", "clicks"} <= cube_names + assert "orders_overview" in view_names + + +def test_orders_cube_fields_populated(): + project, _ = parse_cube_project(FIXTURE) + orders = next(c for c in project.cubes if c.name == "orders") + assert orders.sql_table == "public.orders" + assert {m.name for m in orders.measures} >= {"count", "total_revenue", "completed_revenue"} + assert orders.joins[0].name == "customers" + assert orders.pre_aggregations # captured for unmapped-infra stashing + + +def test_file_level_jinja_is_skipped_and_reported(): + _project, issues = parse_cube_project(FIXTURE) + assert any(i.category == CubeIssueCategory.REQUIRES_TEMPLATING for i in issues) + + +def test_member_level_jinja_skips_member_keeps_cube(): + project, _issues = parse_cube_project(FIXTURE) + tenant = next((c for c in project.cubes if c.name == "tenant_scoped"), None) + assert tenant is not None + dim_names = {d.name for d in tenant.dimensions} + assert "id" in dim_names # plain member kept + assert "tenant" not in dim_names # templated member dropped + + +def test_malformed_cube_is_reported_not_fatal(): + project, issues = parse_cube_project(FIXTURE) + # The nameless cube in malformed.yml is dropped... + assert all(c.name != "orphan" for c in project.cubes) + # ...via a parse_error, and the rest of the project still parsed. + assert any(i.category == CubeIssueCategory.PARSE_ERROR for i in issues) + assert any(c.name == "orders" for c in project.cubes) + + +def test_single_object_cubes_block_tolerated(tmp_path): + (tmp_path / "single.yml").write_text(textwrap.dedent(""" + cubes: + name: solo + sql_table: public.solo + dimensions: + - name: id + sql: "{CUBE}.id" + type: number + """)) + project, _ = parse_cube_project(str(tmp_path)) + assert any(c.name == "solo" for c in project.cubes) + + +def test_single_object_views_block_tolerated(tmp_path): + (tmp_path / "v.yml").write_text(textwrap.dedent(""" + views: + name: solo_view + cubes: + - join_path: orders + includes: ["status"] + """)) + project, _ = parse_cube_project(str(tmp_path)) + assert any(v.name == "solo_view" for v in project.views) + + +def test_member_level_jinja_in_measure_sql_skips_measure(tmp_path): + (tmp_path / "m.yml").write_text(textwrap.dedent(""" + cubes: + - name: orders + sql_table: public.orders + measures: + - name: plain + type: count + - name: templated + type: sum + sql: "{{ user_attr('scale') }} * {CUBE}.amount" + dimensions: + - name: id + sql: "{CUBE}.id" + type: number + """)) + project, issues = parse_cube_project(str(tmp_path)) + orders = next(c for c in project.cubes if c.name == "orders") + names = {m.name for m in orders.measures} + assert "plain" in names + assert "templated" not in names + assert any(i.category == CubeIssueCategory.REQUIRES_TEMPLATING for i in issues) + + +def test_hidden_dirs_and_target_skipped(tmp_path): + (tmp_path / ".hidden").mkdir() + (tmp_path / ".hidden" / "x.yml").write_text( + "cubes:\n - name: ghost\n sql_table: public.ghost\n") + # `target/` (dbt/Cube build output) is skipped too — cover both so the name + # matches the behavior. + (tmp_path / "target").mkdir() + (tmp_path / "target" / "y.yml").write_text( + "cubes:\n - name: built\n sql_table: public.built\n") + project, _ = parse_cube_project(str(tmp_path)) + assert all(c.name not in ("ghost", "built") for c in project.cubes) + + +def test_unreadable_file_is_reported_not_fatal(tmp_path): + # A broken symlink with a .yml extension raises OSError on open — it must be + # reported like a malformed file, not abort the whole import. + (tmp_path / "broken.yml").symlink_to(tmp_path / "nonexistent.yml") + (tmp_path / "good.yml").write_text( + "cubes:\n - name: ok\n sql_table: public.ok\n") + project, issues = parse_cube_project(str(tmp_path)) + assert any(c.name == "ok" for c in project.cubes) + + +def test_invalid_utf8_file_is_reported_not_fatal(tmp_path): + # Invalid UTF-8 raises UnicodeDecodeError (a ValueError, not OSError) on + # read — it must be reported as a PARSE_ERROR, not abort the whole import. + (tmp_path / "bad.yml").write_bytes(b"cubes:\n - name: \xff\n") + (tmp_path / "good.yml").write_text( + "cubes:\n - name: ok\n sql_table: public.ok\n") + project, issues = parse_cube_project(str(tmp_path)) + assert any(c.name == "ok" for c in project.cubes) + assert any(i.category == CubeIssueCategory.PARSE_ERROR for i in issues) + assert any(i.category == CubeIssueCategory.PARSE_ERROR for i in issues) + + +def test_invalid_utf8_js_file_is_reported_not_fatal(tmp_path): + # DEV-1730: a non-UTF-8 .js file raises UnicodeDecodeError on read; the JS + # discovery path must report it, not abort the import. + (tmp_path / "bad.js").write_bytes(b"cube(`\xff`, {})") + (tmp_path / "good.js").write_text("cube(`ok`, { sql_table: `public.ok` });") + project, issues = parse_cube_project(str(tmp_path)) + assert any(c.name == "ok" for c in project.cubes) + assert any(i.category == CubeIssueCategory.PARSE_ERROR for i in issues) + + +def test_js_files_discovered_alongside_yaml(tmp_path): + (tmp_path / "a.yml").write_text( + "cubes:\n - name: y_cube\n sql_table: public.y\n") + (tmp_path / "b.js").write_text("cube(`js_cube`, { sql_table: `public.j` });") + project, _ = parse_cube_project(str(tmp_path)) + names = {c.name for c in project.cubes} + assert {"y_cube", "js_cube"} <= names + + +def test_empty_dir_yields_empty_project(tmp_path): + project, _ = parse_cube_project(str(tmp_path)) + assert project.cubes == [] + assert project.views == [] diff --git a/tests/test_cube_refs.py b/tests/test_cube_refs.py new file mode 100644 index 00000000..b132a86b --- /dev/null +++ b/tests/test_cube_refs.py @@ -0,0 +1,118 @@ +"""Tests for the Cube curly-reference translator (slayer/cube/refs.py). + +DEV-1608 §3 / §4.4. `{CUBE}` / `{member}` / `{cube.member}` are Cube's +single-brace SQL-ref syntax — distinct from Jinja's `{{ }}` / `{% %}`. +""" + + +import pytest + +from slayer.cube.refs import contains_jinja, parse_join_on, translate_cube_refs + + +def test_translate_rejects_invalid_mode(): + with pytest.raises(ValueError): + translate_cube_refs("{CUBE}.x", mode="bogus", cube="orders") + + +# ── Jinja detection ──────────────────────────────────────────────────────── + +def test_contains_jinja_double_brace(): + assert contains_jinja("{{ env_var('SCHEMA') }}.events") + + +def test_contains_jinja_block(): + assert contains_jinja("{% for t in tables %}") + + +def test_single_brace_cube_ref_is_not_jinja(): + assert not contains_jinja("{CUBE}.amount") + assert not contains_jinja("{customers.id}") + assert not contains_jinja("SUM({CUBE}.amount)") + + +# ── Mode A (SQL) translation ─────────────────────────────────────────────── + +def test_cube_dot_col_to_bare(): + assert translate_cube_refs("{CUBE}.amount", mode="sql", cube="orders") == "amount" + + +def test_same_cube_member_to_bare(): + out = translate_cube_refs( + "{first_name} || ' ' || {last_name}", mode="sql", cube="people" + ) + assert out == "first_name || ' ' || last_name" + + +def test_cross_cube_member_single_hop(): + assert ( + translate_cube_refs("{customers.name}", mode="sql", cube="orders") + == "customers.name" + ) + + +def test_cross_cube_member_multi_hop(): + # `{a.b.c}` → SLayer multi-dot `a.b.c` (model-side `_fix_multidot_sql` later + # rewrites it to `a__b.c`). + assert ( + translate_cube_refs("{customers.regions.name}", mode="sql", cube="orders") + == "customers.regions.name" + ) + + +def test_translation_skips_doubled_quote_literal(): + # A `{CUBE}` inside a literal containing an escaped (doubled) quote must be + # left alone — the literal regex has to be doubled-quote aware. + out = translate_cube_refs("status = 'can''t {CUBE}'", mode="sql", cube="orders") + assert out == "status = 'can''t {CUBE}'" + + +def test_translation_skips_string_literals(): + # A `{CUBE}` inside a SQL string literal must be left untouched. + out = translate_cube_refs( + "CASE WHEN {CUBE}.status = '{CUBE}' THEN 1 END", mode="sql", cube="orders" + ) + assert out == "CASE WHEN status = '{CUBE}' THEN 1 END" + + +# ── Mode B (DSL) translation ─────────────────────────────────────────────── + +def test_dsl_measure_refs_to_bare_names(): + out = translate_cube_refs("{revenue} / {count}", mode="dsl", cube="orders") + assert out == "revenue / count" + + +# ── Join ON parsing ──────────────────────────────────────────────────────── + +def test_parse_join_on_simple_equality(): + pairs = parse_join_on( + "{CUBE}.customer_id = {customers.id}", + source_cube="orders", + target_cube="customers", + ) + assert pairs == [["customer_id", "id"]] + + +def test_parse_join_on_composite_key(): + pairs = parse_join_on( + "{CUBE}.a = {t.x} AND {CUBE}.b = {t.y}", + source_cube="o", + target_cube="t", + ) + assert pairs == [["a", "x"], ["b", "y"]] + + +def test_parse_join_on_non_equi_returns_none(): + assert ( + parse_join_on("{CUBE}.ts > {t.start}", source_cube="o", target_cube="t") + is None + ) + + +def test_parse_join_on_function_call_returns_none(): + assert ( + parse_join_on( + "LOWER({CUBE}.email) = {t.email}", source_cube="o", target_cube="t" + ) + is None + ) diff --git a/tests/test_cube_report.py b/tests/test_cube_report.py new file mode 100644 index 00000000..dec32079 --- /dev/null +++ b/tests/test_cube_report.py @@ -0,0 +1,99 @@ +"""Tests for the Cube conversion report shapes (slayer/cube/report.py). + +DEV-1608 §10. +""" + +import json + +from slayer.cube.report import ( + CubeConversionIssue, + CubeConversionReport, + CubeConversionResult, + CubeIssueCategory, +) + + +def test_issue_context_prefers_cube_then_view_then_member(): + assert CubeConversionIssue( + category=CubeIssueCategory.NO_SOURCE, message="x", cube="orders" + ).context == "orders" + assert CubeConversionIssue( + category=CubeIssueCategory.DISCONNECTED_VIEW, message="x", view="ov" + ).context == "ov" + assert CubeConversionIssue( + category=CubeIssueCategory.COMPLEX_MEASURE, message="x", member="aov" + ).context == "aov" + assert CubeConversionIssue( + category=CubeIssueCategory.PARSE_ERROR, message="x" + ).context == "general" + + +def test_report_filters_by_category_and_severity(): + report = CubeConversionReport() + report.add(CubeConversionIssue( + category=CubeIssueCategory.NO_SOURCE, severity="error", message="a")) + report.add(CubeConversionIssue( + category=CubeIssueCategory.LOSSY_MAPPING, severity="info", message="b")) + report.add(CubeConversionIssue( + category=CubeIssueCategory.LOSSY_MAPPING, severity="info", message="c")) + + assert len(report.by_category(CubeIssueCategory.LOSSY_MAPPING)) == 2 + assert len(report.by_severity("info")) == 2 + assert len(report.by_severity("error")) == 1 + assert report.has_errors + + +def test_report_has_no_errors_when_all_info_or_warning(): + report = CubeConversionReport() + report.add(CubeConversionIssue( + category=CubeIssueCategory.SEGMENT_AS_COLUMN, severity="info", message="x")) + assert not report.has_errors + + +def test_result_json_round_trips(): + result = CubeConversionResult( + report=CubeConversionReport( + issues=[CubeConversionIssue( + category=CubeIssueCategory.UNMAPPED_INFRA, + severity="warning", + cube="orders", + message="pre_aggregations dropped", + raw="name: main", + )], + model_count=3, + hidden_count=1, + view_count=1, + ) + ) + blob = result.model_dump_json() + parsed = json.loads(blob) + assert parsed["report"]["model_count"] == 3 + assert parsed["report"]["issues"][0]["category"] == "unmapped_infra" + + # Re-validate from the JSON to confirm the categories survive the round-trip. + restored = CubeConversionResult.model_validate_json(blob) + assert restored.report.issues[0].category == CubeIssueCategory.UNMAPPED_INFRA + assert restored.report.hidden_count == 1 + # `raw` fragment is preserved. + assert restored.report.issues[0].raw == "name: main" + + +def test_converter_derives_report_counts(): + """The converter must populate model/hidden/view counts on the report.""" + from slayer.cube.converter import CubeToSlayerConverter + from slayer.cube.models import CubeCube, CubeDimension, CubeProject, CubeView, CubeViewCubeRef + + project = CubeProject( + cubes=[ + CubeCube(name="orders", sql_table="public.orders", + dimensions=[CubeDimension(name="id", sql="{CUBE}.id", type="number")]), + CubeCube(name="internal", sql_table="public.internal", public=False, + dimensions=[CubeDimension(name="id", sql="{CUBE}.id", type="number")]), + ], + views=[CubeView(name="ov", cubes=[ + CubeViewCubeRef(join_path="orders", includes=["id"])])], + ) + result = CubeToSlayerConverter(project=project, data_source="ds").convert() + assert result.report.model_count == len(result.models) + assert result.report.hidden_count == 1 + assert result.report.view_count == 1 diff --git a/tests/test_cube_smoke.py b/tests/test_cube_smoke.py new file mode 100644 index 00000000..c32eb4c7 --- /dev/null +++ b/tests/test_cube_smoke.py @@ -0,0 +1,120 @@ +"""Smoke tests: enrich/generate SQL against converted Cube models. + +DEV-1608 §12 (Codex test-gap). These don't just assert converter *output* — they +push the converted models through the engine's enrichment + SQL generation, which +is where the §6/§4.4 mapping breaks actually surface (e.g. a facade measure that +referenced a Cube measure *name* instead of the underlying Column would raise +"Column '' not found" at enrichment). +""" + + +from slayer.core.query import ColumnRef, SlayerQuery +from slayer.cube.converter import CubeToSlayerConverter +from slayer.cube.models import ( + CubeCube, + CubeDimension, + CubeJoin, + CubeMeasure, + CubeMeasureFilter, + CubeProject, + CubeView, + CubeViewCubeRef, +) +from slayer.engine.query_engine import SlayerQueryEngine +from slayer.sql.generator import SQLGenerator +from slayer.storage.yaml_storage import YAMLStorage + +DS = "cube_ds" + + +def _orders_customers_view_project() -> CubeProject: + return CubeProject( + cubes=[ + CubeCube( + name="orders", sql_table="public.orders", + joins=[CubeJoin(name="customers", relationship="many_to_one", + sql="{CUBE}.customer_id = {customers.id}")], + measures=[ + CubeMeasure(name="count", type="count"), + CubeMeasure(name="total_revenue", type="sum", sql="{CUBE}.amount"), + CubeMeasure(name="completed_revenue", type="sum", sql="{CUBE}.amount", + filters=[CubeMeasureFilter(sql="{CUBE}.status = 'completed'")]), + ], + dimensions=[ + CubeDimension(name="id", sql="{CUBE}.id", type="number", primary_key=True), + CubeDimension(name="status", sql="{CUBE}.status", type="string"), + CubeDimension(name="customer_id", sql="{CUBE}.customer_id", type="number"), + ], + ), + CubeCube( + name="customers", sql_table="public.customers", + measures=[CubeMeasure(name="lifetime_value", type="sum", sql="{CUBE}.ltv")], + dimensions=[ + CubeDimension(name="id", sql="{CUBE}.id", type="number", primary_key=True), + CubeDimension(name="name", sql="{CUBE}.name", type="string"), + ], + ), + ], + views=[CubeView(name="orders_overview", cubes=[ + CubeViewCubeRef(join_path="orders", includes=["count", "total_revenue", + "completed_revenue", "status"]), + CubeViewCubeRef(join_path="orders.customers", prefix=True, + includes=["name", "lifetime_value"]), + ])], + ) + + +async def _save_converted(tmp_path) -> tuple[SlayerQueryEngine, dict]: + result = CubeToSlayerConverter( + project=_orders_customers_view_project(), data_source=DS).convert() + storage = YAMLStorage(base_dir=str(tmp_path)) + for model in result.models: + await storage.save_model(model) + engine = SlayerQueryEngine(storage=storage) + return engine, {m.name: m for m in result.models} + + +async def _gen_sql(engine: SlayerQueryEngine, query: SlayerQuery, model) -> str: + enriched = await engine._enrich(query=query, model=model) + return SQLGenerator(dialect="sqlite").generate(enriched=enriched) + + +async def test_view_cross_model_measure_resolves_to_underlying_column(tmp_path): + """The facade measure `customers_lifetime_value` must resolve through the + join to the underlying `ltv` column — this is the Codex #1 regression.""" + engine, models = await _save_converted(tmp_path) + view = models["orders_overview"] + query = SlayerQuery( + source_model="orders_overview", + dimensions=[ColumnRef(name="status")], + measures=[{"formula": "customers_lifetime_value"}], + ) + sql = await _gen_sql(engine, query, view) # must not raise "Column not found" + assert "ltv" in sql.lower() + assert "sum" in sql.lower() + + +async def test_view_status_dimension_generates(tmp_path): + engine, models = await _save_converted(tmp_path) + view = models["orders_overview"] + query = SlayerQuery( + source_model="orders_overview", + dimensions=[ColumnRef(name="status")], + measures=[{"formula": "total_revenue"}], + ) + sql = await _gen_sql(engine, query, view) + assert "status" in sql.lower() + + +async def test_filtered_and_unfiltered_measures_emit_distinct_sql(tmp_path): + """Codex #4 at the SQL layer: total_revenue (plain SUM) and + completed_revenue (SUM over CASE WHEN) must both appear, distinctly.""" + engine, models = await _save_converted(tmp_path) + orders = models["orders"] + query = SlayerQuery( + source_model="orders", + measures=[{"formula": "total_revenue"}, {"formula": "completed_revenue"}], + ) + sql = await _gen_sql(engine, query, orders) + assert sql.lower().count("sum(") >= 2 + assert "case" in sql.lower() # the filtered one wraps amount in CASE WHEN diff --git a/tests/test_cube_views.py b/tests/test_cube_views.py new file mode 100644 index 00000000..d253cba7 --- /dev/null +++ b/tests/test_cube_views.py @@ -0,0 +1,296 @@ +"""Tests for Cube views → SLayer facade models (slayer/cube/converter.py). + +DEV-1608 §6 + Codex #1/#3 corrections: facade measures reference the underlying +Column (not the Cube measure name); facade source mirrors the root cube's mode. +""" + +from slayer.core.models import SlayerModel +from slayer.cube.converter import CubeToSlayerConverter +from slayer.cube.models import ( + CubeCube, + CubeDimension, + CubeJoin, + CubeMeasure, + CubeProject, + CubeView, + CubeViewCubeRef, +) +from slayer.cube.report import CubeIssueCategory + +DS = "test_ds" + + +def _orders_customers_cubes(*, orders_sql_mode: bool = False) -> list[CubeCube]: + orders_kwargs = ( + {"sql": "SELECT * FROM public.orders"} if orders_sql_mode + else {"sql_table": "public.orders"} + ) + return [ + CubeCube( + name="orders", **orders_kwargs, + joins=[CubeJoin(name="customers", relationship="many_to_one", + sql="{CUBE}.customer_id = {customers.id}")], + measures=[CubeMeasure(name="count", type="count"), + CubeMeasure(name="total_revenue", type="sum", sql="{CUBE}.amount")], + dimensions=[CubeDimension(name="id", sql="{CUBE}.id", type="number", + primary_key=True), + CubeDimension(name="status", sql="{CUBE}.status", type="string")], + ), + CubeCube( + name="customers", sql_table="public.customers", + measures=[CubeMeasure(name="lifetime_value", type="sum", sql="{CUBE}.ltv")], + dimensions=[CubeDimension(name="id", sql="{CUBE}.id", type="number", + primary_key=True), + CubeDimension(name="name", sql="{CUBE}.name", type="string"), + CubeDimension(name="region", sql="{CUBE}.region", type="string")], + ), + ] + + +def _view() -> CubeView: + return CubeView(name="orders_overview", cubes=[ + CubeViewCubeRef(join_path="orders", + includes=["count", "total_revenue", "status"]), + CubeViewCubeRef(join_path="orders.customers", prefix=True, + includes=["name", "region", "lifetime_value"]), + ]) + + +def _convert(project: CubeProject) -> tuple[dict[str, SlayerModel], object]: + result = CubeToSlayerConverter(project=project, data_source=DS).convert() + return {m.name: m for m in result.models}, result.report + + +def test_view_facade_model_basic_shape(): + project = CubeProject(cubes=_orders_customers_cubes(), views=[_view()]) + models, _ = _convert(project) + view = models["orders_overview"] + assert view.sql_table == "public.orders" # rooted on orders + assert view.meta["cube_kind"] == "view" + # join to customers present on the facade + assert any(j.target_model == "customers" for j in view.joins) + + +def test_view_root_dimension_is_local_derived_column(): + project = CubeProject(cubes=_orders_customers_cubes(), views=[_view()]) + models, _ = _convert(project) + view = models["orders_overview"] + assert view.get_column("status") is not None + + +def test_view_prefixed_joined_dimension_references_joined_column(): + project = CubeProject(cubes=_orders_customers_cubes(), views=[_view()]) + models, _ = _convert(project) + view = models["orders_overview"] + # prefix: true → "_" (Cube prepends the cube name verbatim). + col = view.get_column("customers_name") + assert col is not None + assert col.sql == "customers.name" + + +def test_view_root_measure_carries_underlying_column(): + """Codex #1: a root-cube measure re-export needs the underlying column on + the facade, referenced by `:` (NOT the measure name).""" + project = CubeProject(cubes=_orders_customers_cubes(), views=[_view()]) + models, _ = _convert(project) + view = models["orders_overview"] + m = view.get_measure("total_revenue") + assert m is not None + col_ref = m.formula.split(":")[0].strip() + assert view.get_column(col_ref) is not None # underlying column copied onto facade + assert m.formula.endswith(":sum") + + +def test_view_joined_measure_is_cross_model_underlying_column_ref(): + """Codex #1: joined-cube measure → `customers.:` + (the underlying column `ltv`, never the measure name `lifetime_value`).""" + project = CubeProject(cubes=_orders_customers_cubes(), views=[_view()]) + models, _ = _convert(project) + view = models["orders_overview"] + m = view.get_measure("customers_lifetime_value") # prefix: "_" + assert m is not None + assert m.formula == "customers.ltv:sum" + + +def test_view_count_measure_maps_to_star_count(): + project = CubeProject(cubes=_orders_customers_cubes(), views=[_view()]) + models, _ = _convert(project) + assert models["orders_overview"].get_measure("count").formula == "*:count" + + +def test_view_default_filters_become_model_filters(): + view = _view() + view.default_filters = [{"member": "orders.status", "operator": "equals", + "values": ["completed"]}] + project = CubeProject(cubes=_orders_customers_cubes(), views=[view]) + models, _ = _convert(project) + filters = " ".join(models["orders_overview"].filters) + assert "completed" in filters + + +def test_view_on_sql_backed_root_mirrors_sql_source(): + """Codex #3: facade mirrors the root cube's source mode, not always sql_table.""" + project = CubeProject(cubes=_orders_customers_cubes(orders_sql_mode=True), + views=[_view()]) + models, _ = _convert(project) + view = models["orders_overview"] + assert view.sql_table is None + assert view.sql == "SELECT * FROM public.orders" + + +def test_view_folders_parked_in_meta_and_reported(): + view = _view() + view.folders = [{"name": "Revenue", "includes": ["total_revenue"]}] + project = CubeProject(cubes=_orders_customers_cubes(), views=[view]) + models, report = _convert(project) + assert models["orders_overview"].meta["cube_unmapped"]["folders"] + assert any(i.category == CubeIssueCategory.FOLDERS_UNMAPPED for i in report.issues) + + +def test_view_excludes_drops_member(): + view = CubeView(name="ov", cubes=[ + CubeViewCubeRef(join_path="orders", includes="*", excludes=["status"]), + ]) + project = CubeProject(cubes=_orders_customers_cubes(), views=[view]) + models, _ = _convert(project) + assert models["ov"].get_column("status") is None + assert models["ov"].get_measure("total_revenue") is not None + + +def test_view_includes_star_takes_all_members(): + view = CubeView(name="ov", cubes=[ + CubeViewCubeRef(join_path="orders", includes="*"), + ]) + project = CubeProject(cubes=_orders_customers_cubes(), views=[view]) + models, _ = _convert(project) + ov = models["ov"] + assert ov.get_column("status") is not None + assert ov.get_measure("total_revenue") is not None + assert ov.get_measure("count") is not None + + +def test_view_include_entry_without_name_reported(): + # An includes entry with no valid `name` (Cube requires one) — e.g. + # `{"alias": ...}` or `{}` — is a parse error, not silently dropped or + # reported with a confusing member=None. Valid siblings still convert. + view = CubeView(name="ov", cubes=[ + CubeViewCubeRef(join_path="orders", + includes=[{"alias": "s"}, {}, "status"]), + ]) + project = CubeProject(cubes=_orders_customers_cubes(), views=[view]) + models, report = _convert(project) + assert models["ov"].get_column("status") is not None + parse_errors = [i for i in report.issues + if i.category == CubeIssueCategory.PARSE_ERROR] + assert len(parse_errors) == 2 # one for {"alias": ...}, one for {} + assert all(i.member is None for i in parse_errors) + + +def test_view_disconnected_members_reported(): + cubes = _orders_customers_cubes() + cubes.append(CubeCube(name="weather", sql_table="public.weather", + dimensions=[CubeDimension(name="temp", sql="{CUBE}.temp", + type="number")])) + view = CubeView(name="ov", cubes=[ + CubeViewCubeRef(join_path="orders", includes=["status"]), + CubeViewCubeRef(join_path="weather", includes=["temp"]), # not joined to orders + ]) + project = CubeProject(cubes=cubes, views=[view]) + _models, report = _convert(project) + assert any(i.category == CubeIssueCategory.DISCONNECTED_VIEW for i in report.issues) + + +def test_view_fanout_risk_reported(): + cubes = [ + CubeCube(name="orders", sql_table="public.orders", + joins=[CubeJoin(name="line_items", relationship="one_to_many", + sql="{CUBE}.id = {line_items.order_id}")], + measures=[CubeMeasure(name="total_revenue", type="sum", sql="{CUBE}.amount")], + dimensions=[CubeDimension(name="id", sql="{CUBE}.id", type="number", + primary_key=True)]), + CubeCube(name="line_items", sql_table="public.line_items", + dimensions=[CubeDimension(name="order_id", sql="{CUBE}.order_id", + type="number"), + CubeDimension(name="sku", sql="{CUBE}.sku", type="string")]), + ] + view = CubeView(name="ov", cubes=[ + CubeViewCubeRef(join_path="orders", includes=["total_revenue"]), + CubeViewCubeRef(join_path="orders.line_items", includes=["sku"]), + ]) + project = CubeProject(cubes=cubes, views=[view]) + _models, report = _convert(project) + assert any(i.category == CubeIssueCategory.VIEW_FANOUT_RISK for i in report.issues) + + +def test_view_extends_flattens_member_lists(): + base = CubeView(name="base_view", cubes=[ + CubeViewCubeRef(join_path="orders", includes=["status"])]) + child = CubeView(name="child_view", extends="base_view", cubes=[ + CubeViewCubeRef(join_path="orders", includes=["total_revenue"])]) + project = CubeProject(cubes=_orders_customers_cubes(), views=[base, child]) + models, _ = _convert(project) + child_model = models["child_view"] + assert child_model.get_column("status") is not None # inherited + assert child_model.get_measure("total_revenue") is not None # own + + +def test_view_star_skips_private_member(): + cubes = _orders_customers_cubes() + # mark a root-cube dimension private + cubes[0].dimensions.append(CubeDimension(name="internal_flag", sql="{CUBE}.flag", + type="string", public=False)) + view = CubeView(name="ov", cubes=[CubeViewCubeRef(join_path="orders", includes="*")]) + project = CubeProject(cubes=cubes, views=[view]) + models, _ = _convert(project) + assert models["ov"].get_column("internal_flag") is None + assert models["ov"].get_column("status") is not None + + +def test_view_default_filter_escapes_single_quotes(): + view = _view() + view.default_filters = [{"member": "orders.status", "operator": "equals", + "values": ["O'Reilly"]}] + project = CubeProject(cubes=_orders_customers_cubes(), views=[view]) + models, _ = _convert(project) + filters = " ".join(models["orders_overview"].filters) + assert "O''Reilly" in filters # single quote doubled, not "O'Reilly'" + + +def test_view_object_form_includes_reports_override(): + view = CubeView(name="ov", cubes=[CubeViewCubeRef( + join_path="orders", + includes=[{"name": "status", "format": "upper"}, "total_revenue"])]) + project = CubeProject(cubes=_orders_customers_cubes(), views=[view]) + models, report = _convert(project) + assert models["ov"].get_column("status") is not None # name still extracted + assert models["ov"].get_measure("total_revenue") is not None + assert any(i.category == CubeIssueCategory.UNMAPPED_INFRA for i in report.issues) + + +def test_view_calc_measure_reexport_reported(): + cubes = _orders_customers_cubes() + cubes[0].measures.append(CubeMeasure(name="aov", type="number", + sql="{total_revenue} / {count}")) + view = CubeView(name="ov", cubes=[CubeViewCubeRef(join_path="orders", includes=["aov"])]) + project = CubeProject(cubes=cubes, views=[view]) + models, report = _convert(project) + # A calculated measure can't be re-exported as a facade cross-model measure + # in Stage 1 → reported, not silently skipped. + assert models["ov"].get_measure("aov") is None + assert any(i.category == CubeIssueCategory.COMPLEX_MEASURE for i in report.issues) + + +def test_view_dropped_when_root_cube_not_emitted(): + # orders has no source → not emitted; a view rooted on it can't be built. + cubes = [ + CubeCube(name="orders"), # no source + CubeCube(name="customers", sql_table="public.customers", + dimensions=[CubeDimension(name="id", sql="{CUBE}.id", type="number")]), + ] + view = CubeView(name="ov", cubes=[ + CubeViewCubeRef(join_path="orders", includes=["id"])]) + project = CubeProject(cubes=cubes, views=[view]) + models, report = _convert(project) + assert "ov" not in models + assert any(i.category == CubeIssueCategory.AMBIGUOUS_VIEW_ROOT + for i in report.issues) diff --git a/tests/test_declared_list_variables.py b/tests/test_declared_list_variables.py new file mode 100644 index 00000000..ce7c24ed --- /dev/null +++ b/tests/test_declared_list_variables.py @@ -0,0 +1,306 @@ +"""Declared list-valued ``{variable}`` coercion (DEV-1730 follow-up). + +The generic Mode-A contract leaves quoting to the template author, so a scalar +string renders UNQUOTED — that is what makes ``order_total >= {floor}`` and +``{d}::TIMESTAMP`` work. A machine-generated ``col IN ({var})`` surface has no +author to write the quotes, so the importer declares the variable +``list_valued`` in ``meta.cube_variables`` and the engine wraps a bare scalar +into a one-element list before substitution. + +Covers the two helpers, the engine choke point, and the boundaries that must +NOT change: hand-written models, undeclared variables, empty lists, and the +scalar-position arrow variables. +""" +import sqlite3 +import tempfile + +import pytest + +from slayer.core.enums import DataType +from slayer.core.models import Column, DatasourceConfig, SlayerModel +from slayer.core.query import ( + SlayerQuery, + coerce_declared_list_variables, + declares_variables, + list_valued_variable_names, +) +from slayer.engine.query_engine import ( + SlayerQueryEngine, + _substitute_model_sql_surfaces, +) +from slayer.sql.dialects import SqliteDialect +from slayer.storage.yaml_storage import YAMLStorage + + +def _meta(**flags) -> dict: + """Build a ``meta.cube_variables`` bag: name -> list_valued flag.""" + return { + "cube_variables": { + name: { + "member": name, "required": False, + "kind": "string" if flag else "arrow_value", + "list_valued": flag, "description": None, + } + for name, flag in flags.items() + } + } + + +def _model(*, sql: str, meta: dict | None = None) -> SlayerModel: + return SlayerModel( + name="orders", sql=sql, data_source="ds", meta=meta, + columns=[ + Column(name="id", sql="id", type=DataType.DOUBLE, primary_key=True), + Column(name="region", sql="region", type=DataType.TEXT), + Column(name="amount", sql="amount", type=DataType.DOUBLE), + ], + ) + + +# ── list_valued_variable_names ────────────────────────────────────────────── + + +def test_names_reads_only_the_neutral_flag(): + model = _model(sql="SELECT * FROM orders", meta=_meta(regions=True, cutoff=False)) + assert list_valued_variable_names(model) == {"regions"} + + +def test_names_empty_for_hand_written_model(): + assert list_valued_variable_names(_model(sql="SELECT * FROM orders")) == set() + + +def test_names_ignores_unrelated_or_malformed_meta(): + # A model carrying its own meta must not trip the accessor. + assert list_valued_variable_names( + _model(sql="SELECT 1", meta={"cube_variables": "not-a-dict", "owner": "x"}) + ) == set() + assert list_valued_variable_names( + _model(sql="SELECT 1", meta={"cube_variables": {"a": "not-a-dict"}}) + ) == set() + + +@pytest.mark.parametrize("flag", [1, "true", "false", "yes", [1], {"a": 1}]) +def test_only_a_real_true_enables_coercion(flag): + # meta is user-extensible, so a truthiness match would let a stray 1 — or + # even the string "false" — silently switch substitution semantics. + model = _model( + sql="SELECT 1", + meta={"cube_variables": {"regions": {"member": "r", "list_valued": flag}}}, + ) + assert list_valued_variable_names(model) == set() + + +def test_true_flag_enables_coercion(): + model = _model( + sql="SELECT 1", + meta={"cube_variables": {"regions": {"member": "r", "list_valued": True}}}, + ) + assert list_valued_variable_names(model) == {"regions"} + + +@pytest.mark.parametrize( + "bag", + [ + {"note": {}}, # unrelated bag under the same key + {"note": {"text": "hi"}}, # dict entries, but no 'member' + {"note": {"member": 42}}, # 'member' present but not a string + {"note": {"member": ""}}, # empty member — never importer output + ], +) +def test_unrelated_meta_under_the_same_key_is_not_a_declaration(bag): + """`meta` is free-form user data, so the bag must be SELF-IDENTIFYING. + + A false positive here would be a regression, not a nit: declaring disables + the zero-variable fast path, so a hand-written model with raw brace literals + would start raising on a query that used to work. + """ + model = _model(sql="SELECT * FROM t WHERE tags = '{1,2,3}'", meta={"cube_variables": bag}) + assert declares_variables(model) is False + assert list_valued_variable_names(model) == set() + out = _substitute_model_sql_surfaces( + model=model, variables={}, dialect=SqliteDialect() + ) + assert out.sql == "SELECT * FROM t WHERE tags = '{1,2,3}'" # still protected + + +# ── coerce_declared_list_variables ────────────────────────────────────────── + + +@pytest.mark.parametrize( + "value,expected", + [("US", ["US"]), (5, [5]), (2.5, [2.5]), (True, [True])], +) +def test_scalars_are_wrapped(value, expected): + out = coerce_declared_list_variables({"regions": value}, list_valued={"regions"}) + assert out == {"regions": expected} + + +@pytest.mark.parametrize("value", [["US", "CA"], ("US",), []]) +def test_sequences_pass_through_unchanged(value): + # The empty list is deliberately NOT rescued here — it still raises at + # render time, since 'no filter' belongs to an optional block / sentinel. + out = coerce_declared_list_variables({"regions": value}, list_valued={"regions"}) + assert out["regions"] is value + + +def test_unsupported_types_left_for_the_renderer_to_reject(): + # None/dict keep their own scalar-path error message instead of being + # wrapped into a list and reported as a bad list ELEMENT. + for value in (None, {"a": 1}): + out = coerce_declared_list_variables({"regions": value}, list_valued={"regions"}) + assert out["regions"] is value + + +def test_undeclared_variables_and_absent_names_untouched(): + variables = {"regions": "US", "floor": "500"} + out = coerce_declared_list_variables(variables, list_valued={"regions", "missing"}) + assert out == {"regions": ["US"], "floor": "500"} + + +def test_input_never_mutated_and_returned_as_is_when_no_work(): + variables = {"floor": "500"} + assert coerce_declared_list_variables(variables, list_valued=set()) is variables + assert coerce_declared_list_variables(variables, list_valued={"regions"}) is variables + declared = {"regions": "US"} + coerce_declared_list_variables(declared, list_valued={"regions"}) + assert declared == {"regions": "US"} # caller's dict intact + + +# ── declares_variables / the zero-variable fast-path hole ─────────────────── + + +def test_declares_variables_true_only_for_a_declaring_model(): + assert declares_variables(_model(sql="SELECT 1", meta=_meta(regions=True))) is True + assert declares_variables(_model(sql="SELECT 1", meta=_meta(cutoff=False))) is True + assert declares_variables(_model(sql="SELECT 1")) is False + assert declares_variables(_model(sql="SELECT 1", meta={"owner": "x"})) is False + assert declares_variables( + _model(sql="SELECT 1", meta={"cube_variables": {}}) + ) is False + + +def test_declared_required_var_with_zero_variables_raises(): + # The fast-path hole Codex flagged, closed for GENERATED models: a model + # whose pushdowns are all required has no block to force the pass, so a + # zero-variable call used to emit a bare {var} into the SQL. + model = _model( + sql="SELECT * FROM orders WHERE d >= '{cutoff}'", meta=_meta(cutoff=False) + ) + dialect = SqliteDialect() # hoisted: only one call may throw (sonar S5778) + with pytest.raises(ValueError, match="Undefined variable 'cutoff'"): + _substitute_model_sql_surfaces(model=model, variables={}, dialect=dialect) + + +def test_hand_written_model_keeps_the_brace_literal_protection(): + # The DEV-1625 contract is untouched for models that declare nothing: a + # Postgres array literal must survive a zero-variable call verbatim. + model = _model(sql="SELECT * FROM t WHERE tags = '{1,2,3}'") + out = _substitute_model_sql_surfaces( + model=model, variables={}, dialect=SqliteDialect() + ) + assert out.sql == "SELECT * FROM t WHERE tags = '{1,2,3}'" + assert out is model # untouched, not even copied + + +# ── engine substitution choke point ───────────────────────────────────────── + + +def _sub(model: SlayerModel, variables: dict) -> str: + return _substitute_model_sql_surfaces( + model=model, variables=variables, dialect=SqliteDialect() + ).sql + + +def test_declared_scalar_renders_quoted_in_list(): + model = _model( + sql="SELECT * FROM orders WHERE region IN ({regions})", meta=_meta(regions=True) + ) + assert _sub(model, {"regions": "US"}).endswith("IN ('US')") + assert _sub(model, {"regions": ["US", "CA"]}).endswith("IN ('US', 'CA')") + + +def test_declared_scalar_is_still_escaped(): + model = _model( + sql="SELECT * FROM orders WHERE region IN ({regions})", meta=_meta(regions=True) + ) + assert _sub(model, {"regions": "O'Brien"}).endswith("IN ('O''Brien')") + + +def test_coercion_applies_inside_an_optional_block(): + model = _model( + sql="SELECT * FROM orders WHERE 1=1 AND {? region IN ({regions}) ?}", + meta=_meta(regions=True), + ) + assert _sub(model, {"regions": "US"}).endswith("(region IN ('US'))") + assert _sub(model, {}).endswith("(1=1)") + + +def test_undeclared_scalar_keeps_the_author_written_quote_convention(): + # The generic Mode-A rule is untouched: a hand-written template owns its + # quotes, so {var} still works in numeric / fragment positions. + model = _model(sql="SELECT * FROM orders WHERE amount >= {floor}") + assert _sub(model, {"floor": "500"}).endswith(">= 500") + assert _sub(model, {"floor": 500}).endswith(">= 500") + + +def test_declared_empty_list_still_raises(): + model = _model( + sql="SELECT * FROM orders WHERE region IN ({regions})", meta=_meta(regions=True) + ) + with pytest.raises(ValueError, match="cannot be an empty list"): + _sub(model, {"regions": []}) + + +def test_arrow_style_declared_variable_is_not_wrapped(): + # list_valued is False -> the pre-quoted scalar convention still applies. + model = _model( + sql="SELECT * FROM orders WHERE d >= '{cutoff}'", meta=_meta(cutoff=False) + ) + assert _sub(model, {"cutoff": "2025-01-01"}).endswith(">= '2025-01-01'") + + +# ── end-to-end on SQLite (result data, not just text) ─────────────────────── + + +def _seed(db_path: str) -> None: + conn = sqlite3.connect(db_path) + cur = conn.cursor() + cur.execute("CREATE TABLE orders (id INTEGER PRIMARY KEY, region TEXT, amount REAL)") + cur.executemany( + "INSERT INTO orders VALUES (?, ?, ?)", + [(1, "US", 100.0), (2, "US", 60.0), (3, "EU", 200.0), (4, "CA", 300.0)], + ) + conn.commit() + conn.close() + + +async def _engine_with(model: SlayerModel): + tmp = tempfile.TemporaryDirectory() + _seed(f"{tmp.name}/orders.db") + storage = YAMLStorage(base_dir=tmp.name) + await storage.save_datasource( + DatasourceConfig(name="ds", type="sqlite", database=f"{tmp.name}/orders.db") + ) + await storage.save_model(model) + return SlayerQueryEngine(storage=storage), tmp + + +async def test_e2e_scalar_and_list_return_identical_rows(): + model = _model( + sql="SELECT * FROM orders WHERE 1=1 AND {? region IN ({regions}) ?}", + meta=_meta(regions=True), + ) + engine, tmp = await _engine_with(model) + try: + async def _total(regions): + q = SlayerQuery( + source_model="orders", measures=[{"formula": "amount:sum"}], + variables={"regions": regions}, + ) + resp = await engine.execute(q) + assert resp.row_count == 1, resp.data + return resp.data[0]["orders.amount_sum"] + + assert await _total("US") == await _total(["US"]) == 160.0 + finally: + tmp.cleanup() diff --git a/tests/test_mode_a_optional_blocks_engine.py b/tests/test_mode_a_optional_blocks_engine.py new file mode 100644 index 00000000..063f1b33 --- /dev/null +++ b/tests/test_mode_a_optional_blocks_engine.py @@ -0,0 +1,229 @@ +"""End-to-end engine behavior for Mode-A optional blocks (DEV-1730). + +Blocks + list-valued IN pushdowns are exercised against a real file-backed +SQLite datasource, asserting on RESULT DATA. Also pins the substitution +fast-path boundary (the documented DEV-1625 required-only-zero-vars hole). +""" +import sqlite3 +import tempfile + +import pytest + +from slayer.core.enums import DataType +from slayer.core.models import Column, DatasourceConfig, SlayerModel +from slayer.core.query import SlayerQuery +from slayer.engine.query_engine import ( + SlayerQueryEngine, + _substitute_model_sql_surfaces, +) +from slayer.sql.dialects import SqliteDialect +from slayer.storage.yaml_storage import YAMLStorage + + +def _seed(db_path) -> None: + conn = sqlite3.connect(str(db_path)) + cur = conn.cursor() + cur.execute( + "CREATE TABLE orders (id INTEGER PRIMARY KEY, region TEXT, amount REAL)" + ) + cur.executemany( + "INSERT INTO orders VALUES (?, ?, ?)", + [(1, "US", 100.0), (2, "US", 60.0), (3, "EU", 200.0), + (4, "EU", 75.0), (5, "CA", 300.0)], + ) + conn.commit() + conn.close() + + +async def _engine_with(*models: SlayerModel): + tmp = tempfile.TemporaryDirectory() + _seed(f"{tmp.name}/orders.db") + storage = YAMLStorage(base_dir=tmp.name) + await storage.save_datasource( + DatasourceConfig(name="ds", type="sqlite", database=f"{tmp.name}/orders.db") + ) + for m in models: + await storage.save_model(m) + return SlayerQueryEngine(storage=storage), tmp + + +def _sum(resp, alias: str) -> float: + assert resp.row_count == 1, f"expected 1 row: {resp.data}" + return resp.data[0][alias] + + +def _block_model() -> SlayerModel: + # sql-mode model whose WHERE carries an optional IN-list block. + return SlayerModel( + name="orders", + sql="SELECT * FROM orders WHERE 1=1 AND {? region IN ({regions}) ?}", + data_source="ds", + columns=[ + Column(name="id", sql="id", type=DataType.DOUBLE, primary_key=True), + Column(name="region", sql="region", type=DataType.TEXT), + Column(name="amount", sql="amount", type=DataType.DOUBLE), + ], + ) + + +# ── e2e ───────────────────────────────────────────────────────────────────── + + +async def test_optional_block_collapses_to_unfiltered_when_missing(): + engine, tmp = await _engine_with(_block_model()) + try: + q = SlayerQuery(source_model="orders", measures=[{"formula": "amount:sum"}]) + resp = await engine.execute(q) + assert _sum(resp, "orders.amount_sum") == 735.0 + finally: + tmp.cleanup() + + +async def test_optional_block_filters_with_in_list_when_present(): + engine, tmp = await _engine_with(_block_model()) + try: + q = SlayerQuery( + source_model="orders", + measures=[{"formula": "amount:sum"}], + variables={"regions": ["US", "CA"]}, + ) + resp = await engine.execute(q) + assert _sum(resp, "orders.amount_sum") == 460.0 # US 160 + CA 300 + finally: + tmp.cleanup() + + +async def test_optional_block_single_value_list(): + engine, tmp = await _engine_with(_block_model()) + try: + q = SlayerQuery( + source_model="orders", + measures=[{"formula": "amount:sum"}], + variables={"regions": ["EU"]}, + ) + resp = await engine.execute(q) + assert _sum(resp, "orders.amount_sum") == 275.0 + finally: + tmp.cleanup() + + +async def test_required_var_alongside_block_raises_and_names_it(): + model = SlayerModel( + name="orders", + sql="SELECT * FROM orders WHERE amount >= {min_amount} " + "AND {? region IN ({regions}) ?}", + data_source="ds", + columns=[ + Column(name="id", sql="id", type=DataType.DOUBLE, primary_key=True), + Column(name="region", sql="region", type=DataType.TEXT), + Column(name="amount", sql="amount", type=DataType.DOUBLE), + ], + ) + engine, tmp = await _engine_with(model) + try: + q = SlayerQuery( + source_model="orders", + measures=[{"formula": "amount:sum"}], + variables={"regions": ["US"]}, # min_amount omitted + ) + with pytest.raises(Exception, match="min_amount"): + await engine.execute(q) + finally: + tmp.cleanup() + + +async def test_dimension_query_over_block_model_resolves_types(): + # Exercises get_column_types: blocks must collapse under the defaults probe. + engine, tmp = await _engine_with(_block_model()) + try: + q = SlayerQuery(source_model="orders", dimensions=["region"]) + resp = await engine.execute(q) + assert {r["orders.region"] for r in resp.data} == {"US", "EU", "CA"} + finally: + tmp.cleanup() + + +# ── substitution fast-path boundary (unit) ────────────────────────────────── + + +def test_blockfree_empty_vars_leaves_model_untouched(): + # DEV-1625 brace-literal protection: no vars + no block -> no substitution. + model = SlayerModel( + name="m", sql="SELECT * FROM t WHERE tags = '{1,2,3}'", data_source="ds", + ) + out = _substitute_model_sql_surfaces( + model=model, variables={}, dialect=SqliteDialect()) + assert out.sql == "SELECT * FROM t WHERE tags = '{1,2,3}'" + + +def test_blockfree_required_var_zero_vars_is_untouched_not_raised(): + # The documented boundary hole: a block-free model whose only placeholder is + # a *declared-looking* required var, called with zero variables, is left + # untouched (NOT raised) — substitution only runs when vars OR a block exist. + model = SlayerModel( + name="m", sql="SELECT * FROM t WHERE region = '{region}'", data_source="ds", + ) + out = _substitute_model_sql_surfaces( + model=model, variables={}, dialect=SqliteDialect()) + assert out.sql == "SELECT * FROM t WHERE region = '{region}'" + + +async def test_bare_var_default_satisfies_required_at_runtime(): + # A query_variables default fulfils an otherwise-bare required var so the + # query runs without the caller supplying it. + model = SlayerModel( + name="orders", + sql="SELECT * FROM orders WHERE region = '{region}'", + data_source="ds", + query_variables={"region": "EU"}, + columns=[ + Column(name="id", sql="id", type=DataType.DOUBLE, primary_key=True), + Column(name="region", sql="region", type=DataType.TEXT), + Column(name="amount", sql="amount", type=DataType.DOUBLE), + ], + ) + engine, tmp = await _engine_with(model) + try: + q = SlayerQuery(source_model="orders", measures=[{"formula": "amount:sum"}]) + resp = await engine.execute(q) + assert _sum(resp, "orders.amount_sum") == 275.0 # EU only + finally: + tmp.cleanup() + + +async def test_get_column_types_probes_scalar_and_date_and_list_contexts(): + # The defaults probe must yield valid SQL across all optional-context shapes: + # a scalar-cast block, a date-range block, and an IN-list block. + model = SlayerModel( + name="orders", + sql="SELECT id, region, amount, " + "{? '{d_from}' ?} AS d_scalar " + "FROM orders WHERE 1=1 " + "AND {? amount >= {min_amt} AND amount <= {max_amt} ?} " + "AND {? region IN ({regions}) ?}", + data_source="ds", + columns=[ + Column(name="id", sql="id", type=DataType.DOUBLE, primary_key=True), + Column(name="region", sql="region", type=DataType.TEXT), + Column(name="amount", sql="amount", type=DataType.DOUBLE), + ], + ) + engine, tmp = await _engine_with(model) + try: + # All optional blocks collapse under the empty defaults probe -> valid SQL. + types = await engine.get_column_types("orders") + assert isinstance(types, dict) # degrades gracefully, never raises + finally: + tmp.cleanup() + + +def test_block_bearing_empty_vars_collapses_block(): + # A block-bearing model must collapse even when zero variables are supplied. + model = SlayerModel( + name="m", + sql="SELECT * FROM t WHERE 1=1 AND {? region IN ({regions}) ?}", + data_source="ds", + ) + out = _substitute_model_sql_surfaces( + model=model, variables={}, dialect=SqliteDialect()) + assert out.sql == "SELECT * FROM t WHERE 1=1 AND (1=1)" diff --git a/tests/test_mode_a_variable_substitution.py b/tests/test_mode_a_variable_substitution.py index 5e2807a9..5587e281 100644 --- a/tests/test_mode_a_variable_substitution.py +++ b/tests/test_mode_a_variable_substitution.py @@ -864,9 +864,11 @@ class TestSubstituteVariablesHardened: def test_sql_mode_string_value_doubles_single_quote(self) -> None: from slayer.core.query import substitute_variables - # Mode-A (sqlglot-parsed) surfaces double the single quote. + # Mode-A (sqlglot-parsed) surfaces double the single quote. Standard + # (non-backslash) dialect regime. result = substitute_variables( - filter_str="status = '{v}'", variables={"v": "O'Brien"}, escape="sql" + filter_str="status = '{v}'", variables={"v": "O'Brien"}, + escape="sql", backslash_escapes=False, ) assert result == "status = 'O''Brien'" @@ -874,17 +876,20 @@ def test_sql_mode_string_value_without_quote_unchanged(self) -> None: from slayer.core.query import substitute_variables result = substitute_variables( - filter_str="status = '{v}'", variables={"v": "active"}, escape="sql" + filter_str="status = '{v}'", variables={"v": "active"}, + escape="sql", backslash_escapes=False, ) assert result == "status = 'active'" - def test_sql_mode_backslash_untouched(self) -> None: + def test_sql_mode_backslash_untouched_standard_dialect(self) -> None: from slayer.core.query import substitute_variables - # sqlglot treats backslash as an ordinary char in a string literal, so - # SQL-mode must NOT touch it (only ' is special). + # On a STANDARD dialect (backslash_escapes=False) sqlglot treats + # backslash as an ordinary char, so SQL-mode must NOT touch it (only ' + # is special). DEV-1727 backslash dialects are covered separately. result = substitute_variables( - filter_str="path = '{v}'", variables={"v": r"a\b"}, escape="sql" + filter_str="path = '{v}'", variables={"v": r"a\b"}, + escape="sql", backslash_escapes=False, ) assert result == r"path = 'a\b'" @@ -955,19 +960,50 @@ def test_python_mode_backslash_before_double_quote(self) -> None: ('col = "{v}"', 'a\\"b'), ("col = '{v}'", "abc\\"), ("col = '{v}'", "plain"), + # Control chars must be backslash-escaped so a raw newline/CR/tab/NUL + # doesn't make the single-quoted literal a SyntaxError. + ("col = '{v}'", "a\nb"), + ("col = '{v}'", "a\r\nb"), + ("col = '{v}'", "a\tb"), + ("col = '{v}'", "a\x00b"), ], ) def test_python_mode_ast_roundtrip(self, template: str, value: str) -> None: # Semantic contract: whatever escaping produces, ast.parse of the # substituted filter must recover the ORIGINAL value (both quote - # delimiters, backslash, and backslash-before-quote combinations). + # delimiters, backslash, backslash-before-quote, and control chars). assert _python_mode_roundtrips(template, value) + def test_python_mode_newline_escaped(self) -> None: + from slayer.core.query import substitute_variables + + # A real newline becomes the two-char escape \n so the literal stays on + # one line and re-parses to the original value. + result = substitute_variables( + filter_str="note = '{v}'", variables={"v": "a\nb"}, escape="python" + ) + assert result == "note = 'a\\nb'" + + def test_sql_mode_newline_left_raw(self) -> None: + from slayer.core.query import substitute_variables + + # SQL string literals permit raw newlines, so sql-mode must NOT escape + # them (only the single quote is special there). DEV-1727 made the sql + # regime dialect-aware + fail-closed, so pass the standard-dialect flag. + result = substitute_variables( + filter_str="note = '{v}'", variables={"v": "a\nb"}, + escape="sql", backslash_escapes=False, + ) + assert result == "note = 'a\nb'" + def test_number_value_not_escaped_either_mode(self) -> None: from slayer.core.query import substitute_variables assert ( - substitute_variables(filter_str="amount > {n}", variables={"n": 100}, escape="sql") + substitute_variables( + filter_str="amount > {n}", variables={"n": 100}, + escape="sql", backslash_escapes=False, + ) == "amount > 100" ) assert ( @@ -979,7 +1015,10 @@ def test_float_value_not_escaped(self) -> None: from slayer.core.query import substitute_variables assert ( - substitute_variables(filter_str="rate < {n}", variables={"n": 0.05}, escape="sql") + substitute_variables( + filter_str="rate < {n}", variables={"n": 0.05}, + escape="sql", backslash_escapes=False, + ) == "rate < 0.05" ) @@ -988,7 +1027,10 @@ def test_bool_value_accepted(self) -> None: # bool is an int subclass; kept accepted (renders True/False). assert ( - substitute_variables(filter_str="flag = {v}", variables={"v": True}, escape="sql") + substitute_variables( + filter_str="flag = {v}", variables={"v": True}, + escape="sql", backslash_escapes=False, + ) == "flag = True" ) assert ( @@ -1001,7 +1043,8 @@ def test_nan_value_raises(self) -> None: with pytest.raises(ValueError, match="finite"): substitute_variables( - filter_str="x = {v}", variables={"v": float("nan")}, escape="sql" + filter_str="x = {v}", variables={"v": float("nan")}, + escape="sql", backslash_escapes=False, ) def test_inf_value_raises(self) -> None: @@ -1009,19 +1052,38 @@ def test_inf_value_raises(self) -> None: with pytest.raises(ValueError, match="finite"): substitute_variables( - filter_str="x = {v}", variables={"v": float("inf")}, escape="sql" + filter_str="x = {v}", variables={"v": float("inf")}, + escape="sql", backslash_escapes=False, ) with pytest.raises(ValueError, match="finite"): substitute_variables( filter_str="x = {v}", variables={"v": float("-inf")}, escape="python" ) - def test_list_value_still_raises(self) -> None: + def test_dict_value_raises(self) -> None: from slayer.core.query import substitute_variables - with pytest.raises(ValueError, match="must be a string or number"): + # A dict is neither scalar nor list/tuple → terminal ValueError whose + # message now names list/tuple as an accepted shape (DEV-1730 lists). + with pytest.raises( + ValueError, match="must be a string, number, or list/tuple" + ): substitute_variables( - filter_str="x = '{v}'", variables={"v": [1, 2]}, escape="sql" + filter_str="x = '{v}'", variables={"v": {"a": 1}}, + escape="sql", backslash_escapes=False, + ) + + def test_set_value_raises(self) -> None: + from slayer.core.query import substitute_variables + + # A set is unordered → deliberately NOT accepted (only list/tuple), and + # falls through to the same terminal message. + with pytest.raises( + ValueError, match="must be a string, number, or list/tuple" + ): + substitute_variables( + filter_str="x IN ({v})", variables={"v": {1, 2}}, + escape="sql", backslash_escapes=False, ) def test_escape_is_required_keyword_only(self) -> None: @@ -1037,11 +1099,25 @@ def test_invalid_escape_value_raises(self) -> None: # Literal gives no runtime enforcement; the implementation must reject # an unknown mode deterministically rather than silently pick a branch. - with pytest.raises(ValueError, match="escape"): + # The message must name the VALID modes — so a wrong ordering that hit + # the sql-mode ``backslash_escapes`` guard first (whose message also + # contains the word "escape") could not masquerade as a pass. + with pytest.raises(ValueError, match="sql.*python|python.*sql"): substitute_variables( filter_str="x = '{v}'", variables={"v": "a"}, escape="other" ) + def test_invalid_escape_takes_precedence_over_missing_flag(self) -> None: + from slayer.core.query import substitute_variables + + # An invalid escape mode is rejected BEFORE the sql-mode + # backslash_escapes guard — the error is about the mode, not the flag. + with pytest.raises(ValueError, match="sql.*python|python.*sql") as exc: + substitute_variables( + filter_str="x = '{v}'", variables={"v": "a"}, escape="bogus" + ) + assert "backslash_escapes" not in str(exc.value) + # --------------------------------------------------------------------------- # 12. Unit: _substitute_model_sql_surfaces touches ONLY the four surfaces @@ -1052,6 +1128,7 @@ def test_only_mode_a_surfaces_substituted(self) -> None: # Introduced by DEV-1625; import inline so a missing symbol doesn't # break collection of the whole module during TDD phase 1. from slayer.engine.query_engine import _substitute_model_sql_surfaces + from slayer.sql.dialects import SqliteDialect from slayer.core.models import Aggregation @@ -1082,7 +1159,8 @@ def test_only_mode_a_surfaces_substituted(self) -> None: aggregations=[Aggregation(name="agg", formula="SUM({expr}) * {mult}")], ) out = _substitute_model_sql_surfaces( - model=model, variables={"region": "US", "floor": 5, "mult": 2} + model=model, variables={"region": "US", "floor": 5, "mult": 2}, + dialect=SqliteDialect(), ) # The four Mode-A surfaces are substituted (hidden column included): assert out.sql == "SELECT * FROM t WHERE r = 'US'" @@ -1102,6 +1180,7 @@ def test_only_mode_a_surfaces_substituted(self) -> None: def test_empty_variables_is_noop(self) -> None: from slayer.engine.query_engine import _substitute_model_sql_surfaces + from slayer.sql.dialects import SqliteDialect model = SlayerModel( name="m", @@ -1109,9 +1188,43 @@ def test_empty_variables_is_noop(self) -> None: data_source="ds", columns=[Column(name="j", sql="json_extract(x, '$.a')", type=DataType.DOUBLE)], ) - out = _substitute_model_sql_surfaces(model=model, variables={}) + out = _substitute_model_sql_surfaces( + model=model, variables={}, dialect=SqliteDialect() + ) assert out.get_column("j").sql == "json_extract(x, '$.a')" + def test_list_value_substituted_on_all_four_mode_a_surfaces(self) -> None: + """A single list variable rendered into each of the four Mode-A + surfaces — SlayerModel.sql, SlayerModel.filters, Column.sql, + Column.filter — proving every surface routes list values through + ``_render_variable_value`` (all comma-joined, auto-quoted, sql-escape).""" + from slayer.engine.query_engine import _substitute_model_sql_surfaces + from slayer.sql.dialects import SqliteDialect + + model = SlayerModel( + name="m", + data_source="ds", + sql="SELECT * FROM t WHERE region IN ({regions})", + filters=["region IN ({regions})"], + columns=[ + Column(name="in_flag", sql="region IN ({regions})", type=DataType.BOOLEAN), + Column( + name="amt", + sql="amount", + filter="region IN ({regions})", + type=DataType.DOUBLE, + ), + ], + ) + out = _substitute_model_sql_surfaces( + model=model, variables={"regions": ["US", "CA"]}, dialect=SqliteDialect() + ) + rendered = "region IN ('US', 'CA')" + assert out.sql == f"SELECT * FROM t WHERE {rendered}" + assert out.filters == [rendered] + assert out.get_column("in_flag").sql == rendered + assert out.get_column("amt").filter == rendered + # --------------------------------------------------------------------------- # 13. Regression: query-level (Mode-B) filter substitution + unified escaping @@ -1376,3 +1489,1014 @@ async def test_partial_defaults_degrades_to_empty(self) -> None: assert types == {} finally: tmp.cleanup() + + async def test_list_default_probe_succeeds(self) -> None: + """DEV-1730: a Mode-A ``IN ({var})`` filter with a LIST default in + ``query_variables`` renders cleanly at the type-probe path, so the probe + returns real column types instead of the ``{}`` degradation.""" + model = SlayerModel( + name="orders", + sql_table="orders", + data_source="ds", + query_variables={"regions": ["US", "CA"]}, + filters=["region IN ({regions})"], + columns=[ + Column(name="id", sql="id", type=DataType.DOUBLE, primary_key=True), + Column(name="region", sql="region", type=DataType.TEXT), + Column(name="amount", sql="amount", type=DataType.DOUBLE), + ], + ) + engine, tmp = await _engine_with(model) + try: + types = await engine.get_column_types("orders") + assert types.get("amount") == "number" + assert types.get("region") == "string" + finally: + tmp.cleanup() + + +# --------------------------------------------------------------------------- +# 18. Unit: list/tuple variable rendering (DEV-1730 IN-list pushdown) +# --------------------------------------------------------------------------- + + +class TestListValueRenderingSql: + """``escape="sql"`` list rendering: comma-joined, strings auto-quoted and + single-quote-doubled, numbers/bools bare. Template shape is ``IN ({var})`` + — the author writes the parens, NOT the per-element quotes.""" + + def test_sql_list_strings_auto_quoted(self) -> None: + from slayer.core.query import substitute_variables + + result = substitute_variables( + filter_str="region IN ({v})", + variables={"v": ["US", "CA"]}, + escape="sql", backslash_escapes=False, + ) + assert result == "region IN ('US', 'CA')" + + def test_sql_list_embedded_quote_doubled(self) -> None: + from slayer.core.query import substitute_variables + + # Per-element the same sql escaping as scalars: ' → ''. + result = substitute_variables( + filter_str="name IN ({v})", + variables={"v": ["A", "O'Brien", 3]}, + escape="sql", backslash_escapes=False, + ) + assert result == "name IN ('A', 'O''Brien', 3)" + + def test_sql_list_numbers_and_bools_bare(self) -> None: + from slayer.core.query import substitute_variables + + result = substitute_variables( + filter_str="x IN ({v})", + variables={"v": [1, 2.5, True, False]}, + escape="sql", backslash_escapes=False, + ) + assert result == "x IN (1, 2.5, True, False)" + + def test_sql_single_element_list(self) -> None: + from slayer.core.query import substitute_variables + + result = substitute_variables( + filter_str="region IN ({v})", + variables={"v": ["EU"]}, + escape="sql", backslash_escapes=False, + ) + assert result == "region IN ('EU')" + + def test_sql_tuple_accepted_same_as_list(self) -> None: + from slayer.core.query import substitute_variables + + from_list = substitute_variables( + filter_str="x IN ({v})", variables={"v": ["A", "B"]}, + escape="sql", backslash_escapes=False, + ) + from_tuple = substitute_variables( + filter_str="x IN ({v})", variables={"v": ("A", "B")}, + escape="sql", backslash_escapes=False, + ) + assert from_list == from_tuple == "x IN ('A', 'B')" + + def test_sql_injection_element_stays_inside_literal(self) -> None: + from slayer.core.query import substitute_variables + + # A classic breakout attempt: the closing quote is doubled so the whole + # payload stays a single string literal inside the IN list. + result = substitute_variables( + filter_str="region IN ({v})", + variables={"v": ["x') OR ('1'='1"]}, + escape="sql", backslash_escapes=False, + ) + assert result == "region IN ('x'') OR (''1''=''1')" + + +class TestListValueRenderingPython: + """``escape="python"`` list rendering: comma-joined WITH a trailing comma so + the Mode-B Python-AST parser always reads a tuple, never a bare string + (``x in ('A')`` is string membership; ``x in ('A',)`` is a 1-tuple).""" + + def test_python_list_trailing_comma_single(self) -> None: + from slayer.core.query import substitute_variables + + result = substitute_variables( + filter_str="region in ({v})", + variables={"v": ["A"]}, + escape="python", + ) + assert result == "region in ('A',)" + + def test_python_list_trailing_comma_multi(self) -> None: + from slayer.core.query import substitute_variables + + result = substitute_variables( + filter_str="region in ({v})", + variables={"v": ["A", "B"]}, + escape="python", + ) + assert result == "region in ('A', 'B',)" + + def test_python_list_numbers_bare_trailing_comma(self) -> None: + from slayer.core.query import substitute_variables + + result = substitute_variables( + filter_str="x in ({v})", + variables={"v": [1, 2]}, + escape="python", + ) + assert result == "x in (1, 2,)" + + @pytest.mark.parametrize( + "values", [["A"], ["A", "B"], ["O'Brien", "a\\b"], ["a\nb", "x"]] + ) + def test_python_list_ast_roundtrip_is_tuple(self, values: list) -> None: + # The substituted ``x in (...)`` must parse to an ast.Tuple (never a + # bare Constant) whose elements recover the ORIGINAL string values — + # so single-element lists work and quotes/backslashes round-trip. + import ast + + from slayer.core.query import substitute_variables + + substituted = substitute_variables( + filter_str="region in ({v})", + variables={"v": values}, + escape="python", + ) + expr = ast.parse(substituted, mode="eval") + compare = expr.body + assert isinstance(compare, ast.Compare) + rhs = compare.comparators[0] + assert isinstance(rhs, ast.Tuple) + recovered = [e.value for e in rhs.elts] + assert recovered == values + + def test_python_tuple_accepted_same_as_list(self) -> None: + from slayer.core.query import substitute_variables + + from_list = substitute_variables( + filter_str="x in ({v})", variables={"v": ["A", "B"]}, escape="python" + ) + from_tuple = substitute_variables( + filter_str="x in ({v})", variables={"v": ("A", "B")}, escape="python" + ) + assert from_list == from_tuple == "x in ('A', 'B',)" + + +class TestListValueRenderingErrors: + def test_empty_list_raises_naming_variable(self) -> None: + from slayer.core.query import substitute_variables + + # IN () is invalid SQL; the message names the variable, says "empty", and + # points at the sentinel-default idiom (DEV-1730) for "no filter". + with pytest.raises(ValueError, match="regions") as exc: + substitute_variables( + filter_str="region IN ({regions})", + variables={"regions": []}, + escape="sql", backslash_escapes=False, + ) + msg = str(exc.value).lower() + assert "empty" in msg + # The sentinel-default hint is a required contract (DEV-1730), pinned by + # a stable single-word fragment rather than the full sentence. + assert "sentinel" in msg + + def test_empty_tuple_raises(self) -> None: + from slayer.core.query import substitute_variables + + with pytest.raises(ValueError, match="regions") as exc: + substitute_variables( + filter_str="region IN ({regions})", + variables={"regions": ()}, + escape="python", + ) + assert "empty" in str(exc.value).lower() + + def test_nested_list_element_raises(self) -> None: + from slayer.core.query import substitute_variables + + # ``element`` in the match pins the PER-ELEMENT validation path (a valid + # list with one bad element), not the whole-value type rejection; and the + # variable name must appear so the author can locate it. + with pytest.raises(ValueError, match="element") as exc: + substitute_variables( + filter_str="region IN ({regions})", + variables={"regions": ["A", ["B", "C"]]}, + escape="sql", backslash_escapes=False, + ) + assert "regions" in str(exc.value) + + def test_none_element_raises(self) -> None: + from slayer.core.query import substitute_variables + + with pytest.raises(ValueError, match="element") as exc: + substitute_variables( + filter_str="region IN ({regions})", + variables={"regions": ["A", None]}, + escape="sql", backslash_escapes=False, + ) + assert "regions" in str(exc.value) + + @pytest.mark.parametrize("bad", [float("nan"), float("inf"), float("-inf")]) + def test_non_finite_float_element_raises(self, bad: float) -> None: + from slayer.core.query import substitute_variables + + with pytest.raises(ValueError, match="finite") as exc: + substitute_variables( + filter_str="x IN ({v})", + variables={"v": [1.0, bad]}, + escape="sql", backslash_escapes=False, + ) + assert "v" in str(exc.value) + + def test_dict_element_raises(self) -> None: + from slayer.core.query import substitute_variables + + with pytest.raises(ValueError, match="element") as exc: + substitute_variables( + filter_str="region IN ({regions})", + variables={"regions": ["A", {"b": 1}]}, + escape="python", + ) + assert "regions" in str(exc.value) + + +# --------------------------------------------------------------------------- +# 19. End-to-end Mode-A: WHERE region IN ({regions}) with a list variable +# --------------------------------------------------------------------------- + + +class TestListModeAEndToEnd: + def _model(self, **kw) -> SlayerModel: + return SlayerModel( + name="orders", + sql_table="orders", + data_source="ds", + filters=["region IN ({regions})"], + columns=[ + Column(name="id", sql="id", type=DataType.DOUBLE, primary_key=True), + Column(name="region", sql="region", type=DataType.TEXT), + Column(name="amount", sql="amount", type=DataType.DOUBLE), + ], + **kw, + ) + + async def test_runtime_list_filters_rows(self) -> None: + engine, tmp = await _engine_with(self._model()) + try: + q = SlayerQuery( + source_model="orders", + measures=[{"formula": "amount:sum"}], + variables={"regions": ["US", "CA"]}, + ) + resp = await engine.execute(q) + # US=160, CA=300 → 460 (EU excluded). + assert _sum(resp, "orders.amount_sum") == 460.0 + finally: + tmp.cleanup() + + async def test_single_element_list(self) -> None: + engine, tmp = await _engine_with(self._model()) + try: + q = SlayerQuery( + source_model="orders", + measures=[{"formula": "amount:sum"}], + variables={"regions": ["EU"]}, + ) + resp = await engine.execute(q) + assert _sum(resp, "orders.amount_sum") == 275.0 + finally: + tmp.cleanup() + + async def test_list_default_used_when_kwarg_absent(self) -> None: + engine, tmp = await _engine_with( + self._model(query_variables={"regions": ["EU", "CA"]}) + ) + try: + q = SlayerQuery(source_model="orders", measures=[{"formula": "amount:sum"}]) + resp = await engine.execute(q) + # EU=275, CA=300 → 575. + assert _sum(resp, "orders.amount_sum") == 575.0 + finally: + tmp.cleanup() + + async def test_query_variables_list_overrides_model_default_list(self) -> None: + # Middle precedence layer: model_defaults < query.variables (no runtime). + engine, tmp = await _engine_with( + self._model(query_variables={"regions": ["EU", "CA"]}) + ) + try: + q = SlayerQuery( + source_model="orders", + measures=[{"formula": "amount:sum"}], + variables={"regions": ["US"]}, + ) + resp = await engine.execute(q) + # Query-level ['US'] wins over model default ['EU','CA'] → 160. + assert _sum(resp, "orders.amount_sum") == 160.0 + finally: + tmp.cleanup() + + async def test_runtime_list_overrides_default_list(self) -> None: + engine, tmp = await _engine_with( + self._model(query_variables={"regions": ["EU", "CA"]}) + ) + try: + q = SlayerQuery(source_model="orders", measures=[{"formula": "amount:sum"}]) + resp = await engine.execute(q, variables={"regions": ["US"]}) + # Runtime ['US'] wins over default ['EU','CA'] → 160. + assert _sum(resp, "orders.amount_sum") == 160.0 + finally: + tmp.cleanup() + + async def test_injection_element_matches_no_rows(self) -> None: + engine, tmp = await _engine_with(self._model()) + try: + q = SlayerQuery( + source_model="orders", + measures=[{"formula": "amount:sum"}], + variables={"regions": ["x') OR ('1'='1"]}, + ) + resp = await engine.execute(q) + # The payload stays inside its literal, matching no region, so the + # aggregate is NULL — NOT the whole-table total (735) an escape + # would have produced. + assert resp.data[0]["orders.amount_sum"] is None + finally: + tmp.cleanup() + + +# --------------------------------------------------------------------------- +# 20. End-to-end Mode-B: query filter "region in ({regions})" with a list +# --------------------------------------------------------------------------- + + +class TestListModeBEndToEnd: + def _model(self) -> SlayerModel: + return SlayerModel( + name="orders", + sql_table="orders", + data_source="ds", + columns=[ + Column(name="id", sql="id", type=DataType.DOUBLE, primary_key=True), + Column(name="region", sql="region", type=DataType.TEXT), + Column(name="status", sql="status", type=DataType.TEXT), + Column(name="amount", sql="amount", type=DataType.DOUBLE), + ], + ) + + async def test_query_filter_list_in(self) -> None: + engine, tmp = await _engine_with(self._model()) + try: + q = SlayerQuery( + source_model="orders", + measures=[{"formula": "amount:sum"}], + filters=["region in ({regions})"], + variables={"regions": ["US", "CA"]}, + ) + resp = await engine.execute(q) + assert _sum(resp, "orders.amount_sum") == 460.0 # US 160 + CA 300 + finally: + tmp.cleanup() + + async def test_query_filter_single_element_list(self) -> None: + engine, tmp = await _engine_with(self._model()) + try: + # The trailing-comma render (('EU',)) is what keeps a 1-element list + # a tuple through the Python-AST parser rather than a bare string. + q = SlayerQuery( + source_model="orders", + measures=[{"formula": "amount:sum"}], + filters=["region in ({regions})"], + variables={"regions": ["EU"]}, + ) + resp = await engine.execute(q) + assert _sum(resp, "orders.amount_sum") == 275.0 + finally: + tmp.cleanup() + + async def test_query_filter_not_in_list(self) -> None: + engine, tmp = await _engine_with(self._model()) + try: + q = SlayerQuery( + source_model="orders", + measures=[{"formula": "amount:sum"}], + filters=["region not in ({regions})"], + variables={"regions": ["US"]}, + ) + resp = await engine.execute(q) + # Everything except US(160) → EU 275 + CA 300 = 575. + assert _sum(resp, "orders.amount_sum") == 575.0 + finally: + tmp.cleanup() + + async def test_runtime_list_overrides_query_level_list(self) -> None: + # Mode-B filters read query.variables, which merges runtime kwarg over + # the query-level dict (runtime wins) — verify that merge with lists. + engine, tmp = await _engine_with(self._model()) + try: + q = SlayerQuery( + source_model="orders", + measures=[{"formula": "amount:sum"}], + filters=["region in ({regions})"], + variables={"regions": ["EU", "CA"]}, + ) + resp = await engine.execute(q, variables={"regions": ["US"]}) + assert _sum(resp, "orders.amount_sum") == 160.0 # runtime ['US'] wins + finally: + tmp.cleanup() + + async def test_query_filter_list_escaping_matches_only_its_row(self) -> None: + # Mode-B escaping through the real parse+compile+SQLite pipeline: a + # quote-bearing element must round-trip so it matches only its own row. + engine, tmp = await _engine_with(self._model()) + try: + q = SlayerQuery( + source_model="orders", + measures=[{"formula": "amount:sum"}], + filters=["status in ({statuses})"], + variables={"statuses": ["O'Brien"]}, + ) + resp = await engine.execute(q) + assert _sum(resp, "orders.amount_sum") == 10.0 # row 6 only + finally: + tmp.cleanup() + + +# =========================================================================== +# DEV-1727 — dialect-aware / complete escaping for Mode-A {var} substitution +# =========================================================================== + +import ast # noqa: E402 +import sqlglot # noqa: E402 + +from slayer.core.query import substitute_variables # noqa: E402 +from slayer.sql.dialects import _ALL_DIALECTS # noqa: E402 + + +# --------------------------------------------------------------------------- +# 19. SQL regime is dialect-aware: backslash_escapes flag (Gap 1) +# --------------------------------------------------------------------------- + +class TestSqlModeBackslashEscaping: + """``escape="sql"`` with ``backslash_escapes=`` selects the escaping regime. + + - False (standard: SQLite/Postgres/DuckDB/…): backslash is an ordinary + char, ONLY the single quote is doubled (``'`` → ``''``). Unchanged from + DEV-1625. + - True (backslash dialects: MySQL/ClickHouse/Snowflake/…): backslash is an + escape char, so it is doubled FIRST (``\\`` → ``\\\\``) and the single + quote is backslash-escaped (``'`` → ``\\'``). The double quote is left + untouched (inside a single-quoted literal it is an ordinary char on every + dialect, and ``\\"`` is not a recognised escape on 6 of the 7). + """ + + def test_standard_regime_doubles_quote_leaves_backslash(self) -> None: + result = substitute_variables( + filter_str="p = '{v}'", variables={"v": "a\\'b"}, + escape="sql", backslash_escapes=False, + ) + # a \ ' b → double the quote only; backslash untouched. + assert result == "p = 'a\\''b'" + + def test_backslash_regime_doubles_backslash_and_escapes_quote(self) -> None: + result = substitute_variables( + filter_str="p = '{v}'", variables={"v": "a\\'b"}, + escape="sql", backslash_escapes=True, + ) + # a \ ' b → \\ (doubled backslash) + \' (escaped quote). + assert result == "p = 'a\\\\\\'b'" + + def test_backslash_regime_lone_backslash_doubled(self) -> None: + result = substitute_variables( + filter_str="p = '{v}'", variables={"v": "a\\b"}, + escape="sql", backslash_escapes=True, + ) + assert result == "p = 'a\\\\b'" + + def test_backslash_regime_trailing_backslash_doubled(self) -> None: + # The classic breakout: a trailing backslash must not eat the closing + # quote — it is doubled so the literal stays closed. + result = substitute_variables( + filter_str="p = '{v}'", variables={"v": "abc\\"}, + escape="sql", backslash_escapes=True, + ) + assert result == "p = 'abc\\\\'" + + def test_backslash_regime_single_quote_only(self) -> None: + result = substitute_variables( + filter_str="p = '{v}'", variables={"v": "O'Brien"}, + escape="sql", backslash_escapes=True, + ) + assert result == "p = 'O\\'Brien'" + + def test_backslash_regime_double_quote_untouched(self) -> None: + # Inside a single-quoted literal a bare double quote is an ordinary + # char; escaping it (\") would CORRUPT the value on 6 of 7 backslash + # dialects (only MySQL treats \" as " there). So it stays as-is. + result = substitute_variables( + filter_str="p = '{v}'", variables={"v": 'say "hi"'}, + escape="sql", backslash_escapes=True, + ) + assert result == 'p = \'say "hi"\'' + + def test_backslash_regime_plain_value_unchanged(self) -> None: + result = substitute_variables( + filter_str="p = '{v}'", variables={"v": "active"}, + escape="sql", backslash_escapes=True, + ) + assert result == "p = 'active'" + + def test_number_passthrough_both_regimes(self) -> None: + for be in (True, False): + assert substitute_variables( + filter_str="x > {n}", variables={"n": 100}, + escape="sql", backslash_escapes=be, + ) == "x > 100" + + def test_backslash_regime_list_elements_escaped(self) -> None: + # DEV-1730 list rendering must compose with the backslash regime: + # each string element is auto-quoted AND backslash-escaped. + result = substitute_variables( + filter_str="p IN ({v})", variables={"v": ["a\\b", "O'Brien"]}, + escape="sql", backslash_escapes=True, + ) + assert result == "p IN ('a\\\\b', 'O\\'Brien')" + + def test_backslash_regime_list_elements_standard(self) -> None: + # Same list under the standard regime: quote-doubled, backslash raw. + result = substitute_variables( + filter_str="p IN ({v})", variables={"v": ["a\\b", "O'Brien"]}, + escape="sql", backslash_escapes=False, + ) + assert result == "p IN ('a\\b', 'O''Brien')" + + +# --------------------------------------------------------------------------- +# 20. Fail-closed: escape="sql" requires the backslash_escapes signal +# --------------------------------------------------------------------------- + +class TestSqlModeFailClosed: + """A security-flavoured property (correct escaping) must not silently + default. ``escape="sql"`` requires ``backslash_escapes`` to be specified — + a new/overlooked SQL call site that forgets it FAILS instead of + under-escaping on MySQL/ClickHouse.""" + + def test_sql_without_backslash_escapes_raises(self) -> None: + with pytest.raises(ValueError, match="backslash_escapes"): + substitute_variables( + filter_str="p = '{v}'", variables={"v": "x"}, escape="sql" + ) + + def test_sql_without_flag_raises_even_for_number(self) -> None: + # The guard fires at the sql-mode boundary regardless of value type — + # so no caller can partially bypass it by happening to pass a number. + with pytest.raises(ValueError, match="backslash_escapes"): + substitute_variables( + filter_str="x > {n}", variables={"n": 1}, escape="sql" + ) + + def test_python_mode_ignores_backslash_escapes(self) -> None: + # python mode never needs the flag; passing it is accepted and ignored + # (both True and False yield the identical python-escaped result). + base = substitute_variables( + filter_str="p = '{v}'", variables={"v": "O'Brien"}, escape="python" + ) + for be in (True, False, None): + assert substitute_variables( + filter_str="p = '{v}'", variables={"v": "O'Brien"}, + escape="python", backslash_escapes=be, + ) == base + + +# --------------------------------------------------------------------------- +# 21. Python regime full C0-control pass (Gap 2) +# --------------------------------------------------------------------------- + +def _python_literal_value(substituted_literal: str): + """Parse a single-quoted python literal string (as produced by the + escaping) and return the recovered value via ast.literal_eval.""" + return ast.literal_eval(substituted_literal) + + +class TestPythonModeControlChars: + """After backslash/quote escaping, ``escape="python"`` encodes every C0 + control char (U+0000–U+001F) so SLayer's ast.parse-based Mode-B parser — + which rejects a raw newline / NUL inside a string literal — recovers the + original value.""" + + def test_newline_named_escape(self) -> None: + result = substitute_variables( + filter_str="'{v}'", variables={"v": "a\nb"}, escape="python" + ) + assert result == "'a\\nb'" + # Recover from the ACTUAL produced literal (not a hard-coded one). + assert _python_literal_value(result) == "a\nb" + + def test_carriage_return_named_escape(self) -> None: + result = substitute_variables( + filter_str="p = '{v}'", variables={"v": "a\rb"}, escape="python" + ) + assert result == "p = 'a\\rb'" + + def test_tab_named_escape(self) -> None: + result = substitute_variables( + filter_str="p = '{v}'", variables={"v": "a\tb"}, escape="python" + ) + assert result == "p = 'a\\tb'" + + def test_nul_hex_escape(self) -> None: + result = substitute_variables( + filter_str="p = '{v}'", variables={"v": "a\x00b"}, escape="python" + ) + assert result == "p = 'a\\x00b'" + + def test_other_c0_hex_escape(self) -> None: + # Vertical tab (0x0b) has no named escape → \x0b. + result = substitute_variables( + filter_str="p = '{v}'", variables={"v": "a\x0bb"}, escape="python" + ) + assert result == "p = 'a\\x0bb'" + + @pytest.mark.parametrize("codepoint", list(range(0x00, 0x20))) + def test_every_c0_char_roundtrips_through_ast(self, codepoint: int) -> None: + # The whole C0 range must round-trip: substitute → the produced literal + # is a valid python string literal recovering the original value. + value = f"x{chr(codepoint)}y" + substituted = substitute_variables( + filter_str="'{v}'", variables={"v": value}, escape="python" + ) + # ast.parse must not raise, and the recovered value must be identical. + assert _python_literal_value(substituted) == value + + @pytest.mark.parametrize("codepoint", list(range(0x00, 0x20))) + def test_every_c0_char_exact_rendering(self, codepoint: int) -> None: + # Pin the EXACT encoding (not merely "some valid escape"): \t\n\r use + # their named escape, every other C0 char uses lowercase \xNN. This + # locks the rendering contract so an implementation that used octal or + # \u would still be caught even though ast.parse would accept it. + char = chr(codepoint) + named = {"\t": "\\t", "\n": "\\n", "\r": "\\r"} + expected_escape = named.get(char, f"\\x{codepoint:02x}") + substituted = substitute_variables( + filter_str="'{v}'", variables={"v": char}, escape="python" + ) + assert substituted == f"'{expected_escape}'" + + @pytest.mark.parametrize("codepoint", list(range(0x00, 0x20))) + def test_every_c0_char_roundtrips_in_list_element(self, codepoint: int) -> None: + # DEV-1730 list elements go through the same escaping; a control char in + # an element must round-trip so the ast tuple recovers it. + value = f"x{chr(codepoint)}y" + substituted = substitute_variables( + filter_str="c in ({v})", variables={"v": [value]}, escape="python" + ) + expr = ast.parse(substituted, mode="eval") + rhs = expr.body.comparators[0] + assert isinstance(rhs, ast.Tuple) + assert [e.value for e in rhs.elts] == [value] + + def test_non_control_unicode_untouched(self) -> None: + # A non-C0 char (accented letter, U+2028 line separator) is NOT a C0 + # control char, so the pass leaves it verbatim. + for value in ("café", "a
b", "emoji😀"): + substituted = substitute_variables( + filter_str="'{v}'", variables={"v": value}, escape="python" + ) + assert _python_literal_value(substituted) == value + + @pytest.mark.parametrize( + "codepoint", + [0x2028, 0x2029, 0x0085, 0x00A0, 0x00E9, 0x1F600], + ids=["line-sep", "para-sep", "nel", "nbsp", "e-acute", "emoji"], + ) + def test_non_c0_char_textually_untouched(self, codepoint: int) -> None: + # Non-C0 chars (incl. the deceptively line-break-looking U+2028/U+2029, + # U+0085 NEL) are NOT encoded — they are left in the output byte-for-byte + # AND still round-trip (ast recovers them). A weaker recovery-only check + # would also pass if we'd wrongly \\u-escaped them, so assert BOTH. + char = chr(codepoint) + value = f"x{char}y" + substituted = substitute_variables( + filter_str="'{v}'", variables={"v": value}, escape="python" + ) + assert char in substituted # textually present, not escaped + assert substituted == f"'{value}'" + assert _python_literal_value(substituted) == value + + def test_backslash_before_control_char(self) -> None: + # Order: backslash doubled first, THEN the control char encoded — a + # value of backslash+newline must recover exactly. + value = "\\\n" + substituted = substitute_variables( + filter_str="'{v}'", variables={"v": value}, escape="python" + ) + assert substituted == "'\\\\\\n'" + assert _python_literal_value(substituted) == value + + def test_sql_mode_leaves_control_chars_raw(self) -> None: + # The C0 pass is python-only: SQL literals accept raw newlines and + # sqlglot re-emits them, so sql mode must NOT encode control chars. + result = substitute_variables( + filter_str="p = '{v}'", variables={"v": "a\nb"}, + escape="sql", backslash_escapes=False, + ) + assert result == "p = 'a\nb'" + + +# --------------------------------------------------------------------------- +# 22. Dialect matrix: escaping round-trips through the parser sqlglot uses +# --------------------------------------------------------------------------- + +_MATRIX_VALUES = [ + "O'Brien", # single quote + 'say "hi"', # double quote + "a\\b", # backslash + "a\\'b", # backslash + single quote + 'a\\"b', # backslash + double quote + "abc\\", # trailing backslash (closing-quote eater) + "it's a \"mix\"", # both quote styles + "line1\nline2", # raw newline (legal in a SQL literal) + "tab\there", # raw tab + "plain", # nothing special +] + +# (dialect, value) pairs — the escaping regime flag is read from the dialect's +# own ``backslash_escapes_strings`` property AT TEST TIME (not here), so this +# module still imports cleanly before the property is implemented. +_MATRIX = [ + (d, v) for d in _ALL_DIALECTS for v in _MATRIX_VALUES +] + + +class TestEscapingRegimeDialectMatrix: + """For every dialect, the sql-escaped literal produced by + ``substitute_variables`` — using the regime the dialect's OWN + ``backslash_escapes_strings`` selects — must round-trip through sqlglot's + parser for that dialect (the exact parser SLayer feeds the substituted + Mode-A SQL to), AND survive a parse → emit → re-parse cycle. This is the + core correctness guarantee that the derived flag matches the parser.""" + + @pytest.mark.parametrize( + "dialect,value", + _MATRIX, + ids=[f"{d.sqlglot_name}-{v!r}" for d, v in _MATRIX], + ) + def test_sql_literal_roundtrips_through_dialect_parser( + self, dialect, value: str + ) -> None: + literal = substitute_variables( + filter_str="'{v}'", variables={"v": value}, + escape="sql", backslash_escapes=dialect.backslash_escapes_strings, + ) + parsed = sqlglot.parse_one(literal, dialect=dialect.sqlglot_name) + assert parsed.this == value, (dialect.sqlglot_name, repr(literal)) + # parse → emit → re-parse must preserve the value too. + reparsed = sqlglot.parse_one( + parsed.sql(dialect=dialect.sqlglot_name), dialect=dialect.sqlglot_name + ) + assert reparsed.this == value, (dialect.sqlglot_name, "roundtrip") + + def test_matrix_covers_all_14_dialects(self) -> None: + assert len({d.sqlglot_name for d, _ in _MATRIX}) == 14 + + +# --------------------------------------------------------------------------- +# 23. Engine threading: the resolved dialect reaches the sql escaping +# --------------------------------------------------------------------------- + +class TestSubstituteModelSqlSurfacesDialect: + """``_substitute_model_sql_surfaces`` requires a dialect and derives the + escaping regime from it (fail-closed — no caller can under-escape by + forgetting a bool).""" + + def _model(self) -> SlayerModel: + return SlayerModel( + name="m", + data_source="ds", + sql="SELECT * FROM t WHERE r = '{region}'", + columns=[Column(name="a", sql="amount", type=DataType.DOUBLE)], + ) + + def test_backslash_dialect_escapes_value(self) -> None: + from slayer.engine.query_engine import _substitute_model_sql_surfaces + from slayer.sql.dialects import MysqlDialect + + out = _substitute_model_sql_surfaces( + model=self._model(), variables={"region": "a\\'b"}, + dialect=MysqlDialect(), + ) + assert out.sql == "SELECT * FROM t WHERE r = 'a\\\\\\'b'" + + def test_standard_dialect_doubles_quote_only(self) -> None: + from slayer.engine.query_engine import _substitute_model_sql_surfaces + from slayer.sql.dialects import SqliteDialect + + out = _substitute_model_sql_surfaces( + model=self._model(), variables={"region": "a\\'b"}, + dialect=SqliteDialect(), + ) + assert out.sql == "SELECT * FROM t WHERE r = 'a\\''b'" + + def test_dialect_is_required(self) -> None: + from slayer.engine.query_engine import _substitute_model_sql_surfaces + + # Build the model outside the raises-block so only the call under test + # (missing the required `dialect`) can raise (Sonar S5778). + model = self._model() + with pytest.raises(TypeError): + _substitute_model_sql_surfaces(model=model, variables={"region": "x"}) + + +def _status_literal_from_sql(sql: str, dialect_name: str): + """Parse generated SQL for ``dialect_name`` and return the string value of + the ``status = '...'`` literal (the round-tripped variable value).""" + tree = sqlglot.parse_one(sql, dialect=dialect_name) + for eq in tree.find_all(sqlglot.exp.EQ): + lit = eq.expression + if isinstance(lit, sqlglot.exp.Literal) and lit.is_string: + return lit.this + return None + + +class TestEngineDialectThreadingEndToEnd: + """Through the real engine (dry_run, no live backslash DB): the datasource's + dialect must reach the Mode-A escaping. The generated SQL is parsed+re-emitted + by sqlglot, so we assert the VALUE ROUND-TRIPS (the escaped literal recovers + the original), not a raw substring — the discriminating proof, since a + mis-threaded (standard) regime on MySQL would make sqlglot's MySQL parser + raise on ``'a\\''b'`` and the dry_run would fail outright.""" + + async def _dry_run_sql_for(self, ds_type: str, value: str) -> str: + """Build a one-model engine on a datasource of ``ds_type`` and return + the generated SQL for a Mode-A model filter carrying ``value``. Uses a + SQLite file as the actual backend but pins the datasource *type* so the + dialect (hence escaping regime) is exercised; dry_run skips execution.""" + tmp = tempfile.TemporaryDirectory() + db_path = f"{tmp.name}/x.db" + _seed_orders_db_at(db_path) + storage = YAMLStorage(base_dir=tmp.name) + await storage.save_datasource( + DatasourceConfig(name="ds", type=ds_type, database=db_path) + ) + model = SlayerModel( + name="orders", + sql_table="orders", + data_source="ds", + filters=["status = '{v}'"], + columns=[ + Column(name="id", sql="id", type=DataType.DOUBLE, primary_key=True), + Column(name="status", sql="status", type=DataType.TEXT), + Column(name="amount", sql="amount", type=DataType.DOUBLE), + ], + ) + await storage.save_model(model) + engine = SlayerQueryEngine(storage=storage) + try: + q = SlayerQuery( + source_model="orders", + measures=[{"formula": "amount:sum"}], + variables={"v": value}, + ) + result = await engine.execute(q, dry_run=True) + return result.sql + finally: + tmp.cleanup() + + async def test_backslash_dialect_threads_regime_so_value_roundtrips(self) -> None: + # A backslash+quote value on a mysql-type datasource: the dry_run only + # SUCCEEDS (and the literal round-trips) if the MySQL backslash regime + # was threaded — the naive/standard regime would emit ``'a\\''b'`` which + # sqlglot's MySQL parser rejects, raising before any SQL is returned. + sql = await self._dry_run_sql_for("mysql", "a\\'b") + assert _status_literal_from_sql(sql, "mysql") == "a\\'b" + + async def test_standard_dialect_value_roundtrips(self) -> None: + # Standard (sqlite) path: a quote-bearing (backslash-free) value + # round-trips end-to-end. Backslash values are the pre-existing SQLite + # gap (see the strict-xfail above), so they are deliberately not used. + sql = await self._dry_run_sql_for("sqlite", "O'Brien") + assert _status_literal_from_sql(sql, "sqlite") == "O'Brien" + + +class TestProbeModelDialectThreading: + """``_render_probe_model`` threads the dialect into the Mode-A surfaces it + renders from the model's own defaults.""" + + def _template_model(self) -> SlayerModel: + return SlayerModel( + name="m", + data_source="ds", + sql="SELECT * FROM t WHERE r = '{region}'", + query_variables={"region": "a\\'b"}, + columns=[Column(name="a", sql="amount", type=DataType.DOUBLE)], + ) + + def test_probe_uses_backslash_dialect(self) -> None: + from slayer.engine.query_engine import _render_probe_model + from slayer.sql.dialects import MysqlDialect + + out = _render_probe_model(self._template_model(), dialect=MysqlDialect()) + assert out.sql == "SELECT * FROM t WHERE r = 'a\\\\\\'b'" + + def test_probe_uses_standard_dialect(self) -> None: + from slayer.engine.query_engine import _render_probe_model + from slayer.sql.dialects import SqliteDialect + + out = _render_probe_model(self._template_model(), dialect=SqliteDialect()) + assert out.sql == "SELECT * FROM t WHERE r = 'a\\''b'" + + def test_probe_dialect_required(self) -> None: + from slayer.engine.query_engine import _render_probe_model + + # Build the model outside the raises-block so only the call under test + # (missing the required `dialect`) can raise (Sonar S5778). + model = self._template_model() + with pytest.raises(TypeError): + _render_probe_model(model) + + +# --------------------------------------------------------------------------- +# 24. Mode-B end-to-end: control-char values round-trip through the parser +# --------------------------------------------------------------------------- + +async def _engine_with_ctrl_status(model: SlayerModel) -> tuple: + """Seed a table whose status column carries control-char values, so a + Mode-B (python-escaped) query filter can be shown to match end-to-end.""" + tmp = tempfile.TemporaryDirectory() + db_path = f"{tmp.name}/ctrl.db" + conn = sqlite3.connect(db_path) + cur = conn.cursor() + cur.execute( + "CREATE TABLE t (id INTEGER PRIMARY KEY, status TEXT NOT NULL, amount REAL NOT NULL)" + ) + cur.executemany( + "INSERT INTO t (status, amount) VALUES (?, ?)", + [("line1\nline2", 11.0), ("tab\there", 22.0), ("cr\rhere", 33.0), ("plain", 44.0)], + ) + conn.commit() + conn.close() + storage = YAMLStorage(base_dir=tmp.name) + await storage.save_datasource( + DatasourceConfig(name="ds", type="sqlite", database=db_path) + ) + await storage.save_model(model) + return SlayerQueryEngine(storage=storage), tmp + + +class TestModeBControlCharsEndToEnd: + """Legitimate control-char values (Gap 2): a newline/tab/CR-bearing value in + a Mode-B query filter must parse (no ast.parse SyntaxError) and match ONLY + its row through the real SQLite pipeline.""" + + def _model(self) -> SlayerModel: + return SlayerModel( + name="t", + sql_table="t", + data_source="ds", + columns=[ + Column(name="id", sql="id", type=DataType.DOUBLE, primary_key=True), + Column(name="status", sql="status", type=DataType.TEXT), + Column(name="amount", sql="amount", type=DataType.DOUBLE), + ], + ) + + @pytest.mark.parametrize( + "value,expected", + [("line1\nline2", 11.0), ("tab\there", 22.0), ("cr\rhere", 33.0)], + ) + async def test_control_char_value_matches_only_its_row( + self, value: str, expected: float + ) -> None: + engine, tmp = await _engine_with_ctrl_status(self._model()) + try: + q = SlayerQuery( + source_model="t", + measures=[{"formula": "amount:sum"}], + filters=["status = '{v}'"], + variables={"v": value}, + ) + resp = await engine.execute(q) + assert _sum(resp, "t.amount_sum") == expected + finally: + tmp.cleanup() diff --git a/tests/test_model_variables_inspect.py b/tests/test_model_variables_inspect.py new file mode 100644 index 00000000..00da9111 --- /dev/null +++ b/tests/test_model_variables_inspect.py @@ -0,0 +1,55 @@ +"""Mode-A variable discoverability in inspect renders (DEV-1730). + +An sql-mode model parameterised with ``{var}`` / ``{? ... ?}`` surfaces its +required/optional variables through the model skeleton (additive, no schema +bump), so an agent can learn the query contract without reading the SQL. +""" + +from slayer.core.enums import DataType +from slayer.core.models import Column, SlayerModel +from slayer.inspect.model_render import ( + model_skeleton_fields, + render_model_skeleton, +) + + +def _model() -> SlayerModel: + return SlayerModel( + name="rr", + sql="SELECT '{d_from}' AS d FROM t WHERE 1=1 " + "AND {? brand IN ({brand}) ?} AND {? mkt IN ({market}) ?}", + data_source="ds", + columns=[Column(name="d", sql="d", type=DataType.TIMESTAMP)], + ) + + +def test_skeleton_fields_carry_variables(): + fields = model_skeleton_fields(model=_model()) + assert fields["variables"]["required"] == ["d_from"] + assert fields["variables"]["optional"] == ["brand", "market"] + + +def test_skeleton_render_lists_variables_line(): + out = render_model_skeleton(model=_model()) + assert "Variables:" in out + assert "d_from" in out + assert "brand" in out + assert "market" in out + + +def test_skeleton_render_omits_variables_line_when_none(): + plain = SlayerModel( + name="m", sql_table="t", data_source="ds", + columns=[Column(name="a", sql="a", type=DataType.TEXT)], + ) + out = render_model_skeleton(model=plain) + assert "Variables:" not in out + + +def test_variables_are_derived_not_persisted_no_schema_bump(): + # Discoverability is structural: no new persisted SlayerModel field, so the + # serialized schema version is unchanged and no `variables` key is stored. + model = _model() + dumped = model.model_dump() + assert "variables" not in dumped + assert dumped["version"] == SlayerModel.model_fields["version"].default diff --git a/tests/test_models.py b/tests/test_models.py index fa4637d2..48f4bfac 100644 --- a/tests/test_models.py +++ b/tests/test_models.py @@ -1719,13 +1719,26 @@ def test_invalid_variable_name_raises(self) -> None: def test_invalid_type_raises(self) -> None: from slayer.core.query import substitute_variables - with pytest.raises(ValueError, match="must be a string or number"): + # A dict is neither scalar nor list/tuple → rejected. (Lists ARE now a + # valid IN-list body — DEV-1730 — covered in + # tests/test_mode_a_variable_substitution.py.) + with pytest.raises(ValueError, match="must be a string, number, or list/tuple"): substitute_variables( filter_str="status = '{val}'", - variables={"val": [1, 2, 3]}, + variables={"val": {"a": 1}}, escape="python", ) + def test_list_value_renders_in_list(self) -> None: + from slayer.core.query import substitute_variables + + result = substitute_variables( + filter_str="status in ({vals})", + variables={"vals": ["a", "b"]}, + escape="python", + ) + assert result == "status in ('a', 'b',)" + def test_no_variables_no_change(self) -> None: from slayer.core.query import substitute_variables diff --git a/tests/test_query_optional_blocks.py b/tests/test_query_optional_blocks.py new file mode 100644 index 00000000..d81d5418 --- /dev/null +++ b/tests/test_query_optional_blocks.py @@ -0,0 +1,207 @@ +"""Optional-block ``{? ... ?}`` substitution + structural variable extraction (DEV-1730). + +These pin the Mode-A optional-filter idiom used to represent Cube FILTER_PARAMS +pushdowns: a delimited block that renders its content (parenthesised) when every +``{var}`` inside is supplied, and collapses to the neutral ``(1=1)`` when any is +missing. Plus ``extract_model_variables`` — the structural required/optional +classifier that backs inspect discoverability. +""" + +import pytest + +from slayer.core.models import Column, SlayerModel +from slayer.core.query import extract_model_variables, substitute_variables + +# ── block rendering (escape="sql") ────────────────────────────────────────── + + +def test_block_renders_parenthesised_when_var_present(): + out = substitute_variables( + "WHERE 1=1 AND {? brand IN ({brand}) ?}", + {"brand": ["acme", "zeta"]}, + escape="sql", backslash_escapes=False, + ) + assert out == "WHERE 1=1 AND (brand IN ('acme', 'zeta'))" + + +def test_block_collapses_to_one_equals_one_when_var_missing(): + out = substitute_variables( + "WHERE 1=1 AND {? brand IN ({brand}) ?}", + {}, + escape="sql", backslash_escapes=False, + ) + assert out == "WHERE 1=1 AND (1=1)" + + +def test_block_collapses_if_any_inner_var_missing(): + # from present, to absent -> whole block collapses. + out = substitute_variables( + "AND {? d >= '{d_from}' AND d <= '{d_to}' ?}", + {"d_from": "2025-01-01"}, + escape="sql", backslash_escapes=False, + ) + assert out == "AND (1=1)" + + +def test_block_renders_range_when_all_present(): + out = substitute_variables( + "AND {? d >= '{d_from}' AND d <= '{d_to}' ?}", + {"d_from": "2025-01-01", "d_to": "2025-12-31"}, + escape="sql", backslash_escapes=False, + ) + assert out == "AND (d >= '2025-01-01' AND d <= '2025-12-31')" + + +def test_scalar_position_block_collapse_reproduces_cube_cast_shape(): + # Optional arrow in scalar position -> the (1=1)::TYPE Cube "booby-trap". + out = substitute_variables( + "{? '{d_from}' ?}::TIMESTAMP AS d", {}, escape="sql", backslash_escapes=False + ) + assert out == "(1=1)::TIMESTAMP AS d" + + +def test_multiple_independent_blocks_mix_present_and_missing(): + out = substitute_variables( + "WHERE 1=1 AND {? a IN ({a}) ?} AND {? b IN ({b}) ?}", + {"a": ["x"]}, + escape="sql", backslash_escapes=False, + ) + assert out == "WHERE 1=1 AND (a IN ('x')) AND (1=1)" + + +def test_string_value_inside_block_is_escaped(): + out = substitute_variables( + "{? name = '{name}' ?}", {"name": "O'Brien"}, escape="sql", backslash_escapes=False + ) + assert out == "(name = 'O''Brien')" + + +def test_block_with_list_containing_apostrophe_and_comma(): + # Block + IN-list + SQL escaping interaction: commas inside a value must + # not split it into two list elements; apostrophes are quote-doubled. + out = substitute_variables( + "{? brand IN ({brand}) ?}", + {"brand": ["O'Reilly", "ACME, Inc."]}, + escape="sql", backslash_escapes=False, + ) + assert out == "(brand IN ('O''Reilly', 'ACME, Inc.'))" + + +def test_plain_vars_still_substitute_alongside_blocks(): + out = substitute_variables( + "d >= '{d_from}' AND {? brand IN ({brand}) ?}", + {"d_from": "2025-01-01", "brand": ["a"]}, + escape="sql", backslash_escapes=False, + ) + assert out == "d >= '2025-01-01' AND (brand IN ('a'))" + + +def test_brace_escapes_do_not_open_or_close_a_block(): + # {{ and }} stay literal; no block is parsed here. + out = substitute_variables("{{not a block}}", {}, escape="sql", backslash_escapes=False) + assert out == "{not a block}" + + +# ── block error cases ─────────────────────────────────────────────────────── + + +def test_nested_block_raises(): + with pytest.raises(ValueError, match="[Nn]est"): + substitute_variables("{? a {? {b} ?} ?}", {"b": "1"}, escape="sql", backslash_escapes=False) + + +def test_unterminated_block_raises(): + with pytest.raises(ValueError, match="[Uu]nterminated|unclosed"): + substitute_variables("AND {? brand IN ({brand})", {"brand": ["a"]}, escape="sql", backslash_escapes=False) + + +def test_stray_block_close_raises(): + with pytest.raises(ValueError): + substitute_variables("AND brand ?}", {}, escape="sql", backslash_escapes=False) + + +def test_block_with_no_variables_raises(): + with pytest.raises(ValueError, match="[Vv]ariable"): + substitute_variables("{? 1=1 ?}", {}, escape="sql", backslash_escapes=False) + + +def test_block_in_python_mode_raises(): + # Mode-B query filters must reject the block syntax outright. + with pytest.raises(ValueError): + substitute_variables("{? x IN ({x}) ?}", {"x": ["a"]}, escape="python") + + +# ── extract_model_variables ───────────────────────────────────────────────── + + +def _sql_model(**kw) -> SlayerModel: + kw.setdefault("name", "m") + kw.setdefault("data_source", "ds") + return SlayerModel(**kw) + + +def test_extract_bare_var_without_default_is_required(): + model = _sql_model(sql="SELECT * FROM t WHERE d >= '{d_from}'") + v = extract_model_variables(model) + assert v.required == ["d_from"] + assert v.optional == [] + + +def test_extract_bare_var_with_default_is_optional(): + model = _sql_model( + sql="SELECT * FROM t WHERE region = '{region}'", + query_variables={"region": "EU"}, + ) + v = extract_model_variables(model) + assert v.required == [] + assert v.optional == ["region"] + + +def test_extract_in_block_var_is_optional(): + model = _sql_model(sql="SELECT * FROM t WHERE 1=1 AND {? brand IN ({brand}) ?}") + v = extract_model_variables(model) + assert v.required == [] + assert v.optional == ["brand"] + + +def test_extract_bare_occurrence_wins_over_in_block(): + # A var used bare somewhere AND inside a block classifies as required. + model = _sql_model( + sql="SELECT '{x}' AS a, CASE WHEN 1=1 THEN {? y = {x} ?} END FROM t" + ) + v = extract_model_variables(model) + assert "x" in v.required + assert "x" not in v.optional + + +def test_extract_walks_all_four_mode_a_surfaces(): + model = _sql_model( + sql="SELECT * FROM t WHERE {? a IN ({a}) ?}", + filters=["b >= '{b_from}'"], + columns=[ + Column(name="c1", sql="CASE WHEN {? c IN ({c}) ?} THEN 1 END"), + Column(name="c2", sql="x", filter="d = '{d}'"), + ], + ) + v = extract_model_variables(model) + assert set(v.optional) == {"a", "c"} + assert set(v.required) == {"b_from", "d"} + + +def test_extract_on_malformed_block_does_not_raise(): + # Read-only inspection must never crash on a stored model whose SQL only + # *looks* like a block (a literal '?}' e.g. inside a regex/JSON path). It is + # classified as block-free; execution still raises (DEV-1730 review). + model = _sql_model(sql="SELECT * FROM t WHERE tag ~ 'a?}' AND x = '{y}'") + v = extract_model_variables(model) + assert v.required == ["y"] + + +def test_extract_dedupes_and_sorts(): + model = _sql_model( + sql="WHERE {? a IN ({a}) ?} AND {? a IN ({a}) ?}", + columns=[Column(name="c", sql="z >= '{z_from}'")], + ) + v = extract_model_variables(model) + assert v.optional == ["a"] + assert v.required == ["z_from"] diff --git a/zensical.toml b/zensical.toml index eb151391..256dccbd 100644 --- a/zensical.toml +++ b/zensical.toml @@ -52,6 +52,9 @@ nav = [ { "SLayer vs dbt" = "dbt/slayer_vs_dbt.md" }, { "Importing dbt definitions" = "dbt/dbt_import.md" }, ]}, + { "Cube" = [ + { "Importing Cube definitions" = "cube/cube_import.md" }, + ]}, { "OSI" = [ { "Importing OSI configs" = "osi/osi_import.md" }, ]},